ITI-CERTH
Centre for Research and Technology Hellas (CERTH)
Informatics and Telematics Institute (ITI) (http://www.iti.gr)
Director Prof. Michael Strintzis
Multimedia Knowledge Group (http://mkg.iti.gr)
Head Dr. Yiannis Kompatsiaris
Thessaloniki, GREECE
Amsterdam, Multimedia Semantics XG, 10-11 July 2006
Vasileios Papastathis, Research Fellow
Informatics and Telematics Institute
Centre for Research and Technology Hellas
Presentation Overview
• ITI Multimedia Knowledge Group
• Towards a Common Multimedia Ontology
Framework
• IST BOEMIE Project
Informatics and Telematics Institute
Centre for Research and Technology Hellas
2
ITI Multimedia Knowledge Group
•
•
•
•
•
Semantic Multimedia Analysis
Multimedia Indexing and Retrieval
Multimedia and the Semantic Web
Knowledge Structures, Languages and Tools for Multimedia
Reasoning and Personalization for Multimedia Applications
• Ontology structures in OWL for knowledge-assisted multimedia
analysis and reasoning
• Rules in SWRL including multimedia features for analysis and
reasoning
• MPEG-7 Visual part representation in RDF
• Graphical tool for ontology-based annotation of images (MOntomat)
Informatics and Telematics Institute
Centre for Research and Technology Hellas
3
R&D Projects
• “aceMedia: Integrating knowledge, semantics and content for
•
•
•
•
•
•
user centred intelligent media services”, IP 2004-2007
“KnowledgeWeb: Realizing the Semantic Web”, funded by the
DG XIII, NoE 2004-2007
“MESH: Multimedia Semantic Syndication for Enhanced News
Services”, IST – IP, 2006-2008
“K-Space: Knowledge Space of Semantic Inference for
Automatic Annotation and Retrieval of Multimedia Content, IST NoE, 2006-2009
“X-Media: Knowledge Sharing and Reuse Across Media”, IST –
IP, 2006-2009
“BOEMIE: Bootstrapping Ontology Evolution with Multimedia
Information Extraction”, IST – STREP, 2006-2008
“PATExpert: Advanced Patent Document Processing
Techniques”, IST-STREP, 2006-2008 (collaboration with AUTH)
Informatics and Telematics Institute
Centre for Research and Technology Hellas
4
Towards a common Multimedia
Ontology Framework
Requirements and
Harmonization approaches
Informatics and Telematics Institute
Centre for Research and Technology Hellas
Multimedia Ontology Framework
• Framework for collecting existing multimedia
knowledge representation
• Define approach for harmonization
• Initiated by aceMedia in EWIMT 2005 but not
directly connected to aceMedia work
• http://www.acemedia.org/aceMedia/reference
/multimedia_ontology/index.html
Informatics and Telematics Institute
Centre for Research and Technology Hellas
6
List of contributors
1.
2.
3.
4.
5.
6.
7.
8.
National Technical University of Athens
Istituto per la Matematica Applicata e le Tecnologie
Informatiche, Consiglio Nazionale delle Ricerche,
AIM@SHAPE
University Rennes
Centre for Digital Music, Queen Mary, University of London
Lehrgebiet Multimedia und Internetanwendungen,
Informatikzentrum
University of Southampton, UK
Institute of Informatics & Telecommunications, N.C.S.R.
"Demokritos"
Institute of Information Systems & Information Management,
JOANNEUM RESEARCH Forschungsgesellschaft mbH
Informatics and Telematics Institute
Centre for Research and Technology Hellas
7
List of contributors (2)
9. ISWeb, University of Koblenz
10. Freie Universitaet,Netzbasierte Informationssysteme, Institut
11.
12.
13.
14.
15.
fuer Informatik
'Signals and Images' Laboratory, Institute of Information
Science and Technologies "A. Faedo" (ISTI–CNR), Italian
National Research Council (CNR)
Image Bioinformatics Research Group, Department of
Zoology, University of Oxford
Technical University of Crete,
CWI Amsterdam, Maryland Information and Network Dynamics
Lab, National Technical University of Athens
University of Twente
Informatics and Telematics Institute
Centre for Research and Technology Hellas
8
Requirements for MM ontologies
• Scope and Usage
• Media Description-Description of the
•
•
•
•
•
information object
Representation of Content Structure
Representation of Content Description
Knowledge Representation Formalism
Reasoning Support
Following an established methodology
Informatics and Telematics Institute
Centre for Research and Technology Hellas
9
Requirements (2)
• Scope and Usage






• Media description
Annotation
Analysis
Retrieval
Reasoning
Personalized filtering
Meta-Modeling
Informatics and Telematics Institute
Centre for Research and Technology Hellas






Image
Video
3D Graphics
Audio
AudioVisual
Multimedia
presentation
10
Requirements (3)
• Representation of
• Representation of
content structure



Multimedia structure
representation
Spatiotemporal
relation
representation
Localization
Informatics and Telematics Institute
Centre for Research and Technology Hellas
content description




Low-level description
Multiple audiovisual
descriptors
Alignment to existing
metadata standards
Basic and complex
data types
11
Requirements (4)
• Knowledge representation
formalism
 High complexity of
multimedia modeling
 Compliance to W3C
standards
 Inferencing support
 OWL
• Reasoning support




DL-based reasoning
Rules
Uncertainty
Fuzziness
• Established
methodology


Informatics and Telematics Institute
Centre for Research and Technology Hellas
OntoKnowledge
OntoClean
12
Harmonization approaches (1)
• Modularization


Clear, independent ontology modules
Examples: Visual Descriptors Ontology (VDO), Multimedia
Structure Ontology (MSO), Spatio-Temporal Ontology (STO)
• Ontology cross-domain linking


Alignment techniques for multimedia ontologies linking with
domain-specific ontologies
Merging techniques for multimedia ontology coordination
• Compliance to standards

Semantic Web, W3C, MPEG-7, etc
Informatics and Telematics Institute
Centre for Research and Technology Hellas
13
Harmonization approaches (2)
• Harmonization of core ontologies


Core ontologies cannot be simply mapped but need
to be “harmonized”
Need to apply ontological modeling principles during
harmonization
• Development of a common upper ontology




Starting point for new ontology construction
Bridge between existing ontologies
Act as the ground basis for the establishment of an
integrated ontology framework
Back up the common upper ontology
Informatics and Telematics Institute
Centre for Research and Technology Hellas
14
BOEMIE
Multimedia annotation tool and
A proposed knowledge framework
Informatics and Telematics Institute
Centre for Research and Technology Hellas
M-OntoMat Annotizer
Selection of regions for
merging
Semantic
labeling
Region
merging
functionality
Mid-Level
Concepts
Descriptor
extraction
Automatic
segmentation
Informatics and Telematics Institute
Centre for Research and Technology Hellas
16
XML output
Informatics and Telematics Institute
Centre for Research and Technology Hellas
17
Knowledge framework
Core Ontology-Holistic Representation
Domain Ontology
Spatiotemporal
Constraints
Modality Elements
Multimedia Content
Ontology
Features/ Multimedia
Descriptor Ontology
Classifiers/ Algorithms
Informatics and Telematics Institute
Centre for Research and Technology Hellas
18
An indicative example
High_Jumper_Approach
High_Jumper_Jump_Off
Athletics_Event
Athletics_Event_Inst_1
High_Jumper_Successful _Landin
Jumping_Event
High_Jumper_Failed_Landing
High_
Jump_
Event
Long_Jump_
Event
Pole
Mats
Sandpit
Pole_Vault
_Event
Pole_Vaulter_Jump_Off
Pole_Vaulter_Swing_and_Row
Foam
Mats
Pole_Vaulter_Successful_Landin
Pole_Vaulter_Failed_Landing
Foam_Mats
Horizontal_Bar_Inst_1
Foam_Mats_Inst_1
Horizontal
_Bar
is_Right
is_Below
is_On
Element
Visual
Element
Textual
Element
Horizontal_Bar_ve
Horizontal_Bar_ve_Inst1
Foam_Mats_ve
Sandpit_ve
Multimedia
object
Image
depicts
Foam_Mats_ve_Inst1
Video
Image_Inst1
Video
Still region
segment
Still_region_Inst1
Still_region_Inst2
depicts
Informatics and Telematics Institute
Centre for Research and Technology Hellas
19