A practical approach to the
development of ontology-based
information fusion systems
Juan Gómez-Romero, Miguel A. Serrano,
Jesús García, Miguel Á. Patricio, José M. Molina
Applied Artificial Intelligence Group
University Carlos III of Madrid, Spain
Summary
• Ontologies
– What are ontologies?
– Why should we care?
– How can they be exploited?
– Are there any successful experience?
– How can we contribute?
Outline
• Motivation
• Knowledge representation and reasoning
with ontologies
• Ontologies for HLIF in the maritime
domain
• Proposed architecture
• Implementation
• Discussion, conclusions and future work
Outline
> Motivation
• Knowledge representation and reasoning
with ontologies
• Ontologies for HLIF in the maritime
domain
• Proposed architecture
• Implementation
• Discussion, conclusions and future work
1. Motivation
• Information Fusion
– "theories and methods to effectively combine
data from multiple sensors and related
information to achieve more specific
inferences that could be achieved by using a
single, independent sensor." (Llinas and Hall,
2009)
1. Motivation
• Low-level data fusion
– To process multi-sensor signals to
estimate objects properties
• Tracking: data acquisition, collection, spatial
and temporal alignment
• Video-based tracking: estimate the kinetics of
scene objects in a video sequence
– Surveillance
1. Motivation
• High-level information fusion
– To obtain a symbolic description of
the qualitative relations between the
objects in the scenario
• Actions, intentions, threats, etc. > Situation
assessment
1. Motivation
• High-level information fusion
– Understand the scene, evaluate
threats, support decision making
– Purely numerical techniques are
insufficient
• Cognitive abilities
• Complex and unpredictable world behavior
1. Motivation
• High-level information fusion
– Flexible and dynamic situation models
– Context exploitation
– Symbolic formalisms to represent and
reason with high-level information: abstract
scene objects and relations
Outline
• Motivation
> Knowledge representation and
reasoning with ontologies
• Ontologies for HLIF in the maritime
domain
• Proposed architecture
• Implementation
• Discussion, conclusions and future work
2. Knowledge representation and
reasoning with ontologies
• Ontologies
– Knowledge model that describes the
objects in a domain by means of a
language that can be automatically
processed
• Description Logics (DLs) representation
• Proposed to be the language for metadata
representation in the Semantic Web
2. Knowledge representation
and reasoning with ontologies
• Advantages
– Interpretability
• High level symbolic description
– Interoperability
• Agreed representation of fusion information
– Scalability
• Promote extension and reuse
– Formal
• Reasoning with logic-based formalisms
– Tools
• OWL standard, reasoning engines, programming interfaces, …
2. Knowledge representation
and reasoning with ontologies
• Representation primitives
– Concepts
• Vessel, HarborZone
– Relations
• hasFlag, insideOf
– Instances
• vessel_1, nearShoreZone
– Axioms
• Vessel and (hasFlag some (Flag and (belongsTo
some AlliedCountry)) subclassOf FriendVessel
• transitive(insideOf)
• (vessel_1, nearShoreZone: insideOf)
2. Knowledge representation
and reasoning with ontologies
2. Knowledge representation
and reasoning with ontologies
• Reasoning
– Satisfiability / consistency
• An axiom is satisfiable if it is not a contradiction to the remaining
axioms
– Subsumption
• A (super-) concept includes a (sub-) concept
– Equivalence
• Two concepts include the same instances
– Disjointness
• Two concepts do not have any common instance
– Instance checking
• An instance belongs to a class
Outline
• Motivation
• Knowledge representation and reasoning
with ontologies
> Ontologies for HLIF in the maritime
domain
• Proposed architecture
• Implementation
• Discussion, conclusions and future work
3. Ontologies for HLIF in the
Maritime Domain
• Situation and threat assessment in the harbor
surveillance scenario
– Detected and estimated vessel information from VTS
• Position, AIS identification
– Context knowledge
• Restrictions to the fusion process
– Is the situation plausible?
• Enrich available information
– Link to external information sources
– Normalcy models
• Harbor navigation restrictions
• Expert knowledge about threats
3. Ontologies for HLIF in the
Maritime Domain
• Knowledge representation
– Terminological knowledge base to describe harbor
elements, actors and context
• Concepts, relations, axioms (GCIs)
– Geometrical elements of harbor
– Vessel classification
– Rules of operation
– Assertional knowledge base to represent current
instances of the harbor entities and relevant contextual
information
• Instance axioms
– Static
– Dynamic
isA
divided_into
adjacent_to
delimited_by
extends_to
partially_overlaps
extends_from
isA
isA
isA
isA
isA
isA
3. Ontologies for HLIF in the Maritime Domain
Terminological Knowledge Base: Harbor areas
3. Ontologies for HLIF in the
Maritime Domain
• Qualitative spatial knowledge management
– Zone boundaries and vessel position are
represented according to their relative situation
– RCC (Region Connection Calculus)
• Logic theory for qualitative spatial knowledge
representation and reasoning
– disconnected, externally connected, partial overlap, tangential
inner part, etc.
• Cannot be fully represented with ontologies
• Supported by reasoning engines
3. Ontologies for HLIF in the
Maritime Domain
• Assertional knowledge base (factual)
– Individuals representing current instantiation of the
terminological model
• Scene interpretation is a model-construction
procedure
• Instances representing more abstract entities are inferred
from instances representing concrete measures
> From sensor-based data to situation assessment
3. Ontologies for HLIF in the
Maritime Domain
Individual: vessel1
Types:
PowerDrivenVessel
Facts:
inside_of
middle_harbour,
has_property vessel_size
Individual: vessel1_size
Types:
Size
Facts:
size vessel_size_value
Individual: vessel_size_value
Types:
AbsoluteFloatValue
Facts:
val 25.0f
3. Ontologies for HLIF in the
Maritime Domain
• Rules of operation
• Any vessel inside an area with restricted speed
moving at a speed under the speed limit is
normal
– Classes describing the normal behavior
– Instance checking can be used to classify
a vessel as threatening or non-threatening
according to the “normal behavior” classes
3. Ontologies for HLIF in the
Maritime Domain
3. Ontologies for HLIF in the
Maritime Domain
• Rules of operation
– Vessels that do not accomplish any normalcy rule
are not classified as non-threatening
• It is easy to describe normal scenarios according to
harbor rules
• Better supported by ontologies that abnormalcy models
– Open World Assumption: the knowledge in an ontology is
incomplete
» Default reasoning is not performed
• Any entity not classified as normal remains in an
unknown state
3. Ontologies for HLIF in the
Maritime Domain
Individual: vessel1
Types:
PowerDrivenVessel
Facts:
inside_of
middle_harbour,
has_property vessel_size
has_property vessel_speed
Individual: vessel1_speed
Types:
Speed
Facts:
speed vessel_speed_value
Individual: vessel_speed_value
Types:
AbsoluteFloatValue
Facts:
val 4.0f
reasoner
Individual: vessel1
Types:
PowerDrivenVessel,
NoSpeedViolation, SafeVessel
Facts:
inside_of
middle_harbour,
has_property vessel_size
has_property vessel_speed
Individual: vessel1_speed
Types:
Speed
Facts:
speed vessel_speed_value
Individual: vessel_speed_value
Types:
AbsoluteFloatValue
Facts:
val 4.0f
3. Ontologies for HLIF in the
Maritime Domain Trigger abductive
Individual: vessel1
Types:
PowerDrivenVessel
Facts:
inside_of
middle_harbour,
has_property vessel_size
has_property vessel_speed
Individual: vessel1_speed
Types:
Speed
Facts:
speed vessel_speed_value
Individual: vessel_speed_value
Types:
AbsoluteFloatValue
Facts:
val 6.0f
reasoner
Individual: vessel1
Types:
PowerDrivenVessel,
owl:Thing
Facts:
inside_of
middle_harbour,
has_property vessel_size
has_property vessel_speed
Individual: vessel1_speed
Types:
Speed
Facts:
speed vessel_speed_value
Individual: vessel_speed_value
Types:
AbsoluteFloatValue
Facts:
val 6.0f
reasoning
3. Ontologies for HLIF in the
Maritime Domain
• Abductive reasoning
– Takes a set of facts as inputs and finds a suitable
hypothesis that explains them
• See whether inconsistency is result of low-quality
observations, or this vessel exhibits a possible threatening
behavior
– Increase threat level
– Not directly supported by ontologies
• Monotonic formalisms –do not allow adding or retracting
knowledge while reasoning
– Reasoning engines include extensions to allow
abductive rules (not uncertain)
3. Ontologies for HLIF in the
Maritime Domain
• Reasoning under uncertainty
– Additional reasoning layer
• BAS (Belief Argumentation System)
– Combination of symbolic logic and belief theory
» Compute beliefs supporting or rejecting a
hypothesis (e.g., vessel features or spatiotemporal relations with other vessels and zones)
• Probabilistic ontologies
– Reduction to Bayesian inference
Outline
• Motivation
• Knowledge representation and reasoning
with ontologies
• Ontologies for HLIF in the maritime
domain
> Proposed architecture
• Implementation
• Discussion, conclusions and future work
4. Architecture
Common-sense knowledge
Expert knowledge
Threats and assessment
Scene Objects
Tracking and Sensor Data
Spatial
Reasoning
interpretations
Activities and Situations
recommendations
Recommendations
Heuristics
Ontology
Reasoning
Scene model
Multi-source tracking
Multi-camera
Radar
AIS
...
...
High-level decisionready knowledge
Outline
• Motivation
• Knowledge representation and reasoning
with ontologies
• Ontologies for HLIF in the maritime
domain
• Proposed architecture
> Implementation
• Discussion, conclusions and future work
5. Implementation
• Two layers
– General tracking layer
• Numerical measures (Desnsity functions,
movement vectors)
– Ontology-based contextual layer
• Tracking representation
• Contextual data
• Symbolic reasoning
5. Implementation
• Tracking layer
–
–
–
–
Four modules that run in sequence
Each module has a set of interchangable algorithms
Input: Frames
Output: Track and features
5. Implementation
• Association module
– Fuzzy Region Assignment (FRA)
• Low level granularity, image segmentation operations
• Medium level granularity, smoothness criteria on target
features
• High level granularity, constraints on tracking continuity
5. Implementation
– Fuzzy Region Assignment (FRA)
• Bayesian formulation to determine when a blob is related
with a track
• Heuristic function to update the track situation and
dimensions
• Fuzzy rules derived from experimentation to define the
final group
5. Implementation
• Context layer
– Set of ontologies organized according to the JDL model
•
•
•
•
Tracking entities (TREN)
Scene object (SCOB) e.g. Restricted area, Person, Vessel
Activities (ACTV) e.g. Threatening behaviours
Situation assessment (IMPC) e.g Emergency
– Inputs managed through the OWL 2 API
• Context knowledge given by users or previous executions
• Sensor data (Video tracking)
– Ontologies are instanced in the RACER reasoner
• Abductive nRQL rules and independent RCC implementation
– Scene interpretation and recommendations as output
5. Implementation
• Context layer and mass storage
– Timestamps / snapshots allow temporal dimension
• “A vessel stopped in a restricted area during the last ten
intervals”
– Delays in the overall system
• Query search through a larger number of axioms
– Compromise between data storage and query
performance
– Temporal window
5. Implementation
• Scalability - Dynamic RCC
– Aims
• Representation and reasoning with qualitative spatial properties
• Efficient update of the spatial properties of the objects
– Architecture
• Knowledge base (SCOB spatial features)
• Optimized geometric model: Geometric model (JTS with
OpenGIS) and a data structure
• RCC implementation to store the qualitative spatial
relationships (RACER substrate)
5. Implementation
• What is the problem?
– A full topological analysis has a
quadratic complexity
– Choose only candidate geometries
that could modify the spatial
relations
• How?
– Querying the auxiliary data structure
• Advantages
– Topological relations of a geometry
is obtained by checking only a few
geometries
5. Implementation
• Video examples
– Scene annotation
– Object identification
– Tracking enhancement
– Scene recognition
http://www.giaa.inf.uc3m.es/miembros/jgomez/et/
Outline
• Motivation
• Knowledge representation and reasoning with
ontologies
• Ontologies for HLIF in the maritime domain
• Proposed architecture
• Implementation
> Discussion, conclusions and future work
6. Discussion, conclusions
and future work
• Ontologies for high-level fusion
– Cognitive model
• Symbolic representation of the world
– Formal knowledge model
• Representation
• Reasoning
• Ontologies in the maritime domain
– Heterogeneous information
• Vessel Traffic Systems, AIS
• Security protocols
• Harbor areas and navigation restrictions
6. Discussion, conclusions
and future work
• Ontological model
– Categorization of interesting entities
• Vessels (Temporal properties)
• Harbor areas (Spatial features)
– Normalcy models
• Normal categories of behaviors
– Abduction and uncertainty management
• Extended rule-based reasoning
• Belief-based argumentation (BAS), Bayesian networks
6. Discussion, conclusions
and future work
• Architecture and implementation
– Low-level fusion layer
• Tracker
– High-level fusion layer
• Cognitive scene model
– Reasoner
– Spatial module
Video-surveillance applications
6. Discussion, conclusions
and future work
• Future work
– Practical implementation in real domains (harbor!)
• Multiple information sources
– Acquisition
– Integration
• Expert knowledge
– Acquisition
– Representation
• Real-time demands
• High reliability
– Specific features
• Uncertain and imprecise knowledge
• Interfacing with human users
Questions, comments?
Juan Gómez-Romero, Miguel A. Serrano,
Jesús García, Miguel Á. Patricio, José M. Molina
Applied Artificial Intelligence Group
University Carlos III of Madrid, Spain