Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
From graph matching problem to Assignment
problem :
Application to Content-Based Image Retrieval.
Romain Raveaux
LI lab (EA 6300) – Université de Tours
Rouen, 1rst December, 2013
1 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Graphs are everywhere...
Graphs in Reality
Graphs model objects and
their relationships.
Also referred to as
networks.
All common data
structures can be modeled
as graphs.
2 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Graphs are everywhere...
Graphs in Reality
Graphs model objects and
their relationships.
Also referred to as
networks.
All common data
structures can be modeled
as graphs.
2 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Graphs are everywhere...
Graphs in Reality
Graphs model objects and
their relationships.
Also referred to as
networks.
All common data
structures can be modeled
as graphs.
2 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Definition of a graph
Definition
Let LV and LE denote the set of node and edge labels, respectively.
A labeled graph G is a 4-tuple G = (V , E , µ, ξ) , where
V is the set of nodes,
E ⊆ V × V is the set of edges
µ : V → LV is a function assigning labels to the nodes, and
ξ : E → LE is a function assigning labels to the edges.
Let G1 = (V1 , E1 , µ1 , ξ1 ) be the source graph
And G2 = (V2 , E2 , µ2 , ξ2 ) the target graph
With V1 = (u1 , ..., un ) and V2 = (v1 , ..., vm ) respectively
3 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Definition and notation of a graph
Labels for both nodes and edges can be given by :
Labelling functions
a set of integers L = {1, 2, 3, . . .}
a vector space L = Rn
a finite set of symbolic labels L = {x,y, z, . . .}
...
Definition 1 allows us to handle arbitrarily structured graphs with
unconstrained labelling functions.
4 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Graph comparison
How similar are two graphs?
Graph similarity is the central problem for all learning tasks
such as clustering and classification on graphs.
At least two ways for comparing graphs
Graph Embedding
Graph Matching
5 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Graph comparison
How similar are two graphs?
Graph similarity is the central problem for all learning tasks
such as clustering and classification on graphs.
At least two ways for comparing graphs
Graph Embedding
Graph Matching
5 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Graph comparison
How similar are two graphs?
Graph similarity is the central problem for all learning tasks
such as clustering and classification on graphs.
At least two ways for comparing graphs
Graph Embedding
Graph Matching
5 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Graph isomorphism
Graph isomorphism
Find a mapping f : V1 → V2
i.e. x, y ∈ V1 ⇒ (x, y ) ∈ E1
f is an isomorphism iff (f (x), f (y )) is an edge of G2 .
No polynomial-time algorithm is known for graph isomorphism
Neither it is known to be NP-complete
Subgraph isomorphism
Means finding a subgraph G3 of G2 such that G1 and G3 are
isomorphic.
Subgraph isomorphism is NP-complete
6 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Graph isomorphism
Graph isomorphism
Find a mapping f : V1 → V2
i.e. x, y ∈ V1 ⇒ (x, y ) ∈ E1
f is an isomorphism iff (f (x), f (y )) is an edge of G2 .
No polynomial-time algorithm is known for graph isomorphism
Neither it is known to be NP-complete
Subgraph isomorphism
Means finding a subgraph G3 of G2 such that G1 and G3 are
isomorphic.
Subgraph isomorphism is NP-complete
6 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Graph isomorphism
Graph isomorphism
Find a mapping f : V1 → V2
i.e. x, y ∈ V1 ⇒ (x, y ) ∈ E1
f is an isomorphism iff (f (x), f (y )) is an edge of G2 .
No polynomial-time algorithm is known for graph isomorphism
Neither it is known to be NP-complete
Subgraph isomorphism
Means finding a subgraph G3 of G2 such that G1 and G3 are
isomorphic.
Subgraph isomorphism is NP-complete
6 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Graph isomorphism
Graph isomorphism
Find a mapping f : V1 → V2
i.e. x, y ∈ V1 ⇒ (x, y ) ∈ E1
f is an isomorphism iff (f (x), f (y )) is an edge of G2 .
No polynomial-time algorithm is known for graph isomorphism
Neither it is known to be NP-complete
Subgraph isomorphism
Means finding a subgraph G3 of G2 such that G1 and G3 are
isomorphic.
Subgraph isomorphism is NP-complete
6 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Graph isomorphism
Graph isomorphism
Find a mapping f : V1 → V2
i.e. x, y ∈ V1 ⇒ (x, y ) ∈ E1
f is an isomorphism iff (f (x), f (y )) is an edge of G2 .
No polynomial-time algorithm is known for graph isomorphism
Neither it is known to be NP-complete
Subgraph isomorphism
Means finding a subgraph G3 of G2 such that G1 and G3 are
isomorphic.
Subgraph isomorphism is NP-complete
6 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Graph isomorphism
Graph isomorphism
Find a mapping f : V1 → V2
i.e. x, y ∈ V1 ⇒ (x, y ) ∈ E1
f is an isomorphism iff (f (x), f (y )) is an edge of G2 .
No polynomial-time algorithm is known for graph isomorphism
Neither it is known to be NP-complete
Subgraph isomorphism
Means finding a subgraph G3 of G2 such that G1 and G3 are
isomorphic.
Subgraph isomorphism is NP-complete
6 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Graph isomorphism
Graph isomorphism
Find a mapping f : V1 → V2
i.e. x, y ∈ V1 ⇒ (x, y ) ∈ E1
f is an isomorphism iff (f (x), f (y )) is an edge of G2 .
No polynomial-time algorithm is known for graph isomorphism
Neither it is known to be NP-complete
Subgraph isomorphism
Means finding a subgraph G3 of G2 such that G1 and G3 are
isomorphic.
Subgraph isomorphism is NP-complete
6 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Graph isomorphism
Graph isomorphism
Find a mapping f : V1 → V2
i.e. x, y ∈ V1 ⇒ (x, y ) ∈ E1
f is an isomorphism iff (f (x), f (y )) is an edge of G2 .
No polynomial-time algorithm is known for graph isomorphism
Neither it is known to be NP-complete
Subgraph isomorphism
Means finding a subgraph G3 of G2 such that G1 and G3 are
isomorphic.
Subgraph isomorphism is NP-complete
6 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Graph isomorphism
Graph isomorphism
Find a mapping f : V1 → V2
i.e. x, y ∈ V1 ⇒ (x, y ) ∈ E1
f is an isomorphism iff (f (x), f (y )) is an edge of G2 .
No polynomial-time algorithm is known for graph isomorphism
Neither it is known to be NP-complete
Subgraph isomorphism
Means finding a subgraph G3 of G2 such that G1 and G3 are
isomorphic.
Subgraph isomorphism is NP-complete
6 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Graph isomorphism families
1
Exact Matching : Exact mapping between vertex and/or
edge labels.
2
Inexact Matching : No exact mapping between vertex
and/or edge labels can be found, but when the mapping can
be associated to a cost.
In pattern recognition
Vertices and edges are labeled with measures/features which
may be affected by noise.
Direct and exact comparisons are useless.
The matching problem turns from a decision one to an
optimization one.
7 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Inexact matching
Three sub-problems :
1
Substitution-Tolerant Subgraph Isomorphism
2
Inexact Subgraph Isomorphism
3
Inexact Graph Isomorphism
8 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Substitution-Tolerant Subgraph Isomorphism
1
2
3
This formulation allows differences between labels.
The mapping does not affect the topology.
The mapping should be edge-preserving in one direction.
V E 0.5
0.15 V’ E’ f-­‐1(v’) f(v) f 0.3 0.13 Figure: Substitution-tolerant subgraph isomorphism problem
9 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Inexact Subgraph Isomorphism
1
2
This formulation allows differences between labels.
Topology can be different. The mapping is not
edge-preserving.
V E 0.5 0.16 V’ E’ f Δe’ Matched Unmatched 0.3 f(e) 0.13 Figure: Inexact Subgraph Isomorphism problem
10 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Inexact Graph Isomorphism
1
2
3
Bijective function
This formulation allows differences between labels.
Topology can be different.
V E V’ E’ 0.5 0.16 Δe Δe’ Δv f 0.3 0.13 Figure: Inexact graph isomorphism problem
11 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Graph Matching problems
Figure: Graph isomorphism constraints
12 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Comparison between Classical Graph-Matching Methods
Name
Graph
matching
problem
Models Formulation
Optimality
Complexity
class
Deterministic
Optimization
method
References
Node
Assignment
IGI
Approximate
Sub-optimal
Polynomial
Deterministic
Hungarian algorithm
Neural Network
String
based
methods
Approximated
GED
Genetic algorithm
ISGI
Approximate
Sub-Optimal
Polynomial
Approximate
Sub-Optimal
Polynomial
NonDeterministic
Deterministic
Gradient
ISGI
[Riesen 2009],
[Raveaux 2010],
[Jouili 2009] ,
[Shokoufandeh 2006]
[Kuner 1988],
[Suganthan 2000]
[J. Llados 2001],
[Anjan Dutta 2012]
IGI
Exact
Sub-Optimal
NP-Complete
Deterministic
ISGI
Exact
Sub-Optimal
Polynomial
NonDeterministic
Heuristic Tree
search
Genetic Algorithm
Probabilistic
Relaxation
ISGI
Exact
Sub-Optimal
Polynomial
Deterministic
Probabilistic
Relaxation
Similarity
search
GED
MIGI
Exact
Sub-Optimal
Polynomial
IGI
Exact
Optimal
NP-Complete
NonDeterministic
Deterministic
Greedy search,
Tabu Search
Tree search
GED ILP
IGI
Exact
Optimal
NP-Complete
Deterministic
ILP
STSI
Exact
Optimal
NP-Complete
Deterministic
Indexing
method
Spectral
Theory
Similarity
search
ISGI
Exact
Optimal
NP-Complete
Deterministic
Mathematical
Solver
Mathematical
solver
Indexed search
STSI
Exact
Optimal
Polynomial
Deterministic
MIGI
Exact
Optimal
NP-Complete
Deterministic
Tree Search
Eigen Decomposition
Branch
And
Bound
[Neuhaus M 2006]
[A.D.J. Cross 1997]
,[Ford 1991],
[Jiang 1998]
[W. Christmas 1995],
[Finch 1998],
[Gold 1996]
[Solnon 2005]
[Bunke 1998],
[Bunke 1999]
[Justice D 2006]
[LeBodic 2012]
[Messmer 1998]
[Ferrer 2005] ,
[Umeyama 1988].
[Champin ]
13 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Some highlights
Exact graph matching is useless in many computer vision
applications
Concerning graph matching under noise and distortion
The matching incorporates an error model to identify the
distortions which make one graph a distorted version of the
other
Problems for real world applications
Error-tolerant
To measure the similarity of two graphs.
Runtime may grow exponentially with number of nodes
This is an enormous problem for large datasets of graphs
Wanted: Polynomial-time similarity measure for graphs
14 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Some highlights
Exact graph matching is useless in many computer vision
applications
Concerning graph matching under noise and distortion
The matching incorporates an error model to identify the
distortions which make one graph a distorted version of the
other
Problems for real world applications
Error-tolerant
To measure the similarity of two graphs.
Runtime may grow exponentially with number of nodes
This is an enormous problem for large datasets of graphs
Wanted: Polynomial-time similarity measure for graphs
14 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Some highlights
Exact graph matching is useless in many computer vision
applications
Concerning graph matching under noise and distortion
The matching incorporates an error model to identify the
distortions which make one graph a distorted version of the
other
Problems for real world applications
Error-tolerant
To measure the similarity of two graphs.
Runtime may grow exponentially with number of nodes
This is an enormous problem for large datasets of graphs
Wanted: Polynomial-time similarity measure for graphs
14 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Some highlights
Exact graph matching is useless in many computer vision
applications
Concerning graph matching under noise and distortion
The matching incorporates an error model to identify the
distortions which make one graph a distorted version of the
other
Problems for real world applications
Error-tolerant
To measure the similarity of two graphs.
Runtime may grow exponentially with number of nodes
This is an enormous problem for large datasets of graphs
Wanted: Polynomial-time similarity measure for graphs
14 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Some highlights
Exact graph matching is useless in many computer vision
applications
Concerning graph matching under noise and distortion
The matching incorporates an error model to identify the
distortions which make one graph a distorted version of the
other
Problems for real world applications
Error-tolerant
To measure the similarity of two graphs.
Runtime may grow exponentially with number of nodes
This is an enormous problem for large datasets of graphs
Wanted: Polynomial-time similarity measure for graphs
14 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Some highlights
Exact graph matching is useless in many computer vision
applications
Concerning graph matching under noise and distortion
The matching incorporates an error model to identify the
distortions which make one graph a distorted version of the
other
Problems for real world applications
Error-tolerant
To measure the similarity of two graphs.
Runtime may grow exponentially with number of nodes
This is an enormous problem for large datasets of graphs
Wanted: Polynomial-time similarity measure for graphs
14 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Some highlights
Exact graph matching is useless in many computer vision
applications
Concerning graph matching under noise and distortion
The matching incorporates an error model to identify the
distortions which make one graph a distorted version of the
other
Problems for real world applications
Error-tolerant
To measure the similarity of two graphs.
Runtime may grow exponentially with number of nodes
This is an enormous problem for large datasets of graphs
Wanted: Polynomial-time similarity measure for graphs
14 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Some highlights
Exact graph matching is useless in many computer vision
applications
Concerning graph matching under noise and distortion
The matching incorporates an error model to identify the
distortions which make one graph a distorted version of the
other
Problems for real world applications
Error-tolerant
To measure the similarity of two graphs.
Runtime may grow exponentially with number of nodes
This is an enormous problem for large datasets of graphs
Wanted: Polynomial-time similarity measure for graphs
14 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Problem statement
Graph matching problems
State of the art
Some highlights
Exact graph matching is useless in many computer vision
applications
Concerning graph matching under noise and distortion
The matching incorporates an error model to identify the
distortions which make one graph a distorted version of the
other
Problems for real world applications
Error-tolerant
To measure the similarity of two graphs.
Runtime may grow exponentially with number of nodes
This is an enormous problem for large datasets of graphs
Wanted: Polynomial-time similarity measure for graphs
14 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Definition
Assignment-based Matching
Assignment problem
Jacobi (1890) exposes a polynomial time algorithm to solve the
following problem:
Let aP
i,j be a square matrix nxn, find a permutation σ such
that
ai,σ(i) be maximal.
Such an algorithm was only rediscovered in 1955 par Kuhn,
who called it “Hungarian method”.
It is a well-known problem in mathematical economy, under
the name of “assignment problem”.
15 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Definition
Assignment-based Matching
Assignment problem
Say you have three workers: Jim, Steve and Alan. You need to
have one of them clean the bathroom, another sweep the floors
and the third wash the windows. What’s the best (minimum-cost)
way to assign the jobs? First we need a matrix of the costs of the
workers doing the jobs.
Jim
Steve
Alan
Clean bathroom
1e
3e
3e
Sweep floors
2e
3e
3e
Wash windows
3e
3e
2e
Table: Cost matrix
Then the Hungarian algorithm, when applied to the above table
would give us the minimum cost it can be done with: Jim cleans
the bathroom, Steve sweeps the floors and Alan washes the
windows.
16 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Definition
Assignment-based Matching
Graph comparison through assignment problem
Basic idea:
Methods are based on an optimization procedure mapping
local substructures
Any node(un ) from G1 can be assigned to any node(vm ) of G2 ,
Incurring some cost that depends on the un -vm assignment.
It is required to map all nodes in such a way that the total
cost of the assignment is minimized.
Cost matrix representation (C ):
Cij correspond to the costs of assigning the i th node of G1 to
the j th node of G2 .
17 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Definition
Assignment-based Matching
Graph comparison through assignment problem
Basic idea:
Methods are based on an optimization procedure mapping
local substructures
Any node(un ) from G1 can be assigned to any node(vm ) of G2 ,
Incurring some cost that depends on the un -vm assignment.
It is required to map all nodes in such a way that the total
cost of the assignment is minimized.
Cost matrix representation (C ):
Cij correspond to the costs of assigning the i th node of G1 to
the j th node of G2 .
17 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Definition
Assignment-based Matching
Graph comparison through assignment problem
Basic idea:
Methods are based on an optimization procedure mapping
local substructures
Any node(un ) from G1 can be assigned to any node(vm ) of G2 ,
Incurring some cost that depends on the un -vm assignment.
It is required to map all nodes in such a way that the total
cost of the assignment is minimized.
Cost matrix representation (C ):
Cij correspond to the costs of assigning the i th node of G1 to
the j th node of G2 .
17 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Definition
Assignment-based Matching
Graph comparison through assignment problem
Basic idea:
Methods are based on an optimization procedure mapping
local substructures
Any node(un ) from G1 can be assigned to any node(vm ) of G2 ,
Incurring some cost that depends on the un -vm assignment.
It is required to map all nodes in such a way that the total
cost of the assignment is minimized.
Cost matrix representation (C ):
Cij correspond to the costs of assigning the i th node of G1 to
the j th node of G2 .
17 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Definition
Assignment-based Matching
Graph comparison through assignment problem
Basic idea:
Methods are based on an optimization procedure mapping
local substructures
Any node(un ) from G1 can be assigned to any node(vm ) of G2 ,
Incurring some cost that depends on the un -vm assignment.
It is required to map all nodes in such a way that the total
cost of the assignment is minimized.
Cost matrix representation (C ):
Cij correspond to the costs of assigning the i th node of G1 to
the j th node of G2 .
17 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Definition
Assignment-based Matching
Graph comparison through assignment problem
Basic idea:
Methods are based on an optimization procedure mapping
local substructures
Any node(un ) from G1 can be assigned to any node(vm ) of G2 ,
Incurring some cost that depends on the un -vm assignment.
It is required to map all nodes in such a way that the total
cost of the assignment is minimized.
Cost matrix representation (C ):
Cij correspond to the costs of assigning the i th node of G1 to
the j th node of G2 .
17 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Definition
Assignment-based Matching
Graph comparison through assignment problem
Basic idea:
Methods are based on an optimization procedure mapping
local substructures
Any node(un ) from G1 can be assigned to any node(vm ) of G2 ,
Incurring some cost that depends on the un -vm assignment.
It is required to map all nodes in such a way that the total
cost of the assignment is minimized.
Cost matrix representation (C ):
Cij correspond to the costs of assigning the i th node of G1 to
the j th node of G2 .
17 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Definition
Assignment-based Matching
Graph comparison through assignment problem
Univalent matching
Sets of equal size
Find a bijection b : G1 → G2
Given two sets of subgraphs, G1 and G2 of equal size, the problem
can be expressed as a standard linear program with the objective
function Eq 1 to be minimized.
XX
C (i, j)xij
(1)
i∈G1 j∈G2
The variable xij represents the assignment of a subgraph of G1 to a
subgraph of G2 , taking value 1 if the assignment is done and 0
otherwise.
18 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Definition
Assignment-based Matching
Graph comparison through assignment problem: state of
the art
Node signature
Distance
[Gold 1996]
Node degree+Label
*
[Shokoufandeh 2006]
Eigen vector
L2
[Riesen 2009]
(1)Node+(2)Edge
Edit cost
[Jouili 2009]
Node degree+Weight
L1
[Raveaux 2010]
Sub Graph
Edit Distance
*: Depends on the graph attribute type.
19 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Definition
Assignment-based Matching
Graph comparison through assignment problem: an
example
A subgraph (sg ):
A structure gathering the
edges and their
corresponding ending
vertices from a root
vertex.
Romain Raveaux et al. A
graph matching method
and a graph matching
distance based on
subgraph assignments.
Pattern Recognition
Letters (2010)
20 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Definition
Assignment-based Matching
Graph comparison through assignment problem: an
example
A subgraph (sg ):
A structure gathering the
edges and their
corresponding ending
vertices from a root
vertex.
Romain Raveaux et al. A
graph matching method
and a graph matching
distance based on
subgraph assignments.
Pattern Recognition
Letters (2010)
20 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Definition
Assignment-based Matching
Graph comparison through assignment problem: an
example II
A distance between graph
Subgraph decomposition
Optimization algorithm
21 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Definition
Assignment-based Matching
Graph comparison through assignment problem: an
example II
A distance between graph
Subgraph decomposition
Optimization algorithm
21 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Definition
Assignment-based Matching
Graph comparison through assignment problem: an
example II
A distance between graph
Subgraph decomposition
Optimization algorithm
21 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Definition
Assignment-based Matching
Synthesis
Inexact graph matching problem
Univalent matching
Polynomial algorithm : O(n3 )
Two main directions
Node signature based methods (vectors).
Subgraph based methods. (not a dot in a feature space.)
22 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
CBIR problem
CBIR method
Experiments
How Graph Matching can be used in Pattern Recognition
Problems
Hypothesis :
a graph database consisting of n graphs. D = g 1, g 2, ..., gn
a query graph q
Problem definitions :
1 Full graph search. Find all graphs gi in D s.t. gi is the same
as q.
2 Subgraph search. Find all graphs gi in D containing q or
contained by q.
3 Similarity search. Find all graphs gi in D s.t. gi is similar to
q within a user-specified threshold based on some similarity
measures.
4 Graph mining. Graph mining problem gathers similar graph
or subgraph of D in order to find clusters or prototypes.
23 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
CBIR problem
CBIR method
Experiments
CBIR problem
Figure: Feature clustering stage
24 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
CBIR problem
CBIR method
Experiments
CBIR scheme
Figure: Feature clustering stage
25 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
CBIR problem
CBIR method
Experiments
CBIR scheme
Figure: Images to graphs
26 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
CBIR problem
CBIR method
Experiments
CBIR scheme
Figure: Similarity search
27 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
(a)
CBIR problem
CBIR method
Experiments
(b)
Figure: Regions of Interest found by the SIFT algorithm. Processing
SIFT took 407ms, 60 features were identified and processed
28 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
CBIR problem
CBIR method
Experiments
Figure: A segmentation result. Processing SRM took 1625ms and 26
features were identified and processed
29 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
CBIR problem
CBIR method
Experiments
30 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
CBIR problem
CBIR method
Experiments
Figure: CBIR approach categorization according to the amount of spatial
information captured
31 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
CBIR problem
CBIR method
Experiments
Figure: CBIR taxonomy
32 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
CBIR problem
CBIR method
Experiments
Figure: Image Samples from the Caltech-101 Data set. The 101 object
categories and the background clutter category.
33 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
CBIR problem
CBIR method
Experiments
Figure: Columbia University Image Library
34 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
CBIR problem
CBIR method
Experiments
Recognition rate
(a) Coil-100
(b) Caltech-101
Figure: Comparison between CBIR methods. Summary of results
obtained with the best number of words for each method.
Romain Raveaux et al: Structured representations in a content based image retrieval context. J. Visual
Communication and Image Representation (2013). Elsevier.
35 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
CBIR problem
CBIR method
Experiments
Vocabulary size impact
(a) Coil-100
(b) Caltech-101
Figure: Comparison between the number of words in used by the
methods.
Romain Raveaux et al: Structured representations in a content based image retrieval context. J. Visual
Communication and Image Representation (2013). Elsevier.
36 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
CBIR problem
CBIR method
Experiments
Time experiments
Average response time in seconds
SIFTBoW
IFFSBoW
IFFSGBR
Coil − 100
28
37
1555
Table: Average response time of a query. A speed comparison of the four
methods at the worst case.
37 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Conclusion
To take the stock of :
SIFTBoW is better on Caltech data
IFFS is better on Coil-100
The Graph Based Approach gave similar recognition rate than
BoW methods.
A structural approach requires a fewer number of words to
reach its best performance.
Future work
To avoid visual features clustering.
To enrich the graph representation.
Combination of BoW and GbR : BoW first and then to process
a re-ranking stage with the top k responses based on GbR.
38 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
R.C. Wilson A.D.J. Cross and E.R. Hancock.
Inexact Graph Matching Using Genetic Search.
Pattern Recognition, pages 953–970, 1997.
Josep Llados Anjan Dutta and Umapada Pal.
A symbol spotting approach in graphical documents by
hashing serialized graphs.
Pattern Recognition, pages 752–768, 2012.
Horst Bunke and Kim Shearer.
A graph distance metric based on the maximal common
subgraph.
Pattern Recognition Letters, vol. 2, page 255–259, 1998.
Horst Bunke.
Error Correcting Graph Matching: On the Influence of the
Underlying Cost Function.
38 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, no. 9, pages
917–922, 1999.
Pierre-antoine Champin and Christine Solnon.
Measuring the similarity of labeled graphs.
Miquel Ferrer, Francesc Serratosa and Alberto Sanfeliu.
Synthesis of Median Spectral Graph.
In IbPRIA (2), pages 139–146, 2005.
et al. Finch Wilson.
An energy function and continuous edit process for graph
matching.
Neural Computat, vol. 10, 1998.
G.P. Ford and J. Zhang.
A Structural Graph Matching Approach to Image
Understanding.
38 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Intelligent Robots and Computer Vision X: Algorithms and
Techniques, pages 559–569, 1991.
S. Gold and A Rangarajan.
A graduated assignment algorithm for graph matching.
Workshops SSPR and SPR, pages 377 – 388, 1996.
E. Marti J. Llados and J. J. Villanueva.
Symbol recognition by error-tolerant sub- graph matching
between region adjacency graphs.
Patt. Anal. Mach., pages 1137–1143, 2001.
X. Jiang and H. Bunke.
Marked subgraph isomorphism of ordered graphs.
Workshops SSPR and SPR, pages 122–131, 1998.
Salim Jouili and Salvatore Tabbone.
Graph Matching Based on Node Signatures.
In GbRPR, pages 154–163, 2009.
38 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Hero A Justice D.
A binary linear programming formulation of the graph edit
distance.
IEEE Trans Pattern Anal Mach Intell., vol. 28, page
1200–1214, 2006.
P Kuner and B Ueberreiter.
Pattern Recognition by Graph Matching: Combinatorial versus
Continuous Optimization.
International Journal in Pattern Recognition and Artificial
Intelligence, vol. 2, page 527–542, 1988.
P. LeBodic and H. Heroux.
An integer linear program for substitution-tolerant subgraph
isomorphism and its use for symbol spotting in technical
drawings.
Pattern Recognition, pages 4214–4224, 2012.
38 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
B. Messmer and Bunke.
Error-Correcting Graph Isomorphism using Decision Trees.
Int. Journal of Pattern Recognition and Ar, vol. 12, page
721–742, 1998.
Bunke H Neuhaus M.
A convolution edit kernel for error-tolerant graph matching.
Proceedings of IEEE international conference on pattern
recognition., vol. 30, pages 220–223, 2006.
Burie J. Raveaux R. and Ogier.
A graph matching method and a graph matching distance
based on subgraph assignments.
Pattern Recognition Letters, page 394–406, 2010.
Kaspar Riesen and Horst Bunke.
Approximate graph edit distance computation by means of
bipartite graph matching.
38 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
Image Vision Comput., vol. 27, no. 7, pages 950–959, 2009.
Bretzner L. Macrini D. Fatih Demirci M. Jönsson C.
Shokoufandeh A. and S. Dickinson.
The representation and matching of categorical shape.
Computer Vision and Image Understanding, page 139–154,
2006.
Christine Solnon.
Reactive Tabu Search for Measuring Graph.
pages 172–182, 2005.
P. N. Suganthan.
Attributed relational graph matching by neural-gas networks.
IEEE Signal Processing Society Workshop on Neural Networks
for Signal Processing X, pages 366–374, 2000.
S. Umeyama.
38 / 38
Graphs in pattern recognition
Assignment problem in graph matching
Application to CBIR
Conclusion
An eigendecomposition approach to weighted graph matching
problems.
IEEE Pattern Anal. Mach. Intel, page 695–703, 1988.
J. Kittler W. Christmas and M. Petrou.
Structural matching in computer vision using probabilistic
relaxation.
IEEE Trans. PAMI,, vol. 2, page 749–764, 1995.
38 / 38