Scene recognition by
inexact graph matching
Celso C. Ribeiro *
Claudia Boeres
Isabelle Bloch
IV ALIO/EURO Workshop on
Applied Combinatorial Optimizatio
Pucón, Chile, November 4-6, 2002
November
2002
1/39 ALIO/EURO
Workshop
Scene recognition by inexact graph matching
Summary
• Motivation and applications
• Optimal graph matching formulation
• Model: objective function and
constraints
• Randomized construction algorithm
• Local search
• GRASP heuristic
• Numerical results
• Current
futureScene
work
November
2/39and
ALIO/EURO
recognition by inexact graph matching
2002
Workshop
Motivation and applications
• Recognition and understanding of
complex scenes require a detailed
description of …
– … the objects involved
– … the spatial relationships between the
objects
• The diversity of the forms of the same
object in different instantiations of a
scene, and also the similarities of
different objects in the same scene,
make
the
relationships
between
November
3/39 ALIO/EURO
Scene recognition by inexact graph matching
2002
Workshop
objects of
prime importance to help
Motivation and applications
• Graph-based representations often
used
for scene representation:
– Nodes represent the objects in the scene.
– Edges represent the relationships
between
the objects.
• Relevant information is extracted from
the scene and represented by
relational attributed graphs.
• Model-based recognition: both the
model and the sceneare
represented
Graph
matching
by graphs.
November
4/39 ALIO/EURO
Scene recognition
by inexact graph matching
problem
2002
Workshop
Motivation and applications
• Matching: graph or subgraph
isomorphism
– The assumption of a bijection between the
elements in two instantiations of the same
scene is too strong for some problems.
– Usually, the model has a schematic aspect.
– Construction of the image graph often
relies on segmentation techniques that
mail fail in accurately representing the
image in meaningful entities, therefore no
isomorphism can be expected.
– Recognition can be better expressed as an
inexact (non-bijective) graph matching
November
5/39 ALIO/EURO
Scene recognition by inexact graph matching
problem.
2002
Workshop
Motivation and applications
• Application:
– Recognition of brain structures from 3D
magnetic resonance images previously
processed by a segmentation model (ENST
and hospital Saint Vincent de Paul, Paris)
– Model is an anatomical atlas: nodes
represent the anatomical structures, while
edges represent the spatial relationships
between them.
– Inaccuracies: oversegmented image,
unexpected objects found (e.g.
pathologies),
expected
objects
not
found,
November
6/39 ALIO/EURO
Scene recognition by inexact graph matching
2002
deformations,
imprecise attributes
Workshop
Motivation and applications
• Axial slices of a brain (each grey level
corresponds to a unique connected
structure): middle dark structures
(lateral ventricles) are much bigger in
(b) than in (a), the white hyper-signal
structure (tumor) in (b) does not
appear in (a) or (c).
(a) Normal
(b) Pathological
(c) Brain
November
7/39 ALIO/EURO
Scene
recognition by inexact graph atlas
matching
brain
brain
2002
Workshop
Motivation and applications
• Other applications:
– Aerial or satellite image interpretation
using a map
– Face recognition
– Character recognition
November
2002
8/39 ALIO/EURO
Workshop
Scene recognition by inexact graph matching
Optimal graph matching
formulation
• Attributed graphs: one single vertex
for each image region
• Typical attributes: adjacencies,
distances, relative positions, grey
levels
• Vertex similarities and edge
similarities are computed from the
values of the attributed graphs,
relating each pair of vertices and each
pair of 9/39
edges
(one from
theby model,
the
November
ALIO/EURO
Scene recognition
inexact graph matching
2002
Workshop
Optimal graph matching
formulation
• Model: G1=(N1,E1)
• Oversegmented image: G2=(N2,E2), with
|N2|>|N1|
• Assign one single node of N1 to each node
1, if node i  N1 is associated with node j  N2
of
xij N2:

0, otherwise
A(i )   j  N2 : xij  1
• Set of nodes associated with i  N1:
November
2002
10/39 ALIO/EURO
Workshop
Scene recognition by inexact graph matching
Optimal graph matching
formulation
• Edges (a,b)  E1 and (p,q)  E2 are
associated if node a  N1 is associated
with p  N2 and node b  N1 is
associated with q  N2.
November
2002
11/39 ALIO/EURO
Workshop
Scene recognition by inexact graph matching
Optimal graph matching
formulation
• Search for the optimal graph matching
which best represents the scene
recognition is based on similarity
values.
November
2002
12/39 ALIO/EURO
Workshop
Scene recognition by inexact graph matching
Optimal graph matching
formulation
• Similarity matrices:
sv(i,j ):
vertex-similarity between vertices i  N1
and j  N2.
se(u,v ):
edge-similarity between edges u = (a,b )
 E1 and v = (c,d )  E2.
• A good matching is a solution in which
the vertex associations and the edge
associations correspond to high
November
13/39 ALIO/EURO
Scene recognition by inexact graph matching
2002
similarityWorkshop
values.
Model: objective function
max f (x ) 

N1 N2
f v (x ) 
node-similarities contribution

f v (x )    1 xij  sv (i , j )
i N1 j N2
f e (x ) 
edge-similarities
contribution
 
(a,b )E1 ( p,q )E2
November
2002

(1  ) e
f (x )
E1 E2
1 maxxap ,xaq ,xbp ,xbq   se ((a,b ),(p,q))
14/39 ALIO/EURO
Workshop
Scene recognition by inexact graph matching

Model: objective function
max f (x ) 

N1 N2
f v (x ) 
node-similarities contribution:
f v (x ) 
(1  ) e
f (x )
E1 E2

1 xij  sv (i , j )
i N1 j N2

High similarity associations are privileged:
v
v
s (i,j ) high  xij = 1 maximizes
1 xij  s (i , j )
Low similarity associations are penalized:
v
v
s (i,j ) low  xij = 0 maximizes
1 xij  s (i , j )
November
2002
15/39 ALIO/EURO
Workshop
Scene recognition by inexact graph matching
Model: constraints
• Each vertex of N2 has to be
associated with exactly one vertex of
N1:
– Image segmentation is performed by an
appropriate algorithm which preserves
the boundaries.
– In consequence, it avoids situations in
which one region of the segmented image
is located in the frontier of two
Constraint
Forof
every
j  Nand
exists
2, there
adjacent (1):
regions
the model
no
1
oneundersegmentation
node i  N1 suchoccurs.
that xij = A
1, i.e.
( j )  1.
November
2002
16/39 ALIO/EURO
Workshop
Scene recognition by inexact graph matching
Model: constraints
• Vertices of N2 associated with the
same vertex of N1 should be all
connected among themselves:
– An oversegmentation method can split an
object into several pieces, but it does
not change their relative positions
(connectivity constraint).
Constraint (2): For every i  N1, the graph in
by A (i ) in G2 = (N2,E2) is connected.
November
2002
17/39 ALIO/EURO
Workshop
Scene recognition by inexact graph matching
Model: constraints
• Pairs of vertices with null similarity
cannot be associated:
Constraint (3): sv(i,j ) = 0  xij = 0.
• All objects in the model should appear
in the image:
Constraint (4): For every i  N1, there
exists at least one node j  N2Asuch
(i ) that
1.
xij = 1, i.e.
November
2002
18/39 ALIO/EURO
Workshop
Scene recognition by inexact graph matching
•
•
Randomized construction
algorithm
MaxTrials: maximum number of attempts
to build a feasible solution.
Repeat for at most MaxTrials attempts:
1. Set A (i ) =  for every i  N1.
2. Randomly select a node j  N2 and set
N2N2 – {j }.
3. Randomly select a node i  N1, until sv(i,j ) >
0 and the graph induced in G2 by A (i )  {j }
is connected.
4. Set A (i )  A (i )  {j }.
5. If N2 ≠  go back to step 2 above.
-1
November
Sceneand
recognition
by(j
inexact
6. If A19/39
(i ALIO/EURO
) ≠  i  N
A
) ≠graph
 matching
j  N2,
1
2002
Workshop
then stop.
Randomized construction
algorithm
• The algorithm stops and returns the
first feasible solution found within at
most MaxTrials attempts.
• The algorithm enforces feasibility, good
solutions are not necessarily obtained.
• It may also be extended to return the
2
2
best among the first MaxSolutions
O N1  N 2
feasible solutions built.
• Each attempt has time complexity
November
20/39 ALIO/EURO
Scene recognition by inexact graph matching
.
2002
Workshop


Randomized construction
algorithm
November
2002
21/39 ALIO/EURO
Workshop
Scene recognition by inexact graph matching
Test problems
• 12 test problems
– Some randomly generated, others from
real problems associated with medical
images
– Two difficult instances with isolated nodes
– Largest instance has |N1|=50 and |E1|=88,
|N2|=250 and |E2|=1681.
• Construction algorithm is very fast and
very often finds a feasible solution in
the first attempt.
•November
Instances
with isolated
nodes are
22/39 ALIO/EURO
Scene recognition by inexact graph matching
2002
Workshop
harder, but
even in these cases a
Test problems: I-6 (human
brain)
November
2002
Model:
Image:
12
95 vertices
vertices
1434 edges
23/39 ALIO/EURO 42
Scene recognition by inexact graph matching
Workshop
edges
Test problems: I-7 (muscle)
Model:
14 vertices
27 edges
November
2002
24/39 ALIO/EURO
Workshop
Image:
28 vertices
63 edges
Scene recognition by inexact graph matching
Test problems: I-7 (muscle)
model
November
2002
25/39 ALIO/EURO
Workshop
oversegmented image
Scene recognition by inexact graph matching
Test problems: I-7 (feasible
solutions)
Best of ten
100
First feasible
first feasible
solution: 4
solutions: 12
5
regions
regions
nodes
correctly
correctly
recognized
recognized
November
2002
26/39 ALIO/EURO
Workshop
Scene recognition by inexact graph matching
Local search
• Solutions built by the construction
algorithm are not necessarily optimal.
• Local search algorithm successively
replaces the current solution by a
better one in a neighborhood of the
first, terminating at a local optimum.
• First improving strategy: the current
solution is replaced by the first
neighbor whose cost function value
improves
of the
solution.
November
27/39that
ALIO/EURO
Scenecurrent
recognition by inexact
graph matching
2002
Workshop
Local search: neighborhoods
• Neighborhood A: all solutions that can
be obtained from the current one by
modifying A -1(j ) for some node j  N2
(the new node associated with j has to
enforce all feasibility constraints).
• Neighborhood B: all solutions that can
be obtained from the current one by
exchanging the nodes
A -1(p) and A 1(q ) of N , currently associated with
1
nodes p28/39
and
q of NScene
2. recognition by inexact graph matching
November
ALIO/EURO
2002
Workshop
Local search: neighborhoods
Neighborhood A: associate a new node (a) with node 2
Neighborhood B: exchange the nodes b and
associated with nodes 4 and 7
November
2002
29/39 ALIO/EURO
Workshop
Scene recognition by inexact graph matching
Local search
•
VND procedure using neighborhoods A
and B and a first improving strategy:
1. Start with a solution built by the
construction algorithm.
2. Move to the first improving solution in
neighborhood A of the current solution
until a local optimum is found.
3. If the current solution is also locally
optimal with respect to neighborhood B,
then stop.
4. Otherwise, move to the first improving
solution in neighborhood B and go back to
November
30/39 ALIO/EURO
Scene recognition by inexact graph matching
step
2.
2002
Workshop
Local search
• Each local search iteration (neighborhood
2
search and move) has time
O Ncomplexity
1  N2
.
• Local search improved the solution value for
all test problems.

November
2002
31/39 ALIO/EURO
Workshop

Scene recognition by inexact graph matching
Local search: I-7
Local search applied
to the best of the 100
first feasible solutions
built: 13 regions correc
recognized (out of 28)
November
2002
32/39 ALIO/EURO
Workshop
Scene recognition by inexact graph matching
GRASP heuristic
• GRASP: multistart metaheuristic
• Each iteration has two phases:
construction and local search
• MaxIterations: maximum number of
iterations
• Repeat for MaxIterations iterations:
– Build a feasible solution using the
randomized construction algorithm.
– Use the local search procedure to improve
the solution built.
– Update the best solution found.
November
2002
33/39 ALIO/EURO
Workshop
Scene recognition by inexact graph matching
GRASP heuristic
• Good solutions obtained with a few
hundred GRASP iterations.
• Improved solutions for all test
problems
• Small computation times
• Problem I-7, 20000 GRASP iterations:
216 seconds on a Pentium II 450 MHz
November
2002
34/39 ALIO/EURO
Workshop
Scene recognition by inexact graph matching
GRASP heuristic
• Solution values for α = 0.9:
construction algorithm with MaxTrials
= 500, first improving local search
strategy, and
200 GRASP
|N1|
|E1|
|N2|
|E2| Constructi Local GRASP
iterations
Proble
m
on
search
I-6
12
42
95
1434
0.412
0.425
0.426
I-7
14
27
28
63
0.618
0.634
0.637
I-8
30
39
100
297
0.498
0.518
0.521
I-9
50
88
250
1681
0.505
0.518
0.519
November
2002
35/39 ALIO/EURO
Workshop
Scene recognition by inexact graph matching
GRASP heuristic: I-7
20000 GRASP
iterations, with
local search
starting from
the first
feasible
solution built in
the
construction
phase: 26
regions
correctly
recognized (out
of 28)
November
2002
36/39 ALIO/EURO
Workshop
Scene recognition by inexact graph matching
GRASP heuristic: 1000
iterations
Problem I-6
Problem I-7
November
2002
37/39 ALIO/EURO
Workshop
Scene recognition by inexact graph matching
Current and future current
work
• Improved construction algorithm
incorporating a greedy (randomized)
insertion criterion
• Improvement of the GRASP heuristic
using path-relinking
• Parallelization on a 64-node Linux
cluster using the MPI library for
communication
• Comparison with other approaches in
the literature, in particular with
November
38/39
ALIO/EURO
Scene recognition by inexact graph matching
genetic
algorithms
2002
Workshop
Slides and publications
• Slides of this talk can be downloaded
from: http://www.inf.pucrio/~celso/talks
• Papers about the randomized
construction heuristic with local search
for scene recognition (shortly) available
at:
(also available: other papers about
http://www.inf.pucGRASP,
rio.br/~celso/publicacoes
parallelization, and path-relinking)
November
2002
39/39 ALIO/EURO
Workshop
Scene recognition by inexact graph matching