Image Segmentation
Based on the work of Shi and Malik, Carnegie Mellon and Berkley
and based on the presentation of Jianbo Shi
© 2004 by Davi Geiger
Computer Vision
March 2004
L1.1
Edge-based image segmentation
• Edge detection by gradient operators
• Linking by dynamic programming, voting, relaxation, …
- Natural for encoding curvilinear grouping
- Hard decisions often made prematurely
© 2004 by Davi Geiger
Computer Vision
March 2004
L1.2
Grouping with Bayesian Statistics
Bayes data structure = data generation model + segmentation model
min X E( X ; f )  min X [ log p( f | X )  log p( X )]
Image as
observation f
f1
f2
Texture models
Grouping as
state X
© 2004 by Davi Geiger
X1
X2
Computer Vision
Segmentation is to find a
partitioning of an image, with
generative models explaining
each partition.
Generative models constrain
the observation data, f, and the
prior model constrains the
discrete states, X.
The solution sought is the most
probable state, or the state of
the lowest energy.
March 2004
L1.3
Image segmentation by pairwise similarities
•
•
•
•
Image = { pixels }
Segmentation = partition of image into
segments
Similarity between pixels i and j
Sij = Sji ≥ 0
Sij
Objective: “similar pixels, with large
value of Sij, should be in the same
segment, dissimilar pixels should be
in different segments”
© 2004 by Davi Geiger
Computer Vision
March 2004
L1.4
Relational Graphs

G=(V, E, S)
V: each node denotes a pixel
 E: each edge denotes a pixel-pixel relationship
 S: each edge weight measures pairwise similarity


Segmentation = node partitioning

break V into disjoint sets V1 , V2
© 2004 by Davi Geiger
Computer Vision
March 2004
L1.5
Solving MRF by Graph Partitioning
Some simple MRF models can be translated into graph partitioning
min E ( X ; f ) 
 W
p,q ( X p ,
Xq) 
p qN ( p )
pair relationships
L1
© 2004 by Davi Geiger
Computer Vision
U
p (X p ,
fp)
p
data measures
L2
March 2004
L1.6
Weighted graph partitioning
i
Pixels i I = vertices of graph G
Edges ij = pixel pairs with Sij > 0
Sij
j
i
Similarity matrix S = [ Sij ]
di = Sj Є G Sij
degree of I
A
deg A = Si Є A di degree ofA
G
A
Assoc(A,B) = Si Є A Sj Є B Sij
© 2004 by Davi Geiger
Computer Vision
B
March 2004
L1.7
Cuts in a Graph
•
(edge) cut = set of edges whose removal makes a graph disconnected
•
weight of a cut:
•
the normalized cut
cut( A, B ) = Si
NCut( A,B ) = cut(A, B)(
•
Є A,
Sj Є
B
Sij =Assoc(A,B)
1
1
+
)
deg A deg B
Normalized Cut criteria: minimum cut(A,Ā)
1
Sij 
d ( xi , x j )
© 2004 by Davi Geiger
Computer Vision
March 2004
L1.8
Grouping with Spectral Graph Partitioning
SGP: data structure = a weighted graph, weights describing data affinity
cut ( A, B) cut ( A, B)
min Ncut ( A, B) 

deg( A)
deg( B)
cut ( A, B)   S (i, j )
iA jB
deg( A)   S (i, j )
iA jG

Segmentation is to find a node
partitioning of a relational graph, with
minimum total cut-off affinity.
Discriminative models are used to
evaluate the weights between nodes.
The solution sought is the cuts of the
minimum energy.
NP-Hard!
© 2004 by Davi Geiger
Computer Vision
March 2004
L1.9
Normalized Cut and Normalized Association
cut ( A, B) cut ( A, B)
Ncut ( A, B) 

deg( A)
deg( B)
Nassoc ( A, B) 
Assoc ( A, A) Assoc ( B, B)

deg( A)
deg( B)
Ncut ( A, B)  2  Nassoc ( A, B)
as deg( A)  Assoc ( A, B)  Assoc ( A, A)
•
Minimizing similarity between the groups, and maximizing similarity within
the groups are achieved simultaneously.
© 2004 by Davi Geiger
Computer Vision
March 2004
L1.10
Some definitions
Let S be the similarity matrix, S (i, j )  Si , j ;
Let D be the diag. matrix, D(i, i )   j S (i, j );
Let x be a vector in {1,1}N , x(i )  1  i  A.
•
Rewriting Normalized Cut in matrix form:
Ncut (A, B) 
cut (A, B) cut (A, B)

deg( A)
deg( B)
(1  x)T ( D  S )(1  x) (1  x)T ( D  S )(1  x)


; k
T
T
k1 D1
(1  k )1 D1
 D(i, i)
 D(i, i)
xi  0
i
 ...
© 2004 by Davi Geiger
Computer Vision
March 2004
L1.11
Generalized Eigenvalue problem
•
after simplification, we get
yT ( D  S ) y
T
Ncut ( A, B)
,
with
y

{
1
,

b
},
y
D1  0.
i
T
y Dy
y2i
A
© 2004 by Davi Geiger
i
( D  S ) x  Dx
y2i
i
A
Computer Vision
March 2004
L1.12
© 2004 by Davi Geiger
Computer Vision
March 2004
L1.13
Brightness Image Segmentation
© 2004 by Davi Geiger
Computer Vision
March 2004
L1.14
Brightness Image Segmentation
© 2004 by Davi Geiger
Computer Vision
March 2004
L1.15
© 2004 by Davi Geiger
Computer Vision
March 2004
L1.16
Results on color segmentation
© 2004 by Davi Geiger
Computer Vision
March 2004
L1.17
Motion Segmentation with Normalized Cuts
• Networks of spatial-temporal connections:
 Motion “proto-volume” in space-time
© 2004 by Davi Geiger
Computer Vision
March 2004
L1.18
© 2004 by Davi Geiger
Computer Vision
March 2004
L1.19