1
Image Segmentation
Cuts, Random Walks, and Phase-Space Embedding
Jianbo Shi
Robotics Institute
Carnegie Mellon University
Joint work with: Malik,Malia,Yu
Taxonomy of Vision Problems
 Reconstruction:
– estimate parameters of external 3D world.
 Visual Control:
– visually guided locomotion and manipulation.
 Segmentation:
– partition I(x,y,t) into subsets of separate objects.
 Recognition:
– classes: face vs. non-face,
– activities: gesture, expression.
2
3
We see Objects
4
Outline





Problem formulation
Normalized Cut criterion & algorithm
The Markov random walks view of Normalized Cut
Combining pair-wise attraction & repulsion
Conclusions
5
Edge-based image segmentation
 Edge detection by gradient operators
 Linking by dynamic programming, voting, relaxation, …
Montanari 71, Parent&Zucker 89, Guy&Medioni 96, Shaashua&Ullman 88
Williams&Jacobs 95, Geiger&Kumaran 96, Heitger&von der Heydt 93
-
Natural for encoding curvilinear grouping
Hard decisions often made prematurely
6
Region-based image segmentation
 Region growing, split-and-merge, etc...
- Regions provide closure for free, however,
approaches are ad-hoc.
 Global criterion for image segmentation
• Markov Random Fields e.g. Geman&Geman 84
• Variational approaches e.g. Mumford&Shah 89
• Expectation-Maximization e.g. Ayer&Sawhney 95, Weiss 97
- Global method, but computational complexity precludes exact MAP
estimation
- Problems due to local minima
7
8
Bottom line:
It is hard, nothing worked well, use edge
detection, or just avoid it.
Global good, local bad.
9
 Global decision good, local bad
– Formulate as hierarchical graph partitioning
 Efficient computation
– Draw on ideas from spectral graph theory to define an
eigenvalue problem which can be solved for finding
segmentation.
 Develop suitable encoding of visual cues in terms
of graph weights.
Shi&Malik,97
Image segmentation by pairwise similarities
 Image = { pixels }
 Segmentation = partition of
image into segments
 Similarity between pixels i
and j
Sij = Sji 0
Sij
 Objective: “similar pixels should be in the same
segment, dissimilar pixels should be in different
segments”
10
Segmentation as weighted graph partitioning
Pixels i I = vertices of graph G
Edges ij = pixel pairs with Sij > 0
Similarity matrix S = [ Sij ]
is generalized adjacency matrix
di = Sj Sij
i
i
degree of i
vol A = SiA di volume of A  I
A
Sij
11
j
Cuts in a graph
 (edge) cut = set of edges whose removal makes a
graph disconnected
 weight of a cut
cut( A, B ) = Si  A,j B Sij
 the normalized cut
1 .
1 .
NCut( A,B ) = cut( A,B )(vol A + vol B )
12
Normalized Cut and Normalized Association
Ncut (A, B) 
cut (A, B) cut (A, B)

Vol(A)
Vol ( B)
assoc(A,A) assoc (B, B)
Nassoc(A, B) 

Vol(A)
Vol ( B)
Ncut (A, B) 2  Nassoc (A, B)
 Minimizing similarity between the groups, and maximizing similarity
within the groups can be achieved simultaneously.
13
The Normalized Cut (NCut) criterion
 Criterion
min NCut( A,A )
Small cut between subsets of ~ balanced grouping
NP-Hard!
14
Some definitions
15
Let W be the associatio n matrix, W (i, j )  wi , j ;
Let D be the diag. matrix, D(i, i )   j W (i, j );
Let x be a vector in {1,1}N , x(i )  1  i  A.
Normalized Cut As Generalized Eigenvalue problem
 Rewriting Normalized Cut in matrix form:
Ncut (A, B) 
cut (A, B) cut (A, B)

Vol (A)
Vol (B)
(1  x)T ( D  W )(1  x) (1  x)T ( D  W )(1  x)


; k
T
T
k1 D1
(1  k )1 D1
 ...
 D(i, i)
 D(i, i)
xi  0
i
16
More math…
17
Normalized Cut As Generalized Eigenvalue problem
Ncut (A, B) 
18
cut (A, B) cut (A, B)

Vol (A)
Vol (B)
D(i, i )

(1  x)T ( D  W )(1  x) (1  x)T ( D  W )(1  x)
xi  0


; k
T
T
k1 D1
(1  k )1 D1
i D(i, i)
 ...
 after simplification, we get
yT ( D  W ) y
T
Ncut ( A, B)
,
with
y

{
1
,

b
},
y
D1  0.
i
T
y Dy
y2i
A
i
( D  W ) x  Dx
y2i
A
i
Interpretation as a Dynamical System
19
Interpretation as a Dynamical System
20
Brightness Image Segmentation
21
brightness image segmentation
22
Results on color segmentation
23
Malik,Belongie,Shi,Leung,99
24
25
Motion Segmentation with Normalized Cuts
 Networks of spatial-temporal connections:
26
Motion Segmentation with Normalized Cuts
 Motion “proto-volume” in space-time
 Group correspondence
27
Results
28
Results
Shi&Malik,98
29
Results
30
Results
31
Stereoscopic data
32
Conclusion I
 Global is good, local is bad
– Formulated Ncut grouping criterion
– Efficient Ncut algorithm using generalized eigen-system
 Local pair-wise allows easy encoding and combining
of Gestalt grouping cues
33
FAQs
 Why is this segmentation criterion better?
– 30%
 Is there a good way of picking W(i,j)
– 40%
 How do we integrate prior information?
how is higher level information encoded?
- 20%
34
Learning Segmentation
 Learning Segmentation with Random Walk
Maila & Shi, NIPS ‘00
35
Goals of this work
36
 Better understand why spectral segmentation works
– random walks view for NCut algorithm
– complete characterization of the “ideal” case
 ideal case is more realistic/general than previously thought
 Learning feature combination/object shape model
– Max cross-entropy method for learning
Malia&Shi,00
The random walks view
 Construct the matrix
d1
D=
d2
...
P = D-1S
S =
dn
S11 S12 S1n
S21 S22 S2n
...
Sn1 Sn2 Snn
 P is stochastic matrix Sj Pij = 1
 P is transition matrix of Markov chain with state space I
p=
1 . [ d d . . . d ]T is stationary distribution
1 2
n
vol I
37
Reinterpreting the NCut criterion
NCut( A, A ) = PAA + PAA
PAB = Pr[ A --> B | A ] under P, p
 NCut looks for sets that “trap” the random walk
 Related to Cheeger constant, conductivity in Markov
chains
38
Reinterpreting the NCut algorithm
(D-W)y = mDy
m1=0 m2
y1
y2
...
mn
...
Yn
Px =
1=1 2
x1
mk
x2
39
x
...
...
n
xn
= 1 - k
yk = xk
The NCut algorithm segments based on the second largest eigenvector of P
So far...
 We showed that the NCut criterion & its approximation the
NCut algorithm have simple interpretations in the Markov
chain framework
– criterion - finds “almost ergodic” sets
– algorithm - uses x2 to segment
Now:
 Will use Markov chain view to show when the NCut
algorithm is exact, i.e. when P has K piecewise constant
eigenvectors
40
Piecewise constant eigenvectors: Examples
 Block diagonal P (and S)
S
P
Eigenvalues
 Equal rows in each block
Eigenvalues
Eigenvectors
Eigenvectors
41
Piecewise constant eigenvectors: general case
 Theorem[Meila&Shi] Let P = D-1S with D non-
singular and let D be a partition of I. Then P has K
piecewise constant eigenvectors w.r.t D iff P is
block stochastic w.r.t D and P non-singular.
Eigenvectors
P
Eigenvalues
42
Block stochastic matrices
D = ( A1, A2, . . . AK )
partition of I
 P is block stochastic w.r.t D
SjAs Pij = SjAs Pi’j
for i,i’ in same segment As’
 Intuitively: Markov chain can be aggregated so that
random walk over D is Markov
 P = transition matrix of Markov chain over D
43
Learning image segmentation
Image
fijq
44
Learning
Model

normalize
Sij
Pij
Pij*
Segmentation
 Targets Pij* =
0,
1 .
00,
|A|
A
j A
j
for i in segment A
 Model Sij = exp( Sq qfqij )
The objective function
45
J = - Si I 1 .Sj I Pij* log Pij
|I|
J = KL( P* || P )
where p0 = 1 .and Pi j* = p0 Pij* the flow i j
|I|
 Max entropy formulation maxPij H( j | i )
s.t. <fijq>p0Pij = <fijq>
Convex problem with convex constraints
J
 The gradient 
q
= <fijq>
p0Pij*
for all q
at most one optimum
q>
<f
ij
p0Pij *
p0Pij
Experiments - the features
46
i
 IC - intervening contour
k
fijIC = max Edge( k )
k
Edge( k ) = output of edge filter for pixel k
Discourages transitions across edges
 CL - colinearity/cocircularity
fijCL =
j
(i,j)
2-cos(2ai)-cos(2aj)
1 - cos(al)
+
Edgeness
2-cos(2ai + aj)
1 - cos(a0)
Encourages transitions along flow lines
 Random features
orientation
ai
i
j
aj
Training examples
IC
47
CL
Test examples
Original Image
Edge Map
48
Segmentation
Conclusion II
49
 Showed that the NCut segmentation method has a probabilistic interpretation
Graph Cuts
Generalized Eigen-system
Markov Random Walks
 Introduced
– a principled method for combining features in image segmentation
– supervised learning and synthetic data to find the optimal combination of
features
50
f i ,ICj  max kl (i , j ) Edge(k )
f i ,CLj 
2  cos( 2a i )  cos( 2a j )
1  cos(a l )

2  cos( 2a i  2a j )
1  cos(a o )
51
Si , j  e
IC
CL f iCL


f
,j
IC i , j
Summary
 Grouping is a global process
– Normalized Cuts formalism provides efficient algorithm
based on spectral graph theory
 Grouping is from multiple cues
 Learning Segmentation with Random Walks
52