Tutorial
Graph Based Image Segmentation
Jianbo Shi, David Martin, Charless Fowlkes, Eitan Sharon
Topics
• Computing segmentation with graph cuts
• Segmentation benchmark, evaluation criteria
• Image segmentation cues, and combination
• Muti-grid computation, and cue aggregation
i
Wij
j
Wij
G = {V, E}
V: graph nodes
E: edges connection nodes
Image = { pixels }
Pixel similarity
Topics
• Computing segmentation with graph cuts
• Segmentation benchmark, evaluation criteria
• Image segmentation cues, and combination
• Muti-grid computation, and cue aggregation
Topics
• Computing segmentation with graph cuts
• Segmentation benchmark, evaluation criteria
• Image segmentation cues, and combination
• Muti-grid computation, and cue aggregation
Brightnes
s
Texture
Original Image
Colo
r
L*
Region
Processing
Boundary Processing
Textons
Proximity
D
C
E
A
B
χ2
χ2
Wij
C
A
B
a*
b*
Topics
• Computing segmentation with graph cuts
• Segmentation benchmark, evaluation criteria
• Image segmentation cues, and combination
• Muti-grid computation, and cue aggregation
Part I: Graph and Images
Jianbo Shi
Graph Based Image Segmentation
i
Wij
j
Wij
G = {V, E}
V: graph nodes
E: edges connection nodes
Image = { pixels }
Pixel similarity
Segmentation = Graph partition
Right partition cost function?
Efficient optimization algorithm?
Graph Terminology
adjacency matrix,
degree,
volume,
graph cuts
Graph Terminology
i
Sij
Similarity matrix S = [ Sij ]
j
is generalized adjacency matrix
1
0.9
20
20
0.8
40
0.7
10
0.6
60
0.5
0
80
0.4
0.3
100
!10
0.2
120
0.1
!20
!10
0
10
20
30
40
140
20
40
60
80
100
120
140
0
Graph Terminology
Degree of node:
di =
!
i
Sij
j
1
0.9
20
20
0.8
40
0.7
10
0.6
60
0.5
0
80
0.4
0.3
100
!10
0.2
120
0.1
!20
!10
0
10
20
30
40
140
20
40
60
80
100
120
140
0
Graph Terminology
Volume of set:
vol(A) =
!
di , A ⊆ V
A
i∈A
1
0.9
20
20
0.8
40
0.7
10
0.6
60
0.5
0
80
0.4
0.3
100
!10
0.2
120
0.1
!20
!10
0
10
20
30
40
140
20
40
60
80
100
120
140
0
Cuts in a graph
cut(A, Ā) =
!
Si,j
i∈A,j∈Ā
1
0.9
20
20
0.8
40
0.7
10
0.6
60
0.5
0
80
0.4
0.3
100
!10
0.2
120
0.1
!20
!10
0
10
20
30
40
140
20
40
60
80
100
120
140
0
Graph Terminology
Similarity matrix S = [ Sij ]
i
Sij
j
Volume of set:
A
Degree of node:
di =
i
Graph Cuts
!
j
Sij
Useful Graph Algorithms
• Minimal Spanning Tree
• Shortest path
• s-t Max. graph flow, Min. cut
Minimal/Maximal Spanning Tree
Tree is a graph G without cycle
Graph
Maximal
Minimal
Kruskal’s algorithm
• sort the edges of G in increasing order by length
• for each edge e in sorted order
if the endpoints of e are disconnected in S
add e to S
Randonmalized version can compute Typical cuts
Leakage problem in MST
Leakage
Graph Cut and Flow
Source
Sink
1) Given a source (s) and a sink node (t)
2) Define Capacity on each edge, C_ij = W_ij
3) Find the maximum flow from s->t, satisfying the capacity constraints
Min. Cut = Max. Flow
Problem with min cuts
Min. cuts favors isolated clusters
Normalize cuts in a graph
• (edge) Ncut = balanced cut
1
1
N cut(A, B) = cut(A, B)(
+
)
vol(A) vol(B)
NP-Hard!
Representation
Partition matrix:
segments
X = [X1 , ..., XK ]
pixels
Pair-wise similarity matrix W
Degree matrix D:
D(i, i) =
!
j
Laplacian matrix D-W
Wi,j
Graph weight matrix W
(b)
(a)
W
i1
n=nr * nc
nr
i2
nc
W(i1,j)
(c)
W(i2,j)
n=nr * nc
(d)
Laplacian matrix D-W
Let x = X(1,:) be the indicator of group 1
asso(A, A) = xT W x
xT
W
x
1
0.9
20
0.8
40
0.7
0.6
60
0.5
80
0.4
0.3
100
0.2
120
0.1
140
20
40
60
80
100
120
140
0
Laplacian matrix D-W
T
T
x Wx
x Dx
Cut(A,V-A) =
vol(A)
-
asso(A,A)
1
0.9
20
0.8
40
0.7
0.6
60
0.5
80
0.4
0.3
100
0.2
120
0.1
140
20
40
60
80
100
120
140
0
Cut(A, V − A) = xT (D − W )x
K
!
1
cut(Vl , V − Vl )
N cut(X) =
K
vol(Vl )
l=1
1
0.9
20
0.8
40
0.7
0.6
60
0.5
80
0.4
0.3
100
0.2
120
0.1
140
20
40
60
80
100
120
140
0
K
!
1
XlT (D − W )Xl
=
T DX
K
X
l
l
l=1
X ∈ {0, 1}N ×K , X1K = 1N
Step I: Find Continuous Global Optima
Scaled partition matrix.



X=


1
1
0
0
0
0
0
1
1
0
0
0
0
0
1






− 12
T
Z = X(X DX)





Z=




1
vol(A)
√ 1
vol(A)
0
√
0
0
0
0
0
0
1
vol(B)
√ 1
vol(B)
0
√
0
0
√
1
vol(C)










Step I: Find Continuous Global Optima
K
!
1
XlT (D − W )Xl
=
T DX
K
X
l
l
l=1
Ncut
becomes
1
N cut(Z) = tr(Z T W Z)
K
Eigensolutions
(D − W )z = λDz
∗
Z =
∗
[z1∗ , z2∗ , ..., zk∗ ]
∗
Z T DZ = IK
y2i
i
A
y2i
A
i
Interpretation as a Dynamical System
Interpretation as a Dynamical System
Step I: Find Continuous Global Optima
Partition



X=


1
1
0
0
0
0
0
1
1
0
0
0
0
0
1






Scaled Partition





Z=




1
vol(A)
√ 1
vol(A)
0
0
0
0
1
vol(B)
√ 1
vol(B)
0
√
0
0
0
√
0
0
√
1
vol(C)
Eigenvector solution















∗
Z =




1
vol(A)
√ 1
vol(A)
0
0
0
0
1
vol(B)
√ 1
vol(B)
0
√
0
0
0
√
0
0
√
1
vol(C)



 '

 × cos(θ)

sin(θ)



-sin(θ)
cos(θ)
(D − W )Z ∗ = λDZ ∗
T
If Z* is an optimal, so is {ZR : R R = IK }
(
.
Step II: Discretize Continuous Optima
Target
partition
Z2
Z1



X=


1
1
0
0
0
0
0
1
1
1






Rotation R


Eigenvector Z ∗ = 


solution

1
1
1
1
1
-1.4
-1.3
0.8
0.9
0.7
Rotation R can be found exactly in 2-way partition






Brightness Image
Segmentation
brightness image
segmentation
Part II:
Segmentation Measurement,
Benchmark