S
T
A
N
F
O
R
D
Relaxations and Moves for
MAP Estimation in MRFs
M. Pawan Kumar
QuickTi me™ and a
TIFF ( Uncompressed) decompressor
are needed to see thi s pi ctur e.
Vladimir Kolmogorov
Quic kTime™ and a
TIFF (Unc ompres sed) dec ompres sor
are needed to see this pic ture.
Philip Torr
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Daphne Koller
Our Problem
Label l2
2
0
1
Label l1
5
v1
2
1
0
0
4
2
v2
6
4
3
1
1
3
3
1
0
v3
Random Variables V = {v1, ... ,v4}
Label Set L = {l1, l2}
Labeling f: V  L (shown in red)
7
v4
Our Problem
Label l2
2
0
1
Label l1
5
v1
2
1
0
0
4
2
v2
6
4
3
1
1
3
3
1
0
v3
7
v4
Random Variables V = {v1, ... ,v4}
Label Set L = {l1, l2}
Labeling f: V  L (shown in red)
Energy of Labeling E(f) = 13 (shown in green)
Our Problem
Label l2
2
0
1
Label l1
5
v1
2
1
0
0
4
2
v2
6
4
3
1
1
3
3
1
0
v3
7
v4
Find f* = argminf E(f)
Arbitrary topology, discrete label set, potentials (NP-hard)
Pairwise energy function: unary and pairwise potentials
(still NP-hard)
Outline
• Convex Relaxations
–
–
–
–
–
Integer Programming Formulation
LP Relaxation
SDP Relaxation
SOCP Relaxation
Comparing Relaxations
• Move Making Algorithms
• Some Interesting Open Problems
Integer Programming Formulation
Unary Potentials
Label l2
2
0
1
Label l1
Labeling f shown in red
Unary Potential u = [ 5
5
3
0
v1
2
; 2
Cost of
Cost
v1 =
of1v1 = 2
4
2
v2
4 ]
Integer Programming Formulation
Unary Potentials
Label l2
2
0
1
Label l1
Labeling f shown in red
5
4
3
0
v1
2
v2
Unary Potential u = [ 5
2
; 2
4 ]
Label vector x = [ -1
1
; 1 -1 ]T
2 the optimal x
v1 is vto
11 =
Recall that the aim
find
Integer Programming Formulation
Unary Potentials
Label l2
2
0
1
Label l1
Labeling f shown in red
5
4
3
0
2
v1
v2
Unary Potential u = [ 5
2
; 2
4 ]
Label vector x = [ -1
1
; 1 -1 ]T
Sum of Unary Potentials = 1 ∑i ui (1 + xi)
2
Integer Programming Formulation
Pairwise Potentials
Label l2
2
0
1
Label l1
Labeling f shown in red
5
v1
4
3
0
2
v2
Pairwise Potential P
0
0
3
Cost of v1 = 1 and v1 = 1
0 0
1
0
Cost of v1 = 1 and v2 = 1
0 1
0 0
Cost of v1 = 1 and v2 = 2
3 0
0 0
0
Integer Programming Formulation
Pairwise Potentials
Label l2
2
0
1
Label l1
Labeling f shown in red
Pairwise Potential P
0
0
0
3
0 0
1
0
0 1
0 0
3 0
0 0
5
v1
4
3
0
2
v2
Sum of Pairwise Potentials
1 ∑ P (1 + x )(1+x )
ij ij
i
j
4
Integer Programming Formulation
Pairwise Potentials
Label l2
2
0
1
Label l1
Labeling f shown in red
Pairwise Potential P
0
0
0
3
0 0
1
0
0 1
0 0
3 0
0 0
5
4
3
0
v1
2
v2
Sum of Pairwise Potentials
1 ∑ P (1 + x +x + x x )
ij ij
i
j
i j
4
= 1 ∑ij Pij (1 + xi + xj + Xij)
4
X = x xT
Xij = xi xj
Integer Programming Formulation
Constraints
• Integer Constraints
xi {-1,1}
X = x xT
• Uniqueness Constraint
∑ xi = 2 - |L|
i  va
Integer Programming Formulation
x* = argmin
1 ∑ u (1 + x ) + 1 ∑ P (1 + x + x + X )
ij
i
j
ij
i
i
4
2
∑ xi = 2 - |L|
i  va
Convex
xi {-1,1}
X = x xT
Non-Convex
Outline
• Convex Relaxations
–
–
–
–
–
Integer Programming Formulation
LP Relaxation
SDP Relaxation
SOCP Relaxation
Comparing Relaxations
• Move Making Algorithms
• Some Interesting Open Problems
LP Relaxation
Schlesinger, 1976
Retain Convex Part
x* = argmin
1 ∑ u (1 + x ) + 1 ∑ P (1 + x + x + X )
ij
i
j
ij
i
i
4
2
∑ xi = 2 - |L|
i  va
xi {-1,1}
X = x xT
Relax Non-Convex
Constraint
LP Relaxation
Schlesinger, 1976
Retain Convex Part
x* = argmin
1 ∑ u (1 + x ) + 1 ∑ P (1 + x + x + X )
ij
i
j
ij
i
i
4
2
∑ xi = 2 - |L|
i  va
xi [-1,1]
X = x xT
Relax Non-Convex
Constraint
LP Relaxation
Schlesinger, 1976
X = x xT
Xij [-1,1]
1 + xi + xj + Xij ≥ 0
∑ Xij = (2 - |L|) xi
j  vb
LP Relaxation
Schlesinger, 1976
Retain Convex Part
x* = argmin
1 ∑ u (1 + x ) + 1 ∑ P (1 + x + x + X )
ij
i
j
ij
i
i
4
2
∑ xi = 2 - |L|
i  va
xi [-1,1]
X = x xT
Relax Non-Convex
Constraint
LP Relaxation
Schlesinger, 1976
Retain Convex Part
x* = argmin
1 ∑ u (1 + x ) + 1 ∑ P (1 + x + x + X )
ij
i
j
ij
i
i
4
2
∑ xi = 2 - |L|
i  va
xi [-1,1],
Xij [-1,1]
1 + xi + xj + Xij ≥ 0
∑ Xij = (2 - |L|) xi
j  vb
Outline
• Convex Relaxations
–
–
–
–
–
Integer Programming Formulation
LP Relaxation
SDP Relaxation
SOCP Relaxation
Comparing Relaxations
• Move Making Algorithms
• Some Interesting Open Problems
SDP Relaxation
Lasserre, 2000
Retain Convex Part
x* = argmin
1 ∑ u (1 + x ) + 1 ∑ P (1 + x + x + X )
ij
i
j
ij
i
i
4
2
∑ xi = 2 - |L|
i  va
xi {-1,1}
X = x xT
Relax Non-Convex
Constraint
SDP Relaxation
Lasserre, 2000
Retain Convex Part
x* = argmin
1 ∑ u (1 + x ) + 1 ∑ P (1 + x + x + X )
ij
i
j
ij
i
i
4
2
∑ xi = 2 - |L|
i  va
xi [-1,1]
X = x xT
Relax Non-Convex
Constraint
SDP Relaxation
1
x1
x2
1 x1 x2
...
xn
..
.
=
xn
1
xT
x
X
Xii = 1
Convex
Non-Convex
Positive Semidefinite
Rank = 1
SDP Relaxation
1
x1
x2
1 x1 x2
...
xn
..
.
=
xn
1
xT
x
X
Xii = 1
Convex
Positive Semidefinite
SDP Relaxation
Lasserre, 2000
Retain Convex Part
x* = argmin
1 ∑ u (1 + x ) + 1 ∑ P (1 + x + x + X )
ij
i
j
ij
i
i
4
2
∑ xi = 2 - |L|
i  va
xi [-1,1]
X = x xT
Relax Non-Convex
Constraint
SDP Relaxation
Lasserre, 2000
Retain Convex Part
x* = argmin
1 ∑ u (1 + x ) + 1 ∑ P (1 + x + x + X )
ij
i
j
ij
i
i
4
2
∑ xi = 2 - |L|
i  va
xi [-1,1]
Xii = 1
Accurate
X - xxT
Positive
Semidefinite
0
Inefficient
Outline
• Convex Relaxations
–
–
–
–
–
Integer Programming Formulation
LP Relaxation
SDP Relaxation
SOCP Relaxation
Comparing Relaxations
• Move Making Algorithms
• Some Interesting Open Problems
SOCP Relaxation
Derive SOCP relaxation from the SDP relaxation
x* = argmin
1 ∑ u (1 + x ) + 1 ∑ P (1 + x + x + X )
ij
i
j
ij
i
i
4
2
∑ xi = 2 - |L|
i  va
xi [-1,1]
Xii = 1
X - xxT
0
Further Relaxation
SOCP Relaxation
Kim and Kojima, 2000
Choose a sub-graph G
Variables xG and XG
Choose a matrix C1 = UUT
0
C1 (XG - xGxGT) ≥ 0
Choose a matrix C2 = UUT
REPEAT
0
Outline
• Convex Relaxations
–
–
–
–
–
Integer Programming Formulation
LP Relaxation
SDP Relaxation
SOCP Relaxation
Comparing Relaxations
• Move Making Algorithms
• Some Interesting Open Problems
Dominating Relaxation
≥
A
B
For all MAP Estimation problem (u, P)
A
dominates
B
Dominating relaxations are better
SOCP Relaxation
Kim and Kojima, 2000
Choose a sub-graph G
Variables xG and XG
Choose a matrix C1 = UUT
0
C1 (XG - xGxGT) ≥ 0
If G is a tree, LP dominates SOCP
Examples
Muramatsu and Suzuki, 2003
(MAXCUT)
Ravikumar and Lafferty, 2006
(QP Relaxation)
Kumar, Torr and Zisserman, 2006
(Equivalent SOCP Relaxation)
SOCP Relaxation
Kim and Kojima, 2000
Choose a sub-graph G
Variables xG and XG
Choose a matrix C1 = UUT
0
C1 (XG - xGxGT) ≥ 0
If G is a cycle with non-negative P
SOCP Relaxation
Kim and Kojima, 2000
Choose a sub-graph G
Variables xG and XG
Choose a matrix C1 = UUT
0
C1 (XG - xGxGT) ≥ 0
If G is an even cycle with non-positive P
SOCP Relaxation
Kim and Kojima, 2000
Choose a sub-graph G
Variables xG and XG
Choose a matrix C1 = UUT
0
C1 (XG - xGxGT) ≥ 0
If G is an odd cycle with 1 non-positive P
SOCP Relaxation
Kumar, Kolmogorov and Torr, 2007
What about other cycles?
Dominated by linear cycle inequalities
Cliques?
Dominated by clique inequalities
Outline
• Convex Relaxations
• Move Making Algorithms
– State of the Art
– Comparison with LP Relaxation
– Improved Moves
• Some Interesting Open Problems
MRFs in Vision
Pab(i,k)
lk
li
ua(i) va
Pab(i,k) = wab min{ d(i-k), M }
wab is non-negative
vb ub(k) d(.) is a semi-metric distance
Truncated Linear
Truncated Quadratic
Move Making
Current Solution
Search
Neighbourhood
Energy
Optimal Move
Solution Space
Slide courtesy of Pushmeet Kohli
Outline
• Convex Relaxations
• Move Making Algorithms
– State of the Art
– Comparison with LP Relaxation
– Improved Moves
• Some Interesting Open Problems
Expansion Move
Variables take label  or retain current label
Slide courtesy of Pushmeet Kohli
Boykov, Veksler, Zabih 2001
Expansion Move
Variables take label  or retain current label
Status:
InitializeSky
Expand
Ground
House
with Tree
Slide courtesy of Pushmeet Kohli
Tree
Ground
House
Sky
Boykov,
Veksler,
ZabihZabih]
2001
[Boykov,
Veksler,
Outline
• Convex Relaxations
• Move Making Algorithms
– State of the Art
– Comparison with LP Relaxation
– Improved Moves
• Some Interesting Open Problems
Multiplicative Bounds
Expansion Bounds as bad as ICM Bounds
LP
MoveMaking
Potts
2
2
Truncated
Linear
2 + √2
2M
Truncated
Quadratic
O(√M)
2M
Metric
Labeling
O(log h)
2M
Outline
• Convex Relaxations
• Move Making Algorithms
– State of the Art
– Comparison with LP Relaxation
– Improved Moves
• Some Interesting Open Problems
Randomized Rounding
yi = (1 + xi)/2
y’i = y0 + y1 + … + yi
0
y’0
y’i
y’k
Choose an interval of length L’
y’h = 1
Randomized Rounding
yi = (1 + xi)/2
y’i = y0 + y1 + … + yi
r
0
y’0
y’i
y’k
y’h = 1
Generate a random number r  (0,1]
Randomized Rounding
yi = (1 + xi)/2
y’i = y0 + y1 + … + yi
r
0
y’0
y’i
y’k
y’h = 1
Assign label next to r (if within the interval)
Move Making
• Initialize the labeling
• Choose interval I of L’ labels
• Each variable can
• Retain old label
• Choose a label from I
• Choose best labeling
va
vb
Non-submodular move?
Iterate over intervals
Submodular overestimation
Truncated Convex Models
Pab(i,k) = wab min{ d(i-k), M }
d(.) is convex
Truncated Linear
d(x+1) - 2d(x) + d(x-1) ≥ 0
Truncated Quadratic
Move Making
• Choose interval I of L’ labels
• Each variable can
• Retain old label
• Choose a label from I
• Choose best labeling
Large L’ => Non-submodular
va
vb
Move Making
va
vb
Submodular problem
Move Making
va
vb
Non-submodular
Problem
Move Making
va
vb
Submodular problem
Ishikawa, 2003; Veksler, 2007
Move Making
s
va
am+1
bm+1
am+2
bm+2
am+2
bm+2
an
bn
vb
t
Move Making
LP Bounds
Kumar and Torr, NIPS 08
Kumar and Koller, UAI 09
Type of Problem
Bound
Potts
2
Truncated Linear
2 + √2
Truncated Quadratic
O(√M)
Metric Labeling
O(log h)
Outline
• Convex Relaxations
• Move Making Algorithms
• Some Interesting Open Problems
Problem 1
Relationship between rounding and move-making?
What happens when n < h ??
(Should we even use move-making here??)
What about semi-metric MRFs??
Problem 2
Graph-cuts based image segmentation
Vicente, Kolmogorov, Rother, 2008
Problem 2
Image segmentation with connectivity prior
Vicente, Kolmogorov, Rother, 2008
Problem 2++
Kumar and Koller, 20??
Questions??
http://ai.stanford.edu/~pawan