Energy Minimization
M. Pawan Kumar
http://www.robots.ox.ac.uk/~oval/
Slides available online http://mpawankumar.info
Energy Minimization
Va
Vb
Vc
Vd
Energy Minimization
Label l1
Label l0
2
4
6
3
5
2
Vb
3
7
Va
Q(f;) = ∑a a(f(a))
Vc
Vd
Energy Minimization
Label l1
2
0
1
Label l0
5
Va
1
0
0
4
2
Vb
2
6
3
1
1
4
3
Vc
3
1
0
7
Vd
Q(f;) = ∑a a(f(a)) +∑(a,b) ab(f(a),f(b))
Energy Minimization
Label l1
2
0
1
Label l0
5
Va
1
0
0
4
2
Vb
2
6
3
1
1
4
3
Vc
3
1
0
7
Vd
Q(f;) = ∑a a(f(a)) +∑(a,b) ab(f(a),f(b))
Energy Minimization
Label l1
2
0
1
Label l0
5
Va
1
0
0
4
2
Vb
2
6
3
1
1
4
3
Vc
3
1
0
7
Vd
Q(f;) = ∑a a(f(a)) +∑(a,b) ab(f(a),f(b))
2 + 1 + 2 + 1 + 3 + 1 + 3 = 13
Energy Minimization
Label l1
2
0
1
Label l0
5
Va
1
0
0
4
2
Vb
2
6
3
1
1
4
3
Vc
3
1
0
7
Vd
Q(f;) = ∑a a(f(a)) +∑(a,b) ab(f(a),f(b))
Energy Minimization
Label l1
2
0
1
Label l0
5
Va
1
0
0
4
2
Vb
2
6
3
1
1
4
3
Vc
3
1
0
7
Vd
Q(f;) = ∑a a(f(a)) +∑(a,b) ab(f(a),f(b))
5 + 1 + 4 + 0 + 6 + 4 + 7 = 27
Energy Minimization
Label l1
2
0
1
Label l0
5
Va
1
0
0
4
2
Vb
2
6
3
1
1
4
3
Vc
3
1
0
7
Vd
No loops
Q(f;) = ∑a a(f(a)) +∑(a,b) ab(f(a),f(b))
q* = min Q(f; ) = Q(f*; )
Solve using dynamic programming
Energy Minimization
Q(f;) = ∑a a(f(a)) +∑(a,b) ab(f(a),f(b))
q* = min Q(f; ) = Q(f*; )
In general, NP-hard
Outline
• Minimum Cut Problem
• Two-Label Submodular Energy Functions
• Move-Making Algorithms
Cut
D = (N, A)
10
n1
3
n3
n2
2
5
n4
Let N1 and N2 such that
• N1 “union” N2 = N
• N1 “intersection” N2 = Φ
C is a set of arcs such that
• (n1,n2)  A
• n1  N1
• n2  N2
C is a cut in the digraph D
Cut
D = (N, A)
N1
10
n1
3
n3
What is C?
n2
{(n1,n2),(n1,n4)} ?
2
5
N2
{(n1,n4),(n3,n2)} ?
n4
✓
{(n1,n4)} ?
Weight of a Cut
D = (N, A)
10
n1
3
n3
n2
Sum of length of all
arcs in C
2
5
n4
Weight of a Cut
D = (N, A)
N1
10
n1
3
n3
What is w(C)?
n2
2
5
N2
n4
3
st-Cut
D = (N, A)
s
1
2
10
n1
3
n3
7
n2
A sink “t”
n4
C is a cut such that
• s  N1
• t  N2
2
5
t
A source “s”
3
C is an st-cut
Weight of an st-Cut
D = (N, A)
s
1
2
10
n1
3
n3
7
n2
w(C) = Σ(n1,n2)  C l(n1,n2)
2
5
t
n4
3
Weight of an st-Cut
D = (N, A)
s
1
2
10
n1
3
n3
7
n2
2
5
t
n4
3
What is w(C)?
3
Minimum Cut Problem
D = (N, A)
s
1
2
10
n1
3
n3
n2
2
5
Find a cut with the
minimum weight !!
C* = argminC w(C)
n4
Non-negative arc lengths
7
t
3
Solvers for the Minimum-Cut Problem
Augmenting Path and Push-Relabel
n: #nodes
m: #arcs
U: maximum
arc length
[Slide credit: Andrew Goldberg]
Outline
• Minimum Cut Problem
• Two-Label Submodular Energy Functions
• Move-Making Algorithms
Hammer, 1965; Kolmogorov and Zabih, 2004
Overview
Energy
Q
One node per random variable
+ Additional nodes “s” and “t”
Digraph
D
Compute
Minimum
Cut
Labeling
f*
na  N1 implies f(a) = 0
na  N2 implies f(a) = 1
N=
N1 U N 2
Outline
• Minimum Cut Problem
• Two-Label Submodular Energy Functions
• Unary Potentials
• Pairwise Potentials
• Move-Making Algorithms
Digraph for Unary Potentials
P
Q
f(a) = 0
θa(1)
f(a) = 1
θa(0)
Va
Digraph for Unary Potentials
P
Q
f(a) = 0
s
f(a) = 1
na
t
Digraph for Unary Potentials
P
Q
f(a) = 0
s
f(a) = 1
Constant
P-Q
Q
+
0
Let P ≥ Q
Q
na
P-Q
t
Digraph for Unary Potentials
P
Q
f(a) = 0
s
f(a) = 1
Constant
P-Q
Q
+
0
Let P ≥ Q
Q
f(a) = 1
na
P-Q
t
w(C) = 0
Digraph for Unary Potentials
P
Q
f(a) = 0
s
f(a) = 1
Constant
P-Q
Q
+
0
Let P ≥ Q
Q
f(a) = 0
na
P-Q
t
w(C) = P-Q
Digraph for Unary Potentials
P
Q
f(a) = 0
0
P
+
Q-P
s
Q-P
f(a) = 1
Constant
P
Let P < Q
na
t
Digraph for Unary Potentials
P
Q
f(a) = 0
0
P
+
Q-P
s
Q-P
f(a) = 1
Constant
P
Let P < Q
na
f(a) = 1
w(C) = Q-P
t
Digraph for Unary Potentials
P
Q
f(a) = 0
0
P
+
Q-P
s
Q-P
f(a) = 1
Constant
P
Let P < Q
na
f(a) = 0
w(C) = 0
t
Outline
• Minimum Cut Problem
• Two-Label Submodular Energy Functions
• Unary Potentials
• Pairwise Potentials
• Move-Making Algorithms
Digraph for Pairwise Potentials
f(b) = 0
f(b) = 1
f(a) = 0
f(a) = 1
P
R
Q
θab(1,1)
θab(0,1)
S
θab(1,0)
θab(0,0)
Va
+
P
P
P
P
0
0
Q-P Q-P
0
+
0
Vb
S-Q
S-Q
+
0
R+Q-S-P
0
0
Digraph for Pairwise Potentials
f(a) = 0
f(a) = 1
f(b) = 0
P
R
f(b) = 1
Q
S
s
Constant
+
P
P
P
P
0
0
Q-P Q-P
na
nb
t
0
+
0
S-Q
S-Q
+
0
R+Q-S-P
0
0
Digraph for Pairwise Potentials
f(a) = 0
f(a) = 1
f(b) = 0
P
R
f(b) = 1
Q
S
s
Q-P
na
nb
Unary Potential
f(b) = 1
0
0
Q-P Q-P
t
0
+
0
S-Q
S-Q
+
0
R+Q-S-P
0
0
Digraph for Pairwise Potentials
f(a) = 0
f(a) = 1
f(b) = 0
P
R
f(b) = 1
Q
S
s
S-Q
Q-P
na
nb
t
Unary Potential
f(a) = 1
0
0
S-Q
S-Q
+
0
R+Q-S-P
0
0
Digraph for Pairwise Potentials
f(a) = 0
f(a) = 1
f(b) = 0
P
R
f(b) = 1
Q
S
S-Q
na
s
Q-P
R+Q-S-P
nb
t
Pairwise Potential
f(a) = 1, f(b) = 0
0
R+Q-S-P
0
0
Digraph for Pairwise Potentials
f(a) = 0
f(a) = 1
f(b) = 0
P
R
f(b) = 1
Q
S
S-Q
na
s
Q-P
R+Q-S-P
t
R+Q-S-P ≥ 0
General 2-label problem is NP-hard
nb
Outline
• Minimum Cut Problem
• Two-Label Submodular Energy Functions
• Move-Making Algorithms (Multiple Labels)
Stereo Correspondence
Disparity Map
Stereo Correspondence
L = {disparities}
Pixel (xa,ya) in left
corresponds to
pixel (xa+f(a),ya) in right
Stereo Correspondence
L = {disparities}
θa(i) is proportional to
the difference in intensity values
(IL(xa,ya) – IR(xa+i,ya))2
Stereo Correspondence
L = {disparities}
θab(i,k) = wab d(i,k)
d: Distance function over labels
wab ∝ exp(-|IL(xa,ya)-IL(xb,yb)|)
Move-Making Algorithms
Space of All Labelings
f
Expansion Algorithm
Initialize labeling f = f0 (say f0(a) = 0, for all Va)
For α = 0, 2, … , h-1
fα = argminf’ Q(f’)
s.t. f’(a)  {f(a)} U {lα}
Update f = fα
End
Boykov, Veksler and Zabih, 2001
Repeat
until
convergence
Expansion Algorithm
Variables take label lα or retain current label
Slide courtesy Pushmeet Kohli
Expansion Algorithm
Variables take label lα or retain current label
Status:
InitializeSky
Expand
Ground
House
with Tree
Slide courtesy Pushmeet Kohli
Tree
Ground
House
Sky
Expansion Algorithm
Restriction on pairwise potentials?
θab(i,k) + θab(α,α) ≤ θab(i,α) + θab(α,k)
θab(i,k) = wab d(i,k)
d(.,.) is a metric distance function
Swap Algorithm
Initialize labeling f = f0 (say f0(a) = 0, for all Va)
For α, β = 0, 2, … , h-1
fαβ = argminf’ Q(f’)
s.t.
f’(a)  {lα,lβ} if f(a)  {lα,lβ}
f’(a) = f(a)
otherwise
Update f = fαβ
End
Boykov, Veksler and Zabih, 2001
Repeat
until
convergence
Swap Algorithm
Restriction on pairwise potentials?
θab(α,α) + θab(β,β) ≤ θab(α,β) + θab(β,α)
θab(i,k) = wab d(i,k)
d(.,.) is a semi-metric distance function