Solving NP Hard problems in practice
— lessons from Computer Vision and
Computational Biology
Yair Weiss
School of Computer Science and Engineering
The Hebrew University of Jerusalem
www.cs.huji.ac.il/ ∼yweiss
Collaborators: Talya Meltzer, Chen Yanover
Outline
• Energy minimization in computer vision and computational
biology.
• Linear Programming relaxations.
• Using BP to solve LP.
• Experimental results.
Pairwise energy minimization
E(x) =
X
<ij>
• Stereo vision.
• Side-Chain Prediction.
• Protein Design.
Eij (xi, xj ) +
X
i
Ei(xi)
Stereo Vision
Left
Right
Disparity (Tsukuba University)
Stereo problem as discrete optimization
x is disparity image. E(xi, xj ) is compatability cost and E(xi) is
local data cost.
x∗ = arg min
x
X
i,j
Eij (xi, xj ) +
X
Ei(xi)
i
Old formulation (Marr and Poggio 82) but how do we optimize?
Protein Folding
Gene sequence ⇒ Amino-acid sequence ⇒ 3D structure.
Protein folding in two stages
• Backbone
• Side Chains
Backbone
Backbone plus sidechains
Side chain prediction as combinatorial
optimization
• Search space: position of each side chain defined by 4 angles.
Each angle is one of 3 possibilities. Search space is 81n.
• Cost function: local energy and pairwise energies.
Goal: find set of angles such that energy is minimal.
Protein design
Given desired sequence - find Amino-acid sequence.
Protein design as combinatorial
optimization
• Search space: amino-acids plus angles. Search space is (20×
81)n.
• Cost function: local energy and pairwise energies.
Goal: find set of amino-acids and angles such that energy is
minimal.
Graphical models
Variables = nodes, edges = pairwise energy term
MAP in graphical models
x∗ = arg min
x
X
i,j
Eij (xi, xj ) +
X
Ei(xi)
i
NP Hard for protein folding and stereo vision ⇒ many
approximate algorithms.
What’s wrong with approximations ?
Best approximate minimizers:
• Stereo vision - unsatisfactory results.
• Side chain prediction - approx. 85%
• Protein design - “good structure” often in top 100.
Better minimizer or better energy functions ?
Linear Programming relaxations
J(x) =
X
Eij (xi, xj ) +
i,j
X
Ei(xi)
i
qi(xi), qij (xi, xj ) are indicator variables.
J({q}) =
X X
i,j xi ,xj
qij (xi, xj )Eij (xi, xj ) +
XX
i
xi
qi(xi)Ei(xi)
Integer Programming formulation
minimize:
J({q}) =
X X
i,j xi ,xj
qij (xi, xj )Eij (xi, xj ) +
XX
i
subject to:
qij (xi, xj ) ∈ {0, 1}
X
xi ,xj
X
xi
qij (xi, xj ) = 1
qij (xi, xj ) = qj (xj )
xi
qi(xi)Ei(xi)
Linear Programming relaxation
minimize:
J({q}) =
X X
i,j xi ,xj
qij (xi, xj )Eij (xi, xj ) +
XX
i
subject to:
qij (xi, xj ) ∈ [0, 1]
X
xi ,xj
X
xi
qij (xi, xj ) = 1
qij (xi, xj ) = qj (xj )
xi
qi(xi)Ei(xi)
Guarantee of optimality
If the LP solution is integer (qij (xi, xj ) ∈ {0, 1} for all i, j) then
we have found the global minimum of J.
LP relaxations for vision?
State of the art LP solver can be applied to 39 × 39 subimage in
a machine with 4G memory (34 × 34 with 2G).
8
10
#Variables
6
#Constraints
10
4
10
2
10
LOQO
MATLAB, linprog
MATLAB, bintprog
0
10
50
100
150
200
LP using BP
Surprisingly, belief propagation can be used to solve for the linear
programming relaxation (Wainwright, Jaakkola and Willsky 03;
Weiss, Meltzer and Yanover 05)
Belief Propagation:
• A parallel message-passing algorithm.
• Every node sends a probability density to its neighbors.
From energy function to potentials
1
Ψij (xi, xj ) = exp(− Eij (xi, xj ))
T
1
Ψi(xi) = exp(− Ei(xi))
T
Iterate:
mij (xj ) ← α
X
xi
Ψij (xi, xj )Ψi(xi)
Y
Xk ∈N (Xi)\Xj
From the messages we calclate bi(xi), bij (xi, xj ).
mki(xi)
BP and LP
Fixed points of BP correspond to stationary points of the BetheKikuchi free energy (Yedidia, Freeman, Weiss 01, Kabashima
Saad 98).
Fβ =
X X
bij (xi, xj )Eij (xi, xj ) +
XX
bi(xi)Ei(xi)
i,j xi ,xj
i xi


X
X
X X

bij (xi, xj ) ln bij (xi, xj ) − (di − 1)
bi(xi) ln bi(xi)
+T
xi
i
i,j xi ,xj
BP and LP
minimize:
Fβ =
X X
bij (xi, xj )Eij (xi, xj ) +
XX
bi(xi)Ei(xi)
i,j xi ,xj
i xi


X
X
X X

+T
bij (xi, xj ) ln bij (xi, xj ) − (di − 1)
bi(xi) ln bi(xi)
xi
i,j xi ,xj
i
subject to:
X
xi ,xj
X
xi
bij (xi, xj ) ∈ [0, 1]
bij (xi, xj ) = 1
bij (xi, xj ) = bj (xj )
Visualizing BP and LP
Fβ =
X X
bij (xi, xj )Eij (xi, xj ) +
XX
bi(xi)Ei(xi)
i,j xi ,xj
i xi


X X
X
X
+T 
bij (xi, xj ) ln bij (xi, xj ) − (di − 1)
bi(xi) ln bi(xi)
xi
i,j xi ,xj
i
bij (xi, xj ) =
a
b
b 1 − (a + 2b)
!
Visualizing BP and LP
T=0.00
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Visualizing BP and LP
T=100.00
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Visualizing BP and LP
T=10.00
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Visualizing BP and LP
T=5.00
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Visualizing BP and LP
T=2.00
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Visualizing BP and LP
T=1.00
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Visualizing BP and LP
T=0.50
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Visualizing BP and LP
T=0.25
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Visualizing BP and LP
T=0.10
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Solving LP by running BP at zero
temperature
1
Ψij (xi, xj ) = exp(− Eij (xi, xj ))
T
1
Ψi(xi) = exp(− Ei(xi))
T
Iterate:
mij (xj ) ← α max xiΨij (xi, xj )Ψi(xi)
Y
mki(xi)
Xk ∈N (Xi)\Xj
Claim: If sharpened max-product beliefs are feasible they give
zero-temperature sum-product beliefs.
Visualizing BP and LP
T=100.00
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Visualizing BP and LP
T=10.00
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Visualizing BP and LP
T=5.00
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Visualizing BP and LP
T=2.00
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Visualizing BP and LP
T=1.00
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Visualizing BP and LP
T=0.50
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Visualizing BP and LP
T=0.25
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Visualizing BP and LP
T=0.10
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Is Fβ convex?
Fβ =
X X
bij (xi, xj )Eij (xi, xj ) +
XX
bi(xi)Ei(xi)
i,j xi ,xj
i xi


X X
X
X
+T 
bij (xi, xj ) ln bij (xi, xj ) − (di − 1)
bi(xi) ln bi(xi)
xi
i,j xi ,xj
i
The entropy is always bounded but convexity depends on graph
topology (through degree di) (Pakzad and Ananthram 03, Heskes
03).
Convexifying F
F̃β =
X X
bij (xi, xj )Eij (xi, xj ) +
XX
bi(xi)Ei(xi)
i,j xi ,xj
i xi


X X
+T 
bij (xi, xj ) ln bij (xi, xj )
i,j xi ,xj
(several convexification possible)
Visualizing BP and LP
T=100.00
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Visualizing BP and LP
T=10.00
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Visualizing BP and LP
T=5.00
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Visualizing BP and LP
T=2.00
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Visualizing BP and LP
T=1.00
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Visualizing BP and LP
T=0.50
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Visualizing BP and LP
T=0.25
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Visualizing BP and LP
T=0.10
0.5
0.4
0.3
0.2
0.1
0
0
0.2
0.4
0.6
0.8
1
Minimizing Convexified Fβ
n0
ji(xi )
=
Y
mki(xi)
Xk ∈N (Xi)\Xj
X
0
mij (xj ) = α
Ψij (xi, xj )Ψi(xi)nji(xi)
xi
γ γ −1
j
j
0
0
mij (xj ) ← mij (xj )
nij (xj )
γ −1 γ
j
j
0
0
nij (xj ) ← mij (xj )
nij (xj )
1
γj = 2−1/d
.
j
“two way GBP” (Yedidia, Freeman, Weiss 02)
Minimizing Convexified Fβ at zero
temperature
n0
ji(xi )
Y
=
mki(xi)
Xk ∈N (Xi)\Xj
m0
ij (xj ) ← α max Ψij (xi , xj )Ψi (xi )nji(xi )
xi
γ γ −1
j
j
0
0
mij (xj ) = mij (xj )
nij (xj )
γ −1 γ
j
j
0
0
nij (xj ) ← mij (xj )
nij (xj )
A new LP solver ?
Comparisons
Compare run-times of:
• Generic BP code.
• CPLEX 9.0.
solver.
Perhaps the most powerful commercial LP
Comparisons - stereo
#Variables
#Constraints
5
10
Tomlab
0
10
50
100
150
Grid size
200
BP can solve full sized images.
Comparisons - protein folding
side-chain prediction
design
5
6
x 10
x 10
3
15
Variables
Variables
2.5
2
1.5
1
10
5
0.5
50
100
150
50
Protein length [aa]
4
100
150
100
150
100
150
Protein length [aa]
5
x 10
x 10
4
Constraints
Constraints
4
3
2
1
3
2
1
50
100
150
50
Protein length [aa]
5
Protein length [aa]
7
x 10
x 10
5
Non−zeros
Non−zeros
8
6
4
2
4
3
2
1
50
100
150
Protein length [aa]
50
Protein length [aa]
Run times - stereo
PRIMAL
BP
500
50
400
40
300
200
TRBP
DUAL
BP
TRBP
10
15
NET
30
time [min.]
TRBP
time [min.]
time [min.]
BP
30
20
20
10
10
100
0
5
10
BP
15
non zeros
TRBP
20
0
25
5
BARRIER
10
15
non zeros
BP
TRBP
20
0
25
SIFTING
5
BP
non zeros
TRBP
20
25
CONCURRENT
4
40
2
1
0
time [min.]
3
time [min.]
time [min.]
200
150
100
10
15
non zeros
20
25
0
20
10
50
5
30
5
10
15
non zeros
20
25
0
5
10
15
non zeros
20
25
Run times - side chain prediction
scwrl
BP
TRBP
BP
PRIMAL
TRBP
DUAL
BP
200
time [min.]
300
2
1
5
10
non zeros
BP
TRBP
0
15
5
x 10
10
non zeros
0.6
0.4
BP
TRBP
0
5
10
non zeros
10
non zeros
TRBP
15
5
x 10
CONCURRENT
3
60
40
0
15
5
x 10
5
BP
SIFTING
20
0.2
1
0
time [min.]
0.8
1.5
15
5
x 10
80
time [min.]
time [min.]
5
BARRIER
1
NET
0.5
100
0
TRBP
2
3
400
time [min.]
time [min.]
500
5
10
non zeros
2
1
0
15
5
x 10
5
10
15
5
x 10
non zeros
rosetta
BP
TRBP
PRIMAL
BP
TRBP
DUAL
BP
TRBP
NET
15
time [min.]
time [min.]
500
time [min.]
10
1000
8
6
4
10
5
2
5
10
non zeros
BP
TRBP
0
15
5
x 10
BARRIER
BP
time [min.]
4
2
10
non zeros
6
time [min.]
5
TRBP
5
10
non zeros
15
5
x 10
BP
2000
10
1500
8
1000
0
5
10
non zeros
SIFTING
500
0
0
15
5
x 10
time [min.]
0
TRBP
15
5
x 10
CONCURRENT
6
4
2
5
10
non zeros
15
5
x 10
0
5
10
non zeros
15
5
x 10
Comparisons - protein design
BP
TRBP
BARRIER
time [min.]
250
200
150
100
50
1
2
3
non zeros
4
5
7
x 10
Discussion
• BP almost always faster than all solvers in CPLEX.
• BP can be applied when CPLEX cannot.
• BP is not a general purpose LP solver.
Global Optimum ?
• When LP/BP solution integer.
• Even when BP solution is partly fractional (Meltzer et al. 05)
In about 90% of benchmark problems, we can find global optimum in about 10 minutes.
Does global optimum improve
performance ?
Using global optimizers:
• Stereo vision - unsatisfactory results.
• Side chain prediction - approx. 85 + %
• Protein design - “good structure” often in top 100.
Better energy functions are needed !
Conclusions
• Energy minimization in computer vision and computional biology. Typically NP hard.
• Standard LP solvers cannot handle LPs arising from our applications.
• Convex BP gives a simple, efficient algorithm for solving LP
relaxations.
• Globally optimal solutions can be obtained in minutes.