Decomposition and Reformulation in Mixed-Integer
Programming
IMA New Directions Short Course on Mathematical Optimization
Jim Luedtke
Department of Industrial and Systems Engineering
University of Wisconsin-Madison
August 11, 2016
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
1 / 47
What makes integer programs hard to solve?
1
Weak LP relaxation bounds
Pruning in branch-and-bound is rare – huge search trees
Possible solution: use a better formulation or add (strong) valid
inequalities
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
2 / 47
What makes integer programs hard to solve?
1
Weak LP relaxation bounds
Pruning in branch-and-bound is rare – huge search trees
Possible solution: use a better formulation or add (strong) valid
inequalities
2
Huge number of variables and/or constraints
Just solving LP relaxation is very time-consuming
Possible solution: cutting plane approach – but doesn’t address huge
number of variables
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
2 / 47
What makes integer programs hard to solve?
1
Weak LP relaxation bounds
Pruning in branch-and-bound is rare – huge search trees
Possible solution: use a better formulation or add (strong) valid
inequalities
2
Huge number of variables and/or constraints
Just solving LP relaxation is very time-consuming
Possible solution: cutting plane approach – but doesn’t address huge
number of variables
Another possibility: Lagrangian relaxation or column generation
May be able to help with either or both of these challenges
Particularly useful if enables decomposition – splitting one large
problem into many smaller ones
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
2 / 47
Lagrangian Relaxation
Outline
1
Lagrangian Relaxation
Example
Lagrangian Relaxation Bounds
Solving the Lagrangian Dual
2
Dantzig-Wolfe Reformulation and Column Generation
3
Branch-and-Price
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
3 / 47
Lagrangian Relaxation
Motivation
Consider this IP:
z IP = max
K
X
c>
k xk
k=1
Dx ≤ d
Ak xk ≤ bk ,
k = 1, . . . , K
x = (x1 , . . . , xK ) ∈ ZnK
+
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
4 / 47
Lagrangian Relaxation
Motivation
Consider this IP:
z IP = max
K
X
c>
k xk
k=1
Dx ≤ d
Ak xk ≤ bk ,
k = 1, . . . , K
x = (x1 , . . . , xK ) ∈ ZnK
+
If only we didn’t have those pesky Dx ≤ d constraints . . .
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
4 / 47
Lagrangian Relaxation
Motivation
Drop the pesky constraints:
z R = max
K
X
c>
k xk
k=1
Ak xk ≤ bk ,
k = 1, . . . , K
x = (x1 , . . . , xK ) ∈ ZnK
+
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
5 / 47
Lagrangian Relaxation
Motivation
Drop the pesky constraints:
z R = max
K
X
c>
k xk
k=1
Ak xk ≤ bk ,
k = 1, . . . , K
x = (x1 , . . . , xK ) ∈ ZnK
+
P
The problem decomposes: z R = K
k=1 zk where
n
zk = max c>
k xk : Ak xk ≤ bk , xk ∈ Z+
for k = 1, . . . , K
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
5 / 47
Lagrangian Relaxation
Motivation
Drop the pesky constraints:
z R = max
K
X
c>
k xk
k=1
Ak xk ≤ bk ,
k = 1, . . . , K
x = (x1 , . . . , xK ) ∈ ZnK
+
P
The problem decomposes: z R = K
k=1 zk where
n
zk = max c>
k xk : Ak xk ≤ bk , xk ∈ Z+
for k = 1, . . . , K
z R ≥ z IP , but dropping Dx ≤ d altogether is pretty severe
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
5 / 47
Lagrangian Relaxation
Lagrangian Relaxation
Simplify notation: Let A be the matrix combining all Ak submatrices, and
b = (b1 , . . . , bK ), c = (c1 , . . . , cK )
z IP = max c> x
Dx ≤ d
Ax ≤ b
x ∈ Zn+
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
6 / 47
Lagrangian Relaxation
Lagrangian Relaxation
Simplify notation: Let A be the matrix combining all Ak submatrices, and
b = (b1 , . . . , bK ), c = (c1 , . . . , cK )
z IP = max c> x
Dx ≤ d
Ax ≤ b
x ∈ Zn+
Lagrangian relaxation: relax the constraints Dx ≤ d by dualizing them –
adding them to the objective with a penalty for violation
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
6 / 47
Lagrangian Relaxation
Lagrangian Relaxation
Simplify notation: Let A be the matrix combining all Ak submatrices, and
b = (b1 , . . . , bK ), c = (c1 , . . . , cK )
z IP = max c> x
Dx ≤ d
Ax ≤ b
x ∈ Zn+
Lagrangian relaxation: relax the constraints Dx ≤ d by dualizing them –
adding them to the objective with a penalty for violation
Problem with just constraints Ax ≤ b should be easier to solve
We’ll discuss how to choose the constraints to dualize later
For simplicity we assume x ∈ Zn+
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
6 / 47
Lagrangian Relaxation
Lagrangian Relaxation
First, re-write our IP, with X := {x ∈ Zn+ : Ax ≤ b}
z IP = max c> x
subject to Dx ≤ d
x∈X
Let u ∈ Rm
+ , and define the following Lagrangian Relaxation problem:
IP (u) :
z(u) = max{c> x + u> (d − Dx) : x ∈ X}
Theorem
IP
For any u ∈ Rm
+ , z(u) ≥ z .
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
7 / 47
Lagrangian Relaxation
Lagrangian Relaxation
First, re-write our IP, with X := {x ∈ Zn+ : Ax ≤ b}
z IP = max c> x
subject to Dx ≤ d
x∈X
Let u ∈ Rm
+ , and define the following Lagrangian Relaxation problem:
IP (u) :
z(u) = max{c> x + u> (d − Dx) : x ∈ X}
Theorem
IP
For any u ∈ Rm
+ , z(u) ≥ z .
Why? Let x∗ be an optimal solution to z IP (z IP = c> x∗ and
d − Dx∗ ≥ 0).
z(u) ≥ c> x∗ + u> (d − Dx∗ ) ≥ z IP
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
7 / 47
Lagrangian Relaxation
Lagrangian Dual
For u ∈ Rm
+,
z(u) = max{c> x + u> (d − Dx) : x ∈ X}
Definition
The problem:
wLD = min{z(u) : u ∈ Rm
+}
is called a Lagrangian dual.
Properties
wLD ≥ z IP
wLD ≤ z(u) for all u ∈ Rm
+
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
8 / 47
Lagrangian Relaxation
Lagrangian Dual
For u ∈ Rm
+,
z(u) = max{c> x + u> (d − Dx) : x ∈ X}
Definition
The problem:
wLD = min{z(u) : u ∈ Rm
+}
is called a Lagrangian dual.
Properties
wLD ≥ z IP
wLD ≤ z(u) for all u ∈ Rm
+
Modification with equality constraints Dx = d: variables u are free
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
8 / 47
Lagrangian Relaxation
Lagrangian Dual
IP (u) :
z(u) = max{c> x + u> (d − Dx) : x ∈ X}
Theorem
Let u ∈ Rm
+ and x̂ be an optimal solution to IP (u). If,
(1) Dx̂ ≤ d, and
(2) u> (d − Dx̂) = 0,
then x̂ is an optimal solution to IP.
Note: The second condition is necessary: x̂ could be feasible to IP, but not
optimal.
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
9 / 47
Lagrangian Relaxation
Lagrangian Dual
IP (u) :
z(u) = max{c> x + u> (d − Dx) : x ∈ X}
Theorem
Let u ∈ Rm
+ and x̂ be an optimal solution to IP (u). If,
(1) Dx̂ ≤ d, and
(2) u> (d − Dx̂) = 0,
then x̂ is an optimal solution to IP.
Note: The second condition is necessary: x̂ could be feasible to IP, but not
optimal.
(1) ⇒ x̂ is feasible
(2) ⇒ z IP ≤ z(u) = c> x̂ + u> (d − Dx̂) = c> x̂
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
9 / 47
Lagrangian Relaxation
Example
Example: Stochastic Integer Programming
Extensive form of SIP
z SM IP = min c> x +
S
X
ps qs> ys
s=1
s.t. Ax ≥ b
Jim Luedtke (UW-Madison)
Ts x + Ws ys = hs
s = 1, . . . , S
x ∈ Rn+1 × Zp+1
ys ∈ Rn+2 × Zp+2 ,
s = 1, . . . , S
Decomposition Methods
Lecture Notes
10 / 47
Lagrangian Relaxation
Example
Example: Stochastic Integer Programming
Copy first-stage variables
z
SM IP
= min
S
X
ps (c> xs + qs> ys )
s=1
Axs ≥ b
s = 1, . . . , S
Ts xs + Ws ys = hs
s = 1, . . . , S
S
X
p s0 x s0
s0 =1
xs ∈ Rn+1 × Zp+1 ,
ys ∈ Rn+2 × Zp+2 ,
xs =
Jim Luedtke (UW-Madison)
Decomposition Methods
s = 1, . . . , S
s = 1, . . . , S
s = 1, . . . , S
Lecture Notes
10 / 47
Lagrangian Relaxation
Example
Relax Nonanticipativity
The constraints xs =
constraints.
Jim Luedtke (UW-Madison)
PS
s0 =1 ps0 xs0
are called nonanticipativity
Decomposition Methods
Lecture Notes
11 / 47
Lagrangian Relaxation
Example
Relax Nonanticipativity
P
The constraints xs = Ss0 =1 ps0 xs0 are called nonanticipativity
constraints.
Relax these constraints using Lagrangian Relaxation with dual vectors
λ = (λ1 , . . . , λS ):
PS
PS
PS
> x + q > y )+
> x −
0 x s0
L(λ) := min
p
(c
p
λ
p
0
s
s
s
s
s
s
s
s
s=1
s=1
s =1
Axs ≥ b
s = 1, . . . , S
Ts xs + Ws ys = hs
n1
xs ∈ R+
n2
ys ∈ R +
Jim Luedtke (UW-Madison)
× Zp+1 ,
× Zp+2 ,
s = 1, . . . , S
s = 1, . . . , S
s = 1, . . . , S
Decomposition Methods
Lecture Notes
11 / 47
Lagrangian Relaxation
Example
Relax Nonanticipativity
P
The constraints xs = Ss0 =1 ps0 xs0 are called nonanticipativity
constraints.
Relax these constraints using Lagrangian Relaxation with dual vectors
λ = (λ1 , . . . , λS ):
PS
PS
PS
> x + q > y )+
> x −
0 x s0
L(λ) := min
p
(c
p
λ
p
0
s
s
s
s
s
s
s
s
s=1
s=1
s =1
Axs ≥ b
s = 1, . . . , S
Ts xs + Ws ys = hs
n1
xs ∈ R+
n2
ys ∈ R +
× Zp+1 ,
× Zp+2 ,
s = 1, . . . , S
s = 1, . . . , S
s = 1, . . . , S
P
Rewrite the objective (λ̄ = Ss=1 ps λs ):
>
PS
c + (λs − λ̄) xs + qs> ys
s=1 ps
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
11 / 47
Lagrangian Relaxation
Example
Relax Nonanticipativity
Rewritten objective (λ̄ =
S
X
ps
PS
s=1 ps λs ):
c + (λs − λ̄)
>
xs + qs> ys
s=1
Normalize λs so that λ̄ = 0
Lagrangian relaxation problem decomposes: L(λ) =
where
P
s ps Ds (λs )
Ds (λs ) := min (c + λs )> x + qs> ys
s.t. Ax ≥ b, Ts x + Ws y = hs
x ∈ Rn+1 × Zp+1 , y ∈ Rn+2 × Zp+2
Each subproblem is a deterministic mixed-integer program
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
12 / 47
Lagrangian Relaxation
Example
Lagrangian Dual Problem
For any λ = (λ1 , . . . , λS ) with
P
s ps λs
= 0,
L(λ) ≤ z SM IP
Lagrangian dual
Find best lower bound:
S
n
o
X
wLD := max L(λ) :
ps λs = 0
s=1
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
13 / 47
Lagrangian Relaxation
Lagrangian Relaxation Bounds
How Good is the Bound From the Lagrangian dual?
The integer program we’re trying to solve
z IP = max c> x : Dx ≤ d, x ∈ X
(IP)
where X = {x ∈ Zn+ : Ax ≤ b}
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
14 / 47
Lagrangian Relaxation
Lagrangian Relaxation Bounds
How Good is the Bound From the Lagrangian dual?
The integer program we’re trying to solve
z IP = max c> x : Dx ≤ d, x ∈ X
(IP)
where X = {x ∈ Zn+ : Ax ≤ b}
Theorem
wLD = max c> x : Dx ≤ d, x ∈ conv(X)
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
14 / 47
Lagrangian Relaxation
Lagrangian Relaxation Bounds
How Good is the Bound From the Lagrangian dual?
The integer program we’re trying to solve
z IP = max c> x : Dx ≤ d, x ∈ X
(IP)
where X = {x ∈ Zn+ : Ax ≤ b}
Theorem
wLD = max c> x : Dx ≤ d, x ∈ conv(X)
Let P = {x ∈ Rn+ : Ax ≤ b}
z LP = max c> x : Dx ≤ d, x ∈ P
P ⊇ conv(X), so wLD ≤ z LP
If P = conv(X), then wLD = z LP
wLD < z LP is only possible if P 6= conv(X)
Let’s prove it for special case: X bounded, all integer vars
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
14 / 47
Lagrangian Relaxation
Lagrangian Relaxation Bounds
Strength of Lagrangian Dual of SMIP
Theorem
The Lagrangian dual bound satisfies
(
)
S
X
LD
>
w
= min c x +
ps ys : (x, ys ) ∈ conv(Xs ), s = 1, . . . , S
s=1
where for s = 1, . . . , S
Xs := {(x, y) : Ax ≥ b, Ts x + Ws y = hs
x ∈ Rn+1 × Zp+1 , y ∈ Rn+2 × Zp+2 }
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
15 / 47
Lagrangian Relaxation
Lagrangian Relaxation Bounds
Strength of Lagrangian Dual of SMIP
Theorem
The Lagrangian dual bound satisfies
(
)
S
X
LD
>
w
= min c x +
ps ys : (x, ys ) ∈ conv(Xs ), s = 1, . . . , S
s=1
where for s = 1, . . . , S
Xs := {(x, y) : Ax ≥ b, Ts x + Ws y = hs
x ∈ Rn+1 × Zp+1 , y ∈ Rn+2 × Zp+2 }
In general wLD < z SM IP
But wLD ≥ z SLP (the usual LP relaxation)
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
15 / 47
Lagrangian Relaxation
Lagrangian Relaxation Bounds
Strength of Lagrangian Dual of SMIP
Theorem
The Lagrangian dual bound satisfies
(
)
S
X
LD
>
w
= min c x +
ps ys : (x, ys ) ∈ conv(Xs ), s = 1, . . . , S
s=1
where for s = 1, . . . , S
Xs := {(x, y) : Ax ≥ b, Ts x + Ws y = hs
x ∈ Rn+1 × Zp+1 , y ∈ Rn+2 × Zp+2 }
In general wLD < z SM IP
But wLD ≥ z SLP (the usual LP relaxation)
wLD at least as good as any bound obtained using cuts in single
scenario subproblems
In many test instances, wLD is very close to z SM IP
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
15 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
How to solve the Lagrangian dual?
Lagrangian dual
wLD = min{z(u) : u ≥ 0}
How to find u∗ that solves this optimization problem?
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
16 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
How to solve the Lagrangian dual?
Lagrangian dual
wLD = min{z(u) : u ≥ 0}
How to find u∗ that solves this optimization problem?
Key insight
z(u) is a piecewise linear convex function of u.
We already have seen
z(u) = max c> xt + u> (d − Dxt ) : t = 1, . . . , T
where {xt : t = 1, . . . , T } are the finitely many points in
X = {x ∈ Zn+ : Ax ≤ b}.
Extends also to mixed-integer, unbounded sets (like Benders analysis)
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
16 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
Option 1: Subgradient
Lagrangian dual: A nonsmooth convex optimization problem
wLD = min{z(u) : u ≥ 0}
Assume for this algorithm that X is bounded so conv(X) has no rays
Recall
Let f : Rm → R and u ∈ Rm . A vector γ(u) is called a subgradient of
f at u if
f (v) ≥ f (u) + γ(u)> (v − u) ∀v ∈ Rm
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
17 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
Option 1: Subgradient
Subgradient algorithm for generic convex problem: min{f (u) : u ≥ 0}
1
2
Initialize: u = u0
Iteration k ≥ 0:
Calculate f (uk ) and find a subgradient γ(uk ) of f at uk
Step:
uk+1 = max{uk − µk γ(uk ), 0}
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
18 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
Option 1: Subgradient
Subgradient algorithm for generic convex problem: min{f (u) : u ≥ 0}
1
2
Initialize: u = u0
Iteration k ≥ 0:
Calculate f (uk ) and find a subgradient γ(uk ) of f at uk
Step:
uk+1 = max{uk − µk γ(uk ), 0}
Notes:
The values µk are step sizes that satisfy at a minimum: µk > 0 and
µk → 0 as k → ∞
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
18 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
Option 1: Subgradient
Subgradient algorithm for Lagrangian dual problem:
min{z(u) : u ≥ 0}
1
2
Initialize: u = u0
Iteration k ≥ 0:
Calculate z(uk ) and find a subgradient γ(uk ) of z at uk
Step:
uk+1 = max{uk − µk γ(uk ), 0}
Key question: How to calculate z(uk ) and γ(uk )?
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
19 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
Option 1: Subgradient
Subgradient algorithm for Lagrangian dual problem:
min{z(u) : u ≥ 0}
How to calculate z(uk ) and γ(uk )?
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
20 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
Option 1: Subgradient
Subgradient algorithm for Lagrangian dual problem:
min{z(u) : u ≥ 0}
How to calculate z(uk ) and γ(uk )?
z(uk ) = max c> x + (uk )> (d − Dx) : x ∈ X
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
20 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
Option 1: Subgradient
Subgradient algorithm for Lagrangian dual problem:
min{z(u) : u ≥ 0}
How to calculate z(uk ) and γ(uk )?
z(uk ) = max c> x + (uk )> (d − Dx) : x ∈ X
Subgradients of z
Let xk ∈ X be an optimal solution to calculation of z(uk ). Then
γ(uk ) = d − Dxk
is a subgradient of z at uk .
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
20 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
Option 1: Subgradient
Subgradients of z
Let xk ∈ X be an optimal solution to calculation of z(uk ). Then
γ(uk ) = d − Dxk
is a subgradient of z at uk .
Proof: Let v ∈ Rm .
z(v) = max c>x + v > (d − Dx) : x ∈ X
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
21 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
Option 1: Subgradient
Subgradients of z
Let xk ∈ X be an optimal solution to calculation of z(uk ). Then
γ(uk ) = d − Dxk
is a subgradient of z at uk .
Proof: Let v ∈ Rm .
z(v) = max c>x + v > (d − Dx) : x ∈ X
≥ c> xk + v > (d − Dxk )
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
21 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
Option 1: Subgradient
Subgradients of z
Let xk ∈ X be an optimal solution to calculation of z(uk ). Then
γ(uk ) = d − Dxk
is a subgradient of z at uk .
Proof: Let v ∈ Rm .
z(v) = max c>x + v > (d − Dx) : x ∈ X
≥ c> xk + v > (d − Dxk )
= c> xk + (uk )> (d − Dxk ) + v > (d − Dxk ) − (uk )> (d − Dxk )
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
21 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
Option 1: Subgradient
Subgradients of z
Let xk ∈ X be an optimal solution to calculation of z(uk ). Then
γ(uk ) = d − Dxk
is a subgradient of z at uk .
Proof: Let v ∈ Rm .
z(v) = max c>x + v > (d − Dx) : x ∈ X
≥ c> xk + v > (d − Dxk )
= c> xk + (uk )> (d − Dxk ) + v > (d − Dxk ) − (uk )> (d − Dxk )
= z(uk ) + (v − uk )> (d − Dxk )
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
21 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
Option 1: Subgradient
Subgradients of z
Let xk ∈ X be an optimal solution to calculation of z(uk ). Then
γ(uk ) = d − Dxk
is a subgradient of z at uk .
Proof: Let v ∈ Rm .
z(v) = max c>x + v > (d − Dx) : x ∈ X
≥ c> xk + v > (d − Dxk )
= c> xk + (uk )> (d − Dxk ) + v > (d − Dxk ) − (uk )> (d − Dxk )
= z(uk ) + (v − uk )> (d − Dxk )
= z(uk ) + (v − uk )> γ(uk )
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
21 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
Option 1: Subgradient
Advantages: easy to implement, can potentially get good
improvement in few iterations
Disadvantage: convergence can be slow because it ignores history of
information (past solution values and subgradients)
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
22 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
Solving the Lagrangian Dual: Option 2
We have already seen:
wLD = min{z(u) : u ≥ 0}
= min η + d> u
s.t. η + u> (Dxt ) ≥ c> xt ,
t = 1, . . . , T,
u≥0
This last problem is a linear program with a huge number of constraints
Solve by a cutting plane algorithm – Just like Benders decomposition!
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
23 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
Solving the Lagrangian Dual: Other Options
Lagrangian dual
min{z(u) : u ≥ 0}
Bundle methods: Nonlinear programming methods for nonsmooth
optimization
Use all (or much of) past subgradients as in cutting plane algorithm
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
24 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
Solving the Lagrangian Dual: Other Options
Lagrangian dual
min{z(u) : u ≥ 0}
Bundle methods: Nonlinear programming methods for nonsmooth
optimization
Use all (or much of) past subgradients as in cutting plane algorithm
Use a stabilization technique to limit “bouncing around”
Trust region: restrict k u − uk k≤ αk in master LP
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
24 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
Solving the Lagrangian Dual: Other Options
Lagrangian dual
min{z(u) : u ≥ 0}
Bundle methods: Nonlinear programming methods for nonsmooth
optimization
Use all (or much of) past subgradients as in cutting plane algorithm
Use a stabilization technique to limit “bouncing around”
Trust region: restrict k u − uk k≤ αk in master LP
Augmented objective: add λk k u − uk k to master LP objective
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
24 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
Solving the Lagrangian Dual: Other Options
Lagrangian dual
min{z(u) : u ≥ 0}
Bundle methods: Nonlinear programming methods for nonsmooth
optimization
Use all (or much of) past subgradients as in cutting plane algorithm
Use a stabilization technique to limit “bouncing around”
Trust region: restrict k u − uk k≤ αk in master LP
Augmented objective: add λk k u − uk k to master LP objective
Bundle-level: solve unrestricted LP to obtain L, then solve a second
problem finding closest u to uk that obtains a minimum improvement
in the objective
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
24 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
Choosing the Lagrangian Dual
Trade-offs in choosing constraints to dualize
z IP = max c> x
Dx ≤ d, Ax ≤ b
x ∈ Zn+
Dualize “hard” constraints leaving something easy (convex hull
described by remaining inequalities)
Bound is only as strong as the LP bound
Subproblems (for fixed u) will be easy to solve
Might be more efficient than LP
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
25 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
Choosing the Lagrangian Dual
Trade-offs in choosing constraints to dualize
z IP = max c> x
Dx ≤ d, Ax ≤ b
x ∈ Zn+
Dualize “hard” constraints leaving something easy (convex hull
described by remaining inequalities)
Bound is only as strong as the LP bound
Subproblems (for fixed u) will be easy to solve
Might be more efficient than LP
Leave some “hard” constraints undualized (convex hull not described
by the inequalities)
Bound can be significantly better than LP bound
Solving the subproblems can be more difficult
If enables decomposition, the “hard” subproblems may not be too bad
in practice – e.g. knapsack
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
25 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
What to do After Solving the Lagrangian Dual?
1
Use it as a basis for heuristics
Problem-specific
E.g., fix some variables based on Lagrangian subproblem and solve a
smaller problem
Often, relatively easy to restore feasibility
E.g., in stochastic IP, fix first-stage vars, solve second-stage problems
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
26 / 47
Lagrangian Relaxation
Solving the Lagrangian Dual
What to do After Solving the Lagrangian Dual?
1
Use it as a basis for heuristics
Problem-specific
E.g., fix some variables based on Lagrangian subproblem and solve a
smaller problem
Often, relatively easy to restore feasibility
E.g., in stochastic IP, fix first-stage vars, solve second-stage problems
2
Use it as a relaxation within a branch-and-bound search
Instead of solving LP relaxations!
Branching is problem-specific (we’ll see this later)
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
26 / 47
Dantzig-Wolfe Reformulation and Column Generation
Outline
1
Lagrangian Relaxation
2
Dantzig-Wolfe Reformulation and Column Generation
Dantzig-Wolfe Reformulation
Column Generation
3
Branch-and-Price
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
27 / 47
Dantzig-Wolfe Reformulation and Column Generation
Dantzig-Wolfe Reformulation
Motivation
Consider this IP:
z
IP
= max
K
X
c>
k xk
k=1
Dx ≤ d,
xk ∈ Xk ,
k = 1, . . . , K
Where Xk = {x ∈ Zn+ : Ak x ≤ bk }
Could relax the constraints Dx ≤ d, as in Lagrangian relaxation
Different option: Dantzig-Wolfe Reformulation
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
28 / 47
Dantzig-Wolfe Reformulation and Column Generation
Dantzig-Wolfe Reformulation
Dantzig-Wolfe Reformulation
Assume Xk is bounded, so it has finitely many points:
k
I.e., Xk = {xk,t }Tt=1
Then, xk ∈ Xk if and only if there exists λk,t ∈ {0, 1}, t = 1, . . . , Tk such
that:
xk =
Tk
X
t=1
Jim Luedtke (UW-Madison)
λk,t xk,t ,
Tk
X
λk,t = 1
t=1
Decomposition Methods
Lecture Notes
29 / 47
Dantzig-Wolfe Reformulation and Column Generation
Dantzig-Wolfe Reformulation
Dantzig-Wolfe Reformulation (2)
Replace the constraints xk ∈ Xk with λ formulation yields:
z IP = max
K
X
k=1
K
X
c>
k xk
Dk xk ≤ d,
k=1
xk =
Tk
X
λk,t xk,t ,
t=1
λk,t = 1,
k = 1, . . . , K
t=1
λk,t ∈ {0, 1},
Jim Luedtke (UW-Madison)
Tk
X
t = 1, . . . , Tk , k = 1, . . . , K
Decomposition Methods
Lecture Notes
30 / 47
Dantzig-Wolfe Reformulation and Column Generation
Dantzig-Wolfe Reformulation
Dantzig-Wolfe Reformulation (2)
Replace the constraints xk ∈ Xk with λ formulation yields:
z IP = max
K
X
k=1
K
X
c>
k xk
Dk xk ≤ d,
k=1
xk =
Tk
X
λk,t xk,t ,
t=1
Tk
X
λk,t = 1,
k = 1, . . . , K
t=1
λk,t ∈ {0, 1},
t = 1, . . . , Tk , k = 1, . . . , K
Now, substitute out the xk variables...
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
30 / 47
Dantzig-Wolfe Reformulation and Column Generation
Dantzig-Wolfe Reformulation
Dantzig-Wolfe Reformulation (3)
Obtain the Dantzig-Wolfe Reformulation
max
Tk
K X
X
k=1 t=1
Tk
K X
X
k,t
c>
λk,t
kx
Dk xk,t λk,t ≤ d
k=1 t=1
Tk
X
λk,t = 1,
k = 1, . . . , K
t=1
λk,t ∈ {0, 1},
Jim Luedtke (UW-Madison)
t = 1, . . . , Tk , k = 1, . . . , K
Decomposition Methods
Lecture Notes
31 / 47
Dantzig-Wolfe Reformulation and Column Generation
Dantzig-Wolfe Reformulation
Questions about Dantzig-Wolfe formulation
1
How can we solve the linear programming relaxation? (Since it may
have exponentially many variables)
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
32 / 47
Dantzig-Wolfe Reformulation and Column Generation
Dantzig-Wolfe Reformulation
Questions about Dantzig-Wolfe formulation
1
How can we solve the linear programming relaxation? (Since it may
have exponentially many variables)
2
What is the strength of the LP relaxation, compared to the original
formulation?
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
32 / 47
Dantzig-Wolfe Reformulation and Column Generation
Dantzig-Wolfe Reformulation
Questions about Dantzig-Wolfe formulation
1
How can we solve the linear programming relaxation? (Since it may
have exponentially many variables)
2
What is the strength of the LP relaxation, compared to the original
formulation?
3
If we can solve the LP relaxation, then how should we use it in
branch-and-bound?
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
32 / 47
Dantzig-Wolfe Reformulation and Column Generation
Dantzig-Wolfe Reformulation
Questions about Dantzig-Wolfe formulation
1
How can we solve the linear programming relaxation? (Since it may
have exponentially many variables)
2
What is the strength of the LP relaxation, compared to the original
formulation?
3
If we can solve the LP relaxation, then how should we use it in
branch-and-bound?
We’ve already seen the answer to the first two questions!
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
32 / 47
Dantzig-Wolfe Reformulation and Column Generation
Dantzig-Wolfe Reformulation
Let’s revisit our IP
z IP = max
K
X
k=1
K
X
c>
k xk
Dk xk ≤ d,
k=1
xk ∈ Xk , k = 1, . . . , K
For u ∈ Rm
+ , Lagrangian subproblem is:
K
K
nX
o
X
>
>
z(u) = max
ck xk + u d −
Dk xk : xk ∈ Xk , k = 1, . . . , K
k=1
Jim Luedtke (UW-Madison)
k=1
Decomposition Methods
Lecture Notes
33 / 47
Dantzig-Wolfe Reformulation and Column Generation
Dantzig-Wolfe Reformulation
Let’s revisit our IP
z IP = max
K
X
k=1
K
X
c>
k xk
Dk xk ≤ d,
k=1
xk ∈ Xk , k = 1, . . . , K
For u ∈ Rm
+ , Lagrangian subproblem is:
K
K
nX
o
X
>
>
z(u) = max
ck xk + u d −
Dk xk : xk ∈ Xk , k = 1, . . . , K
k=1
K
X
= u> d +
k=1
n
o
>
max (c>
−
u
D
)x
:
x
∈
X
k
k
k
k=1
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
33 / 47
Dantzig-Wolfe Reformulation and Column Generation
Dantzig-Wolfe Reformulation
Lagrangian Relaxation, cont’d
wLD = min z(u)
u≥0
= min u> d +
u≥0
= min u> d +
ηk +
K
X
n
o
>
max (c>
−
u
D
)x
:
x
∈
X
k
k
k
k=1
K
X
ηk
k=1
u> Dk xk,t
k,t
≥ c>
kx ,
t = 1, . . . , Tk , k = 1, . . . , K
u≥0
where xk,t : t = 1, . . . , Tk are the extreme points of conv(Xk ) for all k
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
34 / 47
Dantzig-Wolfe Reformulation and Column Generation
Dantzig-Wolfe Reformulation
Lagrangian Relaxation: Take the Dual
wLD = min u> d +
ηk +
= max
K
X
ηk
k=1
>
u Dk xk,t
Tk
K X
X
k=1 t=1
Tk
K X
X
k=1 t=1
Tk
X
k,t
≥ c>
kx ,
t = 1, . . . , Tk , k = 1, . . . , K
k,t
c>
λk,t
kx
Dk xk,t λk,t ≤ d,
λk,t = 1,
k = 1, . . . , K,
t=1
λk,t ≥ 0,
Jim Luedtke (UW-Madison)
t = 1, . . . , Tk , k = 1, . . . , K
Decomposition Methods
Lecture Notes
35 / 47
Dantzig-Wolfe Reformulation and Column Generation
Dantzig-Wolfe Reformulation
Dual of the LP Formulation
w
LD
= max
Tk
K X
X
k=1 t=1
Tk
K X
X
k=1 t=1
Tk
X
k,t
c>
λk,t
kx
Dk xk,t λk,t ≤ d,
λk,t = 1,
k = 1, . . . , K,
t=1
λk,t ≥ 0,
t = 1, . . . , Tk , k = 1, . . . , K
Does this look familiar?
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
36 / 47
Dantzig-Wolfe Reformulation and Column Generation
Dantzig-Wolfe Reformulation
Questions About Dantzig-Wolfe Formulation
1
How can we solve the linear programming relaxation?
Column generation: Adding a variable corresponds to adding a
constraint in Lagrangian Dual problem
“Pricing problem” will be identical to Lagrangian relaxation subproblem
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
37 / 47
Dantzig-Wolfe Reformulation and Column Generation
Dantzig-Wolfe Reformulation
Questions About Dantzig-Wolfe Formulation
1
How can we solve the linear programming relaxation?
Column generation: Adding a variable corresponds to adding a
constraint in Lagrangian Dual problem
“Pricing problem” will be identical to Lagrangian relaxation subproblem
2
What is the strength of the LP relaxation, compared to the original
formulation?
Exactly equal to the Lagrangian relaxation bound
Always at least as good as original LP relaxation
Exactly equal to LP relaxation if conv(Xk ) = {x ∈ Rnk : Ak x ≤ bk }
for all k
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
37 / 47
Dantzig-Wolfe Reformulation and Column Generation
Column Generation
Formulating With a Huge Numbers of Variables
DW reformulation ⇒ Formulation with a huge number of variables as
a reformulation of a given problem
In many applications, it is conceptually easier to start with a
formulation having a huge number of variables
Can also be useful in avoiding symmetry in alternative formulations
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
38 / 47
Dantzig-Wolfe Reformulation and Column Generation
Column Generation
Example: Cutting Stock Problem
A steel company makes a set of products I
Product i ∈ I has width wi > 0
Product i ∈ I has demand bi > 0
Products are made by cutting them from a roll of length L
Multiple products can be cut from a roll
E.g., L = 10, w1 = 4, w2 = 3
One roll can make one product 1 and two of product 2
(4 + 2 ∗ 3 = 10)
Let’s formulate an IP to minimize number of rolls used
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
39 / 47
Dantzig-Wolfe Reformulation and Column Generation
Column Generation
Example: Cutting Stock Problem
A steel company makes a set of products I
Product i ∈ I has width wi > 0
Product i ∈ I has demand bi > 0
New set: P - Set of all possible “cutting patterns”
aip = number of product i made by pattern p ∈ P
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
40 / 47
Dantzig-Wolfe Reformulation and Column Generation
Column Generation
Example: Cutting Stock Problem
A steel company makes a set of products I
Product i ∈ I has width wi > 0
Product i ∈ I has demand bi > 0
New set: P - Set of all possible “cutting patterns”
aip = number of product i made by pattern p ∈ P
Let xp = number of pattern p to cut
min
X
xp
p∈P
s.t.
X
aip xp ≥ bi ,
i∈I
p∈P
xp ∈ Z+ ,
Jim Luedtke (UW-Madison)
p∈P
Decomposition Methods
Lecture Notes
40 / 47
Dantzig-Wolfe Reformulation and Column Generation
Column Generation
Column Generation for Cutting Stock
Let P 0 be a given subset of cutting patterns
E.g., include all patterns producing just one product to ensure
feasibilty
Restricted Master LP
Dual
X
X
min
xp
max
bi πi
s.t.
p∈P 0
i∈I
X
X
aip xp ≥ bi ,
i∈I
s.t.
p∈P 0
aip πi ≤ 1,
p ∈ P0
i∈I
xp ≥ 0, p ∈ P 0
xp ≥ 0, p ∈ P
Let (x̂, π̂) be an optimal primal/dual solution. x̂ optimal ⇔ π̂ feasible
0
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
41 / 47
Branch-and-Price
Outline
1
Lagrangian Relaxation
2
Dantzig-Wolfe Reformulation and Column Generation
3
Branch-and-Price
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
42 / 47
Branch-and-Price
Branch-and-Price
What to do after solving the LP relaxation by column generation?
Possibilities:
Heuristics: E.g., start MIP solve with this subset of columns
If find a feasible solution close to column generation LP bound, you
may be happy
If you need an optimal solution (or better than you have found) . . .
Branch-and-price: Do column generation at every node!
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
43 / 47
Branch-and-Price
Branch-and-Price
Difficulty in branching
Fix λk,t = 1, we choose xk,t completely – very restrictive
Fix λk,t = 0, exclude just the one solution xk,t !
Fixing λk,t = 0 also destroys subproblem structure:
Must exclude the solution xk :
max (ck − u> Dk )x : x ∈ Xk , x 6= xk
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
44 / 47
Branch-and-Price
Branch-and-Price: Alternative Branching
Branch on the “original” problem variables xk
E.g. fix xki = 0
In LP Master Problem: Delete columns xk,t with xk,t
i 6= 0
Fix λk,t = 0 for such columns if already generated
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
45 / 47
Branch-and-Price
Branch-and-Price: Alternative Branching
Branch on the “original” problem variables xk
E.g. fix xki = 0
In LP Master Problem: Delete columns xk,t with xk,t
i 6= 0
Fix λk,t = 0 for such columns if already generated
Avoid generating new columns with xk,t
i 6= 0: In subproblem k, fix
xi = 0
max (ck − u> Dk )> x : x ∈ Xk , xi = 0
Often preserves subproblem structure
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
45 / 47
Branch-and-Price
Branch-and-Price: Alternative Branching
Branch on the “original” problem variables xk
E.g. fix xki = 0
In LP Master Problem: Delete columns xk,t with xk,t
i 6= 0
Fix λk,t = 0 for such columns if already generated
Avoid generating new columns with xk,t
i 6= 0: In subproblem k, fix
xi = 0
max (ck − u> Dk )> x : x ∈ Xk , xi = 0
Often preserves subproblem structure
Similarly for alternative branch xki = 1
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
45 / 47
Branch-and-Price
Branching in Cutting Stock
Branching is more difficult in cutting stock formulation
Multiple patterns being selected
Cannot simply say a pattern must make ≤ k or ≥ k + 1 of a product
Would enforce that for all patterns
Must branch by adding constraints (not just bounds on variables)
LP relaxations get larger
Pricing problems become more complicated
We’ll skip these details
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
46 / 47
Branch-and-Price
How to Implement Branch-and-Price?
Solving the LP master problem is not too hard to implement
Good news: in many applications it often yields a provably
near-optimal solution!
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
47 / 47
Branch-and-Price
How to Implement Branch-and-Price?
Solving the LP master problem is not too hard to implement
Good news: in many applications it often yields a provably
near-optimal solution!
Commercial solvers don’t allow you to add variables in the
branch-and-bound tree
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
47 / 47
Branch-and-Price
How to Implement Branch-and-Price?
Solving the LP master problem is not too hard to implement
Good news: in many applications it often yields a provably
near-optimal solution!
Commercial solvers don’t allow you to add variables in the
branch-and-bound tree
SCIP (scip.zib.de - free for academics) supports branch-and-price
Generic Column Generation (www.or.rwth-aachen.de/gcg/) based on SCIP
Some open-source frameworks are available at www.coin-or.org
BCP: Branch-cut-price
CHiPPS and DIP – also supports Lagrangian relaxation
ABACUS
Jim Luedtke (UW-Madison)
Decomposition Methods
Lecture Notes
47 / 47