4.4 Geometry and solution of Lagrangean duals
ο± Solving Lagrangean dual: ππ· = max π(π)
πβ₯0
π
π π = min(πβ²π₯ π + πβ²(π β π΄π₯ )) (unless π π = ββ)
πβπΎ
π
Let ππ = π β π΄π₯
and βπ = πβ²π₯ π . Then
π π = min(βπ + ππβ² π) (piecewise linear and concave)
πβπΎ
ο± π(π) is nondifferentiable, but mimic the idea for differentiable function
ππ‘=1 = ππ‘ + ππ‘ π»π ππ‘ ,
π‘ = 1,2, β¦
ο± Prop 4.5: A function π: π
π β π
is concave if and only if for any π₯ β β π
π ,
there exists a vector π β π
π such that
π π₯ β€ π π₯ β + π β² π₯ β π₯ β ,
for all π₯ β π
π .
ο± The vector π is called the gradient of π at π₯ β (if differentiable). Generalize the
above property to nondifferentiable functions.
Integer Programming 2015
1
ο± Def 4.3: Let π be a concave function. A vector π such that
π π₯ β€ π π₯ β + π β² π₯ β π₯ β ,
for all π₯ β π
π , is called a subgradient of π at π₯ β . The set of all subgradients of
π at π₯ β is denoted as ππ(π₯ β ) and is called the subdifferential of π at π₯ β .
ο± Prop: The subdifferential ππ(π₯ β ) is closed and convex.
Pf) ππ(π₯ β ) is the intersection of closed half spaces.
ο± If π is differentiable at π₯ β , then ππ π₯ β = π»π π₯ β .
For convex functions, subgradient is defined as vector π which satisfies
π π₯ β₯ π π₯ β + π β²(π₯ β π₯ β ).
ο± Geometric intuition: Think in π
π+1 space
π π₯ β€ π π₯ β + π β²(π₯ β π₯ β )
βΊ π β²π₯ β β π π₯ β β€ π β² π₯ β π(π₯)
βΊ (π , β1)β²(π₯ β , π π₯ β ) β€ (π , β1)β²(π₯, π π₯ )
βΊ
π , β1 β² { π₯, π π₯
Integer Programming 2015
β π₯β, π π₯β } β₯ 0
2
ο± Picture:
π , β1 β² { π₯, π π₯
β π₯β, π π₯β } β₯ 0
π π₯ = π π₯ β + π β²(π₯ β π₯ β )
π(π₯)
π₯, π π₯
β (π₯ β , π π₯ β )
(π₯, π π₯ )
(π₯ β , π π₯ β )
β1
π₯β
Integer Programming 2015
π
(π , β1)
π₯
π₯
3
π , β1 β² { π₯, π π₯
ο± For convex functions:
β π₯ β, π π₯β } β€ 0
π π₯ = π π₯ β + π β²(π₯ β π₯ β )
π(π₯)
π₯, π π₯
β (π₯ β , π π₯ β )
(π₯ β , π π₯ β )
(π₯, π π₯ )
β1
π₯
Integer Programming 2015
π₯β
π
(π , β1)
π₯
4
ο± Prop 4.6: Let π: π
π β π
be a concave function. A vector π₯ β maximizes π
over π
π if and only if 0 β ππ π₯ β .
Pf) A vector π₯ β maximizes π over π
π if and only if
π π₯ β€ π π₯ β = π π₯ β + 0β² π₯ β π₯ β , for all π₯ β π
π ,
which is equivalent to 0 β ππ π₯ β .
ο± Prop 4.7: Let
π π = min (βπ + ππβ² π)
πβπΎ
πΈ π = π β πΎ: π π = βπ + ππβ² π .
Then, for every πβ β₯ 0 the following relations hold:
(a) For every π β πΈ πβ , ππβ² is a subgradient of the function π(ο) at πβ .
(b) ππ πβ = conv ππ : π β πΈ πβ , i.e., a vector π is a subgradient of the
function π(ο) at πβ if and only if π is a convex combination of the vectors
ππ , π β πΈ πβ .
Integer Programming 2015
5
Pf) (a) By definition of π(π) we have for every π β πΈ(πβ )
π π β€ βπ + ππβ² π = βπ + ππβ² πβ + ππβ² π β πβ = π πβ + ππβ² π β πβ .
(b) We have shown that {ππ : π β πΈ πβ } β ππ(πβ ) and since a convex
combination of two subgradients is also a subgradient , it follows that
conv ππ : π β πΈ πβ
β ππ πβ .
Assume that there is π β ππ πβ , such that π β conv( ππ : π β πΈ πβ ) and
derive a contradiction. By the separating hyperplane theorem, there exists a
vector π and a scalar ο§, such that
πβ²ππ β₯ πΎ,
for all π β πΈ(πβ )
(4.29)
πβ² π < πΎ
Since π πβ < βπ + ππβ² πβ , for all π β πΈ πβ , then for sufficiently small π > 0,
we have, π πβ + ππ = βπ + ππβ² πβ + ππ , for some π β πΈ(πβ )
Since π πβ = βπ + ππβ² πβ , we obtain that
π πβ + ππ β π πβ = πππβ² π, for some π β πΈ(πβ ).
Integer Programming 2015
6
(continued)
From (4.29), we have
π πβ + ππ β π πβ β₯ ππΎ.
Since π β ππ πβ , we have
π πβ + ππ β€ π πβ + ππ β² π.
From (4.29), we have
π πβ + ππ β π πβ < ππΎ,
contradicting (4.30).
οΏ
Integer Programming 2015
(4.30)
(defβn of subgradient)
( πβ² π < πΎ )
7
ο± Subgradient algorithm
ο± For differentiable functions, the direction π»π(π₯) is the direction of steepest
ascent of π at π₯. But, for nondifferentiable functions, the direction of
subgradient may not be an ascent direction. However, moving along the
direction of a subgradient, we can move closer to the maximum point of a
concave function.
Integer Programming 2015
8
ο± Ex: π π₯1 , π₯2 = min {βπ₯1 , π₯1 β 2π₯2 , π₯1 + 2π₯2 } : concave function
direction of subgradient may not be an ascent direction for π, but we can
move closer to the maximum point.
π₯2
π(π₯) = β2
π 2 = (1,2)
π 1 π₯ β π₯ β = 0
π(π₯) = β1
π₯ β = (β2,0)
π 1 = (1, β2)
Integer Programming 2015
π(π₯) = 0
π₯1
π 2 π₯ β π₯ β = 0
9
ο± The subgradient optimization algorithm
Input: A nondifferentiable concave function π(π).
Output: A maximizer of π(π) subject to π β₯ 0.
Algorithm:
1. Choose a starting point π1 β₯ 0; let π‘ = 1.
2. Given ππ‘ , check whether 0 β ππ(ππ‘ ). If so, then ππ‘ is optimal and the algorithm
terminates. Else, choose a subgradient π π‘ of the function π(ππ‘ ). (ππ β π β π΄π₯ π )
3. Let πππ‘+1 = max{πππ‘ + ππ‘ π ππ‘ , 0} (projection), where ππ‘ is a positive step size
parameter. Increment π‘ and go to Step 2.
Integer Programming 2015
10
ο± Choosing the step lengths:
ο± Thm :
(a) If
β
π‘=1 ππ‘
β β, and lim ππ‘ = 0, then π(ππ‘ ) β ππ· the optimal value of
π‘ββ
π(ο).
(b) If ππ‘ = π0 πΌ π‘ , π‘ = 1,2, β¦ , for some parameter 0 < πΌ < 1, then π(ππ‘ ) β ππ·
if π0 and ο‘ are sufficiently large.
(c) If ππ‘ = π ππ·β β π ππ‘ / π π‘ 2 , where 0 < π < 2, and ππ·β β€ ππ· , then
π ππ‘ β ππ·β , or the algorithm finds ππ‘ with ππ·β β€ π ππ‘ β€ ππ· , for some finite π‘.
ο± (a) guarantees convergence, but slow (e.g., ππ‘ = 1/π‘ )
(b) In practice, may use halving the value of ππ‘ after π£ iterations
(c) ππ·β typically unknown, may use a good primal upper bound in place of ππ·β .
If not converge, decrease ππ·β .
ο± In practice, 0 β ππ(ππ‘ ) is rarely met. Usually, only find approximate optimal
solution fast and resort to B-and-B. Convergence is not monotone.
If ππ· = ππΏπ , solving LP relaxation may give better results (monotone
convergence).
11
Integer Programming 2015
ο± Ex: Traveling salesman problem:
For TSP, we dualize degree constraints for nodes except node 1.
π π = min
( = min
πβπΈ (ππ
β ππ β ππ )π₯π + 2
πβπΈ ππ π₯π
πβπΏ( 1 ) π₯π
πβπΈ(π) π₯π
+
πβπβ{1} ππ (2 β
πβπΏ π
π₯π ) )
= 2,
β€ π β 1,
πβπΈ(πβ 1 ) π₯π
πβπβ{1} ππ
π β π β 1 , π β β
, π β 1 ,
= π β 2,
π₯π β 0,1 .
step direction is
ππ‘+1
= ππ‘π + ππ‘ (2 β
π
πβπΏ π
π₯π ππ‘ ) ( π π‘ = π β π΄π₯ π )
step size using rule (c) is
ππ‘ = π ππ·β β π ππ‘ /
π‘ 2
πβπΏ( π ) π₯π (π ))
π΄π₯(ππ‘ ) 2 )
πβπ(2 β
(ππ‘ = π ππ·β β π ππ‘ / π β
ο± Note that π βth coordinate of subgradient direction is two minus the number of
edges incident to node π in the optimal one-tree. We do not have π β₯ 0 here
since the dualized constraints are equalities.
Integer Programming 2015
12
ο± Lagrangean heuristic and variable fixing:
ο± The solution obtained from solving the Lagrangian relaxation may not be
feasible to IP, but it can be close to a feasible solution to IP.
Obtain a feasible solution to IP using heuristic procedures. Then we may
obtain a good upper bound on optimal value.
Also called βprimal heuristicβ
ο± May fix values of some variables using information from Lagrangian
relaxation (refer to W, pp177-178).
ο± Branching variable selection? (solutions to Lagrangian relaxation are integer
valued)
Integer Programming 2015
13
ο± Choosing a Lagrangean dual:
How to determine the constraints to relax?
οStrength of the Lagrangean dual bound
οEase of solution of π(π)
οEase of solution of Lagrangean dual problem ππ· = max π(π)
πβ₯0
ο± Ex: Generalized Assignment Problem (max problem)(refer to W, pp. 179):
π§ = max
Integer Programming 2015
π
π
π=1 π=1 πππ π₯ππ
π
π=1 π₯ππ β€ 1
π
π=1 πππ π₯ππ β€ ππ
π₯ β π΅ππ
for π = 1, β¦ , π
for π = 1, β¦ , π
14
ο± Dualize both sets of constraints:
ππ·1 = min π1 (π’, π£) , where
π’β₯0,π£β₯0
π1 π’, π£ = max
π
π=1
π
π=1
πππ β π’π β πππ π£π π₯ππ +
π
π=1 π’π
+
π
π=1 π£π ππ
π₯ β π΅ππ
ο± Dualize first set of assignment constraints:
ππ·2 = min π 2 (π’) , where
π’β₯0
π 2 π’ = max
π
π
π=1 π=1(πππ β π’π )π₯ππ
π
π=1 πππ π₯ππ β€ ππ
ππ
+
π
π=1 π’π
for π = 1, β¦ , π
π₯βπ΅
ο± Dualize the knapsack constraints:
ππ·3 = min π 3 π£ , where
π£β₯0
π 3 π£ = max
Integer Programming 2015
π
π
π=1 π=1(πππ β πππ π£π )π₯ππ
π
π=1 π₯ππ β€ 1
π₯ β π΅ππ
+
π
π=1 π£π ππ
for π = 1, β¦ , π
15
ο± ππ·1 = ππ·3 = ππΏπ
For each π,
conv{π₯:
= conv{π₯:
π
π=1 π₯ππ β€ 1, π₯ππ β {0,1} for π
π
π=1 π₯ππ β€ 1, 0 β€ π₯ππ β€ 1 for
= 1, β¦ , π}
π = 1, β¦ , π}
π1 π’, π£ , π 3 (π£) can be solved by inspection. Calculating ππ·3 look easier than
calculating ππ·1 since there are π dual variables compared to π + π for ππ·1 .
ο± ππ·2 β€ ππΏπ
ο± To find π 2 π’ , we need to solve π 0-1 knapsack problems. Also note that the
information that we may obtain while solving the knapsack cannot be stored
and used for subsequent optimization.
Integer Programming 2015
16
ο± May solve the Lagrangean dual by constraint generation (NW p411-412):
ο± Recall that
ππ· = maximize π¦
subject to π¦ + πβ² π΄π₯ π β π β€ πβ²π₯ π ,
πβ² π΄π€ π β€ πβ²π€ π ,
πβπΎ
πβπ½
πβ₯0
ο± Given π¦ β , πβ , with πβ β₯ 0, calculate
π πβ =
min (π β² π₯ + πβ β² π β π΄π₯ )
π₯βconv(πΉ)
(min π β² β πβ β²π΄ π₯ + πβ² π)
If π¦ β β€ π πβ , stop π¦ β = π πβ , π = πβ is an optimal solution
If π¦ β > π πβ , an inequality is violated
οπ πβ β ββ, β ray π€ π s.t. π β² β πβ β²π΄ π€ π < 0, hence πβ² π΄π€ π β€ πβ²π€ π is violated
ο π πβ < β, β extreme point π₯ π s.t. π πβ = πβ²π₯ π + πβ β² π β π΄π₯ π . Since π¦ β >
π πβ , π¦ + πβ²(π΄π₯ π β π) β€ πβ²π₯ π is violated.
ο± Note that max/min is interchanged in NW.
Integer Programming 2015
17
Nonlinear Optimization Problems
ο± Geometric understanding of the strength of the bounds provided by the
Lagrangean dual
ο± Let π: π
π β¦ π
and π: π
π β¦ π
π be continuous functions. Let β± β π
π .
minimize π(π₯)
subject to π(π₯) β€ 0,
(4.31)
π₯ β β±.
Let ππ be the optimal cost. Consider Lagrangean function
π π = ππππ₯ββ± {π π₯ + πβ² π π₯ }.
ο± For all π β₯ 0, π(π) β€ ππ and π(π) is a concave function.
Lagrangean dual: ππ· = πππ₯πβ₯0 π(π).
ο± Let π = { π¦, π§ β² β π
π+1 : π¦ β₯ (=)π π₯ , π§ β₯ (=)π π₯ , for all π₯ β β±}.
Problem (4.31) can be restated as
minimize π¦
subject to (π¦, 0)β² β π. ( π¦, π§ β² β π, π§ β€ 0 )
Integer Programming 2015
18
ο±
π, (π¦)
ππππ£(π)
ππ
π π = π + ππ
ππ·
π, (π§)
Figure 4.7: The geometric interpretation of the Lagrangean dual
Integer Programming 2015
19
ο± Given that π π = ππππ₯ββ± π π₯ + πβ² π(π₯), we have for a fixed ο¬ and for all
π₯ β β± that
π π β€ π¦ + πβ² π§, β(π¦, π§)β² β π.
Geometrically, this means that the hyperplane π π = π¦ + πβ² π§ lies below the
set Y.
For π§ = 0, we obtain π¦ = π(π), that is π(π) is the intercept of the hyperplane
π π = π¦ + πβ² π§ with the vertical axis.
To maximize π(π), this corresponds to finding a hyperplane π π = π¦ + πβ² π§,
which lies below the set Y, such that the intercept of the hyperplane π π =
π¦ + πβ² π§ with the vertical axis is the largest.
ο± Thm 4.10: The value ππ· of the Lagrangean dual is equal to the value of the
following optimization problem:
minimize π¦
subject to (π¦, 0)β² β ππππ£(π). ( π¦, π§ β ππππ£ π , π§ β€ 0 )
Pf) not given here.
Integer Programming 2015
20
ο± Ex 4.7: Ex 4.3 revisited
π π₯1 , π₯2 = 3π₯1 β π₯2 and π π₯1 , π₯2 = β1 β π₯1 + π₯2 ,
β± = {(1, 0), (2,0), (1, 1), (2, 1), (0, 2), (1, 2), (2, 2), (1, 3), (2, 3)}
π, π = {(3, -2), (6, -3), (2, -1), (5, -2), (-2, 1), (1, 0), (4, -1), (0, 1), (3, 0)}
π
ππππ£(π)
ππ = 1
ππ· = β13
Integer Programming 2015
duality gap
π
21
© Copyright 2026 Paperzz