Duality
𝑛
Maximize
Primal:
Subject to
𝑗=1
𝑐𝑗 𝑥𝑗
𝑛
𝑗=1
𝑎𝑖𝑗 𝑥𝑗 ≤ 𝑏𝑖
𝑖 = 1,2, … , 𝑚.
𝑥𝑗 ≥ 0
𝑗 = 1,2, … , 𝑛.
𝑚
Minimize
Dual:
Subject to
𝑖=1
𝑏𝑖 𝑦𝑖
𝑚
𝑖=1
𝑎𝑖𝑗 𝑦𝑗 ≥ 𝑐𝑗
𝑗 = 1,2, … , 𝑛.
𝑦𝑖 ≥ 0
𝑖 = 1,2, … , 𝑚.
Duality
Primal:
Dual:
Maximize
𝑐⊺𝑥
Subject to
𝐴𝑥 ≤ 𝑏
𝑥≥0
Minimize
𝑏⊺𝑦
Subject to
𝐴⊺ 𝑦 ≥ 𝑐
𝑦≥0
The Duality Theorem
Primal P:
Dual D:
Maximize 𝑐 ⊺ 𝑥 subject to 𝐴𝑥 ≤ 𝑏, 𝑥 ≥ 0.
Minimize 𝑏 ⊺ 𝑦 subject to 𝐴⊺ 𝑦 ≥ 𝑐, 𝑦 ≥ 0.
Weak Duality Theorem:
If 𝑥 is feasible for P and 𝑦 is feasible for D, then:
𝑐 ⊺ 𝑥 ≤ 𝑏⊺ 𝑦
(Strong) Dualiy Theorem:
If 𝑥 ∗ is optimal for P and 𝑦 ∗ is optimal for D, then:
𝑐 ⊺𝑥 ∗ = 𝑏⊺ 𝑦∗
Finding Optimal Dual Solution from
Otimal Primal Dictionary
Primal P:
Maximize 𝑐 ⊺ 𝑥 subject to 𝐴𝑥 ≤ 𝑏, 𝑥 ≥ 0.
Solve P using the two phase Simplex method, obtaining
optimal solution 𝑥 ∗ . First row of final dictionary:
𝑧 = 𝑧∗ +
𝑛
𝑗=1
𝑐𝑗∗ 𝑥𝑗 +
𝑚
𝑖=1
Let 𝑦𝑖∗ = −𝑑𝑖∗ , for 𝑖 = 1,2, … , 𝑚.
Then:
1. 𝑦 ∗ is a feasible solution to dual D.
2. 𝑏⊺ 𝑦 ∗ = 𝑧 ∗
𝑑𝑖∗ 𝑥𝑛+𝑖
Explaining the magic
Primal
Dual
Maximize
5𝑥1 + 4𝑥2 + 3𝑥3
Subject to
2𝑥1 + 3𝑥2 + 𝑥3
4𝑥1 + 𝑥2 + 2𝑥3
3𝑥1 + 4𝑥2 + 2𝑥3
𝑥1 , 𝑥2 , 𝑥3
Minimize
5𝑦1 + 11𝑦2 + 8𝑦3
Subject to
2𝑦1 + 4𝑦2 + 3𝑦3
3𝑦1 + 𝑦2 + 4𝑥3
𝑦1 + 2𝑦2 + 2𝑦3
𝑥1 , 𝑥2 , 𝑥3
≤ 5
≤ 11
≤8
≥0
≥ 5
≥4
≥3
≥0
Pivoting primal and dual dictionary
Primal:
4𝑥2
3𝑥2
𝑥2
4𝑥2
+
−
−
−
3𝑥3
𝑥3
2𝑥3
2𝑥3
𝑤 =
− 5𝑦1 − 11𝑦2
(not feasible!)
𝑦4 = −5 + 2𝑦1 + 4𝑦2
𝑦5 = −4 + 3𝑦1 +
𝑦2
𝑦6 = −3 + 𝑦1 + 2𝑦2
−
+
+
+
8𝑦3
3𝑦3
4𝑦3
2𝑦3
Dual:
𝑧 =
𝑥4 = 5 −
𝑥5 = 11 −
𝑥6 = 8 −
5𝑥1
2𝑥1
4𝑥1
3𝑥1
+
−
−
−
Pivoting 𝑥1 and 𝑥4 (and 𝑦4 and 𝑦1 )
Primal:
𝑧 =
𝑥1 =
𝑥5 =
𝑥6 =
25
2
5
2
1
1
2
25
2
7
2
1
−
2
5
2
−
−
+
+
Dual:
𝑤 = −
−
(still not
feasible!)
𝑦5 =
+
𝑥6 =
𝑦1 =
+
+
7
𝑥2 +
2
3
𝑥 −
2 2
5𝑥2
1
𝑥2 −
2
5
𝑦4 −
2
3
𝑦4 −
2
1
𝑦
2 4
1
𝑦4 −
2
1
𝑥
2 3
1
𝑥
2 3
−
−
+
1
𝑥
2 3
+
𝑦2
−
5𝑦2
−
+
2𝑦2
−
5
𝑥
2 4
1
𝑥
2 4
2𝑥4
3
𝑥
2 4
1
𝑦
2 3
1
𝑦
2 3
1
𝑦
2 3
3
𝑦
2 3
Pivoting 𝑥3 and 𝑥6 (and 𝑦6 and 𝑦3 )
Primal:
optimal!
Dual:
Feasible!
(and optimal)
𝑧 = 13 − 3𝑥2
𝑥1 = 2 − 2𝑥2
𝑥5 = 1 + 5𝑥2
𝑥3 = 1 + 𝑥2
𝑤 = −13 −
𝑦5 =
3 +
𝑦1 =
1 +
𝑦3 =
1 −
−
−
+
+
𝑥4 −
2𝑥4 +
2𝑥4
3𝑥4 −
2𝑦4 − 𝑦2
2𝑦4 − 5𝑦2
2𝑦4 − 2𝑦2
𝑦4
𝑥6
𝑥6
2𝑥6
− 𝑦6
− 𝑦6
− 3𝑦6
+ 2𝑦6
Consequence of Strong duality
• Software solving LP programs to optimality
can easily be checked.
• Give the software the primal program as well
as the dual program. The solution to the dual
problem is a certificate that the solution to
the primal program is optimal.
Linear Inequalities Problem
Input:
𝐴 ∈ 𝑅𝑚×𝑛 , 𝑏 ∈ 𝑅𝑚
Output:
If 𝑥 ∈ 𝑅𝑛 ∣ 𝐴𝑥 ≤ 𝑏 is empty, report Infeasible,
otherwise output 𝑥 such that 𝐴𝑥 ≤ b .
Linear Programming
Input:
𝐴 ∈ 𝑅𝑚×𝑛 , 𝑏 ∈ 𝑅𝑚 , 𝑐 ∈ 𝑅𝑛
Output:
𝑥 ∈ 𝐹 maximizing 𝑐 ⊺ 𝑥, where 𝐹 = 𝑥 ∈ 𝑅𝑛 ∣ 𝐴𝑥 ≤
Consequence of Strong duality
• Solving linear programs to optimality is as
easy as solving a system of linear inequalities:
1. Given LP P, check if P is feasible using LI
algorithm. Otherwise report Infeasible.
2. Construct dual D of P. Use LI algorithm to find
(𝑥, 𝑦) so that 𝐴𝑥 ≤ 𝑏, 𝑥 ≥ 0,
𝐴⊺ 𝑦 ≥ 𝑐, 𝑦 ≥ 0, and 𝑐 ⊺ 𝑥 = 𝑏 ⊺ 𝑦. If no such
(𝑥, 𝑦) exist, report Unbounded. Otherwise
return 𝑥.
Complementary Slackness
Primal P:
Dual D:
Maximize 𝑐 ⊺ 𝑥 subject to 𝐴𝑥 ≤ 𝑏, 𝑥 ≥ 0.
Minimize 𝑏 ⊺ 𝑦 subject to 𝐴⊺ 𝑦 ≥ 𝑐, 𝑦 ≥ 0.
Suppose 𝑥 ∗ and 𝑦 ∗ are a pair of optimal solutions to P and
D. By strong duality:
𝑐 ⊺ 𝑥 ∗ = 𝑏⊺ y∗
Consider the proof of weak duality:
𝑐 ⊺ 𝑥 ≤ 𝑦 ⊺ 𝐴 𝑥 = 𝑦 ⊺ 𝐴𝑥 ≤ 𝑦 ⊺ 𝑏
For 𝑥 ∗ and 𝑦 ∗ the inequalities must be equalities.
𝑛
𝑛
𝑚
𝑚
𝑐𝑗 𝑥𝑗 =
𝑗=1
𝑛
𝑚
𝑎𝑖𝑗 𝑦𝑖 𝑥𝑗 =
𝑗=1
𝑖=1
𝑎𝑖𝑗 𝑥𝑗 𝑦𝑖 =
𝑖=1
𝑗=1
𝑏𝑖 𝑦𝑖
𝑖=1
Consider last part:
𝑚
𝑛
𝑚
𝑎𝑖𝑗 𝑥𝑗 𝑦𝑖 =
𝑖=1
𝑗=1
𝑏𝑖 𝑦𝑖
𝑖=1
Rewrite as:
𝑚
𝑛
𝑏𝑖 −
𝑖=1
𝑎𝑖𝑗 𝑥𝑗 𝑦𝑖 = 0
𝑗=1
This is a sum of non-negative numbers (Why?)
summing to 0.
𝑚
𝑛
𝑏𝑖 −
𝑖=1
𝑎𝑖𝑗 𝑥𝑗 𝑦𝑖 = 0
𝑗=1
• All terms must be 0.
• Hence, for all 𝑖 = 1,2, … , 𝑚 (at least) one of
the following must hold:
a) 𝑏𝑖 −
b)
𝑦𝑖 = 0
𝑛
𝑗=1 𝑎𝑖𝑗 𝑥𝑗
= 0 (i.e. slack 𝑥𝑛+𝑖 = 0).
The Complementary Slackness Property
Primal P:
Dual D:
Maximize 𝑐 ⊺ 𝑥 subject to 𝐴𝑥 ≤ 𝑏, 𝑥 ≥ 0.
Minimize 𝑏 ⊺ 𝑦 subject to 𝐴⊺ 𝑦 ≥ 𝑐, 𝑦 ≥ 0.
Solutions 𝑥 and 𝑦 are said to satisfy Complementary Slackness if
and only if:
• for all 𝑖 = 1,2, … , 𝑚 (at least) one of the following holds:
𝑛
a)
𝑗=1 𝑎𝑖𝑗 𝑥𝑗 = 𝑏𝑖 (𝑖th primal constraint has slack 0)
b) 𝑦𝑖 = 0
(𝑖th dual variable is 0)
• for all j= 1,2, … , 𝑛 (at least) one of the following holds:
𝑚
a)
𝑖=1 𝑎𝑖𝑗 𝑦𝑖 = 𝑐𝑗 (𝑗th dual constraint has slack 0)
b) 𝑥𝑗 = 0
(𝑗th primal variable is 0)
Complementary Slackness Theorem
Primal P:
Dual D:
Maximize 𝑐 ⊺ 𝑥 subject to 𝐴𝑥 ≤ 𝑏, 𝑥 ≥ 0.
Minimize 𝑏 ⊺ 𝑦 subject to 𝐴⊺ 𝑦 ≥ 𝑐, 𝑦 ≥ 0.
Theorem
Let 𝑥 and 𝑦 be feasible solutions to P and D. Then:
𝑥 and 𝑦 are both optimal
if and only if
𝑥 and 𝑦 satisfy complementary slackness.
The Dual Simplex Method
• Simple idea: Instead of running the Simplex
Algorithm on the primal problem, run it on the
dual problem.
• Can be usuful as “empiric” number of pivoting
steps of the Simplex Algorithm is roughly ∼
𝑚 log 2 𝑛.
• We can even run the Dual Simplex algorithm on
the primal dictionary.
Dual-Based Phase I Algorithm
• If we change the objective function to
max 0
the dual program is origo feasible!
• The optimal dual solution is feasible in the
primal – this may serve as a replacement for
Phase I.
• We may now proceed as usual in the twophase Simplex algorithm.
We often have feasible origo in dual
program in practice
In many situations the linear program we
formulate is about minimizing cost,
𝑛
min
𝑐𝑗 𝑥𝑗
𝑗=1
which turns into the objective function,
𝑛
max
−𝑐𝑗 𝑥𝑗
𝑗=1
If 𝑐𝑗 ≥ 0 for all 𝑗, then the dual is origo feasible!
Generalized Duality
Primal:
Maximize 𝑐1 𝑥1 + 𝑐2 𝑥2 + 𝑐3 𝑥3
Subject to 𝑦1 : 𝑎11 𝑥1 + 𝑎12 𝑥2 + 𝑎13 𝑥3 ≤ 𝑏1
Dual:
Manimize 𝑏1 𝑦1 + 𝑏2 𝑦2 + 𝑏3 𝑦3
Subject to 𝑥1 : 𝑎11 𝑦1 + 𝑎21 𝑦2 + 𝑎31 𝑦3 ≥ 𝑐1
𝑦2 : 𝑎21 𝑥1 + 𝑎22 𝑥2 + 𝑎23 𝑥3 ≥ 𝑏2
𝑦3 : 𝑎31 𝑥1 + 𝑎32 𝑥2 + 𝑎33 𝑥3 = 𝑏3
𝑥1 ≥ 0, 𝑥2 ≤ 0, 𝑥3 free
𝑥2 : 𝑎12 𝑦1 + 𝑎22 𝑦2 + 𝑎32 𝑦3 ≤ 𝑐2
𝑥3 : 𝑎13 𝑦1 + 𝑎23 𝑦2 + 𝑎33 𝑦3 = 𝑐3
𝑦1 ≥ 0, 𝑦2 ≤ 0, 𝑦3 free
Rules for Taking the Dual
• Constraints in primal corresponds to variables in
dual (and vice versa).
• Coefficients of objective function in primal
corresponds to constants in dual (and vice versa).
Primal (Maximization)
Dual (Minimization)
≤ for constraint
≥0 for variable
≥ for constraint
≤0 for variable
= for constraint
free for variable
≥0 for variable
≥ for constraint
≤0 for variable
≤ for constraint
free for variable
= for constraint