Chap 8. Linear Programming and Game
Theory
KAIST wit Lab
2012. 07. 20
유인철
I. Linear Inequalities
KAIST wit lab
Introduction
KAIST wit lab
 Linear Programming (from wikipedia)
 Mathematical method for determining a way to achieve the best outcome (such as maximum profit or lowest cost) in a given mathematical model for some list of requirements
 A technique for the optimization of a linear objective function, subject to some constraints
 Objective of this chapter
 To see the geometric meaning of linear inequalities
 To find the optimal soloution
 Hyperplane and Halfspace
 Hyperplane :  Halfspace : 3
The Feasible Set and the Cost Function
KAIST wit lab
 Feasible set
 the intersection of halfspaces
 composition of the solutions to • a family of linear inequalities like  Example) 2
4, 0,
0  Fundamental constraint to linear programming
KAIST wit lab
 Problem in linear programming
 Find the point that Cost increases
• lies in the feasible set  The point is feasible
• minimizes or maximize the cost  The point is optimal
 Example : in the previous figure
 Cost : 2
3
 Minimum cost : 2
Cost decreases
∗
3
∗
6,
∗
0
∗
2
• Vector (0,2) is feasible and optimal
 The optimal vector occurs at a corner of the feasible set
 Possible categories 1. The feasible set is empty
2. The cost function is unbounded on the feasible set
3. The cost reaches its minimum (or maximum) on the feasible set
5
Slack Variables
KAIST wit lab
 Slack Variable
 To change the inequality to an equation
• Result : equation + nonnegativity constraints
 Primal problem : 
• Constraints : 0
 Example : in the previous example
• Inequality : 2
4
• Slack variable : 2
• Nonnegativity constraint : 4
0
6
The Diet Problem and Its Dual
KAIST wit lab
 Diet problem in linear programming
 2 sources of protein :
• A pound of peanut butter : a unit of protein
• A steak : 2 unit of protein
 Diet : • at least 4 units are required
• Contains – x pounds of peanut butter
– y steaks
Constraints :
2
4,
0,
0
 Cost :
• A pound of peanut butter : $2
• A steak : $3
the cost of whole diet :
2
3
∴ optimal diet for minimum cost = 2 steaks (
∗
0,
∗
2)
7
KAIST wit lab
 Duality theorem
The minimum in the given “primal problem” equals the maximum in its dual.
 Every linear program has dual.
 Example : Dual problem of previous one
2, 2
 optimal price :
3
∗
0
$1.50
 maximum revenue :4
∗
$6
8
Typical Applications
KAIST wit lab
1. Production Planning : General motors
Chevrolet
Buick
Cadillac
profit
$200
$300
$500
miles per gallon
20
17
14
Assemble duration
1 minute
2 minutes
3 minutes
 Also, the average car must get 18 miles per gallon
What is the maximum profit in 8 hours (480 minutes) ?
9
KAIST wit lab
2. Portfolio Selection
interest
federal bonds
municipals
junk bonds
5%
6%
9%
 We can buy amounts , , not exceeding a total of $100,000
 No more than $20,000can be invested in junk bonds
 The portfolio’s average quality must be no lower than municipals
What is the maximum interest?
10
Problem Set 8.1
KAIST wit lab
.
 Cost : 4
2
1
 Cost : 3
•
2 0,2
12 4,0
 Cost : •
2 0,2
4 4,0
11
II. The Simplex Method
KAIST wit lab
Introduction
KAIST wit lab
 Linear Programming with  n unknowns  m constraints 0
feasible vectors meet conditions
 Minimum problem
 Optimal ∗
 occurs at the point where the planes first touch the feasible set
 Compute ∗ : Find all the corners of the feasible set
 Simplex method
• A systematic way to solve linear programs
13
The Geometry : Movement Along Edges
KAIST wit lab
 Corner in linear algebra
 The meeting point of different planes
given by 1 equation
Subspace of  Possible intersection point of linear programming
 Review the feasible set : the intersection space of
equations (by slack var.)
+
fundamental constraints
0
equations (
,⋯,
)
+
fundamental constraints
equations, but only equations are needed
14
KAIST wit lab
 How equations are chosen for intersection point? 6,
• constraints : 2
2
2
2
6
6 ,
6,
0,
0,
0,
0
0
P : intersection of 0
0&2
6
⇔
∴ equations 0, ⋯ ,
can be chosen out of  The number of possible intersection points : C
0
0
equations
!
! !
15
KAIST wit lab
 The intersection point is a genuine corner if
It also satisfies the remaining inequality constraints.
Otherwise, it is a complete fake.
Ex) in the previous figure
P : ∴
J : J
0&
0&
0
6,
6
0
3,
3
∴
16
KAIST wit lab
 Key idea of simplex methods
 Go from corner to corner along the edges of the feasible set
 Edge of the feasible set
 If one of the intersecting planes is removed, remaining equations form an edge that comes out of the corner
P : 0&
0
6,
1
6
↑ : ↓ : 17
KAIST wit lab
 Newly defined linear programming by slack variable
 Slack variable : ⇔
 Equality constraints and nonnegativity
Minimize
, subject to ,
0
Newly defined
Minimize
, subject to ,
0
18
Example 1.
KAIST wit lab
19
The Simplex Algorithm
 Free variables vs. Basic variables in constraints KAIST wit lab
 Free variables : n components
 Basic variables (Pivot variables) : m components
∵ : ⇔ equations
⇔ pivots
 Basic feasible solutions of A solution is basic when of its components are zero,
and it is feasible when it satisfies 0
 Setting the free variables to zero, basic solution is a genuine corner if its nonzero components are positive
 Phase of simplex method
 Phase I : find one basic feasible solution
 Phase II : move step by step to the optimal ∗
20
KAIST wit lab
 Which corner do we go to next?

1 basic variables remain basic
 Only 1 basic variables will become free (⇔ become zero),
while other 1 basic variables will stay positive
 Terminology
 Entering variable :
the free variable which will become basic
 Leaving variable :
the basic variable which will become free
 Ex) in the previous example
If P moves to Q, ∶ Entering variable, : Leaving variable
21
Example 2
KAIST wit lab
0,
1) 0,
8,
0
7
3
9 : genuine
0
There is possibility that cost can be minimized
2) As : entering variable
is increased,
2
3
0 0 8
9
3) At this point, 7
4
3
3
: leaving variable
9
22
Quick Way
KAIST wit lab
 Observation of Example 2
0,
0
2
3
8
9
 The right sides divided by the coefficients of the entering variable
4,
3
Ratio : hits zero first
23
KAIST wit lab
 The next step of Example 2
2,0,0,0,3
 The current step ends at the new corner • Free variable : , ,
• Basic variable : ,
Leaving variable
Entering variable
9
 Pivot by substituting  Repeat Phase II
•
∗
0,0,0, ⁄ , 3 ⁄
3
24
The Tableau
KAIST wit lab
 Tableau (or large matrix)
 One way to organize the simplex step into matrix
Cost : Constraints :  Applying row operation to tableau  Renumbering if necessary
,⋯,
•
∶
 Tableau at corner ∶
,
∶

:1
,

:1
:
1,
:
1
25
KAIST wit lab
Ex) For Example 2
cost : 0 0 7 1
constraints : 28 42 1160 23 2
8
0
4
2
0
1 6
1 0
7
1
2
3
3
8
9
3
8
9
0
26
KAIST wit lab
 Reduced tableau
The basic variables will stand alone when elimination multiplies by Ex)
2
8
0
4
2
0
1 6
1 0
7
1
2
3
3
8
9
0
1
0
0
1
0
0
6
1
14
14
3 24
14 14
7
1
4
14
5
14
3
10
14
23
14
0
27
KAIST wit lab
 Fully reduced : R=rref(T)
Subtract times the top block row from bottom row
1
2
Ex)
1
0
0
1
0
0
2
6
1
14
14
3 24
14 14
7
1
1
4
14
5
14
3
10
14
23
14
0
28
KAIST wit lab
 Meaning of matrix R
0
Cost :
Constraints : At corner 0 : &
 Stopping test (optimality condition)
r : coefficients of 29
KAIST wit lab
Example 3. Figure 8.3
Minimize
Subject to
2
6, 2
6
<solve problem using tableau>
Slack variable :
2
2
6
6
1
2
1
2
1 0
0
1
6
6
,
1
10
0
30
KAIST wit lab
1) Find initial feasible point
 Let basis variables(free variable) be zero
2
2
6
6
0,
0
6,
6 : fake Find other feasible corner as initial point
0,
0
6,
6 : genuine
We choose this point as initial one
31
KAIST wit lab
2) Renumbering
 Take basic variable(=non‐zeros) firstly
∶
→
This means that problem is changed as below
1
2
1
2
1 0
1
0
0
1
2 1
1 2
6
6
0
1
,
1
6
,
6
10
0
0
11
0
32
KAIST wit lab
3) Make tableau and take row operation
1
0
0
1
0
0
1
0
0
2
1
1
0
1
1
0
1
0
1 0
2 1
1 0
2
3
2 1
1 0
3
2
2 1
1 1
6
6 ,
0
,
6
6
0
6
6
6
6 at point
6
,
6
0
0
33
KAIST wit lab
4) Stopping test
1
0
0
0
1
0
1
6
6
6
3
2
2 1
1 1
1
There is possibility that cost can be minimized
0
Let 0 1
1
→
be variable which becomes basic (=non‐zero)
⟺
3
34
KAIST wit lab
5) Find the variable which becomes free(=zero)
Let : a column of : the index of smallest ratio  Then, 1, ⋯ ,
becomes free variable (=zero)
∵
1
0
0
0
1
0
Ratio : 3
2
2 1
1 1
2
6
6
6
1
3
2
1 : becomes zero
35
KAIST wit lab
6) Find next solution
 Tableau is reconstructed by changing column and 0
1
0
3
2
1
1
0
1
1
0
0
0
1
0
0
1
0
1
2
0 1
0 1
6
6 ,
6
1/3
2/3
2/3 1/3
0
1
2/3
1/3
2/3 1/3
1/3
1/3
,
2
2
6
2
2
4
36
KAIST wit lab
7) Stopping test again
1
0
0
Since 0
1
0
2/3
1/3
2/3 1/3
1/3
1/3
2
2
4
0, this point is optimal.
at the point
4
2
,
2
0
0
37
The Organization of a Simplex Step
KAIST wit lab
 Reduced simplex method
,
 For computational purposes, only 1
0
0
0
1
0
3
2
2 1
1 1
6
6
6
, are used.
3
2
1
1) 2) 0
1
0
1
2
0 1
0 1
6
6
6
0
38
KAIST wit lab
 reduce the computation since has the form as below
39
Karmarkar’s Method
KAIST wit lab
2012. 07. 26.
Wonho Kang
40
Karmarkar’s Method
KAIST wit lab
 Definition
 An iterative algorithm that, given an initial point and parameter , generates a sequence , , … ,
which are the solutions of linear program (LP)
 Assumption
1.
2.
3.
LP has a strictly feasible point, and the set of optimal point is bounded
LP has a special “canonical” form:
minimize
z
1, 0
subject to
0, ∑
The value of the objective at the optimum is known and is equal to zero, ∗ 0
41
Karmarkar’s Method
KAIST wit lab
 Minimize
 Subject to

0
′
 Minimize
 Subject to
0

z 0
′
 Minimize
 Subject to
0

y 0
′
, 0 , ,1
,
,…,
,
⋯
1
,
1
⋯
1
1, ⋯ ,
,
1
42
Karmarkar’s Method
KAIST wit lab
 Minimize
′
 Subject to
0

1

y 0
43
Karmarkar’s Method
KAIST wit lab
 Canonical form
minimize
subject to
z
0, ∑
1,  Denote nullspace of Ω
∈
|
0
 Define simplex
∈
∆
1,
|
0 , 0
1, … , 1
 Denote center of the simplex ∆
,…,
, ∈∆
44
Karmarkar’s Method
KAIST wit lab
 Rewritten canonical form
minimize
z
subject to
∈Ω∩∆
 Note that the constraint set (or feasible set) Ω ∩ ∆ can be represented as
∈
|
0,
1,
0
Ω∩∆
0
Ω∩∆
∈
|
,
0
1
0
∈
|
,
0
Ω∩∆
1
45
Karmarkar’s Method
KAIST wit lab
 Procedure
1. Initialize: Set ≔ 0; 2. Update: Set Ψ
/ ;
where Ψ is an update map
3. Check stopping criterion: If the condition 2
is satisfied, then stop where is a given termination parameter;
4. Iterate: Set ≔
1, go to 2
46
Karmarkar’s Method
KAIST wit lab
 First update step in the algorithm
1. Compute the orthogonal projector onto the nullspace of 2. Compute the normalized orthogonal projection of onto the nullspace of 3. Compute the steepest‐descent direction vector
, 1/
1
4. Compute using
where is the prespecified step size, ∈ 0,1
Karmarkar recommands a value of ¼ in his original paper
47
Karmarkar’s Method
KAIST wit lab
 General update step
1. Compute the matrices
,
⋱
and ,
is, in general, not at the center of the simplex, so whose diagonal entries are the components of the vector is used to transform this point to the center
2. Compute the orthogonal projector onto the nullspace of 48
Karmarkar’s Method
KAIST wit lab
 General update step
3. Compute the normalized orthogonal projection of onto the nullspace of 4. Compute the steepest‐descent direction vector
5. Compute ̅
̅
using
6. Compute by applying the inverse transformation ̅
̅
̅
49
Karmarkar’s Method
KAIST wit lab
 Example
 Minimize
subject to
3
2
3
3
0
0
1
1
,




3 3
1
2
3 1
1 1 1
3
,
0
Ω
Δ
|
|
0
1,
0
50
Karmarkar’s Method
KAIST wit lab
 Solution
1. Initialize:
2. Set ≔ 0;
3.
4.
5.
1 1 1
3
|
7.
|
8.
1/3 1/3 1/3
/
6.
|
/
/
0,
1,
0
,
1
0
,
1
0
0
0
/
/
51
Karmarkar’s Method
KAIST wit lab
Ψ
2. Update: Set where Ψ is an update map
1) Compute the orthogonal projector onto the nullspace of 2
3
4
5
6
3 3
1
2
3 1
1 1 1
2
3 1
1 1 1
0
|
0
7
8
16
4
20
4
1
5
20
5
25
52
Karmarkar’s Method
KAIST wit lab
Ψ
2. Update: Set where Ψ is an update map
2) Compute the normalized orthogonal projection of onto the nullspace of 3
4
5
3
3
16
4
20
1
4
20
1
5
5 25
4
1
5
53
Karmarkar’s Method
KAIST wit lab
Ψ
2. Update: Set
where Ψ is an update map
3) Compute the steepest‐descent direction vector
4
5
1
1
1
3
6
7
1/
1
1/ 6
is the radius of the largest sphere
8) inscribed in the set
9
|
1,
0
54
Karmarkar’s Method
KAIST wit lab
Ψ
2. Update: Set
where Ψ is an update map
3) Compute the steepest‐descent direction vector
4
1
5
4
5
6
1/ 6
4
1
5
55
Karmarkar’s Method
KAIST wit lab
Ψ
2. Update: Set
where Ψ is an update map
4) Compute 5
1/3
1/3
1/3
4
1
5
6
0.0814
0.2703 , 0.6483
7
3) Note:
4
makes 1
be guaranteed
5) to lie in the constraint set
6
|
0,
1,
0
7
∩
|
8) (a strictly interior point of the set)
56
Karmarkar’s Method
KAIST wit lab
3. Check stopping criterion:
2
4. If the condition is satisfied, then stop where is a given termination parameter
5.
3 3
1
1/3 1/3 1/3
6.
7.
0.0814 0.2703 0.6483
8.
0.243
9. if cf) ∗
0 1/4 3/4
2
2.04, then stop, or
4. Iterate: Set ≔
1, go to 2
57
Karmarkar’s Method
KAIST wit lab
c
c0
d0
x (0)
x (1)
1
0.8
0.6
0.4
0.2
0
-0.2
-0.4
-0.6
-0.5
-0.8
0
-1
-0.5
0.5
1
0
0.5
1.5
1
2
1.5
2
2.5
2.5
3
3
58
Karmarkar’s Method

0.081
0.270
0.648
0.081
KAIST wit lab
0.270
0.648
1/3 1/3 1/3


 Minimize
 Subject to


0
1
0
′
Minimize
Subject to
′
′
′
0
1
0
59
Karmarkar’s Method





0.1627
1
0.8017
0.8110 0.6483
1
1
0.2668
KAIST wit lab
0.5349
0.9273 0.1089 0.2184
0.0060 0.4422 0.5517
0.0010 0.2503 0.7487
60
Karmarkar’s Method
KAIST wit lab
c
c0
d0
x (0)
x (1)
1
0.8
0.6
0.4
0.2
0
-0.2
-0.4
-0.6
-0.5
-0.8
0
-1
-0.5
0.5
1
0
0.5
1.5
1
2
1.5
2
2.5
2.5
3
3
61
Karmarkar’s Method
KAIST wit lab
c
c1
d1
x'(1)
x'(2)
x (1)
1
x (2)
0.8
0.6
0.4
0.2
0
-0.2
-0.4
-0.6
-0.5
-0.8
0
-1
-0.5
0.5
1
0
0.5
1.5
1
2
1.5
2
2.5
2.5
3
3
62
III. The Dual Problem
KAIST wit lab
KAIST wit lab
 The Dual Problem
Minimize
Subject to ,
0
Dual
∶n 1
feasible set ∈ Maximize
Subject to
,
0
y ∶ 1
feasible set ∈ Ex) In Section 8.1,
Minimize Subject to Dual
2 3 1 2
Maximize Subject to 4
4
1 2
2 3
64
KAIST wit lab
 Duality theorem
 When both problems have feasible vectors, the minimum cost c
∗
the maximum income ∗
 Weak duality
 If x and y are feasible in the primal and dual problems, then (proof)
x and y are feasible 
y

and 65
KAIST wit lab
 Reverse of duality theorem
 If the vectors x and y are feasible and then x and y are optimal
,
Ex)
Minimize
subject to 2
2
5
4
6, 5
Maximize 6
subject to 2
Dual
:
3 ∶
3 3
7
5
3
0
1
,
3
6
,
7
1 4
7
1,
3
4
1/2
0
66
KAIST wit lab
 Complementary slackness condition
 Relationship between the primal and dual
 Assume (P) means primal problem and have a optimal solution ∗
and (D) means dual problem and have a optimal solution ∗ , then
1. If 2. If ∗
∗
∗
in (P), then in (D), then 0
0
∗
(proof)
∗
∗
c
∗
∗
∗
∗
∗
c
∗
∗
should be always satisfied.
However, from this equation, if ∗ 0,
∗
∴
is ∗
∗
0
67
KAIST wit lab
 The proof of Duality
From simplex method,
,
, ∗
by reordering
In dual problem,
if y is feasible
∴
If ∴
⟹
∗
∗
⟺
∗
,
∗
∗
0
0  optimal
∗
68