Lecture 8
Linear programming
Kin Cheong Sou
Department of Mathematical Sciences
Chalmers University of Technology and Göteborg University
November 17, 2013
Linear programs (LP)
Formulation
Consider a linear program (LP):
z ∗ = infimum
subject to
c T x,
x ∈ P,
where P is a polyhedron (i.e., P = {x | Ax ≤ b}).
◮
A ∈ Rm×n is a given matrix, and b is a given vector,
◮
Inequalities interpreted entry-wise (i.e., (Ax)i ≤ (b)i , i = 1, . . . , m),
◮
Minimize a linear function, over a polyhedron (linear constraints).
TMA947 – Lecture 8
Linear programming
2 / 43
Linear programs (LP)
Formulation
z ∗ = infimum
subject to
c T x,
x ∈ P,
−c
P
TMA947 – Lecture 8
Linear programming
3 / 43
Linear programs (LP)
Formulation
z ∗ = infimum c T x,
subject to
x ∈ P,
x∗
−c
P
TMA947 – Lecture 8
Linear programming
4 / 43
Polyhedra in different forms
Formulation
Inequality constraints Ax ≤ b (i.e., x ∈ P) might look restrictive,
but in fact more general:
◮
x ≥ 0n ⇐⇒ −I n x ≤ 0n ,
◮
Ax ≥ b ⇐⇒ −Ax ≤ −b,
◮
Ax = b ⇐⇒ Ax ≤ b
and − Ax ≤ −b.
In particular, we often consider polyhedron in standard form:
P = {x ∈ Rn | Ax = b, x ≥ 0n }.
P is a polyhedron, since P = {x ∈ Rn | Ãx ≤ b̃} for some à and b̃.
TMA947 – Lecture 8
Linear programming
5 / 43
Standard form linear programs
Standard form
We say that a LP is written in standard form if
z ∗ = infimum c T x,
subject to Ax = b,
x ≥ 0.
◮
Meaning that P = {x ∈ Rn | Ax = b, x ≥ 0}.
◮
Only considering nonnegative variables and equality constraints.
◮
But standard form LP can in fact model all LP’s.
TMA947 – Lecture 8
Linear programming
6 / 43
Rewriting to standard form
◮
For example, if P = {x ∈ Rn | Ax ≤ b, x ≥ 0}, we can add slack
variables, s, in order to write P on standard form.
Ax
x
◮
Standard form
≤ b,
⇐⇒
≥ 0n
Ax + I m s
x
s
= b,
[A I m ] v
≥ 0n ⇐⇒
v
≥ 0m
= b,
≥ 0n+m
If some variable xj is free of sign, substitute it everywhere by
xj = xj+ − xj−
where xj+ , xj− ≥ 0
◮
We also assume that b ≥ 0m . If some element of b is negative,
multiply that constraint by −1.
TMA947 – Lecture 8
Linear programming
7 / 43
Rewriting to standard form, example
minimize −2x
subject to x ≤ 1
x ≥0
Standard form
minimize −2x
subject to x + s = 1
x, s ≥ 0
s
−c
1
−c
P
P
0
1
x
0
1
x
Equivalent LP’s, but different polyhedra!
TMA947 – Lecture 8
Linear programming
8 / 43
Optimality, a geometric motivation
Extreme points
Minimizing c T x over P =⇒ an optimal solution at a vertex of P
(the unique optimal solution over P minimizing c T x for some c):
x∗
−c
P
If P has a vertex and if LP has an optimal solution,
then at least one optimal solution is a vertex.
A point x is a vertex of P ⇐⇒ x is an extreme point of P.
TMA947 – Lecture 8
Linear programming
9 / 43
Extreme points
Extreme points
An extreme point of a convex set S is a point that cannot be
written as a convex combination of two other points in S.
P
TMA947 – Lecture 8
Linear programming
10 / 43
Extreme points solution candidates
Extreme points
We focus on extreme points when searching for optimal solutions, but
◮
When does polyhedron {x ∈ Rn | Ax ≤ b} have extreme points?
◮
Extreme points are geometrical objects... hard to put in algorithm
Work with LP in standard form (i.e., P = {x | Ax = b, x ≥ 0}) because
A nonempty polyhedron in standard form always has an extreme point
Instead of geometric objects such as extreme points, we work with their
algebraic equivalence – basic feasible solution in standard form LP.
◮
We will show, indeed, basic feasible solutions are extreme points
and vice versa.
TMA947 – Lecture 8
Linear programming
11 / 43
Basic feasible solution (BFS)
Extreme points
Standard form polyhedron P = {x | Ax = b, x ≥ 0}, A ∈ Rm×n , rank(A) = m
A point x̄ is a basic feasible solution (BFS) if
◮
x̄ ≥ 0, and x̄ is a basic solution.
A point x̄ is a basic solution if
◮
Ax̄ = b, and
◮
the columns of A corresponding to non-zero components of x̄
are linearly independent (and extendable to a basis of Rm ).
P
(Recall that: Ax̄ = nj=1 aj x̄j , where aj is column j of A.)
TMA947 – Lecture 8
Linear programming
12 / 43
Basic and non-basic variables
Extreme points
For any BFS x̄, we can reorder the variables according to
cB
xB
,
, A = (B, N), c =
x̄ =
cN
xN
such that
◮
B ∈ Rm×m , rank(B) = m.
◮
xN = 0n−m .
◮
xB = B −1 b (as a consequence of Ax̄ = BxB + NxN = b).
We call
◮
xB the basic variables. If xB ≯ 0 then BFS x̄ is called degenerate.
◮
xN the non-basic variables.
◮
B the basis matrix. Each BFS is associated with at least one basis.
TMA947 – Lecture 8
Linear programming
13 / 43
BFS = extreme point
Extreme points
Theorem
Assume rank(A) = m. A point x̄ is an extreme point of the set
{x ∈ Rn | Ax = b, x ≥ 0} if and only if it is a basic feasible solution.
Proof: We show it on blackboard, or consult Theorem 8.7 in text.
Thus,
◮
“vertex = extreme point = basic feasible solution (BFS)”.
◮
So, we focus optimal solution search in BFS’s (extreme points).
Now let’s formally show that the restriction is justified!
TMA947 – Lecture 8
Linear programming
14 / 43
Representation theorem, standard form
Extreme points
◮
P = {x ∈ Rn | Ax = b, x ≥ 0} (i.e., polyhedron in standard form)
◮
V = {v 1 , . . . , v k } be the extreme points of P
◮
C = {x ∈ Rn | Ax = 0, x ≥ 0}
◮
D = {d 1 , . . . , d r } be the extreme directions of C
Representation Theorem (standard form polyhedron)
Every point x ∈ P is the sum of a convex combination of points in V
and a non-negative linear combination of points in D, i.e.
x=
k
X
i =1
where α1 , . . . , αk ≥ 0,
Pk
i =1
i
αi v +
r
X
βj d j ,
j=1
αi = 1 and β1 , . . . , βr ≥ 0
Proof: See text Theorem 8.9 (In the proof, Th. 8.9 should be Th. 3.26).
TMA947 – Lecture 8
Linear programming
15 / 43
Illustration of representation theorem
Extreme points
Representation theorem provides “inner representation” of polyhedron.
◮
(a) x is convex combo. of v 2 and y , and y is convex combo. of v 1
and v 3 =⇒ x is convex combo. of v 1 , v 2 and v 3 .
◮
(b) x is convex combo. of v 1 and v 2 , plus β2 d 2 .
v3
y
d1
v1
v1
d2
x
v2
y
v4
(a) Bounded case
TMA947 – Lecture 8
x
v2
(b) Unbounded case
Linear programming
16 / 43
Optimality of extreme points
Extreme points
Now we can present the theorem regarding optimality of extreme points
Theorem
Consider the standard form LP problem
z ∗ = infimum z = c T x,
subject to x ∈ P,
This problem has a finite optimal solution if and only if P is
nonempty and z is bounded on P, meaning that c T d j ≥ 0 for all
dj ∈ D
Moreover, if the problem has a finite optimal solution, then there
exists an optimal among the extreme points.
Proof: We show it on blackboard, or see Theorem 8.10 in text.
TMA947 – Lecture 8
Linear programming
17 / 43
Summary so far
The simplex algorithm
So far, we have seen
◮
All linear programs can be written in standard form.
◮
Vertex = extreme point = basic feasible solution (BFS).
◮
If a standard form LP has finite optimal solution, then it has an
optimal BFS.
We finally have rationale to search only the BFS’s to solve a standard
form LP. This is the main characteristic of the simplex algorithm.
TMA947 – Lecture 8
Linear programming
18 / 43
Graphic illustration
The simplex algorithm
Start at a BFS, in this case (0, 0)T .
x2
x1 − 2x2 = −1
−c = (1, 1)T
P
x1
T
(0, 0)
3x1 + 2x2 = 9
TMA947 – Lecture 8
Linear programming
19 / 43
Graphic illustration
The simplex algorithm
Find a feasible descent direction towards an adjacent BFS.
x2
x1 − 2x2 = −1
−c = (1, 1)T
P
x1
T
(0, 0)
3x1 + 2x2 = 9
TMA947 – Lecture 8
Linear programming
20 / 43
Graphic illustration
The simplex algorithm
Move along the search direction until a new BFS is found.
x2
x1 − 2x2 = −1
−c = (1, 1)T
P
(3, 0)T
x1
3x1 + 2x2 = 9
TMA947 – Lecture 8
Linear programming
21 / 43
Graphic illustration
The simplex algorithm
Find a new feasible descent direction at the current BFS.
x2
x1 − 2x2 = −1
−c = (1, 1)T
P
(3, 0)T
x1
3x1 + 2x2 = 9
TMA947 – Lecture 8
Linear programming
22 / 43
Graphic illustration
The simplex algorithm
Move along the search direction.
x2
x1 − 2x2 = −1
(2, 3/2)T
−c = (1, 1)T
P
x1
3x1 + 2x2 = 9
TMA947 – Lecture 8
Linear programming
23 / 43
Graphic illustration
The simplex algorithm
If no feasible descent directions exist, the current BFS is declared optimal.
x2
x1 − 2x2 = −1
x = (2, 3/2)T
∗
−c = (1, 1)T
P
x1
3x1 + 2x2 = 9
TMA947 – Lecture 8
Linear programming
24 / 43
Developing the simplex algorithm
The simplex algorithm
To develop the simplex algorithm, we translate geometric intuition into
algebraic manipulations. We need to...
◮
Find a feasible descent direction at any BFS.
◮
Determine the step size to move along a feasible descent direction.
◮
Certify optimality at an optimal BFS.
TMA947 – Lecture 8
Linear programming
25 / 43
Basic feasible solution, recap
The simplex algorithm
A BFS x̄ satisfies Ax̄ = b, x̄ ≥ 0, can reorder the variables such that
cB
xB
,
, A = (B, N), c =
x̄ =
cN
xN
such that
◮
Basis matrix B ∈ Rm×m , rank(B) = m.
◮
Non-basic variables xN = 0n−m .
◮
Basic variables xB ≥ 0, xB = B −1 b (as a consequence of Ax̄ = b).
We call a BFS x̄ degenerate if xB ≯ 0. x̄ non-degenerate if xB > 0.
TMA947 – Lecture 8
Linear programming
26 / 43
Basic feasible solution, example
The simplex algorithm
Consider standard form LP
minimize
subject to
2x1
x1
2x1
x1 ,
+ 0 · x2
+
0 · x3
+
x2
+
x3
+ 0 · x2
+
3x3
x2 ,
x3 , x4 ≥ 0
+
+
+
0 · x4
x4
4x4
= 2
= 2
An example BFS has basic variables xB = (x1 , x2 ) and nonbasic variables
xN = (x3 , x4 ). A = (B, N) with
1 1
1 1
1
0
−1
B=
, N=
, xB = B b =
, xN =
2 0
3 4
1
0
TMA947 – Lecture 8
Linear programming
27 / 43
Search direction
xB
xN
The simplex algorithm
◮
with A = (B, N); iterate update x̄ ← x̄ + θd for
dB
step-size θ ≥ 0 with search direction d =
.
dN
◮
Update only one non-basic variable: Let dN = ej , for
j = 1, . . . , n − m (ej is the j-th unit vector in Rn−m ).
◮
dB is not arbitrary – it is decided by feasibility of x̄ + θd:
At BFS x̄ =
A(x̄ + θd) = b =⇒ Ad = 0 =⇒ dB = −B −1 Nj
−B −1 Nj
◮ search directions become dj =
, j = 1, . . . , n − m
ej

 

−2
−3/2
1
 1/2 
1 1
1 1

 
e.g., B =
, N=
, d1 = 
 1  , d2 =  0 
2 0
3 4
1
0
◮
TMA947 – Lecture 8
Linear programming
28 / 43
Search direction, in picture
The simplex algorithm
Consider feasible search directions changing only one non-basic variable.
−B −1 Nj
, j = 1, . . . , n − m.
That is, dj =
ej
d2
infeasible
direction
BFS
morethanonenonͲbasic
variablechanging
d1
◮
We will show dj indeed goes along the edge of polyhedron.
◮
Question: Is direction dj going to decrease the objective value?
TMA947 – Lecture 8
Linear programming
29 / 43
Reduced costs
◮
The simplex algorithm
From x̄ to x̄ + θdj , objective value change is
−B −1 Nj
:= θ · (c̃N )j
c T (x̄ + θdj − x̄) = θ · c T dj = θ · (cBT , cNT )
ej
(c̃N )j := (cNT − cBT B −1 N)j is the reduced cost for non-basic var (xN )j
T
c̃N := (cNT − cBT B −1 N) are reduced costs for non-basic variables
◮
If (c̃N )j ≥ 0, dj does not decrease objective value.
◮
If (c̃N )j < 0, consider update x̄ + θdj with θ as large as possible
since objective value change is θ · (c̃N )j < 0 as long as θ > 0.
Question: What is the maximum value of θ that we can choose?
TMA947 – Lecture 8
Linear programming
30 / 43
Reduced costs, in picture
The simplex algorithm
To see which directions dj are profitable, we form reduced costs (c̃N )j
which are inner products of cost vector c and directions dj .
d2
-c
BFS
d1
Question: Iterate moves along dj with (c̃N )j < 0, but how far?
TMA947 – Lecture 8
Linear programming
31 / 43
Update along search direction
The simplex algorithm
At BFS x̄ = (xBT , 0)T , negative reduced cost for (xN )j (i.e., (c̃N )j < 0).
◮
Iterate update
xB − θB −1 Nj
dB
xB
, θ≥0
=
+θ
x̄ + θdj =
θej
dN
0
◮
If B −1 Nj ≤ 0 then x̄ + θdj ≥ 0 for all θ ≥ 0. Let θ → ∞, and we
conclude that objective value is unbounded from below.
◮
If B −1 Nj 0 some entry of xB − θB −1 Nj becomes 0 as θ increases.
Thus, θ ≤ θ∗ =
◮
min
k:(B −1 Nj )k >0
(xB )k
(B −1 Nj )k
, and let i be s.t. θ∗ =
(xB )i
(B −1 Nj )i
.
Thus, we arrive at new iterate x̄ + θ∗ dj with (xB − θ∗ B −1 Nj )i = 0.
Note: θ∗ can be zero if x̄ is degenerate!
TMA947 – Lecture 8
Linear programming
32 / 43
Example, continued
1
BFS x̄ = (1, 1, 0, 0), B =
2
Reduced costs:
c̃NT
=
(cNT
−
cBT B −1 N)
=− 2
The simplex algorithm
1
1 1
2
0
,N=
, cB =
, cN =
0
3 4
0
0
1
0
2
−1 1
1
0
3
1
= (−3, −4)
4
Both dir’s d1 = (−3/2, 1/2, 1, 0) and d2 = (−2, 1, 0, 1) are profitable.
3/2
1
B )1
For d1 , B −1 N1 =
, max step-size θ∗ = (B(x
−1 N ) = 3/2 = 2/3.
1 1
−1/2
  

 
0
−3/2
1






4/3
1/2
1
  


Updated iterate: x̄ + θ∗ d1 = 
0 + (2/3)  1  = 2/3
0
0
0
TMA947 – Lecture 8
Linear programming
33 / 43
Iterate update, in picture
The simplex algorithm
Updating x̄ + θdj either tells us objective value is unbounded (left
picture), or a possibly new point x̄ + θ∗ dj is reached (right picture).
BFS
ș d1, ș Æ ’
BFS
ș*d1
newBFS?
Question: What is x̄ + θ∗ dj ? Is it a BFS? How is it related to x̄?
TMA947 – Lecture 8
Linear programming
34 / 43
Change of basis
The simplex algorithm
From x̄ to x̄ + θ∗ dj , the i-th basic variable (xB )i becomes zero, whereas
j-th non-basic variable (xN )j (i.e., the (m + j)-th variable) becomes θ∗ :




..
..
 . 
 . 
(xB )i 
 0 
∗


x̄ = 
x̄ + θ dj = 
 .. 
 .. 
 . 
 . 
0
θ ∗ ej
◮
Can show the columns a1 , . . . , ai −1 , am+j , ai +1 , . . . , am are linearly
independent (forming a new basis), and x̄ + θ∗ dj is indeed a BFS.
◮
We say (xB )i leaves the basis to become non-basic variable,
whereas (xN )j enters the basis to become basic variable.
◮
x̄ and x̄ + θ∗ dj are in fact adjacent BFS’s (see text for formal
definition of adjacency).
TMA947 – Lecture 8
Linear programming
35 / 43
Adjacent BFS’s
The simplex algorithm
In the figure, a is adjacent to b and d, and b is adjacent to a and c
b
a
P
d
c
Search direction dj is along the edge of polyhedron, from a BFS to
a BFS.
TMA947 – Lecture 8
Linear programming
36 / 43
Adjacency
The simplex algorithm
Theorem
Two BFS’s are adjacent if and only if their sets of basic variables
differs in exactly one place.
Remember, (xB )i leaves the basis whereas (xN )j enters the basis during
the change from x̄ to x̄ + θ∗ dj .
◮
In principle, the simplex algorithm starts with a BFS corresponding
to a matrix B and then iteratively changes one of the columns in B
until optimality occurs.
◮
Basis matrix B is modified at every iteration, but BFS might stay
because of degeneracy (a degenerate BFS can be associated with
more than one basis)!
TMA947 – Lecture 8
Linear programming
37 / 43
To recap so far
The simplex algorithm
We have seen so far...
◮
At a BFS with A = (B, N), compute search directions
−B −1 Nj
, j = 1, . . . , n − m.
dj =
ej
◮
Evaluate the reduced costs c̃N := (cNT − cBT B −1 N)T to see which
directions are profitable (which non-basic variable to enter basis).
◮
What happens when we update x̄ + θdj for some dj with (c̃N )j < 0
– either objective value is unbounded or an adjacent BFS is reached.
But...
◮
What if c̃N ≥ 0, as all our search directions are not profitable?
TMA947 – Lecture 8
Linear programming
38 / 43
Optimality criterion
The simplex algorithm
If at a BFS with basis B, all reduced costs are nonnegative (c̃N ≥ 0n−m ),
then it is optimal and the simplex algorithm terminates:
Theorem
Let x̄ be a BFS associated with basis matrix B, and let
T
c̃N = (cNT − cBT B −1 N) be the corresponding vector of reduced
costs for the non-basic variables. If c̃N ≥ 0, then x̄ is optimal.
T
Proof: All feasible directions d at x̄ = (xBT , xNT ) are of the form
−B −1 N dN
=⇒ c T d = (cNT − cBT B −1 N) dN
d=
dN
{z
}
|
c̃NT
c̃N ≥ 0 and dN ≥ 0 (i.e., feasible direction) concludes the statement.
TMA947 – Lecture 8
Linear programming
39 / 43
The conceptual simplex algorithm
The simplex algorithm
T
1. Assume we have an initial BFS x̄ = (xBT , xNT ) with A = (B, N).
2. Compute reduced costs c̃N = (cNT − cBT B −1 N)T .
◮
◮
If c̃N ≥ 0, then current BFS is optimal, terminate.
If c̃N 0, choose some non-basic var index j s.t. (c̃N )j < 0.
3. Compute B −1 Nj .
◮
◮
If B −1 Nj ≤ 0, then objective value is −∞, terminate.
(xB )k
If B −1 Nj 0, compute θ∗ =
min
(B −1 Nj )k .
−1
k:(B
Nj )k >0
4. Update x̄ ← x̄ + θ∗ dj . Let i be s.t. θ∗ =

B ← a1
···
ai −1
am+j
(xB )i
(B −1 Nj )i
ai +1
. Update basis
···

am  .
Reorder variables s.t. the first m variables are basic in x̄ + θ∗ dj .
TMA947 – Lecture 8
Linear programming
40 / 43
Finite termination
The simplex algorithm
Q: Will the simplex algorithm terminate in finite number of steps?
A: Yes, if all BFS’s are non-degenerate.
Theorem
If feasible set is nonempty and every BFS is non-degenerate,
then the simplex algorithm terminates in finite number of
iterations. At termination, two possibilities are allowed:
(a) an optimal basis B found with the associated optimal BFS.
(b) a direction d found s.t. Ad = 0, d ≥ 0 and c T d < 0, thus
optimal objective value is −∞.
The simplex algorithm can be modified to finitely terminate even
with degenerate BFS (e.g., Bland’s rule, see text)!
TMA947 – Lecture 8
Linear programming
41 / 43
Computational complexity
◮
◮
◮
The simplex algorithm
The simplex algorithm works very well in practice.
n
BFS’s
m
before termination – worst-case computation effort is exponential.
The simplex algorithm can, in the worst case, visits all
Polynomial-time algorithms are available (e.g., ellipsoid algorithm,
interior point algorithms). See coming lectures.
TMA947 – Lecture 8
Linear programming
42 / 43
How do we obtain an initial BFS?
The simplex algorithm
We create a so called Phase-I problem by introducing artificial
variables ai in every row.
w ∗ = minimize w = (1m )T a,
subject to Ax + I m a = b,
x
≥ 0n ,
a ≥ 0m .
◮
Why is this easier? Because a = b, x = 0n is an initial BFS.
◮
w ∗ = 0 =⇒ Optimal solution a∗ = 0m
x ∗ BFS in the original problem
◮
w ∗ > 0 =⇒ There is no BFS to the original problem
The original problem is infeasible
TMA947 – Lecture 8
Linear programming
43 / 43