Linear Programming
Computer Science 511
Iowa State University
September 24, 2010
Computer Science 511 (Iowa State University)
Linear Programming
September 24, 2010
1 / 27
Example: Profit Maximization
Joe sells two kinds of beer: Regular Duff and Duff Special Brew.
Profit per pint for Regular Duff is $1.
Profit per pint for Duff Special is $2.
Demand for Regular Duff is at most 200 pints per day.
Demand for Duff Special is at most 400 pints per day.
Duff will sell Joe a combined total of at most 500 pints of Regular
Duff and Duff Special per day.
How many pints of each beer should Joe buy per day so as to
maximize his profit?
Computer Science 511 (Iowa State University)
Linear Programming
September 24, 2010
2 / 27
Example: LP Formulation
Let x1 and x2 denote the number of pints of Regular Duff and Duff Special
that Joe will buy per day.
maximize
subject to
x1 + 2x2
x1
x1 +
x1 ,
Computer Science 511 (Iowa State University)
≤ 200
x2 ≤ 400
x2 ≤ 500
x2 ≥
0
Linear Programming
September 24, 2010
3 / 27
Linear Programming
Maximize (or minimize) a linear function
n
X
cj xj
j=1
subject to linear equalities/inequalities of the form
n
X
aj xj = b,
j=1
Computer Science 511 (Iowa State University)
n
X
aj xj ≤ b,
j=1
or
n
X
aj xj ≥ b.
j=1
Linear Programming
September 24, 2010
4 / 27
Terminology
Feasible solution: Any x = (x1 , . . . , xn ) that satisfies all the constraints.
Feasible region: Set of all feasible solutions.
The intersection of a collection of halfspaces and
hyperplanes.
⇒ A convex polyhedral region.
Objective function: The function of x1 , . . . , xn that we want to maximize.
Objective value: The value of the objective function for some specific
x1 , . . . , xn .
Computer Science 511 (Iowa State University)
Linear Programming
September 24, 2010
5 / 27
Example
!"
maximize
subject to
x1
+
2x2
+
,
x2
x2
x2
x1
x1
x1
400
≤ 200
≤ 400
≤ 500
≥
0
300
200
Obj. val. = 900
100
0
100
200
300
400
500
!!
Obj. val. = 400
Sunday, September 12, 2010
Computer Science 511 (Iowa State University)
Linear Programming
September 24, 2010
6 / 27
Infeasibility and unboundedness
Definition
A linear program is infeasible if it has no feasible solution.
Example
Maximize x subject to x ≤ −1 and x ≥ 0.
Definition
A linear program is unbounded if its objective value can be made
arbitrarily large (small).
Example
Maximize x subject to x ≥ 0.
Computer Science 511 (Iowa State University)
Linear Programming
September 24, 2010
7 / 27
The Simplex Algorithm (Dantzig 1947)
Moves from vertex to neighboring vertex of feasible region.
Objective value improves at each step.
Stop when no neighbor gives a higher value.
By convexity, this vertex must be optimum.
Computer Science 511 (Iowa State University)
Linear Programming
September 24, 2010
8 / 27
Standard Form
Fact
Every linear program can be put into the following standard form.
maximize
subject to
n
X
j=1
n
X
cj xj
aij xj ≤ bi
i = 1, . . . , m
j=1
xj ≥ 0
Computer Science 511 (Iowa State University)
Linear Programming
j = 1, . . . n
September 24, 2010
9 / 27
Converting to Standard Form
1
To turn a minimization problem into maximization problem, multiply
the coefficients of the objective function by −1.
2
To reverse the direction of an inequality, multiply both sides by −1.
3
To
Pnchange an equality constraint into inequalities rewrite
j=1 aj xj = b as
n
X
aj x j ≤ b
and
j=1
4
n
X
aj xj ≥ b.
j=1
If x is a variable that is unrestricted in sign, introduce two nonnegative
variables, x + , x − ≥ 0 and replace every occurrence of x by x + − x − .
Computer Science 511 (Iowa State University)
Linear Programming
September 24, 2010
10 / 27
Slack Form
For the ith constraint, define a nonnegative slack variable xn+i :
xn+i = 0
⇐⇒
constraint i is tight.
At all times, express constraints as a set of equalities where the n
variables in RHS are 0.
I
RHS variables correspond to tight inequalities.
Tight inequalities define a vertex of the feasible region.
I
A point in Rn is defined by n hyperplanes.
Computer Science 511 (Iowa State University)
Linear Programming
September 24, 2010
11 / 27
Original
n
X
maximize
cj xj
j=1
n
X
subject to
aij xj ≤ bi
i = 1, . . . , m
j=1
xj ≥ 0
j = 1, . . . n
Slack Form
z=
n
X
cj xj
j=1
xn+i = bi −
n
X
aij xj
i = 1, . . . , m
j=1
(x1 , . . . , xn+m ≥ 0)
Computer Science 511 (Iowa State University)
Linear Programming
September 24, 2010
12 / 27
Slack Form: Example
Original
maximize
subject to
Slack Form
2x1 + 5x2
2x1 − x2
x1 + 2x2
−x1 + x2
x1
x2
Computer Science 511 (Iowa State University)
≤
≤
≤
≥
≥
4
9
3
0
0
z
x3
x4
x5
Linear Programming
=
=
=
=
0
4
9
3
+ 2x1
− 2x1
− x1
+ x1
+ 5x2
+ x2
− 2x2
− x2
(x1 , x2 , x3 , x4 , x5 ≥ 0)
September 24, 2010
13 / 27
Dictionaries
At all times, simplex maintains a dictionary:
X
z = v0 +
cj0 xj
j∈N
xi =
bi0
−
X
aij0 xj
for i ∈ B
j∈N
Variables in RHS are called non-basic.
I
For each j ∈ N, xj = 0.
Variables in LHS are called basic.
I
For each i ∈ B, xi = bi0 .
The values of the variables implied by a dictionary give a basic
feasible solution (b.f.s.).
I
z = v 0 is the objective value of the b.f.s.
Computer Science 511 (Iowa State University)
Linear Programming
September 24, 2010
14 / 27
Pivoting
Greedy Approach
Find a j ∈ N such that cj0 > 0.
I
z =v +
X
xi = bi0 −
X
0
I
cj0 xj
j∈N
aij0 xj
for i ∈ B
j∈N
z can be increased by increasing xj .
If no such j exists, current b.f.s. is
optimal.
Increase xj as much as possible,
keeping xi ≥ 0 for all i ∈ B.
I
xj enters the basis.
For some i ∈ B, xi drops to 0.
I
Computer Science 511 (Iowa State University)
xi leaves the basis.
Linear Programming
September 24, 2010
15 / 27
Dictionary 1
Basic feasible solution:
z
x3
x4
x5
=
=
=
=
0
4
9
3
+ 2x1
− 2x1
− x1
+ x1
+ 5x2
+ x2
− 2x2
− x2
(x1 , x2 , x3 , x4 , x5 ) = (0, 0, 4, 9, 3)
Objective value: 0
Entering variable: x2
Leaving variable x5
Dictionary 2
Basic feasible solution:
z
x3
x4
x2
= 15 + 7x1
= 7 − x1
= 3 − 3x1
= 3 + x1
Computer Science 511 (Iowa State University)
− 5x5
− x5
− 2x5
− x5
(x1 , x2 , x3 , x4 , x5 ) = (0, 3, 7, 3, 0)
Objective value: 15
Entering variable: x1
Leaving variable x4
Linear Programming
September 24, 2010
16 / 27
Dictionary 2
Basic feasible solution:
z
x3
x4
x2
= 15 + 7x1
= 7 − x1
= 3 − 3x1
= 3 + x1
− 5x5
− x5
− 2x5
− x5
(x1 , x2 , x3 , x4 , x5 ) = (0, 3, 7, 3, 0)
Objective value: 15
Entering variable: x1
Leaving variable x4
Dictionary 3
z
= 22 −
x3 =
6 +
x1 =
1 −
x2 =
4 −
7
3 x4
1
3 x4
−
1
3 x4
1
3 x4
+
Computer Science 511 (Iowa State University)
−
−
1
3 x5
5
3 x5
Basic feasible solution:
2
3 x5
1
3 x5
Objective value: 22
(x1 , x2 , x3 , x4 , x5 ) = (1, 4, 6, 0, 0)
Linear Programming
Optimal
September 24, 2010
17 / 27
Unboundedness
Fact
Simplex can be modified to discover unboundedness.
Method
If a linear program is unbounded, at some point, Simplex will find a
non-basic xi such that
xi has a positive coefficient in the objective function and
none of the constraints binds xi .
At this point, stop and return unbounded.
Computer Science 511 (Iowa State University)
Linear Programming
September 24, 2010
18 / 27
Cycling
Successive pivots may encounter solutions with same objective value.
Reason: the same vertex may be defined by different sets of
constraints.
Cycling — i.e., getting stuck at a vertex — is a possibility.
Bland’s rule
When choosing entering or leaving variable, always pick the one with
smallest index.
Fact
Using Bland’s rule, Simplex will never cycle.
Computer Science 511 (Iowa State University)
Linear Programming
September 24, 2010
19 / 27
Finding an Initial Basic Feasible Solution
The origin, (x1 , . . . , xn ) = (0, . . . , 0), is not always feasible.
Happens when bi < 0 for some i.
Example
maximize
subject to
2x1 + 5x2
2x1
x1
−x1
−x1
x1
Computer Science 511 (Iowa State University)
− x2
+ 2x2
+ x2
− x2
,
x2
Linear Programming
≤
4
≤
9
≤
3
≤ −1
≥
0
September 24, 2010
20 / 27
Finding an Initial Basic Feasible Solution
Set up and solve an auxiliary LP.
Auxiliary LP always has a solution.
I
An initial b.f.s. for auxiliary problem can be obtained easily.
Optimal solution to auxiliary LP gives an initial b.f.s. for original LP.
Optimal solution to auxiliary problem also tells us if original problem
is feasible.
Computer Science 511 (Iowa State University)
Linear Programming
September 24, 2010
21 / 27
The Auxiliary Problem: Example
Original Problem
max 2x1
subj. to
2x1
x1
−x1
−x1
x1
+
Auxiliary Problem
5x2
− x2
+ 2x2
+ x2
− x2
,
x2
≤
4
≤
9
≤
3
≤ −1
≥
0
max −x0
subj. to
−x0
−x0
−x0
−x0
x0
+
+
−
−
,
2x1
x1
x1
x1
x1
−
x2
+ 2x2
+
x2
−
x2
,
x2
≤
4
≤
9
≤
3
≤ −1
≥
0
The auxiliary problem has an obvious solution with x1 , x2 = 0.
I
Not a b.f.s., though.
The original problem also has a solution, but it is less obvious.
Computer Science 511 (Iowa State University)
Linear Programming
September 24, 2010
22 / 27
A B.F.S. for the Auxiliary Problem: One Pivot is Enough
Auxiliary Problem in Slack Form
z
x3
x4
x5
x6
=
0 − x0
=
4 + x0
=
9 + x0
=
3 + x0
= −1 + x0
Not a legal dictionary:
− 2x1
−
x1
+
x1
+
x1
+
−
−
+
x2
2x2
x2
x2
(x0 = x1 = x2 = 0) ⇒ (x6 = −1)
Legal Starting Dictionary
Make x0 enter the basis
and x6 leave.
Not a “normal” pivot,
since objective value
decreases.
Computer Science 511 (Iowa State University)
z
x3
x4
x5
x0
=
=
=
=
=
Linear Programming
−1
5
10
4
1
+
−
−
x1
3x1
2x1
−
x1
+
x2
− 3x2
− 2x2
− x2
−
+
+
+
+
September 24, 2010
x6
x6
x6
x6
x6
23 / 27
The Auxiliary Problem: General Case
Original
maximize
n
X
cj xj
j=1
subject to
n
X
aij xj ≤ bi
i = 1, . . . , m
j=1
xj ≥ 0
j = 1, . . . n
Auxiliary
−x0
maximize
subject to
n
X
aij xj − x0 ≤ bi
i = 1, . . . , m
j=1
xj ≥ 0
Computer Science 511 (Iowa State University)
Linear Programming
j = 0, . . . n
September 24, 2010
24 / 27
The Auxiliary Problem
Fact
The original LP is feasible if and only if optimum objective value for the
auxiliary LP is 0.
Proof.
Suppose (x1 , . . . , xn ) is a feasible solution to the original problem.
I
I
I
⇒ x = (0, x1 , . . . , xn ) is a feasible solution to the auxiliary problem.
The objective value of x is 0.
Since we require x0 ≥ 0, x must be an optimal solution.
Suppose the optimum solution to the auxiliary problem has value 0.
I
I
⇒ x0 = 0.
⇒ (x1 , . . . , xn ) is a feasible solution for the original problem.
Computer Science 511 (Iowa State University)
Linear Programming
September 24, 2010
25 / 27
The Fundamental Theorem of Linear Programming
Theorem
For any linear program in standard form:
1
if there is no optimal solution, then the problem is either infeasible or
unbounded,
2
if a feasible solution exists, then a basic feasible solution exists,
3
if an optimum solution exists, then there exists a basic feasible
solution that is also an optimum solution.
Computer Science 511 (Iowa State University)
Linear Programming
September 24, 2010
26 / 27
Running Time of the Simplex Algorithm
Time per pivot is polynomial.
Number of basic feasible solutions is ≤ m+n
n .
In the worst case, Simplex may have to run through most of them.
I
Run time of Simplex is exponential in worst case.
However, this rarely happens.
I
In practice Simplex is fast.
There are polynomial time algorithms for LP (Khachiyan 1979,
Karmarkar 1984)
Computer Science 511 (Iowa State University)
Linear Programming
September 24, 2010
27 / 27