Linear Programming
• LP is a general method to solve
optimization problem with linear
objective function and linear constraints.
• Diet Problem Example: Feeding an army.
• 4 menu choices: Fish, Pizza, Hamburger,
Burrito.
• Each choice has benefits (nutrients) and a
cost (calories).
F
P
H
B
RDA
A
200
75
275
500
2000
C
90
80
80
95
300
D
100
250
510
200
475
Calories
600
800
550
450
• Determine least-caloric diet that satisfies
RDA of nutrients.
Subhash Suri
UC Santa Barbara
LP Formulation
• Introduce variables xF , xP , xH , xB for the
amount (units) consumed of each food.
• Obj. Function is total calories.
minimize 600xF + 800xP + 550xH + 450xB
• Each of the nutrient constraint can be
expressed as a linear function:
200xF + 75xP + 275xH + 500xB
≥ 2000
90xF + 80xP + 80xH + 95xB
≥ 300
100xF + 250xP + 510xH + 200xB
≥ 475
• Finally, all food quantities consumed
should be non-negative:
xF ≥ 0; xP ≥ 0; xH ≥ 0; xB ≥ 0
• This is an example of Linear Prog.
Subhash Suri
UC Santa Barbara
Manufacturing Example
• A plant has 3 types of machines, A, B, C.
• Choice of 4 products: 1, 2, 3, 4.
• Resource consumption (how much
machine time a product uses) and profit
per unit.
1
2
3
4
Avail.
A
2
2
0.5
1.5
20/wk
B
0.5
2
1
2
30/wk
C
1.5
1
3
1.5
15/wk
Profit
3.5
4.2
6.5
3.8
• Machine shop operates 60 hrs/week.
• Determine most profitable product mix.
Subhash Suri
UC Santa Barbara
LP Formulation
• Introduce variables x1, x2, x3, x4 for the
units of different products manufactured.
• Obj. Function is total profit.
maximize 3.5x1 + 4.2x2 + 6.5x3 + 3.8x4
• Each of the machines provides a
constraint, expressed as a linear function:
2x1 + 2x2 + 0.5x3 + 1.5x4 ≤ 1200
0.5x1 + 2x2 + x3 + 2x4 ≤ 1800
1.5x1 + x2 + 3x3 + 1.5x4 ≤ 900
• Finally, all product quantities should be
non-negative:
x 1 , x 2 , x 3 , x4 ≥ 0
Subhash Suri
UC Santa Barbara
Transportation Example
• m supply sites, and n demand sites.
• Supply site i produces ai units;
demand site j requires bj units.
• cij cost of shipping one unit from i to j.
Supply Nodes
Demand Nodes
b1
a1
..
b2
a2
am
..
bn
• Determine optimal shipping cost.
Subhash Suri
UC Santa Barbara
LP Formulation
• Let xij be the amount shipped from i to j.
Then, the LP is
n
m X
X
cij xij
min
i=1 j=1
• subject to
n
X
xij
= ai,
i = 1, 2, . . . , m
xij
= bj ,
j = 1, 2, . . . , n
j=1
m
X
i=1
m
X
ai =
i=1
xij
Subhash Suri
n
X
bj
j=1
≥ 0
UC Santa Barbara
General Form of LP
• n unknowns x1, . . . , xn, and m constraints.
max c1x1 + c2x2 + · · · + cnxn
• subject to
a11x1 + a12x2 + · · · + a1nxn ≤ b1
a21x1 + a22x2 + · · · + a2nxn ≤ b2
..
am1x1 + am2x2 + · · · + amnxn ≤ bm
x 1 , x2 , · · · , xn ≥ 0
• A company making n products, each
requiring some amount each of m
resources. Item i has profit ci.
• bj is total resource of type j. aij is the
amount of resource j needed by item i.
Subhash Suri
UC Santa Barbara
Matrix Form
• maximize
• subject to
cx
Ax ≤ b
• x≥0
• The input data are:
– the m × n coefficient matrix A,
– the m-vector of right hand sides b, and
– the n-vector of objective costs c.
Subhash Suri
UC Santa Barbara
Slack Variables
• An inequality ai1x1 + · · · + ainxn ≤ bi can be
converted to an equation by introducing a
slack variable si ≥ 0:
ai1x1 + · · · + ainxn + si = bi
• Similarly, an inequality like
can be written as
Pn
j=1 aij xj
≥ bi
ai1x1 + · · · + ainxn − si = bi
• The Standard form Linear Program is
maximize
cx
subject to
Ax = b
x≥0
• Slack variable coefficients in the obj
function are typically zero.
Subhash Suri
UC Santa Barbara
Geometrical Method
• 2-variable LP can be solved graphically.
• Useful teaching tool: Helps build intuition
about the problem.
• Not a practical method for practical linear
programming problems.
• We plot constraints on a rectangular
coordinate system.
• A constraint is drawn as a line. The
inequality constraint restricts the feasible
region to one side of the line.
• The non-negative constraints x1, x2 ≥ 0
restrict the feasible region to the first
quadrant.
Subhash Suri
UC Santa Barbara
Example
• We will consider the following LP:
max z = 2x1 + 3x2
• subject to
x1 − 2x2 ≤ 4
2x1 + x2 ≤ 18
x2 ≤ 10
x1, x2 ≥ 0
Subhash Suri
UC Santa Barbara
Illustration
• Building the feasible region, one
constraint at a time.
x2
x2
x1 − 2x2 = 4
x1
x1
x2
x2
x2 = 10
2x1 + x2 = 18
Feasible
Region
x1
Subhash Suri
x1
UC Santa Barbara
Optimal Solution
• Examine Level curves: z = 1, 2 etc.
• E.g. z = 24 defines the line 2x1 + 3x2 = 24.
Similarly, for z = 30.
• Parallel lines–they have same slope.
• Line with highest z value that intersects
feasible region.
x2
(0,10)
.
(4,10)
.
Optimal Extreme
Point
z* = 38
z = 30
z = 24
. (8,2)
(2,3)
(0,0)
Subhash Suri
.
(4,0)
x1
UC Santa Barbara
Algebra of LP
x2
(0,10)
.
(4,10)
.
Optimal Extreme
Point
z* = 38
z = 30
z = 24
. (8,2)
(2,3)
(0,0)
.
(4,0)
x1
• Important points:
Feasible region is convex polyhedron,
optimal solution is a corner.
• Simplex Method is a famous method for
solving LP, invented by George Dantzig.
• Introduce algebra and geometry to
explain how it works.
Subhash Suri
UC Santa Barbara
Linear Algebra
H+ = {x : ax >= b}
a
H: ax = b
H− = {x : ax <= b}
• A hyperplane in Rn is the set of points
x = {x1, x2, . . . , xn} that satisfy ax = b,
where a ∈ Rn and b ∈ R.
• Examples: Line a1x1 + a2x2 = b in R2.
• H + = {x : ax ≥ b} is the positive halfspace.
• Vector a is gradient of the linear function
f (x) = ax, and so is normal to H.
• A polyhedral set is the intersection of
halfspaces.
Subhash Suri
UC Santa Barbara
Convexity
Convex Sets
x
x
y
y
Non−Convex Sets
• A set S is convex if for any two points
x, y ∈ S, the line segment xy also lies in S.
• That is, αx + (1 − α)y ∈ S if x, y ∈ S, for all
α ∈ [0, 1].
• Examples: A hyperplane is convex.
A halfspace is convex.
Intersection of convex sets is convex.
• αx + (1 − α)y, where α ∈ [0, 1] is called
convex combination of x, y.
• αx + (1 − α)y, for α ∈ (−∞, ∞) is called
linear combination of x, y.
Subhash Suri
UC Santa Barbara
Extreme Points
.
.
.
.
.
Extreme Points
All boundary points
extreme
• A point x is extreme point of S if it is not
a strict convex combination of two other
points of S.
• Corners of a polyhedron are extreme.
All boundary points of a sphere are
extreme.
• Let S = {x : Ax = b, x ≥ 0}, where A is
m × n matrix of rank m ≤ n.
• A point x is an extreme point of S iff x is
the intersection of n linearly independent
hyperplanes.
Subhash Suri
UC Santa Barbara
Basic Feasible Solution
• LP Ax = b. Assume rank(A) = m ≤ n.
• Partition cols as A = (B : N ), where
B is m × m basis matrix
N is m × (n − m) non-basis matrix.
• The LP system becomes
BxB + N xN = b
• Because B is non-singular, it is invertible:
xB = B −1b − B −1N xN
• Set xN = 0. We get xB = B −1b.
• xB is vector of basic variables,
xN vector of non-basic vectors.
• If xB ≥ 0, then the solution x =
called a basic feasible solution.
Subhash Suri
¡B −1b¢
0
is
UC Santa Barbara
Example of BFS
• Consider linear system:
2x1 + x2 + x3 + x4 = 15
x1 + 3x2 + x3 − x5 = 12
µ
• A=
2 1 1 1
0
1 3 1 0 −1
¶
µ
, b=
¶
15
.
12
• Consider basis matrix of 1st and 3rd cols:
µ
¶
2 1
B=
.
1 1
¶
µ
1 −1
−1
.
• Inverse matrix is B =
−1
2
µ
¶
xB
• BFS is x =
, where xN = 0, and
x
µ N¶
x1
xB =
= B −1b
x3
µ
¶µ
¶ µ ¶
1 −1
15
3
=
=
−1
2
12
9
Subhash Suri
UC Santa Barbara
Fundamental Theorem
Theorem: x is an extreme point of
S = {x : Ax = b, x ≥ 0} iff x is a BFS.
x2
Linear System
(0,10)
.
x5=0
(4,10)
.
x1 − 2*x2 + x3 = 4
2*x1 + x2 + x4 = 18
x2 + x5 = 10
x1, x2, x3, x4, x5 >= 0
x4=0
x1 = 0
. (8,2)
(0,0)
x2=0
.
(4,0)
x3=0
x1
• Each hyperplane ↔ variable, which is 0 on
the hyperplane.
• Each corner has 2 variables zero, and 3
non-zero.
Subhash Suri
UC Santa Barbara
Simplex Example
• Consider LP max z = 2x1 + 3x2
• subject to
x1 − 2x2 + x3 = 4
2x1 + x2 + x4 = 18
x2 + x5 = 10
x1, x2, x3, x4, x5 ≥ 0
• Initial basic feasible solution:
(x3, x4, x5) = (4, 18, 10), and (x1, x2) = (0, 0).
• Objective function value is zero.
• Entering variable. Pick a xi with ci > 0
(max) in the objective, and make it basic.
(E.g. x2)
• Departing variable. Increase x1 from zero,
until some basic variable becomes zero
(and non-basic).
• Repeat until all vars in objective function
have negative coefficients.
Subhash Suri
UC Santa Barbara
Simplex Example
• Consider LP max z = 2x1 + 3x2
• subject to
x1 − 2x2 + x3 = 4
2x1 + x2 + x4 = 18
x2 + x5 = 10
x1, x2, x3, x4, x5 ≥ 0
• Initial BFS form:
x3 = 4 − x1 + 2x2
x4 = 18 − 2x1 − x2
x5 = 10 − x2
• Set x1, x2 = 0, x3 = 4, x4 = 18, x5 = 10. z = 0.
• Entering variable is x2.
• Max value of x2 is min{∞, 18, 10}
• Departing variable is x5.
Subhash Suri
UC Santa Barbara
Simplex Example
• Re-arrange the linear system:
x2 = 10 − x5
x3 = 24 − x1 − 2x5
x4 = 8 − 2x1 + x5
• Set x1 = x5 = 0, and x2 = 10, x3 = 24, x4 = 8.
• Obj. function becomes
z = 30 + 2x1 − 3x5 = 30.
• Entering variable is x1.
• Max value of x1 is min{∞, 24, 4}.
• Departing variable is x4.
Subhash Suri
UC Santa Barbara
Simplex Example
• Re-arrange the linear system:
1
1
x1 = 4 − x4 + x5
2
2
x2 = 10 − x5
1
5
x3 = 20 + x4 − x2
2
2
• Set x4 = x5 = 0, and x1 = 4, x2 = 10, x3 = 20.
• Obj. function becomes
z = 38 − x4 − 2x5 = 38.
• This is optimal because all coeffs are
negative.
Subhash Suri
UC Santa Barbara
Complexity of LP
• Simplex invented by G. Dantzig in 1947.
• One of the most popular and widely used
optimization method. Many commercial
software packages.
• Extremely fast in practice, solves LP with
105 constraints and variables in secs or
mins.
• Worst-case complexity exponential!.
Klee-Minty (1972)
• Poly-time LP algorithms in 1980’s.
Khachian (1979). Much worse in practice!
• Interior point method of Karmakar (1984)
first method to be theoretically in P and
comparable in practice.
• Open Problem: Is there a pivoting rule
for which simplex is polynomial?
Subhash Suri
UC Santa Barbara
LP Duality
• Duality turns out to be a deep property of
linear programming.
• Many famous theorems in other areas are
an easy consequence of LP Duality.
• In order to motivate its power, we
consider an investment game.
• Investment Game played during Mon-Fri
period.
• Rules: If invest x one morning, and 2x
next morning, then receive 4x on third
morning.
• Amount received on day D can be
invested same day for next round.
• Inability to pony up 2x on second day
forfeits original x.
• Determine optimal investment strategy.
Subhash Suri
UC Santa Barbara
Strategy 1
• Starting fortune is $1.
• Invest $1/3 on Mon, and $2/3 on Tues.
• You get $4/3 on Wed, of which invest
$4/9.
• Invest remaining $8/9 on Thurs.
• Collect $16/9 on Friday.
• Total gain is $7/9.
• Can we do better?
Subhash Suri
UC Santa Barbara
Strategy 2
Mon
1/4
Tues
Wed
1/2
1/4
(1)
1/2
1/2
Thurs
Fri
(1)
1
(2)
• Total gain is $1.
• Can we do still better?
• This is a maximization LP, and we want
an upper bound on the solution.
• LP duality aims to answer this question.
Subhash Suri
UC Santa Barbara
Illustrating Example
• Consider LP max z = 4x1 + x2 + 5x3 + 3x4
• subject to
x1 − x2 − x3 + 3x4 ≤ 1
5x1 + x2 + 3x3 + 8x4 ≤ 55
−x1 + 2x2 + 3x3 − 5x4 ≤ 3
xi ≥ 0
• Suppose instead of computing optimal z ∗,
we just want a quick estimate of the max
possible (upper bound).
• We can get lower bounds by trying
feasible solutions:
(0, 0, 1, 0) → z ∗ = 5
(2, 1, 1, 1/3) → z ∗ = 15
(3, 0, 2, 0) → z ∗ = 22
• However, this method still gives no
estimate of an upper bound.
Subhash Suri
UC Santa Barbara
Illustrating Example
• LP max z = 4x1 + x2 + 5x3 + 3x4
• subject to
x1 − x2 − x3 + 3x4 ≤ 1
5x1 + x2 + 3x3 + 8x4 ≤ 55
−x1 + 2x2 + 3x3 − 5x4 ≤ 3
xi ≥ 0
• Multiply 2nd constraint by 5/3:
5
15
40
25
x1 + x2 + x3 + x4 ≤
3
3
3
3
275
3
• This implies z ∗ ≤ 275
3 , because term by
term this constraint dominates obj.
function z.
• One more. Add constraints (2) and (3),
4x1 + 3x2 + 6x3 + 3x4 ≤ 58
• This implies z ∗ ≤ 58.
Subhash Suri
UC Santa Barbara
General Strategy
• Form linear combinations of constraints.
• Multiply constraint (i) by yi, then add
y1(x1 − x2 − x3 + 3x4)+
y2(5x1 + x2 + 3x3 + 8x4)+
y3(−x1 + 2x2 + 3x3 − 5x4) ≤ y1 + 55y2 + 3y3.
• Rewrite
(y1 + 5y2 − y3)x1+ (−y1 + y2 + 2y3)x2+
(−y1 + 3y2 + 3y3)x3+ (3y1 + 8y2 − 5y3)x4 ≤
y1 + 55y2 + 3y3.
• For this to dominate z term by term
y1 + 5y2 − y3 ≥ 4
−y1 + y2 + 2y3 ≥ 1
−y1 + 3y2 + 3y3 ≥ 5
3y1 + 8y2 − 5y3 ≥ 3
yi ≥ 0
• We then have z ≤ y1 + 55y2 + 3y3.
Subhash Suri
UC Santa Barbara
Dual LP
• Our goal is to find smallest z subject to
constraints on yi.
• We have a linear program:
• minimize u = y1 + 55y2 + 3y3
• subject to
y1 + 5y2 − y3 ≥ 4
−y1 + y2 + 2y3 ≥ 1
−y1 + 3y2 + 3y3 ≥ 5
3y1 + 8y2 − 5y3 ≥ 3
yi ≥ 0
• This LP is called the Dual of the original
LP.
Subhash Suri
UC Santa Barbara
Duality Theorems
Weak Duality: If z and u are feasible
solutions of the primal and dual LPs, then
z≤u
• Follows by definition of dual LP. Each
feasible solution of dual is an upper bound
on primal.
Strong Duality: If primal LP has finite
optimal solution z ∗, then the dual also has a
finite optimal solution u∗ and
z ∗ = u∗
Subhash Suri
UC Santa Barbara
Back to Investment Game
• Variables xi to denote investment
amounts.
Mon
x1
Tues
2x1
x2
Wed
(4x1)
2x2
x3
Thur
Fri
(4x2)
2x3
(4x3)
• The initial amount is $1.
• The final amount is 1 + x1 + x2 + x3.
• Formulate as LP: max 1 + x1 + x2 + x3
1 − x1 ≥ 0
1 − 3x1 − x2 ≥ 0
1 + x1 − 3x2 − x3 ≥ 0
1 + x1 + x2 − 3x3 ≥ 0
xi ≥ 0
Subhash Suri
UC Santa Barbara
Back to Investment Game
• Investment game LP: max 1 + x1 + x2 + x3
• subject to
1 − x1 ≥ 0
1 − 3x1 − x2 ≥ 0
1 + x1 − 3x2 − x3 ≥ 0
1 + x1 + x2 − 3x3 ≥ 0
xi ≥ 0
• Linear combination 2 ∗ (2) + (3) + (4):
4x1 + 4x2 + 4x3 ≤ 4, which implies
1 + (x1 + x2 + x3) ≤ 2
• Thus, strategy 2 is optimal.
Subhash Suri
UC Santa Barbara
Integer Programming
• maximize
cx
• subject to
• x integer
Ax ≤ b
(only diff from LP)
• IP feasible space is discrete, and smaller
than LP feasible space, which is
continuous.
IP sol
LP sol
. . .
. . . . . .
. . . . .
Subhash Suri
UC Santa Barbara
Integer Programming
• One might think that restricting variables
makes problem easier to solve.
• In fact, IP is NP-Complete.
• Reduction from Boolean 3-SAT.
• Φ = (x1 ∨ x3 ∨ x4) ∧ (x2 ∨ x3 ∨ x4) ∧ · · ·
• Boolean Satisfiability of Φ can be encoded
as an IP:
• minimize
x1
• subject to
x1 + (1 − x3) + x4 ≥ 1
(1 − x2) + x3 + (1 − x4) ≥ 1
..
xi
Subhash Suri
∈
{0, 1}
UC Santa Barbara
Formulating IP Problems
• Knapsack Problem: n items, weights ai,
capacity b.
Pn
• maximize
i=1 ci xi
• subject to
n
X
aj xj
≤ b
i=1
xi
binary
• Vertex Cover: G = (V, E)
Pn
• minimize
i=1 xi
• subject to
xi + xj
xi
Subhash Suri
≥ 1, for each edge (i, j) ∈ E
binary
UC Santa Barbara
Network Problems using IP
• Min Cost Network Flows.
• Given is a network G = (V, E), supply bi for
each node i, capacity cij for each edge, and
lower and upper bounds on flow for each
edge.
P
• minimize
cij xij
• subject to
X
j:(i,j)∈E
xij −
X
xji = bi
j:(j,i)∈E
lij ≤ xij ≤ uij , for all (i, j) ∈ E
• No need to restrict xij to integers.
Subhash Suri
UC Santa Barbara
Network Problems using IP
• Shortest Path: 1 unit flow from s to t.
P
• minimize
cij xij
• subject to
X
xsj
= 1
j:(s,j)∈E
X
X
xjt = −1
j:(j,t)∈E
xij −
X
xji = 0
xij
∈
if i 6= s, t
{0, 1} for all (i, j) ∈ E
s
Subhash Suri
UC Santa Barbara
Network Problems using IP
• Assignment Problem.
P
• minimize
cij xij
• subject to
X
xij
= 1
for all i ∈ X
xji = 1
for all i ∈ Y
j:(i,j)∈E
X
j:(j,i)∈E
xij
∈
X
Subhash Suri
{0, 1}
for all (i, j) ∈ E
Y
UC Santa Barbara
LP Relaxation of IP
• An IP formulation with integrality
constraint removed is called the LP
relaxation.
• If IP is max cx, with optimal solution zI ,
and zL is the optimal solution of relaxed
LP, then
zI ≤ zL
• It follows because IP feasible space is a
subset of LP feasible space.
• When can one guarantee that zL = zI ?
• Specifically, shortest path and assignment
formulations require integer variables.
But they have combinatorial poly-time
algorithms.
Subhash Suri
UC Santa Barbara
Total Unimodularity
• Consider an IP: max cx s.t. Ax ≤ b, and
x integer.
• The constraint matrix A is called Totally
Unimodular (TUM) if det(B) is in
{0, 1, −1} for all 2 × 2 submatrices B of A.
• E.g. if each col of A has exactly one 1, one
-1, and all other zero entries, then A is
TUM.
• If A is the incidence matrix of a graph,
then A is TUM.
• Thus, our network problems (shortest
path, maxflow, assignment) can be solved
through LP relaxation.
Subhash Suri
UC Santa Barbara