Linear Programming
Reading: CLRS, Ch. 29 or reference
CSE 6331 Algorithms
Steve Lai
Linear functions and linear constraints
Let a1 , a2 ,
let x1 , x2 ,
, an , b be real numbers and
, xn be variables.
Linear function:
f ( x1 , x2 , , xn ) a1 x1 a2 x2
a n xn
Linear equality:
f ( x1 , x2 , , xn ) b
(hyperplane)
Linear inequalities:
f ( x1 , x2 , , xn ) b
(half-space)
, xn ) b
(half-space)
f ( x1 , x2 ,
2
General Linear Programming Problems
Given real numbers bi , c j , aij , where 1 i m, 1 j n,
find real numbers x1 , x2 ,
, xn to
optimize a linear objective function
c1 x1 c1 x2
cn xn
subject to linear constraints
aij x j bi
j 1
n
for 1 i m.
3
Standard Maximum Form
Given real numbers
aij 1 i m, 1 j n
bi 1 i m
cj 1 j n
find real rnumbers x1 , x2 ,
maximize
c1 x1 c2 x2
n
subject to
, xn that
a
j 1
ij
cn xn
xj
bi
for 1 i m
xj
for 1 j n
0
4
In Matrix Notation
( c1 x1 c2 x2
maximize
cT x
subject to
Ax b and x 0
a11
a
21
am1
a12
a22
a m1
cn xn )
a1n x1 b1
a2 n x2 b2
amn xn bn
See the next slide for notation.
5
Notation
n-vectors: x T x1 , x2 ,
, xn , cT c1 , c2 ,
m-vectors: bT b1 , b2 ,
, bm , y T y1 , y2 ,
, cn
, ym
m n-matrix:
a11
a
A 21
am1
a12
a22
am1
a1n
a2 n
amn
6
Standard Minimum Form (in Matrix Notation)
minimize
yT b
subject to
y T A cT
y1
y2
a11
a
ym 21
am1
( y1b1 y2 b2
ym bm )
and y 0
a12
a22
am1
a1n
a2 n
c1
amn
c2
cn
7
Terminology
Variables: x1 , x2 ,
, xn
Feasible solution: a vector (x1 , x2 ,
, xn ) that
satisfies all the constraints.
Feasible region : the set of all feasible solutions.
Optimal solution: a feasible solution that optimizes the
objective function.
Optimal objective value f *:
A linear program may be
feasible ( is nonempty) or infeasible ( is empty)
(feasible) bounded or unbounded (f * is finite or infinite)
8
Example
x1 x2
maximize
objective function
subject to
x1
2 x2
4 x1
2 x2
12
x1
x2
1
x1 , x2
0
4
main constraints
nonnegativity constraints
9
Source: Linear programming, Thomas S.
Ferguson, UCLA
10
Equivalence of Linear Programs
Two maximization LPs, L and L, are equivalent if
for each feasible solution x to L with objective value z
there is a corresponding feasible solution x to L with
objective value z, and vice versa.
A maximization LP, L, and a minimization LP, L,
are equivalent if for each feasible solution x to L with
objective value z there is a corresponding feasible
solution x to L with objective value z, and
vice versa.
11
Converting to Standard Maximum Form
We can always convert an LP into an equivalent LP in
standard maximum (or minimum) form.
A linear program may not be in standard maximum form
for any of these reasons:
1. The objective funtion may be a minimization.
2. There may be " " (instaed of " ") inequality constraints.
3. There may be equality constraints.
4. There may be some xi without nonnegativity constraints.
12
1. Convert minimization to maximixation:
Minimize f ( x1 , , xn ) maximize f ( x1 ,
, xn )
2. Convert a " " constraint to a " " constraint:
n
a
j 1
ij
x j bi
n
( a
j 1
ij
) x j bi
3. Convert an equality constraint into two inequalities:
n
n
aij x j bi
aij x j bi
n
j 1
j 1
aij x j bi n
n
j 1
a x b
( a ) x b
ij j
i
ij
j
i
j 1
j 1
13
4. If x j is without nonnegativity constraint,
we replace x j with x j x j and add x j 0, x j 0.
Thus, c j x j becomes c j x j c j x j , and
a j x j becomes a j x j a j x j .
A feasible solution x1 ,
, x j , x j ,
corresponds to a feasible solution x1 ,
, xn to the new LP
, xj,
, xn
to the original LP, where x j x j x j .
14
Example
minimize 2 x1 3x2
maximize 2 x1 3 x2
subject to
subject to
x1
x2
7
x1
2 x2
4
x1
0
x1
x2
x1
x2
7
x1
x1
2 x2
7
4
0
15
Replace x2 with x2 x2 :
maximize 2 x1 3x2
maximize 2 x1 3x2 3x2
subject to
subject to
x1
x1
x1
x1
x2
x2
2 x2
7
7
4
0
x2
x2
x2
x2
x1
2 x2 2 x2
x1 , x2 ,
x2
x1
x1
7
7
4
0
16
Renaming x2 , x2 as x2 , x3 yields:
maximize 2 x1 3x2 3x3
subject to
x1
x1
x1
x2
x2
2 x2
x1 , x2 ,
x3
x3
x3
2 x3
7
7
4
0
17
Duality
Each LP has an intimately related LP, called its dual.
The original LP is called the primal.
Primal (standard maximum LP):
maximize
cT x c1 x1 c2 x2
subject to
Ax b and x 0
cn xn
Dual (standard minimum LP):
minimize
y T b y1b1 y2 b2
subject to
y T A cT
ym bm
and y 0
18
The dual of the Standard Minimum LP is the
Standard Maximum LP.
The primal max LP and the dual min LP can be
simultaneously displayed as:
y1
y2
ym
x1
a11
a21
x2
a12
a22
xn
a1n
a2 n
am1
c1
am 2
c2
amn
cn
b1
b2
bm
19
Example
Primal problem:
maximize
Dual problem:
x1 x2
minimize
subject to
4y1 12 y2 y3
subject to
x1
4 x1
2 x2
2 x2
4
12
x1
x2
1
x1 , x2
0
y1
2 y1
4 y2
2 y2
y3
y3
y1 , y2 , y3
1
1
0
20
Example
x1
x2
y1
1
2
4
y2
4
2
12
y3
1
1
1
1 1
21
Weak Duality Theorem
Theorem. If x is a feasible solution to the standard maximum
LP (A,b,c ) and y a feasible solution to its dual, then cT x y T b.
Proof.
Ax b, x 0 y T Ax y T b
T
T
c
x
y
b.
T
T
T
T
y A c , y 0 y Ax c x
Corollary. If a standard LP and its dual are both feasible,
then they are bounded.
Corollary. If x and y are feasible solutions to the primal and
dual LPs, respectively, and cT x y T b, then
x and y are optimal solutions to their respective problems.
22
Strong Duality Theorem
Theorem. A feasible solution x to the primal linear program
(A,b,c ) is optimal iff there is a feasible solution y to its dual
linear program such that cT x y T b. In this case, y is also
optimal.
Corollary: The primal LP has an optimal solution iff its dual
has an optimal solution, in which case, their optimal values
are equal.
23
Primal
feasible bounded feasible unbounded infeasible
Dual
f.b.
yes
no
no
f.u.
no
no
yes
i
no
yes
yes
24
LP in equality form
Find
x
to
maximize cT x
subject to Ax b and x 0
where
xT x1 , x2 ,
, xn m , bT b1 , b2 ,
cT c1 , c2 ,
, cn m
, bm
a11 x1
a21 x1
+
+
a1n xn
a2 n xn
a1, n m xn m b1
a2,n m xn m b2
am1 x1
+
amn xn
am, n m xn m bm
25
Basic Solutions
Assume rank( A) m.
Let B be a non-singular matrix formed by choosing m
columns out of the m n columns of A.
The m variables xi associated with the columns in B are
called basic variables; the other n variables are non-basic
variables.
If all the n non-basic variables are set to zero, then the
solution to the resulting system of equations is called a
basic solution.
If a basic solution is feasible, it is a basic feasible solution.
The number of basic solutions is at most C
mn
m
( m n )!
.
m ! n ! 26
Basic feasible solutions are extreme points
A set C is said to be convex if for all points x1 , x2 C
we have x1 (1 ) x2 C for all 0 1.
A point x in a convex set C is called an extreme point
if there exist no x1 , x2 C , x1 x2 , and 0 1,
such that x x1 (1 ) x2 .
The feasible region of an LP is a convex set.
Basic feasible solutions of an LP (in the equality form)
are extreme points of the feasible region.
If the LP is feasible bounded, then at least one
optimum solution occurs at an extreme point.
27
Simplex Method
A system of linear
inequalities defines a
polytope (or simplex) as
a feasible region.
v̂
The simplex algorithm
begins at a starting
vertex and moves along
the edges of the polytope
until it reaches the vertex
of the optimum solution.
Source: Linear programming, Thomas S.
Ferguson, UCLA
28
Example
x1 x2
maximize
objective function
subject to
x1
2 x2
4 x1
2 x2
12
x1
x2
1
x1 , x2
0
4
main constraints
nonnegativity constraints
29
Example
Standard max form
Slack form
x1 x2
maximize
x1 x2
maximize
subject to
subject to
x1
4 x1
2 x2
2 x2
4
12
x1
4 x1
2 x2
2 x2
s1
s2
4
12
x1
x2
1
x1
s3
1
x1 , x2
0
x1 , x2 , s1 , s2 , s3
0
x2
30
Source: Linear programming, Thomas S.
Ferguson, UCLA
31
Converting standard form to slack form
Standard maximum form:
Find x to
maximize
cT x
subject to
Ax b and x 0
Equivalent slack form:
Find x and sT ( s1 , s2 ,
, sm ) to
(or minimize cT x )
maximize
cT x
subject to
Ax s b and x 0, s 0
Ax s b may be written as s b Ax or s Ax b.
32
s1 , s2 ,
, sm are called slack variables.
The equality s Ax b and the objective
function z cT x v can be displayed in a tableau:
(to minimize
z cT x v, where v 0)
1 x1
s1 a11
s2 a21
x2
a12
a22
xn
a1n
a2 n
1
b1
b2
sm
z
am 2
cm
amn
cn
bm
v
am1
c1
33
Basic and non-basic variables
maximize f ( x ) cT x
(or minimize cT x)
subject to s Ax b and x 0, s 0:
Initial basic variables: B s1 , s2 ,
, sm
Initial non-basic variables: N x1 , x2 ,
, xn
The solution (x 0, s b) is a basic solution.
If b 0, (x 0, s b) is a basic feasible solution.
If in b 0, c 0, the solution (x 0, s b) is optimum.
All feasible solutions (x, s) have objective value f ( x) 0.
The solution (x 0, s b) has objective value f ( x ) 0.
34
Pivot Operation
Maximize z cT x v s.t. s Ax b and x 0, s 0:
1
s1
x1
a11
xj
a1 j
xn
a1n
1
b1
si
ai1
aij
ain
bi
sm
z
am1
c1
amj
c j
amn
cn
bm
v where v 0 initially
If aij 0, we can pivot around aij , switching the roles
of si and x j
35
Dividing both sides of the equation
si ai1 x1
ai , j 1 x j 1 aij x j ai , j 1 x j 1
ain xn bi
by aij , we have an equivalent equation:
ai1
x j
x
a 1
ij
ai , j 1
aij
si ai , j 1
x j 1
x j 1
aij
aij
ain bi
xn .
aij aij
Substituting this into every other equation yields:
akj ai1
sk ak 1
x1
aij
akj bi
bk
aij
akj
a
ij
si
akj ain
akn
aij
x
n
36
Also, substitute it into z cT x v
( c1 ) x1
( cn ) xn v
and we have:
(c j )ai1
z (c1 )
x1
aij
(c j )ain
( cn )
aij
( c j )
si
aij
(c j )bi
xn v
aij
37
After pivoting around aij , we have the equivalent LP:
1
x1
s1
aˆ11
xj
ai1
aij
sm
aˆm1
z
cˆ1
si
a1 j
aij
1
aij
amj
aij
cˆ j
1
xn
a1n
a1 j ain
aij
ain
aij
a mn
aij
bi
aij
amj ain
cˆn
b1
a1 j bi
aij
bm
amj bi
aij
vˆ
38
Summary of Pivot Operation
p
r
c
q
1 p
r p
c p
q rc p
1. Pivot element: p 1 p
2. Other elements in the pivot row: r r p
3. Other elements in the pivot column: c c p
4. All other elements: q q rc p
Example:
3
0
5
2
3 1
0
4
3
3 2
9 2
12
3 2 1 2
0
4
7 2
3
39
Converting minimum form to slack form
Standard minimum form:
Find
y to
minimize
yT b
subject to
y T A cT
and y 0
Equivalent slack form:
Find y and sT ( s1 , s1 ,
minimize
yT b
subject to
sT y T A cT
, sn ) to
and y 0, s 0
40
The equality sT y T A cT and objective function y T b
can be displayed in a tableau:
s1
s2
sn
y1
a11
a12
a1n
b1
y2
a21
a22
a2 n
b2
ym
am1
am 2
amn
bm
c1
cm
cn
0
B s1 , s2 ,
N y1 , y2 ,
, sm
, yn
The solution y 0, s c is a basic solution.
If c 0, the solution is a basic feasible solution.
If b 0 and c 0, the solution is an optimum solution
41
Pivoting for the dual follows the same rule:
p
r
c
q
1 p
r p
c p
q rc p
So we display both the primal and the dual in the
same tableau.
42
y1
y2
x1
a11
a21
x2
a12
a22
xn
a1n
a2 n
b1
b2
ym
z
am1
c1
am 2
cm
amn
cn
bm
0
43
The Pivot Madly Method
Pivot madly until we suddently find that all entries in the last
row and last column (exclusive of the corner) are nonnegative.
r1
r2
rn
t1
aˆ11
aˆ12
aˆ1n
t2
aˆ21
aˆ22
aˆ2 n
tm
aˆ m1
cˆ1
aˆm 2
cˆm
aˆmn
cˆn
Solution for the primal:
bˆ1
bˆ
2
bˆm
vˆ
x j 0 for x j r1 ,
, rn
xi bˆi for xi t1 ,
, tm
Solution for the dual:
y j cˆ j for r1 ,
, rn cˆ j
yi 0 for yi t1 ,
, tm 0
Optimal value vˆ
44
Example
x1
y2
Max solution for primal:
y3
y1
1
x2
4
y4
x3
1
3
5
2
4
16
x1 0
x2 4, x3 3
Min solution for dual:
y1 0 y4 0
y2 2, y3 4
Optimal value 16
45
Suppose after pivoting for a while we have the tableau:
t
r
A
c
b
v
If b 0, then r 0, t b is a feasible solution for the
maximization problem.
If c 0, then t 0, r c is a feasible solution for the
minimization problem.
If b 0 and c 0, then we have an optimal solution
for both problems.
46
The Simplex Method
The same as the pivot madly method except that we now
choose the point elements more systematically.
The goal is to make all entries in the last row and last
column (exclusive of the corner) nonnegative.
t
r
A
b
c
v
47
Pivot rules for the simplex method (1)
Case 1: b 0. Take any column j0 with c j0 0. Among
those i with ai , j0 0, choose i0 to be the i for which the ratio
bi ai , j0 is smallest. Pivot around ai0 , j0 . (If no such i0 ,
the max problem is feasible unbounded.)
r1
r2
r3
r4
r5
t1
1
1
3
1
6
6
t2
0
3
2
4
4
t3
2
0
3
1
t4
5
1 4
5 2
4
r1
r2
r3
t1
1
3
3
1
4
t2
0
1
2
4
7
7
t3
2
2
3
1
2
3
0
t4
5
1
4
2
2
1 0
5 2
4
0
48
Properties of rule 1:
After the pivot, b stays nonnegative, the entry c j0
becomes positive, and the value v never gets less
(v gets greater if bi0 0).
p
r
c
q
ai0 , j0
bi0
ai , j0
bi
c j0
v
1 p
r p
c p
q rc p
1 ai0 , j0
bi0 ai0 , j0
ai , j0 ai0 , j0
bi ai , j0 bi0 ai0 , j0
c j0 ai0 , j0
v c j0 bi0 ai0 , j0
49
Pivot rules for the simplex method (2)
Case 2: some bi are negative. Take the first bk 0. Find any
negative entry in row k , say ak , j0 0. (If there is no such j0 ,
the max problem is infeasible.) Compare bk ak , j0 with bi ai , j0
for which bi 0, ai0 , j0 0 and choose i0 such that bi0 ai0 , j0
is smallest (i0 may be k .) Pivot around ai0 , j0 .
r1
r4
4
2
1
2
4
4
5
1
3
1
2
1
4
2
0
5
2
4
2
0
1
t2
1
t3
2
1
r3
3
t1
t4
r2
r1
r2
r3
t1
1
3
3
1
t2
0
1
2
4
t3
2
2
3
1
t4
5
1
4
0
50
Properties of rule 2:
The objective in Case 2 is to get to Case 1.
With the pivot, the nonnegative bi stay nonnegative,
and bk becomes no smaller (it gets larger if bi0 0).
p
c
r
q
ai0 , j0
bi0
ai , j0
bi
ak j0
bk
1 p
c p
r p
q rc p
1 ai0 , j0
bi0 ai0 , j0
a
ai , j0 ai0 , j0
bi ai , j0 bi0 ai0 , j0
ak j0 ai0 , j0
bk
b
k j0 i0
ai0 , j0
51
Pivot rules for the simplex method (3)
Case 3: c 0. Treat it as the dual problem.
Take any row i0 such that bi0 0. Among those j with
ai0 , j 0, choose the j0 for which the ratio c j ai0 , j is
closest to zero. Pivot around ai0 , j0 . (If there is no such j0 ,
the dual problem is feasible unbounded.)
r1
r2
r3
r4
t1
1
1
3
1 6
t2
0
3
2
4
4
t3
2
0
3
1
2
5
1
4
2
3
52
Example
Primal: maximize x1 x2 2 x3 subject to all xi 0 and
x1
x2
2 x1
x2
2 x3 3
3 x3 2
x3
1
Dual: minimize 3y1 2 y2 y3 subject to all yi 0 and
y1
y2
2 y3
y3
1
1
2 y1
3 y2
y3
2
53
y1
x1
0
x2
1
x3
2
y2
1
0
3
y3
2
1
1
3
2
y2
1
x2
1 1 2
x1
y3
y1
0
y2
y3
1
x3
1
2
1
0
3
2
2
1
1
1
1
1
1 1
Maximum solution for primal:
y1
5 3 1 1 3 4 3
x3
1 3
x2
7 3
1 1 3
23
1
0
x1
2
13 23
13
13 53
x1 0, x2 1 3, x3 2 3
Minimum solution for dual
y1 0, y2 1 3, y3 1
Optimal value for both: 5 3
54
Cycling
The simplex rules as stated may lead to cycling,
even though it is rare in practical problems.
There are strategies (modified simplex methods) for
avoiding cycling.
Smallest-subscript rule: If there is a choice of pivot rows
or a choice of pivot columns, select the row (or column)
with the x variable having the lowest subscript, or if there
are no x variables, with the y variable having the lowest
subscript.
55
Formulating problems as linear
programs
Single-pair shortest path
Problem: Given a weighted directed graph G (V , E ) and
two vertices s, t V , find the shortest distance from s to t.
For each vertex v V , let d v denote the shortest distance
from s to v. (We want to find d t .)
The values d v satisfy the following constraints:
d s 0 and for each edge ( u, v ) E , d v d u w(u, v ).
If values xv satisfy the above constraints, then xt d t .
Thus, we obtain the following LP (with variables xv v V ):
Maximize xt
Subject to xv xu w( u, v ) for each edge ( u, v) E
xs 0 and xv 0 for all v V {s}.
57
An alternative formulation
For each edge (i, j ) E , introduce a variable xij .
Minimize
( i , j )E
xij wij
//wij is the weight of (i, j )//
Subject to
1
j xij j x ji 1
0
and xij 0 for all (i,
if i s
if i t
for all i V ;
otherwise
j ) E.
Theorem: Every basic optimal solution (when exists) has
all variables equal to 0 or 1, and the set of edges whose
variables equal 1 form an s-t directed path.
58
Maximum Flows
G (V , E ), capacity function c, source s, sink t.
A flow is a real-valued function f : V V satisfying
Capacity constraint: u, v V , f ( u, v ) c( u, v ).
Skew symmetry: u, v V , f ( u, v ) f ( v, u ).
Flow conservation: u V {s, t},
f ( u,V )
f ( u, v ) 0
vV
The value of a flow is f f ( s, v ) f ( s,V ).
vV
The maxflow problem is to find a maximum flow.
59
References
• T.S. Ferguson, “Linear Programming: A
Concise Introduction,”
http://www.math.ucla.edu/~tom/LP.pdf .
• Lecture notes:
http://www.math.cuhk.edu.hk/~wei/LP11.html
60
© Copyright 2026 Paperzz