University of Massachusetts at Amherst
665 Computer Algorithm
Mathematical Programming
Professor Maciej Ciesielski
Spring 2004
(DFG)
OUTLINE
1. Non Linear Programming
2. Quadratic Programming
3. Linear Programming
A. Geometric Solutions
B. Basic Solutions
C. Simplex Method
D. LP modifications
4. Integer Programming
A. Unimodularity
B. Network Flow Problems
5. Conclusions
Mathematical Programming
Nonlinear Programming
Minimize f(x)
subject to:
{
g(x) = 0
h(x) ≥ 0
where x = [x1,...xn]T, and all functions f, g, h, are
differentiable.
Convexity
To obtain global optimum, it is important that the
constraint set
{x | g(x) = 0,h(x) ≥ 0}
and the function f(x) be convex.
If the function is convex over a convex constraint
set, then local minimun is also a globa1 minimum
Set convexity
Convex set
non-convex set
Function convexity
Convex function
non-convex function
Classical optimization deals with unconstrained
optimization problems.
Solution: move the constraints to the function and solve
the unconstrained optimization problem
For the minimization problem
Min f(x) subject to g(x) = 0
1. Penalty function method: replace f(x) by
F(x) = f(x)+ M( g(x))2, M >> 0
2. Lagrangian multipliers: replace f(x) by
L(x,λ) = f(x) + Σ λi gi(x)
i
At the optimum point, x*, L(x*, λ) = f(x*)
Quadratic Programming (QP)
Minimize xTQx
Subject to:
{Ax ≥ b
Where Q is an n x n symmetric matrix, A is a
constraint matrix, and b is a constraint vector
Matrix Q is positive definite, i.e.,
xTQx > 0
For every x  0. Thus f is convex.
Linear Programming (LP)
Minimize cT
Subject to
{
Ax ≥ b
x≥0
Where cT is a coefficient vector for f, A is a constraint
matrix, and b es a constraint vector.
The constraint set
{x | Ax ≥ b}
Is a convex polyhedron, and f is linear, convex.
Linear Programming (LP)
Optimization problem:
• 2 types of products are made by the factory
• 3 machines are needed to make each
product
Constraints:
Machines are available for a
limited time during one production session:
• time on A ≤ 50 hours
• time on B ≤ 35 hours
• time on C ≤ 80 hours
To complete the product, each lot must be processed
by all three machines for a certain number of hours
Required manufacturing time
1
Machines
A
B
C
10
5
5
5
2
Time Constraint:
50
5
15
35
80
Profit
• Product 1 can be sold for $100 per lot
• Product 2 can be sold for $80 per lot
Optimization Problem:
Design the production schedule which
rnaximizes the profit of both products
manufactured during one production session.
Formulation
x1 = number of lots of product 1
x2 = number of lots of product 2
max f (x) = 100x1 + 80x2
{
10x1 + 5x2 ≤ 50
5x1 + 5x2 ≤ 35
5x1 + 15x2 ≤ 80
Convert the problem to a standard Linear
Program (LP):
Max f(x) = 100x1 + 8Ox2
s. to
{
10x1 + 5x2+ s1
5x1 + 5x2+ s2
5x1 + 15x2+ s3
Basis
New Matrix
A
= 50
= 35
= 80
x2
15
Geometric solution
Family of curves, constant for x1, x2,
F(x) = 100 x1, + 80 x2 = 800 (const)
10
F(x) = 800
(a) 10x1, +
5x2  50
(b) 5x1, +
5x2  35
(c) 5x1, + 15x2  80
5
4
P0
P2
3
P1
5
a
b
10
15
c
x1
Basic Solution
Given a system of equalities
Ax=b
where x = n - vector, b = m - vector, A = m x n matrix,
select a set of m linearly independent columns such that
(m x m) matrix B is nonsingular, i.e. /B /  0. Then one
may uniquely solve the equation
Bxb= b
where Xb is an m - element subvector of x.
By putting x=(xb,0) we obtain a solution to
Ax = b
Such a solution (with n - m components of x not
associated with columns of B m x m ) to the resulting set of
equations is said to be a basic solution with regard to the
basis B.
The Xb variables, associated with columns of B, are called
basic variables.
Basic Feasible Solution
If a feasible solution (i.e. the one which satisfies all
contraints) is also basic, it is called a basic feasible
solution.
Fundamental Theorem of Linear Programming
Given an L.P. in standard form
min cTx
s. to
Ax = b
x0
where A is m x n matrix of rank m (i.e. m < n and
the m rows are linear independent),
 If there is a feasible solution, there is a basic
feasible solution
 If there is an optimal feasible solution, there
is an optimal basic feasible solution
This theorem shows that it is necessary only to
consider basic feasible solutions when seeking an
optimal solution to a linear program, as the optimal
value is always achieved as such a solution
Ultimate goal:
find a basic feasible solution with a base B
composed of original variables only, and which is
optimum.
ƒ (x) = 100 x1, + 80 x2 + 0 s1 + 0 s2 + 0 s3
x1
10
5
100
5
5
010
s1
001
s2,
5
15
50
x2
=
s3
35
80
basis
Solution: x1 = 0, x2 = 0, s1 = 50, s2 = 35, s3 = 80, is
a basic feasible solution (point P0 (0,0))
Function value: F(0, 0. 50, 35, 80) = 0
Simplex Method
Proceed from one basic feasible solution (extreme
point) to another, in such a way as to continually
decrease the value of the f (x) until a minimum is
reached.
General comments
It is easy to find initial basic feasible solutions with
slack variables.
Finding initial basic solution is part of the Simplex
method (Lue 84).
Simplex Algorithm
1. Select the column, such that the new resulting
basic feasible solution will yield a lower value to
f(x) than the previous one.
2. Select the pivot element in that column.
By an elementary evaluation determine
• Which vector aj should enter the basis (so that
f(x) is reduced), and
• Which vector should leave the basis.
Construct a Simplex Tableau, [A, b ] appended by
a row at the top with cost coefficients and current
cost.
j=
1
2
3
4
5
0
rj
100
80
0
0
0
0
y1j
10
5
1
0
0
50
y2j
5
5
0
1
0
35
y3j
5
15
0
0
1
80
yi1
yi2
yi3
yi4
yi5
yi0
Point P0 (0,0)
F(0,0,50,35,80) = 0
Step 1. Select column q with rq > 0.
(xq enters the basis)
q : rq = max ri
Step 2. Select pivot element p in column q
Which minimizes ratio:
p:
yp0
ypq
= min
i
yi0
yiq
yi0
yiq
for positive yiq
Step 3. Pivot on element ypq and update the
tableau (update all rows, including the top
one)
• In row p divide ypj by ypq , j = 0, 1,…, n
• Subtract from each row i  p
yi q
yp q
•
ypj , j = 0, 1,…, n
Pivot on y1 1
j=
rj
y1j
y2j
y3j
1
2
3
4
5
0
100
80
0
0
0
0
10
5
1
0
0
50
5
5
0
1
0
35
5
15
0
0
1
80
yi1
yi2
yi3
yi4
yi5
yi0
Point P0 (0,0)
F(0,0,50,35,80) = 0
Update the table
j=
rj
y1j
y2j
y3j
1
2
3
4
5
0
0
30
-10
0
0
-500
1
0.5
0.1
0
0
5
0
2.5
-0.5
1
0
10
0
12.5
-0.5
0
1
55
yi1
yi2
yi3
yi4
yi5
yi0
Point P1 (5,0)
F(5,0,0,10,55) = 500
Pivot on y2 2
j=
rj
y1j
y2j
y3j
1
2
3
4
5
0
0
30
-10
0
0
-500
1
0.5
1
0
0
5
0
2.5
-0.5
1
0
10
0
12.5
-0.5
0
1
55
yi1
yi2
yi3
yi4
yi5
yi0
Point P1 (5,0)
F(5,0,0,10,55) = 500
Update the table
j=
rj
y1j
y2j
y3j
1
2
3
4
5
0
0
0
-4
-12
0
-620
1
0
0.2
-0.2
0
3
0
1
-0.2
0.4
0
4
0
0
2
-5
1
5
yi1
yi2
yi3
yi4
yi5
yi0
Point P1 (3,4)
F(3,4,0,0,5) = 620
LP Modifications
1. Maximization
max c t x = min (-c t x)
2. Unconstrained x
let x = x+ - x - ; x+, x -  0)
min (c t x+ - c t x -)
[A, -A]
s. to
x+
x-
x+, x -  0
= b
3. Constraint set:
Ax  b
Slack variables, xs : Ax + xs = b
min
[cT
, 0]
x
xs
[A, I ]
x
xs
= b
x , xs  0
Columns corresponding to xs form a basic solution
4. Constraint set:
Ax  b
Surplus variables, xp : Ax - xp = b
min
[cT
, 0]
x
xp
[A, -I ]
x
xp
x , xp  0
= b
Artificial variables, xa : Ax - xp + xa = b
(cannot star with negative basis of xp )
min [cT , 0 , k I ]
[A, -I , I ]
S. to
x
xp
xa
x
xp
xa
=
b
x , xp , xa  0
Columns corresponding to xa form a basic solution
Integer Programming
Integer Program
max cTx
s. to
Ax  b
xi integer
Typically, xi = 0,1
(0,1 Integer Programming)
IP Example: matching problem
Given a graph G , find maximal set of edges in G, such
that no two edges are adjacent to the same vertex.
• Maximum matching:
cardinality.
matching
• Weighted matching: matching with
max  w (ei )
2
e1
1
e2
e3
3
of
maximum
1 if ei is included in matching
xi  
0 otherwise
max x1  x2  x3
1
 x1  x2
x 
 x3  1
 1
s. to  
x2  x3  1

 xi  0, 1
2
e1
1
e2
e3
3
In matrix form:
T
max c x
 Ax  b
s. to  
 xi  0, 1
Where b = [ 1,…1]T ,
A = incidence matrix of G
1
2
3
e1 1 1 0


e.g. A  e 2 1 0 1
e3 0 1 1
max 1x1 + 1x2 + 1x3
s.to
1 1 0  x1 
1
1 0 1  x    1 

  2
 
0 1 1  x3 
 1 
x1 , x2 , x3 = 0, 1
One solution to this IP problem:
x1 = 1 , x2 = 0 , x3 = 0
Other possible solutions:
x1 = 0 , x2 = 1 , x3 = 0, or x1 = 0 , x2 = 0 , x3 = 1
Question
Can the solution to this IP problem be obtained by
dropping the integrality constraint:
xi = 0, 1
And solving the LP problem instead?
In our example, solution to the IP is not obtainable
from LP.
Reason: matrix A does not have certain property
(total unimodularity) needed to guarantee integer
solutions.
If xi = 0, 1 is relaxed, such that xi  0 , then the
solution to the associated LP problem is non –
integer:
1
x1  x2  x3 
2
1 1 0
Reason: the A matrix 1 0 1


0 1 1
Is not totally unimodular: |A| = - 2
Unimodularity: LP -> IP
Given a constraint set in standard form
 Ax  b

 xi  0
where A, b are integer
Partition A = [B/N ]; x = [xB, xN ]
B is nonsingular m x m basis, N is non-basic
 xB 
Ax  B / N     BxB  NxN  b
 xN 
Basic solution is
 xN  0

1
x

B
b
 B
In particular, when B = I
and B -1 = I
then
XB = b
a solution can be obtained by inspection (as in the
initial step of Simplex method).
Since xB = B –1 b with xN = 0, b integer
A sufficient condition for a basic solution xB to be
integer is that
B –1 be an integer matrix
Unimodularity
A square matrix B is called unimodular if
D = |det B| = 1
An integer matrix A is totally unimodular if every
square, nonsingular submatrix of A is unimodular.
Equivalently:
A is totally unimodular if every
subdeterminant of A is 0, +1, or –1.
Recall that for B nonsingular:
Where B
+

B
B 1 
det B
, adjoint matrix
= [ i j cofactor of element a i j in det A ]T
B + and det B are integer if B –1 is integer
b11 b12
B  b21 b22
b31 b32
B
b13 
b23 
b33 
(b21 b33  b23 b31 )
(b21 b32  b22 b31 ) 
 (b22 b33  b23 b32 )
  (b12 b33  b32 b13 ) (b11 b33  b13 b31 )  (b11 b32  b12 b31 )
 (b12 b23  b13 b22 )  (b11 b23  b21 b13 ) (b11 b22  b12 b21 ) 
Cofactor of ai j : Determinant obtained by omitting
the ith row and the jth column of A and then
multiplying by (-1) i + j .
T
For B unimodular, B
–1
integer:
If A is totally unimodular, every basis matrix B is
unimodular and every basic solution
( xB , yN) = ( B
–1
b, 0 )
Is integer.
In particular, the optimal solution is integer
Theorem 1:
If A is totally unimodular then every basic
solution of Ax = b is integer.
For LP’s with equality constraints total
unimodularity is sufficient but not necessary.
For LP with inequality constraints,
Ax  b,
total unimodularity of A is both necessary
and sufficient for all extreme points of
s = {x : Ax  b , x  0 } to be integer for every
integer vector b.
Conclusion:
Any IP with totally unimodular constraint matrix
can be solved as an LP.
Totally unimodular matrix : a i j = 0 , +1, -1
Also, every determinant of A must be 0, +1 or -1
Special Linear Programs with Integer Solutions
Network flow problems:
• max flow
• min – cost flow
• assignment problem
• shortest path
• transportation problem
are LP with the property that they possess
optimal solutions in integers.
Some Totally Unimodular Linear Programs
Assignment problem
(special case of min – cost capacitated flow
problems)
[ m jobs x m men ]
1 if man i is assigned to job j
xi j  
0 otherwise
c i j = cost of assigning man i to job j
Max  c ij x ij
ij
s. to.
m
 x ij  1, j  1 m (1 man per job)
i 1
m
 x ij  1, i  1 m (1 job per man)
j 1
xij  0, 1
X11 X12 X13 X21 X22 X23 X31 X32 X33
1

 1

1

1 1 1




 1

1
1
1 
 
1
1
1
 X    

1

1
1 1 1

 
1 1 1
 1 
1
1
In matrix notation:
Ax = 1
T
max c x
where
m
1

 

1
A
1  1

0


2m
...
m
1


 
1



0

1  1
1

1
0
1  1
0
Exactly m 1’s in each row
Exactly 2 1’s in each column

2
Network Flow Problems:
Given:
• a set of origins V1 ; each origin i  V1 ; supplies a1 of
commodity.
• a set of destinations V2 ; each destination j  V2 ; has a
demand bi of commodity.
• cost per unit commodity; cij associated with sending
commodity through (i, j ).
Constraint Set: (Totally unimodular)

xij

(i , j )  Γ  (i )
0  xi j  ci j

x ji
( j , i )  Γ  (i )
 ai for i V1 ( Γ  (i )   )

 0 for i V1
  b for i V ( Γ  (i )   )
i
2

Special case: for single
destination, max flow problem

xij

(i , j )  Γ  (i )
in any case:

x ji
( j , i )  Γ  (i )
source,
single
 f for source node

0 for int ernal nodes
 f for destinatio n node

Ax  b
min  cost flow
No matter which problem it is 
max flow
min cut

the general format is A’x  b
It can be shown that
• If A is totally unimodular then [A/I ] is also totally
unimodular
• The transpose of totally unimodular matrix is also
totally unimodular
 A
• A'
. 
I 
 
where A = incidence matrix of the corresponding
digraph, totally unimodular. I = identity matrix, A’ is
totally unimodular.
No efficient algorithms exist for general Integer
Programming problems
• Exploit a special structure of the problem
(total unimodularity, etc.) to obtain integer
solutions by solving simpler problems.
• Transform the problem to another problem for
which an approximate solution is easier to
find.
• Once the structure of the problem is well
understood, use heuristic, but… stay away
from brute force approach.
Conclusions
A large number of CAD problems can be cast in
analytical form:
• Graph Theory
• Mathematical Optimization
For some problems – efficient algorithms exist.
For others – need to resort to heuristic,
suboptimal solutions.
Some known successful heuristic approaches
• Simulated annealing
• LP rounding