The Simplex Algorithm
An approach to optimization
problems for linear real
arithmetic constraints
Sebastian Bitzer ([email protected])
Seminar Constraint Logic Programming
University of Osnabrueck
November 11th 2002
Content
•
•
•
•
Optimization – what is that?
The Simplex Algorithm – background
Simplex form
Basic feasible solved form / basic feasible
solution
• The algorithm
• Initial basic feasible solved form
11 November, 2002
The Simplex Algorithm
2
Content
•
•
•
•
Optimization – what is that?
The Simplex Algorithm – background
Simplex form
Basic feasible solved form / basic feasible
solution
• The algorithm
• Initial basic feasible solved form
11 November, 2002
The Simplex Algorithm
3
Optimization – what is that?
• In general
• Optimization problem (C,f) with constraint
C and objective function f
e.g.
C : X  Y  4
11 November, 2002
The Simplex Algorithm
4
Optimization – what is that?
Objective function f:
– expression over variables V in constraint C
– evaluates to a real number
– e.g.
f : X ²  Y ²
11 November, 2002
The Simplex Algorithm
5
Optimization – what is that?
a valuation θ (substituting variables by values):
   X 1  v1 , X 2  v2 ,  , X n  vn 
X 1n variables
v1n values
solution of objective function using θ:
f   : f v1 , v2 ,  , vn 
11 November, 2002
The Simplex Algorithm
6
Optimization – what is that?
preferred valuations:
– valuation θ is preferred to valuation θ',
if f(θ) < f(θ')
optimal solution:
– θ is optimal,
if f(θ) < f(θ') for all solutions θ' θ
(there is no solution that is preferred to θ)
11 November, 2002
The Simplex Algorithm
7
Optimization – what is that?
Do all problems have an optimal solution?
X  7  X  49
X  77 with f ( X )  X
11 November, 2002
The Simplex Algorithm
8
Optimization Example
An optimization problem
(C  X  Y  4, f  X 2  Y 2 )
Find the closest point to the
origin satisfying the C.
Some solutions and f value
{ X  0, Y  4}
16
{ X  3, Y  3}
18
{ X  2, Y  2}
8
Optimal solution
{ X  2, Y  2}
Content
•
•
•
•
Optimization – what is that?
The Simplex Algorithm – background
Simplex form
Basic feasible solved form / basic feasible
solution
• The algorithm
• Initial basic feasible solved form
11 November, 2002
The Simplex Algorithm
10
Background
• George Dantzig
• born 8.11.1914,
Portland
• invented "Simplex
Method of
Optimisation" in 1947
• this grew out of his
work with the USAF
11 November, 2002
The Simplex Algorithm
11
Background
• originates from planning tasks:
– plans or schedules for training
– logistical supply
– deployment of men
• has in practice usually polynomial cost
11 November, 2002
The Simplex Algorithm
12
Quotes
Eugene Lawler (1980):
[Linear programming] is used to allocate
resources, plan production, schedule workers,
plan investment portfolios and formulate
marketing (and military) strategies. The
versatility and economic impact of linear
programming in today's industrial world is
truly awesome.
11 November, 2002
The Simplex Algorithm
13
Quotes
Dantzig I:
The tremendous power of the simplex method is a
constant surprise to me.
Dantzig II:
... it is interesting to note that the original problem that
started my research is still outstanding - namely the
problem of planning or scheduling dynamically over
time, particularly planning dynamically under
uncertainty. If such a problem could be successfully
solved it could eventually through better planning
contribute to the well- being and stability of the world.
11 November, 2002
The Simplex Algorithm
14
Content
•
•
•
•
Optimization – what is that?
The Simplex Algorithm – background
Simplex form
Basic feasible solved form / basic feasible
solution
• The algorithm
• Initial basic feasible solved form
11 November, 2002
The Simplex Algorithm
15
The example
Y
minimize X  Y subject to
Y  0
4
X  1
3
X  3
2Y  X  3
-2
-1
0
preferred
solutions
1
2
2
1
X
0
1
2
3
4
An optimization problem
showing contours of the
objective function
Simplex form
• (C,f) is in simplex form,
if C has the form CE  CI
• CE is a conjunction of linear arithmetic
equations
• CI is a term that constrains all variables in C
to be 0
11 November, 2002
The Simplex Algorithm
17
Simplex form
allowed conversions to get simplex form:
– X not constrained to be non-negative:
X  X   X  with X   0 and X   0
– inequality e  r (e=expression and r=number)
e  r  e  S  r with S  0
11 November, 2002
The Simplex Algorithm
18
The example
Y
minimize X  Y subject to
Y  0
4
X  1
3
-2
X  3
-1
0
preferred
solutions
1
2
2Y  X  3
2
1
X
An equivalent simplex form is:
 S2
X
 S3
X
X
 1
 2Y
 S1
 3
 3
X  0  Y  0  S1  0  S 2  0  S3  0
0
1
2
3
4
An optimization problem
showing contours of the
objective function
Content
•
•
•
•
Optimization – what is that?
The Simplex Algorithm – background
Simplex form
Basic feasible solved form / basic feasible
solution
• The algorithm
• Initial basic feasible solved form
11 November, 2002
The Simplex Algorithm
20
Basic feasible solved form
feasible = practicable, able to be carried out
(durchführbar, anwendbar)
• a simplex form optimization problem is in
basic feasible solved form,
if all equations in CE (of the simplex form)
have the form:
X 0  b  a1 X 1    an X n
11 November, 2002
The Simplex Algorithm
21
Basic feasible solved form
X 0  b  a1 X 1    an X n
• X0 is called basic variable, does not occur
anywhere else (neither in objective function)
• X1...n are parameters
• b, a1...n are constants
• b0
11 November, 2002
The Simplex Algorithm
22
Basic feasible solution
X 0  b  a1 X 1    an X n
• corresponding basic feasible solution to a
basic feasible solved form:
– setting each X1...n = 0
 X0  b
11 November, 2002
The Simplex Algorithm
23
The example
Basic
feasible
minimize
X  Ysolved
subjectform:
to
minimize 0  0Y.5S100.5S3 subject to
Y
3  X0.5S11   0.5S3
S2  2
X

X  3   S3

3 2Y  X  3  S3

Y
4
-2
-1
0
preferred
solutions
3
1
2
2
1
X
An equivalent simplex form is:
 S2
X
 S3
X
X
 1
 2Y
 S1
 3
 3
X  0  Y  0  S1  0  S 2  0  S3  0
0
1
2
3
4
An optimization problem
showing contours of the
objective function
Content
•
•
•
•
Optimization – what is that?
The Simplex Algorithm – background
Simplex form
Basic feasible solved form / basic feasible
solution
• The algorithm
• Initial basic feasible solved form
11 November, 2002
The Simplex Algorithm
25
The algorithm
• idea: optimal solution has to be in one of the
vertices
• so: go from one vertex to the preferred next
vertex
• end: if there is no preferred vertex, the
actual has to be the optimal solution
11 November, 2002
The Simplex Algorithm
26
The algorithm
in other words:
– take a basic feasible solved form
– look for an "adjacent" basic feasible solved
form whose basic feasible solution decreases
the value of the objective function
– if there is no such adjacent basic feasible solved
form, then the optimum has been found
11 November, 2002
The Simplex Algorithm
27
The algorithm
• adjacent  just one single pivot
• pivoting  move one variable out of basic
variables ( exit variable) and
another in ( entry variable)
X  49  7Y  21Z
Y  7  3Z  X
1
7
11 November, 2002
The Simplex Algorithm
28
The algorithm
m
f  e   d jY j
Problem:
Which variables should be
exiting resp. entering?
Entering Variable:
j 1
m




X

b

a
Y


i
i
ij
j



i 1 
j 1

n
m
  X i  0   Y j  0
n
i 1
j 1
– choose one YJ with dJ<0
 pivoting on this YJ can only decrease f (see next slide)
– no such YJ  optimum has been found
11 November, 2002
The Simplex Algorithm
29
Why pivoting on a Yj with dj<0
decreases objective function f
f  e  d1Y1    d J YJ
• looking at the basic feasible solution (bfs) every
parametric variable (Yj) is set to 0
• pivoting on such a variable (var. becomes basic)
leads to an increase of this variable in the bfs: Yj≥0
 a Yj with negative dj decreases f
11 November, 2002
The Simplex Algorithm
30
The algorithm
Exiting variable:
– we have to maintain basic feasible solved form
 all bi's have to be 0
 choose a Xi so that –bI/aIJ is a minimum of:
  bi

M 
aiJ  0 and 1  i  n
 aiJ

– M={Ø}  optimization problem unbounded
11 November, 2002
The Simplex Algorithm
31
Simplex Example
minimize 10  Y  Z subject to
X 3
T
Y
4  2Y

 2Z

X  0Y  0 Z  0T  0
Choose variable Y, the first
eqn is only one with neg.
coeff Y  3  X
minimize 7  X  Z subject to
Y
3
X
T  10  2 X

 2Z

X  0Y  0 Z  0T  0
Choose variable Z, the 2nd
eqn is only one with neg.
.T
coeff Z  5  X  05
minimize 2  2 X  0.5T subject to
Y
3 X
Z 5 X

 0.5T

X  0Y  0 Z  0T  0
No variable can be chosen,
optimal value 2 is found
The algorithm
starting from a problem in bfs form
repeat
Choose a variable y with negative coefficient in the obj. func.
Find the equation x = b + cy + ... where c<0 and -b/c is minimal
Rewrite this equation with y the subject y = -b/c + 1/c x + ...
Substitute -b/c + 1/c x + ... for y in all other eqns and obj. func.
until no such variable y exists or no such equation exists
if no such y exists optimum is found
else there is no optimal solution
11 November, 2002
The Simplex Algorithm
33
The example
Y
Basic feasible solution form: circle
minimize 0  0.5S1  0.5S3 subject to
Y  3  0.5S1  0.5S3 
S2  2
 S3

X 3
 S3

4
-2
-1
0
3
1
2
2
1
Choose S3, replace using 2nd eq
minimize  1  0.5S1  0.5S 2 subject to
Y
2  0.5S1
 0.5S 2

S3  2
 S2

X 1
 S2

X
0
1
2
3
4
Optimal solution: box
Content
•
•
•
•
Optimization – what is that?
The Simplex Algorithm – background
Simplex form
Basic feasible solved form / basic feasible
solution
• The algorithm
• Initial basic feasible solved form
11 November, 2002
The Simplex Algorithm
35
Initial basic feasible solved form
idea:
– solve a different optimization problem
this optimization problem should have an
initial basic feasible solved form, which:
– can be found trivially
– has an optimal solution that leads to an initial
basic feasible solved form of the original
problem
11 November, 2002
The Simplex Algorithm
36
Initial basic feasible solved form
 m

f  e   d j x j :    aij x j  bi  
i 1
j 1
 j 1

m
n
 x j  0
m
j 1
add artificial variables and minimize on them:
m


f   zi :   zi  bi   aij x j  
i 1
i 1
j 1


n
n
 x j  0 
m
j 1
11 November, 2002
  zi  0 
n
i 1
The Simplex Algorithm
37
Initial basic feasible solved form
to get basic feasible solved form:
m


f   zi    bi   aij x j 
i 1
i 1 
j 1

n
n
 solve this problem
11 November, 2002
The Simplex Algorithm
38
Initial basic feasible solved form
possible outcomes:
– (f > 0)  original problem unsatisfiable
– (f = 0)  (zi...n parametric)  got a basic
feasible solved form for original problem
– (f = 0)  (zi...n parametric)  zi must occur in
such an equation:
n
m
i 1
j 1
z  0   d 'i zi   a' j x j
11 November, 2002
The Simplex Algorithm
39
Such an equation is no problem,
because
n
m
i 1
j 1
z  0   d 'i zi   a' j x j
• if all a’j = 0 → equation is redundant
• if one a’j ≠ 0 → use according xj for
pivoting z out of basic variables (this
maintains basic feasible solved form since
z = 0 +…)
 all z become parametric
11 November, 2002
The Simplex Algorithm
40
The example
Y
minimize X  Y subject to
Y  0
4
X  1
3
-2
X  3
-1
0
preferred
solutions
1
2
2Y  X  3
2
1
X
An equivalent simplex form is:
 S2
X
 S3
X
X
 1
 2Y
 S1
 3
 3
X  0  Y  0  S1  0  S 2  0  S3  0
0
1
2
3
4
An optimization problem
showing contours of the
objective function
The example
Original simplex form equations
 S2
X
 S3
X
X
 1
 2Y
 3
 S1
3
With artificial vars in bfs form:
A1  1  X
 S2
A2  3  X
A3  3  X

 S3
 2Y
 S1
Objective function: minimize
A1  A2  A3
 7  X  2Y  S1  S2  S3

The example
Problem after minimization of objective function
minimize A1  A2  A3 subject to
Y
 3  0.5S1
 0.5S3
S2
 2
 S3
X
 3
 S3
 0.5 A2
 A1
 A2
 0.5 A3


 A2
Removing the artificial variables, the original problem
Y
3  0.5S1
 0.5S3

S2  2
 S3

X
 S3

3
Simplex solver
finding a basic feasible solution is exactly a
constraint satisfaction problem
 efficient constraint solver for linear
inequalities
11 November, 2002
The Simplex Algorithm
44
Cycling
Problem:
– if for one of the basic variables is valid:
Xi = 0 + … , a pivot could be performed which
does not change the corresponding basic
feasible solution
 danger of pivoting back
Solution:
– use e.g. Bland’s anti-cycling rule (always select
candidate with smallest index: x2 instead of x4)
11 November, 2002
The Simplex Algorithm
45
Summary
We have seen that optimisations of linear real
arithmetic constraints play an important role
in many applications.
The Simplex Method which was introduced
here provides a very efficient algorithm to
determine whether there exists an optimal
solution to linear real arithmetic constraints
and if there exists one, to compute it.
11 November, 2002
The Simplex Algorithm
46
Literature
• books:
George B. Dantzig, Mukund N. Thapa
"Linear Programming I: Introduction"
Springer Verlag
Kim Marriott & Peter J. Stuckey
"Programming with Constraints: An Introduction"
MIT Press
11 November, 2002
The Simplex Algorithm
47
Literature
• examples are taken from a presentation of
Marriott & Stuckey and could be accessed
via internet:
http://www.cs.mu.oz.au/~pjs/book/course.html
• this presentation in the net:
http://www-lehre.inf.uos.de/~sbitzer/clp
11 November, 2002
The Simplex Algorithm
48