Optimization – Class III
Simplex method

Example:
Max z  300x1  250x2
Subject to:
Max z  300x1  250x2  0s1  0s2  0s3
2x1  x2  40
Subject to:
x1  3x2  45
x1  3x2  s2  45
x1  12
x1  s3  12
x1, x2  0


2x1  x2  s1  40
x1, x2 , s1, s2 , s3  0
Definitions:
o With the exception of the variables that appear on the left (i.e., the
variables that we have been referring to as the dependent variables) are
called basic variables. Those on the right (i.e., the independent variables)
are called nonbasic variables. The solutions we have obtained by setting
the nonbasic variables to zero are called basic feasible solutions.
General task
n
o Maximize :
   c jx j
j=1
n
a x
o subject to

j 1
ij
j
 bi ; i  1..m
x j  0; j  1..n
Definition
n
o Slack Variables:

wi  bi   aij x j ; i  1..m
j 1
Other notation
( x1 , . . . , xn , w1 , . . . , wm )  ( x1 , . . . , xn , xn1 , . . . , xnm ).
n
   c jx j
j=1
n
xni  bi   aij x j ; i  1..m
j 1
xi  0; i  1..n  m

Each dictionary has m basic variables and n nonbasic variables. Let B denote the
collection of indices from {1, 2, . . . , n  m} corresponding to the basic
variables, and let N denote the indices corresponding to the nonbasic variables.
Initially, we have N  {1, 2, . . . , n} and B  {n  1, n  2, . . . , n  m} .

In the following steps, we have:
     c jx j  
jN
xi  bi   aij x j ; i  B
jN

When pivoting, we must enlarge

thus choose the k with ck maximal and
positive, and ensure that all x stay positive, thus choose a i so that
i  B, bi  aik xk  0
In other words:

xk  min  bi / aik  or xk  max aik i / bi
iB , aik  0


iB

1
Identify the Incoming and Outgoing Variables
Incoming Variable -An incoming variable is currently a nonbasic variable (the
current value is zero) and will be changed to a basic variable (introduced into the
solution).
o -For the maximization problems, the incoming variable is the variable
with the largest positive value(coefficient) in the c j  z j row.
o - For the minimization problems, the incoming variable is the variable
with the largest negative value in the c j  z j row.

Outgoing Variable -An outgoing variable is currently a basic variable that is first
reduced to zero when increasing the value of the incoming variable and will be
changed to a nonbasic variable (removed from the solution).
o To determine the outgoing variable, compute the ratio of the Quantity to
the coefficient of the incoming variable for each basis row.

o For both the maximization and minimization problems, the outgoing
variable is the basic variable with the smallest ratio.
o The coefficient of the incoming variable in the outgoing row is called the
pivot element.
Summary: The Simplex Procedure
o Step 1: Standardize the problem.
o Step 2: Generate an Initial Solution.
o Step 3: Test for Optimality. If the solution is optimal, go to Step 6.
Otherwise, go to Step 4.
o Step 4: Identify the Incoming and Outgoing Variables.
o Step 5: Generate an Improved Solution. Go to Step 3.
o Step 6: Check for other Optimal Solutions.
Initialization

If b is negative:
( x1 , . . . , xn , w1 , . . . , wm )  ( x1 , . . . , xn , xn 1 , . . . , xn  m ).
n
   c j x j     x0
j=1
n
xn i  bi   aij x j  x0 ; i  1..m
j 1
xi  0; i  0..n  m

Example:
max imize - 2 x1 - x2
subject to - x1  x 2  -1
- x1 - 2 x2  - 2
x2  1
x1 , x2  0.
Unboudness

Example:
  5  x3 - x1
x2  5  2 x3 - 3 x1
x4  7 - 4 x1
x5  x1.

If xk  max  aik i / bi  is non-positive – the problem diverges.

THEOREM 3.4. For an arbitrary linear program in standard form, the following
statements are true:
o (1) If there is no optimal solution, then the problem is either infeasible or
unbounded.
o (2) If a feasible solution exists, then a basic feasible solution exists (nonebase values are zero).
o (3) If an optimal solution exists, then a basic optimal solution exists.
iB
Dual problem

Example
  4 x1  x2  3x3
(a) x1  4 x2  1
(b)3x1 - x2  x3  3
( x1 , x2 , x3 )  (1, 0, 0), ( x1 , x2 , x3 )  (0, 0, 3)
  2* a  3* b  11

Example continuation
  4 x1  x2  3 x3
y1 ( x1  4 x2 )  1 y1
y2  3 x1 - x2  x3   3 y2
(y1 + 3y2 )x1 + (4y1 - y2 )x2 + (y2 )x3  y1 + 3y2 .
  4 x1  x2  3x3
y1 + 3y2  4
4y1 -y2  1
y2  3
min imize    y1 + 3y2 .
n
o Maximize :
   c jx j
j=1
n
a x
o subject to

j 1
ij
j
 bi ; i  1..m
x j  0; j  1..n
Becomes:
m
o Minimize :
   bi yi
i 1
m
ya
o subject to
i 1
i ij
 c j ; i  1..n
yi  0; j  1..m
m
o
maximize    bi yi
i 1
m
 y (a )  (c ); i  1..n
o subject to
i 1
i
ij
yi  0; j  1..m
j

Weak Duality Theorem: If (x1, x2, . . . , xn) is feasible for the primal and (y1,
y2, . . . , ym) is feasible for the dual, then
n
m
c x  b y
j
j
j=1

i 1
i
i
Proof
m
n
m
 m

c j x j     yi aij  x j   yi   aij x j    bi yi

j=1
j=1  i 1
i 1
j=1
i 1

n
n

Strong Duality Theorem: If the primal problem has an optimal solution, x_ =
(x_1 , x_2 , . . . , x_n), then the dual also has an optimal solution, y_ = (y_1, y_2, .
. . , y_m ), such that:
n
m
c x *  b y *
j 1


j
j
i 1
i i
Note that the numbers in the dual dictionary are simply the negative of the
numbers in the primal dictionary arranged with the rows and columns
interchanged.
Include function into array form (just as the first line).


That is the dual dictionary is the negative transpose of the primal dictionary
This property is preserved from one dictionary to the next. Let’s observe the
effect of one pivot. To keep notations uncluttered, we shall consider only four
generic entries in a table of coefficients: the pivot element, which we shall denote
by a, one other element on the pivot element’s row, call it b, one other in its
column, call it c, and a fourth element, denoted d, chosen to make these four
entries into a rectangle.
o The pivot element gets replaced by its reciprocal;
o Elements in the pivot row get negated and divided by the pivot element;
o Elements in the pivot column get divided by the pivot element; and
o All other elements, such as d, get decreased by bc/a.
In the dual



Thus feasibility of the dual is optimality of the primal
Duality theory is often useful in that it provides a certificate of optimality.
The strong duality theorem tells us that, whenever the primal problem has an
optimal solution, the dual problem has one also and there is no duality gap. But
what if the primal problem does not have an optimal solution? For example,

suppose that it is unbounded. The unboundedness of the primal together with the
weak duality theorem tells us immediately that the dual problem must be
infeasible. Similarly, if the dual problem is unbounded, then the primal problem
must be infeasible. It is natural to hope that these three cases are the only
possibilities, because if they were we could then think of the strong duality
theorem holding globally. That is, even if, say, the primal is unbounded, the fact
that then the dual is infeasible is like saying that the primal and dual have a zero
duality gap sitting out at +infinity. Similarly, an infeasible primal together with an
unbounded dual could be viewed as a pair in which the gap is zero and sits at
−infinity. But it turns out that there is a fourth possibility that sometimes occurs—
it can happen that both the primal and the dual problems are infeasible. For
example, consider the following problem
Example
max imize 2 x1 - x2
subject to :
x1  x 2  1
- x1  x2  - 2
x1 , x2  0.

Complementary Slackness Theorem : Suppose that x = (x1, x2, . . . , xn) is
primal feasible and that y = (y1, y2, . . . , ym) is dual feasible. Let (w1,w2, . . .
,wm) denote the corresponding primal slack variables, and let (z1, z2, . . . , zn)
denote the corresponding dual slack variables. Then x and y are optimal for their
respective problems if and only if:
x j z j  0, for j  1, 2, . . . , n,
wi yi  0, for i  1, 2, . . . , m.

Proof
m
n
m
 m

c j x j     yi aij  x j   yi   aij x j    bi yi

j=1
j=1  i 1
i 1
j=1
i 1

n
n
m
 x j  0,  yi aij  c j
i 1
m
ya
i 1
i ij
m
 c j ; z j   yi aij  c j  0  x j z j  0
i 1
The Dual Simplex Method


One can actually apply the simplex method to the dual problem without ever
writing down the dual problem or its dictionaries. Instead, the so-called dual
simplex method is seen simply as a new way of picking the entering and leaving
variables in a sequence of primal dictionaries.
Example:
  - x1  x2
x3  4  2 x1  x2
x4   8  2 x1  4x2
x5  -7+x1  3x2
Solve the dual and perform the changes on the primal too and get an optimal
solution
First step:
Second step


If the dual is feasible. This means that all the coefficients of the nonbasic
variables in the primal objective function must be nonpositive, we proceed as
follows.
o First we select the leaving variable by picking that basic variable whose
constant term in the dictionary is the most negative (if there are none, then
the current dictionary is optimal).
o Then we pick the entering variable by scanning across this row of the
dictionary and comparing ratios of the coefficients in this row to the
corresponding coefficients in the objective row, looking for the largest
negated ratio just as we did in the primal simplex method.
o Once the entering and leaving variable are identified, we pivot to the next
dictionary and continue from there.
If the dual is non-feasible, it is enough to change the score function.
Equalities in the constraints
maximize cT x subject to Ax = b, x  0
 maximize cT x subject to Ax  b, Ax  b,x  0
 maximize cT x subject to -Ax  b, Ax  b,x  0

Lets associate y+ to the first set of constraints and y- to the second set of
constraints.
minimize bT y+ -bT y- subject to AT y+ - AT y- = c, y  0
y  y+ -y-  minimize bT y subject to AT y= c
In other words, equations lead to unconstrained values in the dual.
Matrix notation
maximize cT x subject to Ax = b, x  0
 a11

A m*(n+m) = 


 am1
a1n
amn
 x1   x1 
 c1 

  
 

  
 

  
 
1
 b1 


  
 



x
x
1
 , b=   , c   cn  , x   n    n 



0
 
1 
xn 1
w1 







 
1

  
 
bm 

  
 

  
 
 0 
 xn  m   wm 
 x x 
A  [ N m*n , Bm*m ], x      N n   AX  N m*n * xN  Bm*m * xB
 w  xB m 
c 
c   N n   cT X  cN n * xN  cB m * xB  cB m * xB
cB m 
AX  b  N * xN  B * xB  b  xB  B1 b  N * xN 
  cT x  cN T * xN  cBT * xB  cN T * xN  cBT * B 1  b  N * xN 
 cBT * B 1b   cBT * B 1 N  cN T  * xN

 cBT * B 1b   B 1 N  cB  cN

T
 *x
T
N
In the optimal solution
x *N  0
 *  cBT * B 1b
x *B  B 1b

Dual problem
m
maximize -   bi yi  bT y
i 1
m
 y (  a )  ( c )   A
T
subject to
i 1
i
ij
j
y  z  c
 z   zN n 
y 
  AT Y   N m*nT * z  Bm*mT * y

 y   zB m 
T Y 
 AT Y  BT z  c    A -B     c
Z 

  cB * B b   B N  c B  c N
1
T
1
T
 *x
T
N
xB  B 1  b  N * xN 
cBT B -1b  
cN - ( B -1 N )T cB  [c j ]
B -1b  bi 
B -1 N  [aij ]
-  - cBT B -1b - ( B -1b)T zB
zN  ( B-1 N )T cB - cN  ( B -1 N )T zB .

In the optimal solution
zB *  0
-  - cBT B -1b
z N *  ( B -1 N )T cB - cN

Back to the primal problem we get:
   *  z *N T xN
xB  xB *  B 1 NxN
And the dual is
-  -  *- xBT zB
z N  z N *  ( B -1 N )T zB .

The (primal) simplex method can be described briefly as follows. The starting
assumptions are that we are given
o A partition of the n +m indices into a collection B of m basic indices and a
collection N of n nonbasic ones with the property that the basis matrix B is
invertible,
o An associated current primal solution
o An associated current dual solution
xB  0
zN  0
(and
(with
xN  0 )
zB  0 )

The simplex method then produces a sequence of steps to “adjacent” bases such
that the current value
*
of the objective function increases at each step (or, at
least, would increase if the step size were positive), updating

and
zN
along
the way.
Two bases are said to be adjacent to each other if they differ in only one index.
That is, given a basis B, an adjacent basis is determined by removing one basic
index and replacing it with a nonbasic index.
z*N  0

Check for Optimality. If

Select Entering Variable. Pick an index
x j is the entering variable.

xB
……
stop.
jN
for which
z j *  0 Variable
Problems in general form

Maximize CTx, subject to:
a  Ax  b; l  x  u
li  x  ui

Define slack variables:
w  Ax
a  w  b; l  x  u

If lower bound of everybody is feasible, simply apply simplex method and move
to upper boud, until ended
Dual of general form

Maximize CTx, subject to:
Ax  b;  Ax  a; x  u;  x  l

Adding slack variables:
Ax  f  b;  Ax  p  a; x  t  u;  x  g  l
bT v  aT q  uT s  l T h

Minimize:

Subject to:


Residual slackness:
V,q and s,h must be complementary, so only one can be non-zero. Otherwise we
could reduce them both by a factor and reduce the bjective function
AT v  AT q   h  s   c
fv  pq  ts  gh  0
Minimize
bT y   aT y   uT z   l T z 
Subject to :
Example:
AT y  z  c
Put all non-basic variables to zero and move in both positive and negative directions