Complexity of linear programming: outline
I
Assessing computational efficiency of algorithms
I
Computational efficiency of the Simplex method
I
Ellipsoid algorithm for LP and its computational efficiency
IOE 610: LP II, Fall 2013
Complexity of linear programming
Page 139
Problem, instance, algorithm
I
I
“Linear Programming in inequality form” is a problem;
An instance of this problem is given by d = (A, b, c)
Definition 8.1
An instance of an optimization problem consists of a feasible set F
and a cost function c : F ! <. An optimization problem is defined
as a collection of instances.
I
I
I
An algorithm is a finite set of instructions, such as arithmetic
operators, conditional statements, read and write statements,
etc., and the running time of the algorithm is the total
number of steps involved in carrying out these instructions
until termination is reached
Algorithms are described for problems, but then applied to
individual instances.
An algorithm solves the problem if it terminates in finite time
and produces a correct answer for all instances of the problem.
IOE 610: LP II, Fall 2013
Complexity of linear programming
Page 140
Algorithms for Solving (Optimization) Problems
I
It is reasonable to expect the running time of an algorithm to
depend on the size of the instance to which it is applied.
Definition 8.2
The size of an instance is defined as the number of bits used to
describe the instance, according to a prespecified format.
.... geared towards binary computation and integer input data
I r 2 Z, |r |  U: need blog2 Uc + 2 bits (binary representation)
r = ak 2k +ak
I
k 1
+· · ·+a1 w 1 +a0 20 ,
12
ai 2 {0, 1} 8i, k  blog2 Uc
An LP instance with d 2 Zmn+m+n , max{|aij |, |bi |, |cj |} = U:
the size of d is O(mn log2 U)
IOE 610: LP II, Fall 2013
Complexity of linear programming
Page 141
Computational efficiency analysis of algorithms
Question: Given an algorithm that solves our problem and an
instance of our problem of a particular size, how long will the
algorithm take to solve it? Two types of answers:
I Average performance on a typical problem instance
I
I
I
Mathematically difficult to define or analyze rigorously
Observations from empirical experience give some hints
Worst case performance: how long for the most difficult (for
this algorithm) instance of the given size?
I
I
I
I
IOE 610: LP II, Fall 2013
Often too pessimistic to predict practical behavior
Relatively easy to define and analyze rigorously
Provides a worst-case guarantee
Important for the theory of complexity: Suggests where the
limits of our computational abilities are
Complexity of linear programming
Page 142
Worst-case analysis and efficient algorithms
I
We will be looking at T (S): the worst-case running time of
the algorithm on instances of the size (at most) S.
I
I
As before, we will use the arithmetic model of computation,
and count the number of arithmetic operations to estimate
T (S)
The algorithm is usually considered efficient if T (S) can be
bounded above by a polynomial function
Definition 8.3
An algorithm runs in polynomial time (poly-time) if 9k 2 Z such
that T (S) = O(S k ).
IOE 610: LP II, Fall 2013
Complexity of linear programming
Page 143
How can we measure the running time of an algorithm?
I
Arithmetic model of computation: each operation (including
arithmetic operations) takes unit time. (Easy to analyze.)
I
Bit model of computation: each operation is decomposed into
bit operations, and it’s these elementary operations that are
assumed to take unit time (for example, adding two numbers
takes longer if the numbers are large). (Better estimate of the
true running time)
Fact Suppose
I
I
I
an algorithm takes poly-time under the arithmetic model, and
on instances of size S, any integer produced in the course of
the algorithm execution has size bounded by a polynomial in S.
Then the algorithm runs in polynomial time under the bit
model.
IOE 610: LP II, Fall 2013
Complexity of linear programming
Page 144
Computational efficiency of simplex method
I
An LP instance in n variables with m constraints
min cT x s.t. Ax
I
I
We have shown that each iteration takes polynomial time
(O(mn) with revised implementation)
Total number of iterations?
I
I
I
b
On problems arising in practice simplex method is usually
extremely fast
Conventional wisdom suggests that number of iterations in
practice is about 3m
Worst case — number of iterations exponential in n (Klee and
Minty, 1972)
IOE 610: LP II, Fall 2013
Complexity of linear programming
Page 145
Klee-Minty example
max
s.t.
n
X
2n j xj
j=1
i 1
X
2
j=1
xj
2i j xj + xi  100i
1
, i = 1, . . . , n
0, j = 1, . . . , n
For n = 3:
max 4x1 + 2x2 + x3
s.t. x1

1
4x1 + x2

100
8x1 + 4x2 + x3  10000
x1 , x 2 , x3
0
IOE 610: LP II, Fall 2013
Complexity of linear programming
Page 146
5
Klee-Minty
cube
ed Cube
Feasible region is a distortion to
a stretched n-dimensional cube
Case n = 3:
0  x1  1
0  x2  100
..
.
10000
esent a “minor” disn-dimensional hyper-
0  xn  100n
9600
1
 100
9992
9592
 100n 1.
100
1
96
5
1
Starting at x = 0, using
“steepest” pivot rule, simplex
method will visit all 2n vertices
before finding the optimal
solution
... what if we use a di↵erent
pivoting rule?
Thanks to R. Vanderbei for the illustration
IOE 610: LP II, Fall 2013
Complexity of linear programming
Page 147
Simplex method and diameter of polyhedra
Examples of “bad” polyhedra have been given for all known
pivoting rules. Is there an exponential example for any pivoting
rule, perhaps one not yet invented?
I If x and y are two vertices of a polyhedron, consider traveling
from x to y via adjacent vertices;
I d(x, y) — # of steps in the shortest such path
I Diameter of a polyhedron: maxx,y d(x, y)
I
(n, m) - maximum diameter of all bounded polyhedra in <n
that can be represented by m inequalities
I
I
With a “perfect” pivoting rule, simplex shouldn’t need more
than (n, m) iterations
Hirsch Conjecture proposed in 1957:
I
I
I
n
True for n  3 and other special cases
Disproven by Francisco Santos, 2010: a polyhedron with
n = 86 and m = 43 with diameter bigger than 43
Still open: (n, m)  m? Known: (n, m)  m1+log2 n
Even if we can bound
IOE 610: LP II, Fall 2013
(n, m)  m
(n, m), still need the right pivoting rule...
Complexity of linear programming
Page 148
Some LP history
I
1930’s — 1940’s
I
I
Late 1940’s
I
I
I
General LP theory (John Von Neumann) and solution method
(Simplex algorithm, George Dantzig) developed in US
Simplex runs quite fast in practice; LP used for military
operations and gains widespread use after the war
1972
I
I
(Specialized) LP models and solution approaches developed
independently in the Soviet Union and the West for a variety
of optimal resource allocation and planning applications
Klee and Minty show that the simplex algorithm is not
efficient, i.e., it does not run in polynomial time
1975: Nobel Prize in Economics is awarded for “for their
contributions to the theory of optimum allocation of
resources” via LP to Leonid V. Kantorovich and Tjalling C.
Koopmans
IOE 610: LP II, Fall 2013
Complexity of linear programming
Page 149
LP history, continued
I
1970’s
I
I
THE BIG QUESTION: Does there exist an algorithm for
solving LPs that’s poly-time in the worst case?
1979, in the Soviet Union...
I
I
IOE 610: LP II, Fall 2013
Leonid G. Khachiyan: YES, THERE IS — the Ellipsoid
algorithm!
NY Times publishes several articles about this discovery;
makes some mathematical blunders about the implications of
the result and has to print retractions
Complexity of linear programming
Page 150
Ellipsoid Algorithm for LP: outline
I
Develop general ideas for the Ellipsoid algorithm
I
Specify the algorithm for finding a point in
P = {x 2 <n : Ax b}
I
Modify the algorithm for solving min cT x s.t. Ax
IOE 610: LP II, Fall 2013
Ellipsoid algorithm
b
Page 151
Volumes and Affine Transformations
Definition
If L ⇢ <n , the volume of L is
Vol(L) =
Z
dx
x2L
Definition 8.6
If D 2 <n⇥n is nonsingular and b 2 <n , the mapping
S(x) = Dx + b is an affine transformation.
I
Note: by definition, affine transformation is invertible
Lemma 8.1
Let L ⇢ <n . If S(x) = Dx + b then Vol(S(L)) = | det(D)| · Vol(L).
IOE 610: LP II, Fall 2013
Ellipsoid algorithm
Page 152
Assumptions for Ellipsoid Algorithm
Goal of the algorithm: find a point in a suitably described convex
set P ⇢ <n .
Assumptions:
I
I
P is bounded: 9B(x0 , r ) ◆ P. Let V Vol(B(x0 , r )).
P is full-dimensional, i.e., has positive volume (Definition
8.7)
I
I
Let Vol(P) > v > 0.
P can be described via a separation oracle: given a vector
y 2 <n , the oracle either reports that y 2 P, or finds a
separating hyperplane: a 2 <n such that aT y < aT x 8x 2 P.
IOE 610: LP II, Fall 2013
Ellipsoid algorithm
Page 153
Positive definite matrices and Ellipsoids
Definition 8.4
Let Q be an n ⇥ n symmetric matrix, i.e., QT = Q. Q is called
positive definite (notation: Q 0) if xT Qx > 0 8x 6= 0.
I
I
I
n or Q
Symmetric: Q 2 S n ; symmetric PD: Q 2 S+
0
Q
Q 2 S n has n eigenvalues 1 , . . . , n ; det(Q) = ni=1
n i↵
Q 2 S+
>0
Definition 8.5
A set E ⇢ <n
E = E (z, Q) = {x : (x
where Q
IOE 610: LP II, Fall 2013
z)T Q
1
(x
z)  1}
0 is an ellipsoid with center z 2 <n .
Ellipsoid algorithm
Page 154
i
Ellipsoids as affine transformations of balls
I
A ball centered at z with radius r > 0:
E (z, r 2 I) = {x : (x
I
z)T (x
z)  r 2 } = {x : kx
zk  r }
Unit ball: ball centered at 0 with radius 1:
B(0, 1) = E (0, I)
I
I
1
1
1
1
n , 9Q 2 2 S n : Q 2 · Q 2 = Q and det(Q 2 ) =
If Q 2 S+
+
p
det(Q)
Note: an ellipsoid is an affine transformation of the unit ball:
1
E (z, Q) = Q 2 E (0, I) + z
Corollary:
Let Q
0. Then Vol(E (z, Q)) =
IOE 610: LP II, Fall 2013
Ellipsoid algorithm
p
det(Q) · Vol(B(0, 1)).
Page 155
Central-cut ellipsoid algorithm: general idea
Input: Ellipsoid E0 , constants V and v as above.
Initialization: E0 = E (x0 , Q0 ) ◆ P, t = 0.
Iteration t: (Et = E (xt , Qt ) ◆ P)
I
Call the separation oracle with xt as input.
I
If xt 2 P – terminate.
I
Otherwise, oracle returns a such that
P ✓ Et \ {x : aT x
aT xt }.
Construct ellipsoid Et+1 = E (xt+1 , Qt+1 ) of smaller volume
containing the set on the right.
I
Set t
IOE 610: LP II, Fall 2013
t + 1 and continue.
Ellipsoid algorithm
Page 156
Central-cut ellipsoid algorithm
Iteration details
How to construct Et+1 ◆ Et \ {x : aT x
aT xt }
Theorem 8.1
Let E = E (z, Q) be an ellipsoid in <n , and let 0 6= a 2 <n .
Consider the halfspace H = {x : aT x aT z} and let
1
Qa
p
,
n + 1 aT Qa
z̄ = z +
Q̄ =
n2
n2
1
✓
2 QaaT Q
n + 1 aT Qa
Q
◆
.
Then Q̄ 0, thus E 0 = E (z̄, Q̄) is an ellipsoid. Moreover,
(a) E \ H ⇢ E 0
(b) Vol(E 0 ) < e
1
2(n+1)
Vol(E ).
E 0 is the smallest-volume ellipsoid that contains E \ H
IOE 610: LP II, Fall 2013
Ellipsoid algorithm
Page 157
Central-cut ellipsoid algorithm
Termination and running time
I
At iteration t,
⇣
1
2(n+1)
Vol(Et ) < e
I
I
e
t
2(n+1)
P ✓ Et , so v  Vol(P)  Vol(Et )
v <e
t
2(n+1)
⌘t
Vol(E0 )
·V
· V , hence
t < 2(n + 1) ln(V /v )
I
Since t is an integer, we have
t  d2(n + 1) ln(V /v )e
I
1 ⌘ t?
1,
i.e., we must have xt ? 1 2 P.
So, the ellipsoid algorithm will terminate in at most t ?
iterations, each iteration consisting of a call to the separation
oracle and some arithmetics
IOE 610: LP II, Fall 2013
Ellipsoid algorithm
Page 158
Ellipsoid algorithm for linear inequalities
P = {x 2 <n : Ax
b}
Goal: find out whether P = ;.
I
I
Assume P is bounded and either empty or full-dimensional.
Also, presume E0 , v and V are known.
Separation oracle: for given y 2 <n , check if
aT
bi , i = 1, . . . , m
i y
I
I
If all constraints satisfied, y 2 P
If ith constraint is violated, aT
i x
hyperplane
aT
i y is a separating
The oracle requires O(mn) arithmetic operations
I
We assume for now calculations are made in infinite precision
and taking a square root takes the same time as any other
arithmetic operation
IOE 610: LP II, Fall 2013
Ellipsoid algorithm
Page 159
Central-cut ellipsoid algorithm for linear inequalities
Input: Ellipsoid E0 , constants V and v as above.
Initialization: E0 = E (x0 , Q0 ) ◆ P, t = 0. Let
t ? = d2(n + 1) ln(V /v )e.
Iteration t: (Et = E (xt , Qt ) ◆ P)
I
I
I
If t = t ? , stop; P = ;.
If xt 2 P, stop; P 6= ;.
Otherwise, find a violated constraint: aT
i xt < bi . Construct
ellipsoid
Et+1 = E (xt+1 , Qt+1 )
Et \ {x : aT
i x
(use Thm. 8.1).
I
Set t
IOE 610: LP II, Fall 2013
t + 1 and continue.
Ellipsoid algorithm
Page 160
aT
i xt }
Assumptions revisited: V
Lemma 8.2 (modified)
Let A 2 Zm⇥n , and b 2 <m . Let U = max{|aij |, |bi |}.
(a) Every basic solution of {x : Ax b} satisfies
(nU)n  xj  (nU)n , j = 1, . . . , n
(b) Every basic solution of P = {x : Ax = b, x
0} satisfies
(mU)m  xj  (mU)m , j = 1, . . . , n
Proof of (a):
I Basic solution is  1 b̂, where  2 <n⇥n is a submatrix of A
and b̂ 2 <n is a subvector of b.
I Cramer’s rule gives a formula of each component xj
If rows of A span <n , P 6= ; i↵ it has extreme points, all of which
are contained in
2n
n
E0 = E (0, n(nU)
I ), Vol(E0 ) < V = (2n)
(nU)n
IOE 610: LP II, Fall 2013
Ellipsoid algorithm
Page 161
2
Assumptions revisited: v
Lemma 8.4
Let P = {x : Ax b} be full-dimensional and bounded. Assume A
and b have integer entries of magnitude at most U. Then
2
Vol(P) n n (nU) n (n+1)
Idea of the proof:
I
P is the convex hull of its extreme points
I
If P 2 <n is full-dimensional, it has n + 1 extreme points that
do not belong to a common hyperplane
I
Vol(P)
I
The latter can be bounded by bounding components of
extreme points
IOE 610: LP II, Fall 2013
volume of convex hull of these extreme points
Ellipsoid algorithm
Page 162
Assumptions revisited: P full-dimensional
Lemma 8.3
Let P = {x : Ax b}. Assume A and b have integer entries of
magnitude at most U. Let
✏=
1
((n + 1))U)
2(n + 1)
(n+1)
.
Let P✏ = {x : Ax b ✏e}.
(a) If P is empty, then P✏ is empty
(b) If P is nonempty, then P✏ is full-dimensional.
IOE 610: LP II, Fall 2013
Ellipsoid algorithm
Page 163
Run-time of the ellipsoid algorithm: number of iterations
P = {x : Ax
b}
Assume A, b have integer components with magnitude bounded by
U, rows of A span <n .
Case 1: If we know that P is bounded, and either empty or
full-dimensional, take
E0 = E (0, n(nU)2n I ), v = n
n
(nU)
n2 (n+1)
, V = (2n)n (nU)n2 ,
and the algorithm with these inputs will terminate in
O(n4 log(nU)) iterations (Recall: instance size is O(mn log U))
Case 2: If P is arbitrary,
I Construct PB by adding bounds on variables as in Lemma 8.2
I
I
I
PB contains all extreme points of P
Construct PB,✏ as in Lemma 8.3
To decide if P = ;, apply EA to PB,✏ ; it will terminate in
O(n6 log(nU)) iterations.
IOE 610: LP II, Fall 2013
Ellipsoid algorithm
Page 164
Run-time of the ellipsoid algorithm: overall running time
I
I
I
We showed that number of iterations is polynomial in the size
of the problem
To show the algorithm takes polynomial time, need to show
that the number of operations/run-time per iteration is also
polynomial
Issues:
I
I
I
Need to take square roots — cannot be done accurately in
finite-precision calculations
Need to make sure the numbers generated have polynomial size
These problems can be resolved by showing that computations
need only be carried out with finite (polynomial) precision.
The result:
Theorem 8.3
The LP feasibility problem with integer data can be solved in
polynomial time.
IOE 610: LP II, Fall 2013
Ellipsoid algorithm
Page 165
Ellipsoid algorithm for LP optimization
(P) min cT x
s.t. Ax b
(D) max pT b
s.t. pT A = cT
p 0
Option 1: Apply ellipsoid algorithm to
Q = {(x, p) : bT p = cT x, Ax
b, AT p = c, p
0}.
Option 2: Sliding objective: start by finding x0 2 P; for each t
apply ellipsoid algorithm to
P \ {x : cT x < cT xt }.
If the set is empty, xt is a solution. Otherwise, we find a point
xt+1 2 P that’s better than xt , and continue.
IOE 610: LP II, Fall 2013
Ellipsoid algorithm
Page 166
Practical implications?
I
Although a great theoretical accomplishment, Ellipsoid
algorithm never became a method of choice for solving LPs in
practice
I
Its observed running time ⇡ its worst-case running time...
I
I
...unlike the simplex method
...or the Interior Point (Barrier) Methods
IOE 610: LP II, Fall 2013
Ellipsoid algorithm
Page 167