Mathematical
Programming
ECE 665
Professor Maciej Ciesielski
By DFG
Computer Algorithms
Overview

Classical optimization deals with unconstrained
optimization problems.

For the minimization problem: Min f(x) subject to g(x) = 0
The solution is to move the constraints to the function and
solve the unconstrained optimization problem.
•
Penalty function method: replace f(x) by
F(x) = f(x)+ M( g(x))2, M >> 0
•
Lagrangian multipliers: replace f(x) by
L(x,λ) = f(x) + Σ λi gi(x)
At the optimum point, x*, L(x*, λ) = f(x*)
Department of Electrical Engineering - UMASS
2
Outline
 Mathematical programming paradigm
 Feasible solutions?
• Convexity
• Concave
• Concept example
 Geometric solution
 Fundamental theorem of LP
 Simplex
• Example
 LP Modifications
Department of Electrical Engineering - UMASS
3
Mathematical Programming Paradigm
A mathematical program is an optimization problem of the
form:
• Maximize (or Minimize) f(x)
subject to:
• g(x) = 0
• h(x) ≥ 0
Mathematical
Programming
paradigm
where x = [x1,...xn] is a subset of Rn, the functions g and
h are called constraints, and f is called the objective
function.
Department of Electrical Engineering - UMASS
4
The Nature of Mathematical Programming
Of course there are some forms that deviate from this
paradigm.
Important examples are:





Fuzzy mathematical program
Goal program
Multiple objectives
Randomized program
Stochastic program
Therefore it is typically a modeling issue to find an
standard form to solve the mathematical programming.
Department of Electrical Engineering - UMASS
5
When is it feasible?
A point x is feasible if it is in X and satisfies the
constraints:
Mathematical Programming
Paradigm
• g(x) = 0 and h(x) ≥ 0.
Min or Max
f(x)
Subject to:
g(x) = 0
h(x) ≥ 0
A point x* is optimal if it is feasible and if the value of the
objective function is:
• For Maximization
not less than that of any other feasible solution:
f(x*) ≥ f(x) for all feasible x.
• For Minimization
not greater than that of any other feasible solution:
f(x*) ≤ f(x) for all feasible x.
Department of Electrical Engineering - UMASS
6
Convexity
 To obtain global optimum, it is important that the
constraint set
Mathematical Programming
Paradigm
{x | g(x) = 0, h(x) ≥ 0}
Min or Max
f(x)
Subject to:
g(x) = 0
h(x) ≥ 0
and the function f(x) be convex.
If the function is convex over a convex constraint
set, then local minimun is also a globa1 minimum
Department of Electrical Engineering - UMASS
7
Convexity
 Set convexity
(If any two points are in the set, so is their line segment)
Convex set
 Function convexity
non-convex set
???
Convex function
Department of Electrical Engineering - UMASS
non-convex function
8
Convexity
A function f:X–>R is said to be convex if:
 Its epigraph is convex.
Epigraph: region on
or above the graph
 X is a convex set, and for x,y  X and a[0, 1]:
f(ax + (1-a)y) ≤ af(x) + (1-a)f(y)
Department of Electrical Engineering - UMASS
9
Convexity
If the function is convex over a convex constraint
set, then local minimun is also a globa1 minimum
Department of Electrical Engineering - UMASS
10
Concave
 It is define for maximization, and it is the
negative of convexity.
 A function f is concave if its hypograph is convex
Hypograph: region
below the graph
Department of Electrical Engineering - UMASS
11
Concave
If the function is concave over a convex constraint
set, then local maximun is also a globa1 maximum
Department of Electrical Engineering - UMASS
12
Concept Example
 Can we minimize based on the local minimum?
f(x) = xTQx
 Subject to:
Ax ≥ b
x≥0
Quadratic
Programming
 Where Q is an n x n symmetric matrix, A is a constraint
matrix, and b is a constraint vector.
Department of Electrical Engineering - UMASS
13
Concept Example
 If xTQx > 0 for every x  0.
The matrix Q is positive definite. In which case its
eigenvalues are non-negative, and the quadratic form is a
convex function. Thus f is convex.
As Q is a symmetric matrix, these eingenvalues will be
reals.
 Ax ≥ b defines a convex constraint set, as its solution is a
Rk space. (A convex polyhedron)
 f and the constraint set are convex, therefore, finding the
local minimum also finds the global minimum.
Department of Electrical Engineering - UMASS
14
Linear Programming (LP)
 The goal of Linear programming is to:
(Max)Minimize:
Subject to:
CTx
Ax ≥ b
x≥0
Where CT is a coefficient vector for f, A is a constraint
matrix, and b es a constraint vector.
 The constraint set: {x | Ax ≥ b} is a convex polyhedron,
and f is linear so it is convex.
Therefore, LP has convexity, and the local min/max is the
global min/max
Department of Electrical Engineering - UMASS
15
LP Example (Problem Enunciation)
Optimization problem:
 2 types of products are made by the factory
 3 machines are needed to make each product
Constraints:
Machines are available for a
limited time during one production session:
 time on A ≤ 50 hours
 time on B ≤ 35 hours
 time on C ≤ 80 hours
Department of Electrical Engineering - UMASS
16
LP Example (Problem Enunciation)
To complete the product, each lot must be processed by
all three machines for a certain number of hours
Required manufacturing time
Machines
A
B
1
Products
10
5
5
5
5
15
2
Time Constraint:
C
50
Department of Electrical Engineering - UMASS
35
80
17
LP Example (Problem Enunciation)
Profit:
 Product 1 can be sold for $100 per lot
 Product 2 can be sold for $80 per lot
Optimization Problem:
 Design the production schedule which
maximizes the profit of both products
manufactured during one production session.
Department of Electrical Engineering - UMASS
18
LP Example (Problem Formulation)
 Formulation
x1 = number of lots of product 1
x2 = number of lots of product 2
max f (x) = 100x1 + 80x2
10x1 +
5x1 +
5x1 +
5x2 ≤ 50
5x2 ≤ 35
15x2 ≤ 80
Department of Electrical Engineering - UMASS
19
LP Example (Geometric Solution)
x2
 Geometric solution: Family of curves,
constant for x1, x2
10
F(x) = 100 x1 + 80 x2 = 800 (const)
F(x) = 800
(a)
(b)
(c)
5
4
Extreme
point P
0
P2
10x1 + 5x2  50
5x1 + 5x2  35
5x1 + 15x2  80
Sol:
X1 = 3
X2 = 4
3
P1
5
a
b
10
Department of Electrical Engineering - UMASS
15
c
x1
20
LP Example (Standard Linear Program Approach)
 Convert the problem to a standard Linear Program (LP):
Max f(x) = 100x1 + 8Ox2 + 0s1 + 0s2 + 0s3
Subject to:
10x1 + 5x2 + s1
= 50
5x1 + 5x2 + s2
= 35
5x1 + 15x2+
s3 = 80
Why the
Basis??
Basis
New Matrix A
Department of Electrical Engineering - UMASS
21
Basis and Basic solution
Why the Basis?
• Because it finds a basic solution
Max
f(x) = 100x1 + 8Ox2 + 0s1 + 0s2 + 0s3
Subject to:
10x1 + 5x2 + s1
= 50
5x1 + 5x2 +
s2
= 35
5x1 + 15x2 +
s3 = 80
Given a system of equalities: Ax = b
where:
x = n-vector,
b = m-vector,
A = mn matrix
select a set of m linearly independent columns such that (mm)
matrix B is nonsingular, i.e. |B|  0.
Then one may uniquely solve the equation:
where:
Bxb= b
xb is an m-element subvector of x.
Department of Electrical Engineering - UMASS
22
Basis and Basic solution
By putting x=(xb,0) we obtain a solution to
Max
f(x) = 100x1 + 8Ox2 + 0s1 + 0s2 + 0s3
Subject to:
10x1 + 5x2 + s1
= 50
5x1 + 5x2 +
s2
= 35
5x1 + 15x2 +
s3 = 80
Ax = b
Such a solution (with n-m components of x not associated with
columns of Bmm) to the resulting set of equations is said to be a
basic solution with regard to the basis B.
The Xb variables, associated with columns of B, are called basic
variables.
basic variables
Basic Feasible Solution
B.F.S x1 = 0
x2 = 0
s1 = 50
s2 = 35
s3 = 80
If a feasible solution (i.e. the one which satisfies all contraints) is
also basic, it is called a basic feasible solution.
Department of Electrical Engineering - UMASS
23
Fundamental Theorem of Linear Programming
Given an LP in standard form
min CTx
subject to Ax = b
x0
where A is mn matrix of rank m (i.e. m < n and the m
rows are linear independent),
• If there is a feasible solution, there is a basic feasible
solution
• If there is an optimal feasible solution, there is an optimal
basic feasible solution
Department of Electrical Engineering - UMASS
24
Fundamental Theorem of Linear Programming
This theorem shows that it is necessary only to consider
basic feasible solutions when seeking an optimal solution
to a linear program, as the optimal value is always
achieved as such a solution
Ultimate goal:
• find a basic feasible solution with a base B composed of
original variables only, and which is optimum.
Department of Electrical Engineering - UMASS
25
LP Example (Basic Feasible solution)
ƒ (x) = 100 x1 + 80 x2 + 0 s1 + 0 s2 + 0 s3
 x1 
 
10 5 1 0 0   x2  50 
 5 5 0 1 0   s1   35

   
 5 15 0 0 1   s 2  80 
 s3 
basis
 Solution: x1 = 0, x2 = 0, s1 = 50, s2 = 35, s3 = 80, is a
basic feasible solution: point P0(0,0) Extreme point
 Function value: F(0, 0, 50, 35, 80) = 0
Department of Electrical Engineering - UMASS
26
LP Example (Basic Feasible solution)
x2
The basic feasible solution in the
Geometric approach is in the Extreme
Point.
10
5
4
P2
Extreme
point
P0
3
P1
5
a
b
10
Department of Electrical Engineering - UMASS
15
c
x1
27
Simplex Method
 Proceed from one basic feasible solution (extreme point)
to another, in such a way as to continually
decrease / increase
the value of the f (x) until a
minimum / maximum
is reached.
General comments:
 It is easy find initial basic feasible solutions with slack
variables.
 Finding initial basic solution is part of the Simplex
method (Lue 84).
Department of Electrical Engineering - UMASS
28
Simplex Algorithm
1. Select the column, such that the new resulting basic
feasible solution will yield a lower / greater value to f(x)
than the previous one.
2. Select the pivot element in that column.
By an elementary evaluation determine:
•
Which vector aj should enter the basis (so that f(x) is
reduced), and
•
Which vector should leave the basis.
Department of Electrical Engineering - UMASS
29
Simplex Algorithm
Construct a Simplex Table, [A, b ] appended by a row at
the top with cost coefficients and current cost.
j=
1
2
3
4
5
0
rj
100
80
0
0
0
0
y1j
10
5
1
0
0
50
Point P0 (0,0)
F(0,0,50,35,80) = 0
y2j
5
5
0
1
0
35
y3j
5
15
0
0
1
80
yi1
yi2
yi3
yi4
yi5
yi0
Department of Electrical Engineering - UMASS
30
Simplex Algorithm

Step 1.
Select column q with rq > 0.
(xq enters the basis)
q : rq = max ri

Step 2.
Select row p, which defines pivot element p in
column q Which minimizes ratio:
yi 0
yiq
p:
y p0
y pq
yi 0
= min
yiq
i
Department of Electrical Engineering - UMASS
for positive yiq
31
Simplex Algorithm

Step 3.
Pivot on element ypq and update the table
(update all rows, including the top one)
•
In row p divide ypj by ypq ,
•
Subtract from each row i  p
y p0
y pq
j = 0, 1,…, n
 ypj , j = 0, 1,…, n
Department of Electrical Engineering - UMASS
32
Simplex Algorithm Example
 Pivot on y11
Step2: min of
y10 50

5
y11 10
Step1: max rq
j=
1
2
3
4
5
0
rj
100
80
0
0
0
0
y1j
10
5
1
0
0
50
y2j
5
5
0
1
0
35
y3j
5
15
0
0
1
80
yi1
yi2
yi3
yi4
yi5
yi0
Department of Electrical Engineering - UMASS
y20 35

7
y21 5
y30 80

 16
y31 5
Therfore:
ypq = 10
Point P0 (0,0)
F(0,0,50,35,80) = 0
33
Simplex Algorithm Example
Step 3.B: in row i ≠ p
 Update table
yij  yij 
jj =
=
yiq
y pq
y pj  yij 
yiq
1
y pj
11
22
33
44
55
00
rjj
100
0
80
30
0
-10
0
0
0
-500
y1j
1j
10
1
5
0.5
1
0.1
0
0
0
0
50
5
y 2j
5
0
5
2.5
0
-0.5
1
1
0
0
35
10
y3j
3j
5
0
15
12.5
0
-0.5
0
1
80
55
yi1
yi2
yi3
yi4
yi5
yi0
Department of Electrical Engineering - UMASS
Step 3.A: in row p
y pj
y pq

y1 j
y11

y1 j
10
Point P
P0
(0,0)
1 (5,0)
F(5,0,0,10,55)
F(0,0,50,35,80) ==
0 500
34
Simplex Algorithm Example
 Pivot on y22
Step2: min of
y10
5

 10
y12 0.5
Step1: max rq
j=
1
2
3
4
5
0
rj
0
30
-10
0
0
-500
y1j
1
0.5
0.1
0
0
5
y2j
0
2.5
-0.5
1
0
10
y3j
0
12.5
-0.5
0
1
55
yi1
yi2
yi3
yi4
yi5
yi0
Department of Electrical Engineering - UMASS
y20 10

4
y22 2.5
y30
55

 4.4
y32 12.5
Therfore:
ypq = 4
Point P1 (5,0)
F(5,0,0,10,55) = 500
35
Simplex Algorithm Example
Step 3.B: in row i ≠ p
 Update table
yij  yij 
jj =
=
yiq
y pq
y pj  yij 
yiq
1
y pj
1
1
2
2
33
44
55
00
rrj
j
100
0
80
30
0
0
-10
-4
0
-12
0
0
-500
-620
y
y1j
1j
1
0.5
0
0.1
1
0
0
5
3
y
y2j
2j
5
0
5
1
0
-0.2
1
0.4
0
35
4
y3j
3j
5
0
15
12.5
0
0
-0.5
2
0
-5
1
80
55
5
y i1
i1
y i2
i2
y i3
i3
y i4
i4
y i5
i5
y i0
i0
Department of Electrical Engineering - UMASS
Step 3.A: in row p
y pj
y pq

y2 j
y22

y2 j
2.5
Point
Point P
P1
(5,0)
2 (3,4)
F(3,4,0,0,5)
620
F(5,0,0,10,55) =
= 500
36
Simplex Algorithm Example
 Pivot on
Step1: all rq < 0
Terminate algorithm
j=
1
2
3
4
5
0
rj
0
0
-4
-12
0
-620
y1j
1
0
1
0
0
3
y2j
0
1
-0.2
0.4
0
4
y3j
0
0
2
-5
1
5
yi1
yi2
yi3
yi4
yi5
yi0
Department of Electrical Engineering - UMASS
Sol:
X1 = 3
X2 = 4
Point P2 (3,4)
F(3,4,0,0,5) = 620
37
Linear Programming Modifications
1. Maximization
max c t x = min (-c t x)
2. Unconstrained x
let x = x+ - x - ; x+, x -  0)
min (c t x+ - c t x -)
s. to
[A, -A]
x+
x-
x+, x -  0
Department of Electrical Engineering - UMASS
= b
38
Linear Programming Modifications
3. Constraint set:
Ax  b
Slack variables, xs : Ax + xs = b
min
[cT
, 0]
x
xs
[A, I ]
x
xs
= b
x , xs  0
Columns corresponding to xs form a basic solution
Department of Electrical Engineering - UMASS
39
Linear Programming Modifications
4. Constraint set:
Ax ≥ b
Surplus variables, xp : Ax - xp = b
min
[cT
, 0]
x
xp
Department of Electrical Engineering - UMASS
[A, -I ]
x
xp
= b
x , xp  0
40
Linear Programming Modifications
5. Artificial variables, xa : Ax - xp + xa = b
(cannot star with negative basis of xp )
min [cT , 0 , k I ]
x
xp
xa
[A, -I , I ]
S. to
x
xp
xa
= b
x , xp , xa  0
Columns corresponding to xa form a basic solution
Department of Electrical Engineering - UMASS
41