Dynamic Programming 2
– Review
– More examples
1
MIT and James Orlin © 2003
Match game example

Suppose that there are 20 matches on a table,
and the person who picks up the last match wins.
At each alternating turn, my opponent or I can
pick up 1, 2 or 6 matches. Assuming that I go
first, how can I be sure of winning the game?
2
MIT and James Orlin © 2003
Determining the strategy using DP


n = number of matches left (n is the state/stage)
g(n) = 1 if you can force a win at n matches.
g(n) = 0 otherwise g(n) = optimal value function.
At each state/stage you can make one of three decisions: take 1,
2 or 6 matches.


g(1) = g(2) = g(6) = 1 (boundary conditions)
g(3) = 0; g(4) = g(5) = 1.
The recursion:
 g(n) = 1 if g(n-1) = 0 or g(n-2) = 0 or g(n-6) = 0;
g(n) = 0 otherwise.
 Equivalently, g(n) = 1 – min (g(n-1), g(n-2), g(n-6)).
3
MIT and James Orlin © 2003
Winning and Losing Positions
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
4
MIT and James Orlin © 2003
Principle of Optimality

Any optimal policy has the property that
whatever the current state and decision,
the remaining decisions must constitute
an optimal policy with regard to the state
resulting from the current decision.

Whatever node j is selected, the remaining
path from j to the end is the shortest path
starting at j.
5
MIT and James Orlin © 2003
Capital Budgeting, again
Investment budget = $14,000
Investment 1
Cash
Required
(1000s)
NPV
added
(1000s)
$5
2
3
4
5
6
$7
$4
$3
$4
$6
$16 $22 $12 $8
$11 $19
6
MIT and James Orlin © 2003
Stages, and the DP recursion

Solve the knapsack problem as a
sequence of problems
 At stage k we consider whether to add
item k or not to the knapsack.

Let f(k,B) be the best NPV limited to stocks 1, 2,
…, k only and using a budget of at most B.
Knapsack problem via DP
7
MIT and James Orlin © 2003
Solving the Stockco Problem

At stage 1 compute for each budget from 0 to
$14,000 compute the best NPV while limited to
stock 1.

At stage 2 compute for each budget from 0 to
$14,000 compute the best NPV while limited to
stocks 1 and 2.

At stage 3 compute for each budget from 0 to
$14,000 compute the best NPV while limited to
stocks 1, 2 and 3.
8
MIT and James Orlin © 2003
Capital budgeting in general
Max
s.t
Sj=1..n
Sj=1..n
cj xj
aj xj  b
x binary
Let f(k, B) = Max
s.t
Sj=1..k cj xj
Sj=1..k aj xj 
B
x binary
9
MIT and James Orlin © 2003
The recursion

f(0,0) = 0; f(0,k) is undefined for k > 0

f(k, B) = min ( f(k-1, B), f(k-1, B-ak) + ck)
either item k is included, or it is not
The optimum solution to the original problem is
max { f(n, B) : 0  B  b }.
Note: we solve the capital budgeting problem for all
right hand sides less than b.
10
MIT and James Orlin © 2003
The Knapsack Problem
Is there a feasible solution to
31 x + 35 y + 37 z = 422
x, y, z are non-negative integers.
let f(1, j) = T if there is a solution to 31 x = j,
f(1, j) = F if there is no solution to 31 x = j.
where x is a non-negative integer
let f(2, j) = T if there is a solution to
31 x + 35 y = j,
x and y are non-negative integer
MIT and James Orlin © 2003
11
The Knapsack Problem
Is there a feasible solution to
31 x + 35 y + 37 z = 422
x, y, z are non-negative integers.
let f(3 ,j) = T if there is a solution to
31 x + 35 y + 37 z = j,
x, y, and z are non-negative integer
Note: there are three stages to this problem.
12
MIT and James Orlin © 2003
The recursion for the knapsack problem
Is there a feasible solution to
31 x = 422
x non-negative integer?
f(0,0) = T, f(0,k) = F for k > 0
f(1,k) = T
if f(0,k)
f(1, k-31)
f(1,k) = F
otherwise.
=T
=T
or
for k = 31 to 422
Does this really work?
13
MIT and James Orlin © 2003
The recursion for the knapsack problem
Is there a feasible solution to
31 x + 35 y = 422
x, y are non-negative integers.
f(2,k) = T
or
if f(1,k) = T
f(2, k-35) = T for k = 0 to 422
f(2,k) = F otherwise
14
MIT and James Orlin © 2003
The recursion for the knapsack problem
Is there a feasible solution to
31 x + 35 y + 37 z = 422
x, y, z are non-negative integers?
f(3,k) = T
f(3,k) = F
if f(2,k)
=T
or f(3, k-37) = T
otherwise
15
MIT and James Orlin © 2003
The DP again, without stages
Is there a feasible solution to
31 x + 35 y + 37 z = 422
x, y, z are non-negative integers.
let g(j) = T if there is a solution to
31 x + 35 y + 37 z = j,
x, y, and z are non-negative integer
Then g(0) = T.
For j > 0, g(j) is true if
i.
g(j-31) is true or
ii.
g(j-35) is true or
iii.
g(j-37) is true
16
MIT and James Orlin © 2003
Optimal Capacity Expansion: What is the
least cost way of building plants?
Year
Cum. Demand
Cost per plant in
$millions
2002
1
54
2003
2
56
2004
4
58
2005
6
57
2006
7
55
2007
8
52
Cost of $15 million in any year in which a plant is
built. At most 3 plants a year can be built
17
MIT and James Orlin © 2003
Cost of Building plants
cost of building in each year
3
177 183 189 186 180 171
2
123 127 131 129 125 119
1
69
71
73
72
70
67
0
0
0
0
0
0
0
years to go
6
5
4
3
2
1
0
Minimum number of plants needed
# of plants
years to go
6
1
5
2
4
4
3
6
2
7
1
8
0
18
MIT and James Orlin © 2003
Step 1. For each year, identify those table entries that are
feasible, taking into account the lower bounds and the fact
that at most 3 plants can be built per year.
8
7
6
5
4
3
2
1
0
6
5 4 3
Years to go
2
1
0
(Y, k) is a state denoting having k plants with Y years
to go.
MIT and James Orlin © 2003
19
Step 2. f(Y, k) = remaining cost of capacity expansion
starting with k plants with Y years to go.
Fill in the entries for 0 years to go. Then fill in the entries
for 1 year to go. Then fill in entries for 2 years to go, and
continue.
8
7
6
5
4
3
2
1
0
6
MIT and James Orlin © 2003
5 4 3
Years to go
2
1
0
20
Determining state and stage

Stage: Y = years to go
 State: number of plants
 f(j, Y) = optimum cost of capacity
expansion starting with j plants with Y
years to go
 We want to compute f(0, 6).

What is f(j, 0) for different j?
21
MIT and James Orlin © 2003
The recursion





Let g(Y, k) be the cost of building k plants with Y
years to go.
Let l(Y) be the minimum number of plants needed
with Y years to go.
Let M(Y) be the maximum number of plants that
can be built with Y years to go.
Let f(Y,k) be the remaining cost of capacity
expansion starting with k plants with Y years to
go.
What is f(0,k)? Can you compute f(Y,k) assuming
that f(Y-1, j) is already computed for each j?
22
MIT and James Orlin © 2003
Summary for Dynamic Programming

Recursion
 Principle of optimality
 states and stages and decisions
 useful in a wide range of situations
– games
– shortest paths
– capacity expansion
– more will be presented in next lecture.
23
MIT and James Orlin © 2003
Dynamic Programming Review

Next lecture: DP under uncertainty.

Applications
– meeting a financial goal
(a very simplified version)
– getting a plane through enemy territory
24
MIT and James Orlin © 2003