Dynamic Programming
Recursive Equations
1
D Nagesh Kumar, IISc
Optimization Methods: M5L2
Introduction and Objectives
Introduction

Recursive equations are used to solve a problem in sequence

These equations are fundamental to the dynamic programming
Objectives
To formulate recursive equations for a multistage decision
process
In a backward manner and
In a forward manner
2
D Nagesh Kumar, IISc
Optimization Methods: M5L2
Recursive Equations
3

Recursive equations are used to structure a multistage decision
problem as a sequential process

Each recursive equation represents a stage at which a decision
is required

A series of equations are successively solved, each equation
depending on the output values of the previous equations

A multistage problem is solved by breaking into a number of
single stage problems through recursion

Approached can be done in a backward manner or in a forward
manner
D Nagesh Kumar, IISc
Optimization Methods: M5L2
Backward Recursion

A problem is solved by writing equations first for the final stage and
then proceeding backwards to the first stage

Consider a serial multistage problem
NB1
S1
NBt
S2
Stage 1
St
X1

Stage t
NBT
St+1
Xt
ST
Stage T
ST+1
XT
Let the objective function for this problem is
T
T
t 1
t 1
f   NBt   ht  X t , S t 
4
 h1  X 1 , S1   h2  X 2 , S 2   ...  ht  X t , S t   ...  hT 1  X T 1 , S T 1   hT  X T , S T 
D Nagesh Kumar, IISc
Optimization Methods: M5L2
...(1
Backward Recursion …contd.

The relation between the stage variables and decision variables are
St+1 = g(Xt, St), t = 1,2,…, T.

Consider the final stage as the first sub-problem. The input variable to
this stage is ST.

Principle of optimality: XT should be selected such that hT  X T , ST  is
optimum for the input ST

The objective function fT for this stage is

fT ( ST )  opt hT  X T , ST 
XT
Next, group the last two stages
together as the second sub-problem.
The objective function is
fT1 ( ST 1 )  opt hT 1  X T 1 , ST 1   hT  X T , ST 
X T 1 , X T
5
D Nagesh Kumar, IISc
Optimization Methods: M5L2
Backward Recursion …contd.


By using the stage transformation equation, fT 1 ( ST 1 ) can be rewritten
as




fT 1 ( ST 1 )  opt hT 1  X T 1 , ST 1   fT gT 1  X T 1 , ST 1 
X T 1

Thus, a multivariate problem is divided into two single variable
problems as shown

In general, the i+1th sub-problem can be expressed as
fTi ( ST i ) 
opt
hT i  X T i , ST i   ...  hT 1  X T 1 , ST 1   hT  X T , ST 
X T i ,..., X T 1 , X T

Converting this to a single variable problem


fTi ( ST i )  opt hT i  X T i , ST i   fT(i 1) gT i  X T i , ST i 
X T i
6
D Nagesh Kumar, IISc
Optimization Methods: M5L2
Backward Recursion …contd.

fT(i 1) denotes the optimal value of the objective function for
the last i stages

Principle of optimality for backward recursion can be stated as,

No matter in what state of stage one may be, in order
for a policy to be optimal, one must proceed from that
state and stage in an optimal manner sing the stage
transformation equation
7
D Nagesh Kumar, IISc
Optimization Methods: M5L2
Forward Recursion

The problem is solved by starting from the stage 1 and proceeding
towards the last stage

Consider a serial multistage problem
NB1
S1
NBt
S2
Stage 1
St
X1

Stage t
NBT
St+1
Xt
ST
Stage T
ST+1
XT
Let the objective function for this problem is
T
T
t 1
t 1
f   NBt   ht  X t , S t 
8
 h1  X 1 , S1   h2  X 2 , S 2   ...  ht  X t , S t   ...  hT 1  X T 1 , S T 1   hT  X T , S T 
D Nagesh Kumar, IISc
Optimization Methods: M5L2
...(1
Forward Recursion …contd.

The relation between the stage variables and decision variables are
St  g  X t 1 , St 1 
t  1,2,...., T
where St is the input available to the stages 1 to t



Consider the stage 1 as the first sub-problem. The input variable to
this stage is S1
Principle of optimality: X1 should be selected such that h1  X 1 , S1  is
optimum for the input S1
The objective function f1 for this stage is
f1 ( S1 )  opt h1  X 1 , S1 
X1
9
D Nagesh Kumar, IISc
Optimization Methods: M5L2
Backward Recursion …contd.

Group the first and second stages together as the second subproblem. The objective function is
f 2 ( S 2 )  opt h2  X 2 , S 2   h1  X 1 , S1 
X 2 , X1


By using the stage transformation equation, f 2 ( S 2 ) can be rewritten
as
f 2 ( S 2 )  opt h2  X 2 , S 2   f1 g 2  X 2 , S 2 


X2

In general, the ith sub-problem can be expressed as
f i  ( Si ) 
opt
hi  X i , Si   ...  h2  X 2 , S2   h1  X 1 , S1 
X 1 , X 2 ,..., X i
10
D Nagesh Kumar, IISc
Optimization Methods: M5L2
Backward Recursion …contd.

Converting this to a single variable problem


f i  ( Si )  opt hi  X i , Si   f (i 1) g i X i , Si 
Xi

f i  denotes the optimal value of the objective function for the
last i stages

Principle of optimality for forward recursion can be stated as,

11
No matter in what state of stage one may be, in order
for a policy to be optimal, one had to get to that state
and stage in an optimal manner
D Nagesh Kumar, IISc
Optimization Methods: M5L2
Thank You
12
D Nagesh Kumar, IISc
Optimization Methods: M5L2