O PTIMAL C ONTROL T HEORY: A PPLICATIONS TO
M ANAGEMENT S CIENCE AND E CONOMICS
(S ECOND E DITION , 2000)
Suresh P. Sethi
Gerald. L. Thompson
Springer
Chapter 1 – p. 1/37
C HAPTER 1
W HAT IS O PTIMAL C ONTROL T HEORY ?
Chapter 1 – p. 2/37
W HAT IS O PTIMAL C ONTROL T HEORY ?
• Dynamic Systems: Evolving over time.
Chapter 1 – p. 3/37
W HAT IS O PTIMAL C ONTROL T HEORY ?
• Dynamic Systems: Evolving over time.
• Time: Discrete or continuous; Optimal way to control a
dynamic system.
Chapter 1 – p. 3/37
W HAT IS O PTIMAL C ONTROL T HEORY ?
• Dynamic Systems: Evolving over time.
• Time: Discrete or continuous; Optimal way to control a
dynamic system.
• Prerequisites: Calculus, Vectors and Matrices, ODE and
PDE.
Chapter 1 – p. 3/37
W HAT IS O PTIMAL C ONTROL T HEORY ?
• Dynamic Systems: Evolving over time.
• Time: Discrete or continuous; Optimal way to control a
dynamic system.
• Prerequisites: Calculus, Vectors and Matrices, ODE and
PDE.
• Applications: Production, Finance, Economics, Marketing
and others.
Chapter 1 – p. 3/37
B ASIC C ONCEPTS AND D EFINITIONS
• A dynamic system is described by state equation:
ẋ(t) = f (x(t), u(t), t),
x(0) = x0 ,
(1)
where x(t) is state variable, u(t) is control variable.
• The control aim is to maximize the objective function:
J=
Z
T
F (x(t), u(t), t)dt + S[x(T ), T ].
(2)
0
• Usually the control variable u(t) will be constrained as
follows:
u(t) ∈ Ω(t),
t ∈ [0, T ],
(3)
Chapter 1 – p. 4/37
B ASIC C ONCEPTS AND D EFINITIONS CONT.
• Sometimes, we consider the following constraints:
(1) Inequality constraints (mixed)
g(x(t), u(t), t) ≥ 0, t ∈ [0, T ] .
(4)
(2) Constraints involving only state variables (pure)
h(x(t), t) ≥ 0, t ∈ [0, T ].
(5)
(3) Terminal state
x(T ) ∈ X,
(6)
where X is called the reachable set of the state variable at
time T.
Chapter 1 – p. 5/37
F ORMULATIONS OF S IMPLE C ONTROL M ODELS
• Example 1.1 A Production-Inventory Model.
We consider the production and inventory storage of a given
good in order to meet an exogenous demand at minimum
cost.
Chapter 1 – p. 6/37
T HE P RODUCTION -I NVENTORY M ODEL
State Variable
I(t) = Inventory Level
Control Variable
State Equation
P (t) = Production Rate
˙
I(t)
= P (t) − S(t), I(0) = I0
Objective Function
Maximize J =
State Constraint
I(t) ≥ 0
Control Constraints
0 ≤ Pmin ≤ P (t) ≤ Pmax
Terminal Condition
I(T ) ≥ Imin
Exogenous
S(t) = Demand Rate
Functions
h(I) = Inventory Holding Cost
n
RT
0
−[h(I(t)) + c(P (t))]dt
o
c(P ) = Production Cost
Parameters
T = Terminal Time
Imin = Minimum Ending Inventory
Pmin = Minimum Possible Production Rate
Pmax = Maximum Possible Production Rate
I0 = Initial Inventory Level
Chapter 1 – p. 7/37
F ORMULATIONS OF S IMPLE C ONTROL M ODELS
• Example 1.2 An Advertising Model.
We consider a special case of the Nerlove-Arrow advertising
model.
Chapter 1 – p. 8/37
T HE A DVERTISING M ODEL
State Variable
G(t) = Advertising Goodwill
Control Variable
u(t) = Advertising Rate
State Equation
Ġ(t) = u(t) − δG(t), G(0) = G0
Objective Function
Maximize J =
R∞
0
···
State Constraint
Control Constraints
[π(G(t)) −
u(t)]e−ρt dt
0 ≤ u(t) ≤ Q
···
Terminal Condition
Exogenous Function
π(G) = Gross Profit Rate
Parameters
δ = Goodwill Decay Constant
ρ = Discount Rate
Q = Upper Bound on Advertising Rate
G0 = Initial Goodwill Level
Chapter 1 – p. 9/37
F ORMULATIONS OF S IMPLE C ONTROL M ODELS
• Example 1.3 A Consumption Model.
We consider a problem of an agent’s consumption of his
wealth over time in a way that maximizes his consumption
utility over his lifetime.
Chapter 1 – p. 10/37
T HE C ONSUMPTION M ODEL
State Variable
W (t) = Wealth
Control Variable
C(t) = Consumption Rate
State Equation
Ẇ (t) = rW (t) − C(t), W (0) = W0
Objective Function
Max J =
State Constraint
W (t) ≥ 0
Control Constraint
C(t) ≥ 0
n
RT
0
U (C(t))e−ρt dt
Functions
Parameters
o
···
Terminal Condition
Exogenous
+
B[W (T )]e−ρT
U (C) = Utility of Consumption
B(W ) = Bequest Function
T = Terminal Time
W0 = Initial Wealth
ρ = Discount Rate
r = Interest Rate
Chapter 1 – p. 11/37
H ISTORY OF O PTIMAL C ONTROL T HEORY
• Calculus of Variations.
• Brachistochrone problem: path of least time
• Newton, Leibniz, Bernoulli brothers, Jacobi, Bolza.
• Pontryagin et al.(1958): Maximum Principle.
Chapter 1 – p. 12/37
F IGURE 1.1 T HE B RACHISTOCHRONE P ROBLEM
Chapter 1 – p. 13/37
N OTATION AND C ONCEPTS U SED
• “=” “is equal to” or “is defined to be equal to” or “is
identically equal to”
• “:=” “is defined to be equal to,”
• “≡” “is identically equal to,”
• “≈” “is approximately equal to.”
• “⇒” “implies”
• “∈” “is a member of.”
Chapter 1 – p. 14/37
N OTATION AND C ONCEPTS U SED CONT.
Let y be an n-component column vector and z be an
m-component row vector, i.e.,


y1


 y 
 2 
y =  .  = (y1 , . . . , yn )T and z = (z1 , . . . , zm ),
 .. 


yn
ẏ := dy/dt and ż := dz/dt are defined as
dz
dy
T
= (ẏ1 , · · · , ẏn ) and ż =
= (ż1 , . . . , żm ),
ẏ =
dt
dt
When n = m, we can define the inner product
zy = Σni=1 zi yi .
(7)
Chapter 1 – p. 15/37
N OTATION AND C ONCEPTS U SED CONT.




More generally, if A = {aij } = 


a11
a21
..
.
a12
a22
..
.
···
···
···
a1k
a2k
..
.







am1 am2 · · · amk
is an m × k matrix and B = {bij } is a k × n matrix, we define
the matrix product C = {cij } = AB, which is an m × n matrix
with components
cij = Σkr=1 air brj .
(8)
Chapter 1 – p. 16/37
D IFFERENTIATING V ECTORS AND M ATRICES
W. R . T.
S CALARS
• Let f : E 1 → E k be a k-dimensional function of a scalar
variable t. If f is a row vector, then
df
= ft = (f1t , f2t , · · · , fkt ), a row vector.
dt
• If f is a column vector, then


f1t


 f 
df
 2t 
= ft =  .  = (f1t , f2t , · · · , fkt )T , a column vector.
 .. 
dt


fkt
Chapter 1 – p. 17/37
D IFFERENTIATING S CALARS W. R . T. V ECTORS
• If F (y, z) is a scalar function of n-dimensional column
vector y and m-dimensional row vector z, n ≥ 2, m ≥ 2,
then the gradients Fy and Fz are defined, respectively, as
Fy = (Fy1 , · · · , Fyn ), a row vector,
(9)
and


Fz1
 . 
..  , a column vector,
Fz = 


Fzm
(10)
Chapter 1 – p. 18/37
D IFFERENTIATING V ECTORS W. R . T. V ECTORS
If f : E n × E m → E k is a k-dimensional vector function f
either row or column, k ≥ 2, i.e.,
f = (f1 , · · · , fk ) or f = (f1 , · · · , fk )T ,
where fi = fi (y, z) depends on column y ∈ E n and row
z ∈ E m , n ≥ 2, m ≥ 2, then fz denotes k × m matrix:


∂f1 /∂z1 , ∂f1 /∂z2 , · · · ∂f1 /∂zm


 ∂f2 /∂z1 , ∂f2 /∂z2 , · · · ∂f2 /∂zm 


fz = 
 = {∂fi /∂zj },
.
.
.
..
..
..


·
·
·


∂fk /∂z1 , ∂fk /∂z2 , · · · ∂fk /∂zm
(11)
Chapter 1 – p. 19/37
D IFFERENTIATING V ECTORS W. R . T. V ECTORS
CONT.
fy will denote the k × n matrix

∂f1 /∂y1 ∂f1 /∂y2 · · ·

 ∂f /∂y ∂f /∂y · · ·
1
2
2
 2
fy = 
..
..

.
···
.

∂fk /∂y1 ∂fk /∂y2 · · ·
∂f1 /∂yn


∂f2 /∂yn 

 = {∂fi /∂yj }.
..

.

∂fk /∂yn
(12)
Matrices fz and fy are known as Jacobian matrices. Note that
fz = fzT = fzT = fzTT ,
where by fzT we mean (f T )z and not (fz )T .
Chapter 1 – p. 20/37
D IFFERENTIATING V ECTORS W. R . T. V ECTORS
CONT.
Applying the rule (11) to Fy in (9) and the rule (12) to Fz in (10),
respectively, we obtain Fyz = (Fy )z to be the n × m matrix
Fyz

Fy1 z1 Fy1 z2 · · ·

 F
 y2 z1 Fy2 z2 · · ·
= .
..
 ..
.
···

Fyn z1 Fyn z2 · · ·
Fy1 zm
Fy2 zm
..
.
Fyn zm

 
2
∂ F

,
=

∂yi ∂zj

(13)
Chapter 1 – p. 21/37
D IFFERENTIATING V ECTORS W. R . T. V ECTORS
CONT.
and Fzy = (Fz )y to be the m × n matrix
Fzy

Fz1 y1
Fz1 y2
···

 F
 z2 y1 Fz2 y2 · · ·
= .
..
 ..
.
···

Fzm y1 Fzm y2 · · ·
Fz1 yn
Fz2 yn
..
.
Fzm yn

 
2
∂ F

.
=

∂zi ∂yj

(14)
Chapter 1 – p. 22/37
P RODUCT RULE FOR D IFFERENTIATION
• Let g be an n-component row vector function and f be an
n-component column vector function of an n-component
vector x, Then
(gf )x = gfx + f T gxT = gfx + f T gx .
(15)
If g = Fx , then
(gf )x =(Fx f )x =Fx fx + f T Fxx = Fx fx + (Fxx f )T .
(16)
Chapter 1 – p. 23/37
V ECTOR N ORM AND N EIGHBORHOOD
• The norm of an m-component row or column vector z is
defined to be
k z k=
q
2 .
z12 + · · · + zm
(17)
• Neighborhood Nz0 of a point, e.g.,
Nz0 = {z| k z − z0 k< ε} ,
(18)
where ε > 0 is a small positive real number.
Chapter 1 – p. 24/37
L ITTLE - O N OTATION AND N ORM OF A F UNCTION
• A function F (z) : E m → E 1 is said to be of the order o(z),
if
F (z)
= 0.
lim
kzk→0 k z k
• The norm of an m-dimensional row or column vector
function z(t), t ∈ [0, T ], is defined to be
Z
k z k= Σm
j=1
T
zj2 (τ )dτ
0
21
.
(19)
Chapter 1 – p. 25/37
S OME S PECIAL N OTATION
• The concepts of left and right limits
x(t− ) = lim x(t − ε) and x(t+ ) = lim x(t + ε).
ε↓0
ε↓0
(20)
• Discrete-time models (in Chapter 8-9)
xk : state variable at time k.
uk : control variable at time k.
λk : adjoint variables, respectively, at time k,
∆xk := xk+1 − xk : difference operator.
xk∗ and uk∗ : quantities along an optimal path.
Chapter 1 – p. 26/37
S OME S PECIAL N OTATION CONT.
• bang function
bang[b1 , b2 ; W ] =
• sat function




b1
if
W < 0,
undefined
if
W = 0,
b2
if
W > 0.



sat[y1 , y2 ; W ] =



 y1



if
W < y1 ,
W
if
y1 ≤ W ≤ y2 ,
y2
if
W > y2 .
(21)
(22)
Chapter 1 – p. 27/37
S OME S PECIAL N OTATION CONT.
Impulse Control
imp(x1 , x2 , t) = limε→0
Z
t+ε
F (x, u, τ )dτ.
(23)
t
If the impulse is applied only at time t, then we can calculate the
objective function as
Z t
J =
F (x, u, τ )dτ + imp(x1 , x2 , t)
(24)
0
+
Z
T
F (x, u, τ )dτ + S[x(T ), T ].
t
Chapter 1 – p. 28/37
C ONVEX S ET
• A set D ⊂ E n is a convex set if for each pair of points
y, z ∈ D, the entire line segment joining these two points is
also in D, i.e.,
py + (1 − p)z ∈ D, for each p ∈ [0, 1].
• Given xi ∈ E n , i = 1, 2, . . . , l, we define y ∈ E n to be a
convex combination of xi ∈ E n , if there exists pi ≥ 0 such
that
l
l
X
X
p i xi .
pi = 1 and y =
i=1
i=1
Chapter 1 – p. 29/37
C ONVEX H ULL
•
The convex hull of a set D ⊂ E n is
coD :=
( l
X
i=1
pi xi :
l
X
i=1
)
pi = 1, pi ≥ 0, xi ∈ D, i = 1, 2, . . . , l .
Chapter 1 – p. 30/37
C ONCAVE F UNCTION
• ψ : D → E 1 , is concave, if for each pair of points y, z ∈ D
and for all p ∈ [0, 1],
ψ(py + (1 − p)z) ≥ pψ(y) + (1 − p)ψ(z).
• If ≥ is changed to > for all y, z ∈ D with y 6= z, and
0 < p < 1, then ψ is called a strictly concave function.
• If ψ(x) is a differentiable function on the interval [a, b], then
it is concave, if for each pair of points y, z ∈ [a, b],
ψ(z) ≤ ψ(y) + ψx (y)(z − y).
Chapter 1 – p. 31/37
F IGURE 1.2 A C ONCAVE F UNCTION
Chapter 1 – p. 32/37
C ONCAVE AND C ONVEX F UNCTIONS
• If the function ψ is twice differentiable, then it is concave, if
at each point in [a, b],
ψxx ≤ 0.
In case x is a vector, ψxx needs to be a negative definite
matrix.
• If ψ : D → E 1 defined on a convex set D ⊂ E n is a concave
function, then the negative of the function ψ, i.e.,
−ψ : D → E 1 , is a convex function.
Chapter 1 – p. 33/37
A FFINE AND H OMOGENEOUS F UNCTIONS
• ψ : E n → E 1 is said to be affine, if ψ(x) − ψ(0) is linear.
ψ : E n → E 1 is said to be homogeneous of degree one, if
ψ(bx) = bψ(x), where b is a scalar constant.
• Saddle Point ψ : E n × E m → E 1 . For maxx miny , a point
(x̂, ŷ) ∈ E n × E m is called a saddle point of ψ(x, y), if
ψ(x, ŷ) ≤ ψ(x̂, ŷ) ≤ ψ(x̂, y) for all x ∈ E n and y ∈ E m .
Note also that
ψ(x̂, ŷ) = max ψ(x, ŷ) = min ψ(x̂, y).
x
y
Similarly, for minx maxy . Just reverse the inequalities.
Chapter 1 – p. 34/37
F IGURE 1.3 A S ADDLE P OINT
Chapter 1 – p. 35/37
L INEAR I NDEPENDENCE AND R ANK OF A M ATRIX
• A set of vectors a1 , a2 , . . . , an from E n is said to be linearly
dependent if there exist scalars pi not all zero such that
n
X
pi ai = 0.
(25)
i=1
• If the only set of pi for which (1.25) holds is
p1 = p2 = · · · = pn = 0, then the vectors are said to be
linearly independent.
Chapter 1 – p. 36/37
L INEAR I NDEPENDENCE AND R ANK OF A M ATRIX
CONT.
• The rank (or more precisely the column rank) of an m × n
matrix A, written rank(A), is the maximum number of
linearly independent columns in A.
• An m × n matrix is of full rank if
rank(A) = n.
Chapter 1 – p. 37/37