Optimal Control
Prof. Lutz Hendricks
Econ720
September 27, 2016
1 / 42
Topics
Optimal control is a method for solving dynamic optimization
problems in continuous time.
2 / 42
Generic Optimal control problem
Choose functions of time c (t) and k (t) so as to
Z T
max
v[k(t), c(t), t]dt
(1)
0
Constraints:
1. Law of motion of the state variable k (t):
k̇(t) = g[k(t), c(t), t]
(2)
2. Feasible set for control variable c (t):
c (t) ∈ Y (t)
(3)
3. Boundary conditions, such as:
k(0) = k0 ,given
(4)
k(T) ≥ kT
(5)
3 / 42
Generic Optimal control problem
I
c and k can be vectors.
I
Y (t) is a compact, nonempty set.
I
T could be infinite.
I
I
Then the boundary conditions change
Important: the state cannot jump; the control can.
4 / 42
Example
A household chooses optimal consumption to
Z T
u[c(t)]dt
(6)
k̇(t) = rk(t) − c(t)
(7)
c (t) ∈ [0,c̄]
(8)
k(0) = k0 ,given
(9)
k(T) ≥ 0
(10)
max
0
subject to
5 / 42
A Recipe for Solving Optimal Control
Problems
A Recipe
Step1. Write down the Hamiltonian
H(t) = v(k, c, t) + µ(t)g(k, c, t)
(11)
µ is essentially a Lagrange multiplier (called a “co-state variable).
Step 2. Derive the first order conditions which are necessary for
an optimum:
∂ H/∂ c = 0
(12)
∂ H/∂ k = −µ̇
(13)
7 / 42
A Recipe
Step 3. Impose the transversality condition:
I
I
for finite horizon:
µ (T) = 0
(14)
lim H(t) = 0
(15)
for infinite horizon:
t→∞
This depends on the terminal condition (see below).
8 / 42
A Recipe
Step 4. A solution is the a set of functions [c(t), k(t), µ(t)] which
satisfy
I
the FOCs
I
the law of motion for the state
I
the boundary / transversality conditions
9 / 42
Example: Growth Model
Z ∞
max
e−ρt u (c (t)) dt
(16)
0
subject to
k̇ (t) = f (k (t)) − c (t) − δ k (t)
(17)
k (0) given
(18)
10 / 42
Growth Model: Hamiltonian
H (k, c, µ) = e−ρt u (c (t)) + µ (t) [f (k (t)) − c (t) − δ k (t)]
(19)
Necessary conditions:
Hc = e−ρt u0 (c) − µ = 0
Hk = µ f 0 (k) − δ = −µ̇
11 / 42
Growth Model
Substitute out the co-state:
µ̇ = e−ρt u00 (c) ċ − ρ µ
ċ =
µ̇ + ρ µ
e−ρt u00 (c)
= − f 0 (k) − δ − ρ
(20)
(21)
u0 (c)
u00 (c)
(22)
Solution: ct , kt that solve Euler equation and resource constraint,
plus boundary conditions.
12 / 42
Details
First order conditions are necessary, not sufficient.
They are necessary only if we assume that
1. a continuous, interior solution exists;
2. the objective function v and the constraint function g are
continuously differentiable.
Acemoglu (2009), ch. 7, offers some insight into why the FOCs are
necessary.
13 / 42
Details
If there are multiple states and controls, simply write down one
FOC for each separately:
δ H/δ ci = 0
∂ H/∂ kj = −µ̇j
There is a large variety of cases depending on the length of the
horizon (finite or infinite) and the kinds of boundary conditions.
I
Each has its transversality condition (see Leonard and
Van Long 1992).
14 / 42
Next steps
Typical useful next things to do:
1. Eliminate µ from the system. Obtain two differential
equations in (c, k) .
2. Find the steady state by imposing ċ = k̇ = 0.
15 / 42
Sufficient conditions
First-order conditions are sufficient, if the programming problem is
concave.
This can be checked in various ways.
16 / 42
Sufficient conditions I
The objective function and the constraints are concave functions of
the controls and the states.
I
The co-state must be positive.
This condition is easy to check, but very stringent.
In the growth model:
I
u (c) is concave in c (and, trivially, k)
I
f (k) − δ k − c is concave in c and k
I
µ = u0 (c) > 0
17 / 42
Sufficient Conditions II
First-order conditions are sufficient, if
the Hamiltonian is concave in controls and states,
where the co-state is evaluated at the optimal level.
This, too is very stringent.
18 / 42
Sufficient Conditions III
Arrow and Kurz (1970)
First-order conditions are sufficient, if the maximized Hamiltonian
is concave in the states.
Maximized Hamiltonian:
Substitute controls and co-states out, so that the Hamiltonian is
only a function of the states.
This is less stringent and by far the most useful set of sufficient
conditions.
19 / 42
Discounting: Current value Hamiltonian
Problems with discounting
Current utility depends on time only through an exponential
discounting term e−ρt .
The generic discounted problem is
Z T
max
e−ρt v[k(t), c(t)]dt
(23)
0
subject to the same constraints as above.
20 / 42
Applying the Recipe
H (t) = eρt v (k, c) + µ̂g (k, c)
(24)
∂H
= 0 =⇒ e−ρt vc (kt, ct ) = −µ̂t gc (kt , ct )
∂ ct
(25)
∂H
= e−ρt vk (kt , ct ) + µ̂t gk (kt , ct ) = −µ̂˙ t
∂ kt
(26)
21 / 42
Applying the Recipe
Let
µt = eρt µ̂t
(27)
and multiply through by eρt :
vc (t) = −µt gk (t)
This is the standard FOC, but with µ instead of µ̂.
22 / 42
Applying the Recipe
vk (t) + eρt µ̂t gk (t) = −eρt µ̂˙ t
(28)
Substitute out µ̂˙ t using
µ̇t =
deρt µ̂t
= ρ µt + eρt µ̂˙ t
dt
we have
vk (t) + µt gk (t) = −µ̇t + ρ µt
This is the standard condition with an additional ρ µ term.
23 / 42
Shortcut
We now have a shortcut for discounted problems.
Hamiltonian (drop the discounting term):
H = v (k, c) + µg (k, c)
(29)
∂ H/∂ c = 0
(30)
∂ H/∂ k = µ(t)ρ − µ̇(t)
| {z }
added
(31)
lim e−ρT µ(T)k(T) = 0
(32)
FOCs:
and the TVC
T→∞
24 / 42
Equality constraints
Equality constraints of the form
h[c(t), k(t), t] = 0
(33)
are simply added to the Hamiltonian as in a Lagrangian problem:
H(t) = v(k, c, t) + µ(t)g(k, c, t) + λ (t)h(k, c, t)
(34)
FOCs are unchanged:
∂ H/∂ c = 0
∂ H/∂ k = −µ̇
For inequality constraints:
h (c, k, t) ≥ 0; λ h = 0
(35)
25 / 42
Transversality Conditions
Finite horizon: Scrap value problems
The horizon is T.
The objective function assigns a scrap value to the terminal state
variable: e−ρT φ (k(T)):
Z T
max
e−ρt v[k(t), c(t), t]dt + e−ρT φ (k(T))
(36)
0
Hamiltonian and FOCs: unchanged.
The TVC is
µ (T) = φ 0 (k (T))
(37)
Intuition: µ is the marginal value of the state k.
27 / 42
Scrap value examples
1. Household with bequest motive
Z T
U=
0
eρt u (c (t)) + V (kT )
(38)
with k̇ = w + rk − c.
2. Maximizing the present value of earnings
Z T
Y=
e−rt wh (t) [1 − l (t)]
(39)
0
subject to ḣ (t) = Ah (t)α l (t)β − δ h (t)
Scrap value is 0. TVC: µ (T) = 0.
28 / 42
Infinite horizon TVC
The finite horizon TVC with the boundary condition k (T) ≥ kT is
µ (T) = 0.
I
Intuition: capital has no value at the end of time.
But the infinite horizon boundary condition is NOT
limt→∞ µ (t) = 0.
The next example illustrates why.
29 / 42
Infinite horizon TVC: Example
Z ∞
max
[ln (c (t)) − ln (c∗ )] dt
0
subject to
k̇ (t) = k (t)α − c (t) − δ k (t)
k (0) = 1
lim k (t) ≥ 0
t→∞
c∗ is the max steady state (golden rule) consumption.
No discounting - subtracting c∗ makes utility finite.
30 / 42
Infinite horizon TVC
Hamiltonian
H (k, c, λ ) = ln c − ln c∗ + λ [kα − c − δ k]
(40)
Necessary FOCs
Hc = 1/c − λ = 0
Hk = λ αkα−1 − δ = −λ˙
(41)
(42)
31 / 42
Infinite horizon TVC
We show: limt→∞ c (t) = c∗ [why?]
Limiting steady state solves
λ˙ /λ
= αkα−1 − δ = 0
k̇ = kα − 1/λ − δ k = 0
Solution is the golden rule:
k∗ = (α/δ )1/(1−α)
(43)
Verify that this max’s steady state consumption.
32 / 42
Infinite horizon TVC
Implications for the TVC...
λ (t) = 1/c (t) implies limt→∞ λ (t) = 1/c∗ .
Therefore, neither λ (t) nor λ (t) k (t) converge to 0.
The correct TVC: limt→∞ H (t) = 0.
The only reason why the standard TVC does not work: there is no
discounting in the example.
33 / 42
Infinite horizon TVC: Discounting
With discounting, the TVC is easier to check.
Assume:
the objective function is e−ρt v [k (t) , c (t)]
it only depends on t through the discount factor
v and g are weakly monotone
Then the TVC becomes
lim e−ρt µ (t) k (t) = 0
t→∞
(44)
where µ is the costate of the current value Hamiltonian.
This is exactly analogous to the discrete time version
lim β t u0 (ct ) kt = 0
t→∞
(45)
34 / 42
Example: renewable resource
Example: Renewable resource
Z ∞
max
e−ρt u (y (t)) dt
(46)
0
subject to
(47)
ẋ (t) = −y (t)
(48)
x (0) = 1
(49)
x (t) ≥ 0
(50)
36 / 42
Example: Renewable resource
Current value Hamiltonian
Necessary FOCs
37 / 42
Example: Renewable resource
FOC
Therefore:
µ (t) = µ (0) eρt
(51)
y (t) = u0−1 [µ (0) eρt ]
(52)
38 / 42
Solution
The optimal path has lim x (t) = 0 or
Z ∞
Z ∞
y (t) dt =
0
u0−1 [µ (0) eρt ] dt = 1
(53)
0
This solves for µ (0).
39 / 42
Example: Renewable resource
TVC for infinite horizon case:
lim e−ρt µ (0) eρt x (t) = 0
(54)
lim x (t) = 0
(55)
Equivalent to
40 / 42
Reading
I
Acemoglu (2009), ch. 7. Proves the Theorems of Optimal
Control.
I
Barro and Martin (1995), appendix.
I
Leonard and Van Long (1992): A fairly comprehensive
treatment. Contains many variations on boundary conditions.
41 / 42
References I
Acemoglu, D. (2009): Introduction to modern economic growth,
MIT Press.
Arrow, K. J. and M. Kurz (1970): “Optimal growth with irreversible
investment in a Ramsey model,” Econometrica: Journal of the
Econometric Society, 331–344.
Barro, R. and S.-i. Martin (1995): “X., 1995. Economic growth,”
Boston, MA.
Leonard, D. and N. Van Long (1992): Optimal control theory and
static optimization in economics, Cambridge University Press.
42 / 42