Advanced Macroeconomics I:
Deterministic Dynamic Programming
Krzysztof Makarski
1
Deterministic Dynamic Programming
Introduction
• Our goal is to show how to solve simple one sector growth model with value function iterations.
• It is a global method that allows to solve the increasing number of models (nonlinearities, asymmetry).
• Recursive formulation.
• Some math background.
• Example: finite horizon model.
• Value function iterations.
• Analytical example.
1.1
One sector growth model (Ramsey)
Households
Representative household offers labor supply to firms lts , for the real wage rate wt . They also own capital,
and make investment decisions xt 1 . Furthermore, households own firms and if there are profits Πt they are
transferred to households, nevertheless, since it is a competitive equilibrium profits are zero.
Households life infinitely and they have to choose the consumption path c̃ = (c0 , c1 , c2 , ..., ct , ...). Instantaneous utility (utility in period t) is given by the following function u (ct ) = log ct , which is increasing in
ct but its derivative is decreasing in ct (positive and decreasing marginal utility). Households are impatient,
thus from the point of view of period 0 utility in period t is discounted by a factor β < 1. Thus, the lifetime
utility is
U (c̃) = β 0 u(c0 ) + β 1 u(c1 ) + ... + β t u(ct ) + ...
which can be also expressed as
U (c̃) =
∞
X
β t u (ct )
t=0
Households in each period face the following budget constraint
ct + x t
=
wt lt + rt kt
kt+1
=
(1 − δ)kt + xt
where δ− depreciation rate. Next we can eliminate xt from the above equations, by substituting from the
firs equation into the second to get
ct + kt+1 = wt lt + (rt + 1 − δ) kt
1 This is different than in reality, but it simplifies the analysis and it can be shown that in a large class of models if firms
own capital the result is the same, and it does not matter who owns capital.
1
Now we can write down the household problem
∞
X
max
s s ∞
{ct ,kt ,lt }t=0
β t u (ct )
t=0
s
s.t. ct + kt+1
= wt lt + (rt + 1 − δ) kt
lt ∈ [0, 1] , kt > 0
Solving the consumer problem.
First, notice that since there is no disutility from labor (no labor in the utility function) our consumers
work all the time, which we normalize to be 1
lt = 1
Next we construct Lagrangian (denote as λt the Lagrangian multiplier on the period t budget constraint):
L =
(β 0 u (c0 ) + ...β t u( ct ) + β t+1 u (ct+1 ) + ...)
− [λ0 (c0 + k1 − w0 l0 − (r0 + 1 − δ)k0 ) + ...
+λt ( ct + kt+1 − wt lt − −(rt + 1 − δ)kt ) + ...
+ λt+1 (ct+1 + kt+2 − wt+1 lt+1 − (rt+1 + 1 − δ) kt+1 ) + ...
=
∞
X
∞
X
β t u (ct ) −
t=0
i
λt (ct + kt+1 − wt lt − (rt + 1 − δ)kt )
t=0
Next we compute the first order conditions (FOCs), i.e. derivatives of the Lagrangian w.r.t. ct i kt+1 )
and the transversality condition
Lct
=
β t u0 (t) − λt = 0
Lkt+1
=
−λt + λt+1 (rt+1 + 1 − δ) = 0
2
lim λt kt+1 = 0
t→∞
Simplifying
β t u0 (t)
=
λt
λt
=
λt+1 (rt+1 + 1 − δ)
lim λt kt+1
=
0
t→∞
Next we eliminate λs. Substituting from the first for λt = β t u0 (t) and λt+1 = β t+1 u0 (t + 1) we get
β t u0 (t)
=
β t+1 u0 (t + 1) (rt+1 + 1 − δ)
lim β t u0 (t) kt+1
=
0
t→∞
Rearranging
u0 (t)
= rt+1 + 1 − δ
βu0 (t + 1)
lim β t u0 (t) kt+1 = 0
t→∞
Henceforth from the consumer problem we get (1) - (3)
u0 (t)
βu0 (t + 1)
lt
t 0
lim β u (t) kt+1
t→∞
=
rt+1 + 1 − δ
(1)
=
1
(2)
=
0
(3)
Note that these equations are identical to (7) - (9).
2 Transversality condition will not be derived. It is a necessary condition and it excludes some of the dynamic paths of the
economy (for example it excludes such a path that we save everything and never consume, which obviously is not optimal).
2
Firms
Firms rent capital and labor to produce output. While making their decisions they seek the maximal
possible profits. Each firm is too small to impact prices, thus they take prices as given. In each period, the
representative firm solves the following problem
yt − rt kt − wt lt
max
(yt ,lt ,kt )
yt = Aktα lt1−α
s.t.
Solution of the firm problem
Firms problem
max yt − rt kt − wt lt
(yt ,lt ,kt )
p.w.yt = Aktα lt1−α
We construct Lagrangian
L = yt − rt kt − wt lt − ξt yt − Aktα lt1−α
FOCs
yt
:
1 = ξt
lt
: rt = ξt αAktα lt1−α
kt
: wt = ξt (1 − α) Aktα−1 lt−α
Substituting for ξt = 1 we get
rt
wt
= Aαktα−1 lt1−α
= A(1 −
α)ktα lt−α
(4)
(5)
and the production function
yt = Aktα lt1−α
(6)
Market clearing
We have the market clearing conditions for the labor market and the capital market which are introducing
into the definition by notation (namely there is the same l and k in both consumer and firm problem).
Furthermore, in the goods market demand must be equal to supply
ct + x t = y t
and since
kt+1 = (1 − δ)kt + xt
we get
ct + kt+1 = yt + (1 − δ)kt
Competitive equilibrium
Definition 1. A competitive equilibrium is an endogenous allocation {ct , lt , kt , yt }∞
t=0 and endogenous
prices {rt , wt }∞
satisfying
(given
exogenous
(k
,
A,
β,
δ))
0
t=0
• {ct , lt , kt+1 }∞
t=0 solves
P∞ the consumer problem given prices and k0 .
max{ct ,kt+1 ,lt } t=0 ln ct
s.t. ct + kt+1 = wt lt + (rt + 1 − δ)kt , lt ∈ [0, 1], k0 given
• for all t, (lt , kt , yt ) solves the producer problem given prices
max(yt ,lt ,kt ) yt − rt kt − wt lt
s.t. yt = Aktα lt1−α
• markets clear
ct + kt+1 = yt + (1 − δ)kt .
3
Solution
Equations describing the economy
• The consumer problem.
Euler
u0 (t)
= rt+1 + 1 − δ
βu0 (t + 1)
(7)
lt = 1
(8)
lim βu0 (t) kt+1 = 0
(9)
Labor
and the transversality condition (TVC)
t→∞
• The producer problem
=
Aαktα−1 lt1−α
(10)
wt
=
(11)
yt
=
A(1 − α)ktα lt−α
Aktα lt1−α
rt
(12)
• Market clearing
ct + kt+1 = Aktα lt1−α + (1 − δ)kt
(13)
Summarizing a competitive allocation is fully characterized by equations (7) - (13).
Properties
Next, we check whether a competitive allocation is efficient. First, we need to define what efficient means.
Definition 2. An allocation {ct , lt , kt }∞
t=0 is efficient if it solves the following benevolent social planner
problem
∞
X
max
{ct ,kt+1 ,lt }∞
t=0
s.t.
β t ln ct
t=0
ct + kt+1 = yt = Aktα lt1−α + (1 − δ)kt
lt ∈ [0, 1] , given k0 ∈
Since there is no disutility from labor it is optimal to
lt = 1
for all t.
Constructing Lagrangian, note that we have one feasibility constraint for each period:
max
{ct ,kt ,lt }∞
t=0
β 0 u(c0 ) + ...β t u(ct ) + β t+1 u(ct+1 ) + ...
s.t.
c0 + k1 = Ak0α + (1 − δ) k0
c1 + k2 = Ak1α + (1 − δ) k1
..
.
ct + kt+1 = Aktα lt1−α + (1 − δ) kt
1−α
α
ct+1 + kt+2 = Akt+1
lt+1
+ (1 − δ) kt+1
..
.
4
(14)
Lagrangian
L = (β 0 u (c0 ) + ...β t u( ct ) + β t+1 u (ct+1 ) + ...)
− [λ0 (c0 + k1 − Ak0α − (1 − δ) k0 ) + ...
+ λt ( ct + kt+1 − Aktα − (1 − δ) kt ) + ...
α
+λt+1 (ct+1 + kt+2 − A kt+1
− (1 − δ) kt+1 ) + ...
i
Or in a shorter version
L=
∞
X
β t u (ct ) −
t=0
∞
X
λt [ct + kt+1 − Aktα − (1 − δ) kt ]
t=0
Next, we can compute the FOCs (derivatives w.r.t. ct and kt+1 ).
= β t u0 (t) − λt = 0
Lct
α−1
= −λt + λt+1 Aαkt+1
+ (1 − δ) = 0
Lkt+1
and the transversality condition
lim λt kt+1 = 0
t→∞
Simplifying
β t u0 (t)
= λt
α−1
= λt+1 Aαkt+1
+ (1 − δ)
λt
lim λt kt+1
=
t→∞
0
Next, we eliminate λs. Substituting from the first for λt = β t u0 (t) and λt+1 = β t+1 u0 (t + 1) we get
α−1
β t u0 (t) = β t+1 u0 (t + 1) Aαkt+1
+ (1 − δ)
lim β t u0 (t) kt+1
=
t→∞
Rearranging
0
u0 (t)
α−1
= Aαkt+1
+ (1 − δ)
βu0 (t + 1)
(15)
lim β t u0 (t) kt+1 = 0
(16)
t→∞
Additionally, we have the production function
yt = Aktα
(17)
ct + kt+1 = yt + (1 − δ) kt
(18)
and the feasibility condition
Optimal allocation is fully characterized by equations (14) - (18). Note that these equations are identical to
(19) - (23).
Summarizing the optimal allocation is characterized by
lt
0
u (t)
βu0 (t + 1)
ct + kt+1
yt
0
lim βu (t) kt+1
t→∞
=
1
(19)
α−1
= Aαkt+1
+ (1 − δ)
(20)
= yt + (1 − δ)kt
(21)
= Aktα
(22)
=
(23)
Next we have
5
0
Theorem 1. First Welfare Theorem: Any competitive allocation is efficient.
Proof. First we simplify equations(7) - (13). From (7) - (8) and (10) we get
u0 (t)
α−1
= rt+1 + 1 − δ = Aαkt+1
+1−δ
βu0 (t + 1)
(24)
Thus competitive allocation satisfies equations (24), (8), (9), (12) and (13). Note, that those equations are
the same as equations characterizing an efficient allocation. Hence, any competitive allocation is efficient.
and
Theorem 2. Second Welfare Theorem: Any efficient allocation can be decentralized as a competitive allocation.
Proof. Consider an efficient allocation {ct , lt , kt , yt }∞
t=0 that satisfies equations (19) - (23). Construct prices
{wt , rt }∞
using
(10)
(11).
Then
the
efficient
allocation
and the prices is a competitive equilibrium, since
t=0
they satisfy equations (7) - (13).
Both theorems are important. The first one tells us that no resources are wasted in competitive equilibrium. And the second tells us that any efficient allocation can be decentralized as an efficient allocation.
Furthermore, they are technically useful, since one does not need to find an equilibrium directly, instead one
can find an efficient allocation first, and then prices using equations (10) - (11). Thus, one can just solve the
benevolent social planner problem to get an equilibrium.
Furthermore they reveal the informative role of the prices in the competitive economy. And any time the
price system is disturbed, it leads to an inefficient outcome.
1.2
Value function iterations
Recursive problem
• Consider the following problem
∞
X
max
{ct ,kt+1 ,lt }∞
t=0
β t u (ct )
t=0
s.t. ct + kt+1 = Aktα lt1−α + (1 − δ) kt
Since lt = 1
∞
X
max
{ct ,kt+1 }∞
t=0
β t u (ct )
t=0
s.t. ct + kt+1 = Aktα + (1 − δ) kt
• If we start in period t (instead of period 0) the problem becomes
∞
X
max
{ct+τ ,kt+1+τ }∞
τ =0
β τ u (ct+τ )
τ =0
α
s.t. ct+τ + kt+1+τ = Akt+τ
+ (1 − δ) kt+τ
Substituting out ct
V (kt ) =
max ∞
{kt+1+τ }τ =0
∞
X
α
β τ u(Akt+τ
+ (1 − δ) kt+τ − kt+1+τ )
τ =0
rearranging
V (kt ) = max β 0 u(Aktα + (1 − δ) kt − kt+1 ) +
kt+1
max ∞
{kt+1+τ }τ =1
∞
X
α
β τ u(Akt+τ
+
τ =1
+(1 − δ)kt+τ −kt+1+τ )
6
(25)
adjusting the indices (we start the second sum from 0 therefore we replace τ with τ + 1)
V (kt ) = max u(Aktα + (1 − δ) kt − kt+1 ) + β
kt+1
max ∞
{kt+2+τ }τ =0
∞
X
α
β τ u(Akt+1+τ
+
τ =0
+(1 − δ)kt+1+τ − kt+2+τ )
P∞ τ
α
using (25) we get V (kt+1 ) = max{kt+2+τ }∞
τ =0 β u(Akt+1+τ + (1 − δ) kt+1+τ − kt+2+τ ) substiτ =0
tuting above we get a Bellman equation
0
β u(Aktα + (1 − δ) kt − kt+1 ) + βV (kt+1 )
V (kt ) =
max
α
0≤kt+1 ≤Akt +(1−δ)kt
0
• Next (it the problem is stationary) we replace kt with k and kt+1 with k in order to express the
problem in recursive form
V (k) =
0≤k
0
0
0
[u(Ak α + (1 − δ) k − k ) + βV (k )]
max
≤Akα +(1−δ)k
0
0
• We can also simplify the notation further by replacing 0 ≤ k ≤ Ak α + (1 − δ) k with k ∈ Γ (k)
0
0
V (k) = 0max [u(Ak α + (1 − δ) k − k ) + βV (k )]
k ∈Γ(k)
where Γ(·) is a correspondence.
• In general form the Bellman equation takes the following form
V (x) = max [u (x, y) + βV (y)]
y∈Γ(x)
First Order Conditions
• Envelope theorem. Let function f (x, y) be differentiable with respect to both x and y. Suppose
that for all x there exists differentiable function F (x)
F (x) = max f (x, y)
y
and differentiable function g(x)
y = g(x) = arg max f (x, y)
y
then
0
F (x) = f1 (x, g(x))
• From
V (x) = max [u (x, y) + βV (y)]
y∈Γ(x)
0
– Euler equation (derivative w.r.t. k )
0
u0 (c) = βV 0 (k )
– From envelope
V 0 (k) = u0 (c) (αAk α−1 + 1 − δ)
– and TVC
0
lim β t V (kt )kt = 0
t→∞
0
0
• Substituting for V (k ) from the second equation into the first we get the standard Euler equation
0
0
α−1
u (ct ) = βu (ct+1 )(αAkt+1
+ 1 − δ)
7
Contraction mapping theorem
Definition 3. A metric space is a set S, together with a metric (distance function) d : S × S → R, such
that for all x, y, z ∈ S:
(i) d (x, y) ≥ 0, with equality iff x = y;
(ii) d (x, y) = d (y, x); and
(iii) d (x, y) ≤ d (x, z) + d (z, y) (triangle inequality).
Examples: The following are metric spaces
1. The set of integers, S with d (x, y) = |x − y|.
2. The set of integers S with d (x, y) = 0, if x = y and d (x, y) = 1, if x 6= y.
3. The set of strictly increasing functions on [a, b], S with d (x, y) = maxt∈[a,b] |x (t) − y (t)|.
Rb
4. The set of all continuous, strictly increasing functions on [a, b], S with d (x, y) = maxt∈[a,b] a |x (t) − y (t)| dt.
Definition 4. A metric space (S, d) is complete if every Cauchy sequence in S converges to an element in
S.
Examples:
1. Closed interval [a, b] with a metric d(x, y) = |x − y| is a complete metric space.
2. Open interval (a, b) with a metric d(x, y) = |x − y| is not a complete metric space.
3. R2 with a standard metric is a complete metric space.
Definition 5. A normed vector space is a vector space S, together with a norm k·k : S → R, such that for
all x, y ∈ S and α ∈ R: (i)kxk ≥ 0, with equality iff x = θ; (ii)kaxk = |a| kxk; and (iii)kx + yk ≤ kxk + kyk
(triangle inequality).
Examples.
The following spaces are normedPvector spaces.
l
(i) Let S = Rl , with kxk = [ i=1 x2i ]1/2 (Euclidean space).
(ii) Let S = Rl , with kxk = maxi |xi |.
Pl
(iii) Let S = Rl , with kxk = i=1 |xi |.
(iv) Let S be the set of all bounded infinite sequences of real numbers (x1 , x2 , ..., xk , ...) with kxk =
supk |xk |. (It is called l∞ )
(v) Let S be the set of all continuous functions on [a, b] with kxk = supa≤t≤b |x(t)|. (It is called C[a, b]).
FACT: Any normed vector space (S, k·k) is a metric space, by defining d (x, y) = kx − yk.
Definition 6. Banach space - complete normed vector space. It is a normed vector space in which any
Cauchy sequence converges to an element of this space.
FACT: Any normed vector space (S, k·k) is a metric space, to define metric one can use d(x, y) = kx − yk.
Examples:
1. Closed interval [a, b] with a norm kxk = |x| is a Banach space.
2. Open interval (a, b) with a norm kxk = |x| is not a Banach space.
P2
1
3. R2 with a norm kxk = [ i=1 x2i ] 2 is a Banach space.
4. C (X), space of bonded continuous functions f : X → R, where X ⊆ Rn with a norm kf k = supx∈X |f (x)|
is a Banach space (note, if X is compact, any function on X is bounded).
Definition 7. Let (S, d) be a metric space and T : S → S be a function mapping S into itself. T is a
contraction mapping (with modulus β) if for some β ∈ (0, 1), d (T x, T y) ≤ βd (x, y), for all x, y ∈ S.
Example: Consider a closed interval S = [a, b] with metric d (x, y) = |x − y|, then T : S → S is a contraction
mapping if for some β ∈ (0, 1)
|T x − T y|
≤ β < 1, for all x, y ∈ S where x 6= y
|x − y|
Exercise: Check if the following mappings are contraction
1. T x = 0, 2x + 0, 4.
2. T x = 1, 2x + 0, 4.
8
Definition 8. A fixed point of a mapping T : X → X is an element x ∈ X for which T x = x.
Example: Fixed points of T on a space S = [a, b] are given by the intersection of T x with 45o line. It is
easy to show that any contraction on [a, b] has unique fixed point (FIGURE).
Notation: Define iterations of T as T 0 x = x and T n x = T T n−1 x.
Theorem 3. (Contraction Mapping Theorem, Banach Theorem). If (S, d) is a complete metric
space and T : S → S is a contraction mapping with modulus β, than
(i) T unique fixed point v in S, and
(ii) for any V0 ∈ S, d(T n V0 , V ) ≤ β n d(V0 , V ) , n = 0, 1, 2, .....
Counterexample: Consider the following mapping T x = 0, 2x + 0, 8 on S = (0, 1). It does not have a fixed
point. (why?).
In order to use the Banach theorem for our problem, we need to show that T
(T V ) (x) = max [u (x, y) + βV (y)]
y∈Γ(x)
maps from a complete metric space into itself and that it is contraction (for details see Appendix):
1. From Theorem 1 follows that C(X), space of bounded continuous functions is complete.
2. Next we need to show that T maps the space of bounded continuous functions C(X) into itself (theorem
of maximum, Berge).
3. A useful way of verifying that certain operators are contractions is due to Blackwell.
Facts
• If we deal with bounded form above problem, then under certain conditions:
– There exists a unique value function V (·), which solves the Bellman equation.
– V (·) can be found by value function iterations.
– Under certain (stronger) assumptions (u(·) is strictly concave, strictly increasing, continuously
differentiable and Γ(·) is convex and monotone) V (·) is strictly concave, strictly increasing and
continuously differentiable.
0
– dynamics of k can be described by the policy function g(·), k = g(k).
Finite horizon Ramsey model
• Consider a model with finite time horizon t = 1, 2, ..., T . Let u(ct ) = ln ct and δ = 1 then the social
problem in period 0 has the following form
max
T
X
{ct ,kt+1 }T
t=0
β t u(ct )
t=0
s.t. ct + kt+1 = Aktα
and in period t
T
−t
X
max
−t
{ct+τ ,kt+1+τ ,lt+τ }T
τ =0 τ =0
β τ u(ct+τ )
1−α
α
s.t. ct+τ + kt+1+τ = Akt+τ
lt+τ
We can define Vt (kt )
Vt (kt ) ≡
T
−t
X
max
−t
{ct+τ ,kt+1+τ }T
τ =0 τ =0
β τ u(ct+τ )
1−α
α
s.t. ct+τ + kt+1+τ = Akt+τ
lt+τ
9
Rearranging
Vt (kt ) ≡

 max
(ct ,kt+1 )
s.t. c + k
α
t
t+1 = Akt 

 max
Vt (kt ) ≡



u(ct )+
(ct ,kt+1 )
β τ u(ct+τ )
−t
{ct+τ ,kt+1+τ }T
τ =0
τ =1
s.t.
α
kt+1+τ = Akt+τ



u(ct )+
T
−t
X
max
ct+τ +
max
−t
{ct+τ ,kt+1+τ }T
τ =0
s.t. c + k
α
t
t+1 = Akt 
s.t. ct+τ +
β·
T
−t
X
β τ −1 u(ct+τ )
τ =1
α
kt+1+τ = Akt+τ
Replacing indices τ with τ + 1
Vt,T (kt ) ≡

 max
(ct ,kt+1 )



u(ct ) + β· 
s.t. c + k
α
t
t+1 = Akt 

T −(t+1)
X
max
−t
{ct+τ ,kt+1+τ }T
τ =0
s.t. ct+1+τ +
β τ u(ct+1+τ )
τ =0
α
kt+1+1+τ = Akt+1+τ
Vt (kt ) ≡ max u(ct ) + βVt+1 (kt+1 )
(ct ,kt+1 )
s.t. ct + kt+1 = Aktα
• Solving by backward induction starting t = T
VT (kT ) ≡
max u(cT )
(cT ,kT +1 )
p.w. cT + kT +1 = AkTα
Since (it is straightforward) kT +1 = 0 then
VT (kT ) =u(AkTα )
Substituting u(c) = ln c
VT (kT ) = ln(AkTα ) = ln A + α ln kT
• Next in period t = T − 1. Period T − 1 problem
VT −1 (kT −1 ) ≡
max u(cT −1 ) + βVT (kT )
(cT −1 ,kT )
p.w. cT −1 + kT = AkTα−1
VT −1 (kT −1 ) ≡ max ln(AkTα−1 − kT ) + β(ln A + α ln kT )
kT
Derivative
1
−1
+ βα
=0
kT
− kT
AkTα−1
αβ
1
1
=
α
kT
AkT −1 − kT
αβ(AkTα−1 − kT ) = kT
10
αβAkTα−1 = kT + αβkT
kT =
αβ
Ak α
1 + αβ T −1
Substituting
VT −1 (kT −1 ) = ln(AkTα−1 − kT ) + β(ln A + α ln kT )
αβ
Ak α )
1 + αβ T −1
αβ
Ak α )
+ β(ln A + α ln
1 + αβ T −1
VT −1 (kT −1 ) = ln(AkTα−1 −
VT −1 (kT −1 ) = ln(
VT −1 (kT −1 ) = ln(
1
Ak α ) + β ln A+
1 + αβ T −1
αβ
+ αβ ln
Ak α
1 + αβ T −1
1
)+ ln AkTα−1 + β ln A+
1 + αβ
αβ
+ αβ ln AkTα−1
+ αβ ln
1 + αβ
1
) + β ln A+
1 + αβ
αβ
+ (1 + αβ) ln AkTα−1
+ αβ ln
1 + αβ
VT −1 (kT −1 ) = ln(
VT −1 (kT −1 ) = − ln(1 + αβ)+β ln A + (1 + αβ) ln A + αβ ln αβ−
− αβ ln(1 + αβ) + (1 + αβ)α ln kT −1
VT −1 (kT −1 ) =(1 + β + αβ) ln A + αβ ln αβ
− (1 + αβ) ln(1 + αβ) + α(1 + αβ) ln kT −1
Which can be expressed as
VT −1 (kT −1 ) =aT −1 + bT −1 ln kT −1
where
aT −1 =(1 + β + αβ) ln A + αβ ln αβ − (1 + αβ) ln(1 + αβ)
bT −1 =α(1 + αβ)
Since the value function in period T has the following form
VT (kT ) =aT + bT ln kT
where
aT = ln A
bT = α
we can conclude that in any iteration T − n it will also take the following form
Vt (kt ) =at + bt ln kt
11
• Next, we can find the policy function
Vt (kt ) ≡ max u(ct ) + βVt+1 (kt+1 )
(ct ,kt+1 )
p.w. ct + kt+1 = Aktα
Vt (kt ) ≡ max ln(ct ) + β(at+1 + bt+1 ln kt+1 )
(ct ,kt+1 )
p.w. ct + kt+1 = Aktα
Vt (kt ) ≡ max ln(Aktα − kt+1 ) + β(at+1 + bt+1 ln kt+1 )
kt+1
Derivative
−1
1
+ βbt+1
=0
Aktα − kt+1
kt+1
kt+1 = βbt+1 (Aktα − kt+1 )
and we get the policy function
βbt+1
Ak α
1 + βbt+1 t
kt+1 =
Special case
P∞
• We consider the special case of the Ramsey model with the following utility function t=0 β t ln ct ,
production function yt = Aktα and δ = 1. We solve a social planner problem using the Bellman
equation 3 .
• Optimal allocation {ct , kt+1 }∞
t=0 satisfies the social planner problem (given k0 )
∞
X
max
{ct ,kt+1 }∞
t=0
β t ln ct
t=0
s.t. ct + kt+1 = f (kt )
k0 dane
• Recursively it becomes
v(k) = max
ln c + βv(k 0 )
0
(c,k )
s.t. c + k 0 = Ak α
• Educated guess (one source of educated guess can be solution of the finite horizon case):
v(k) = a0 + a1 ln k
Substituting
v(k) = max
ln c + β(a0 + a1 ln k 0 )
0
(c,k )
p.w. c + k 0 = Ak α
eliminating c
v(k) = max
ln(Ak α − k 0 ) + β(a0 + a1 ln k 0 )
0
k
3 This
case can be solved analytically. Most problems can be only solved numerically.
12
(26)
FOCs:
−1
1
+ βa1 0
Ak α − k 0
k
1
βa1 0
k
=
=
0
1
− k0
Ak α
k 0 = βa1 (Ak α − k 0 )
Solving for k 0
βa1
Ak α
1 + βa1
k0 =
• Substituting into (26) we get
v(k) = ln(Ak α −
βa1
βa1
Ak α ) + β(a0 + a1 ln(
Ak α ))
1 + βa1
1 + βa1
rearranging
v(k)
=
v(k)
=
v(k)
=
βa1
1
+ βa0 + βa1 ln
+ ln Ak α + βa1 ln Ak α
1 + βa1
1 + βa1
βa1
1
+ βa0 + βa1 ln
+ (1 + βa1 ) ln Ak α
ln
1 + βa1
1 + βa1
βa1
1
+ βa0 + βa1 ln
+ (1 + βa1 ) ln A
ln
1 + βa1
1 + βa1
+(1 + βa1 )α ln k
ln
in order to find a0 and a1 we compare the above parameters to the parameters from v(k) = a0 + a1 ln k
1
βa1
v(k) = ln
+ βa0 + βa1 ln
+ (1 + βa1 ) ln A
1 + βa1
1 + βa1
{z
}
|
a0
+ (1 + βa1 )α ln k
|
{z
}
a1 ln k
which implies
a0
=
a1
=
1
βa1
+ βa0 + βa1 ln
+ (1 + βa1 ) ln A
1 + βa1
1 + βa1
(1 + βa1 )α
ln
From the second equation
a1 =
α
1 − βα
substituting to the first one
a0 = ln
α
β 1−βα
α
1
+
βa
+
β
ln
0
α
α
1 + β 1−βα
1 − βα 1 + β 1−βα
α
+ (1 + β
) ln A
1 − βα
a0 − βa0 = ln
1
1−βα+βα
1−βα
+(
α
+ βa0 + β
ln
1 − βα
βα
1−βα
1−βα+βα
1−βα
1 − βα + βα
) ln A
1 − βα
1
βα
1
a0 =
[ln(1 − βα) +
ln(βα) +
ln A]
1−β
1 − βα
1 − βα
13
substituting into the policy function
k0 =
βa1
Ak α = αβAk α
1 + βa1
(27)
Therefore we get the following value function (strictly increasing, strictly concave and continuously
differentiable)
v(k) =
1
βα
1
α
[ln(1 − βα) +
ln(βα) +
ln A] +
ln k
1−β
1 − βα
1 − βα
1 − βα
using feasibility
= Aktα
ct + kt+1
ct
=
(1 − αβ)Aktα
(28)
Solution is given by equation (27) - (28). Note that the savings rate is given by αβ.
Value function iterations
• Pick acceptable error ε.
• Discretize the state space k = {k1 , k2 , ..., ki , ..., kik } (use the steady state).
• Guess v0 (k). v0 (·) is a vector of length n.
• Find updated value function using equation
vn+1 (k) =
0<k
0
0
max
<Akα +(1−δ)k
• Compute the distance between functions
d(vn+1 , vn ) =
0
[u(Ak α + (1 − δ)k − k ) + βvn (k )]
X
|vn+1 (ki ) − vn (ki )|
i
or
d(vn+1 , vn ) = max |vn+1 (ki ) − vn (ki )|
i
• Iterate until d(vn+1 , vn ) ≤ ε.
Summary
• Solving the Ramsey problem using value function iterations.
• Recursive problem.
• It is a contraction.
• Finite horizon problem.
• Special case, which can solved analytically.
• Algorithm for value function iterations.
• It can be easily extended for the stochastic case.
14
Appendix: Fixed point and Contraction.
Math
Definition 1. A metric space is a set S, together with a metric (distance function) d : S × S → R, such
that for all x, y, z ∈ S:
(i)d (x, y) ≥ 0, with equality iff x = y;
(ii)d (x, y) = d (y, x); and
(iii)d (x, y) ≤ d (x, z) + d (z, y) (triangle inequality).
Examples.
Show that the following are metric spaces
(i) The set of integers S with d (x, y) = |x − y|.
(ii) The set of integers, S with d (x, y) = 0, if x = y and d (x, y) = 1, if x 6= y.
(iii) The set of strictly increasing functions on [a, b], S with d (x, y) = maxt∈[a,b] |x (t) − y (t)|.
Rb
(iv) The set of all continuous, strictly increasing functions on [a, b], S with d (x, y) = maxt∈[a,b] a |x (t) − y (t)| dt.
For vector spaces metrics are usually defined so that the distance between two points is equal to the
distance of their difference from zero point θ, d (x, y) = d (x − y, θ).
Definition 2. A normed vector space is a vector space S, together with a norm k·k : S → R, such that for
all x, y ∈ S and α ∈ R: (i)kxk ≥ 0, with equality iff x = θ; (ii)kaxk = |a| kxk; and (iii)kx + yk ≤ kxk + kyk
(triangle inequality).
Examples.
Show that the following spaces are
Pl normed vector spaces.
(i) Let S = Rl , with kxk = [ i=1 x2i ]1/2 (Euclidean space).
(ii) Let S = Rl , with kxk = maxi |xi |.
Pl
(iii) Let S = Rl , with kxk = i=1 |xi |.
(iv) Let S be the set of all bounded infinite sequences of real numbers (x1 , x2 , ..., xk , ...) with kxk =
supk |xk |. (It is called l∞ )
(v) Let S be the set of all continuous functions on [a, b] with kxk = supa≤t≤b |x(t)|. (It is called C[a, b]).
FACT: Any normed vector space (S, k·k) is a metric space, by defining d (x, y) = kx − yk.
Before we move on to the Banach theorem we first need to define.
∞
Definition 3. A sequence {xn }n=0 in S converges to x ∈ S, if for each ε > 0, there exists Nε , such that
d (xn, x) ≤ ε, for all n ≥ Nε
∞
Definition 4. A sequence {xn }n=0 in S is a Cauchy sequence if for each ε > 0, there exists Nε , such that
d (xn, xm ) ≤ ε, for all n, m ≥ Nε
Definition 5. A metric space (S, d) is complete if every Cauchy sequence in S converges to an element in
S.
For example [a, b] ∈ R is complete (with metric d (x, y) = |x − y|), but (a, b) is not (with the same metric).
Examples.
Consider the following spaces
(i) Let S be the set of integers with d (x, y) = |x − y|.
(ii) Let S be the set of integers with d (x, y) = 0, if x = y and d (x, y) = 1, if x 6= y.
(iii) Let S be the set of strictly increasing functions on [a, b], with d (x, y) = maxt∈[a,b] |x (t) − y (t)|.
(Hint: x (t) = 1 + t/n.)
15
(iv)Let S be the set of all continuous, strictly increasing functions on [a, b], with d (x, y) = maxt∈[a,b]
(Hint: same as above).
show that (i) − (ii) are complete and (iii) − (iv) are not complete.
Rb
a
|x (t) − y (t)| dt.
FACT. The set of real numbers R with the metric d (x, y) = |x − y| is a complete metric space.
Note a complete normed vector space is called a Banach space.
Theorem 1. Let X ⊆ Rn and let C (X) be the set of bounded continuous functions f : X → R with the sup
norm, kf k = supx∈X |f (x)|. Then C (X) is a complete normed vector space. (Note that if X is compact
then every continuous function is bounded, otherwise the restriction to bounded functions mus be added.)
We can now define a contraction mapping.
Definition 6. Let (S, d) be a metric space and T : S → S be a function mapping S into itself. T is a
contraction mapping (with modulus β) if for some β ∈ (0, 1), d (T x, T y) ≤ βd (x, y), for all x, y ∈ S.
Consider a closed interval S = [a, b] with d (x, y) = |x − y|, then T : S → S is a contraction if for some
β ∈ (0, 1)
|T x − T y|
≤ β < 1, for all x, y ∈ S with x 6= y
|x − y|
That is T is a contraction on S if it is uniformly less than one in absolute value. Note also that if T is
contraction on S then T is uniformly continuous on S = [a, b]. Therefore, the fixed points of T , the elements
of S satisfying T x = x, are the intersections of T x with the 45o line. It is easy to show that any contraction
on S = [a, b] has unique fixed point (FIGURE). This is much more general conclusion.
Before we state the Banach theorem define the iterates of T , as T 0 x = x and T n x = T T n−1 x .
Theorem 2. (Contraction Mapping Theorem, Banach Theorem). If (S, d) is a complete metric
space and T : S → S is a contraction mapping with modulus β, than
(i) T unique fixed point v in S, and
(ii) for any v0 ∈ S, d (T n v0 , v) ≤ β n d (v0 , v), n = 0, 1, 2, ....
Recall,
(T v) (x) = max [u (x, y) + βv (y)]
y∈Γ(x)
In order to use the Banach theorem for our problem, we need to show that T maps from a complete metric
space into itself and that it is contraction. First notice that from Theorem 1 follows that C(X), space of
bounded continuous functions is complete, next we need to show that T maps C(X) into itself and that T
is contraction. For the contraction part a useful way of verifying that certain operators are contractions is
due to Blackwell.
Theorem 3. (Blackwell’s sufficient conditions for contraction). Let X ⊆ Rn and let B (X) be a
space of bounded functions f : X → R, with the sump norm. Let T : B (X) → B (X) be an operator satisfying
(i) (monotonicity) f, g ∈ B (X) and f (x) ≤ g (x), for all x ∈ X, implies (T f ) (x) ≤ (T g) (x), for all x ∈ X.
(ii) (discounting) there exists some β ∈ (0, 1) such that
[T (f + a)] (x) ≤ (T f ) (x) + βa, all f ∈ B (X) , a > 0, x ∈ X
[Note, here (f + a) (x) is the function defined by (f + a) (x) = f (x) + a.] Then T is a contraction with
modulus β.
In order to show that T maps bounded continuous functions into itself we can use the Theorem of
Maximum. But, first we need to introduce the following terms.
Definition 7. A correspondence Γ: X → Y is a mapping which associates each x ∈ X with a subset Γ(x)
of Y . Two considerations are important here:
(i) A correspondence Γ(x) can be empty
(ii) A correspondence can be regarded as a function if Γ(x) consists of one element for each x ∈ X.
16
Next, Assume Γ is non-empty and compact valued (the set Γ(x) is non empty and compact for all x ∈ X.).
Definition 8. Upper Hemi Continuity (u.h.c.) at x: for any pair of sequences {xn } and {yn } with
xn → x and yn ∈ Γ(xn ) there exists a subsequence of {yn } that converges to a point y ∈ Γ(x).
Definition 9. Lower Hemi Continuity (l.h.c.) at x: for any sequence {xn } with xn → x and for every
y ∈ Γ(x) there exists a sequence {yn } with yn ∈ Γ(xn ) such that yn → y.
Examples.
Consider the following
correspondences:
{0, 1} , for x 6= 1
(i) Γ (x) =
(l.h.c. but not u.h.c)
 {0} , for x = 1
 {1} for x > 1
[0, 1] , for x 6= 1 (u.h.c. but not l.h.c).
(ii) Γ (x) =

{0} , for x < 1
Definition 10. Γ (x) is continuous at x if it is both upper and lower hemi continuous at x.
Define
h(x) = max f (x, y)
y∈Γ(x)
G(x)
=
arg max f (x, y)
y∈Γ(x)
Theorem 4. Theorem of Maximum (Berge) Let X ⊂ Rl and Y ⊂ Rm . Let f : X ×Y → R be continuous
and Γ : X → Y be compact-valued and continuous. Then h : X → R is continuous and G : X → Y is nonempty, compact valued, and u.h.c.
Let C (X) be a set of bounded and continuous functions C (X) = {f : X → R, f is continuous, and kf k ≡ supx∈X |f (x)| < ∞
where d(f, g) = kf − gk ≡ supx∈X |f (x) − g (x)|. Define T as before
(T v) (x) = max [u (x, y) + βv (y)]
y∈Γ(x)
Theorem 5. Suppose X is convex and Γ is nonempty, compact-valued and continuous. T maps the set of
bounded continuous functions C (X) into itself. Moreover T is a contraction.
Proof. First, T maps bounded continuous functions into itself follows from the Theorem of Maximum.
To check monotonicity, take w (x) ≤ v (x) for all values of x, then since u is the same in both cases (T w) (x) ≤
(T v) (x).
In order to check discounting
T (v + a) (x)
=
max [u (x, y) + β[v (y) + a]]
y∈Γ(x)
=
max [u (x, y) + β[v (y)]] + βa
y∈Γ(x)
=
(T v) (x) + βa
Thus we can now apply contraction mapping theorem and it is gives an unique value function that is
bounded and continuous. Furthermore, it gives us convergence through the iteration of value function.
Theorem 6. Suppose {fn (x, y)} and f (x, y) are concave in y that and Γ is convex and compact valued.
Then if fn → f in the sup-norm (uniformly). Define
gn (x)
=
arg max fn (x, y)
g(x)
=
arg max f (x, y)
y∈Γ(x)
y∈Γ(x)
then gn (x) → g(x) for all x (pointwise convergence); if X is compact then the convergence is uniform.
17
We need concavity of the Value function for convergence of the policy function. We can use the following
theorems.
Theorem 7. If in addition to the assumptions in the contraction mapping theorem we have that (i) u(·, y)
is increasing, and (ii) Γ (x) is monotone, i.e. Γ (x) ⊆ Γ(x0 ) for x ≤ x0 . Then, the unique fixed point v
satisfying v = T v is increasing. If u(·, y) is strictly increasing, so is v.
Theorem 8. If in addition to the assumptions in the contraction mapping theorem we have that (i) u is
strictly concave in (x, y), and (ii) Γ is convex, i.e. for all x, x0 ∈ X and λ ∈ [0, 1], if y ∈ Γ(x) and y 0 ∈ Γ(x0 )
we have λy 0 + (1 − λ)y ∈ Γ(λx0 + (1 − λ)x). Then the fixed point v satisfying v = T v is strictly concave and
an associated policy function g is a continuous, single-valued function.
With this we have both value function and a policy function that is single valued and converges. Thus,
we can move on to finding the value function on the computer. Finally, we can also have differentiability.
Theorem 9. If in addition to the assumptions in the contraction mapping theorem we have that (i) u is
strictly concave and differentiable in the interior of A = {(x, y) ∈ X × X; y ∈ Γ(x)}, and (ii) Γ is convex.
Then if x0 ∈ intX and g(x0 ) ∈ intΓ(x0 ), then the fixed point v satisfying v = T v is continuously differentiable
at x0 .
18