Lecture IV
Value Function Iteration with Discretization
Gianluca Violante
New York University
Quantitative Macroeconomics
G. Violante, ”VFI with Discretization”
p. 1 /13
Stochastic growth model with leisure
• Consider the planner’s problem:
V (k, z)
=
s.t.
c + k′ =
c ≥
k′ ≥
h ∈
max
u (c) + v (1 − h) + β
′
c,k ,h
X
π (z, z ′ ) V (k ′ , z ′ )
z ′ ∈Z
ez f (k, h) + (1 − δ) k
0
0
[0, 1]
• We want to solve for the policy functions gc (z, k), gk (z, k), gh (z, k)
• We do it by VFI + discretization of the state space (Z, K)
• V is a function defined only on grid points (a 2 × 2 matrix)
G. Violante, ”VFI with Discretization”
p. 2 /13
Setting up the grids
• Grid of Nz points {z1 , ..., zS } for productivity shock (e.g., Tauchen)
• Evenly spaced grid of Nk points for capital:
ki = k1 + (i − 1) η
, for i = 1, ...Nk
where
η=
k Nk − k 1
Nk − 1
but better: more points where decision rules have curvature
• How do we choose bounds {k1 , kNk }? Solve for SS system:
1
−1+δ
β
fk (k ∗ , 1 − h∗ )
=
c∗ + δk ∗
=
f (k ∗ , h∗ )
uc (c∗ ) fh (k ∗ , h∗ )
=
−vh (1 − h∗ )
• {k1 , kNk } must be below and above k ⋆ . Grid size depends on the
size and persistence of z. Keep k1 > 0.
G. Violante, ”VFI with Discretization”
p. 3 /13
Dealing with choice of leisure
• At every (ki , kj , zs ) corresponding to a grid point for the variables
(k, k ′ , z) we have the intratemporal FOC:
uc (ezs f (ki , h) + (1 − δ) ki − kj ) · ezs fh (ki , h) = −vh (1 − h)
• Note that the LHS is decreasing in h and the RHS is increasing
• h = 0 ruled out by Inada condition on f and h = 1 ruled out by
Inada condition on v. Interior solution between (0, 1).
• Use bisection method to solve for h
• Call the solution g̃h (ki , kj , zs ), but not yet a policy function
because it depends on k ′
G. Violante, ”VFI with Discretization”
p. 4 /13
Algorithm
1. Define a three dimensional array R of size (Nk , Nk , Nz ) with
typical element (ki , kj , zs ) containing the return function
R (ki , kj , zs ) = u (ezs f (ki ) + (1 − δ) ki − kj )+v (1 − g̃h (ki , kj , zs ))
Check whether the argument of u at point (i, j, s) is negative: if so,
set R (ki , kj , zs ) to a very large negative number.
2. Start with a guess of the (Nk , Nz ) matrix V 0 , say the null array so
the first iteration is a static problem. Or
∗
∗
u
(z
f
(k
,
h
)
−
δk
)
+
v
(1
−
h
)
s
i
i
0
V (ki , zs ) =
1−β
Denote the iteration number by t
3. You enter each iteration t with an (Nk , Nz ) matrix V t with generic
element V t (kj , zs )
G. Violante, ”VFI with Discretization”
p. 5 /13
Algorithm
4. Compute the (Nk , Nz ) matrix Vt that represents the conditional
expected value with generic element
Vt (kj , zs ) =
S
X
π (zs , zs′ ) V t (kj , zs′ )
s′ =1
5. Update value function by selecting
V
t+1
t
(ki , zs ) = max R (ki , kj , zs ) + βV (kj , zs )
j
Careful here! With the max operator you are squeezing a 3-dim
array into a 2-dim array. In Matlab use command called squeeze
Exploit monotonicity of the policy function kj (ki , zs) when
searching for the max over successive grid points!
G. Violante, ”VFI with Discretization”
p. 6 /13
Algorithm
6. Store the arg max, i.e., the decision rules:
gkt+1 (ki , zs )
=
ght+1 (ki , zs )
=
gct+1 (ki , zs )
=
arg max R (ki , kj , zs ) + βVt (kj , zs )
j
t+1
g̃h ki , gk (ki , zs ) , zs
t+1
zs
e f k i , gh
+ (1 − δ) ki − gkt+1 (ki , zs )
7. Howard improvement step here (see later)
8. Check convergence. If ||V t+1 − V t || < ε, stop and report success,
otherwise go back to step [3.] and update V t with V t+1
G. Violante, ”VFI with Discretization”
p. 7 /13
Additional checks
1. Check that the policy function isn’t constrained by the discrete
state space. If gk is equal to the highest or lowest value of capital
in the grid for some i, relax the bounds of the grid for k and redo
the value function iteration. Especially likely for upper bound.
2. Check that the error tolerance is small enough. If a small
reduction in ε results in large changes in the value or policy
functions, the tolerance is too high. Reduce ε until the solution to
the value and policy functions are insensitive to further reductions.
3. Check that grid size is dense enough. If an increase in number of
grid points results in a substantially different solution, the grid
might be too sparse. Keep increasing grid size until the value and
policy functions are insensitive to further increases.
Discretization is considerably slower compared with other methods that
you will learn (curse of dimensionality), but it’s the most robust method
G. Violante, ”VFI with Discretization”
p. 8 /13
Howard Improvement Step: idea
• Selecting maximizer in step [5.] to compute the policy functions is
the most time-consuming step in the value function iteration
• Howard’s improvement reduces the number of times we update
the policy function relative to the number of times we update the
value function
• Idea: on some iterations, we simply use the current approximation
to the policy function to update the value function, i.e. we do not
update the policy function at each VFI
• Updating the value function using the current estimate of the
policy function will bring the value function closer to the true value
since the policy function tends to converge faster than the value
function
G. Violante, ”VFI with Discretization”
p. 9 /13
Why VFI slower than PFI?
• Convergence rate of VFI (in infinite horizon problems)
approximately β and convergence (from any initial condition) is
assured if it is a contraction mapping
• Den Haan example: consider the problem
max
x1 ,x2
x1−γ
+ x1−γ
1
2
s.t.
x1 + x2 = 2
with solution
x1
x2
γ
=1
• At {x∗1 = x∗2 = 1}, M RS ∗ = 1 and V ∗ = 2
• Assume γ = 0.9, consider a huge deviation: {x̂1 = 2, x̂2 = 0}
• M ˆRS = ∞ but V̂ = 1.87, so V comparatively much flatter
G. Violante, ”VFI with Discretization”
p. 10 /13
Howard Improvement Step: implementation
• Let V0t+1 (ki , zs ) = V t+1 (ki , zs ) and iterate h = 1, ..., H times:
t+1
(ki , zs )
Vh+1
=R
ki , gkt+1
(ki , zs ) , zs +β
Nz
X
π (zs , zs′ ) Vht+1
gkt+1
(ki , zs ) , zs′
s′ =1
Note: gk stays the same at each iteration h
• Need to choose H. Too high H may result in a value function
moving further from the true one since the policy function is not
the optimal policy.
1. A good idea is to increase H after each iteration
2. Or, use the HIS only after a few steps of the VFI
3. Or stop when ||Vh+1 − Vh || improvement is small enough
G. Violante, ”VFI with Discretization”
p. 11 /13
Policy function iteration
1. At iteration t, gkt is given. Need an initial guess of policy: get
smart!
2.a Solve the system of Nk × Nz equations for each element of V t :
t
V (ki , zs ) = R
t
ki , gkt (ki , zs ), zs
+β
S
X
π (zs , zs′ ) V
t
gkt (ki , zs ), zs′
s′ =1
2.b Iterate h = 1, ... until convergence of V t (slower than 2.a but more
stable).
t
Vh+1
(ki , zs ) = R
t
ki , gkt (ki , zs ), zs
+β
S
X
π (zs , zs′ ) Vht
s′ =1
gkt (ki , zs ), zs′
This is basically an HIS algorithm with H = ∞
G. Violante, ”VFI with Discretization”
p. 12 /13
Policy function iteration
3 Update the policy function:
(
gkt+1 (ki , zs ) = arg max R (ki , kj , zs ) + β
j
S
X
s′ =1
π (zs , zs′ ) V t (kj , zs′ )
)
4. Go back to step 2. with new policy
G. Violante, ”VFI with Discretization”
p. 13 /13
Lecture V
Linear-Quadratic Approximations
Gianluca Violante
New York University
Quantitative Macroeconomics
G. Violante, ”LQ Approximations”
p. 1 /15
LQ approximation methods
• The LQ method locally approximates the period utility function
around the steady state using a quadratic function
• If u is (approximated as) quadratic, the value function is quadratic,
too. The decision rules are linear in the state variables.
• LQ approximation and the method of linearizing FOCs around the
steady state are equivalent. In both methods, the optimal decision
rules are going to be linear in state variables
• Since both methods rely on the local approximation, both methods
are valid only locally around the steady state of the model
• The method is easy to implement where the welfare theorems
hold and can solve the Social Planner’s problem. For solving
equilibrium of wider class of economies with distortions,
linearization of FOCs is simpler.
G. Violante, ”LQ Approximations”
p. 2 /15
Stochastic growth model with leisure
V (z, k) = max
u (c, 1 − h) + β
′
c,k ,h
X
π (z, z ′ ) V (k ′ , z ′ )
z ′ ∈Z
where:
h
i1−γ
cθ (1 − h)1−θ
u (c, 1 − h)
=
y
=
ez k α h1−α
y
=
c + k ′ − (1 − δ) k
1−γ
z follows an AR(1) with mean zero
G. Violante, ”LQ Approximations”
p. 3 /15
Steps for LQ Approximation
1. Solve for the steady state
2. Identify the endogenous and exogenous states, and the choice
variables
3. Redefine the utility as a function of (endogenous and exogenous)
state variables and choice variables
4. Then, approximate the utility function around the steady state,
using a 2nd order Taylor approximation
5. Use value function iteration to find the optimal value function
G. Violante, ”LQ Approximations”
p. 4 /15
Step 1: Solve for SS
k̄, c̄, h̄
1. Euler equation:
1 = β 1 + αk̄
α−1 1−α
h̄
−δ
2. Intratemporal FOC:
θ 1 − h̄ · (1 − α) k̄ α h̄−α
=
(1 − θ) c̄
3. From the resource constraint:
c̄ + δ k̄ = k̄ α h̄1−α
substituting c̄ out using the last two equations: (2+3) →
α 1−α
α −α
− δ k̄
θ 1 − h̄ · (1 − α) k̄ h̄ = (1 − θ) k̄ h̄
2 equations in 2 unknowns (k̄, h̄). Use root finding method to find SS
G. Violante, ”LQ Approximations”
p. 5 /15
Steps 2/3: Identify variables and redefine u
• Exogenous state variables: z
• Endogenous state variables: k
• (2 × 1) vector of control/decision variables: d = (k ′ , h)
• Eliminate c from utility function as u (z, k, d) by substituting
aggregate resource constraint
• Compute the number ū ≡ u z̄, k̄, d¯ , period utulity evaluated at SS
• Define the (4 × 1) vector:

z
z


  k
w= k = ′
 k
d
h

G. Violante, ”LQ Approximations”






p. 6 /15
Steps 4: Approximate u with quadratic function
• Using a second order approximation of u around the SS:
1
u (z, k, d) ≃ ū + (w − w̄)T J¯ + (w − w̄)T H̄ (w − w̄)
2
¯
¯
where J and H̄ are Jacobian and Hessian evaluated at z̄, k̄, d .
u (z, k, d)
≃
=
=
=
1
ū + (w − w̄)T J¯ + (w − w̄)T H̄ (w − w̄)
2
1
1 T
T ¯
T ¯
ū − w̄ J + w̄ H̄ w̄ + w J − H̄ w̄ + wH̄w
2
2



T
h
i
ū − w̄T J¯ + 12 w̄T H̄ w̄ 21 J¯ − H̄ w̄
1
T




1 w
1 ¯
1
w
H̄
J − H̄ w̄
2
2


h
i
1

1 wT Q 
w
A quadratic form, where the matrix of coefficients Q is (5 × 5) .
G. Violante, ”LQ Approximations”
p. 7 /15
Step 5: VFI
• We can write the VFI at iteration t as:
"
(
#
)
i
h
X
1
T
Q
Vt+1 (z, k) = max
+β
π (z, z ′ ) Vt (z ′ , k ′ )
1 w
d
w
z′
• We know the value function has the same quadratic form as u.
Define the vector of states:


1


s= z 
k
• We postulate a quadratic form:
Vt (z, k) = sT Pt s
where P is negative semi-definite and symmetric. Need to find P!
G. Violante, ”LQ Approximations”
p. 8 /15
Step 5: VFI
• The approximated Bellman equation looks like
T
s Pt+1 s = max
d
(
h
1
w
T
i
Q
"
1
w
#
h
T
+ βE (s′ ) Pt s′
)
i
where the law of motion for the state can be written as a function
of the vector w as:


1


 




0
1 0 0 0 0  z 
1
 

 


s′ =  z ′  =  0 ρ 0 0 0   k  +  ε′ 
 ′ 
′
0
k
0 0 0 1 0  k 
h


"
#
0
1


s′ = B
+  ε′ 
w
0
G. Violante, ”LQ Approximations”
p. 9 /15
Step 5: VFI
• This is useful because we can rewrite:
T
s Pt+1 s
=
max
d
h
i
Q
1
w


+β
h
1
i
wT
i

B T Pt B 
1
w



0


′

+βE 0 ε′ 0 Pt 
 ε 
0



h
i
h
1
T


max
Q
+β 1
1 w
d 
w
h
=
1
wT

wT
i

B T Pt B 
1
w

 + βσε2 Pt,22



• You can see the certainty equivalence property here: the FOC wrt
d (a component of w) will not depend on σε
G. Violante, ”LQ Approximations”
p. 10 /15
Step 5: VFI
• Collecting terms
sT Pt+1 s = max
d
h
1 w
T
i
[Q + Mt ]
"
1
w
#
where




Mt = βB T Pt B + 


G. Violante, ”LQ Approximations”
βσε2 Pt,22
0
...
0
...
0
...
...

0
... 





0
p. 11 /15
Step 5: VFI
• Let’s rewrite the Bellman equation as follows:
T
s Pt+1 s
=
max
d
h
sT
dT
i
("
Qss
Qsd
QTsd
Qdd
#
+
"
Mt,ss
Mt,sd
T
Mt,sd
Mt,dd
#) "
s
d
#
• Recall that s is a (3 × 1) vector of constant plus states and d a
(2 × 1) vector of decisions
• The Q and Mt matrices are (5 × 5). Qs· and Mt,s· are (2 × 3) and
Qdd and Mt,dd are (2 × 2)
• Thus, multiplying through:
sT Pt+1 s = max sT (Qss + Mt,ss ) s + 2dT (Qsd + Mt,sd ) s
d
+
G. Violante, ”LQ Approximations”
dT (Qdd + Mt,dd ) d
p. 12 /15
Step 5: VFI
• It is a concave program, so the FOC is sufficient
• The FOC wrt d yields:
0 =
2 (Qsd + Mt,sd ) s + 2 (Qdd + Mt,dd ) d
d =
− (Qdd + Mt,dd )
d =
ΩTt s
−1
(Qsd + Mt,sd ) s
where Ω is (3 × 2) given by:
Ωt = −
G. Violante, ”LQ Approximations”
QTsd
+
T
Mt,sd
(Qdd + Mt,dd )−1
p. 13 /15
Step 5: VFI
• If we use the FOC to substitute out d in the Bellman equation:
sT Pt+1 s = sT (Qss + Mt,ss ) s + 2sT Ωt Qsd + Mt,sd s + sT Ωt Qdd + Mt,dd ΩT
t s
• Using the expression for Ωt :
h
i
−1
T
T
Qw̃d + Mt,sd s
s Pt+1 s = s Qss + Mt,ss − Qsd + Mt,sd Qdd + Mt,dd
T
T
• This suggests a recursion for V :
1. Given Pt (needed to construct Mt ):
Pt+1
=
Qss + Mt,ss −
QTsd
+
T
Mt,sd
−1
(Qdd + Mt,dd )
(Qsd + Mt,sd )
2. Use Pt+1 to construct Mt+1 and go back to step 1 til convergence
G. Violante, ”LQ Approximations”
p. 14 /15
Summary of LQ-VFI procedure
1. Guess P0 . Since the value function is concave, we guess a
negative semidefinite matrix, for example P0 = −I.
2. Given Pt , update the value function using recursion above and
obtain Pt+1 . The Q matrix is defined by approximating the return
function and the Mt matrix by the formula we obtained above
3. Compare Pt and Pt+1 . If the distance (measured in sup norm) is
smaller than the predetermined tolerance level, stop. Otherwise
go back to the updating step (step 2) with Pt+1 .
4. With the optimal P ∗ , we can compute the decision rules. Check
solution is correct by evaluating decision rules at SS.
5. Use the decision rules to simulate the economy.
G. Violante, ”LQ Approximations”
p. 15 /15