Discrete Choice Models
Dynamic Programming Models
Discrete-Choice Dynamic-Programming Models
Estimating Discrete Choice Dynamic
Programming Models
Katsumi Shimotsu1
1 Department
2 School
Ken Yamada2
of Economics, Hitotsubashi University
of Economics, Singapore Management University
Japanese Economic Association Spring Meeting
Tutorial Session (May, 2011)
References
Discrete Choice Models
Dynamic Programming Models
Discrete-Choice Dynamic-Programming Models
Introduction to Part II
• The second part focuses on econometric implementation of
discrete choice dynamic programming models.
• A simple machine replacement model is estimated using the
nested fixed point algorithm (Rust, 1987).
• For more applications, see Aguirregabiria and Mira (2010) and
Keane, Todd, and Wolpin (2010).
• For audience with different backgrounds, I will briefly review
1. Discrete choice models and
2. Numerical methods for (i) maximum likelihood estimation and
(ii) dynamic programming.
References
Discrete Choice Models
Dynamic Programming Models
Discrete-Choice Dynamic-Programming Models
Outline
Discrete Choice Models
The Random Utility Model
Maximum Likelihood Estimation
Dynamic Programming Models
Machine Replacement
Numerical Dynamic Programming
Discrete-Choice Dynamic-Programming Models
The Nested Fixed Point Algorithm
The Nested Pseudo Likelihood Algorithm
References
References
Discrete Choice Models
Dynamic Programming Models
Discrete-Choice Dynamic-Programming Models
Outline
Discrete Choice Models
The Random Utility Model
Maximum Likelihood Estimation
Dynamic Programming Models
Machine Replacement
Numerical Dynamic Programming
Discrete-Choice Dynamic-Programming Models
The Nested Fixed Point Algorithm
The Nested Pseudo Likelihood Algorithm
References
References
Discrete Choice Models
Dynamic Programming Models
Discrete-Choice Dynamic-Programming Models
References
The Random Utility Model
• Consider a static problem in which the agent chooses among J
alternatives, such as transportation mode and brand choice.
• Assume that the choice-specific utility can be expressed as
Vij = uij + εij
i = 1, . . . , N, j = 0 . . . , J,
where uij is a deterministic component while εij is a stochastic
component.
• A simple example is uij = xij θ , where xij is a vector of observed
characteristics such as price and income.
• Let A = {1, . . . , J} denote the choice set. The choice probability
is
Z
P (a) =
(
)
1 a = arg max (uij + εij ) f (εi ) dεi ,
ε
j∈A
where f (εi ) is a joint density of εi = (εi1 , . . . , εiJ )0 .
Discrete Choice Models
Dynamic Programming Models
Discrete-Choice Dynamic-Programming Models
The Conditional Logit Model
• Assume that each εj is independently, identically distributed
extreme value. The density is f (εj ) = e −εj exp (−e −εj ). The
choice probability follows the logit model (McFadden, 1981).
pij = P (a = j) =
exp (uij )
J
∑j=1 exp (uij )
=
exp (uij − ui1 )
1 + ∑Jj=2 exp (uij − ui1 )
• Note that this is a system of J − 1 equations. For J = 2,
pi2 =
exp (ui2 − ui1 )
= Λ (ui2 − ui1 ) .
1 + exp (ui2 − ui1 )
There exists a mapping between the utility differences and
choice probabilities.
pi2
−1
ui2 − ui1 = Λ (pi2 )
or
ui2 − ui1 = ln
1 − pi2
See Hotz and Miller (1993) for the details on invertibility.
References
Discrete Choice Models
Dynamic Programming Models
Discrete-Choice Dynamic-Programming Models
Outline
Discrete Choice Models
The Random Utility Model
Maximum Likelihood Estimation
Dynamic Programming Models
Machine Replacement
Numerical Dynamic Programming
Discrete-Choice Dynamic-Programming Models
The Nested Fixed Point Algorithm
The Nested Pseudo Likelihood Algorithm
References
References
Discrete Choice Models
Discrete-Choice Dynamic-Programming Models
Dynamic Programming Models
MLE
• The choice probability is
P (a = j) =
exp (uij )
J
∑j=1 exp (uij )
• Suppose Vij = xij θ + εij . The log-likelihood function is
N
l (θ ) =
N
J
∑ li (θ ) = ∑ ∑ 1 (ai = j) ln P ( ai = j| xij ) .
i=1 j=1
i=1
• If the model is correctly specified,
d
θb → N
−1 !
∂ 2l
θ,E −
.
∂θ∂θ0
References
Discrete Choice Models
Dynamic Programming Models
Discrete-Choice Dynamic-Programming Models
Numerical Maximization (Train, 2009)
References
Discrete Choice Models
Dynamic Programming Models
Discrete-Choice Dynamic-Programming Models
Numerical Maximization (Train, 2009)
References
Discrete Choice Models
Dynamic Programming Models
Discrete-Choice Dynamic-Programming Models
Numerical Methods for MLE
• The Newton-Raphson procedure uses the following formula.
θk+1 = θk + −Hk−1 gk ,
where gk = ∂ l/ ∂ θ |θk and Hk = ∂ 2 l ∂ θ ∂ θ 0 θ .
k
• Quasi-Newton methods:
[fval, theta, g, H] = csminwel (0 fun0 , theta0, H0, g, crit, nit) by
Christopher Sims
• Non-gradient based algorithm (simplex methods):
[theta, fval] = fminsearch (0 fun0 , theta0, options)
References
Discrete Choice Models
Dynamic Programming Models
Discrete-Choice Dynamic-Programming Models
References
Extension to Dynamic Models
• We may be concerned with dynamics in some applications such
as firm’s adjustment of labor and capital, educational and
occupational choices.
• The value function V is the unique solution to the Bellman
equation:
V (xt , εt ) = max {Vjt = V (xt , εt , at = j)} ,
j∈A
where the choice-specific value function (including ε) is
V (xt , εt , at ) = u (xt , at ) + εt (at ) + β E [ V (xt+1 , εt+1 )| xt , εt , at ]
= {u (xt , at ) + β E [ V (xt+1 , εt+1 )| xt , εt , at ]} + εt (at )
= v (xt , at ) + εt (at )
for β ∈ (0, 1). The model considered here is a natural extension
of the conditional logit model (or the random utility model) to a
dynamic setting.
Discrete Choice Models
Dynamic Programming Models
Discrete-Choice Dynamic-Programming Models
Outline
Discrete Choice Models
The Random Utility Model
Maximum Likelihood Estimation
Dynamic Programming Models
Machine Replacement
Numerical Dynamic Programming
Discrete-Choice Dynamic-Programming Models
The Nested Fixed Point Algorithm
The Nested Pseudo Likelihood Algorithm
References
References
Discrete Choice Models
Dynamic Programming Models
Discrete-Choice Dynamic-Programming Models
References
Plant Investment Decisions
• Here we consider the plant’s investment decision problem set out
by Kasahara at UBC. The tradeoff is that investment (machine
replacement), a ∈ {0, 1},
1. requires large fixed costs for now but
2. increases profits and lowers replacement costs in the future.
• The profit function is
u (zt , at−1 , at ) = R (zt , at ) − FC (at , at−1 ) ,
where R is revenues net of variable input costs, zt is plant’s
productivity, and FC is fixed costs.
• Assume that zt = ρzt−1 + ηt , where ηt ∼ N 0, ση2 . Specify
the profit function as
u (zt , at−1 , at ) = u (zt−1 , at−1 , at )
= exp (θ0 + θ1 zt−1 + θ2 at ) − [θ3 at + θ4 (1 − at−1 ) at ] .
Discrete Choice Models
Dynamic Programming Models
Discrete-Choice Dynamic-Programming Models
References
Transition Probability
• Let Z = {z1 , . . . , zg , . . . , zG }.
• Denote the transition probability by
qgh = q ( zt = zh | zt−1 = zg ) = Pr ( z ∈ [zh , zh+1 ]| z ∈ [zg , zg+1 ]) .
By the Tauchen method, this can be approximated by

Φ ( (z 1 − ρzg )/ ση )
if h = 1,

Φ ( (z h − ρxg )/ ση ) − Φ ( (z h−1 − ρzg )/ ση ) if 1 < h < G,
qgh =

1 − Φ ( (z G − ρzg )/ ση )
if h = G,
where z g = (zg + zg+1 )/ 2 and Φ is the standard normal cdf.
Discrete Choice Models
Discrete-Choice Dynamic-Programming Models
Dynamic Programming Models
References
The Choice-Specific Value Function
• Define the integrated value function as
Z
V (zt−1 , at−1 , εt ) dF (εt ) .
V (zt−1 , at−1 ) =
εt
• Assume that each εt (at ) is iid extreme value.
!
J
V (zt−1 , at−1 ) = γ + ln
∑ exp (v (zt−1 , at−1 , at = j))
,
j=1
where γ is the Euler constant.
• The choice-specific value function is
v (zt−1 , at−1 , at ) = u (zt−1 , at−1 , at )+β ∑ q ( zt | zt−1 ) V (zt−1 , at−1 ) .
xt
Discrete Choice Models
Dynamic Programming Models
Discrete-Choice Dynamic-Programming Models
Outline
Discrete Choice Models
The Random Utility Model
Maximum Likelihood Estimation
Dynamic Programming Models
Machine Replacement
Numerical Dynamic Programming
Discrete-Choice Dynamic-Programming Models
The Nested Fixed Point Algorithm
The Nested Pseudo Likelihood Algorithm
References
References
Discrete Choice Models
Dynamic Programming Models
Discrete-Choice Dynamic-Programming Models
References
Infinite Horizon Problem
• In the infinite-horizon problem with a finite number of states, the
Bellman equation is a system of nonlinear equations.
#
"
G
Vg = max u (xg , a) + β
a∈A
∑ qgh (a) Vh
,
h=1
where a ∈ {1, . . . , J} is a choice variable,
x ∈ {x1 , . . . , xg , , . . . , xG } is a state variable, and qgh (a) is the
transition probability from state g to state h.
• Let a = (a1 , . . . , aG )0 denote the policy function and
ua = (u (x1 , a1 ) , . . . , u (xG , aG ))0 the return. Then, in vector
notation, Va = ua + β Qa Va , which leads to the solution:
Va = (I − β Qa )−1 ua ,
where the ghth element of Qa is qgh (a). For details, see Judd
(1998) and Adda and Cooper (2003).
Discrete Choice Models
Dynamic Programming Models
Discrete-Choice Dynamic-Programming Models
Value Iteration Algorithm
1. Choose a grid X = {x1 , . . . , xg , , . . . , xG }, where xg < xh for
g < h, and specify u (x , a) and qgh (a).
2. Make an initial guess V 0 and choose stopping criterion c > 0.
3. For g = 1, . . . , G, compute
(
Vgk+1
= max u (xg , a) + β
a∈A
)
G
∑
qgh (a) Vhk
h=1
4. If V k − V k+1 < c, stop; else go to step 3.
.
References
Discrete Choice Models
Dynamic Programming Models
Discrete-Choice Dynamic-Programming Models
Policy Iteration Algorithm
1. Choose a grid X = {x1 , . . . , xg , , . . . , xG }, where xg < xh for
g < h, and specify u (x , a) and qgh (a).
2. Make an initial guess V 0 and choose stopping criterion c > 0.
0
3. Compute ak+1 = a1k+1 , . . . , aGk+1 , where
(
agk+1 = arg max u (xg , a) + β
a∈A
G
∑ qgh (a) Vhk
h=1
0
4. Compute uk+1 = u x1 , a1k+1 , . . . , u xG , aGk+1 .
5. Compute
k+1 −1 k+1
Vk+1 = I − β Qa
u .
6. If V k − V k+1 < c, stop; else go to step 3.
)
.
References
Discrete Choice Models
Dynamic Programming Models
Discrete-Choice Dynamic-Programming Models
Outline
Discrete Choice Models
The Random Utility Model
Maximum Likelihood Estimation
Dynamic Programming Models
Machine Replacement
Numerical Dynamic Programming
Discrete-Choice Dynamic-Programming Models
The Nested Fixed Point Algorithm
The Nested Pseudo Likelihood Algorithm
References
References
Discrete Choice Models
Dynamic Programming Models
Discrete-Choice Dynamic-Programming Models
The Nested Fixed Point Algorithm (Rust, 1987)
• The conditional choice probability takes the form:
P ( a| x ) =
exp (u (x , a, θ ) + β EV (x 0 , ε 0 ))
.
∑Jj=1 exp (u (x , j, θ ) + β EV (x 0 , ε 0 ))
• The transition probability can be obtained separately.
1. Inner loop: Given θ , compute the value function V using the
value iteration algorithm or the policy iteration algorithm, and
obtain the conditional choice probability P.
2. Outer loop: Given P, obtain a pseudo-likelihood estimate of θ
using the Newton method.
3. Repeat steps 1 and 2 until convergence.
References
Discrete Choice Models
Dynamic Programming Models
Discrete-Choice Dynamic-Programming Models
Outline
Discrete Choice Models
The Random Utility Model
Maximum Likelihood Estimation
Dynamic Programming Models
Machine Replacement
Numerical Dynamic Programming
Discrete-Choice Dynamic-Programming Models
The Nested Fixed Point Algorithm
The Nested Pseudo Likelihood Algorithm
References
References
Discrete Choice Models
Dynamic Programming Models
Discrete-Choice Dynamic-Programming Models
References
Hotz and Miller (1993) and Aguirregabiria and Mira (2002)
• Denote the optimal decision rule by djto = 1 { at = j| xt , εt }.
(
V (xt ) = E
T
)
J
∑ ∑ β τ−t djto [u (xt , at ) + εt (at )]
τ=t j=1
The Bellman equation can be rewritten as
"
0
V (x ) = ∑ P ( a| x ) u (x , a) + E [ ε| x , a] + β ∑ q x x , a V x
0
#
x0
a
• The system of equations can be solved for the value function. In
matrix notation,
−1
V = I − β ∑ P (a) ◦ Q (a)
∑ P (a)◦[u (a) + (γ − ln P (a))] ,
a
a
where ◦ is the element-by-element product.
.
Discrete-Choice Dynamic-Programming Models
Dynamic Programming Models
Discrete Choice Models
The Conditional Logit Form
• Suppose u (x , a) = z (x , a) α. The policy iteration operator is
Ψ (P) =
exp (e
z (x , a) α + e
e (x , a))
,
+e
e (x , a))
z (x , a) α
∑Jj=1 exp (e
where
e
z (x , a) = z (x , a) + β ∑ q x 0 x , a Wz x 0 , P ,
0
x 0
e
e (x , a) = β ∑ q x x , a We x 0 , P ,
x0
#−1
"
Wz x 0 , P
=
I −β ∑P ◦Q
j
∑ P ◦ z,
a
−1
0
We x , P = I − β ∑ P ◦ Q
∑ P ◦ (γ − ln (P)) .
a
a
References
Discrete Choice Models
Discrete-Choice Dynamic-Programming Models
Dynamic Programming Models
References
The Nested Pseudo Likelihood Algorithm
• Make an initial guess P 0 and choose the number of iterations K .
1. Given e
z (x , a) = z (x , a) + β ∑x 0 q ( x 0 | x ,a) Wz x 0 , P k−1 and
e
e (x , a) = β ∑x 0 q ( x 0 | x , a) We x 0 , P k−1 , obtain a
pseudo-likelihood estimate of α k as
N
ak = arg max ∑ Ψ P k−1 .
α
i=1
2. Given ak , update P using
P k = Ψ P k−1 .
3. Iterate steps 1 and 2 until k = K .
Discrete Choice Models
Dynamic Programming Models
Discrete-Choice Dynamic-Programming Models
References
References
1.
Adda, J. and R. Cooper, (2003): Dynamic Economics: Quantitative Methods and
Applications, MIT Press.
2.
Aguirregabiria, V. and P. Mira, (2002): “Swapping the nested fixed point algorithm: A
class of estimators for discrete Markov decision models,” Econometrica.
3.
Aguirregabiria, V. and P. Mira, (2010): “Dynamic discrete choice structural models: A
survey,” Journal of Econometrics.
4.
Hotz, J. and R. A. Miller, (1993). “Conditional choice probabilities and the estimation of
dynamic models,” Review of Economic Studies.
5.
6.
Judd, K. L. (1998): Numerical Methods in Economics, MIT Press.
7.
McFadden, D. L. (1981): “Structural discrete probability models derived from theories of
choice,” in C. F. Manski and D. L. McFadden (eds.) Structural Analysis of Discrete Data
and Economic Applications, MIT Press.
8.
Rust, J. (1987): “Optimal replacement of GMC bus engines: An empirical model of
Harold Zurcher” Econometrica.
9.
Train, K. (2009): Discrete Choice Methods with Simulation, Cambridge University
Press.
Keane, M. P., P. E. Todd, and K. I. Wolpin, (2010). “The structural estimation of
behavioral models: discrete choice dynamic programming methods and applications,”
Handbook of Labor Economics, Vol. 4A.