The Empirical Implications of Models
with Multiple Equilibria
Alberto Bisin, Andrea Moro and Giorgio Topa§
April 2007
§ Bisin:
NYU; Moro & Topa: FRBNY. Many thanks to Chris Huckfeldt for
his excellent research assistance.
1
General Motivation
• Large variety of economic models have multiple equilibria.
— Social interaction models
— Search models with endogenous intensity
— Discrete choice models in I/O and urban economics
— Growth models with externalities
• Wish to examine conditions under which ‘deep’ model parameters are identified, in a general framework that encompasses
several concrete applications
• Focus on consistent, “easy to compute” estimator
2
Existing Literature
— Jovanovic (1989) - general condition for identification
Specific Settings:
— Bresnahan and Reiss (1991) — Tamer (2001)
(discrete games - multiplicity can help with identification)
— Dagsvik and Jovanovic (1994) (a repeated static model)
— Brock and Durlauf (2001) (social interactions)
— Aguirregabiria-Mira (2007)
— Bajari-Hong-Ryan (2006); Bajari-Hong-Krainer-Nekipelov (2006)
— Moro (2002) (human capital/ income inequality)
3
The Setup
• Agent i preferences:
³
´
i
i
−i
0
0
U x ,x ;θ ,u
xi ∈ X compact; θ0 preference parameters; u0 is a random
vector (note: allows for strategic interactions across agents).
• Agent i constraint set:
xi ∈ X i(p, x−i; θ00, u00)
p ∈ P is a vector of endogenous variables (e.g. prices); θ00 is
a vector of parameters; u00 is a random vector.
θ ≡ [θ0, θ00] ∈ Θ
u ≡ [u0, u00] ∈ U ∼ pdf f (·)
4
... under some regularity conditions the solution to the agent’s
problem is represented by a continuous function mapping (p, x−i)
into xi,
³
´
i
i
−i
x = x p, x ; θ, u
³
´
i
i
−i
x(p, x; θ, u) = the composition mapping, from x = x p, x ; θ, u ,
over i.
5
Definition: An equilibrium of the economy is a vector π ≡ (p, x)
such that
F (p, x(p, x; θ, u)) = 0
— Let F denote a vector valued mapping defined on π ≡ (p, x).
— With some extra regularity and dimensionality assumptions, the
equilibrium can be represented in general as a map from (θ, u) into
π which has the property of a smooth manifold.
— From now on we write the equilibrium condition as:
F (π; θ, u) = 0
6
Observations
• Let y ∈ Y denote a vector of observable variables.
• We assume there exists a map g from π, θ and random vector
v into y.
v ∼ h(v) is a vector of disturbances affecting the observations
but not the equilibrium (for example, observation errors)
• Fix θ ∈ Θ and a realization v; the map g(π, θ; v) is smooth,
one-to-one, and onto in π.
7
θ1, u1
Fundamentals, Shocks
θ2, u2
¡@
¡
@
¡
@
¡
@
ª
¡
R
@
Equilibria
Obs. error
π1
6
?
π2
@
@
@
@
@
@
R
@
v1
v2
π3
v3
?
¡
¡
ª
v ∼ h, u ∼ f
8
F (π; θ, u) = 0
¡
¡
¡
¡
¡
y
Data
θ(π, u)
g(π, θ, v)
π6
Multiple equilibria and global identification
-
(θ, u)
π6
Unique equilibrium and no identification
-
(θ, u)
π6
Multiple equilibria, no identification
-
(θ, u)
9
Identification
Let L(·| θ) be the set of probability distributions over the realization of y given θ (a correspondence).
The parameter vector θ0 is identified if for all θ ∈ Θ, θ 6= θ0,
∀ l ∈ L(· | θ0),
l∈
/ L(· | θ1).
Sufficient Condition
(for generic f, and given π)
θ(π, u) is single valued in a subset (possibly strict) of its domain,
and not defined everywhere else. In other words, for any (π, u),
either one and only one value of θ exists such that π is an equilibrium, or none.
10
A simple example: Global Interactions
in a City (Brock/Durlauf)
• I agents in a single city (model can be extended to N cities).
• Agents are characterized by a scalar characteristic Xi, observed by the econometrician.
• Agents choose an outcome yi ∈ {−1, 1}, as a solution of
max V (yi, Xi, E(y−i), εi(yi)) = h(Xi)·yi+
yi
X
Jij yiE(yj )+εi(yi).
j6=i
J ,
• We assume “global” interactions within each city, Jij = I−1
and a linear specification for h(Xi):
h(Xi) = cXi.
11
• The individual errors εi are assumed extreme value distributed
which implies
1
Pr (εi(−1) − εi(1) ≤ z) =
;
1 + exp(−βz)
• The probability that agent i chooses yi is
Pr (yi|Xi, Yn, E(y−i)) = Pr (V (yi, ·) > V (−yi, ·))
Ã
!
cXi · yi + JyiE(y−i) + εi(yi) >
.
= Pr
−cXi · yi − JyiE(y−i) + εi(−yi)
• Given the logistic specification of the individual error terms,
Pr (yi|Xi, Yn, E(y−i)) ∼ exp (cXi · yi + JyiE(y−i))
12
• Since the random utility terms are independent across individuals, one obtains
Pr (y|X, E(y−i)) =
Y
i
Y
i
Pr (yi|Xi, E(y−i)) ∼
exp (cXi · yi + JyiE(y−i)) .
• The equilibrium average choice π is determined by
π=
Z
tanh (cX + Jπ) dF (X).
13
Direct estimator:
1) Fix a candidate value for θ ≡ [ c, J]
2) calculate all equilibria {π k }K
k=1 and compute Pr (y | X, π k ; θ)
[this gives us L(y|π, θ, u)]
3) maximize over equilibria [this gives us L (y| θ)]
4) maximize over θ
14
Two-step estimator, first step:
1) Fix a value for θ ≡ [c, J] and for π
2) compute Pr (y | X, π; θ) [this gives us L (y |π, θ)]
3) maximize over θ and π.
b = samNote: here L (y |π, θ) = L (y |π) independent of θ, and π
ple mean of y.
15
Two-step estimator, second step:
1) fix a value for θ
b ; θ) [this gives us L (y |π
b , θ)]
2) compute Pr (y | X, π
3) maximize over θ.
16
Montecarlo experiment
• We draw N random values for θtrue
≡
j
n
o
ctrue
, Jjtrue ;
j
³
´
true
true
(always choos• For each draw θj , we generate data ye θj
ing medium equilibrium);
³
´
, we estimate
• For each artificial dataset ye θtrue
j
17
½
¾
2s
d
θbj , θbj .
Results
• Two-step estimation procedure is much faster than the Direct
method;
2s
• Two-step estimator θb is also more accurate than the direct
d
one θb :
µ
b2s
MSE θ
¶
µ
bd
< M SE θ
¶
• BUT: distinguish efficiency from computational issues.
18
Number of draws in Montecarlo Experiment:
160
Number of Cities: 1
Summary Statistics: True Parameters
C
Minimum
Maximum
Mean
-0.52608
0.50371
0.02641
J
1.23269
2.99158
2.36880
Probability of Landing on Different Equilbria: P1=0, P2=1, P3=0
Estimator Comparison
direct method
RMSE,
Bias,
RMSE,
Bias,
C
C
J
J
min time
max time
mean time
median time
two-step method
direct w/ 2s estimates
0.08220
0.00601
0.78321
0.24166
0.00808
0.00005
0.68947
0.10600
0.00810
0.00010
0.68514
0.09644
158.67007
212.67449
188.24622
188.36760
0.18436
0.30282
0.25126
0.25601
158.04850
218.42600
187.85952
188.70405
| J, direct
All simulations
RMSE
| 0.78321
Bias
| 0.24166
no. of simulations | (160)
| J, two-step
| J, direct+2s
| 0.68947
| 0.10600
| (160)
| 0.68514
| 0.09644
| (160)
Within 2 Standard Deviations
RMSE
| 0.39216
Bias
| 0.15905
no. of simulations | (157)
| 0.18550
| 0.03237
| (158)
| 0.16853
| 0.02269
| (158)
Within 1 Standard Deviation
RMSE
| 0.27176
Bias
| 0.09944
no. of simulations | (149)
| 0.10922
| 0.01232
| (155)
| 0.12550
| 0.01379
| (157)
Within .5 Standard
RMSE
Bias
no. of simulations
| 0.09858
| 0.00854
| (154)
| 0.09743
| 0.00494
| (155)
Within .25 Standard Deviations
RMSE
| 0.23035
Bias
| 0.07128
no. of simulations | ( 30)
| 0.08845
| 0.00838
| (152)
| 0.08976
| 0.00952
| (153)
Within .15 Standard Deviations
RMSE
| 0.28790
Bias
| 0.10165
no. of simulations | ( 13)
| 0.12805
| 0.01767
| ( 62)
| 0.11957
| 0.01805
| ( 64)
Deviations
| 0.15330
| 0.03829
| (137)
Number of draws in Montecarlo Experiment:
160
Number of Cities: 1
Summary Statistics: True Parameters
C
Minimum
Maximum
Mean
-0.52608
0.50371
0.02641
J
1.23269
2.99158
2.36880
Probability of Landing on Different Equilbria: P1=1, P2=0, P3=0
Estimator Comparison
direct method
RMSE,
Bias,
RMSE,
Bias,
C
C
J
J
min time
max time
mean time
median time
two-step method
direct w/ 2s estimates
0.05220
-0.00144
0.11604
-0.00749
0.05043
-0.00251
0.10708
-0.00542
0.05043
-0.00251
0.10706
-0.00390
163.37621
220.05349
188.47448
188.14973
0.22234
0.34216
0.26496
0.26686
157.19373
217.32402
187.49023
187.09089
| J, direct
All simulations
RMSE
| 0.11604
Bias
| -0.00749
no. of simulations | (160)
| J, two-step
| J, direct+2s
| 0.10708
| -0.00542
| (160)
| 0.10706
| -0.00390
| (160)
Within 2 Standard Deviations
RMSE
| 0.08954
Bias
| 0.00120
no. of simulations | (153)
| 0.08526
| -0.00179
| (153)
| 0.08521
| -0.00025
| (153)
Within 1 Standard Deviation
RMSE
| 0.07002
Bias
| 0.00446
no. of simulations | (138)
| 0.06823
| 0.00260
| (138)
| 0.06734
| 0.00527
| (137)
Within .5 Standard
RMSE
Bias
no. of simulations
| 0.07553
| 0.00852
| ( 69)
| 0.07530
| 0.00929
| ( 67)
Within .25 Standard Deviations
RMSE
| 0.07962
Bias
| 0.00350
no. of simulations | ( 39)
| 0.07661
| 0.00044
| ( 36)
| 0.07805
| -0.00158
| ( 37)
Within .15 Standard Deviations
RMSE
| 0.08078
Bias
| 0.00857
no. of simulations | ( 24)
| 0.07617
| 0.01727
| ( 23)
| 0.07642
| 0.00348
| ( 23)
Deviations
| 0.08028
| 0.01469
| ( 73)
Error in two-step estimation of C
20
18
16
14
12
10
8
6
4
2
0
-0.25
-0.2
-0.15
-0.1
-0.05
0
0.05
0.1
0.15
0.2
0.25
0.1
0.15
0.2
0.25
Error in direct estimation of C
60
50
40
30
20
10
0
-0.25
-0.2
-0.15
-0.1
-0.05
0
0.05
Error in two-step estimation of C
16
14
12
10
8
6
4
2
0
-0.02
-0.015
-0.01
-0.005
0
0.005
0.01
0.015
0.02
0.015
0.02
Error in direct estimation of C - without "runaway" runs
18
16
14
12
10
8
6
4
2
0
-0.02
-0.015
-0.01
-0.005
0
0.005
0.01
Error in two-step estimation of J
45
40
35
30
25
20
15
10
5
0
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
2.5
1
1.5
2
2.5
Error in direct estimation of J
45
40
35
30
25
20
15
10
5
0
-2
-1.5
-1
-0.5
0
0.5
EXAMPLE OF FAILURE OF DIRECT METHOD
_______________________________________________________________________________
true_CC
error-2s
error-di
0.2224
0.1007
0.0156
0.0105
0.3743
0.0442
0.1469
0.1283
0.2081
-0.0021
0.0002
-0.0059
-0.0124
0.0045
-0.0063
0.0174
-0.0001
0.0058
-0.0027
-0.0011
-0.0060
-0.0111
0.1538
-0.0063
0.0176
-0.0005
0.0058
true_JJ
error-2s
error-di
%% run that gets “stuck” on high eq.m
2.5834
-0.0177
-0.0164
1.9890
-0.0095
-0.0164
2.4235
-0.0029
0.0579
1.3605
-0.4753
-0.4182
2.4398
0.0142
0.5226 %% run that gets “stuck” on high eq.m
2.0817
-0.0755
-0.0707
2.1188
0.1651
0.1695
2.0555
-0.0261
-0.0213
2.2368
0.0497
0.0517
_______________________________________________________________________________
true_CC
twostep
direct
0.2224
0.1007
0.0156
0.0105
0.3743
0.0442
0.1469
0.1283
0.2081
0.2246
0.1005
0.0215
0.0228
0.3698
0.0505
0.1295
0.1284
0.2023
0.2251
0.1018
0.0216
0.0215
0.2205
0.0506
0.1293
0.1288
0.2024
true_JJ
twostep
direct
2.5834
1.9890
2.4235
1.3605
2.4398
2.0817
2.1188
2.0555
2.2368
2.6011
1.9985
2.4264
1.8358
2.4256
2.1571
1.9537
2.0816
2.1871
2.5998
2.0054
2.3655
1.7787
1.9173
2.1524
1.9492
2.0768
2.1851
%% run that gets “stuck” on high eq.m
%% run that gets “stuck” on high eq.m
EXAMPLE OF FAILURE OF DIRECT METHOD
______
True parameter values:
trueC =
trueJ =
0.4554
2.7991
Equilibria given these parameters:
eqa =
-0.9990
0.6330
0.8800
Mean of individual choices, given parameters and middle eq.m:
0.6333
_______________________________________________________________________________
Two-step estimates:
twostepC_sa =
twostepJ_sa =
0.4601
2.8200
logL (at max)= -8.4651e+003
_______________________________________________________________________________
Direct estimates, using arbitrary initial values for parameters:
Initial values:
startC = 0
startJ = 2
Estimates:
directC_sa =
directJ_sa =
0.1289
1.6048
logL (at max)= -8.9940e+003
candC
candJ
low-eqm
med-eqm
hig-eqm
mean(Omega)
0.1289
1.6048
-0.9420
0.5550
0.6320
0.6333
candC
candJ
lik-med
lik-high
lik-low
which-eqm
0.1289
1.6048 -57.7457
-9.0901
-8.9940
3.0000
_______________________________________________________________________________
Direct estimates, using the two-step estimates as initial values:
Initial values:
startC = 0.4601
startJ = 2.8200
Estimates:
directC_sa_2s =
directJ_sa_2s =
logL (at max) =
0.4601
2.8169
-8.4651e+003
candC
candJ
low-eqm
med-eqm
hig-eqm
mean(Omega)
0.4601
2.8169
-0.9990
0.6340
0.8820
0.6333
candC
candJ
lik-low
lik-med
lik-high
which-eqm
0.4601
2.8169 -119.6801
-8.4651 -10.4715
2.0000
_______________________________________________________________________________
Likelihood
• Let the likelihood of y given parameters θ be
L(y|θ) ≡ p(y; θ)
where p(·) is the pdf of y.
• In our setup, the likelihood is defined as:
where
R
L(y|θ) =
Z
(u,v):g(π(θ,u),θ,v)=y
denotes the Aumann integral
19
h(v)dvf (u)du
Direct Method
( R
v:g(π,θ;v)=y h(v)dv if
L(y|π, θ, u) =
F (π; θ, u) = 0
otherwise
0
Estimator:
⎛ Ã
⎞
!
Z
θb = arg max ⎝
max
L(y|π, θ, u) f (u)du.⎠
θ
π:F (π;θ,u)=0
u
20
Algorithm:
1. Choose a candidate value for θ; fix a vector of disturbances u
2. Compute all possible equilibria π (θ, u) = {π 1 (θ, u) , ..., π K (θ, u)}
3. Compute maxk L(y|π k , θ, u)
4. Integrate with respect to u
5. Maximize over θ.
21
The two-step method
Define:
L(y|π, θ) ≡
• First Step:
Z ÃZ
u
v:g(π,θ;v)=y
!
h(v)dv f (u)du
b ) = arg max L(y|π, θ)
b 1, θ
(π
1
π,θ
b 1 to max the full likelihood:
• Second Step use π
⎛
⎞
Z
⎜
⎟
b
⎜
b 1, θ, u)f (u)du.⎟
θ2 = arg max ⎝
L(y|π
⎠
θ
u:θ=θ(π
b1,u)
22
The second step does NOT involve computation of all the equilibria
• Algorithm for the computation of θb2:
— Choose a value for θ
b 1 is an equilib— Find the set of values for u s.t. the given π
rium for the posited value of θ
b 1, θ)
— Integrate the density f (u) over this set to obtain L(y | π
— Maximize over θ
23
Consistency
L(y | π, θ) is a well behaved likelihood function, continuous in
(y, π, θ) and bounded; the parameter spaces Θ and P × X are
compact by definition. Therefore, standard consistency arguments
go through.
Note: θ may not be identified in the first step (we only observe
one realization of π)
24
Multiple Equilibria in the Data
• different cities or geographic regions, industrial sectors, ..., in
cross-sectional context may be in different equilibria;
• different time periods in a static, repeated framework (as in
Dagsvik and Jovanovic (1994));
⇒ use previous methods to define each city’s contribution to the
likelihood, summing them together.
• Provides additional source of identification
25
Multiple Equilibria in the Data
• In a dynamic setting with a Markov structure, different observations may pertain to different equilibria.
⇒ Here, we can also entertain the possibility that the different
units have a pre-specified correlation structure, to be estimated
jointly with θ.
26
Jovanovic-Dagsvik Method
• Likelihood for each city is a mixture of likelihoods conditional
on equilibria, with weights equal to probabilities of equilibria
given data:
L(y | θ) =
J
Y
j=1
L (j|j − 1, j − 2, ...) ,
where
L (j|j − 1, j − 2, ...) =
³
X
k
³
´
L yj |π k , θ · φj (π k )
´
where φj (π k ) = Pr city j in eqm π k | yj−1, yj−2, ... .
• Equilibrium selection mechanism is modeled as a SAR(2) process.
27
• Equilibrium selection mechanism:
— with probability a1 city j adopts equilibrium of city j − 1;
— with probability a2 city j adopts equilibrium of city j − 2;
— with probability (1 − a1 − a2) city j chooses equilibrium
independently, with probabilities (p1, p2).
• Conditional equilibrium probabilities φj (π k ) are computed recursively.
28
Number of draws in Montecarlo Experiment:
640
Number of Cities: 10
Summary Statistics: True Parameters
C
Minimum
Maximum
Mean
-0.52063
0.50179
-0.00461
J
1.03381
2.99826
2.36532
Correlation between Cities: A1=1/3, A2=1/3
Probability of Different Equilbria: P1=1/3, P2=1/3
Estimator Comparison
direct method
two-step method
RMSE, C
Bias, C
0.04118
-0.00138
0.00571
-0.00002
RMSE, J
Bias, J
0.16222
0.02906
0.03810
0.00263
min time
max time
mean time
median time
161.99815
538.02297
350.93698
390.35940
exits
23
No. of Function Evaluations
Direct Method
min
2101.00000
max
3000.00000
mean
2664.40156
median
2701.00000
(computed from
640 simulations)
Two-step Method
min
2201.00000
max
2901.00000
mean
2622.56250
median
2601.00000
(computed from
640 simulations)
2.25910
11.33234
4.95842
5.43274
0
| J, direct
| J, two-step
All simulations
RMSE
| 0.16222
Bias
| 0.02906
no. of simulations | (640)
| 0.03810
| 0.00263
| (640)
Within 2 Standard Deviations
RMSE
| 0.03814
Bias
| 0.00338
no. of simulations | (620)
| 0.01653
| 0.00053
| (635)
Within 1 Standard Deviation
RMSE
| 0.02316
Bias
| 0.00105
no. of simulations | (612)
| 0.01551
| 0.00063
| (628)
Within .5 Standard
RMSE
Bias
no. of simulations
| 0.01275
| 0.00083
| (593)
Deviations
| 0.01933
| 0.00002
| (608)
Within .25 Standard Deviations
RMSE
| 0.01970
Bias
| -0.00019
no. of simulations | (450)
| 0.01362
| 0.00086
| (327)
Within .15 Standard Deviations
RMSE
| 0.03187
Bias
| -0.00303
no. of simulations | (125)
| 0.01402
| 0.00047
| (187)
Number of draws in Montecarlo Experiment:
672
Number of Cities: 10
Summary Statistics: True Parameters
C
Minimum
Maximum
Mean
-0.52495
0.50542
-0.00531
J
1.14736
2.99971
2.38348
Correlation between Cities: A1=1/10, A2=8/10
Probability of Different Equilibria: P1=1/3, P2=1/3
Estimator Comparison
direct method
two-step method
RMSE, C
Bias, C
0.10920
-0.00628
0.09355
-0.00400
RMSE, J
Bias, J
0.35842
0.12616
0.28770
0.09020
min time
max time
mean time
median time
exits
0.20495
550.88008
295.89813
382.98843
173
No. of Function Evaluations
Direct Method
min
3.00000
max
3000.00000
mean
2035.23162
median
2601.00000
(computed from
544 simulations)
Two-step Method
min
3.00000
max
2901.00000
mean
2006.80147
median
2601.00000
(computed from
544 simulations)
0.00325
11.43535
4.22500
5.41913
128
| J, direct
| J, two-step
All simulations
RMSE
| 0.35842
Bias
| 0.12616
no. of simulations | (672)
| 0.28770
| 0.09020
| (672)
Within 2 Standard Deviations
RMSE
| 0.20467
Bias
| 0.05504
no. of simulations | (624)
| 0.14293
| 0.02828
| (620)
Within 1 Standard Deviation
RMSE
| 0.10695
Bias
| 0.02167
no. of simulations | (575)
| 0.06156
| 0.01185
| (580)
Within .5 Standard
RMSE
Bias
no. of simulations
| 0.04493
| 0.00759
| (570)
Deviations
| 0.06518
| 0.01351
| (432)
Within .25 Standard Deviations
RMSE
| 0.15576
Bias
| 0.07545
no. of simulations | ( 37)
| 0.11393
| 0.05403
| ( 43)
Within .15 Standard Deviations
RMSE
| 0.15909
Bias
| 0.09290
no. of simulations | ( 18)
| 0.12166
| 0.06245
| ( 25)
Number of draws in Montecarlo Experiment:
155
Number of Cities: 25
Number of Agents: 150
Summary Statistics: True Parameters
C
Minimum
Maximum
Mean
-0.47979
0.46909
0.00859
J
1.21660
2.99796
2.40884
A1 = 1/3, A2 = 1/3, P1 = 1/3, P2 = 1/3
Estimator Comparison
Jovanovic method
RMSE,
Bias,
RMSE,
Bias,
C
C
J
J
RMSE,
Bias,
RMSE,
Bias,
A1
A1
A2
A2
0.21504
0.00579
0.20590
-0.10844
0.27215
-0.01959
0.18979
-0.05694
RMSE,
Bias,
RMSE,
Bias,
P1
P1
P2
P2
0.18832
-0.01138
0.19455
-0.02221
0.20338
-0.01643
0.22299
-0.00585
min time
max time
mean time
median time
0.04429
-0.00173
0.17752
0.01694
Two-step method
386.44514
473.91435
418.64079
418.54813
No. of Function Evaluations
Jovanovic Method
min
3000.00000
max
3000.00000
mean
3000.00000
median
3000.00000
Two-step Method
min
2101.00000
max
2701.00000
mean
2394.54839
median
2401.00000
0.02649
-0.00272
0.15981
0.06149
0.08633
0.11836
0.10133
0.10144
| J, direct
| J, two-step
All simulations
RMSE
| 0.17752
Bias
| 0.01694
no. of simulations | (155)
| 0.15981
| 0.06149
| (155)
Within 2 Standard Deviations
RMSE
| 0.10912
Bias
| -0.00124
no. of simulations | (152)
| 0.12584
| 0.04282
| (148)
Within 1 Standard Deviation
RMSE
| 0.09039
Bias
| -0.00701
no. of simulations | (145)
| 0.08924
| 0.02597
| (132)
Within .5 Standard
RMSE
Bias
no. of simulations
| 0.10087
| 0.03866
| ( 60)
Deviations
| 0.08791
| -0.01118
| ( 99)
Within .25 Standard Deviations
RMSE
| 0.08464
Bias
| -0.01666
no. of simulations | ( 39)
| 0.11209
| 0.06377
| ( 23)
Within .15 Standard Deviations
RMSE
| 0.09072
Bias
| -0.02647
no. of simulations | ( 22)
| 0.11150
| 0.05399
| ( 16)