Measuring Lack-of-Fit of a Bayesian Model
(A Simple Way to Broaden your Posterior)
David Fletcher
1
Peter Dillingham
2
1 Department of Mathematics and Statistics
University of Otago, Dunedin, New Zealand
2 School of Science and Technology
University of New England, Armidale, Australia
Overview
◮
Count (and binomial) data
◮
Testing vs Measuring lack-of-fit
Overdispersion adjustment transferred to Bayesian setting
◮
1. Estimating the amount
2. Broadening the posterior
◮
Example: lapwing mark-recovery data
◮
Simulations: simple setting (coverage of CIs)
Posterior predictive p-value
◮
◮
Response variable Y (n × 1)
Parameters β (p × 1)
◮
Model M(β) for µ ≡ E (Y )
◮
D(y , β) = discrepancy between y and model M(β)
p = Pr (D(y ∗ , β) ≥ D(y , β)|y ), where y ∗ ∼ M(β)
Posterior predictive p-value
D(y , β) for count data:
− µi )2
P
i (yi
P
i
(yi −µi )2
µi
P
i
(yi −µi )2
µi
P √
i
yi −
.P
yi
i µi
√ 2
µi
(Fletcher 2012)
(Freeman-Tukey 1950)
Mark-Recovery Example
◮
British lapwings ringed as nestlings (Barry et al 2003)
◮
Product-multinomial (assume constant reporting probability)
Year
1970
1971
1972
...
1990
1991
1992
Released
1963
2463
3092
...
4170
4314
3480
Age at recovery
0
1
2
3
8
3
2
0
4
1
1
2
7
2
2
2
... ... ... ...
12
3
3
9
4
18
...
...
...
...
20
0
0
0
21
0
0
22
0
Mark-Recovery Example
◮
Logit of survival depends on
Model 1: Age group (first-year of life or older)
Model 2: Age group + Random Error
◮
2 chains, each with 105 iterations (burn-in = 103 )
◮
Ignore year effect here (Barry et al 2003)
800
600
400
D*
1000
Model 1: Age group
400
600
800
D
p=0.03
1000
700
500
D*
900
1100
Model 2: Age group + Random Error
500
600
700
800
D
p=0.43
900
1000 1100
Posterior predictive p-value
◮
p close to 0.5 ⇒ no evidence of lack-of-fit
◮
Issues
1. How close to 0.5? Need for calibration (Hjort et al 2006)
2. Lack-of-fit test : power depends on n
◮
Testing vs measuring lack-of-fit?
Measuring lack-of-fit
◮
p = Pr (D(y ∗ , β) ≥ D(y , β)|y ), with y ∗ ∼ M(β)
◮
Measure lack-of-fit:
◮
p = Pr (φ ≤ 1|y )
φ=
D(y ,β)
D(y ∗ ,β)
800
600
400
D*
1000
Model 1: Age group
400
600
800
D
p=0.03
1000
0.0
0.5
1.0
1.5
Model 1: Age group
0.5
1.0
1.5
φ
Mean = 1.39
2.0
2.5
700
500
D*
900
1100
Model 2: Age group + Random Error
500
600
700
800
D
p=0.43
900
1000 1100
0.0
0.5
1.0
1.5
Model 2: Age group + Random Error
0.5
1.0
1.5
φ
Mean = 1.06
2.0
Overdispersion
Frequentist setting: widening your confidence interval
◮
Wald interval
β̂ ± tn−p φ̂1/2 sβ̂
◮
Profile likelihood
2{log Lp (β̂)−log Lp (β0 )}
φ̂
∼ F1,n−p
Overdispersion
Bayesian setting: broadening your posterior
1/2 −1
F̂ β j
β∗j = F̂ −1 Φ φj
Φ
F̂ (·) = empirical c.d.f. of β j
Φ(·) = N(0, 1) c.d.f.
Replace β j by more (or less) extreme value, as determined by φj
Mark-Recovery Example
Age Group
0.76
0.78
Age Group (Adjusted)
0.80
0.82
0.84
Survival after first year
0.86
0.88
Mark-Recovery Example
Age Group
Age Group (Adjusted)
0.76
0.78
0.80
0.82
Age Group + Random Error
0.84
Survival after first year
0.86
0.88
Simulations: Simple Setting
Yi ∼ NBin(µ), var(Yi ) = φµ
n
10
10
50
50
n
10
10
50
50
φ
2
3
2
3
φ
2
3
2
3
Poisson
0.83
0.74
0.84
0.74
Poisson
2.75
2.74
1.24
1.24
95% CI for µ = 5 (104 simns )
Coverage
Adj. Poisson
0.95
0.92
0.95
0.94
Median Width
Adj. Poisson
4.26
4.78
1.78
2.13
True Model (NBin)
0.98
0.97
0.96
0.95
True Model (NBin)
5.82
6.55
1.85
2.31
Discussion
◮
Simple, quick method for broadening your posterior
◮
More robust than adding a random effect? (Cox et al 2010)
◮
Use with hierarchical models?
◮
n/φ = “effective sample size” (McCullagh & Pregibon 1985)
◮
log L(β) →
1
φ
log L(β)
(QAIC and Fitzmaurice 1997)
◮
“Quasi-posterior” (Rodrigues 1997, Lin 2006, Annis 2007)
◮
Alternative methods for broadening the posterior?
◮
Different adjustment needed for each parameter?
◮
Different φ for different components of Y ?
Measuring Lack-of-Fit of a Bayesian Model
(A Simple Way to Broaden your Posterior)
David Fletcher
1
Peter Dillingham
2
1 Department of Mathematics and Statistics
University of Otago, Dunedin, New Zealand
2 School of Science and Technology
University of New England, Armidale, Australia