Mixed Effects
Models
T. Mielke
Review
Example 1
Design
Example 2
Approximation of the Fisher information
and design in nonlinear mixed effects
models
Information
Approximation
Example 3
Example 1
Tobias Mielke
Optimum Design for Mixed Effects Non-Linear and
Generalized Linear Models
PODE - 2011
Mixed Effects
Models
Outline
T. Mielke
Review
Example 1
Design
Example 2
Information
Approximation
1 Review
Example 1
Example 3
Example 1
2 Design
Example 2
3 Information Approximation
Example 3
Example 1
Mixed Effects
Models
Mixed Effects Models
T. Mielke
Review
Example 1
Design
Example 2
Information
Approximation
Example 3
Example 1
• Similar functions for different individuals
• Every individual has its own individual parameters
• Vectors of individual parameters are realizations of
random vectors
→ Mixed Effects Models
Mixed Effects
Models
Mixed Effects Models
T. Mielke
Review
Example 1
Design
Two-stage-model:
Example 2
Information
Approximation
• 1. stage (intra-individual variation):
Example 3
Example 1
Yij
= η(βi , xij ) + ij , j = 1, ..., mi , ij ∼ N(0, σ 2 )
• 2. stage (inter-individual variation):
βi = β + bi , i = 1, ..., N, bi ∼ Np (0, σ 2 D)
• bi and ij are assumed to be independent.
• Variance parameters σ 2 and D assumed to be known.
• Aim: Estimation of β.
Mixed Effects
Models
Review
T. Mielke
Review
Example 1
Design
Individual observation vector:
Example 2
Information
Approximation
Yi
= η(βi , ξi ) + i ,
Example 3
Example 1
- mi - number of observations,
- ξi = (xi1 , ..., ximi ) - experimental settings,
- η(βi , ξi ) := (η(βi , xi1 ), ..., η(βi , ximi ))T .
Design matrix:
Fβ := (
∂η(βi , ξi )
|βi =β ).
∂βiT
Mixed Effects
Models
Review
T. Mielke
Review
Example 1
Design
Example 2
Information
Approximation
Example 3
Example 1
• Optimal designs for estimating β → cov (β̂)?
• For Yi with density fYi (yi ) and
l(β; yi ) := log(fYi (yi )) :
Maximization of the Fisher information
Mβ = E(
∂l(β; Yi ) ∂l(β; Yi )
).
∂β
∂β T
• Often applied: Linearization.
Mixed Effects
Models
Review
T. Mielke
Review
Example 1
For η nonlinear in βi :
Design
Example 2
Information
Approximation
Example 3
Example 1
• Linearization of the model (1):
Yi
= η(βi , ξi ) + i
≈ η(β0 , ξi ) + Fβ0 (β − β0 ) + Fβ0 (βi − β) + i ,
with a guess β0 of β (or βi ).
• Distribution assumptions yield:
Yi ∼ N(η(β0 , ξi ) + Fβ0 (β − β0 ), σ 2 Vβ0 )
with Vβ0 := Imi + Fβ0 DFβT0
→ Linear mixed effects model.
Mixed Effects
Models
Review
T. Mielke
Review
Example 1
For η nonlinear in βi :
Design
Example 2
Information
Approximation
Example 3
Example 1
• Linearization of the model (2):
Yi
= η(βi , ξi ) + i
≈ η(β, ξi ) + Fβ (βi − β) + i
with the true β.
• Distribution assumptions yield:
Yi ∼ N(η(β, ξi ), σ 2 Vβ )
with Vβ := Imi + Fβ DFβT
→ Heteroscedastic nonlinear normal model.
Mixed Effects
Models
Review
T. Mielke
Review
Example 1
Design
Example 2
Calculation of the Fisher information:
Assumption: new model is true.
• Linear Mixed Information [Retout et. al (2001)]:
Information
Approximation
Example 3
Example 1
⇒ M1,β (ξ) =
1 T −1
F V Fβ
σ 2 β0 β0 0
• Nonlinear heteroscedastic Information
[Retout et. al (2003)]:
⇒ M2,β (ξ) =
1 T −1
1
Fβ Vβ Fβ + S,
2
2
σ
where S ≥ 0, with
Sj,k = Tr [Vβ−1
∂Vβ −1 ∂Vβ
V
], j, k = 1, ..., p.
∂βj β ∂βk
Mixed Effects
Models
Review
T. Mielke
Review
Example: Observations as
Example 1
Design
Example 2
Information
Approximation
Example 3
Yi
= exp(βi ) + i , βi ∼ N(β, d), i ∼ N(0, 1) yield
M1,β =
exp(2β)
and
1 + d exp(2β)
M2,β =
exp(2β)
2d 2 exp(4β)
.
+
1 + d exp(2β) (1 + d exp(2β))2
Example 1
Mixed Effects
Models
Design
T. Mielke
Review
Example 1
Design
Example 2
Information
Approximation
Example 3
• Experiment settings ξi for individuals: xij ∈ X
ξi = xi1 ... ximi ∈ X mi .
Example 1
• Alternative: Approximate individual designs - not here!
• Population design ζ:
ζ=
ξ1 ... ξk
ω1 ... ωk
X
k
,
ωj = 1,
j=1
ωj : weight of indivdual design ξj in the population.
• Population design =
ˆ approximate design on X m
Mixed Effects
Models
Design
T. Mielke
Review
Example 1
Design
Example 2
Information
Approximation
Example 3
• Individual information: Mβ (ξi )
• Population information: Mβ;pop (ζ) =
k
P
ωi Mβ (ξi )
i=1
Example 1
• Population design ζ ∗ D-optimal if and only if:
Tr Mβ;pop (ζ ∗ )−1 Mβ (ξ) ≤ p, ∀ξ ∈ X m
• Efficiency:
δ(ζ) :=
det(Mβ;pop (ζ))
det(Mβ;pop (ζ ∗ ))
1
p
.
Mixed Effects
Models
T. Mielke
Example [Schmelter 2007]:
Review
Example 1
Design
Example 2
Information
Approximation
Example 3
Example 1
• Yi = η(βi , xi ) exp(i ), with i ∼ N(0, σ 2 )
β
β
• η(βi , xi ) := β β 1,i−β [exp(− β2,i xi ) − exp(−β1,i xi )],
3,i 1,i
2,i
3,i
• βi,k = βk exp(bik ) with bik ∼ N(0, σ 2 dk ), k = 1, 2, 3,
Mixed Effects
Models
Example: Information matters
T. Mielke
Review
Example 1
Design
Example 2
Information
Approximation
Example 3
Example 1
M1,β -Information:
(0.10) (4.18) (24.00)
→ ζ1 =
, δ(ζ2 ) = 0.66
0.33
0.33
0.33
M2,β -Information:
→ ζ2 =
(3.10) (5.18) (24.00)
, δ(ζ1 ) = 0.55
0.61
0.08
0.31
Mixed Effects
Models
Information Approximation
T. Mielke
Review
Remember:
Example 1
Design
Yi = η(βi , ξi ) + i , βi ∼ N(β, σ 2 D), i ∼ N(0, σ 2 Imi )
Example 2
Information
Approximation
log-Likelihood function:
Example 3
Example 1
l(β; yi ) := log(fYi (yi )), with
Z
1
1
fYi (yi ) :=
exp[− 2 l̃(βi , β; yi )]dβi and
2σ
Rp c1
l̃(βi , β; yi ) :=
(yi − η(βi , ξi ))T (yi − η(βi , ξi ))
+(βi − β)T D −1 (βi − β).
We search:
Mβ = E(
∂l(β; Yi ) ∂l(β; Yi )
).
∂β
∂β T
Mixed Effects
Models
T. Mielke
Information Approximation
Review
Example 1
Alternative representation:
Design
Example 2
Information
Approximation
Example 3
Example 1
• With Eyi (βi ) := E(βi |Yi = yi ):
∂l(β; yi )
1
= 2 D −1 (Eyi (βi ) − β).
∂β
σ
• For Varyi (βi ) = Var (βi |Yi = yi ) follows:
Mβ =
=
1 −1
1
D Var (EYi (βi ))D −1 2
σ2
σ
1 −1
1
1
D − 2 D −1 E(VarYi (βi ))D −1 2
2
σ
σ
σ
→ Approximation of conditional moments.
Mixed Effects
Models
Remark
T. Mielke
Review
Example 1
Design
More general inter-individual model:
Example 2
Information
Approximation
Example 3
Example 1
Differentiable function g : Rp1 → Rp :
βi = g(β) + bi , with β ∈ Rp1 , bi ∼ Np (0, σ 2 D)
With the (p × p1 )-Jacobi-matrix G(β) follows:
Mβ = G(β)T
1
1 −1
D Var (EYi (βi ))D −1 2 G(β)
2
σ
σ
= G(β)T (
1 −1
1
1
D − 2 D −1 E(VarYi (βi ))D −1 2 )G(β)T
2
σ
σ
σ
Mixed Effects
Models
T. Mielke
Information Approximation
Review
Example 1
Design
Example 2
Information
Approximation
Example 3
Example 1
Approximation of conditional moments:
• Quadrature rules
• Simulations
→ dependence on experimental settings?
→ computational intensive.
• Laplace approximation
→ accuracy?
Mixed Effects
Models
T. Mielke
Review
Information Approximation
Laplace approximation:
Example 1
Design
• Problem in nonlinear mixed models:
Example 2
E(EYi (βi )EYi (βi )T )
Information
Approximation
Example 3
Example 1
• Approximation of Eyi (βi ) given yi :
R
Eyi (βi )
=
l̃(βi , β; yi ) :=
˜ ,β;y )
i
i
βi exp(− l(β2σ
)dβi
2
, with
˜l(β ,β;y )
R
i
i
exp(− 2σ2 )dβi
(yi − η(βi , ξi ))T (yi − η(βi , ξi ))
+(βi − β)T D −1 (βi − β).
• Laplace approximation of both integrals.
[Tierney and Kadane (1986)]
Mixed Effects
Models
T. Mielke
Review
Example 1
Information Approximation
Laplace approximation:
• Idea: Approximation of
Design
Example 2
Information
Approximation
Z
fYi (yi ) =
Example 3
l̃(βi , β; yi )
1
exp(−
)dβi
c1
2σ 2
Example 1
where l̃(βi , β; yi ) is minimial.
• Second order Taylor expansion around βi∗ with
βi∗ := argmin{l̃(βi , β; yi )}
βi ∈Rp
→ Linear part vanishes → normal density.
fYi (yi ) ≈
l̃(β ∗ , β; yi )
1
).
exp(− i 2
c2 (yi , β)
2σ
• Dependence of βi∗ on yi .
Mixed Effects
Models
Information Approximation
T. Mielke
Review
Example 1
• For approximating
Design
Example 2
fβi |Yi =yi =
Information
Approximation
φYi |βi (yi )φβi (βi )
fYi (yi )
Example 3
Example 1
• Apply similar approximation to the numerator
⇒ βi |Yi =yi
Mβi∗
app.
∼
N(βi∗ , σ 2 Mβ−1
∗ ), with
i
:=
1 ∂ 2 l̃(βi , β; yi )
|βi =βi∗ .
2 ∂βi ∂βiT
• Problems:
• Dependence of βi∗ on yi
• Dependence of Mβ−1
∗ on yi
i
Mixed Effects
Models
Information Approximation
T. Mielke
Review
Example 1
Design
• Alternative: Just approximation of η(βi , ξi ):
Example 2
Information
Approximation
η(βi , ξi ) ≈ η(β̂i , ξi ) + Fβ̂i (βi − β̂i ).
Example 3
Example 1
in l̃(βi , β; yi ) with some β̂i ∈ Rp .
• Same approximation to numerator yields
app.
∼
N(µ(yi , β̂i , β), σ 2 M−1 ), with
Mβ̂i
:=
Fβ̂T Fβ̂i + D −1
µ(yi , β̂i , β)
:=
M−1 (Fβ̂T (yi − η(β̂i , ξi )) + Fβ̂i β̂i ) + D −1 β).
⇒ βi |Yi =yi
β̂i
i
β̂i
i
• Appropriate choice of β̂i ?
Mixed Effects
Models
Information Approximation
T. Mielke
Review
Example 1
Design
• For β̂i = βi∗ minimizing l̃(βi , β; yi ):
Example 2
Information
Approximation
app.
⇒ βi |Yi =yi ∼ N(βi∗ , σ 2 M−1
β ∗ ).
i
Example 3
Example 1
Problem: Dependence on yi .
• For β̂i = β:
app.
T
2 −1
⇒ βi |Yi =yi ∼ N(β + M−1
β Fβ (yi − η(β, ξi )), σ Mβ ).
• For β̂i = βi :
app.
⇒ βi |Yi =yi ∼ N(µ(yi , βi , β), σ 2 M−1
βi ).
Mixed Effects
Models
T. Mielke
Review
Information Approximation
Fisher information approximations in β̂i = β:
Example 1
Design
Example 2
Information
Approximation
• With Vβ := Imi + Fβ DFβT :
• Conditional variance:
Example 3
Example 1
Mβ =
≈
1 −1
1
1
D − 2 D −1 E(VarYi (βi ))D −1 2
2
σ
σ
σ
1 T −1
F V Fβ =: M1,β
σ2 β β
• Conditional expectation:
Mβ =
≈
1 −1
1
D Var (EYi (βi ))D −1 2
2
σ
σ
1 T −1
F V Var (Yi )Vβ−1 Fβ =: M3,β .
σ4 β β
Mixed Effects
Models
T. Mielke
Review
Example 1
Design
Information Approximation
Fisher information approximation in β̂i = βi :
• With Vβi := Imi + Fβi DFβT :
i
Example 2
Information
Approximation
Example 3
Example 1
• Conditional variance:
M4,βi
1 T −1
F V Fβi .
σ 2 βi βi
:=
→ consider here M4,β :=
1
E(FβTi Vβ−1
Fβi ).
σ2
i
Some other possible approximations:
• Nonlinear heteroscedastic normal model:
M2,β :=
1 T −1
1
Fβ Vβ Fβ + S.
2
2
σ
• Quasi-Information:
T
M5,β := Fβ,QL
Var (Yi )−1 Fβ,QL
Mixed Effects
Models
T. Mielke
Information Approximation
Review
Example 1
Design
Summary: With Vβi := Imi + Fβi DFβTi :
Example 2
Information
Approximation
Example 3
M1,β :=
1 T −1
F V Fβ
σ2 β β
M2,β :=
1 T −1
1
Fβ Vβ Fβ + S
2
2
σ
M3,β :=
1 T −1
F V Var (Yi )Vβ−1 Fβ
σ4 β β
M4,β :=
1
Fβi )
E(FβTi Vβ−1
i
σ2
Example 1
T
M5,β := Fβ,QL
Var (Yi )−1 Fβ,QL
Mixed Effects
Models
Example 3:
T. Mielke
Review
Example 1
Design
Example 2
Information
Approximation
Example 3
Example 1
• Log-Concentration:
Yi
•
= −β2i − xi exp(β1i − β2i ) + i , i = 1, ..., N
- i ∼ N(0, σ 2 ), ξi = (xi ) ∈ [tmin , tmax ],
- βi = (β1i , β2i )T ∼ N(β, σ 2 D),
- β = (β1 , β2 )T , D = diag(d1 , d2 ).
Mixed Effects
Models
T. Mielke
Example 3
Review
Example 1
Design
Example 2
Information
Approximation
Example 3
- M1,β : Red
- M2,β : Blue
- M3,β : Yellow
Example 1
- M5,β : Green
Mixed Effects
Models
Example 3
T. Mielke
Review
Example 1
Design
Example 2
Information
Approximation
Example 3
Information matrices M1,β (β) or M5,β (β):
For every pair of experimental settings x1 , x2 ∈ [tmin , tmax ]
with
Example 1
α(x1 , x2 ) := σ 2 + cov (Y (x1 ), Y (x2 )) = 0,
the population design
1
ζ =
(x1 ) (x2 )
0.5 0.5
is D-optimal in the case of one allowed observation per
individual.
Mixed Effects
Models
Example 3
T. Mielke
Review
Example 1
How to see this:
• Equivalence theorem for D-optimality: ζ optimal
Design
Example 2
Information
Approximation
Example 3
Example 1
gζ (ξ) := Tr Mβ;pop (ζ ∗ )−1 Mβ (ξ) ≤ p, ∀ξ ∈ X m
a b
• With general nonsingular M(ζ) =
and ξ = x:
b c
c −b
g̃ζ (x) := Fβ (x)
Fβ (ξ)T − 2(ac − b2 )Vβ (x) ≤ 0
−b a
• 2 support points, weight: ω = 0.5.
• For
(x1 ) (x2 )
ζ=
0.5 0.5
⇒ g̃ζ (x) =
α(x1 ,x2 )
V (x1 )V (x2 ) (x
− x2 )(x − x1 ).
Mixed Effects
Models
Example 3
T. Mielke
Review
M1,β -Information:
Example 1
Design
Example 2
Information
Approximation
Example 3
(1.13) (11.98)
→ ζ1 =
, δ1 (ζ1 ) = 1.0000
0.5
0.5
M2,β -Information:
Example 1
(0.96) ( 6.00)
→ ζ2 =
, δ1 (ζ2 ) = 0.9895
0.47
0.53
M3,β -Information:
(1.70) (24.00)
→ ζ3 =
, δ1 (ζ3 ) = 0.9794
0.5
0.5
M5,β -Information:
(0.93) (11.99)
, δ1 (ζ5 ) = 0.9949
→ ζ5 =
0.5
0.5
Mixed Effects
Models
Example 1
T. Mielke
Review
Example 1
Example: Observations as
Design
Example 2
Information
Approximation
Yi
= exp(βi ) + i , βi ∼ N(β, d), i ∼ N(0, 1) yield
M1,β =
exp(2β)
,
1 + d exp(2β)
M2,β =
2d 2 exp(4β)
exp(2β)
+
,
1 + d exp(2β) (1 + d exp(2β))2
M3,β =
exp(2β)(exp(2β + d)(exp(d) − 1) + 1)
and
(1 + d exp(2β))2
M5,β =
exp(2β + d)
.
exp(2β + d)(exp(d) − 1) + 1
Example 3
Example 1
Mixed Effects
Models
Example 1
T. Mielke
Review
Example 1
Yi = exp(βi ) + i ; βi ∼ N(−2, d =
ρ
1−ρ ); i
∼ N(0, 1)
Design
Example 2
Information
Approximation
Example 3
Example 1
• Red: Simulated Fisher information; Blue: M4,β
• M1,β : solid, M2,β : dashed, M3,β : dot-dash, M5,β : dotted
Mixed Effects
Models
Example 1
T. Mielke
Review
Example 1
Yi = exp(βi ) + i ; βi ∼ N(−1, d =
ρ
1−ρ ); i
∼ N(0, 1)
Design
Example 2
Information
Approximation
Example 3
Example 1
• Red: Simulated Fisher information; Blue: M4,β
• M1,β : solid, M2,β : dashed, M3,β : dot-dash, M5,β : dotted
Mixed Effects
Models
Example 1
T. Mielke
Review
Example 1
Yi = exp(βi ) + i ; βi ∼ N(0, d =
ρ
1−ρ ); i
∼ N(0, 1)
Design
Example 2
Information
Approximation
Example 3
Example 1
• Red: Simulated Fisher information; Blue: M4,β
• M1,β : solid, M2,β : dashed, M3,β : dot-dash, M5,β : dotted
Mixed Effects
Models
Example 1
T. Mielke
Review
Example 1
Yi = exp(βi ) + i ; βi ∼ N(1, d =
ρ
1−ρ ); i
∼ N(0, 1)
Design
Example 2
Information
Approximation
Example 3
Example 1
• Red: Simulated Fisher information; Blue: M4,β
• M1,β : solid, M2,β : dashed, M3,β : dot-dash, M5,β : dotted
Mixed Effects
Models
Example 1
T. Mielke
Review
Example 1
Yi = exp(βi ) + i ; βi ∼ N(2, d =
ρ
1−ρ ); i
∼ N(0, 1)
Design
Example 2
Information
Approximation
Example 3
Example 1
• Red: Simulated Fisher information; Blue: M4,β
• M1,β : solid, M2,β : dashed, M3,β : dot-dash, M5,β : dotted
Mixed Effects
Models
Example 1
T. Mielke
Review
Example 1
Design
Example 2
Remember:
Var (βi ) = E(VarYi (βi )) + Var (EYi (βi ))
Information
Approximation
Example 3
Example 1
Mβ =
=
1 −1
1
D Var (EYi (βi ))D −1 2
2
σ
σ
1
1
1 −1
D − 2 D −1 E(VarYi (βi ))D −1 2
2
σ
σ
σ
Some explanation:
• d →∞
⇒ Mβ → 0.
• d → 0 ⇒ Nonlinear regression
⇒ Mβ →
1 T
F Fβ
σ2 β
Mixed Effects
Models
Conclusions/Outlook
T. Mielke
Review
Example 1
Design
Example 2
Information
Approximation
Example 3
Example 1
• Conclusions:
- Motivation of model approximation
- Different information → different design
- Big influence of inter-individual variance on the
accuracy of approximations
• Outlook:
- More insight needed on:
- appropriateness of the approximations
- approximation in βi or βi∗
- Inclusion of variance parameters.
Mixed Effects
Models
T. Mielke
Review
Example 1
Design
Example 2
Information
Approximation
Example 3
Example 1
Thank you for your attention!
Further informations needed (M6,β ,...)?