Mixed Effects
Models
T. Mielke
Information
Approximation
Experimental
designs
Optimal Design in Nonlinear Mixed Effects
Models
Tobias Mielke
Experiments for Processes with Time and Space
Dynamics 2011
Mixed Effects
Models
Outline
T. Mielke
Information
Approximation
Experimental
designs
1 Information Approximation
Mixed Effects
Models
Outline
T. Mielke
Information
Approximation
Experimental
designs
1 Information Approximation
2 Experimental designs
Mixed Effects
Models
T. Mielke
Information
Approximation
Experimental
designs
Mixed Effects Models
Mixed Effects
Models
T. Mielke
Mixed Effects Models
Information
Approximation
Experimental
designs
• Similar functions for different individuals
Mixed Effects
Models
T. Mielke
Mixed Effects Models
Information
Approximation
Experimental
designs
• Similar functions for different individuals
• Every individual has its own individual parameters
Mixed Effects
Models
Mixed Effects Models
T. Mielke
Information
Approximation
Experimental
designs
• Similar functions for different individuals
• Every individual has its own individual parameters
• Vectors of individual parameters are realizations of
random vectors
Mixed Effects
Models
Mixed Effects Models
T. Mielke
Information
Approximation
Experimental
designs
• Similar functions for different individuals
• Every individual has its own individual parameters
• Vectors of individual parameters are realizations of
random vectors
→ Mixed Effects Models
Mixed Effects
Models
Mixed Effects Models
T. Mielke
Information
Approximation
Experimental
designs
Two-stage-model:
• 1. stage (intra-individual variation):
Yij
= η(xij , βi ) + ij , j = 1, ..., mi , ij ∼ N(0, σ 2 )
Mixed Effects
Models
Mixed Effects Models
T. Mielke
Information
Approximation
Experimental
designs
Two-stage-model:
• 1. stage (intra-individual variation):
Yij
= η(xij , βi ) + ij , j = 1, ..., mi , ij ∼ N(0, σ 2 )
• 2. stage (inter-individual variation):
βi = β + bi , i = 1, ..., N, bi ∼ Np (0, σ 2 D)
Mixed Effects
Models
Mixed Effects Models
T. Mielke
Information
Approximation
Experimental
designs
Two-stage-model:
• 1. stage (intra-individual variation):
Yij
= η(xij , βi ) + ij , j = 1, ..., mi , ij ∼ N(0, σ 2 )
• 2. stage (inter-individual variation):
βi = β + bi , i = 1, ..., N, bi ∼ Np (0, σ 2 D)
• bi and ij are assumed to be independent.
Mixed Effects
Models
Mixed Effects Models
T. Mielke
Information
Approximation
Experimental
designs
Two-stage-model:
• 1. stage (intra-individual variation):
Yij
= η(xij , βi ) + ij , j = 1, ..., mi , ij ∼ N(0, σ 2 )
• 2. stage (inter-individual variation):
βi = β + bi , i = 1, ..., N, bi ∼ Np (0, σ 2 D)
• bi and ij are assumed to be independent.
• Aim: Estimation of β.
Mixed Effects
Models
Information Approximation
T. Mielke
Information
Approximation
Experimental
designs
Individual observation vector:
Yi
= η(ξi , βi ) + i ,
- mi - number of observations,
- ξi = (xi1 , ..., ximi ) - experimental settings,
- η(ξi , βi ) := (η(xi1 , βi ), ..., η(ximi , βi ))T .
Design matrix:
Fβ := (
∂η(ξi , βi )
|βi =β ).
∂βiT
Mixed Effects
Models
T. Mielke
Information
Approximation
Experimental
designs
Information Approximation
• Optimal designs for estimating β → cov (β̂)
Mixed Effects
Models
T. Mielke
Information
Approximation
Experimental
designs
Information Approximation
• Optimal designs for estimating β → cov (β̂)
• For Yi with density fYi (yi ) and
l(yi , β, θ) := log(fYi (yi )) :
Maximization of the Fisher information
Mβ = E(
∂l(Yi , β, θ) ∂l(Yi , β, θ)
).
∂β
∂β T
Mixed Effects
Models
Information Approximation
T. Mielke
Information
Approximation
Experimental
designs
• Optimal designs for estimating β → cov (β̂)
• For Yi with density fYi (yi ) and
l(yi , β, θ) := log(fYi (yi )) :
Maximization of the Fisher information
Mβ = E(
∂l(Yi , β, θ) ∂l(Yi , β, θ)
).
∂β
∂β T
• Here:
Z
fYi (yi ) =
Rp
φYi |βi (yi )φβi (βi )dβi .
Mixed Effects
Models
T. Mielke
Information Approximation
Information
Approximation
Experimental
designs
For η nonlinear in βi :
• Linearization of the model (1):
Yi ≈ η(ξi , β) + Fβ (βi − β) + i
→ Heteroscedastic nonlinear normal model.
Mixed Effects
Models
T. Mielke
Information Approximation
Information
Approximation
Experimental
designs
For η nonlinear in βi :
• Linearization of the model (1):
Yi ≈ η(ξi , β) + Fβ (βi − β) + i
→ Heteroscedastic nonlinear normal model.
• Linearization of the model (2):
Yi ≈ η(ξi , β0 ) + Fβ0 (β − β0 ) + Fβ0 (βi − β) + i
→ Linear mixed effects model.
Mixed Effects
Models
T. Mielke
Information Approximation
Information
Approximation
Experimental
designs
For η nonlinear in βi :
• Linearization of the model (1):
Yi ≈ η(ξi , β) + Fβ (βi − β) + i
→ Heteroscedastic nonlinear normal model.
• Linearization of the model (2):
Yi ≈ η(ξi , β0 ) + Fβ0 (β − β0 ) + Fβ0 (βi − β) + i
→ Linear mixed effects model.
• Fisher information: Assume the new model is true.
Mixed Effects
Models
Information Approximation
T. Mielke
Information
Approximation
Alternative:
Experimental
designs
• With Eyi (βi ) := E(βi |Yi = yi ):
1
∂l(yi , β, θ)
= 2 D −1 (Eyi (βi ) − β).
∂β
σ
Mixed Effects
Models
Information Approximation
T. Mielke
Information
Approximation
Alternative:
Experimental
designs
• With Eyi (βi ) := E(βi |Yi = yi ):
1
∂l(yi , β, θ)
= 2 D −1 (Eyi (βi ) − β).
∂β
σ
• For Varyi (βi ) = Var (βi |Yi = yi ) follows:
Mβ =
=
1 −1
1
D Var (EYi (βi ))D −1 2
2
σ
σ
1 −1
1 −1
1
D − 2 D E(VarYi (βi ))D −1 2 .
2
σ
σ
σ
Mixed Effects
Models
Information Approximation
T. Mielke
Information
Approximation
Alternative:
Experimental
designs
• With Eyi (βi ) := E(βi |Yi = yi ):
1
∂l(yi , β, θ)
= 2 D −1 (Eyi (βi ) − β).
∂β
σ
• For Varyi (βi ) = Var (βi |Yi = yi ) follows:
Mβ =
=
1 −1
1
D Var (EYi (βi ))D −1 2
2
σ
σ
1 −1
1 −1
1
D − 2 D E(VarYi (βi ))D −1 2 .
2
σ
σ
σ
→ Approximation of conditional moments.
Mixed Effects
Models
T. Mielke
Information
Approximation
Experimental
designs
Information Approximation
Laplace approximation:
• Idea: Approximation of
Z
fYi (yi ) =
by a normal density.
c · exp(
−h(β, D, yi , βi )
)dβi
2σ 2
Mixed Effects
Models
T. Mielke
Information
Approximation
Experimental
designs
Information Approximation
Laplace approximation:
• Idea: Approximation of
Z
c · exp(
fYi (yi ) =
−h(β, D, yi , βi )
)dβi
2σ 2
by a normal density.
• Taylor expansion around β̂i with
β̂i := argmin{h(β, D, yi , βi )}
βi ∈Rp
Mixed Effects
Models
T. Mielke
Information
Approximation
Experimental
designs
Information Approximation
Laplace approximation:
• Idea: Approximation of
Z
c · exp(
fYi (yi ) =
−h(β, D, yi , βi )
)dβi
2σ 2
by a normal density.
• Taylor expansion around β̂i with
β̂i := argmin{h(β, D, yi , βi )}
βi ∈Rp
• Problem: β̂i depends on yi → not normal.
Mixed Effects
Models
T. Mielke
Information
Approximation
Experimental
designs
Information Approximation
Laplace approximation:
• Idea: Approximation of
Z
c · exp(
fYi (yi ) =
−h(β, D, yi , βi )
)dβi
2σ 2
by a normal density.
• Taylor expansion around β̂i with
β̂i := argmin{h(β, D, yi , βi )}
βi ∈Rp
• Problem: β̂i depends on yi → not normal.
• Alternative: linearization of η(ξi , βi ) around β̂i .
Mixed Effects
Models
Information Approximation
T. Mielke
Information
Approximation
Experimental
designs
Laplace approximation:
• Note:
fβi |Yi =yi =
φYi |βi (yi )φβi (βi )
fYi (yi )
• Similar approximation to the numerator:
app.
βi |Yi =yi ∼ N(β̂i , σ 2 M −1 )
β̂i
with Mβ̂i = Fβ̂T Fβ̂i + D −1 .
i
Mixed Effects
Models
Information Approximation
T. Mielke
Information
Approximation
Experimental
designs
Laplace approximation:
• Note:
fβi |Yi =yi =
φYi |βi (yi )φβi (βi )
fYi (yi )
• Similar approximation to the numerator:
app.
βi |Yi =yi ∼ N(β̂i , σ 2 M −1 )
β̂i
with Mβ̂i = Fβ̂T Fβ̂i + D −1 .
i
• Try β:
app.
βi |Yi =yi ∼ N(β + Mβ−1 FβT (yi − η(ξi , β)), σ 2 Mβ−1 )
Mixed Effects
Models
T. Mielke
Information
Approximation
Experimental
designs
Information Approximation
Fisher information approximations
• With Vβ := σ 2 (Imi + Fβ DFβT ):
Mixed Effects
Models
T. Mielke
Information
Approximation
Experimental
designs
Information Approximation
Fisher information approximations
• With Vβ := σ 2 (Imi + Fβ DFβT ):
• Conditional expectation:
M1,β :=
=
1
1 −1
D Var (EYi (βi ))D −1 2
2
σ
σ
FβT Vβ−1 Var (Yi )Vβ−1 Fβ .
Mixed Effects
Models
T. Mielke
Information
Approximation
Experimental
designs
Information Approximation
Fisher information approximations
• With Vβ := σ 2 (Imi + Fβ DFβT ):
• Conditional expectation:
M1,β :=
=
1
1 −1
D Var (EYi (βi ))D −1 2
2
σ
σ
FβT Vβ−1 Var (Yi )Vβ−1 Fβ .
• Conditional variance:
M2,β :=
=
1 −1
1
1
D − 2 D −1 E(VarYi (βi ))D −1 2
2
σ
σ
σ
FβT Vβ−1 Fβ .
Mixed Effects
Models
T. Mielke
Information Approximation
Information
Approximation
Experimental
designs
Other possible approximations:
• For known moments E(Yi ) and Var (Yi ):
T
M3,β := Fβ,QL
Var (Yi )−1 Fβ,QL
Mixed Effects
Models
T. Mielke
Information Approximation
Information
Approximation
Experimental
designs
Other possible approximations:
• For known moments E(Yi ) and Var (Yi ):
T
M3,β := Fβ,QL
Var (Yi )−1 Fβ,QL
• Nonlinear heteroscedastic normal model:
1
M4,β := FβT Vβ−1 Fβ + S, with
2
Sjk
=
Tr Vβ−1
∂Vβ −1 ∂Vβ
V
.
∂βj β ∂βk
Mixed Effects
Models
Example
T. Mielke
Information
Approximation
Experimental
designs
• Log-Concentration:
Yi
= −β2i − xi exp(β1i − β2i ) + i , i = 1, ..., N
Mixed Effects
Models
Example
T. Mielke
Information
Approximation
Experimental
designs
• Log-Concentration:
Yi
•
= −β2i − xi exp(β1i − β2i ) + i , i = 1, ..., N
- i ∼ N(0, σ 2 ), ξi = (xi ) ∈ [tmin , tmax ],
- βi = (β1i , β2i )T ∼ N(β, σ 2 D),
- β = (β1 , β2 )T , D = diag(d1 , d2 ).
Mixed Effects
Models
T. Mielke
Information
Approximation
Experimental
designs
Example
- M1,β : Orange
- M2,β : Red
- M3,β : Green
- M4,β : Blue
Mixed Effects
Models
Design-Problem
T. Mielke
Information
Approximation
Experimental
designs
• Experiment settings ξi for individuals: xij ∈ X with mij
observations:
ξi = xi1 ... ximi ∈ X mi .
Mixed Effects
Models
Design-Problem
T. Mielke
Information
Approximation
Experimental
designs
• Experiment settings ξi for individuals: xij ∈ X with mij
observations:
ξi = xi1 ... ximi ∈ X mi .
• Population design ζ:
ζ=
ξ1 ... ξk
ω1 ... ωk
X
k
,
ωj = 1,
j=1
ωj : weight of indivdual design ξj in the population.
• Population design =
ˆ approximate design on X m
Mixed Effects
Models
Design-Problem
T. Mielke
Information
Approximation
• Choose design, such that:
Experimental
designs
Cov (β̂) ≈ Mpop (ζ)−1
is minimal.
Mixed Effects
Models
Design-Problem
T. Mielke
Information
Approximation
• Choose design, such that:
Experimental
designs
Cov (β̂) ≈ Mpop (ζ)−1
is minimal. → Optimality-criteria
- c-optimality: Φc (ζ)
- A-optimality: ΦA (ζ)
- D-optimality: ΦD (ζ)
- ...
Mixed Effects
Models
Design-Problem
T. Mielke
Information
Approximation
• Choose design, such that:
Experimental
designs
Cov (β̂) ≈ Mpop (ζ)−1
is minimal. → Optimality-criteria
- c-optimality: Φc (ζ)
- A-optimality: ΦA (ζ)
- D-optimality: ΦD (ζ)
- ...
• How to determine optimal designs?
→ Use of equivalence theorems
→ Use of algorithms
Mixed Effects
Models
Design-Problem
T. Mielke
Information
Approximation
Experimental
designs
Theorem
A design ζ ∗ is ΦL -optimal for estimating β ∈ Rp if and only if
the function gζ∗ (ξ) is for all ξ ∈ X m nonpositive:
Tr [Mpop (ζ ∗ )−1 LMpop (ζ ∗ )−1 (Mβ (ξ) − Mpop (ζ ∗ ))] ≤ 0.
Mixed Effects
Models
Design-Problem
T. Mielke
Information
Approximation
Experimental
designs
Theorem
A design ζ ∗ is ΦL -optimal for estimating β ∈ Rp if and only if
the function gζ∗ (ξ) is for all ξ ∈ X m nonpositive:
Tr [Mpop (ζ ∗ )−1 LMpop (ζ ∗ )−1 (Mβ (ξ) − Mpop (ζ ∗ ))] ≤ 0.
• For D-optimality: L = Mpop (ζ ∗ )
Mixed Effects
Models
Design-Problem
T. Mielke
Information
Approximation
Experimental
designs
Theorem
A design ζ ∗ is ΦL -optimal for estimating β ∈ Rp if and only if
the function gζ∗ (ξ) is for all ξ ∈ X m nonpositive:
Tr [Mpop (ζ ∗ )−1 LMpop (ζ ∗ )−1 (Mβ (ξ) − Mpop (ζ ∗ ))] ≤ 0.
• For D-optimality: L = Mpop (ζ ∗ )
→ In our situation:
- Individual information: Mβ (ξ)
- Population information: Mpop (ζ) :=
k
P
i=1
ωi Mβ (ξi ).
Mixed Effects
Models
Example
T. Mielke
Information
Approximation
Experimental
designs
• Log-Concentration:
Yi
= −β2i − xi exp(β1i − β2i ) + i , i = 1, ..., N
Mixed Effects
Models
Example
T. Mielke
Information
Approximation
Experimental
designs
• Log-Concentration:
Yi
•
= −β2i − xi exp(β1i − β2i ) + i , i = 1, ..., N
- i ∼ N(0, σ 2 ), ξi = (xi ) ∈ [tmin , tmax ],
- βi = (β1i , β2i )T ∼ N(β, σ 2 D),
- β = (β1 , β2 )T , D = diag(d1 , d2 ).
Mixed Effects
Models
T. Mielke
Information
Approximation
Experimental
designs
Optimal Design - Analytically
Simplest situation: One observation per individual: ξ = (x).
• Information matrix Fβ (x)T Vβ (x)−1 Fβ (x):
c −b
g̃ζ (x) := Fβ (x)
Fβ (x)T − 2(ac − b2 )Vβ (x) ≤ 0
−b a
Mixed Effects
Models
T. Mielke
Information
Approximation
Experimental
designs
Optimal Design - Analytically
Simplest situation: One observation per individual: ξ = (x).
• Information matrix Fβ (x)T Vβ (x)−1 Fβ (x):
c −b
g̃ζ (x) := Fβ (x)
Fβ (x)T − 2(ac − b2 )Vβ (x) ≤ 0
−b a
• Optimal population designs:
(x1 ) (x2 )
ζ =
,
0.5 0.5
∗
with x1 , x2 ∈ [tmin , tmax ], such that
σ 2 + cov (Y (x1 ), Y (x2 )) = 0.
Mixed Effects
Models
First Order Kinetics
T. Mielke
Information
Approximation
Experimental
designs
M1,β -Information:
(1.70) (24.00)
→ ζ1 =
, δ2 (ζ1 ) = 0.9794
0.5
0.5
M2,β -Information:
(1.13) (11.98)
→ ζ2 =
, δ2 (ζ2 ) = 1.0000
0.5
0.5
M3,β -Information:
(0.93) (11.99)
→ ζ3 =
, δ2 (ζ3 ) = 0.9949
0.5
0.5
M4,β -Information:
(0.96) ( 6.00)
→ ζ4 =
, δ2 (ζ4 ) = 0.9895
0.47
0.53
Mixed Effects
Models
First Order Kinetics
T. Mielke
Information
Approximation
Experimental
designs
Yi
=
kei
Vi (kei −
Cli
Vi )
(exp(−xi
Cli
) − exp(−xi kei ))exp(i )
Vi
- i ∼ N(0, σ 2 ), ξi = (xi ) ∈ [tmin , tmax ],
- kei = ke exp(b1 ), Cli = Cl exp(b2 ), Vi = V exp(b3 )
- (b1 , b2 , b3 ) ∼ N(0, σ 2 D), D = diag(d1 , d2 , d3 ).
Mixed Effects
Models
T. Mielke
First Order Kinetics
Information
Approximation
Experimental
designs
M2,β -Information:
(0.10) (4.18) (24.00)
→ ζ2 =
, δ4 (ζ2 ) = 0.55
0.33
0.33
0.33
M4,β -Information:
(3.10) (5.18) (24.00)
, δ2 (ζ4 ) = 0.66
→ ζ4 =
0.61
0.08
0.31
Mixed Effects
Models
Conclusions/Outlook
T. Mielke
Information
Approximation
Experimental
designs
• Conclusions:
- Motivation of model approximation
- Different information → different design
Mixed Effects
Models
Conclusions/Outlook
T. Mielke
Information
Approximation
Experimental
designs
• Conclusions:
- Motivation of model approximation
- Different information → different design
• Outlook:
- M2,β -Information realistic?
- Laplace approximation in βi ?
- Efficiency of algorithms?
Mixed Effects
Models
T. Mielke
Information
Approximation
Experimental
designs
Thank you for your attention!