ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Estimating θ
Recall the general mean-variance specification
E(Y |x) = f (x, β),
var(Y |x) = σ 2 g (β, θ, x)2 .
We have, so far, considered θ as a known constant.
In many cases, we instead need to estimate it.
1 / 21
Estimating Variance Parameters
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
One approach is to continue to use the Gaussian likelihood:
−n log σ −
n
X
j=1
n
1 X {Yj − f (xj , β)}2
log g (β, θ, xj ) −
.
2 j=1 σ 2 g (β, θ, xj )2
Differentiating w.r.t. θ leads to
#
"
n
X
{Yj − f (xj , β)}2 − σ 2 g (β, θ, xj )2
νθ (β, θ, xj ) = 0.
σ 2 g (β, θ, xj )2
j=1
2 / 21
Estimating Variance Parameters
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Here
νθ (β, θ, xj ) =
∂ log g (β, θ, xj )
gθ (β, θ, xj )
=
.
∂θ
g (β, θ, xj )
For consistency with earlier estimating equations, rewrite as
"
#
n
X
{Yj − f (xj , β)}2 − σ 2 g (β, θ, xj )2
2σ 2 g (β, θ, xj )2 νθ (β, θ, xj )
4 g (β, θ, x )4
2σ
j
j=1
= 0.
3 / 21
Estimating Variance Parameters
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Combining with the σ 2 equation derived earlier,
"
#
n
X
{Yj − f (xj , β)}2 − σ 2 g (β, θ, xj )2
2σ 4 g (β, θ, xj )4
j=1
2σg (β, θ, xj )2
= 0.
×
2σ 2 g (β, θ, xj )2 νθ (β, θ, xj )
4 / 21
Estimating Variance Parameters
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
The σ 2 and θ equations may be put in the standard form
n
X
DTj Vj−1 (sj − mj ) =
j=1
0
.
((q+1)×1)
We need
sj = {Yj − f (xj , β)}2
mj = σ 2 g (β, θ, xj )2
Dj =
2σg (β, θ, xj )2
2σ 2 g (β, θ, xj )2 νθ (β, θ, xj )
Vj = 2σ 4 g (β, θ, xj )4 .
5 / 21
Estimating Variance Parameters
!T
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Since β is also unknown, we must also solve the corresponding
equation
n
X
{Yj − f (xj , β)} fβ (xj , β)
σ 2 g (β, θ, xj )2
j=1
"
#
n
2
X
{Yj − f (xj , β)}
+
− 1 νβ (β, θ, xj ) = 0.
σ 2 g (β, θ, xj )2
j=1
In principle, we must solve the (q + 1) variance parameter equations
and the p β equations jointly.
6 / 21
Estimating Variance Parameters
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Joint Estimating Equations
Gaussian ML:
n
X

fβj

0

fβj
j=1

2 2
2σ 2 gj2 νβj
σ gj

1/σ
0
2σ 2 gj2
νθj
0
2σ 4 gj4
−1 0
2σ 4 gj4
−1 Yj − fj
(Yj − fj )2 − σ 2 gj2
Yj − fj
(Yj − fj )2 − σ 2 gj2
= 0.
GLS:
n
X

j=1
7 / 21
0
2σ 2 gj2
0

1/σ
νθj

σ 2 gj2
0
Estimating Variance Parameters
= 0.
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Remarks
Solve (p + q + 1) estimating equations jointly.
Approach in GLS is called pseudo-likelihood (PL) approach; that is,
some parameters are estimated by ML while others are estimated by
ad hoc but consistent methods. — GLS-PL
When g (·) does not depend on β but does involve unknown θ,
νβ (β, θ, xj ) = 0,
and the normal theory ML and GLS equations coincide.
8 / 21
Estimating Variance Parameters
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Implementation
Write estimating equations in this form, then solve using an idea
similar to IRWLS
n
X
DTj (α)Vj−1 (α) {sj (α) − mj (α)} = 0
j=1
where
DTj (α) =
9 / 21
∂
mj (α).
∂α
Estimating Variance Parameters
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Two strategies
Solve the entire set of parameters together:


β
α =  σ .
θ
Iterate between: σ
Given α =
, solve for β;
θ
Given β, solve for α.
10 / 21
Estimating Variance Parameters
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Implementation
Use Taylor series approximation,
−1
α ≈ α∗ + DT (α∗ )V−1 (α∗ )D(α∗ )
DT (α∗ )V−1 (α∗ ) {s(α∗ ) − m(α∗ )}
where
V(α) = block diag{V1 (α), . . . , Vn (α)},
and
DT (α) = {DT1 (α), . . . , DTn (α)}
11 / 21
Estimating Variance Parameters
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Iterative update
n
o−1
−1
−1
α(a+1) = α(a) + DT(a) V(a)
D(a)
DT(a) V(a)
s(a) − m(a)
Remark: The ML or PL quadratic equations are more unstable than
are the linear equations, and can be ill-behaved in practice.
12 / 21
Estimating Variance Parameters
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
A General Class of Estimators of θ
Motivation
Estimating equations for θ would be based on the residuals
{Yj − f (xj , β)}.
For squared residuals, the effect of “outlying” or “unusual”
observations can be magnified.
Other functions of the residuals might be considered.
13 / 21
Estimating Variance Parameters
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
14 / 21
Estimating Variance Parameters
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Recall
The standardized residual:
j =
Yj − f (xj , β)
σg (β, θ, xj )
Box-Cox transformation:
 λ
u − 1
h(u, λ) =
λ
log |u|
λ 6= 0
λ = 0.
Key idea
Consider forming estimating equations based on general power
transformations of absolute residuals.
15 / 21
Estimating Variance Parameters
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Key assumption
The appropriate moments of j are not dependent on xj and are
constant for all j:
E |j |λ xj = E |j |λ = constant ∀j,
E |j |2λ xj = E |j |2λ = constant ∀j.
Definitions
Define η such that
e λη = σ λ E (|j |λ )
Identify |Yj − f (xj , β)|λ as the “response”.
16 / 21
Estimating Variance Parameters
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
It follows that
E |Yj − f (xj , β)|λ xj = σ λ g (β, θ, xj )λ E |j |λ = e λη g (β, θ, xj )λ
and
var |Yj − f (xj , β)|λ xj = σ 2λ g (β, θ, xj )2λ var |j |λ
n
2 o
= σ 2λ g (β, θ, xj )2λ E |j |2λ − E |j |λ
= σ 2λ E |j |2λ − e 2λη g (β, θ, xj )2λ
17 / 21
Estimating Variance Parameters
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Estimating equations
n X
|Yj − f (xj , β)|λ − e λη g (β, θ, xj )λ
λe λη g (β, θ, xj )λ τθ (β, θ, xj ) = 0
g (β, θ, xj )2λ
j=1
defines a family of estimating equations for θ, indexed by λ.
The quadratic PL equation is obtained when λ = 2.
18 / 21
Estimating Variance Parameters
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Extended quasi-likelihood
Recall that quasi-likelihood (QL) was an attempt to give a
“distributional” justification for GLS-type estimation of β in models
in which the variance depends on β through the mean, but there are
no additional, unknown, variance parameters θ (that is, θ is known).
When θ is unknown, the idea here is to define a “scaled exponential
family-like” “loglikelihood” to be used as a basis for joint estimation
of β, σ, and θ.
19 / 21
Estimating Variance Parameters
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Recall
The log quasi-likelihood function
1
`QL (β, σ; y ) = 2
σ
Z
µ
y
y −u
du.
g (u)2
where µ = f (x, β).
Estimate β and σ by maximizing
n
X
`QL (β, σ; yj ) .
j=1
20 / 21
Estimating Variance Parameters
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Define the log extended quasi-likelihood function
Z µ
1
y −u
1
2 2
`EQL (β, σ, θ; y ) = 2
du
−
log
2πσ
g
(y
,
θ)
σ y g (u)2
2
where µ = f (x, β).
Estimate β, σ, and θ by maximizing
n
X
`EQL (β, σ, θ; yj ) .
j=1
21 / 21
Estimating Variance Parameters