ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Implementing GLS
Recall the assumptions of Approach 9:
E(Y |x) = f (x, β),
var(Y |x) = σ 2 g (β, θ, x)2 .
Here:
β is, as before, the vector of parameters in the mean function;
θ is a possible additional parameter in the structure of the
variance function;
σ 2 is an additional dispersion parameter.
1 / 17
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Ad hoc estimation scheme:
i
Get initial estimate of β using OLS;
ii
Get initial estimate of θ, if needed, and construct initial
estimated weights
1
ŵj = 2
g β̂ OLS , θ̂, x
iii
Re-estimate β using WLS, treating ŵj as fixed: solve the
estimating equation
n
X
j=1
2 / 17
ŵj {Yj − f (xj , β)} fβ (xj , β) = 0.
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Digression
Gauss-Newton method for WLS (including OLS):
The equation
n
X
wj {Yj − f (xj , β)} fβ (xj , β) = 0.
j=1
generally cannot be solved in closed form.
But if β ∗ is close to the solution β,
f (xj , β) ≈ f (xj , β ∗ ) + fβ (xj , β ∗ )T (β − β ∗ ) ,
fβ (xj , β) ≈ fβ (xj , β ∗ ) + fββ (xj , β ∗ ) (β − β ∗ )
3 / 17
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Omitting terms likely to be small, the WLS estimating equation
becomes
( n
)
X
∗
∗ T
(β − β ∗ )
wj fβ (xj , β ) fβ (xj , β )
j=1
≈
n
X
j=1
4 / 17
wj {Yj − f (xj , β ∗ )} fβ (xj , β ∗ )
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Write

fβ (x1 , β)T


..
X(β) = 

.
(n×p)
T
fβ (xn , β)



f (x1 , β)


..
f(β) = 

.
(n×1)
f (xn , β)



=

W

(n×n)
5 / 17
w1 0 . . . 0
0 w2 . . . 0
.
..
.. . .
. ..
.
.
0 0 . . . wn





ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Then the approximate equation may be written
X(β ∗ )T WX(β ∗ ) (β − β ∗ ) ≈ X(β ∗ )T W {Y − f(β ∗ )}
or (if the inverse exists)
−1
β ≈ β ∗ + X(β ∗ )T WX(β ∗ )
X(β ∗ )T W {Y − f(β ∗ )}
Iterative solution:
β (a+1) = β (a) +
n
T
o−1
T
X β (a) WX β (a)
X β (a) W Y − f β (a)
Note that W is fixed throughout the inner circle of iteration.
6 / 17
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Note that if the iteration converges, β (a) → β (∞) , and
T
X β (a) WX β (a) converges to a non-singular matrix, then
X β (∞)
T
W Y − f β (∞) = 0
which is the original estimating equation written in matrix form.
That is, the limit β (∞) does solve the original WLS problem.
This algorithm, suitably refined, is the default method in both R’s
nls() and SAS’s proc nlin.
7 / 17
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Instead of the nested iterations, we could solve the last equation
directly.
Note
This is not equivalent to minimizing
Sg (β) =
n
X
j=1
1
2
2 {Yj − f (xj , β)} ,
g (β, θ, xj )
because differentiating Sg (β) w.r.t. β also brings in the derivative
of g (·).
8 / 17
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
The equation can be solved by a Gauss-Newton method; a similar
approximation leads to the iteration
β (a+1) = β (a) +
n
T
T
o−1
X β (a) W(a) Y − f β (a)
X β (a) W(a) X β (a)
where the weight matrix W is now updated using the current value of
β (a) .
That is, the weights are updated within the iteration, instead of being
held constant, as in the WLS iteration; a nonlinear instance of
Iteratively Reweighted Least Squares, IRWLS (note the redundancy!)
or IWLS.
9 / 17
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Estimating σ 2
By analogy with WLS, the natural estimator of σ 2 is either
n
o2
n
Y
−
f
x
,
β̂
X
j
j
GLS
1
2
n j=1
g β̂ GLS , θ, xj
or
1
n−p
10 / 17
n
X
j=1
n
o2
Yj − f xj , β̂ GLS
2 .
g β̂ GLS , θ, xj
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
For fixed weights, the latter is unbiased.
For fixed weights and Gaussian Y , the former is ML and the latter is
REML.
The latter is reported by most software.
11 / 17
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Summary
Model
E(Y |x) = f (x, β),
var(Y |x) = σ 2 g (β, θ, x)2 .
Generalized least squares (GLS)
n
X
j=1
12 / 17
wj {Yj − f (xj , β)} fβ (xj , β) = 0.
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Motivation
Loss function
Sg (β) =
n
X
wj {Yj − f (xj , β)}2 ,
j=1
with plugged-in weights wj = g (β, θ, xj )−2 .
Gaussian distribution with mean f (xj , β) and plugged-in
weights wj .
13 / 17
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Implementations
3-step GLS
(0)
1
2
Get initial estimate β̂
Repeat until convergence, or for a fixed number of C steps:
1
2
Update the weights ŵj = g (β̂
Estimate β using WLS:
Repeat until convergence
(t)
, x)−2
β (a+1) = β (a) +
T
−1 T
n
o
X β (a) WX β (a)
X β (a) W Y − f β (a)
3
14 / 17
Update the estimate β̂
(t+1)
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Comment
There are 2 loops
In the inner loop (step 2.2), the weight matrix W is fixed
15 / 17
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Implementations (continued)
Iteratively reweighted least squares (IRWLS)
(0)
1
2
Get initial estimate β̂
Repeat until convergence:
1
2
Update the weights ŵj = g (β̂
Estimate β using WLS
(t)
, x)−2
β (a+1) = β (a) +
T
−1 T
n
o
X β (a) W(a) X β (a)
X β (a) W(a) Y − f β (a)
3
16 / 17
Update the estimate β̂
(t+1)
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Comment
There is only 1 loop
The weight matrix W is iteratively updated
17 / 17