ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Second-Order Effect of Estimated Weights
Recall the general mean-variance specification
E(Y |x) = f (x, β),
var(Y |x) = σ 2 g (β, θ, x)2 .
The large sample approximate distribution of the GLS estimator is
√ L
n β̂ GLS − β 0 −→ N 0, σ02 ΣWLS
where
ΣWLS =
lim n−1 XT WX
n→∞
−1
,
if the working variance function is the true variance function.
1 / 15
Second-Order Effect of Estimated Weights
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
The folklore theorem says the asymptotic distribution is the same
no matter if the weights are known (β and θ are held equal to
their true values),
√ or if the weights are estimated (β and θ are
estimated by n-consistent estimators β̂ and θ̂).
regardless of the number of iterations of the GLS algorithm (the
same for all values of C ).
Such results hold only to this order of approximation (first order).
The standard errors obtained this way tend to understate the
variability associated with β̂ GLS , particularly for smaller n.
2 / 15
Second-Order Effect of Estimated Weights
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
This level of approximation suggests that
1
var β̂ GLS ≈ σ02 ΣWLS
n
Or say,
var β̂ GLS
1
= σ02 ΣWLS + op
n
1
.
n
We can sometimes go further to a more refined approximation:
1
1
1
2
var β̂ GLS = σ0 ΣWLS + 2 V + o
,
n
n
n2
where the second-order term n−2 V may capture the effect of using
estimated weights and the choice of number of iterations.
3 / 15
Second-Order Effect of Estimated Weights
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Generally, arguments to establish such second order results are very
tedious. So we will pursue some simple, special cases.
Throughout, we assume the variance function g (·) is correctly
specified, as our focus is on understanding the performance of the
first order and second order results when the model is correct.
The second order results can provide some useful theoretical insight.
However, they do not translate into improvements that may be used
in practice, as the necessary calculations are much too difficult to be
implemented easily, even for those simple special cases.
Later, we will consider the bootstrap as an alternative way of
effecting the same sort of improvement “automatically” under certain
circumstances.
4 / 15
Second-Order Effect of Estimated Weights
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Case I
When g (·) does not depend on β [Rothenberg, 1984]
Assumptions
g (·) does not depend on β;
The variance parameter θ is a scalar, and is estimated by
·
θ̂ ∼ N(θ0 , τ 2 /n);
Linear model E (Yj |xj ) = xTj β, and Gaussian distribution of Yj
given xj .
5 / 15
Second-Order Effect of Estimated Weights
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Then
var β̂ GLS
1
1
= σ02 ΣWLS + 2 V + o
n
n
1
n2
,
and V is an increasing function of τ 2 .
Heuristic argument shows that V = τ 2 ΣD , but ΣD is hard to
calculate.
The number of iterations C appears not to matter here.
6 / 15
Second-Order Effect of Estimated Weights
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
The precision of estimation of β by β̂ GLS is dictated by the precision
of estimation of θ. In particular, the more precise θ̂ is, the more
precise β̂ GLS is, to second order.
The role of estimation of θ only shows up in the second order term
n−2 V; for large n, this term is dominated by the leading term, and its
effect is negligible. For small n, however, the effect may be more
pronounced.
The form of V may be very difficult to derive, so it may not be
practical to use this additional correction term to calculate improved
standard errors of β̂ GLS .
7 / 15
Second-Order Effect of Estimated Weights
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Case II
When g (·) depends on β [Carroll, Wu, and Ruppert, 1988]
Weaker assumptions
g (·) may depend on β, i.e., a general g (β, θ, x);
The variance parameter θ does not have to be a scalar, and is
estimated by a “reasonable” estimator θ̂;
Mean model need not be linear, and errors need not be Gaussian.
(C )
Then the C -step estimator β̂ GLS satisfies
var
(C )
β̂ GLS
1
1
= σ02 ΣWLS + 2 V(C ) + o
n
n
1
n2
where the second-order term V(C ) may depend on C .
8 / 15
Second-Order Effect of Estimated Weights
,
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
In general, V(3) = V(4) = · · · = V(∞).
In addition, V(2) = V(3) = · · · = V(∞) if either
g (·) does not depend on β and the errors are symmetrically
distributed;
(0)
β̂ GLS is β̂ OLS .
In addition, V(1) = V(2) = · · · = V(∞) if both the above conditions
hold.
9 / 15
Second-Order Effect of Estimated Weights
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
When V(1) 6= V(3), there is no general ordering; both V(1) > V(3)
and V(1) < V(3) are possible.
If g (·) does not depend on β and var 2j = 2 + κ for all j, then for
all C and for some matrix V∗ ,
V(C ) = (2 − κ)V∗ .
10 / 15
Second-Order Effect of Estimated Weights
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
There is no “optimal” number of iterations of the GLS algorithm, in
a second order sense:
It could well be that iterating past C = 1 could be detrimental!
The usual practice of taking C = ∞ could be suboptimal in
some situations.
After C = 3 in general, or after C = 2 or 1, additional iteration
has, to second order, no effect.
The form of V(C ) is always complicated, and again bootstrap
variance estimation is preferred.
Main point
From a second order perspective, how one estimates θ does matter!
This has motivated research into determining the “best” way to
estimate θ under different circumstances.
11 / 15
Second-Order Effect of Estimated Weights
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Recall
This approach is based on the first-order approximation that
√ n β̂ GLS − β 0
converges in law to some limiting Gaussian distribution.
The central limit theorem does not need to be written in this way.
All we really need in practice is that
·
β̂ GLS ∼ N (β 0 , Σn )
where Σn is some matrix we can calculate.
12 / 15
Second-Order Effect of Estimated Weights
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Outline of bootstrap
Depends on the assumption that
j =
Yj − f (xj , β)
, 1 ≤ j ≤ n,
g (β, θ, xj )
are i.i.d. (note: not true for Poisson regression, for example).
At step C + 1, get residuals
sj =
13 / 15
(C +1)
Yj − f xj , β̂
(C +1) (C ) .
g β̂
, θ̂ , xj
Second-Order Effect of Estimated Weights
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
For b = 1, 2, . . . , B:
sample s1b , s2b , . . . , snb with replacement from the finite
population {s1 , s2 , . . . , sn }
Form the bootstrap responses
(C +1) (C ) (C +1)
b
, θ̂ , xj sjb .
Yj = f xj , β̂
+ g β̂
b
Get the bootstrap estimate β̂ from these responses, using the
C + 1 iterate.
14 / 15
Second-Order Effect of Estimated Weights
ST 762
Nonlinear Statistical Models for Univariate and Multivariate Response
Summarize these B bootstrap estimates as
β̂ boot
B
1X b
β̂ ,
=
B b=1
Σ̂boot
b
T
1 X b
=
β̂ − β̂ boot β̂ − β̂ boot .
B − 1 b=1
B
A heuristic argument shows that, when the second-order
approximation works, Σ̂boot estimates
(C +1)the
sum of both the first-order
and second-order terms in var β̂
.
That is, it does “automatic” adjustment for using estimated
weights.
15 / 15
Second-Order Effect of Estimated Weights