'.
o:a...
..-- .
~
,
.'
e
f.:
,f.
~
.
.
,
"
,
-- _"
:to
~
,
...
.
.
,
,.
'
CONSIS'rENCY OF PPJW<IETER-ESTDIATES IN A
LINEAR TIME- SERIES MODEL
~
,
..
•
. -1-
by
FriedheIm Eicker
University
of~grth
~.-. -
Carolina
,!.
.
'
- ~.
-~
;
.. :
.
~~.
'J
r.
This research was supported by the Office of Naval
Research under Contract No. Nonr-855(06) for research
in probability and statistics at Chapel Hill. Reproductionin whole or in part is permitted for any purpose
of the United States Government
"
[
!
i
rt-·
.
\
.
.~
'.
Institute of Statistics
. Mimeograph Series No. 265
October, 1960
J
7'
-,
,
..
'.
"
e,"
Consistency of Parameter-Estimates in a
l .
Linear Time-Series
Hodel
.-a=-=m eez-c.
..
..........
___
••
•
#
•
rr
••
." • __
~
....
~.~~
".~"=I
~
by
-.
Friedhelm Eicker
~HE
university of North Carolina*
: ....
.
LIBRARY
.U1UV
}t
0
1. Introduction and results.
•
•
I
,
·The equation of linear regression under consideration is
.~
r
(1)
••• -+,
••• +
~ qxt+e
q
t
'
~;
r
The quanitities
y, Y l' ... , Y1'
o
-
-p
as ~elr-as the 'regression
variables are considered as given constants in a practical problem.
The errors
e
t
are independently distributed and have mean O.
Their second and fourth moments are assumed to be inclu'ded betvleen
,~
~..;,
i
tt'10 positive constants independent of' t .
«1' ••• , «p ,
~l
' •••. '
Pq
~.
;
For the unknotm constants
consistent estimates are wanted.
It is assumed that the process described by
(1)
is stable, i.e.,
that all the roots' are in modulus smaller than one in the equation
,
(2)
••• - «p-l~ -«p
=0
;
•
f
i
!
!
~
:f
I
*
On leave from University of Mainz.
"
1. This research ~as supported by the Office of Naval Research under
Contract No .. Nonr-855(06) for research in probability andstetistics at
Chapel Hill. Reproduction in whole or in part is permitted for any purpose of the United States Government.
..
-.
~,~ .
.
•
,
;.. 2 -
.
..-
-.
-."
.
The equations, (1)
can be described in matrix notation, writing for
"
any
N,
t
1, 2, ... , N :'
=0
"e.
... , xiN')1
" '1 .. 1 ., 2,
• ••
J
, •• , q,
~q )t
t,
, ,'and as, regression matrix
"
i ,r:·,·
~',
','fl'
'
"._The subscript N on vectors and
eN x
matrice,~
t,
,
'r; ,
I
.,'
I
,,:J,
I"
&:
,
1-
f'
q).
vri1l
be-9mittedwh~re confusion
"i,s unlikely; the prime denotes the transpose.
(1)
.Equation
as~umptions and vrith different methods.
1943)
Mann and Wald (
proved the consistency of'the least squares estimates
&:
(a1 ,a2 , '0' .' a p )" in thespecia1 case
_1'0' vI.
Andsl'son gives
in his
notes
some conditions that' assure consistency.
q = 1,
<[,\.7,
'.
!: .
[J.IJ,
,
!
,
"~
has been treated by several authors under different
-
a of
;.
t'
,.
~'
'f
XI =(-1,1, ••• , 1).
1949, p. 82)
.
For instance the distur-
).
t
;
bances, tt
,,.
are assumed to be identically and normally distributed
1.
,
. and the matrix
(5)
lim
N·~oo
M,
'N-Ix IX
, is assumed to exist.
. •1!.~N •
,
0
(N). . A
in the notes (188),
i
ll1our'notation:
t
i
,
.,
•"
This implies
Simi1~r
t. '
t.i
,
,
t
f
I
.. 1.
~.
"
·.ssumption concerning the exogenous variables
~iN has been made in a paper by Koopmans, Rubin and Leipnik ([loJ ,
j,
;.
i·
,
;
'.
~. '
,",,'
f
I
,
. f'
..
-
e·',,······,
.,'1950,
1.34,
3.68).
p.
eq.
.
0
"
!k.,!L elN = o(N);
J..
)
r
..
.J ,
1
In our notation it is required that
"
,
L is given by (9 rand 8 = 0, 1, ......
existence of (5) . is also assumed in a paper by T
CI
vi.
lJhe
Anderson :and
.H. "Rubin <["2'.], 1950, p. 572, Assumption F).
"While all papers mentioned so far do not make use of spectral
theory, this method leads to a number of conditions of a different
nature for the exogenous variables ~
i-
(£7J,
So for instance ,." Ch'enander
i954) and Grenander and Rosenblatt
<[""8_7,
["'9_7, 1956 and
1957) consider properties of estimates in linear stochastic difference
, ,"
/ submitted to a stationarity assumption'
equations under the conditions that
,
.as
N ->00, ¢(r) / '
N+h
.
.
¢<;)
N
.-.
_> 1-
and
(
~-
.l
I
has a limit.
' .·(r)
Coil'
~'2
(r)
our notation
__
,
...
~(r»
"'N
(~rN)"
' is assuF£d to be real
(see (13».
and. h is any positive integer.
~'A
Both rand
S
and e~uals
~
in
run from 1 to q
simpl'e comparison "lith our condi-
tions given below is not possible because in our model theTis would
jn'volve parameters
",
0..,
which are assumed to be unkno"ltrn.
1 ·
On the
other hand a process, with autoregressiva disturbances' 'like (1) whi<.:ll
. is stab~e, can asymptoti~ally also be considered as a stationary pro,,
cess.if the errors are identically distributed.
"The rr,ethod and notation used in this paper is similar to that of
Durbin
<Lb.?, !:5J,
.
1959 and 1&60), 'Nho considers various models
I
"
,~
f
I
,.f
f
!
[
r
f
It
..
-;
.
~ ..
'-:.
•
.. 4 -
"~
and· their properties l applied in time series analysis.
Section 2 of this paper contains some
le~~as
helpful to prove the main
theorem in Section 31 whi?h states the basic conditions under which
. consistency c~n be assured.
Some narrOVler conditions are derived
· from ~hese, which may be easier to verify in some applications.
It
may be emphasized that these conditions shc)'tv the existence of consistent least 'squares estimates also for some expqnentially increasing
· regression vectors • This case seems not to have been handled so far.'
· Xt is not included in the conditions of Grenander and· Rosenblatt,
"which seem to be the broadest given up to nmv.
t·
..eIn.Section 4 applications are made to polymonia11 exponential" and
t
f
i
,.f
·r~·
trigonometric regression.
...
t.
i·
~
Allresu1ts'presehted here are 'obtained by means of elementary theo'rems of matrix theory.
It will be noted that all·the proofs are '
simple and straightforward.
This fact suggests that by refinement 'and
extension of the method further results maybe obtainable ~
~
'.
.,,..
.
~
Thus fur-
1.~
ther asymptotic properties can be treated for the model studied here
as well as for related models, e.g., asymptotic distributionst
But
also the eJ=plosive autoregression model, or considerations about the
'goodness of the asymptotic approximations for finite samples can be
.~.
~ .
•
handled. 'fork in these directions is under way.
~
.'
'.
'
•
....
·2.
.
.
- S-
Notations and normal equations •
---~--
let L
~
&::
..
(9)
o
0
".
'0
a1yoof-a 2 Y-1 + ... + o.pY1 _p
0.2 Yo
+ ••• + apY2_p
• ••• •
o
••
••
:
o
•
0
(Nxl) •
Then (1) can be written
(11) 0
=(
~. avLv ) 1. + l ~
Xli
v=o
.,understanding
a
o
as
= -1.
~!
'
The soluti.on with respect to l ' leads to
: t
;
,
.
~.
•\
where'"'
'•0
\.l. L
v
•
v
~The \.l.vts are determined from CC- l
=
I
..I
'.
e',
-cr. ~p. =1
.
o.
,0
a1
'.
(..1.1
so
~al(..l.l + (..1.2
(14)a 2 ....
~3 .. -,cr. 2(..1.1 - ~(..I.t (..1.3'
• • •
..
• • • • • • • •
r.
I
• •
-..
1
. -a (..1.p_l + (..I.p ,
I'
I•
v
==
1 1 2, ••• ,N-p-l•..
The homogenuous equat,ions are satisfied by the unique real solution
,
-
v>pd-"l
'
where the summation is to be taken over all different roots ~i
of
is the multiplicity of ~i' thenpi(v) is an arbitrary
i
. ' polynomial of degree;: ffi -1 in v •. Together they involve p coni
stants i-Ihich are used to aclf\ut the solution of (1.5) to th3.t of
(2.) •. If
ffi
.
.
It is seen that
(14).
(..I..
l.
is a uniquely determined polynomial of de-
a,' s' for i, j = 1, 2, .... ,p. It fo11.01<1S further
J
from our assumption of stability (all 1~i I:: .; < 1) that*
gree
(17)
i
in the
I (..L I
v
*The
constant.
<
const
~v
for all v ,
symbol const in mathenlatical expressions denotes a certain
IJ.- --> 0
V
as v
~
00.
Ci ~~ 2.
:This has the immediate consequence that
as·N ~
00" .
i.e.,
.-
the initial values .- Yo ' ••• 'Yt-p are damped out and hence there is
-'no loss of gene:r::ality in putting
I
=
0 . .
In passing we remark'that addition of all the eyuations
(14) and
(15) Ie ads to
-
'1
1-0:1-0:2 -" •
.. O:p
=
co
Z
v=o
IJ. v •
-.
"The least squares estimates of ~ = (0:1 , a2' ••• , O:p)'
d~noted
by ~ and
and ~ are
k respectively. It is convenient to introduce
_~e ~atrix
...
Y = (Ly,
2
Ll' •.. , L Py ) •
-
-
Then tbe normal equations, derived from the
miminum condition for &2,
.
-
namely
~ (e_.2)= ·..~~
(e~
: OJ. i
ao:
.A
.... )
i
~an
ot'j
=
1, 2, ••• , p,
j
= 1,
2, "f, q '•
be written for any N
-
(20) ll1" = yaYa
.... + YIXb
..
"
..
Here (11) has been used ll1 the form
e
(22) ....t
= ...y
.. Ya
.... - XA~ •
I
.
'": 8 ~r~tiplying
(22) by It
resp.
XI
and
a~ding
lsads to
.'...
~
.
These tW9.
matr~
.
equations are the subject of the lemmas in the next
sect~on.
As a. matter of fact, rank X has to be
all N
othe~vis~
assu~£d
equal to q for almost
the estimator b is not even uniquely determined.
Abbreviating,
(24)
can hence be written for all N > N ,
o
say,
·
)
( 26) P-1·
Xt & = P- 1
XIY(a-a
+
...
--
In the next paragraph the consistency of the left sides and of the
matrices of the coefficien~s of ~-~, £-~ in (26)
and
(23)
(the.
latter after multiplication' with a suitable factor) will be shown.
The. left
si~es
will be shown to converge in probability to
£.
.
.
minimum characteristic value of the coefficient matrix Ivill be
later
~n
sho~vn
,
.'
to be bounded away from 0 uniformly for
N > No under
suitable conditions, which finally proves the consistency of
2.
The
.
gon~i~y of expressions in the normal equ~ti~~~.
I
. For later purposes ive need the follm-ring lemmas.
~
-
and b.
.
.
.. ....
~
,.,
,.,
..
".,.
.. y'-r-- • ".
. --..:...'
,'.'1
"".:"
.
.'~
.e
.-
Var (p":1/2X1t )
stant is independent of N.
'.
.'
,:
2
.' .E~t = O' and.,E(st)
< const ,where the con-
It is assumed that
< const
for all t.
Remark: This lemma applies also to ordinary linear regression without
..............
lagged variables (p = 0) ~6-7. The necessary and sufficient· condition
tor the consistency of the least squares estimates of the regression
parameters
is easily seen to be A.
(p) --> 00.
.
.
DUn
•
.
Proof: (I) Clearly E(p-l/2x1e ) =
et"'I:
p-I/2X1Ee
0
=:
. . . . .
p-l/2XI e
-
(II) The ·variance of each component of the vector
. is bounded if and only if E(-e rxp-Ix, eJ < const.
Bec,::use of the as-
~.on E(£~).< oonst, that :xpreSSi:n is essentially
=
tr I . = q, which is constant.
q
This completes the proof.
Remark:
•
~ .a:.-~
If q 158mall, A.min (p) can be ivritten dOl1n explicitly •. The same may
hold for larger qrs if P has a simple pattern or if something is
kno~n
about its limiting matrix for
for . A..
(p) -->
mn
r!.t! , X·I '-';.I.
=~
Necessary conditions
are for instance that --1
x~~
00
Ix.
""~
N-? co.
X.,
~
'Quattatively speaking,
~>
00
00
and
for all i, j = 1, ""
A..
(p)
llll.n
~~ CO
"
q, i
means that the
=1'
j •
col~~ns
of
X tend in modulus to infinity and that they must not be at the same
~t~TH·F·H~fS:~3 LiDR:~PY
4~ '~ ~
i
··
_~
t·
"r-'
~-.
-10 ..
\
,
l·
. t ..
•
(.
"
f-
I-
.~e too close to linear dependence.
i"
r.
,.
-::
i"'
.f
· . f
..- r~
f
As a sufficient condition the method of
..
Gersgo~in
.:;J
" t:·
assures that
.
;.
,
"
°t,."
(
~.,
. !:
,-1.
. 2·
~. ' N' -];
il,
.J.
Jr:L
I'd
N x. N I ~
-.oJ..,-:L,
for all i
00.
= 1,
2;
..
C),
• • • J
as
N
-;>
0:>.
'
~
;t
f"
Marcus ~12~, 1960, p. 11)
(See e.g.
"'
•
"":Inorder' to find the asymptotic b:.havioul' of . 'p-l/2X'Y ",it is helpful
.' to introduce the folloHing matrices.
". ~
.~
...
}..
Let
f
,.f
\
.."
1
.
,"
.-
t
...z = Of,... ,
li
.
.r
= CXLi •
.. t
~'
Then
~'-
.-
:".
;,
,:
and
~
i
r ..
Y = Z +W.
::
..
.~:
Clearly E(p..lXIY)
=
P-1X1\V.
The variance of each matrix
element is bounded if and only if the expectation of the modulus of
;,
each column vector is bounded.
e-
that .ID
(!.'(Lf)kCIXP-lxIC~~)
< const q A
H~r6
<
canst
tr(l..l')kCIXp-~XtCLk)
«LI)kCIXp..lXfCLk) < const~ for any k out of 1, 2,
we use the fact that for any
.' max
••• , p.
N0"t:l, we have, using E( ~~) < const for all.
.~ -
•
'\
-11-
,-
square matrices ,A and B, where B is
'">.
'. max' ' ( A
IBA)
.
{min)
Lemma 3:
~-~
<
(»
A
,max
(min)
sy~netric
(AlA) A
max
..
(B);
".' .
(min)
For any N
o < const
< '" (CCI) < const <
00
•
There is a vector x of modulus
Proof:
........
we have
1
such that
~
"f
t
••
~
i.
=
-
XICICX
. '·1
•
"
~.
L
(:
l
•I'
This has the upper bound
I,
, l- '
r, '
l
co
'const 2:
.;"n'm < const)
1l,m=O
~py
.
mmcing use of (11)
and the Schwartz inequality.
f·.,
Similarly
·\
,I
.-'J·.. .
"
~.s.t"~...._""'''''~'''''
'1
min
(CIC)
"
".As a consequence of lemmas 1 and, 2 we have the
i "
-Remark:
tends to
As N
-~
co
the set of the normal equations &iven by ,(26) ~
(This of course does not yet state the existence of the p- limits, of
...
. a ,andb
.... ).
.e-
".
We 'also have
LeIl1l11a'"".'--:.0
4: E (y Ie)
...
-.s _
=
0 :.
The vari ance of each component of this vector
satisfies with a ~onstant independent of Nand
,
I
2
i
the inequality
2
,', E(ll(LI).J.~) <canst (!! + N) <canst (N+i1(p», i= 1,2, ... , p.
Here' :\ 1 (p)
: '~o..fl
The
P.
is the largest characteristic root of
ith
component 0.1'
".
The first
~i s
term is linear in the
and hence has
0
'k
The second one is a sUm of terms containing §.' L £,k
,are
~ree
2
~t
of pure qu&dratic terms
expectation.
:: 1, 2, ... , which
and hence also have
0 expectation.
, 2
As to the variance, we m~ke use of the uniform boimdedness of E(e ).
He~ceE(~'L~)2 <canst ~1(L,)iL~< canst w2
,
t
Further
t'L\:r~ 'Lic~ is a cubic form all whose terms contain at least one e
t
in the first pO~'Ter; hence the expectation is
,
i
E(~'L C£)
'
2 '
only the squares of the terms of
zero. !inally, in
"i
elL Ce give a contribution
. ,,
)
"
whose sum is smaller' than a constant times the sum of squares of elements
;'01'
__
LiC,
Le~
< const tr
(N+(lJ-l)~2.~
(N-2)
, ,1
C1(Lt)iLic < const
11 2 + ••• ) < canst
2
tr CtC:: canst times
NCI + ~ +l;2+ ...... ) < canst
N.
-"._- -._-_ .... _ ._-«1' _. _. _._._. _ '.,:._ •
t
t.
a'
i'
,
,
.
r
i
- 13
,'. e
~
...•
Lemma , •
E(Y'i:) = "fiX. The variance of each matrix' element is
.
.
.
2
:'smaller than canst tr P or canst max
!.k'. The constants do not!
k=lJ •• ,q,
• •_ _ IlQoel:a . . . . ~
'depend on
,
N.
,
.",:.
': . Proof':
The varfance of the
C?~st~~c
<
Lj(Lt )jcl~ <
'-".
(j, k) element in Z'X
const,;~,
using lemma
3.
",
"Le~a
6:
.- .,
,.. ..
-
-..
E(YtY) = E (Z't.) + vJt\'l.
For the (j, k).th matrix element
hOlds:
N
E(Z't Z)J',', k = 2:
2 N-s- max(j,k).
s=l
":
'Be &s)
Z
r=O
rr . IIrr+ tj... k,
II.
I
.
" ',w.e
Iilod~lus is
N.
< canst
The variance of the
.
'j2
k2
.
, .var (YIY)j,k < canst {(L~) +(I, E) +N(11J.:i1 +
canst i ~2 + N I!! I +
N'
(j, k).·th
element is
.
,
I r;~I) + N } <
,
1
~'
!.,..
F
i\.
independent of N, j, and k.
with constants
,!!?2f! ~Any element in
,Z'\v ha~ 0 ex:pe.ctaxion~
>: '
This proves the first
;.
.
'
;
statement.
.
;
!. .
It is helpful to introduce
them to be equal to
O.
~ 's
Then
.
"II
with negative indices and define
(Ln C)'
r,s
=
~
r-s-n '
any n
c
1, 2,
... ,
.
and'
:-
;
.'
,.
I
i
e.
.
..
.
..
.
"
- 14
e
'
\
which proves the second statement.
; const
,,
'.
~
.'
-.
~
."
,,'
..-:
The modulus of the second ,sum is
~ Jj~kt . ~ ~2v.< COtlst..
,,=0
,
.
.
E(e 2s ) .< const for aIls.
the last statement it is convenient" to write
Furthermore,
'To
p~6ve
tjz,
m.J, k = E(Z'Z).J, k
- .
'_z, =
...J
.
O
~j
forLj! '
Then
"":-.
E. { (ztz -'. m. )2
. -J-'
k
-
Jl<:
(z.tw)
-J-k
+
2
+
(wtz)
-J-k
2 "
+ 2(ztz ..m'k)('~'ik'z. + 't'1~z,) +
-J~k J
- -J
-J-k
+ 2 .w!
Z.'.R,} •
....J ,z
-r-J-,K
-::;;.. .
. The first term becomes
,r
,t
.
r
!-
. ~ .
i
~.
f·
N
+
~
sf}t=l
sr t
N
{E(~~),E(e~) r~l lLr-s-j lLr~t-k
'
..< const (N +
~
I'·
N',
•
U:l (lL~_s_j!J.~_t_k +lJ.~.-t-jlJ.u-s~k)
I "
>
}
<
.
,
Z
s>t
~ 2{s-t) < const
NJ
'by making use of the assumed uniformboundedness of the first
,.t
»'
"
,
~.
';
h'
moments
'"0
.. 15 -
For any positive integers a, b, c, d it follows from the Schwartz in-
~quality
"
that
E(~~~'~:!d)= !~~LbC
------,themodulus being < const
-'
,
l~tafl :1d l
diag
{E(e;)}(l~XN)CI(i.r)~l"d '
)
< const
.
The remaining term contributes
l·
N
""
x" 2: (r1k -l·!J.t
~
-s-J.+'101·J t lJ.t -5-'ok)'
"t=l
e
"the modulus being < const N{f;!k I ... I~jl}
by
'
UShlg
(Z .
)2 < II const Z ~ 2t
t :v.kt l-'-t-s-j
~{
t ..
•
.
.
Hence finally
2
var (yty).J, k < const l
(N-F,.t
I + pI. I )
. . k. .+.w~
J + N ( I ill1
_.{f
c=>J
,
'Which completes the proof,
3. The consistency of the estimates
-a
and b •
~
.
We noVI prove the basic
e·
..
Theorem:-.
~.~.-
equation (1),
t
..
The least squares estimates a and b of the parameters in
= 1, 2, •••
\-There Yo' Y-l'
"',Y~p~l and
jt ; j = '1, 2, .•• ,p;
are given constants, are consistent if the follmving
X
,'"
;
\
0
.. 16 ...
...
conditions are satisfied:
. (A) All
t
li
IS
are independently distributed with 0 means and such that
uniformly in t
holds
..
.
'E(&~)
(B)
> eonst > 0,
E'(&~)
< const;
the modulus of the roots of (2) are smaller than 1;
if the set {Nj of those N for which
(Cl)
g (N) -->0 for N= 1, 2,
with some
..
... ,
is infinite, then
:e
.-.
-
-=-
~.
'(40)
NV1""i.Tp).........+
(N +
Al(p».-1/2
K s:: (X, LX"
~1(P)
Here
A
min (Ktl() -->00""
is the largest root of P
If the comp~~mentary set
:(C2)
A.
.2
~
.
p..
on
~.
.
a·....;> ai.p.
-
.-
on
rm
For
it is sufficient to have in'addition that the elements
of p.l XILjX are uniformly bounded for
Remark:
.......
= XtX
(N} to {Nlis infinite,
(p) '~oo is sufficient for
mn .
,
... ,
,
-
Ne'{ N}
j
z
1, 2, •••
'.
o.,
The subdivision into conditions like (Cl) and (C2)
is de..
sir able to prove the consistency in such cases where. 'X is a constant
.
vector for ,·rhich (tlO) does not hold.
Compare also the remark at the
t
.~
,'..
.
'
- 17 -
At~f example (5)
,Proof:
Case
~
in Section
::
2.
~.
4.
'".;
!
.' t·
The
equations
F
.r·
1··
.
.'
•
o·
.
·_1
can be put into the form
".'
- -:
~
.
t.'.;·
~".
.
. (43)
-
. ,wheref(N) -~o
as N ~co •
The' left side tends to the
,
'thereby
using
.
"
0
vector Lp. according to lenunas
1
and
. .CX{3:
.o..We.note by the-=way that.. ..w2 =
l-T
:::
.
..
Lemmas
.
. - .
2 and 6 indicate that the matrix in (43) tends'i.p. to
"
thereby using W::: O.
This matrix is symmetric, and it will be shown
to have a positive minirilUIn characteristic value !J.min' say.
unimodular characteristic vector to >...
nun (E(Z' z)
, (&t'tt-l.' ... )St_p+1)· rlhere £'0 :: £'-1
== ••• ::
If
.!! is the
, and '. .,.£t' =.
.
,\
•~
0 then
;-
'0
_
•
(45) Amin (E( z'
z»::
.
N-l
f
!
2
E(~'Z •. ~) > const l: E (t.{,~) } > const
t==l',
..-
- .. -
N > O.
,
::'."
.~~
- .. -.. - ---- -.- __
.
.-.:"_-~
_..
_._--~
_-
,
"
.. 18 -
'e
of the uniform boundGdne'Ss of the val'
Here assumption (A)
from beloH has been applied.
.
(46)
If
.~t
Nm'1
> const > O. '
lJ.. :: min
mn
all h
",
~e now square (43) and let
N ~oo
then, using
.
-.
Slutsky1s theorem
(Cra'~~ ["3.;.7, p. 255), the left side tends toiero~ and the right
,side is greater than or equal to
.
,
If we choose f slowly enough decreasing then because of (46)
,··e·, ~_£)2~
0', Le.,!
is consistent and
E(~f:' £.
b
A i.p., because vlith a suitable vector
_ ~
, c.
x'
p!
C
KtK~)::
(xl, Q')
\nin (KIK),
-"7
,Similarly'
l'Ir'.i:Jl
. (p)
c::
and hence With assumption
'(C1):
£2 1\.
m~n
(p) > const
J/2 f 2
> const > 0 ;
.
'respectively from assumption (C2) follows for a suitable f (N)
.'
:r2,mm
I\.
Case
(p)
l! I Q J
>
{N}
cons t > 0
0
infinite.
I
,.--.1.
..
"" 19 '.
. Squaring the normal equations, multiplied by
(48)
K2>"~n2
_:
."
(}1) «::-£t..)2 +
.(!?_~)2)
gives
J.i
.
~
.
;
,
"
.
,-
as the left side tends in probability to zs!'o as can be seen from
: , lemmas 1 and
. sam~.
4
and from Slutsky·ls theorem, the right side must do the
M means the matrix
=(YIY
XIY
'e.
: , Again by Slutsky 1 s
-, -7,
'/
theorem (Craner
Tj
.~ }"min (H) -> '
p.
2,5>
one has
.'
XIX
i.p.
,(So)
•
,I< >.:nun
by using
lemmas
S and 60
The latter matrix can be i'1ritten as
,
C'C (V, C-l X) ,
e
;.
(,
i·
where
V = (LX~, L2x~, .• e,
LPX~).
,
. ii
',
..
!
...
, - 20 ..
~'With (11):
-1
C ,=
~
E
v=O
Ct
,-
.'.
.
= K
~
•
0
••
•
•t Ct I
q
• 2••
I
a
cip!q
.
constant
ir~dependent
f g.
,.I. ',
ro,
..
,
.'.
i
The second matrix on the right is of the form (pvl) q x (p+q)
has full rank if" and only if .E.
'
0 I Ct1lq
~
"
,
...
-Iq
Q
..
(V, C..lX)
I·
L\J , it comes
'w
and
"re thus h'a-ve with le nun a, 3 and a
of N
f
':
f
"
~f
we multiply with Hand choose feN)
appropriately, the last expression
and therefore (50) remains above a'positive constant.
Case
tN]
Hence in (48)
infinite.
-- Lemmas 4' to 6 state that the normal equations (23) and (26) after
, inultiplicationi-rithj( given in (h?) tend for N-~ co
with probabilit.y:
one to the system of equations
".;
.,
The second set can be used for elimination of ~ -
f,
thus obtaining
.
~
.
- 21 ..
..•.
.
.
(IN being the Nth identity matrix).
":Squaring gives
2 2 2
. .
'. 2
(a-a)
I~).,.·
_ _
mn (E(ZIZ)} '>const (_a-_a)
..
.
. with a positive constant coming from
(t~6);
.
further, as ).l(P) N..
2
<.
g(N) ,
for a suitable f(N).- Thus follovrs a -~ a if A. (p) -~ co •
.
....
-i.p.mm
Further follO\.J's from (C2) that for any fixed
are uniformly bounded vectors fora11 j,
k
and p..1 XlvI
£.~~ ~
= 1, 2, ••• ,
are bounded.
Thus as
l?
the expressions
hence because of (17):
p,
a .. a ~ 0
,.l
i·
in (32)
also
•
,.i
t
:.
~
·i
i.p.
·This completes the proof.
•
!
i
!
!
.~
..
.
- 22 -
·e·
.-
l!0dified forms of conditions (Cl) and (C2) •
··.Let
.
.
. (55)
...
~r,s) ._ X_",,',N
_xs ,N-h
..
. ~ N
oD
co
diag
'j. I ~r,N I
e I :'IN I, r ~N I,
••• ,
I ~s,N-hl ,
r ~q, N-pJ)
•
\'lith a matrix R whosa elements are of the form (55) .we have
KIK
=DR
D,
Hence:
2
l.,,~,., (KIK) > min
x
..,........
- • 1
. -:i, N-p
].:= , •• ,q
-lmin (R)
0
.
,
Using. 'A (p) :: tr
1
P we may then replace assumption eel) by the
following stronger one on the set
(el) I
(a.)
0,
(p)
l N}
:
R(r,s). exists for all possible r, s, h,
lim
h,N
N->oo
the limiting matrix R is n0l1singu1ar,
min
(N+N~P
+ tr p)-1/2
-~
00
i
"
or stronger:
min x 2
i
-i,N-p
(trp)-1/2.~
co
plus
tr
P
--~.oo
•
,
!
1
i
ii
•
,
!
;
.,
......
~
-.
- 2,3 -
The conditions (a) and
(~)
are close to some conditions already
.
,
.
, -knovm (see .e.g. Gren~nder and Rosenblatt
.
["'9_7:
p. 233), but con-
.' ditio~ (~7 allows for a faster grovJth of the regr'ession vectors than
that admitted before.
Another set of sufficient conditions (Cl)", (C2)n can easily be
derived using the Ge1'6gorin method' as in the remarks to lemma 1 •
C
. ·These do not require the existence of any limit like (SO), just like
this existence is not required by the original assumption (Cl), (C2) •
. r
So one has for instance (Cl)lI:
,.·e'
,(2. .. Z R(r,s)
s, h · h,N
for all r, i
x~ N-p.
-2.,
= 1, .1.,
(N + 1;.r p)Ml/2 ->
00
q.
The f6llo1ving simple case may illuminate the necessity of
a distinc-
tion between different rates of growth of A1(p) in some sort
l~ke
~ha_"\:,
(the
measured by (39).
If q :::" 1 and X1,t
caseccnsidered by Mann and Wald
satisfied for any choice of g (N)o
;-11
""
7.
-"
~
const for all t
1943),
(Cl~
is not
But there is a sequence g (N)
! .
!
such that (C2) holds.
Condition (C2) holds if the following condition is satisfied
,
,
--
A1 (P)
l'A.
Dan
A -
min
(p)
(p) >< const for all N,_
arid'
·r....
•;
-_-.. co.
.
,.
....
- 24 -
e.
Some applications.
.Regression
on polynomials.
.......-
(1) "The question m~y be raised as to '~'rhich po't-Iers in a regression
'OI1polynorl1ials are allovled such that condition (Cl) , of oUr theorem
,
, j
ri~ fulfilled. Assume x =' t°l. J i = 1, 2 J ••• , qj t = 1, 2, ... " N;
it
0
Then
1
2
,'-
X =.
(S8)
jl
1
• • •
j
2,2
j1
j"
l"~'
'j . ,
2 q
j2
• • •
• • • • • • • • • • •
-
j
jl
N' 2
N
•
·•
... ,
If we vlrite
then
K*
~
which is obtained from K
~ of the first
(X,LX,
000'
rh) , by
rows and a simple reordering of the collli~s, can
p
be written
.
J
(1+1)' I
J
2 I
,K*--
..
-
'
J
(2+1) .1 • • • (2+p) I
•
•
•
•
...
J
(N-p) I
.r
J
(l+p) 1
'
•
0
~2
j
(N-p+l)' 1
2
I
•
•
I
•
J
If,I
,
I
•
•
••• (I+p) q
•••
J
•
omission
I
.,
..
j
'r'
(2+p). q
,,
t
t' • •
~ (N-P) ,2
I
I e· •
•
J
-=
•
t
•
I
N: q ,
, ....
I
j
..
(N",p)x(p+l)q
!..
.r
,
I·
i
,
.',----
-~-..,..
---~-"
------_ _--_._----:--...---_..
....
_---~-
-:-:.-;--.---
._..
\
'
.
....: ... -
;'..
.... -.
~~';~"""
.,
••
-
~25
.
.~)
-
.... (tC:1
.
,
. j -1
t< .1.-,.) J
_
'teO»~ x
,-.
• ••
.
.
.
'.
.
1
,
10
.
.
(j~ )20
C)
. j
.
(j~_ll21
•
•
•
•
•
•
•
•
•
•
.
I
.1
.'
j
..
• • '. jl P0
. (:~_l)ll
x.':
+ 1)
j
0
---
} .
{:~)
j
cr·1
(N-p) x
• •
0
.
I
.
0
•
•
I
(ji~l) pI
.'
0
"
e'
e-.
..
(:1 )
I
o
•
I
• • •
·
.
..
.• ••
.
• • •
1
J1
,.
t 1)
.. 0
(~~ )
1.1
o
• • • •
::..
• • •
• • •
• • •
.
• • • • • • • •
o oq.J J. j
C
q
-
e-
I
. 0
(~21
j p
2,
I
(~)
~I
o P I
• • •
\
. I
o
..
'.
1
t
. 2,1
j
.
.0
('
• • •
• • •
,
I
0
.
f
•
•
0
1
0
• • •
•
• • •
•
• • •
•
(:2 )1
32
..
.
,
,'~
•
... 26 ..
A
(KIK)
min
In (01)
~ 00
is required.for consistency.
As
IA.
m~n
(KIK)
'., lmin (K*IK*) I < qonst independ~mt of Ifj' for th~t require.
'
ment 'to be .fulfille d it is ~ecessary that
In the factorization (60)
(p+l) q.
sufficiently large
-
determinant.
K*
has full rank, i.e.,
o:t. K* the first matrix for
N always contains a non-singular van der Monde
For full rank of the second matrix it is obviously neces-
sBr,y that
j ~ >
(61)
(q+l.. ~) (p+1) -1"
As K* has full rank if and only if this holds for the second matrix,
in
e>
.-
we may state:
*
'I£mma
... _7:.....
~quations
For rank K = (p+l) q
(61)
.
.,
and for
.
-----
Amin (KI1\) ~co
must be satisfiedo
'We are unable to give sufficient conditions for the jf such that
~
has full rank.
~his problem.
full
ra~k
The literature seems to offer little help for
One may re~rrite the problem obs611ving that
if and only if this is true for, say
,W has
.. 27'-
el
.
j ) 1 j1
1 0
,
.
j') 11 '1-1
11 .
..... •
..
"W'~
•••
.
j1
(3 1 '0 p'
•••
) 31-11
(jll
P
•• •
(j2)0
P"l
,
I
I
•
•
I
-.--
;
..
,
(j ).
1 j1-1 - j 2
'. 1 J2+1
,
•••
,
•
I
(jl)j -lP
. 1.
I
l
• • •
"tlhich is of the form (jl+l.. q) x qp
in_ the lot-Ter rO\vs.
I
I
(j ).
1 j1+1 - j 2
1
1 J2-
(j1)' 11
.
Jl-
,
•
• • •
•.
(jq)oP~~
I
.'
Jl j +11 1
q
.
,
•••
I"
..
. j -1"j
. q
(.)
(j2)01 j1
j
•
.
I j l+1 - j q
(j).
1 J q-1
.
I
I
-
.I
I
(j2)'
J1-
11 ••• (j2) j -lP
.
I
~
of •
1,
(jq) jlP11"
and contains a number of zeros
A similarity to a "lfTronskian is apparent.
A
W~onskian,' submatrix ofM.r,~ could have been obtain~d also starting
directly from r~ asa criteria for independence of the p~lynomia1s
jl
jl j1
j2
j ;)l
J (x+1)
J ••• J (x + p)
J X
, ... J (x + p) q
,.
. "In some special cases one can determine rank
(2)
411hg
If P
.
M ~- •
= 1,' jl> j2 > ... > jq and if at least one of the fo1loH-
I
t
f-
inequalities is true:
I
r
1
II
I
I•
I
. i
- 28 -
··e:
then If contains a non-singular submatrix according to
q 1
.generalformulae (see e.g. Muir [1.3J ~ 1930~ p. 688~ § 739) •
.' 3 + e ';'~J
.With
t:f
If+1
o (~)
-n+r + ~- +
. ,
_.J
2"1'
one has
A1(P)=O(NJ'.t+)
and A.
IIlJ.n
'.~ 2" "2 +1
(KIK)=o(N J 1{-Q ) ,
.
~ 1£ ~ is the smallest nt~~ber out of l~ 2~ .•• ~ q
..
•
K
.... jK+1
•
such that
> q + 1..' Condition (el) is hence satisfied if
.
! '
o ' (N2 j v. -2q .- jl + )i.~ ->
00
..
, or
One observes that nec~ssarily
(3)
-~,
e (4)
If. q = 1
and
j1
2:
p
51
~
2 q.
one sees easily from (60)
N2(j1-P) + ' l - j l _1/2--:;;. co', i.e.
jl
~
that
.
2 p.
A general rule to obtain a lower bOtmd for A.
mJ.n
(KIK)
is gj. VEln
in
...
~min(KIK)
4I!.ma 6.
,the~mallest subscript
~,29 ..
is
0
(N2S+1) as
, . ,',:possible rov; number within
/J'
00
where S is
occurring in that (pqx pq) stibmatrix of M*
.', : which is a) of full rank, and b)
,:,,~or ,S
~
N
vlhose first :r:ow has the highest
rf • Thus co~di tion eel) holds for '28 ~~.
(j
holds the inequality S ~ min'
l'
- (q+1-r)(p+l)+1) :: S ,
'
,max
r=l" ••• ,CJ
•
, say.
.
'(S) If p :: q :: 2 one finds that in the inequality for S the equality
s~gn. holds
0
To prove this
W3
distinguish betvleen- the cases:
(<<) j2 = 2
e"'(~) j2 = 3,
(1-) j2 ;:
l~,
(~2)
(~l) jl ;: 6,
( 1) jl:: j2+1"
jl =
(. 2)
Because of (61) we need only to consider
*
are always submatrices in M
,
'.
(a) 1
."
.• ."
"
ji
(j1)3
1
16
Bjl
2
16
0
( j) j1
jl :: j2+ 2 ,
j1 ~ "
j2;: 2.
~ j2+3 •
There
of full rank and in th'e required shape:
16
2(j1 )3
S- -
o
..
.\",
'
r
f
r 0,
'I
S:: 0
:: ,S
max.
i'·
.' i.'
i .'
I
. (j1 )l~
(j1)4
0
- 0
t
!.,
i
I
i
.
I '
i
I
I
. (~1)
•
.
8j
jl
3
1
6
.
.' ·(j1)2
4(jl)2
6
6
(jl~4
2(jl)4
o
o
(j1)5
(~1)5
o
0
.:
.-.-
S=l=S
.'
.
m~
'.
-.'
>-
1
1
40
3
80
6
120
o
20
120
(-y.1)
(jl) j2··4
(j~) .. .".
J2""3
..
41
16
6 \
'.
s:
6
o\
8(jl)'J2--4
1
h~jl)j2-.3
4
(jl)j2':2
2(jl)j2-2
(jl)j2-1
(jl)j~-l
..
0, S = 0 = smax ·
".'
8 .
6
1
5 2-.:; 4
16
N
24.
1
2
1
bJ 12 .3
tv 2h .3
20
60
f 0,
s=j-4=S'
2
max •
(12)
--
e
(jl)jl- 5
8(jl)jl-5
(j1- 2 ). 5
J1-
4(j1-2).
.
JI - 5
(j1)jl~4
4(jl)j
-4
1
(jl- 2 ).J - 4
1
2(jl- 2 ) . 4
Jl-
(jl-2)j -3
1
(jl-2)ji.. 3
--
-3
(jl)jl-3
2(jl)j
(jl) jf"l
(jl)j:t-1
S ~ jl~'
= Smax
1
0
0
".
""
..r
1
4\
2L
3
6\,,1 0,
2l'
6
6!
-:.
"
.,; 31-
:-- .
'"
(f3)
"
(jl)
.
j2-2
..
8(j1)j2-2
(j2)j2-2
2(j2)j2- 2
4(jl)j -1
. 2
(j2)j2-1 "
(j2)j2-1
-"
(jl)j2-1
"
{:
""
(jl)j2+1
2(j1)j2+1
0
oJ
0
I)
(jl )j2+2
(jl) j2+2"
0
0
-.":.
S=
" j2 - 2
,-..
= Smax
•
From the preceding lemrrla follows that condition (el)
. in cases (a) and (~)
is not fulfilled
and thus consistency oftheLS - estimates can "
not be assured from our theorem in the given form.
is fulfilled in cases (7)
But con~ition (el)
if
or
or
It is possible -to continue this table for different values of p and
q which are
. values
0
general~y
in statistical applications confined to small
Unfortunately the theorem in its present form
require~
con-
siderably fast increasing regression vectors though the necessity of this
is not always plausible.
But one may expect that for certain
binations of jls no consistency can be assured.
com"~
So one might at
:least conjecture that it is necessary that K has full rank.,
.-
.
.
- 32-
.
,
--
-- ----
Cases of exponential regression.
(6) If "it
.
~
"t
ci > 1, i'
"i"
~ 1,*
2, •• " q,' the m$triX j('
.
.~
: tormed _as in (1) contains the submatrix
...• .
.".i',
'";:
.
"
..
•
. N+P
c.).
N+p-l
• • •
c.1.
>lhich h$S only rank 1
such th$t condition (el). is not fulfilled.
n
But if we consider for q 1 "t t
, t
1, 2, ...the
~
* . is
typical ro\'! of K
c~
}
.
t P
_ l)n c + +\
~
~~
.
n
.. , t c t .
Ms a rank defect if thel"l exists a linear combination between
*
.
j("
,its columns, i.e.
~ 1,
t
~
~'p
(10) '(t + p)n ct+P, (t
2, •• ,
beWeen the functions of t
• As
~hese
given in (70)
for
functions are analytic thiS condition is
ident:lOal with the vanishing of the linear combination for all t.
:rntroducing siL-nilar1y to (1)
_( ,,),cr-;
(1,"i
c , 2" c2
, ·'''J N" c),
we have like in (60)'
t
N
t
* = (t(n)
_ c-,
tt(n-l)
_
et
K
e
(~)
••
•
(n)
n
'0' i
(~)(p_l)n
(~) prY.
-
,
1
t·-
~)(p_l)n:l
1
•
0
-t '(0) e--Zt)
.•
'. N>c(n+1)
~
1
0
• •
-
••
•
(n)
n
"
• • •
...
•
0
x diag (op,
p-1
CJ
• • • J
1 )
-:i I
'
..
.'
.. :33 -
4It
, ~.
'.
where all three factor matrices, and hence K* " have full rank if
n
~
N
~
p (compare (3».
NOtv as for any
v
approximatel~
tV ct ~ conste~p (N log c) we have condition (el):
"tel
•....
fulfilled if only c > 1 and n > p.
n·
t
(7)
vIe notl1 consider x',J.
= t ). c.
• .
1.,).
,
the integers n. > po
..
~f
the c
i
are equal but the nls
).
-
If several
are different the matrix K contains
a sUbmatrix in which only this particular
c. occurs. and which re~
.
.).
.
••
sembles K* in example (1)
:and
apart from maltiplication from the left
the right t·dth non~singula+ diagonal
for this
. matrices.: Hence
.
submatrj~
the considerations of (1) apply.
Because of even greater
complications in K we cannot make general statements.
lfnow c > c? > ••• > cq >1 is assumed (ni ~ p),K is seen to be
1
.
2N
of full rank under utilization of (6), and ''''znin (K!K) = 0 (Cq ) •
. '.
.. With Al (p) =
2
0
(c1 N), (el) is seen to be fulfilled if
o
(8) For a simultaneous regression on a poiynoni~al and an exponential"
(el) is not fulfilled •. We have, "Ji th c > 1" Amin (KIK) < ~. (poly;~
no~tal in N), while A1(P) = 0 (c~) •
-.i
."
~" .
. I.
}..
.;
\.-
-).
.
,
.
;.
.. 34 ..
··e·
Cases of trigometric regression.
••
...-
+
..
-
Let . :Kit be. cos ""vit or elin wit.
As long. as no scale factors ap-
pear condition (C2) . has to be fulfilled for consistancy.·
.'
(9) If q = I
and w < 2 Rj the scalar
P is
o(N).
Also XiLjX
at ~ost o(N), hence (C2) is fulfilled, both for xt = cos wt
is
= cos wt, x
=. sin "1 t , w < 2n, p. tends
2t
it
.. to a diagonal matrix whose non-zero elements are 6 (N). The elements
and x
t
t
are a1so amos
0
(N) ,
hence (C2) is fulfilled. Rera'p\
as in (9) is unrestricted •
.(11) If p and q are any positive integers, if further in
_ , t
2i 1
X
'.
=
cos wi t,
x 2i ,t .= ·sin wi t
for i =J j and all
1'1J.
<
2n
then
again P tends to a diagonal matrix 't'lhose non-zero elements are
o (N),
and again (C2)
is fulfilled.
Acknm'1iedgements
~ .
... -...
The author is highly indebted to Professor James Durbin for dra,w-
--
. ing his interest and giving an introduction to th~ problems discussed
,
above.
:
.,
(10) If q =2
of, X ILjX
..
or
sin w t.
,
'
He is also very grateful to Professor Harold Hotelling for
continued encouragement aDd stimulation~
He finally expresses. his.
thanks to Dr. K. Murthy for hints to related literat?re, and to Mro
John Ada~B for going through the manuscripto
"
til.
,
'.
~
....
.. . aiterature :
·[1..7
T. \II. j.~derson, "l'Iotes on stochastic difference equations, II
Mimeographed notes for a course, New York 1949, unpublished •
.["2_7
T. W. P.r1derson and H. Rubin, I'The asymptotic properties of
. estimates o~ the parameters of a single equation in a complete
Stat~,
system of stochastic equations, II Ann. Math.
,
.(i950), pp. 570-582.
..
.£3_7
H. cramer, IIl'1athematical Hethbds of Statistics," Princeton
.{4_7
J. Durbin, liThe fitting of time-series models,lI Institute of
··"eO
.</
Vol. 21
1945•
.. ' statistics; Mimeograph Series No. 244, 1959.
["5J
J. Durbin" "Estimation of P~amet~rs in time-series regression
models, II J. Royal Statist. Soc., Sarll.B
£6_7
1960, p. 1)9.
F. Ei6ker, IIA necessary and sufficient condition for consistency of the LS-estimates in linear regression," Institute of
statistics, Himeographed Series. No.
t:7J
~66
, 1960. '
Ulf Grenander,· liOn the estimation of regression coefficients
:in the case of ·an autocorrelated dis ttlrb ance ;"
Ann. lvlath.
Stat., Vol. 25(1954), pp. 252-272.
Ulf Grenander and l:Iurray Rosenblatt, "Some problems in "estimating the spectrum of a time series, II Third
Berke~ey
Symposiu.m
on Hath. Stat. and Probability, Vol. I (1956), PP" 77-94.
~..'
.•
..
..,36 ...
..........•
::..
. . ··A.
'.'"
.. ::
,
'., '.. L'] J
______-, ItStatistical analysis of stationary time serlos,1t
1957•.
. John 'tr[iley, Nei-T York,
[10_7
T.
c.
Koopmans, H. Rubin, and R. B. Leipnik," Neasuring lohe
equa~i6n systems of dynafuic economics," COi'1'les' Conunissioll for
Research in Economics, Monograph No.
./:11_7
10 (1950), pp.
53-2J7~
H. B. Mann and A. lva1d, "On the statistical treatment of
.-
linear stochastic difference equations," Econometrica"
.
,
Vol.
11 (1943), pp. 173-220.
H. Narcus, "Basic Theorems in Natrix Theory," NBS Al'18
No.
57, 19600 ,
"
I~
treatise on the theory of determinants," .
Privately published, Albany, N.
Y.,
19300
,"
" .
'e
.. ~.