Comment.Math.Univ.Carolin. 39,3 (1998)551–561
Change-point estimator in gradually changing sequences
Daniela Jarušková
Abstract. Recently Hušková (1998) has studied the least squares estimator of a changepoint in gradually changing sequence supposing that the sequence increases (or decreases) linearly after the change-point. The present paper shows that the limit behavior
of the change-point estimator for more complicated gradual changes is similar. The limit
variance of the estimator can be easily calculated from the covariance function of a limit
process.
Keywords: gradual type of change, polynomial regression, estimator, limit distribution
Classification: 62G20, 62E20, 60F17
1. Introduction
In applications we often observe a sequence of variables that at some unknown
time point starts gradually changing its behaviour. Such series we encounter in
engineering problems as well as in ecology. Sometimes, we even know from our
experience how the series behaves before and after change point. The inference
about broken line regression considered by Hinkley (1969) or Siegmund and Zhang
(1994) belongs to such problems. Recently Hušková (1998) has studied the model
where at an unknown time point a linear trend appears in the mean of an observed
time series {Yi , i = 1, . . . , n}:
i − k⋆ Yi = µ + δn
+ ei ,
n
+
where a+ = max{a, 0}, µ, δn , k ⋆ are unknown parameters and {ei , i = 1, . . . , n}
are random errors. We show that her approach can be generalized to a case
of a gradual change in polynomial regression. We consider a sequence {Yi , i =
1, . . . , n} satisfying
i
i − k ⋆ m
i p
Yi = α˜0 + α˜1
+β
+ ei , i = 1, . . . , n,
+ . . . + α˜p
n
n
n
+
p = 0, 1, . . . , m ≥ 1 are known integers, α˜0 , . . . , α˜p , β ∈ R1 and k ⋆ ∈ N are
unknown parameters. The errors {ei } are i.i.d. satisfying Eei = 0, Ee2i = σ 2 ,
E|ei |2+δ < ∞ for some δ > 0. The parameter σ 2 is supposed to be known and
Partially supported by grant GAČR - 201/97/1163.
551
552
D. Jarušková
we can assume without loss of generality that it is equal to 1. The aim of this
paper is to estimate the change point k ⋆ and to find the limit distribution of this
estimator. Therefore, we suppose that β may depend on the sample size n and
may go to zero as n tends to infinity. Similarly as in Hušková (1998) we use the
least squares method.
c⋆ in more apparent way it is convenient
To present the least squares estimator k
to express the variables {Yi , i = 1, . . . , n} with the help of orthogonal vectors
n
n
′
n
n
n
′
φn
0 = (φ0 (1), . . . , φ0 (n)) , . . . , φp = (φp (1), . . . , φp (n)) . More precisely
(1.1)
n
Yi = α0 φn
0 (i) + · · · + αp φp (i) + βn
i − k ⋆ m
n
+
+ ei ,
i = 1, . . . , n,
p = 0, 1, . . . and m ≥ 1 are some known integers, α0 , . . . , αp , βn ∈ R1 and k ⋆ ∈ N
are unknown parameters. The errors {ei } have the same properties as given above.
The orthogonal vectors may be chosen so that
φn
0 (i) = 1,
i = 1, . . . , n,
and for j = 1, . . . , p
i j−1
i j
+
C
(j,
n)
+ · · · + C0 (j, n),
(i)
=
φn
j−1
j
n
n
so that for every j 6= j ′
n
X
i=1
i = 1, . . . , n,
n
φn
j (i)φj ′ (i) = 0,
see Anderson (1971). The first few vectors are
φn
0 (i) = 1,
i 1
1
−
1+
,
φn
1 (i) =
n
2
n
i 2 i 1 1
1 2
φn
1+
+
1+
1+
,
−
2 (i) =
n
n
n
6
n
n
..
.
As n → ∞, the functions {φn
j ([nt]), t ∈ [0, 1]} j = 0, . . . , p converge on [0,1] to
functions {φj (t)} j = 0, . . . , p forming an orthogonal system, i.e., for j 6= j ′
Z 1
φj (t)φj ′ (t) dt = 0
0
and
n
1X n
(φj (i))2 →
n
i=1
Z
0
1
(φj (t))2 dt.
Change-point estimator in gradually changing sequences
The first few functions are
φ0 (t) = 1,
1
φ1 (t) = t − ,
2
1
φ2 (t) = t2 − t + ,
6
..
.
n
n
Denote X the design matrix, i.e., X = (φn
0 , φ1 , . . . , φp ). It holds

 P n
0
...
0
1/ i (φ0 (i))2
P n


0
0
1/ i (φ1 (i))2 . . .


′
−1
(X X) = 
.
.
.


.
0
0
P 0n
2
0
0
0 1/ i (φp (i))
If we denote
1 m
n − k m ′
ck = 0, . . . , 0,
,...,
,
n
n
c⋆ of the change point k ⋆ can be expressed
then the least squares estimator k
(ck̃ ′ M Y )2
(ck ′ M Y )2
⋆
c
= max
(1.2)
k = min k̃,
ck̃ ′ M ck̃
p≤k<n−p ck ′ M ck
where the matrix M = I − X(X ′X)−1 X ′ . It is clear that if the variables {ei } are
c⋆ is the maxdistributed according to a normal distribution, then the estimator k
imum likelihood estimator of k ⋆ in model (1.1) where all parameters α0 , . . . αp , β
are unknown.
The paper is organized as follows. Section 2 contains our main result. To prove
the assertion of the main theorem several auxiliary lemmas have to be stated and
proved. Section 3 presents three examples how to apply the main theorem to
obtain the limit distribution of k ⋆ in different situations.
2. Main result
The aim of this section is to derive asymptotic distribution of k ⋆ . First, for
n ∈ N we introduce variables {S n (k), k = p, . . . , n − p − 1} by
(2.1)
n
1
1 X i − k m
S n (k) = √ ck ′ M e = √
ei
n
n
n
i=k
. X
p X
n n
n
X
1 X n
i − k m n
1
1
2
√
(φn
(i))
.
φj (i)
φj (i)ei
−
j
n
n
n
n
j=0
i=k
i=1
i=1
553
554
D. Jarušková
The covariance function of the sequence {S n (k), k = p, . . . , n − p − 1} satisfies for
l≤k
n
1 ′
1 X i − k m i − l m
ck M cl =
n
n
n
n
i=k
p
n
n
n
. 1 X
X
1 X i − k m n
1 X i − l m n
n
2
−
(φj (i))
.
φj (i)
φj (i)
n
n
n
n
n
Rn (k, l) =
(2.2)
j=0
i=k
i=1
i=l
Further, for n ∈ N we define the processes {Y n (t)} on [0, 1] by
Y n (t) = S n ([nt]),
t ∈ [0, 1].
Using the limit theorem for random processes, see Theorem 15.6 of Billingsley
(1968), the processes {Y n (t)} converge in distribution on D[0, 1] to a process
(2.3)
−
Y (t) =
p Z
X
j=0
1
t
m
(z−t) φj (z) dz
Z
1
(z−t)m dW (z)
t
Z
1
0
.Z 1
2
φj (z) dW (z)
(φj (z)) dz
, t ∈ [0, 1],
0
where {W (t), t ≥ 0} denotes Wiener process. The covariance function of the
process {Y (t)} satisfies
(2.4)
−
R(t, s) =
p Z
X
j=0
1
t
m
(z−t) φj (z) dz
Z
s
Z
1
(z−t)m (z−s)m dz
t
1
m
(z−s) φj (z) dz
.Z
0
1
(φj (z)) dz
, s ≤ t.
2
Moreover, for n ∈ N we introduce a sequence {Ṡ n (k), k = p, . . . , n − p − 1} by
n
1 X i − k m−1
Ṡ n (k) = − √
ei
m
n
n
i=k
. X
p X
n
n
n
i − k m−1
X
1
1
1 X n
2
n
√
(i)e
φ
+
(i)
φn
(i))
m
(φ
i
j
j
j
n
n
n
n
(2.5)
j=0
i=1
i=k
and processes {Ẏ n (t)} by
Ẏ n (t) = Ṡ n ([nt]),
t ∈ [0, 1].
i=1
Change-point estimator in gradually changing sequences
The processes {Ẏ n (t)} converge in distribution on D[0, 1] to the process
(2.6)
+
p Z
X
j=0
t
Ẏ (t) = −
1
Z
1
t
m(z − t)m−1 φj (z) dz
m(z − t)m−1 dW (z)
Z
1
0
.Z 1
(φj (z))2 dz
,
φj (z) dW (z)
0
that is the derivative of the process {Y (t)}.
In the main theorem given below it is stated that under certain conditions
on the limit behavior of βn the asymptotic distribution of k ⋆ is normal with the
variance which may be computed from the covariance function of the process
{Y (t)}.
Theorem. Let random variables {Yi , i = 1, . . . , n} satisfy the properties of model
(1.1) with k ⋆ = [nθ⋆ ] for some θ⋆ ∈ (0, 1). Let
(2.7)
βn = O(1) and
βn2 n
→ ∞ as n → ∞.
(ln ln n)2
Then, as n → ∞,
c⋆ − k ⋆ p
k
D
√
A(θ⋆ ) −→ N (0, 1),
n
2
.
∂ R(t,s)
∂R(t,s) 2
where A(θ⋆ ) = ∂t∂s
R(θ⋆ , θ⋆ ).
−
∂t
⋆
⋆
(2.8)
βn
t=s=θ
t=s=θ
c⋆ can be defined as the
Proof: Similarly as in Hušková (1998), the estimator k
solution of the following maximization problem
′
(ck M Y )2
(ck⋆ ′ M Y )2
(2.9)
max Dk = max
−
.
ck ′ M ck
ck ⋆ ′ M ck ⋆
p≤k<n−p
p≤k<n−p
Under the assumptions that the variables {Yi , i = 1, . . . , n} satisfy model (1.1),
the variables Dk can be expressed as follows:
Dk = βn2 Ck + 2βn Bk + Ak ,
(2.10)
where
(Rn (k, k ⋆ ))2
n ⋆ ⋆
−
R
(k
,
k
)
,
Rn (k, k)
√ Rn (k, k ⋆ ) − Rn (k, k)
S n (k) + S n (k) − S n (k ⋆ ),
Bk = n
Rn (k, k)
Ck = n
Ak =
(S n (k))2
(S n (k ⋆ ))2
− n ⋆ ⋆ .
n
R (k, k)
R (k , k )
555
556
D. Jarušková
We consider a sequence {rn } such that
√
|βn | n
→ ∞.
rn ln ln n
rn → ∞,
To prove the main result, we have to verify the validity of two assertions
(2.11)
P ( max
p≤k<n−p
Dk =
max
|βn |
|k−k⋆ |
√
≤rn
n
Dk ) → 1
as n → ∞
√
and uniformly for |βn (k − k ⋆ )|/ n ≤ rn
(2.12)
k ⋆ k ⋆ |k − k ⋆| k k⋆ √
,
Ck = n Q ,
−Q
+o
,
n n
n n
n
(2.13)
Bk =
(2.14)
|k − k ⋆ | √
k − k⋆
k k⋆ √
,
+ oP
n Z
,
− 1 S n (k ⋆ ) + Ṡ n (k ⋆ ) √
n
n
n n
|k − k ⋆ | √
,
n
2
where Q(t, s) = R(t, s) /R(t, t) and Z(t, s) = R(t, s)/R(t, t).
The proof of (2.12, 2.13, 2.14) follows from the following lemmas.
√
Lemma 1. It holds uniformly in |βn (k − k ⋆ )|/ n ≤ rn
Ak = oP
(2.15)
k ⋆ k ⋆ (Rn (k, k ⋆ ))2
|k − k ⋆ | k k⋆ n ⋆ ⋆
Q ,
,
−Q
−
,
−
R
(k
,
k
)
=
o
n n
n n
Rn (k, k)
n3/2
|k − k ⋆ | Rn (k, k ⋆ )
k k⋆ (2.16)
.
−1 −
,
−
1
=
o
Z
n n
Rn (k, k)
n3/2
Proof: For every integer q ≥ 0, j ≥ 0 and k < k ⋆ it holds:
Z
Z
n
1
k q
1 X i − k q n
z−
,
φj (z) dz =
φj (i) + O
n
n
n
n
k/n
1
k ⋆ /n k/n
z−
k q
n
φj (z) dz =
Clearly for arbitrary integer k and
1
k k
− Rn (k, k) = O
R ,
n n
n
1
n
i=k
⋆ −1
kX
i=k
(k ⋆ − k) i − k q n
φj (i) + O
.
n
n
k⋆
and R
k k⋆ 1
,
− Rn (k, k ⋆ ) = O
.
n n
n
557
Change-point estimator in gradually changing sequences
Moreover
k k⋆ k k
|k − k ⋆ | |k − k ⋆ | R ,
−R ,
=O
, Rn (k, k ⋆ ) − Rn (k, k) = O
.
n n
n n
n
n
Using the equality
k − k ⋆ m k m k ⋆ m
k m−1 k − k ⋆ z−
+ ...+
,
− z−
=− m z−
n
n
n
n
n
it can be proved that
|k − k ⋆ | k k⋆ k k
n
⋆
n
.
R ,
−R ,
− R (k, k ) − R (k, k) = O
n n
n n
n2
Notice that all the equalities above hold uniformly for k < k ⋆ . The assertion of
Lemma 1 is their consequence.
√
⋆
⋆
Lemma 2. For k < k, uniformly in |βn (k − k )|/ n ≤ rn , we have
k
k − k⋆ 1 X i − k m
√
ei = oP
,
n
n
n
⋆
(2.17)
i=k
(2.18)
p k
X
1 X i − k m
j=0
n
i=k ⋆
n
. X
n
n
1 X n
1
2
n
√
(i)
φn
(i))
(φ
(i)e
φ
i
j
j
j
n
n
= oP
The analogous result holds for k < k ⋆ .
k
i=1
− k⋆ n
i=1
.
Proof: Assertion (2.17) follows from the law of iterated logarithm, see Theo⋆
rem 3,
√ Chapter 7, par. 3 of Petrov (1971). For k < k, uniformly in |βn (k −
⋆
k )|/ n ≤ rn :
√1 Pk ⋆ i−k m e k
i
i=k
n
k − k ⋆ m+1/2
1 X i − k m
n
√
ei =
2m+1
n
n
n
k−k ⋆
2
⋆
i=k
n
k − k ⋆ m+1/2 k − k ⋆ r1/2 O (√ln ln n)
k − k⋆ √
p
n
=
o
= OP ( ln ln n)
≤
.
P
1/2
n
n
n
βn n1/4
√
Assertion (2.18) follows from the obvious fact that, uniformly in |βn (k−k ⋆ )|/ n ≤
rn ,
k
1 X i − k m n
k − k ⋆ m+1
.
φj (i) = O
n
n
n
⋆
i=k
558
D. Jarušková
√
Lemma 3. It holds uniformly for |βn (k − k ⋆ )|/ n ≤ rn
S n (k) − S n (k ⋆ ) =
(2.19)
k − k⋆ k − k⋆ n ⋆
.
Ṡ (k ) + oP
n
n
Proof: For k > k ⋆ we have
n
n
1 X i − k m
1 X i − k ⋆ m
S (k) − S (k ) = √
ei − √
ei
n
n
n
n
⋆
n
n
⋆
i=k
−
p
X
P
n
1
n
j=0
i=k
i−k m
n
φn
j (i) −
1
n
i=k
Pn
i=k ⋆
P
n
1
n
Pn
n (i)e
√1
φn
φ
i
i=1
j (i)
j
n
i−k ⋆ m
n
n
2
i=1 (φj (i))
n
k
i − k ⋆ m 1 X i − k m
1 X i − k m
√
√
−
ei
=
ei −
n
n
n
n
n
i=k ⋆
i=k ⋆
P
Pn
n
1
i−k m − i−k ⋆ m φn (i)
n (i)e
p
√1
φ
X
i
i=k ⋆
i=1
j
j
n
n
n
n
P
−
n
1
n
2
j=0
i=1 (φj (i))
n
P
Pn
k
i−k m φn (i)
1
n (i)e
p
√1
φ
⋆
X
i
i=1 j
i=k
j
n
n
n
P
+
.
n
1
n
2
(i))
(φ
j=0
i=1 j
n
The result follows from Lemma 2 and the expansion
i − k m
n
−
i − k ⋆ m
n
= −m
i − k ⋆ m−1 k − k ⋆ n
n
k − k⋆ +o
.
n
Assertion (2.12) follows from Lemma 1, assertion (2.13) from Lemma 1 and
Lemma 3 and assertion (2.14) from Lemma 2 and Lemma 3. The proof of (2.11)
follows the same pattern as in Hušková (1998).
Now,
∂Q(t, s) = 0,
∂t
t=s=θ ⋆
∂ 2 Q(t, s) n ∂R(t, s) 2
o
.
∂ 2 R(t, s) ⋆ ⋆
=
2
R(θ
,
θ
)
−
∂t2
∂t
∂t∂s
t=s=θ ⋆
t=s=θ ⋆
t=s=θ ⋆
⋆
= −2A(θ ),
.
∂R(t, s) ∂Z(t, s) R(θ⋆ , θ⋆ ).
=
∂t
∂t
t=s=θ ⋆
t=s=θ ⋆
Change-point estimator in gradually changing sequences
√
Hence, uniformly in |βn (k − k ⋆ )|/ n ≤ rn ,
k − k ⋆ 2 1 ∂Q(t, s) √
Dk = βn2
n
2
∂t2
t=s=θ ⋆
|k − k ⋆ | ⋆
∂Z(s, t)
k−k
n ⋆
n ⋆
√
.
S
(k
)
+
Ṡ
(k
)
+
o
+2βn √
P
∂t
n
n
t=s=θ ⋆
c⋆ , the variable
Similarly as in Hušková (1998), regarding the definition of k
⋆
⋆
βn k √−k
A(θ⋆ ) has the same limit distribution as the variable
n
∂Z(t, s) S n (k ⋆ ) + Ṡ n (k ⋆ )
∂t
t=s=θ ⋆
that is asymptotically normal with the limit variance A(θ⋆ ). Thus the asymptotic
⋆p
⋆
distribution of βn k √−k
A(θ⋆ ) is standard normal.
n
Remark 1. It is clear that the exact form of A(θ⋆ ) depends on the value m
and p.
Remark 2. If σ 2 is known but not equal to 1, then
c⋆ − k ⋆ p
βn k
D
√
A(θ⋆ ) −→ N (0, 1) as n → ∞.
σ
n
If σ 2 is unknown it can be replaced by its usual estimator based on residual sum
of squares obtained by least squares method.
3. Examples
Example 1 (Hušková). We consider variables {Yi , i = 1, . . . , n}:
i − k⋆ Yi = µ + βn
+ ei , i = 1, . . . , n,
n
+
where µ, βn ∈ R1 , k ⋆ ∈ N are unknown parameters, k ⋆ = [nθ⋆ ] for some θ⋆ ∈
(0, 1). The errors {ei } are i.i.d. satisfying Eei = 0, Ee2i = σ 2 , E|ei |2+δ < ∞ for
some δ > 0. As n → ∞, the parameter βn satisfies condition (2.7).
The limit process Y (t) has the form
Z 1
(1 − t)2
W (1)
Y (t) =
(z − t) dW (z) −
2
t
and its covariance function
(1 − t)3
(1 − t)2
(1 − s)2 (1 − t)2
R(t, s) =
+ (t − s)
−
, s ≤ t.
3
2
4
The limit distribution of kc⋆ is given by:
r
c⋆ − k ⋆ θ⋆ (1 − θ⋆ ) D
βn k
√
−→ N (0, 1).
n
σ
1 + 3θ⋆
559
560
D. Jarušková
Example 2. We consider variables {Yi , i = 1, . . . , n}:
i
i − k⋆ Yi = µ + α
+ βn
+ ei , i = 1, . . . , n,
n
n
+
where µ, α, βn ∈ R1 , k ⋆ ∈ N are unknown parameters, k ⋆ = [nθ⋆ ] for some
θ⋆ ∈ (0, 1). The errors have the same properties as in Example 1. As n → ∞, the
parameter βn satisfies condition (2.7).
The limit process Y (t) has the form
Z 1
Z 1
1
(1 − t)2
z−
dW (z)
(z − t) dW (z) −
W (1) − (1 − t)2 (1 + 2t)
Y (t) =
2
2
0
t
and its covariance function is
(1 − t)2
(1 − t)3
+ (t − s)
3
2
(1 − t)2 (1 − s)2 (1 + 2t)(1 + 2s)
(1 − s)2 (1 − t)2
−
, s ≤ t.
−
4
12
The limit distribution of kc⋆ is given by:
r
c⋆ − k ⋆ θ⋆ (1 − θ⋆ ) D
βn k
√
−→ N (0, 1).
n
σ
4
Example 3. We consider variables {Yi , i = 1, . . . , n}:
i − k ⋆ 2
i
+ βn
+ ei , i = 1, . . . , n,
Yi = µ + α
n
n
+
where µ, α, βn ∈ R1 , k ⋆ ∈ N are unknown parameters, k ⋆ = [nθ⋆ ] for some
θ⋆ ∈ (0, 1). The errors have the same properties as in Example 1. As n → ∞, the
parameter βn satisfies condition (2.7).
The limit process Y (t) has the form
Z 1
(1 − t)3
W (1)
Y (t) =
(z − t)2 dW (z) −
3
t
Z
Z 1
1 1
1
(z − t)2 z −
− 12
z−
dz
dW (z)
2
2
t
0
and its covariance function is
(1 − t)5
(1 − t)4
(1 − t)3
(1 − s)3 (1 − t)3
R(t, s) =
+ (t − s)
+ (t − s)2
−
5
2
3
9
(1 − t)3 (1 − s)4 (1 − s)3 1
1
, s ≤ t.
+ s−
− (1 − t)4 + t −
2
3
4
2
3
The limit distribution of kc⋆ is given by:
s
c⋆ − k ⋆
(θ⋆ )3 (1 − θ⋆ )3 (4 + 5θ⋆ )
βn k
D
√
−→ N (0, 1).
n
σ
3 + 15θ⋆ + 45(θ⋆ )2 + 45(θ⋆ )3
R(t, s) =
Change-point estimator in gradually changing sequences
References
[1] Anderson T.W., The Statistical Analysis of Time Series, John Wiley, New York, 1971.
[2] Billingsley P., Convergence of Probability Measures, John Wiley, New York, 1968.
[3] Davies R.B., Hypothesis testing when a nuisance parameter is present only under the
alternative, Biometrika 74 (1987), 33–43.
[4] Hinkley D., Inference about the intersection in two-phase regression, Biometrika 56 (1969),
495–504.
[5] Hušková M., Estimation in location model with gradual changes, Comment. Math. Univ.
Carolinae 39 (1998), 147–157.
[6] Knowles M., Siegmund D., Zhang H., Confidence regions in semilinear regression, Biometrika 78 (1991), 15–31.
[7] Petrov V.V., Predel’nyje teoremy dlja sum nezavisimych slucajnych velicin, Nauka, Moskva,
1987.
[8] Siegmund D., Zhang H., Confidence region in broken line regression, Change-point problems, IMS Lecture Notes - Monograph Series, vol. 23, 1994, pp. 292–316.
Department of Mathematics, Faculty of Civil Engineering, Czech Technical University, Thákurova 7, CZ–166 29 Prague 6, Czech Republic
E-mail : [email protected]
(Received July 21, 1997, revised October 27, 1997)
561