4.2
Autoregressive (AR)
Moving average models are causal linear processes by definition. There is another
class of models, based on a recursive formulation similar to the exponentially
weighted moving average.
Definition 4.11 (Autoregressive AR(p)). Suppose φ1 , . . . , φp ∈ R are constants
and (Wi ) ∼ WN(σ 2 ). The AR(p) process with parameters σ 2 , φ1 , . . . , φp is defined
through
p
X
φj Xi−j ,
(3)
X i = Wi +
j=1
whenever such stationary process (Xi ) exists.
Remark 4.12. The process in Definition 4.11 is sometimes called a stationary
AR(p) process. It is possible to consider a ‘non-stationary AR(p) process’ for any
φ1 , . . . , φp satisfying (3) for i ≥ 0 by letting for example Xi = 0 for i ∈ [−p + 1, 0].
Example 4.13 (Variance and autocorrelation of AR(1) process). For the AR(1)
process, whenever it exits, we must have
γ0 = Var(Xi ) = Var(φ1 Xi−1 + Wi ) = φ21 γ0 + σ 2 ,
which implies that we must have |φ1 | < 1, and
γ0 =
σ2
.
1 − φ21
We may also calculate for j ≥ 1
γj = E[Xi Xi−j ] = E[(φ1 Xi−1 + Wi )Xi−j ] = φ1 E[Xi−1 Xi−j ] = φj1 γ0 ,
which gives that ρj = φj1 .
Example 4.14. Simulation of an AR(1) process.
phi_1 <- 0.7
x <- arima.sim(model=list(ar=phi_1), 140)
# This is the explicit simulation:
gamma_0 <- 1/(1-phi_1^2)
x_0 <- rnorm(1)*sqrt(gamma_0)
x <- filter(rnorm(140), phi_1, method = "r", init = x_0)
Example 4.15. Consider a stationary AR(1) process. We may write
Xi = φ1 Xi−1 + Wi = · · · = φn1 Xi−n +
n−1
X
φj1 Wi−j .
j=0
29
4
2
0
−4
Simulated values
20
40
60
80
100
120
140
0.6
−0.2 0.2
ACF
1.0
0
0
5
10
15
Figure 23: Simulation of AR(1) process in Example 4.14.
phi_1 = −0.9
0.0
−0.5
0.4
0.5
0.8
phi_1 = 0.9
0
5
10
15
0
10
15
phi_1 = −0.7
0.0
−0.5
0.4
0.5
0.8
phi_1 = 0.5
5
0
5
10
15
0
5
10
15
Figure 24: Autocorrelations of AR(1) with different parameters.
30
P
j
Define the causal linear process Yi = ∞
j=0 φ1 Wi−j , then we may write (detailed
proof not examinable)
∞
2 1/2
X
n
j
2 1/2
= Eφ1 Xi−n −
E|Xi − Yi |
φ1 Wi−j j=n
≤ |φ1 |
n
= |φ1 |n
1/2
2
EXi−n
+
∞
X
2
|φ1 |j EWi−j
j=n
σ n→∞
−−−→ 0,
σX +
1 − |φ1 |
1/2
2
where σX
= EX12 . This implies Xi = Yi (almost surely).
We may write the autoregressive process also in terms of the backshift
operator, as
p
X
φ j B j X i = Wi ,
(4)
Xi −
j=1
or φ(B)Xi = Wi , where
Definition 4.16 (Characteristic polynomial of AR(p)).
φ(z) := 1 −
p
X
φj z j .
j=1
Remark 4.17. Note the minus sign in the AR polynomial, contrary to the plus in
the MA polynomial. In some contexts (esp. signal processing), the AR coefficients
are often defined φ̃i = −φi , so that the AR polynomial will look exactly like the
MA polynomial.
Theorem 4.18. The (stationary) AR(p) process exists and can be written as a
causal linear process if and only if
φ(z) 6= 0
for all z ∈ C with |z| ≤ 1,
that is, the roots of the complex polynomial φ(z) lie strictly outside the unit disc.
For full proof, see for example Theorem 3.1.1 of Brockwell and Davis.
However, to get the idea, we may write informally
Xi = φ(B)−1 Wi ,
and we may write the reciprocal of the characteristic function as
∞
X
1
=
cj z j ,
φ(z)
j=0
for |z| ≤ 1 + ǫ,
31
This means that we may write the AR(p) as a causal linear process
∞
X
Xi =
cj Wi−j ,
j=0
where the coefficients satisfy10 |cj | ≤ K(1 + ǫ/2)−j .
Remark 4.19. This justifies viewing AR(p) as a ‘MA(∞)’ with coefficients (cj )j≥1 .
This also implies that we may apporximate AR(p) with ‘arbitrary precision’ by
MA(q) with large enough q.
4.3
Invertibility of MA
Example 4.20. Let θ1 ∈ (0, 1) and σ 2 > 0 be some parameters, and consider two
MA(1) models,
i.i.d.
Xi = Wi + θ1 Wi−1 ,
(Wn ) ∼ N (0, σ 2 )
X̃i = W̃i + θ̃1 W̃i−1 ,
(W̃n ) ∼ N (0, σ̃ 2 ),
i.i.d.
where θ̃1 = 1/θ1 and σ̃ 2 = σ 2 θ12 . We have
γ0 = σ 2 (1 + θ12 ),
γ 1 = σ 2 θ1
γ̃1 = σ˜2 θ̃1 .
γ̃0 = σ̃ 2 (1 + θ̃12 )
What do you observe?
It turns out that the following invertibility condition resolves the MA(q)
identifiability problem, and therefore it is standard that the roots of the characteristic polynomial are assumed to lie outside the unit disc.
Theorem 4.21. If the roots of the characteristic polynomial of MA(q) are strictly
outside the unit circle, the MA(q) is invertible in the sense that it satisfies
∞
X
Wi =
βj Xi−j ,
j=0
where the constants satisfy β0 = 1 and |βj | ≤ K(1 + ǫ)−j for some constants
K < ∞ and ǫ > 0.
As with Theorem 4.18, we may write symbolically, from Xi = θ(B)Wi ,
that
∞
X
1
Xi =
βj Xi−j ,
Wi =
θ(B)
j=0
where the constants βj are uniquely determined by 1/θ(z) =
roots of θ(z) lie outside the unit disc.
10. Because
32
P
cj (1 + ǫ/2)j → 0 as j → ∞.
P∞
j=0
βj z j , as the
4.4
Autoregressive moving average (ARMA)
Definition 4.22 (Autoregressive moving average ARMA(p,q) process). Suppose
φ1 , . . . , φp ∈ R are coefficients of a (stationary) AR(p) process and θ1 , . . . , θq ∈ R,
and (Wi ) ∼ WN(σ 2 ). The (stationary) ARMA(p,q) process with these parameters
is a process satisfying
Xi =
p
X
j=1
φj Xi−j +
q
X
θj Wi−j ,
(5)
j=0
with the convention θ0 = 1 and where the first sum vanishes if p = 0.
Remark 4.23. AR(p) is ARMA(p,0) and MA(q) is ARMA(0,q).
We may write ARMA(p,q) briefly with the characteristic polynomials of
the AR and MA and the backshift operator as
φ(B)Xi = θ(B)Wi .
Simulation of a general ARMA(p,q) model is not straightforward exactly,
but we can approximately simulate it by setting X−p+1 = · · · = X0 = 0 (say)
and then following (5). Then, Xb , Xb+1 , . . . , Xb+n is an approximate sample of a
stationary ARMA(p,q) if b is ‘large enough’. This is what R function arima.sim
does; the parameter n.start is b above.
Example 4.24. Simulation of ARMA(2,1) model with φ1 = 0.3, φ2 = −0.4, θ1 =
−0.8.
x <- arima.sim(list(ma = c(-0.8), ar=c(.3,-.4)),
140, n.start = 1e5)
This is the same as
q
z
z
x
<<<<-
2; n <- 140; n.start <- 1e5
filter(rnorm(n.start+n), c(1, -0.8), sides=1)
tail(z, n.start+n-q)
tail(filter(z, c(.3,-.4), method="r"), n)
(The latter may sometimes be necessary, because arima.sim checks the stability
of the AR part by calculating the roots of φ(z) numerically, which is notoriously
unstable if the order of φ is large. Sometimes arima.sim refuses to simulate a
stable ARMA. . . )
Remark 4.25. If the characteristic polynomials θ(z) and φ(z) of an ARMA(p,q)
share a (complex) root, say x1 = y1 , then
(z − x1 )(z − x2 ) · · · (z − xq )
θ(z)
=
φ(z)
(z − y1 )(z − y2 ) · · · (z − yp )
33
2
0
−4 −2
Simulateed values
20
40
60
80
100
120
140
0.5
−0.5
ACF
1.0
0
0
5
10
15
Figure 25: Simulation of ARMA(2,1) in Example 10.6.
=
(z − x2 ) · · · (z − xq )
θ̃(z)
,
=
(z − y2 ) · · · (z − yp )
φ̃(z)
where θ̃(z) is of order q − 1 and φ̃(z) is of order p − 1, and it turns out that
φ̃(B)Xi = θ̃(B)Wi ,
which means that the model reduces to ARMA(p − 1,q − 1).
Condition 4.26 (Regularity conditions for ARMA). In what follows, we shall
assume the following:
(a) The roots of the AR characteristic polynomial are strictly outside the unit
disc (cf Theorem 4.18).
(b) The roots of the MA characteristic polynomial are strictly outside the unit
disc (cf. Theorem 4.21).
(c) The AR and MA characteristic polynomials do not have common roots
(cf. Remark 4.25).
34
Theorem 4.27. A stationary ARMA(p,q) model satisfying Condition 4.26 exists, is invertible and can be written as a causal linear process
Xi =
∞
X
ξj Wi−j ,
Wi =
j=0
∞
X
βj Xi−j ,
j=0
where the constants ξj and βj satisfy
∞
X
θ(z)
ξj z =
φ(z)
j=0
j
and
∞
X
j=0
βj z j =
φ(z)
.
θ(z)
In addition, β0 = 1 and there exist constants K < ∞ and ǫ > 0 such that
max{|ξj |, |βj |} ≤ K(1 + ǫ)−j for all j ≥ 0.
Remark 4.28. In fact, the coefficients ξj (or βj ) related to any ARMA(p,q) can be
calculated numerically from the parameters easily. Also the autocovariance can
be calculated numerically up to any lag in a straightforward way; cf. Brockwell
and Davis p. 91–95. In R, the autocorrelation coefficients can be calculated with
ARMAacf.
4.5
Integrated models
Autoregressive moving average models are pretty flexible models for stationary
series. However, in many practical time series, it might be more useful to consider
the differenced series (Definition 2.10). This brings us to the general notion of
Definition 4.29 (Difference operator). Suppose (Xi ) is a stochastic process. Its
d:th order difference process is defined as (∇d Xi ), where the d:th order difference
operator may be written in terms of the backshift operator as ∇d = (1 − B)d for
d ≥ 1.
Definition 4.30 (Autoregressive integrated moving average ARIMA(p,d,q) process). If the d-th difference of the process (∇d Xi ) follows ARMA(p,q), then we
say (Xi ) is ARIMA(p,d,q).
Remark 4.31. Suppose that (∇d Xi ) is a stationary ARMA(p, q).
(i) The ARIMA(p,d,q) process (Xi ) is not unique (why?).
(ii) The ARIMA(p,d,q) process (Xi ) is not, in general, stationary.
The process (Xi ) (or the data x1 , . . . , xn ) is said to be difference stationary.
Example 4.32. Simple random walk
Xi = Xi−1 + Wi
is an ARIMA(0,1,0).
35