ST430 Introduction to Regression Analysis
ST430: Introduction to Regression Analysis, Chapter 10,
Sections 5,6,8,9
Luo Xiao
November 23, 2015
1 / 27
ST430 Introduction to Regression Analysis
Time Series Analysis
2 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
Time series
A time series is a collection of data obtained by observing a response
variable at various times.
Examples
The monthly Case-Shiller house price index;
Daily Facebook stock price;
Yearly average global temperature.
3 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
Forecasting by regression
setwd("~/Dropbox/teaching/2015Fall/R_datasets/Exercises&Exampl
load("SALES35.RData")
with(SALES35, plot(T, SALES,
xlim = c(0,40),ylim = c(0,200),pch = 20))
fit = lm(SALES ~ T, SALES35)
summary(fit)
curve(predict(fit, data.frame(T = x)),
add = TRUE, lty = 2, lwd=2)
# forecast next 5 values:
newTimes = 36:40
p = predict(fit, data.frame(T = newTimes),
interval = "prediction")
matlines(newTimes, p, col = c("red", "blue", "blue"),
lty = c(2, 3, 3), lwd =2)
4 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
100
50
0
SALES
150
200
Predict by extrapolation
0
10
20
30
40
T
5 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
Issues with regression-based forecasts
Forecasting involves extrapolation, and depends on continued validity of the
fitted model.
Less of an issue if we limit forecasts to the short term.
The errors in time series regressions are often correlated with each other,
which violates the usual assumptions.
In particular, we cannot trust reported standard errors and P-values.
But using time series models, we can exploit the correlation to improve
forecasts.
6 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
Autocorrelation
When a regression model is fitted to time series data, the residuals often
violate the standard assumptions by being correlated.
Because one residual is correlated with another in the same series, we refer
to auto-correlation.
Notation, emphasizing the autocorrelated nature of the residuals:
Yt = E (Yt |Xt ) + Rt
= β0 + β1 Xt,1 + β2 Xt,2 + · · · + βk Xt,k + Rt .
7 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
The correlation of Rt with Rt−1 is called a lagged correlation, specifically
the lag 1 correlation.
More generally, the correlation of Rt with Rt−m is the lag m correlation,
m > 0.
Autocorrelation is usually strongest for small lags and decays to zero for
large lags.
Autocorrelation depends on the lag m, but often does not depend on t.
8 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
When the correlation of Rt with Rt−m does not depend on t, and in
addition:
the expected value E (Rt ) is constant;
the variance var(Rt ) is constant;
the residuals are said to be stationary.
Regression residuals have zero expected value, hence constant expected
value, if the model is correctly specified.
They also have constant variance, perhaps after the data have been
transformed appropriately, or if weighted least squares has been used.
9 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
Correlogram
The graph of corr(Rt , Rt−m ) against lag m is the correlogram, or
auto-correlation function (ACF).
The ACF is usually plotted including the lag 0 correlation, which is of
course exactly 1.
The blue lines indicate significance at α = .05.
For the SALES35 example:
setwd("~/Dropbox/teaching/2015Fall/R_datasets/Exercises&Exampl
load("SALES35.RData")
fit = lm(SALES ~ T, SALES35)
acf(residuals(fit))
10 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
ACF plot
0.4
0.0
−0.4
ACF
0.8
Series residuals(fit)
0
5
10
15
Lag
11 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
Modeling autocorrelation
Sometimes the ACF appears to decay exponentially:
ACF(m) ≈ φm , m ≥ 0, −1 < φ < 1.
The 1st order autoregressive model, AR(1), has this property; if
Rt = φRt−1 + t ,
where t has constant variance and is uncorrelated with t−m for all m > 0,
then
corr(Rt , Rt−m ) = φm , m > 0.
12 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
Autocorrelation functions for several first-order auto
regressive error models: Rt = φRt−1 + t
13 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
Autocorrelation functions for several first-order auto
regressive error models: Rt = φRt−1 + t
14 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
Autocorrelation functions for several first-order auto
regressive error models: Rt = φRt−1 + t
15 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
General autoregressive model, AR(p):
Rt = φ1 Rt−1 + φ2 Rt−2 + · · · + φp Rt−p + t .
ACF decays exponentially, but not exactly as φm for any φ.
Moving average models, MA(q):
Rt = t + θ1 t−1 + θ2 t−2 + · · · + θq t−q .
ACF is zero for lag m > q.
16 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
Autocorrelations for a first-order moving average model:
Rt = t + θt−1
17 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
Combined ARMA(p, q) model:
Rt = φ1 Rt−1 + φ2 Rt−2 + · · · + φp Rt−p
+ t + θ1 t−1 + θ2 t−2 + · · · + θq t−q .
ACF again decays exponentially, but not exactly as φm for any φ.
18 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
Regression with autocorrelated errors
To fit the regression model
Yt = E (Yt |Xt ) + Rt
we must specify both the regression part
E (Yt |Xt ) = β0 + β1 Xt,1 + β2 Xt,2 + · · · + βk Xt,k
and a model for the autocorrelation of Rt , say ARMA(p, q) for specified p
and q.
Time series software fits both submodels simultaneously.
19 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
In SAS:
proc autoreg can handle AR(p) errors, but not ARMA(p, q) for
q > 0.
proc arima can handle ARMA(p, q).
In R, try AR(1) errors:
ar1 <- arima(SALES35$SALES,
order = c(1, 0, 0), xreg = SALES35$T)
print(ar1)
tsdiag(ar1)
The ARMA(p, q) model is specified by order = c(p, 0, q). The middle
part of the order is d, for differencing.
20 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
−1 2
Standardized Residuals
0
5
10
15
20
25
30
35
Time
ACF
−0.2
ACF of Residuals
0
5
10
15
Lag
p value
0.0 1.0
p values for Ljung−Box statistic
2
4
6
8
10
lag
21 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
Note that φ̂, the ar1 coefficient, is significantly different from zero.
Note that the trend (coefficient of T) is similar to the original regression:
4.2959 versus 4.2956.
Its reported standard error is higher: 0.1760 versus 0.1069.
The original, smaller, standard error is not credible, because it was
calculated on the assumption that the residuals are not correlated.
The new standard error recognizes autocorrelation, and is more credible, but
is still calculated on an assumption: that the residuals are AR(1).
22 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
Which model to use?
ACF of sales residuals decays something like an exponential, but is also not
significantly different from zero for m > 1, so it could be MA(1).
ma1 <- arima(SALES35$SALES, order = c(0, 0, 1),
xreg = SALES35$T)
print(ma1)
tsdiag(ma1)
AIC(ma1)
AIC(ar1)
AIC(fit)
θ̂1 , the ma1 coefficient, is also significantly different from zero.
But AIC is higher than for AR(1), so AR(1) is preferred.
23 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
Be systematic; use BIC to find a good model (AIC tends to overfit):
P = Q = 0:3
BICtable <- matrix(NA, length(P), length(Q))
dimnames(BICtable) <- list(paste("p:", P), paste("q:", Q))
for (p in P) for (q in Q) {
apq <- arima(SALES35$SALES, order = c(p, 0, q),
xreg = SALES35$T)
BICtable[p+1, q+1] <- AIC(apq,k=log(nrow(SALES35)))
}
BICtable
AR(1) gives the minimum BIC out of these choices.
24 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
Forecasting: R code
with(SALES35,
plot(T,SALES,xlim = c(0,40), ylim = c(0,200)))
ar1Fit <- coefficients(ar1)["intercept"] +
coefficients(ar1)["SALES35$T"] * SALES35$T
lines(SALES35$T, ar1Fit, lty = 2,lwd=2)
newTimes <- 36:40
p <- predict(ar1, n.ahead = length(newTimes),
newxreg = newTimes)
pCL <- p$se * qnorm(.975)
matlines(newTimes, p$pred + cbind(0, -pCL, pCL),
col = c("red", "blue", "blue"),
lty = c(2, 3, 3),lwd=2)
25 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
100
50
0
SALES
150
200
Forecasting: output
0
10
20
30
40
T
26 / 27
Time Series Analysis
ST430 Introduction to Regression Analysis
The predicted values remain the same as those from linear regression.
The width of prediction interval is actually smaller than that from linear
regression.
Question:
Isn’t that we will have larger standard errors for the coefficients, then why
narrower prediction intervals?
Answer:
Because the estimated error variance for the AR(1) regression model is
substantially smaller.
27 / 27
Time Series Analysis