H. Madsen, Time Series Analysis, Chapmann Hall
Time Series Analysis
Henrik Madsen
[email protected]
Informatics and Mathematical Modelling
Technical University of Denmark
DK-2800 Kgs. Lyngby
Henrik Madsen
1
H. Madsen, Time Series Analysis, Chapmann Hall
Outline of the lecture
Practical information
Introductory examples (See also Chapter 1)
A brief outline of the course
Chapter 2:
Multivariate random variables
The multivariate normal distribution
Linear projections
Example
Henrik Madsen
2
H. Madsen, Time Series Analysis, Chapmann Hall
Introductory example – shares (COLO B 18m)
From www.cse.dk
Henrik Madsen
3
H. Madsen, Time Series Analysis, Chapmann Hall
2000
1200
800
Heat Consumption (GJ/h)
Consumption of District Heating (VEKS) – data
Nov
Dec
Jan
Feb
Apr
Mar
Apr
1996
5
0
−5
−10
Air Temperature (°C)
10
1995
Mar
Nov
Dec
Jan
Feb
1995
Henrik Madsen
1996
4
H. Madsen, Time Series Analysis, Chapmann Hall
2000
1500
1000
Heat Consumption (GJ/h)
2500
Consumption of DH – simple model
−15
−10
−5
0
Air Temperature (°C)
Henrik Madsen
5
5
10
H. Madsen, Time Series Analysis, Chapmann Hall
Consumption of DH – model error
400
0
−400
Model Error (GJ/h)
Model Error
Nov
Dec
Jan
Feb
1995
Mar
Apr
Mar
Apr
1996
400
0
−400
Model Error (GJ/h)
Model Error as it should be if the model were OK
Nov
Dec
Jan
Feb
1995
Henrik Madsen
1996
6
H. Madsen, Time Series Analysis, Chapmann Hall
A brief outline of the course
General aspects of multivariate random variables
Prediction using the general linear model
Time series models
Some theory on linear systems
Time series models with external input
Some goals:
Characterization of time series / signals; correlation functions,
covariance functions, spectral distributions, stationarity,
ergodicity, linearity, . . .
Signal processing; filtering, sampling, smoothing
Modelling; with or without external input
Prediction / Control
Henrik Madsen
7
H. Madsen, Time Series Analysis, Chapmann Hall
Multivariate random variables
Joint and marginal densities
Conditional distributions
Expectations and moments
Moments of multivariate random variables
Conditional expectation
The multivariate normal distribution
Distributions derived from the normal distribution
Linear projections
Henrik Madsen
8
H. Madsen, Time Series Analysis, Chapmann Hall
Multivariate random variables
Definition (n-dimensional random variable; random vector)


X1
 X2 
 
X= . 
 .. 
Xn
Joint distribution function:
F (x1 , · · · , xn ) = P{X1 ≤ x1 , · · · , Xn ≤ xn }
Henrik Madsen
9
H. Madsen, Time Series Analysis, Chapmann Hall
Multivariate random variables
Probability density function (continuous case):
∂ n F (x1 , · · · , xn )
f (x1 , · · · , xn ) =
∂x1 · · · ∂xn
F (x1 , · · · , xn ) =
Z
x1
Z
···
−∞
xn
−∞
f (t1 , · · · , tn ) dt1 . . . dtn
Probability density function (discrete case):
f (x1 , · · · , xn ) = P{X1 = x1 , · · · , Xn = xn }
Henrik Madsen
10
H. Madsen, Time Series Analysis, Chapmann Hall
The Multivariate Normal Distribution
The joint p.d.f.
1
1
√
fX (x) =
exp − (x − µ)T Σ−1 (x − µ)
2
(2π)n/2 det Σ
Σ must be positive semidefinite
Notation: X ∼ N(µ, Σ)
Standardized multivariate normal: X ∼ N(0, I)
N(µ, Σ) = µ + T N(0, I), where Σ = T T T
If X ∼ N(µ, Σ) and Y = a + BX then
Y ∼ N(a + Bµ, BΣB T )
More relations between distributions in Sec. 2.7
Henrik Madsen
11
H. Madsen, Time Series Analysis, Chapmann Hall
Marginal density function
Sub-vector: (X1 , · · · , Xk )T (k < n)
Marginal density function:
Z ∞ Z
fS (x1 , · · · , xk ) =
···
−∞
∞
−∞
f (x1 , · · · , xn ) dxk+1 · · · dxn
Marginal histogram of 100000 samples
4
0.10
15
Density
0.08
0.06
0
0.04
−2
0.02
−4
x2
Percent of Total
x2
2
10
5
0.00
−4
−2
0
2
x1
4
0
−4
−2
0
x1
x1
Henrik Madsen
12
2
4
H. Madsen, Time Series Analysis, Chapmann Hall
Conditional distributions
The conditional density
of Y given X = x is
defined as (fX (x) > 0):
0.10
2
0.08
0.06
y
fX,Y (x, y)
fY |X=x (y) =
fX (x)
4
0
0.04
−2
(joint density of (X, Y )
divided by the marginal
density of X evaluated at
x)
0.02
−4
0.00
−4
−2
0
x
Henrik Madsen
13
2
4
H. Madsen, Time Series Analysis, Chapmann Hall
Independence
If knowledge of X does not give information about Y we get
fY |X=x (y) = fY (y)
This leads to the following definition of independence:
fX,Y (x, y) = fX (x)fY (y)
Henrik Madsen
14
H. Madsen, Time Series Analysis, Chapmann Hall
Expectation
Let X be a univariate random variable with density fX (x). The
expectation of X is then defined as:
Z ∞
E[X] =
xfX (x)dx (continuous case)
−∞
E[X] =
X
xP (X = x)
(discrete case)
all x
Calculation rule:
E[a + bX1 + cX2 ] = a + b E[X1 ] + c E[X2 ]
Henrik Madsen
15
H. Madsen, Time Series Analysis, Chapmann Hall
Moments and variance
n’th moment:
n
E[X ] =
Z
∞
xn fX (x) dx
−∞
n’th central moment:
E[(X − E[X])n ] =
Z
∞
−∞
(x − E[X])n fX (x) dx
The 2’nd central moment is called the variance:
V [X] = E[(X − E[X])2 ] = E[X 2 ] − (E[X])2
Henrik Madsen
16
H. Madsen, Time Series Analysis, Chapmann Hall
Covariance
Covariance:
Cov[X1 , X2 ] = E[(X1 −E[X1 ])(X2 −E[X2 ])] = E[X1 X2 ]−E[X1 ]E[X2 ]
Variance and covariance:
V [X] = Cov[X, X]
Calculation rule:
Cov[aX1 + bX2 , cX3 + dX4 ] =
ac Cov[X1 , X3 ] + ad Cov[X1 , X4 ] + bc Cov[X2 , X3 ] + bd Cov[X2 , X4 ]
The calculation rule can be used for the variance also
Henrik Madsen
17
H. Madsen, Time Series Analysis, Chapmann Hall
Expectation and Variance for Random Vectors
Expectation: E[X] = [E[X1 ] E[X2 ] . . . E[Xn ]]T
Variance-covariance (matrix):
ΣX = V [X] = E[(X − µ)(X − µ)T ] =

V [X1 ]
Cov[X1 , X2 ] · · ·
 Cov[X2 , X1 ]
V [X2 ]
···


..

.
Cov[Xn , X1 ] Cov[Xn , X2 ] · · ·

Cov[X1 , Xn ]
Cov[X2 , Xn ]


..

.
Correlation:
Cov[Xi , Xj ]
σij
ρij = p
=
σi σj
V [Xi ]V [Xj ]
Henrik Madsen
18
V [Xn ]
H. Madsen, Time Series Analysis, Chapmann Hall
Expectation and Variance for Random Vectors
The correlation matrix R = ρ is an arrangement of ρij in a
matrix
Covariance matrix between X (dim. p) and Y (dim. q ):
ΣXY
T
= C[X, Y ] = E (X − µ)(Y − ν)


Cov[X1 , Y1 ] · · · Cov[X1 , Yq ]


..
..
= 

.
.
Cov[Xp , Y1 ] · · ·
Cov[Xp , Yq ]
Calculation rules – see the book.
The special case of the variance C[X, X] = V [X] results in
V [AX] = AV [X]AT
Henrik Madsen
19
H. Madsen, Time Series Analysis, Chapmann Hall
Conditional expectation
E[Y |X = x] =
Z
∞
−∞
yfY |X=x (y) dy
E[Y |X] = E[Y ] if andY are independent
E[Y ] = E E[Y |X]
E[g(X)Y |X] = g(X)E[Y |X]
E[g(X)Y ] = E g(X)E[Y |X]
E[a|X] = a
E[g(X)|X] = g(X)
E[cX + dZ|Y ] = cE[X|Y ] + dE[Z|Y ]
Henrik Madsen
20
H. Madsen, Time Series Analysis, Chapmann Hall
Variance separation
Definition of conditional variance and covariance:
T V [Y |X] = E Y − E[Y |X] Y − E[Y |X] |X
T C[Y , Z|X] = E Y − E[Y |X] Z − E[Z|X] |X
The variance separation theorem:
V [Y ] = E V [Y |X] + V E[Y |X]
C[Y , Z] = E C[Y , Z|X] + C E[Y |X], E[Z|X]
Henrik Madsen
21
H. Madsen, Time Series Analysis, Chapmann Hall
Linear Projections
Consider two random vectors Y and X , then
E
Y
X
=
µY
µX
and V
Y
X
=
ΣY Y
ΣXY
ΣY X
ΣXX
Consider the linear projection: E[Y |X] = a + BX
Then:
E[Y |X] = µY + ΣY X ΣXX −1 (X − µX )
V [Y − E[Y |X]] = ΣY Y − ΣY X ΣXX −1 ΣY X T
C Y − E[Y |X], X = 0
The linear projection above has minimal variance among all
linear projections.
Henrik Madsen
22
H. Madsen, Time Series Analysis, Chapmann Hall
Air pollution in cities
Carstensen (1990) has used time series analysis to set up
models for N O and N O2 at Jagtvej in Copenhagen
Measurements of N O and N O2 available every third hour (00,
03, 06, 09, 12, . . . )
We have µN O2 = 48µg/m3 and µN O = 79µg/m3
In the model X1,t = N O2,t − µN O2 and X2,t = N Ot − µN O is
used
Henrik Madsen
23
H. Madsen, Time Series Analysis, Chapmann Hall
Air pollution in cities – model and forecast
X1,t
X2,t
=
0.9 −0.1
0.4
0.8
X1,t−1
X2,t−1
+
(µg/m3 )2
ξ1,t
ξ2,t
X t = ΦX t−1 + ξt
V [ξt ] = Σ =
σ12
σ21
σ12
σ22
=
30 21
21 23
Assume that t corresponds to 09:00 today and we have
measurements 64 µg/m3 N O2 and 93 µg/m3 N O
Forecast the concentrations at 12:00 (t + 1)
What is the variance-covariance of this forecast?
Henrik Madsen
24
H. Madsen, Time Series Analysis, Chapmann Hall
Air pollution in cities – linear projection
At 12:00 (t + 1) we now assume that N O2 is measured with
67 µg/m3 as the result, but N O cannot be measured due to
some trouble with the equipment.
Estimate the missing N O measurement.
What is the variance of the error of the estimation?
Henrik Madsen
25