Multivariate General Linear Model
Multivariate General Linear Model
I
In the univariate GLM, N responses x1 , x2 , . . . , xN are assumed to be
independent, and the αth response is related to q covariate values,
represented by the (q × 1) vector zα , through
xα ∼ N β 0 zα , σ 2 , α = 1, 2, . . . , N.
I
In the multivariate case, the scalar response xα is replaced by a
(p × 1) vector response xα , and the (1 × q) parameter vector β 0 is
replaced by a (p × q) matrix β:
xα ∼ Np (βzα , Σ) ,
NC STATE UNIVERSITY
1 / 10
α = 1, 2, . . . , N.
Statistics 784
Multivariate Analysis
Multivariate General Linear Model
I
Note: if xi,α is the i th component of xα , and β 0i is the i th row of β,
then xi,α satisfies the (marginal) univariate model
xi,α ∼ N β 0i zα , σi,i , α = 1, 2, . . . , N.
I
That is, each component of the response vector is related to the same
covariates zα , but with its own parameter vector β 0i .
I
We could generalize this to allow different covariates for each
response by including all covariates in zα , but then β would be
constrained to contain structural zeros.
I
The classical theory that follows does not allow this more general
form of the model.
NC STATE UNIVERSITY
2 / 10
Statistics 784
Multivariate Analysis
Multivariate General Linear Model
Likelihood
I
The likelihood, as a function of running variables β ∗ and Σ∗ , is
L(β ∗ , Σ∗ )
=
=
#
N
1X
0 ∗ −1
∗
∗
(xα − β zα )
exp −
(xα − β zα ) Σ
2
α=1
(
"
#)
N
X
1
exp − trace Σ∗ −1
(xα − β ∗ zα ) (xα − β ∗ zα )0
2
"
1
1
det(2πΣ∗ ) 2 N
1
1
det(2πΣ∗ ) 2 N
NC STATE UNIVERSITY
α=1
3 / 10
Statistics 784
Multivariate Analysis
Multivariate General Linear Model
I
Write
A=
N
X
zα z0α ,
C=
xα z0α ,
B = CA−1
α=1
α=1
I
N
X
∗
Then for any (p × q) matrix β we have
N
X
(xα − β ∗ zα ) (xα − β ∗ zα )0
α=1
=
=
N
X
α=1
N
X
0
(xα − Bzα ) (xα − Bzα ) + (B − β ∗ )
N
X
!
zα z0α
(B − β ∗ )0
α=1
(xα − Bzα ) (xα − Bzα )0 + (B − β ∗ )A(B − β ∗ )0
α=1
NC STATE UNIVERSITY
4 / 10
Statistics 784
Multivariate Analysis
Multivariate General Linear Model
I
Since
i
h
trace Σ∗ −1 (B − β ∗ )A(B − β ∗ )0 ≥ 0,
with equality only if β ∗ = B, the likelihood is maximized, for any Σ∗ ,
at
β̂ = B = CA−1
=
N
X
!
xα z0α
α=1
I
N
X
!−1
zα z0α
α=1
Then the likelihood is maximized wrt Σ∗ at
N
0
1 X
xα − β̂zα xα − β̂zα .
Σ̂ =
N
α=1
NC STATE UNIVERSITY
5 / 10
Statistics 784
Multivariate Analysis
Multivariate General Linear Model
I
0
Note: the i th row of β̂, β̂ i , is given by
0
β̂ i = C0i A−1 ,
where C0i , the i th row of C, is given by
C0i =
N
X
xi,α z0α .
α=1
0
I
That is, β̂ i is the least squares estimate of the parameters in the
regression of the i th response xi,α on zα .
I
So we can construct β̂ by regressing each of the responses on the
covariates in turn.
NC STATE UNIVERSITY
6 / 10
Statistics 784
Multivariate Analysis
Multivariate General Linear Model
Sampling Distributions
I
Since each element of β̂ is a linear combination of the responses, the
elements jointly follow a multivariate normal distribution.
I
From the row-by-row construction, β̂ is unbiased:
E β̂ = β.
I
The joint distribution is typically described in terms of the (pq × 1)
vector
0 0
0 0
vec β̂ = β̂ 1 , β̂ 2 , . . . , β̂ p
I
Then unbiasedness is written
h i
E vec β̂ = vec(β)
NC STATE UNIVERSITY
7 / 10
Statistics 784
Multivariate Analysis
Multivariate General Linear Model
I
Also
E
I
β̂ i − β i
β̂ j − β j
0 = σi,j A−1 .
So

C
h
i 

vec β̂ = 

σ1,1 A−1
σ2,1 A−1
..
.
σp,1 A−1
σ1,2 A−1 . . .
σ2,2 A−1 . . .
..
..
.
.
−1
σp,2 A
...
σ1,p A−1
σ2,p A−1
..
.
σp,p A−1





= Σ ⊗ A−1 ,
the Kronecker (or direct) product of Σ and A−1 .
NC STATE UNIVERSITY
8 / 10
Statistics 784
Multivariate Analysis
Multivariate General Linear Model
I
So the sampling distribution of β̂ can be written
vec β̂ ∼ Npq vec(β) , Σ ⊗ A−1 .
I
The sampling distribution of Σ̂ is found by writing
N Σ̂ =
=
N X
α=1
N−q
X
xα − β̂zα
xα − β̂zα
0
yα yα0
α=1
where y1 , y2 , . . . , yN−q are iid Np (0, Σ).
NC STATE UNIVERSITY
9 / 10
Statistics 784
Multivariate Analysis
Multivariate General Linear Model
I
This shows that
N Σ̂ ∼ Wp (Σ, N − q),
and hence that
S=
=
N
Σ̂
N −q
N
1 X
N −q
xα − β̂zα
xα − β̂zα
0
α=1
is an unbiased estimator of Σ.
NC STATE UNIVERSITY
10 / 10
Statistics 784
Multivariate Analysis