THE ASYMPTOTIC COVARIANCE MATRIX OF
ESTIMATES OF FACTOR LOADINGS WITH
NORMALIZED VARIMAX ROTATION
by
Kentaro Hayashi and Pranab K. Sen
Department of Biostatistics
University of North Carolina
Institute of Statistics
Mimeo Series No. 2176
July 1997
..
•
•
The Asymptotic Covariance Matrix of Estimates of Factor Loadings
with Normalized Varimax Rotation
KENTARO HAYASHI
and
PRANAB KUMAR SEN
University of North Carolina at Chapel Hill
•
...
.
Mailing address:es: Kentaro Hayashi, Department of Psychology, The
University of North Carolina, Chapel Hill, NC 27599-3270, U.S.A.; Pranab K.
Sen, Department of Biostatistics, School of Public Health, The University of
North Carolina, Chapel Hill, NC 27599-7400
Emails: [email protected]; [email protected]
Running head: Normal varimax in factor analysis
-1-
ABSTRACT
We derive the asymptotic covariance matrix of the estimates of normalized
varimax-rotated factor loadings.
•
We partition the process of obtaining the
estimates of normalized varimax-rotated factor loadings into the three stages: (i)
•
normalization, (ii) raw varimax rotation, and (iii) denormalization, and use the
chain rule to combine the matrix of partial derivatives from each of the three
stages.
For the stage (ii) we
make use of the existing formulas for the
asymptotic covariance matrix of the estimates of raw varimax-rotated factor
loadings.
Key words and phrases: asymptotic covariance matrix, factor analysis, factor
loadings, Kronecker product, vee operator, normalized varimax rotation.
of
- 2-
1. Introduction
•
We derive the asymptotic covariance matrix of the estimates of normalized
varimax-rotated factor loadings (Kaiser (1958), see also Neudecker (1981) and
Sherin (1·966», which is an extension of work by Archer and Jennrich (1973)
and Jennrich (1974) who derived the asymptotic covariance matrix of the
estimates of raw varimax-rotated factor loadings.
The normalized varimax
rotation is by far the most frequently used method of rotation in practice of
applied research using factor analysis.
Thus it is important to report the
asymptotic covariance matrix of the estimates of normalized varimax-rotated
factor loadings.
Let A* = (Air*) be the p x m matrix of normalized varimax-rotated factor
loadings (i.e., after the nomalized varimax rotation); A
•
= (Air) be the matrix of
unrotated factor loadings (i.e., before the normalized varimax rotation); T = (tsr)
be the m x m orthogonal rotation (i.e., transformation) matrix; 'II be the p x p
positive definite diagonal matrix of residual variances. Then the factor analysis
model is given by the following decomposition of the covariance matrix L:
(1 )
L = AA' + 'II
= (AT)(TA') + 'II,
where A' is the transpose of A. The equation (1) indicates that the matrix of
factor loadings in the first term of the decomposition is not unique (called the
indeterminacy regarding the orthogonal rotation).
Thus we need the extra
conditions regarding the rotation to make the model uniquely defined.
normal varimax rotation is an orthogonal rotation
•
A*
=AT
and
TT =1m
such that the variance of squared normalized factor loadings
-3-
The
is maximized, where hi is the square-root of the i-th diagonal element of the p
P diagonal normalization matrix H
= Ip#(AA') = (h 2),
j
X
with the Hadamard (i.e.,
•
elementwise) product # and the p-dimensional identity matrix Ip . (Note that H is
unchanged by the orthogonal rotation. Thus the * sign which indicates "after the
normalized varimax rotation" is not with H.)
2. The asymptotic covariance matrix
Let
i =vec(A) and i. * =vec(A *),
where vec(A) denotes the pm x 1 vector
listing m columns of the p x m matrix A starting from the first column. Then the
asymptotic covariance matrix of the estimates of normalized varimax-rotated
factor loadings is expressed as
A
Cov(J.. *)
dt..*
A
dt..*
= (--)Cov(J.. )(--)'
dt..'
dt..'
I
A
where Cov(J..) is the asymptotic covariance matrix of the MLEs of unrotated
factor loadings which was given by Jennrich and Thayer (1973)
Lawley and Maxwell (1971) and Hayashi and Sen (1996a).);
(See also
dt..*
-a:;:
is the pm x
A
pm matrix consisting of the partial derivatives which maps the differentials of J..
A
A
to the differentials of J.. * evaluated at J.. = t...
The following four matrices of factor loadings are involved in the mapping of
the normalized varimax rotation:
(i)
Unrotated factor loadings: A,
-4-
•
=H-1/2A,
(ii)
Normalized unrotated factor loadings: A#
(iii)
Normalized raw varimax-rotated factor loadings: Ab = A#T (= H-1/2AT
=H-1/2A*),
(iv)
•
Normalized varimax-rotated factor loadings: A* = AT (= H1/2Ab).
(Note that the mapping in the stage (iii) is the raw varimax rotation.) Therefore,
by the chain rule,
(2)
where
iJA*
iJAb'
is the matrix of partial derivatives which maps the differentials of
A
A
A
A b to the differentials of A * evaluated at A b =Ab;
iJAb
#
iJA I
is the matrix of partial
i
•
i
derivatives which maps the differentials of # to the differentials of
b
iJA#
evaluated at A # A#; - - is the matrix of partial derivatives which maps the
iJA'
=
A
differentials of
i
to the differentials of
i
# evaluated at
i
= A.
Thus our task is to obtain these three matrices of partial derivatives on the
RHS of the above equation. The elementwise expressions for the matrix of
partial derivatives of Ab with respect to A# in the middle of the RHS have already
been given by Archer and Jennrich (1973), and we will present the matrix of
partial derivatives of Ab with respect to A# using a matrix approach. (See also
Hayashi and Sen (199Gb).)
3. The matrices of partial derivatives
Now, we give the expressions for the three matrices of partial derivatives
necessary to compute the asymptotic covariance matrix of the estimates of
normalized varimax-rotated factor loadings.
- 5-
3.1 Normalization: A ---> A#
The first process is to normalize the matrix of unrotated factor loadings A by
premultiplication by H-1I2. The matrix of partial derivatives that maps A to
i..# is
given by
(3)
aA#
aA'
•
= - (1 12)(A'®H-3/2>{«veC(lp» 1pm')#((1p2 + Kpp)(A®l p»}+ Im®H-1I2,
where ® is the Kronecker product; 1pm is the pm-dimensional column vector
whose elements are all 1's; Kpp is a unit matrix defined such that Kppvec(A) =
vec(A') for any p x p matrix A (See, e.g., Magnus and Neudecker (1988».
3.2 Raw varimax rotation: A# ---> Ab
Once the initial unrotated factor loadings are normalized, the rotation is
exactly identical to the raw varimax rotation.
The elements of the matrix of
partial derivatives for the raw varimax rotation were obtained by Archer and
Jennrich (1973). We give a matrix version of the identical results, which are as
follows:
where
(5)
= Jm(2)'(Kmm -1 m2)
f.
{(Jm,iJm,i'®lm)Kmm(
1=1
:~~~
)(Jm,iJm,i'®lp)}
with
(6)
:~~~ =3(1 m1m'®(Ab#Ab)')#(1 m'®Ab'®1 m) - 1m'®(Ab#Ab#Ab)'®1 m
- (lIP)[(1 m'®Ab'®1 m)#{1m®Jm(3)(vec(Ab'Ab»1 pm'
-6-
•
where Km = (K(1)', K(2)', ... , K(m)')' is the m2 x (1/2)m(m - 1) matrix with the m x
(1/2)m(m - 1) unit matrix K(r) which has 1's in the (i, I(i,r» elements, 1 sis r - 1;
-1's in the (r + i, I(r, r + i» elements, 1 sis m - r; O's in the rest of the elements;
and I(i, r) is defined such that I(i, r)
=(1I2)(i - 1)(2m - i) + (r - i),
1 s i < r s m; Jm,i
=
(0, ... ,0, 1, 0, ... , 0)' is a m-dimensional unit vector whose i-th element is 1 and
the rest are O's; J m(1) is the m2 x (1I2)m(m - 1) unit matrix which consists of m(m
- 1) submatrices with the (i, i) submatrix (of order m x (m - i» being of the form
(O(m-i)xi, Im-i)', 1 sis m - 1 (The last m rows of J m(1) are all O's.); Jm(2) is the m2 x
(1/2)m(m - 1) unit matrix which consists of m(m - 1) submatrices with the (i + 1, i)
submatrix (of order m x i) being of the form (Ii, Oix(m-i)', 1 sis m - 1 (The first m
rows of Jm(2) are all O's.) (See Tables 1 and 2 for the components of J m(1) and
J m(2), respectively.); J m(3)
=(vec(Jm,1Jm,1')', ... , vec(Jm,mJm,m')')' is the m x m2
unit matrix which has 1's in the (i, (i - 1)m + i) elements, 1 sis m, and O's in the
•
rest of the elements.
3.3 Denormalization: Ab ---> A*
After performing the raw-varimax rotation on the matrix of normalized
unrotated factor loadings, the matrix of normalized rotated factor loadings need
to be denormalized to cancel out the effects of initial normalization. The matrix
of partial derivatives that maps the differentials of Ab to the differentials of A* is
given by
(7)
dAb
=(1 12)(Ab'®H1I2){«veC(lp»1 pm')#((Ip2 + Kpp)(A®lp)(~r1)}
+ Im®H1/2,
where
-7-
(8)
OAb
OA'
=- (1I2)((T'A')®H-3/2){((veC(l p»1 pm ')#((l p2 + Kpp)(A®l p»} + T'®H-1/2
- (I m®(H-1/2A»[(l m®T)Km{(
:~b' )(l m®Ab)(J m(2) - Jm(1)}1
o~
OA# ]
( OAb' )(T'®lp)(~) ,
where
OA#
OA' '
o~
OAb'
•
, Jm(1), Jm(2), and Km have already been defined in
sections 3.1 and 3.2.
4. Conclusion
In sections 2 and 3, we derived the asymptotic covariance matrix of the
estimates of normalized varimax-rotated factor loadings.
The normalized
varimax rotation is by far the most frequently used method of rotation in practice
of factor analysis. Thus we think that it is important to report the analytical
formulas for the asymptotic covariance matrix of the estimates of normalized
varimax-rotated factor loadings.
In section 2, we presented the process of obtaining the estimates of
normalized varimax-rotated factor loading as consisting of the three stages: (i)
normalization, (ii) raw varimax rotation, and (iii) denormalization. Each stage
has its corresponding matrix of partial derivatives, which was obtained in
section 3. The three matrices of partial derivatives from the three stages were
then combined by the chain rule to derive the matrix of partial derivatives
corresponding to the normalized varimax rotation given in equation (2). For the
stage (ii) we used a matrix version of the existing elementwise formulas for the
matrix of partial derivatives corresponding to the raw varimax rotation.
substantially reduced the burden of the process of derivation.
-8-
It
•
•
REFERENCES
Archer, C. 0., and Jennrich, R. I. (1973). Standard errors for orthogonally
rotated factor loadings. Psychometrika, 38, 581-592.
Hayashi, K., and Sen, P. K. (1996a). The asymptotic covariance matrix for
covariance estimators in factor analysis. Institute of Statistics Mimeo
Series, No. 2172, The University of North Carolina.
Hayashi, K., and Sen, P. K. (1996b). The asymptotic covariance matrix for
covariance estimators with standardization and raw-varimax rotation in
factor analysis. Institute of Statistics Mimeo Series, No. 2173, The
University of North Carolina.
Jennrich, R. I. (1974). Simplified formulae for standard errors in maximumlikelihood factor analysis. British J. Math. Statist. Psychol. 27, 122-131.
"
•
Kaiser, H. F. (1958). The varimax criterion for analytic rotation in factor analysis.
Psychometrika, 2 3, 187-200.
Magnus, J. R., and Neudecker, H. (1988). Matrix differential calculus with
applications in statistics and econometrics. Wiley, New York.
Neudecker, H. (1981). On the matrix formulation of Kaiser's varimax criterion,
Psychometrika, 4 6, 343-345.
Sherin, R.J. (1966). A matrix formulation of Kaiser's varimax criterion.
Psychometrika, 3 1, 535-538.
- 9-
Table 1. Components of unit matrix J m(1)
01 x(m-1)
01 x (m-2)
1 m-1
(m-1)x(m-2)
2x (m-2)
2x (m-1)
(m-2)x(m-1) 1 m-2
0
3x(m-2)
o
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
o (m-2) x2
12
o (m-1) x2
o (m-2) x 1
o 2x 1
o (m-1)x1
°1x2
Omx2
o mx1
o
o
o
o
11
Note: lis are the identity (sub)matrices and O's are the null (sub)matrices. The
subscripts indicate the orders of the submatrices.
Table 2. Components of unit matrix J m(2)
•
o mx1
o mx2
11
(m-1)x1
2x 1
(m-2) x 1
o
o
o
01x2
(m-1)x2
12
(m-2) x2
o
o
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
o 3 x (m-2)
0
1 m-2
o (m-2)x(m-1)
o 2x(m-2) o 2x(m-1)
o (m-1)x(m-2) 1m-1
01 x (m-2)
o 1x(m-1)
Note: lis are the identity (sub)matrices and OIS are the null (sub)matrices. The
subscripts indicate the orders of the submatrices.
- 10-
•
PROOFS
(1) Proof for the matrix of partial derivatives of
,..# with respect to A in (3)
dA#
dh- 1I2
(i) Express ---;;:;.: as a matrix function involving dA'
A# = H-1/2A
dA# = (dH-1/2)A + H-1/2(dA)
dA# = vec(dA#)
= vec«dH-1/2)A) + vec(H-1I2(dA»
= (A'®l p)vec(dH-1/2) + (Im®H-1/2)vec(dA)
= (A'®lp)(dh- 1I2) + (I m®H-1/2)(dA)
dA#
dh- 1/2
- - = (A'®l p)(
) + Im®H-1/2
dA'
dA'
•
dh
(ii) Obtain the expression for - dA'
H
dH
=lp#(M')
= Ip#(d(M'»
= Ip#«dA)A' + A(dA'»
dh
= vec(dH)
= vec(lp#«dA)A' + A(dA'»)
= vec(lp)#Vec«dA)A' + A(dA'»
= vec(lp)#(vec«dA)A') + vec(A(dA'»
It
= vec(lp)#«A®lp)vec(dA) + (lp®A)vec(dA'»
= vec(lp)#«A®lp)(dA) + (lp®A)Kpm(dA»
- 11 -
= vec(lp)#«A®l p + (lp®A)Kpm)(dA))
=vec(lp)#«A®l p + Kpp(A®lp))(dA))
=vec(l p)#((Ip2 + Kpp)(A®lp)(dA))
ah
aA'
=«vec(l p))1 pm ')#((1p2 + Kpp)(A®lp))
(iii) Obtain the expression for
dh- 1I2
aA'
dH-1/2 = - (1/2)H-3/2(dH)
~
dh- 1/2
(since H is diagonal)
= vec(dH-1/2)
= - (1/2)vec(H-3/2(dH))
= - (112)(l p®H-3/2)vec(dH)
= - (112)(l p®H-3/2)(dh)
~
~
ah- 1/2
d~
dh- 1/2
dA'
•
dh
= - (112)(l p®H-3/2)(--)
a~
=- (1 12)(lp®H-3/2)«(vec(lp)) 1pm')#((Ip2 + Kpp)(A®l p)))
ah
U)
(We used the formula for - - in (II .)
dA'
(iv) Combine (i) and (iii) to obtain (3)
dA#
--;;:;:
=- (1 12)(A'®Ip)(l p®H-3/2)«(vec(l p)) 1pm')#((Ip2 + Kpp)(A®l p)))
+ Im®H-1I2
= - (1 12)(A'®H-3/2)«(vec(l p)) 1pm')#((Ip2 + Kpp)(A®l p))) + Im®H-1I2
(2) Outline of proof for the expression for the matrix of partial derivatives of Ab
with respect to A# given in (4) - (6)
-12-
•
Although the matrix expression for the partial derivatives necessary for the
raw varimax rotation has already been given in Hayashi and Sen (1996b), we
give the outline of proof here for completeness.
First, the coordinatewise
expression for the elements of the matrix of partial derivatives given by Archer
and Jennrich (1973) are
m-1
m
= Oij1sr - ~
(9)
~
m (
a;uv
)
'\" eiruv ( aAI'tb ) tst .
u=1 v=u+1 ~
Here, E
=(eiruv) has the elements
m
~1
eiruv = ~ AitbL!(t,r),I(u,v) - ~ AitbL!(r,t),I(u,v),
t=1
t=r+1
(10)
where
Uj is the (i, j) element of the inverse of the matrix L, and the (I(r,s), I(u,v»
element of L is
•
_
(11 )
~
LJ(r,s),I(u,v) - 4.
(A' b(
IU
1=1
with I(r, s)
a;rsb ) - A'IVai.·
b( a;rs »)
ai..
b '
'''IV
=(112)(r - 1)(2m - r) + (s - r),
'''IU
1 s r < ssm, and I(u, v)
=(1/2)(u - 1)(2m -
u) + (v - u), 1 s u < v s m. For 1 s r < ssm,
p
b
b
{{Ajr )2 - (Ajs b)2) + 2 Air }: (AjrbAjsb)},
J=1
a;rs
aAisb
a;rs
aAitb
=0 , for all t
pi!
s.
•
Now, we can easily verify that (5) and (6) are the matrix expression
corresponding to (12). (Note that 1m1m'®Ab' and 1m1m'®(Ab#Ab)' correspond
- 13-
to Airb and (Airb)2; Ab'®1 m1m' and (Ab#Ab#Ab)'®1 m1m' corresponds to Aisb and
(Aisb)3,
respectively.
Note
also
that
1m®J m(3)(vec(Ab'Ab» 1pm'
and
J m(3)(veC(Ab'Ab»®1 m1 pm ' correspond to (Ab'Ab)rr and (Ab'Ab)ss, respectively.)
Next, the matrix expressions for (10) and (11) are
(13)
(14)
respectively, where Km, Jm(1), and Jm(2) have already been defined in the text.
Thus, using (13) and (14), we obtain the matrix expression for (9) as
dAb
dA#
d~
= (I pm - E ( dAb' »(T'®lp)
=(I pm - (I m®Ab)KmL-1(
d~
dAb' »(T'®lp)
=[I pm - (Im®Ab)Km{( :~b' )(l m®Ab)(J m(2) - Jm(1)}1( d~~'
)]
(T'®lp).
•
(3) Proof for the matrix of partial derivatives of A* with respect to Ab in (7)
(i) Express
dh 1/2
as a matrix function involving - dA'
A* = H1/2Ab
dA*
dAb'
1l2
=(Ab'®l p)( dhdAb'
) + Im®H1/2
dh 1/2
(ii) Obtain the expression for - dA'
•
dH1I2 = (1I2)H1I2(dH)
=>
dh 1l2
=vec(dH1I2)
-14-
=(1I2)vec(H1/2(dH»
=(1/2)(l p®H1/2)(dh)
=(1I2)(lp®H1/2){«veC(lp»1 pm')#((Ip2 + Kpp)(A®lp)( a~:'
»}
aAb
•
= (1I2)(l p®H1/2){«veC(l p»1 pm')#((Ip2 + Kpp)(A®lp)(a~;-·-r1)}
(We used the expression for the differential dh in section (1) in the proofs.)
(iii) Combine (i) and (ii) to obtain (7)
(4) Proof for the matrix of partial derivatives of Ab with respect to A in (8)
aAb
ah
at
.
(i) Express - - as a matrix function involving - - and - - , and Insert
a~
a~
aA'
ah
the expression for - - in section (1) in the proofs
aA'
,
Ab
=A#T =H-1I2AT
dAb = (dH-1/2)AT + H-1/2(dA)T + H-1I2A(dT)
=- (1I2)H-3/2(dH)AT + H-1/2(dA)T + H-1I2A(dT)
dAb
=vec(dAb)
=- (1I2)vec(H-3/2(dH)AT) + vec(H-1/2(dA)T) + vec(H-1I2A(dT»
= - (1I2)«T'A')®H-3/2)vec(dH) + (T'®H-1I2)vec(dA)
+ (I m®(H-1/2A»vec(dT)
=- (1I2)«T'A')®H-3/2)(dh) + (T'®H-1/2)(dA) + (I m®(H-1/2A»(dt)
ah
at
=- (112)«T'A')®H-3/2)(--)
+ T'®H-1I2 + (Im®(H-1/2A»(--)
.
a~
a~
=- (1 12)«T'A')®H-3/2){«veC(lp»1 pm ')#((1p2 + Kpp)(A®l p»}
-15 -
at
+ r®H-1/2 + (I m®(H-1/2A»(--)
a'A'
(ii) Obtain the matrix expression for
at
a:;:
We can obtain the matrix expression for
~
a'A'
in the identical way to the
expression for the matrix of partial derivatives of t with respect to the sample
covariance matrix arranged as a vector given in Hayashi and Sen (1996b).
First, equation (12) in Archer and Jennrich (1973) gives
dT = - (TT')-1TL-1(d;((dA#)T»
= - TL-1 (d;((dA#)T»,
which is, in our notation, equivalent to
f
where
~1
m
giruv = ~ ~tL'(t,r),I(u,v) - ~ ~tL'(r,t),I(u,v),
t=1
t=r+1
with I(r,s)
arrange
=(112)(r - 1)(2m - r) + (s - r); 1 s i, r, ssm, 1 s j s p, 1 s u < v sm.
d~r
Now,
as
(15)
and construct the matrix expression for giruv:
(16)
-16-
•
.
as
where the order of G m IS m 2 x (1I2)m(m - 1), and --b ,J m(1), J m(2), and Km are
a'A '
all defined in the text. Then the above coordinatewise expression in (15) can
be written in matrix form as
dt
= d(vec(T))
=_G m( as
a'A.*'
)vec((dA#)T)
as
=- (Im®T)Km{( :~b' )(Im®Ab)(Jm(2) - Jm(1))}-1( a'A.*'
)(T'®lp)(d'A.#).
(We used the expression for G m in (16).) Thus
as )(Im®Ab)(J m(2) - J m(1))}-1 ( as
-at- =- (Im®T)Km{ (--b
a'A'
a'A*'
a'A '
(iii) Combine (i) and (ii) to obtain (8)
•
- 17-
a'A.#
)(T'®l p)(--)
a'A.'
Appendix: Some Properties of Matrix Operations Used in Proofs Section
(which are given in Magnus and Neudecker (1988»
(i)
(ii)
(A®B)(C®D)
= (AC)®(BD)
(A®B)' =A'®B'
(iii)
Kmnvec(A) = vec(A ')
(iv)
Kpm(A®B) = (B®A)Kqn (for m x n matrix A and p x q matrix B)
(v)
vec(ABC) = (C'®A)vec(B)
(for m x n matrix A)
(via)
vec(AB)
= (B'®lm)vec(A)
(vib)
vec(AB)
= (lq®A)vec(B)
(vii)
vec(A#B) = vec(A)#Vec(B)
(viii)
d(AB) = (dA)B + A(dB)
-18-