The Robustness of the Likelihood Ratio Chi-Square Test for Structural Equation Models: A
Meta-Analysis
Author(s): Douglas A. Powell and William D. Schafer
Reviewed work(s):
Source: Journal of Educational and Behavioral Statistics, Vol. 26, No. 1 (Spring, 2001), pp. 105132
Published by: American Educational Research Association and American Statistical Association
Stable URL: http://www.jstor.org/stable/3657940 .
Accessed: 15/09/2012 17:44
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .
http://www.jstor.org/page/info/about/policies/terms.jsp
.
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact [email protected].
.
American Educational Research Association and American Statistical Association are collaborating with
JSTOR to digitize, preserve and extend access to Journal of Educational and Behavioral Statistics.
http://www.jstor.org
Journalof Educational
andBehavioralStatistics
Vol.
2001,
26, No. 1,pp. 105-132
Spring
The Robustness of the Likelihood Ratio Chi-Square
Test for Structural Equation Models: A Meta-Analysis
Douglas A. Powell
William D. Schafer
Universityof Maryland,College Park
structural
robustness,
equations
Keywords:meta-analysis,
The robustnessliteraturefor the structuralequation model was synthesizedfol-
lowingthe methodof Harwellwhichemploysmeta-analysisas developedby
ontheexplanation
HedgesandVevea.Thestudyfocused
ofempiricalTypeI error
ratesfor six principal classes of estimators: two that assume multivariatenor-
leastsquares),ellipticalestimators,
likelihoodandgeneralized
mality(maximum
two distribution-freeestimators (asymptoticand others), and latent projection.
Generally,the chi-square testsfor overall modelfit werefound to be sensitive to
non-normalityand the size of the model for all estimators (with the possible
exceptionof the elliptical estimatorswith respectto modelsize and the latentpro-
Theasymptotic
distributionjectiontechniqueswithrespectto non-normality).
free (ADF)and latentprojectiontechniqueswerealsofoundto be sensitiveto
samplesizes. Distribution-freemethodsotherthanADF showed,in general, much
less sensitivityto all factors considered.
The use of meta-analyticmethods to summarizeMonte Carlo(MC) robustness
and power studies was suggested by Harwell (1992). The rationalefor the use of
this procedurewas to correctdefects typically associatedwith the use of MC studies. The generalmeta-analyticframeworkof Hedges andOlkin (1985) permitsthe
modeling of both empirical Type I errorrates and power values that have been
reportedin MC studies, by means of weighted least squaresregressionanalyses.
Various study characteristicsserve as explanatoryvariables.
Harwell (1992) applied the procedureto synthesize the results of MC studies
of the robustnessof the Bartletttests for homogeneityof variancesagainstviolation of the normalityassumption.A second applicationwas providedby Harwell,
Rubinstein,Hayes, and Olds (1992), who meta-analyzed34 studies that involved
robustnessfor the F test, the Welch test, and the Kruskal-Wallistest for the oneway ANOVA model and the F test for two-way ANOVA. A thirdapplicationof
the Harwell (1992) procedure was the work of Keselman, Lix, and Keselman
(1993), who synthesized findings of 15 differentstudies on the robustnessof four
differenttest statistics used in repeatedmeasuresdesigns.
An area of much attentionin the statisticalliteratureis the structuralequation
model (SEM) (Jtreskog, 1973), which has been shown to have a diversityof applications in many domains(e.g., education,psychology, sociology, criminology).In
105
Powell and Schafer
the psychometricfield the model has had an impact in increasedusage of confirmatoryfactoranalysis(CFA).This modeling strategypermitsthe testingof particularfactormodelswherethe observedvariablesaremodeledas a linearcombination
of a set of latent variables(factors).More complex models permit modeling the
structureof latentvariables.
The estimatesof modelparametersfor SEM areobtainedby the minimizationof
a fittingfunctionbasedon a given estimationmethod(see "EstimationMethodsfor
SEM").Assessmentof modelfit is accomplishedvia the likelihoodratiochi-square
statistic.The appropriatevalue of the chi-squarefit statisticis calculatedby multiplying the fittingfunctionfor the estimationmethodby n - 1, where n represents
the sample size. The formulafor the likelihoodratiochi-squareis thus given by:
x2 = (n-
1)F,,
(1)
where F, representsthe fittingfunctionfor estimationmethodcc.
The sensitivity of the chi-squaretest to deviations from multivariatenormality
andto otherfactorssuch as small sample size, underdifferentestimationmethods,
has received a considerableamountof attentionvia Monte Carlo(MC) studies(see
"FactorsAffecting Type I ErrorRates for SEM").This study attemptsa synthesis
of thatrobustnessliterature.
Estimation Methods for SEM
Normal TheoryEstimators
The normaltheoryestimatorsassume the data are sampledfrom a multivariate
normaldistribution.The methodof maximumlikelihood (ML) is one of the more
common estimationmethodsused in SEM. Anothercommon estimationmethod
for SEM is the methodof generalizedleast squares(GLS).Both methodsare summarizedby Bollen (1989). The fittingfunctionsfor these two estimationmethods
are:
FML= lntL(8)l+ tr[S2-7'(0)]- InlSI- (p + q),
(2)
where
1(0) = the populationcovariancematriximplied by the model,
S = the sample covariancematrixfor observedvariables,
p = the numberof observedexogenous variables,and
q = the numberof observedendogenousvariables.
FGLS=
.50tr({[S -
(O)]W-1}2),
(3)
where Wis a weight matrixthatis typically chosen to be equal to S, althoughany
positive definitematrixof constantsor any matrixthatconverges in probabilityto
a positive definite matrixmay be used. A special case of GLS is the unweighted
106
Robustnessfor StructuralEquationModels
least squaresestimator,ULS, which occurs when W is taken to be equal to I, the
identitymatrix.
Elliptical Estimators
The elliptical estimators assume the multivariatedistributionof the observed
datais symmetricbut permitsunivariatekurtosesthatdeviate from the kurtosisof
a normaldistribution.However,the same univariatekurtosisparameteris assumed
for all observedvariables.Fourellipticalestimatorshave been studiedand aresummarizedby Harlow (1985). These estimators (El, E2, E3, E4) differ only in the
methodby which the commonkurtosisparameteris estimated(Harlow, 1985). The
fittingfunctionfor the elliptical estimatorsis given by Bollen (1989):
FE
= .50(K + l)-'tr{[S - Z(O)]W-'}2 - C{tr[S - l()]W-'}2,
(4)
where K is the common kurtosisparameter,C1is a constantcomputed by means
of K,p, andq, and Wis the weight matrix,which is typically selected to be the sample covariancematrix,S. The four differentestimatorsof the common univariate
kurtosisparameterare given by Harlow (1985).
The first estimator(k1)is given by the Mardia(1970) coefficient of multivariate relative kurtosis minus unity [see (15) below]. The second estimator (KC2)is
given by:
K2 = (S2S4)(S2S2) -
1,
(5)
where s2 is a vector with elements (susji+ sikSjl+ silsjk) and s4 is a vector with elements(Sijkl),wheresijis the samplecovariancefor variablesi,j andsijklis the fourthordersample productmomentfor variablesi, j, k, 1.
The thirdestimator(iK3)
is the mean of the scaled univariatekurtoses.The scaled
univariatekurtosis(y2) involves the ratio of the fourthsample moment for a variable to the squareof the second sample moment of that variable,with three subtractedfrom the resultingratio.The estimatorK3 is thus given (forp variables)by:
P
K3 =
i=1
2i /3p ,
(6)
wherey2,iis the scaled univariatekurtosisfor the ith variable.
The fourthestimator(k4)is given by:
K4
i=1
,
Kjkl /m
(7)
= (Sijkl)/(SijSkl
+ SikSjil
+ SilSjk),with sil and Sijkldefined as in (5) and m =
where Kijkl
p(p + 1)(p + 2)(p + 3)/24.
107
Powell and Schafer
In additionto the trueellipticalestimators,five elliptically correctedestimators
have been studied.These estimatorsproducetheirchi-squarestatisticsby dividing
a normaltheorychi-squarestatisticby one of the estimatorsof the common kurtosis parameterplus unity. The estimators CML1, CML2, CML3, and CML4
involve the correctionby division of the ML estimatorfor non-normalkurtosis,
wherethe correctionfactorinvolves one of the fourestimatorsof the common kurtosis. The elliptically correctedquadraticform estimator(CQF) involves the use
of the firstestimateof the common univariatekurtosisto correctthe GLS estimator by means of division.
Distribution-FreeEstimators
The distribution-freeestimatorsmakeminimalor no assumptionsaboutthe populationdistributionof the observedvariablesother than thatthey have a continuous probabilitydensity function.In otherwords, these estimatorstypically permit
both non-normalskewness and non-normalkurtosis.
The most widely used of the distribution-freeestimators is the asymptotic
distribution-free(ADF) estimator(Browne, 1984). Anotherdistribution-freeestimator is the DADF estimatorpresentedin Henly (1991). This estimatorhas the
same fittingfunctionas ADF butreplacesf by a weight matrixcontainingonly the
diagonalelements ofIF.
The fittingfunctionto be minimizedfor ADF is given by Harlow (1985):
FADF= (s* -
*)t 1(s* -
*),
(8)
wheres* is a (p + q)(p + q + 1)/2 column vectorcontainingthe nonredundantelementsof S and cY*is a (p + q)(p + q + 1)/2 columnvectorcontainingthe nonredundantelementsof X(O).F is a weightmatrixwhose elementsaregiven by:
=
Y"ijkl
Yijkl -
Gij(klY ,
(9)
where
= E[(yi - gi)(Yj - pj)(Yk - Pk)(YI- 91)],
- i)(yj and
Yij= E[(yi
•gl)],
aijki
=i=
E(yi).
IFis obtainedby replacingthe populationmoments by the correspondingsample
moments.
Otherapproachesare groupedunderthe distribution-freerubric.The heterogeneous kurtosisestimator(HK) was utilizedby Hu, Bentler,and Kano (1992). This
estimatorrequiresa symmetricdistributionbut, unlike the elliptical estimators,
permitsthe univariatekurtosisparametersto be different for the observed variables. The fittingfunctionfor HK is given by:
FHK= .50tr[S-
108
()iC'•]2,
(10)
Robustnessfor StructuralEquationModels
where
C = A*S and* indicatorselementwisemultiplicationformatricesof sameorder,
A = (da), and
+ 2,j1/22
a2 [(i2,i1/)2
=
The SCALED estimator, developed by Satorra and Bentler (1988) and also
employedby Hu et al. (1992), uses a correctiontechniquesimilarto the elliptically
correctedestimators.However, the estimatorpermitsan arbitrarycontinuousdistribution.The techniqueinvolves the division of the ML chi-squarestatistic by a
factorof k. The factork is given by:
k = UF-',
(11)
whereU = IF- 'P-1,and a is the vectorof model impliedeleT-))
F'mentsof the covariancematrix(1) and F is the weightmatrixfor ADF.
LatentProjectionEstimators
The latent projectionestimators attemptto estimate the parametersof a SEM
with continuous variables that have been either categorized or censored by the
measurementprocess.
The CVM methodof Muthen(1984) whichwas studiedby Potthast(1991) is used
with categoricalobservedvariables.This method,featuredin the computerpackage
LISCOMP,operatesessentiallyby analyzingthe matrixof polychoriccorrelations
by means of the GLS estimationmethod. A similar approachis employed by the
LISREL computerpackage and is denoted the WLS PRELIS estimator(Dolan,
1994).
The TOBIT approachis a final latent projectiontechnique,which involves the
fittingof the SEM for the underlyingcontinuousvariablesif the observedvariables
have been censored.This approachwas developed by Muthen(1985).
Factors Affecting Type I Error Rates for SEM
From an assessment of the exact theoreticalassumptionsfor the various estimationmethods, it follows that the empiricalType I errorratesfor normaltheory
estimatorsshouldbe sensitive to deviationsfrommultivariatenormality.Likewise,
the elliptical estimatorsshould be sensitive to skewness, thoughperhapsnot kurtosis. Both the distribution-freeand latentprojectionmethodsare not theoretically
dependent on any particulardistributionalform; however, this claim requires
empiricalconfirmation.
Because of the size of the weight matrixassociated with both the distributionfree and the latentprojectiontechniques,these estimationclasses should be associated with computationaldifficulties for large models (i.e., those with many
estimatedparameters-typically assessed in MC studiesby degreesof freedomfor
the model). These computationaldifficultiesmay lead to inflatedchi-squarevalues
for these estimationmethods (Muthen& Kaplan, 1992; Potthast,1991).
109
Powell and Schafer
Lastly, sample size has been cited as a complicationin the attainmentof proper
chi-squarevalues for SEM models (Harlow, 1985). Sample size was investigated
in a majorMC study by Boomsma (1983) for normaltheory (ML) estimatorsand
by Henly (1991) and Hu et al. (1992) for a varietyof otherestimationmethods.In
general, normaltheory estimatorshave shown limited sensitivity to sample size,
while distribution-freemethods,particularlythe ADF method,have shown inflated
chi-squarevalues at low sample sizes.
Meta-Analysis of Structural Modeling Robustness Literature
Problem Formulation
The fundamentalresearchquestion for this study was: What factors explain
empirical Type I errorrates for the chi-square tests in MC studies of structural
equationmodels?Forthe purposesof this study,to be considereda structuralequation model, a model must contain at least one continuouslatent variable (i.e., no
studies involving only observedvariableswere included). EmpiricalType I error
rates were modeled by meta-analytictechniques.Explanatoryvariableswere various studycharacteristics.
Data Collection
A populationof MC studiesfor linearstructuralmodels was identifiedby searching variousdatabases.The databasessearchedwere: ERIC,DissertationAbstracts
International,Psyclit, Econolit,Sociofile, BusinessIndex,LIFE,and Mathsci.Key
words used in the searchesinclude:LISREL,LISCOMP,EQS, Monte Carlo,chisquare,robustness,simulation,multivariate,normality,nonnormality,skewness,
kurtosis, asymptotic distribution-free,maximum likelihood, generalized least
squares,two-stageleast squares,ADF, ML, GLS, 2SLS, factoranalysis, structural
equationsmodeling, Type I error,and covariancestructures.
The databaseswere searchedby each of the keywords. Items were examinedby
means of the abstractprovidedin the database.If an item could not be positively
excluded as an eligible studybased on the readingof the abstract,its citation was
recordedand the correspondingarticle, conference paper, dissertation,or technical reportwas examined.A total of 219 reportswere selected for examination.
These studies were examined to determine:(a) did the model that was investigated qualify as a structuralequationmodel? (i.e., did it contain at least one continuouslatentvariable?);(b) did the methodologyof the studyinvolve MonteCarlo
methods?;(c) could multivariateskewness and multivariatekurtosisbe estimated
for the data;and(d) assumingthe use of MonteCarlomethods,did the studyreport
empiricalType I errorrates?If the answerto either(a), (b) or (c) was no, the study
was excluded.If the answersto (a), (b), and (c) were yes, but the answerto (d) was
no, the authorof the studywas contacted(if possible) to obtainthe empiricalType I
error rates. Five studies fell into this category and in one case (i.e., Benson &
Fleishman, 1994), the empiricalType I errorrateswere obtainedfrom the authors
(J. Fleishman, personal communication,July 10, 1995). In four other cases, the
110
Robustnessfor StructuralEquationModels
authorswere unableto provideempiricalType I errorrates.However, in one case
an applicablestudy was obtainedfrom an author(J. Finch, personalcommunication, August 5, 1995) and includedin the study.
Additionally, five journals were searcheddirectly by examining every article
appearingin them since 1980. These journalswere: Applied Psychological Measurement,BritishJournalofMathematicaland StatisticalPsychology,Multivariate
BehavioralResearch,Psychological Bulletin,and Psychometrika.These journals
were chosen becausethey were majorjournalsin the field of quantitativemethods
and were the most commonjournalsin which applicablestudieswere found.
The foregoing proceduresyielded 25 applicablestudies. As a final component
of the search,five scholarsin the field of structuralequationsmodeling were contacted, suppliedwith a list of these 25 studies, and asked to supply the references
for any additional studies that met the criteriafor applicability. These scholars
were: Dr. PeterBentler,Dr. Anne Boomsma,Dr. Lisa Harlow,Dr. David Kaplan,
andDr. Bengt Muthen.As a resultof these communications,one additionalapplicable studywas received (P. Bentler,personalcommunication,September8, 1995).
Those cases with empirical Type I errorrates equal to either zero or one were
excludedfromthe meta-analysisbecausethese cases did not permitthe calculation
of the samplingvariancefor the empiricalType I errorrate. Of the original 2089
cases, 161 were excluded for this reason.
While only 26 studies were included in the meta-analysis, all of the studies
yielded multipleempiricalType I errorrates.EachreportedempiricalType I error
rate was treatedas a separatecase.
Estimating Non-Normality Variables. Since the studies to be meta-analyzed
employ differentnumbersof variableswith variouscombinationsof skewness and
kurtosis(e.g., Harlowuses 13 differentcombinationsof univariateskewness and
kurtosiswith six observed variables),it is necessaryto have indices of skewness
and kurtosis that compactly representthe non-normalityof the data across all
conditionsin themeta-analysis.Tihemultivariateskewnessandkurtosiscoefficients
of Mardia(1970) wereused. Fora finitepopulationof size N, multivariateskewness
and multivariatekurtosis(P2,p) are definedas follows:
(Pp)
3
NN
=
•i,,
N-2X
t1
,
-
i=1 j=1 [(Yi
-(Y
-
•y
2
NN
p2.p = N-2Xi=1
(12)
,)]
j=1
[(Yi -
Ity)
,
(13)
ly)t•-1(Yi
where
Y,= the ith multivariate(p variables)observationin the population,
= the populationcentroidfor the p variables,and
ty=
p x p covariancematrixfor the p variables.
111
Powell and Schafer
The sample estimatesfor P,p and
2,pin a sample of
=n-2 [(Y
- Y)S-(Y
b.,
i=1 j=1
size n are given by:
3
- Y)]
(14)
2
b2,p
=n
[(
i=1 j=1
-
S-'(Y - Y) ,
(15)
where
Y= the populationcentroidfor the p variables,and
S = p x p sample covariancematrixfor the p variables.
The values of Pi,pandP2,pare0 andp(p + 2), respectively,for a multinormaldistribution.Multivariaterelative kurtosis (Muthen & Kaplan, 1985) is defined as
+ 2)] and representsthe ratioof the multivariatekurtosisof a given disk2,p/[p(p to thatof a multivariatenormaldistribution.
tribution
Since the values of the multivariateskewness (MS) andthe multivariaterelative
kurtosis(MRK)were notreportedin the studiesto be meta-analyzed,a programwas
writtento estimatethem.In the multivariatenormalcase, the MS andMRK values
were estimatedfor samplesizes of 1000 or less. For samplesizes greaterthan 1000,
E(MS) andE(MRK)were used because,at or above this samplesize, estimatedvalues approximatedthe expectationsvery closely. The expected values of MS and
MRKareequaltop(p + 1)(p + 2)/n and(n - 1)/(n+ 1), respectively(Mardia,1970).
In the cases where non-normaldata were generated,two typical methods were
observed. In the more typical scenario, data are generatedas multivariatenormal
accordingto a "true"model covariancematrix.The dataaretheneithercategorized
(i.e., into Likertscale type variables)or censored from above or below (only two
studies involve censoring).In the less typical scenario,continuousdataare generated accordingto the "true"model covariancematrixwith prespecifiedunivariate
skewness andkurtosiscoefficients.This is accomplishedby meansof the Vale and
Maurelli(1983) procedureor by transformingdatafrom a multivariatenormaldistribution(e.g., multivariateX2or multivariatet).
The studies availablefor the meta-analysisare listed in Table 1 along with several key variables. These variables include: Samp (the range of sample sizes is
listed), Reps (the numberof replicationsemployed in the study), Skew/Kurt(the
methodof inducingnon-normality,i.e., C = categorizationor censoring,V = ValeMaurelli,T = transformationof multinormaldata, N = multinormaldata), Model
Size (the range of the degrees of freedom for the chi-squaretest that yielded the
empirical Type I errorrates is listed), EstimationClass (N = normaltheory, E =
elliptical estimationmethods,D = ADF andothermembersof the distribution-free
class, L = latent projection),and # (the numberof cases for this study which met
the criterionfor inclusion in the meta-analysis,i.e., the empiricalType I errorrate
is not equal to either zero or one).
112
Robustnessfor StructuralEquationModels
TABLE 1
Availablestudies
Study
Samp
Reps
Skew/
Kurt
Model
Size
Estim
Class
#
T
4
300
1000
3
Satorra& Bentler
N,D
(1990)
300
75-9600
N148
T,N
5
Henly (1991)
200
219
150-5000
87-93
Hu et al. (1992)
T,N
N,E,D
N
75-500
50
50-51
N
6
Silvia (1988)
C
4
1050
100
60
Muthen (1989)
N,D
Chou & Bentler
N
N
4
100-800
1000
49
(1990)
500
20
20
8
Browne (1984)
T,N
N,E,D
Yung & Bentler
250-500
200
47
87
T,N
N,D
(1994)
Muthen & Kaplan
25
2
1000
16
C,N
N,D
(1985)
Muthen & Kaplan
1000
500-1000
8-87
98
C,N
N,D
(1992)
L
100
C
500-1000
32
Potthast(1991)
2-98
100
23
200
C
2-5
Brown (1989a)
N,L
25-400
100
2
35
Brown (1989b)
C,N
N,D,L
200-400
100-300
8-14
Harlow (1985)
517
V,N
N,E,D
Chou, Bentler, &
100
V
72
200-400
8-14
Satorra(1991)
N,E,D
Harlow, Chou, &
100
200-400
48-60
79
Bentler (1986)
V,N
N,E,D
Curran,West, &
200
100-1000
24
Finch (1995)
36
N,E,D
V,N
Waller & Muthen
100-1000
500
3
24
C,N
N,E,D
(1992)
Yuan & Bentler
500
D
150-1000
87
36
T,N
(1995)
100
N
300-1000
C
36
1
Salomaa (1990)
300
2
N
Babakus (1985)
100-300
160
C,N
Wendler (1993)
250-2000
50
C
26-100
31
N,L
100-1000
100
28-60
73
N,D,L
C,N
Kaplan(1991)
200-1000
100
C
Dolan (1994)
20-104
86
N,L
4-20
N
25-800
100-300
94
Boomsma (1983)
C,N
of multinormal
Note. Skew/Kurt= methodof inducingnonnormality
data,
(T = transformation
N = multinormal).
V = Vale-Maurelli,
C = categorization,
ModelSize = degreesof freedomfor
model.EstimClass= generalclassof estimationmethod(N = normaltheory,E = ellipticalclass,
L = latentprojection).
# = numberof cases.
D = distribution-free,
113
Powell and Schafer
Data Evaluation
The studies were screenedfor methodologicalcriteria.For example, if a study
used an inadequatenumberof replicationsof the simulationprogram(e.g., Fuller
andHemmerle(1966) used only a single replication),it would not affordsufficient
data for estimation of Type I errorrate and was thus excluded from the analysis.
Two aspects of the simulationprocess were scrutinizedas suggested by Harwell
(1992). First, the data generationitself was evaluated(i.e., is an establishedrandom numbergeneratorbeing used, e.g., a IMSLFORTRANsubroutineor the simulation routine of a major structuralmodeling program such as LISCOMP
[Muthen,1987]?) Second, do the patternof empiricalType I errorratesapproximate theirnominalvalues when all relevantassumptionshave been met?
The Benson and Fleishman(1994) study was excluded from the meta-analysis
on this basis. Of the eight cases in the study where normaldata were used, six of
these cases hadempiricalType I errorratesmorethantwo standarderrorsin excess
of the expected level of .05. All eight of the cases were more than one standard
errorabove the expected level. This was truefor boththe ML andADF estimators.
Thus, in accordancewith the recommendationsof Harwell (1992), the study was
excluded because empiricalType I errorrateswere substantiallydeviantfrom the
expected level when all assumptionswere met.
In studies where multipleestimationmethodswere investigated,the same simulateddatawas often used to fit structuralmodels underdifferentestimationmethods. This leads to dependenciesamongthe empiricalType I errorrateswhich were
calculatedfrom the same simulateddata.
Data Analysis
Mixed-effects regressionmodels were developed as describedin Hedges and
Vevea(1998). Thedependentvariablefortheregressionmodelsis theempiricalType
I errorrate.The empiricalTypeI errorrateis by definitiontheproportionof timesthat
thechi-squarestatisticwas foundto rejectformodelfit (( = .05).The fittingof mixedeffects regressionmodelswas achievedthrougha two-stepprocedure.First,a fixedeffects model was fit to the data.This is a weightedleast squaresregressionmodel
whereeachcase is weightedby theinverseof its samplingvariance(Hedges& Olkin,
1985).The samplingvariancefor an empiricalType I errorrateis equalto:
Var(TYPE I) = (TYPE I)(1 - TYPE I)/REPS,
(16)
where REPS is the numberof replicationsused to generatethe data on which the
empiricalTYPE I errorrate was computed.
The various studycharacteristicsserve as explanatoryvariables.The regression
equationfor a weighted least squaresregressionmodel is given by:
(TYPE I)i = b0 +
114
p
_ biX,.
i=1
+ ej,
(17)
Robustnessfor StructuralEquationModels
where
bo= interceptterm,
bi = slope term for ith explanatoryvariable,
Xii= value of the ith explanatoryvariablefor thejth case, and
ej = errorterm forjth case.
After having fit a fixed-effects regression model, the QE statistic (Hedges &
Olkin, 1985) is obtained and used to calculate an estimate of the between-study
In the mixed-effects model, effect sizes have two
variance component (i.e.,
,2).
sourcesof variation,a conditional
samplingvariancecomponent,estimatedby (16)
above, and a between-studyvariancecomponent,reflectingthe fact thatthe effect
sizes includedin the meta-analysisarenot fixed but unknownconstants,butrather
randomvariableswith a distributionof theirown (Hedges & Vevea, 1998). The t2
term incorporatesthe variationamong the effect sizes and is estimatedby r2 as follows (Hedges & Vevea, 1998):
2 = [QE- (k - 1)]/c
=0
2 k -1
if
QE
if
QE< k-l,
(18)
where
= QE statisticfrom fixed-effects analysis,
k = numberof effects sizes (empiricalType I rates)includedin the analysis,
QE
k
k
c=
i=1
wi-
i
k
(wi)2
i
=
wi , and
wi = the weight for the ith effect size in the fixed-effects analysis [i.e., wi = the
reciprocal of the fixed-effects sampling variance as given by (16)].
The unconditionalsamplingvarianceof an empiricalType I errorrateis thus:
v* = v, + T2,
(19)
where
vi = the conditionalsamplingvarianceestimatedby (16), and
1?= the between-studyvariancecomponentestimatedby (18).
The properweights for the mixed-effects analysis arethen obtainedby reciprocating the estimatesof the v*, where the vi's are estimatedby (16) and the t2 terms
are estimatedby (18). The mixed-effects models are "mixed"becausethe explanatory variablesare fixed variablesand the between-studyeffects are randomvariables. The analyses arethenrunagainwith the same explanatoryvariablesbut with
the new regressionweights.
115
Powell and Schafer
Analytical Scheme. The relevant literature as well as exploratory analyses
suggested thatthe four majorfactorsin the explanationof empiricalType I error
rates were: sample size, multivariateskewness, multivariatekurtosis, and model
size. Additionally, different classes and subclasses of estimation methods are
likely to be relevantin the explanationof empiricalType I errorrates. Based on
these considerations,separaterandom-effectsregressionmodels were developed
for eachof the following six classes of estimators:(1) MaximumLikelihood(ML),
(2) GeneralizedLeast Squares (GLS), (3) Elliptical, (4) ADF, (5) DistributionFree (otherthanADF), and (6) LatentProjectionTechniques.
The independentvariableswere the samplesize (SAMP),multivariateskewness
(MS), multivariaterelative kurtosis (MRK), and model size as measuredby the
degrees of freedom for the model (DF). Each of the independentvariables were
enteredin both linearand quadraticforms.
Foreach of the six regressionmodels, standardizedresidualswere calculatedby
multiplyingthe rawresiduals(i.e., the actualobservedempiricalType I errorrates
minus the correspondingpredictedType I errorrates) by the squareroots of the
regressionweights (i.e., the l/v* terms). Standardizedresidualswith an absolute
value greaterthan3.0 were examined.
DescriptiveStatistics
Univariatedescriptivestatisticsare reportedfor the majorvariablesin the study
in Tables 2-6. The statisticsarereportedboth for the overalldataset as well as the
six groupsfor which separateregressionmodels were developed(ML, GLS, Elliptical, ADF, Distribution-Free(non-ADF),LatentProjection).
RegressionAnalyses
As was outlinedpreviously (see "AnalyticalScheme"), mixed-effects models
were systematically developed for each of the six different groups. Regression
summarytables are presentedin Tables 7, 9, 11, 13, 15, and 17. For each regression, the linearand quadratictermsof sample size, the linear and quadraticterms
of multivariateskewness, the linear and quadraticterms of multivariaterelative
TABLE2
statisticsfor empiricalTypeI errorrate
Univariate
descriptive
SD
Skew
M
Subclass
Kurt
N
Overall
ML
.137
.131
.201
.176
2.980
3.226
8.548
11.166
1888
528
GLS
Ellipt.
ADF
Non-ADF
Latent
.189
.068
.201
.118
.136
.258
.057
.241
.224
.170
2.084
2.696
2.078
3.012
2.838
3.155
10.890
3.432
8.016
8.284
278
482
281
172
125
Note. Non-ADF refersto Distribution-FreeEstimatorsotherthanADF.
116
Robustnessfor StructuralEquationModels
TABLE 3
Univariatedescriptivestatisticsfor sample size
Subclass
M
SD
Skew
Kurt
N
Overall
ML
GLS
Ellipt.
ADF
Non-ADF
Latent
668.62
517.14
806.84
571.58
875.98
981.98
531.60
1192.97
1033.64
1377.63
1134.54
1437.05
1377.41
394.65
4.911
6.419
4.560
5.604
4.113
2.147
1.447
28.473
47.940
23.856
35.983
19.418
3.450
2.568
1888
528
278
482
281
172
125
Note.Non-ADFrefersto Distribution-Free
Estimators
otherthanADF.
TABLE 4
Univariatedescriptivestatisticsfor multivariateskewness
Subclass
M
SD
Skew
Kurt
N
Overall
ML
ML(d)
GLS
Ellipt.
ADF
Non-ADF
Latent
10.55
8.19
7.22
11.35
6.16
12.57
20.65
15.07
19.82
25.21
11.53
14.68
8.42
17.69
23.60
26.78
10.767
16.486
16.828
2.280
2.834
2.446
1.565
2.974
240.146
330.124
3.584
6.179
9.120
6.255
1.563
8.836
1888
528
527
278
482
281
172
125
estimators
Note.Non-ADFrefersto distribution-free
otherthanADF.
skewness= 522 wasdeleted).
ML(d)refersto statisticsforMLwhenoutlier(multivariate
TABLE 5
Univariatedescriptivestatisticsfor MRK
Subclass
M
SD
Skew
Kurt
N
Overall
ML
GLS
Ellipt.
ADF
Non-ADF
Latent
1.17
1.10
1.22
1.14
1.25
1.24
1.31
.38
.35
.48
.22
.38
.31
.69
4.672
5.616
4.571
1.430
3.500
2.606
3.225
33.844
52.581
26.272
1.091
21.225
11.418
11.324
1888
528
278
482
281
172
125
Note.Non-ADFrefersto distribution-free
estimatorsotherthanADF;MRK,multivariate
relative
kurtosis.
117
Powell and Schafer
TABLE 6
Univariatedescriptivestatisticsfor degrees offreedom
Subclass
M
SD
Skew
Kurtosis
Overall
ML
GLS
Ellipt.
ADF
Non-ADF
Latent
28.38
15.42
33.95
21.24
32.96
62.78
29.86
31.29
22.58
32.88
24.26
30.76
35.90
29.11
1.102
2.301
.679
1.972
.802
-.787
1.198
-.411
4.470
-1.138
2.449
-.895
-1.338
.482
N
1888
528
278
482
281
172
125
Note.Non-ADFrefersto distribution-free
estimators
otherthanADF.
kurtosis,and the linearandquadratictermsof model size (degrees of freedom)are
reported.These tables reportresults for each independentvariableif it is entered
firstand for each independentvariableif it is enteredlast.
In each of Tables 7, 9, 11, 13, 15, and 17, the AQRis presentedfor each of the
variables.Additionallythe QEstatisticis presented.This statisticis a test of model
specificationand is distributedas a chi-square(Hedges & Olkin, 1985). The AR2
is also reportedfor each of the variables under each order of entry. The final
weighted least squaresregression equation for the mixed model is provided for
each groupin Tables 8, 10, 12, 14, 16, and 18. The equationprovides a predicted
value for empiricalType I errorrates as a linear function of eight predictorvariables (both linearand quadratictermsfor each of the four independentvariables).
TABLE7
Maineffects-MLestimators
MainEffect (Linear,Quadratic)
AQR
df
AR2
QE
df
2
2
.004
.011
812.46
630.39
525
519
SampleSize
Enteredfirst
Enteredlast
MultivariateSkewness
Enteredfirst
3.19*
8.98
142.60
2
.175
673.04
525
21.59
2
.026
630.39
519
Enteredfirst
84.34
2
.103
731.31
525
Enteredlast
26.95
2
.033
630.39
519
Degreesof Freedom
Enteredfirst
Enteredlast
76.42
21.96
2
2
.094
.027
739.23
630.39
525
519
Enteredlast
Multivariate
Kurtosis
Note.All QEandQRstatisticsaresignificant
(p < .05),exceptthosemarkedwithanasterisk.
118
Robustness for Structural Equation Models
TABLE 8
Regressioncoefficients-ML estimators
Variable
SE B
Beta
T
B
Sig T
MS2
-4.281E-06
2.714E-07
-.290143
-15.772
.0000
.003522
MS
1.309E-04
.505716
26.912
.0000
SAMP2
4.406E-09
3.198E-10
.216058
13.777
.0000
MRK
.318705
.009722
.633186
32.780
.0000
SAMP
-4.833E-05
2.724E-06
-.288341
-17.746
.0000
MRK2
-.065742
.002488
-.586805
-26.427
.0000
DF2
-2.184E-06
1.799E-06
-.024527
-1.213
.2251
DF
.001723
1.594E-04
.223839
10.810
.0000
-0.166038
.007482
-22.193
.0000
(Intercept)
Note.DF,degreesof freedom;DF2,degreesof freedom(squared);
skewness;MS2,
MS,multivariate
multivariate
skewness(squared);
relativekurtosis;MRK,multivariate
relative
MRK,multivariate
kurtosis(squared);
SAMP,samplesize;SAMP2,samplesize(squared).
AdditionallyFigures 1-6 (on pp. 124-129) provide a graphicalrepresentationof
the relationshipbetweenempiricalType I errorratesandtwo independentvariables:
multivariateskewness and multivariatekurtosis.
Summary and Conclusions
Maximumlikelihoodestimatorsshow very limited sensitivityto samplesize but
are sensitive to the non-normalityfactors(Table7). The effect of model size is also
importantfor ML estimators.Of the 528 effect sizes, 19 (3.6%) had standardized
TABLE9
Maineffects-GLSestimators
Main Effect (Linear,Quadratic)
AQR
df
AR2
QE
df
Sample Size
Enteredfirst
3.82*
2
.009
401.06
275
Enteredlast
0.55*
2
.001
206.75*
269
168.45
41.80
2
2
.416
.103
236.42*
206.75*
275
269
Enteredfirst
Enteredlast
97.85
23.94
2
2
.242
.059
307.01"
206.75*
275
269
Degreesof Freedom
Enteredfirst
Enteredlast
24.97
7.09
2
2
.061
.017
379.90
206.75*
275
269
MultivariateSkewness
Enteredfirst
Enteredlast
Multivariate
Kurtosis
Note.All QEandQRstatisticsaresignificant
(p < .05)exceptthosemarkedwithanasterisk.
119
Powell and Schafer
TABLE 10
Regressioncoefficients-GLS estimators
Variable
B
SE B
Beta
T
Sig T
MRK2
-26.361
.0000
-.119095
.004518
-1.056390
MRK
.0000
.592783
.024157
24.539
1.096171
2.556
.0106
SAMP2
1.59787E-09
6.2522E-10
.071934
-7.4859E-06
-.026053
-1.028
.3038
MS2
7.2795E-06
16.642
.0000
.520131
MS
.009159
5.5038E-04
-1.8715E-05
-3.448
.0006
SAMP
5.4282E-06
-.100117
.0000
DF2
-4.1944E-05
-11.971
3.5039E-06
-.494397
.0000
DF
13.863
.004643
3.3494E-04
.591559
.0000
-.485651
.021144
-22.969
(Intercept)
of
multivariate
Note.DF,degrees freedom;DF2,degreesof freedom(squared);
skewness;MS2,
MS,
relativekurtosis;MRK,multivariate
relative
multivariate
skewness(squared);
MRK,multivariate
kurtosis(squared);
SAMP,samplesize;SAMP2,samplesize (squared).
residualsin excess of 3.0 in absolutevalue. The largestof these was 6.26 and they
came fromthreeof the 19 studiesthatcontributedML estimators.These resultsdid
not cause the authorsto questionthe adequacyof the model.
The outcome for GLS estimatorsis similarto thatfor ML estimators(Table 9).
Sample size was not a majorfactor in accounting for variationamong empirical
Type I errorrates. Non-normalityis even more profound a factor for GLS estimators,due principallyto the differentialimpact of MS between the two types of
estimators.Residuals were examined for the 278 effect sizes involving GLS esti-
TABLE11
Main effects--elliptical estimators
MainEffect(Linear,Quadratic)
AQR
df
AR2
QE
df
Sample Size
Enteredfirst
Enteredlast
18.63
9.99
2
2
.018
.010
1,004.22
801.32
479
473
MultivariateSkewness
Enteredfirst
Enteredlast
158.95
130.91
2
2
.155
.128
863.90
801.32
479
473
MultivariateKurtosis
Enteredfirst
Enteredlast
53.08
24.85
2
2
.052
.024
969.77
801.32
479
473
Degrees of Freedom
Enteredfirst
Enteredlast
14.37
3.63*
2
2
.014
.004
1,008.48
801.32
479
473
Note.All QRandQEstatisticsaresignificant
(p < .05),exceptthosemarkedwithanasterisk.
120
Robustness for Structural Equation Models
TABLE12
Regression coefficients-elliptical estimators
B
SEB
Beta
T
SigT
MRK2
MRK
DF
SAMP2
.128316
-.368699
-1.7784E-04
1.2445E-09
.001317
.003528
1.4071E-05
2.0611E-11
1.573839
-1.723644
-.094707
.274715
97.439
-104.521
-12.639
60.382
.0000
.0000
.0000
.0000
SAMP
-1.2910E-05
Variable
DF2
MS2
MS
(Intercept)
1.8107E-07
-.352948
-71.297
.0000
3.5537E-06
1.2959E-04
-.003037
1.5103E-07
6.2165E-07
2.7252E-05
.175648
.767163
-.496353
23.531
208.453
-111.458
.0000
.0000
.0000
.316026
.002260
139.833
.0000
Note. DF, degrees of freedom;DF2, degrees of freedom (squared);MS, multivariateskewness; MS2,
multivariateskewness (squared);MRK, multivariaterelative kurtosis;MRK, multivariaterelative
kurtosis (squared);SAMP, sample size; SAMP2, sample size (squared).
mators. Of the 278 standardizedresiduals, four (1.4%) were in excess of 3.0 in
absolute value. The maximum was 3.78 and they came from two of the 12 studies that contributed GLS estimators. As in the case of the ML estimators, the
examinationof residualsdid not cause the authorsto questionthe adequacyof the
model.
Like the normaltheoryestimators,ellipticalestimatorsshowedlimitedsensitivity
to sample size (Table 11). Non-normalityplayeda role in the explanationof empirical Type I errorrates, but model size had only a small role in the explanation
TABLE13
Maineffects--ADFestimators
MainEffect(Linear,Quadratic)
AQR
df
AR2
QE
df
Enteredfirst
Enteredlast
123.11
133.32
2
2
.103
.111
1,071.55
439.14
278
272
Multivariate
Skewness
Enteredfirst
SampleSize
295.16
2
.247
899.29
278
Enteredlast
70.38
2
.059
439.14
272
Multivariate
Kurtosis
Enteredfirst
Enteredlast
3.49*
26.68
2
2
.003
.022
1,190.97
439.14
278
272
Degreesof Freedom
Enteredfirst
375.22
2
.314
819.24
278
Enteredlast
222.85
2
.186
439.14
272
Note. All QEand QRstatisticsare significant(p < .05), except those markedwith an asterisk.
121
Powell and Schafer
TABLE 14
Regression coefficients--ADF estimators
B
Variable
SE B
Beta
T
Sig T
DF2
45.817
7.63630E-05
1.6667E-06
.0000
.904539
DF
-18.575
.0000
-.002993
1.6111E-04
-.381575
SAMP2
66.982
.0000
1.78279E-08
2.6616E-10
.882354
-7.7035E-05
MS2
2.5072E-06
-.414335
-30.726
.0000
SAMP
-1.7676E-04
2.2964E-06
-1.066363
-76.971
.0000
.010427
2.2120E-04
47.139
.0000
MS
.766011
.045580
.002470
MRK2
18.453
.0000
.284721
MRK
-.307213
.011254
-.476517
-27.298
.0000
.458724
46.883
.0000
.009784
(Intercept)
Note.DF,degreesof freedom;DF2,degreesof freedom(squared);
skewness;MS2,
MS,multivariate
multivariate
skewness(squared);
relative
relativekurtosis;MRK,multivariate
MRK,multivariate
kurtosis(squared);
SAMP,samplesize;SAMP2,samplesize(squared).
of empiricalType I errorratesfor the elliptical estimators.The general weakness
of model size as a predictorin the elliptical case is likely to be due to the limited
numberof studies contributingcases for the elliptical estimators.Three-quarters
of the cases came from one study, while 82% of the model sizes were either 5, 8,
or 14. The effects of model size on empiricalType I errorrates for elliptical estimatorsdeserves more extensive study in future research.Standardizedresiduals
were examined for the regression model for the elliptical estimators.Of the 482
effect sizes, 15 (3.1%)had standardizedresidualsin excess of 3.0 in absolutevalue.
TABLE15
Main effects-distribution-free (other thanADF) estimators
Main Effect (Linear,Quadratic)
AQR
df
AR2
QE
df
Sample Size
Enteredfirst
Enteredlast
6.02
3.68*
2
2
.051
.031
111.31*
100.78*
169
163
MultivariateSkewness
Enteredfirst
Enteredlast
7.43
2.52*
2
2
.063
.021
109.89*
100.78*
169
163
0.80*
0.41"
2
2
.007
.003
116.52*
100.78*
169
163
5.49*
3.42*
2
2
.047
.029
111.84*
100.78*
169
163
MultivariateKurtosis
Enteredfirst
Enteredlast
Degrees of Freedom
Enteredfirst
Enteredlast
Note.All QEandQRstatisticsaresignificant
(p < .05),exceptthosemarkedwithanasterisk.
122
Robustnessfor StructuralEquationModels
TABLE 16
Regression coefficients-distribution-free (other than ADF) estimators
Variable
B
SE B
Beta
T
Sig T
DF2
1.5833E-05
-3.9589E-05
-.622040
-2.500
.0125
DF
.005504
.001556
.882447
3.537
.0004
SAMP2
2.2497E-08
3.0357E-09
7.411
.697805
.0000
MS2
1.0801E-05
5.5465E-05
.477077
5.135
.0000
MS
-.003452
.001027
-.364043
-3.360
.0008
SAMP
-1.5027E-04
1.7494E-05
-.925361
.0000
-8.589
MRK2
-.023470
.020328
-.107280
-1.155
.2484
MRK
.016626
.070656
.8140
.022653
.235
.097383
.056462
.0847
1.725
(Intercept)
Note.DF,degreesof freedom;DF2,degreesof freedom(squared);
skewness;MS2,
MS,multivariate
multivariate
skewness(squared);
relativekurtosis;MRK,multivariate
relative
MRK,multivariate
kurtosis(squared);
SAMP,samplesize;SAMP2,samplesize (squared).
These 15 were contributedby three of the five studies that includedellipticalestimators. The maximum of these residuals was 7.54. The results did not lead the
authorsto question the adequacyof the regressionmodel.
The ADF estimators had the strongest and most easily interpretedregression
model in the entire meta-analysis (Table 13). The estimatorsappearsensitive to
sample size, non-normality,and model size. The findings suggest that the ADF
estimatoris affected when sample size is small or when model size is large. The
user shouldapproachthe employmentof ADF skepticallyin these situationsunless
TABLE17
Maineffects-latentprojectionestimators
Main Effect (Linear,Quadratic)
AQR
df
AR2
QE
df
Sample Size
Enteredfirst
Enteredlast
24.51
35.37
2
2
.051
.074
453.29
214.59
122
116
MultivariateSkewness
Enteredfirst
Enteredlast
80.76
5.74*
2
2
.169
.012
397.03
214.59
122
116
Multivariate
Kurtosis
Enteredfirst
Enteredlast
6.28
3.32*
2
2
.013
.006
471.52
214.59
122
116
Degrees of Freedom
Enteredfirst
213.30
2
.446
264.50
122
Enteredlast
123.75
2
.259
214.59
116
Note. All QEand QRstatisticsare significant(p < .05), except those markedwith an asterisk.
123
TABLE 18
Regression coefficients-latent projectionestimators
Variable
MRK2
MRK
SAMP2
MS2
SAMP
MS
DF
DF2
(Intercept)
B
SE B
Beta
T
Sig T
-.017735
.086306
-4.1151E-08
7.0843E-06
-5.9196E-05
3.8624E-05
.003974
2.2789E-06
-.025633
.001077
.006373
3.444E-09
1.163E-06
6.110E-06
1.608E-04
1.175E-04
1.063E-06
.006339
-.395921
.389099
-.169744
.137699
-.145559
.006578
.713739
.040445
-16.463
13.542
-11.949
6.088
-9.689
.240
33.835
2.145
-4.044
.0000
.0000
.0000
.0000
.0000
.8102
.0000
.0320
.0001
Note: DF, degrees of freedom;DF2, degrees of freedom(squared);MS, multivariateskewness; MS2,
multivariateskewness (squared);MRK, multivariaterelative kurtosis;MRK, multivariaterelative
kurtosis(squared);SAMP, sample size; SAMP2, sample size (squared).
O
FIGURE 1. ML estimators-Type I error rate as a function of multivariate skewness
and multivariaterelative kurtosis.
124
Robustnessfor StructuralEquationModels
Cq)
tob
q)
Qc5
FIGURE2. GLSEstimators-TypeI errorrateas a functionof multivariateskewness
and multivariaterelative kurtosis.
furtherresearchcontradictsthese results. As in the case of the other regression
models, an examinationof standardizedresidualsdid not lead the authorsto question the adequacyof the regressionmodel. Of the 281 effect sizes, nine (3.2%) had
standardizedresidualsin excess of 3.0 in absolutevalue. The maximumwas 4.10
and these nine came from five of the 16 studiesthatcontributedADF estimatorsto
the meta-analysis.
The remainingdistribution-freeestimators(Table 15) representa disparatecollection of procedures,some of which are in the experimentalstage. None of the
estimatorsareheavily representedin this study,but as a groupthey show substantially less sensitivity to non-normalitythan any of the other groups of estimators.
Taken as a group,they are substantiallymore robustthanADF. Of the 172 effect
sizes, only one had a standardizedresidualin excess of 3.0 in absolutevalue. This
standardizedresidualwas equal to 3.02 and these results did not lead the authors
to questionthe adequacyof the regressionmodel.
125
Powell and Schafer
g03
c,
P>c5
9'
'O5
FIGURE3. Ellipticalestimators-TypeI errorrateas afunctionof multivariaterelativekurtosis.
The latentprojectiontechniquesshow sensitivitywith respectto sample size and
to model size (Table 17). The studycontainedonly 125 effect sizes involving latent
projectiontechniques.While the findingsof this study do not suggest these techniques are sensitive to MS or MRK, the relatively sparse numberof effect sizes
precludesany strongconclusions. These proceduresrequirefurtherstudy. Of the
125 effect sizes involving latent projectionmethods, seven (5.6%) had standardized residuals in excess of 3.0 in absolute value. The most extreme of the standardizedresiduals had a value of -5.85, and they came from three of the seven
studiesthatcontributedlatentprojectionestimatorsto the meta-analysis.As in the
case of the otherestimationmethods,the adequacyof the regression models was
not called into questionby the resultsof this analysis.
None of these estimation methods for SEM can be endorsed without considerable qualification. Both the ML and GLS methods depend on having data that
126
Robustness for Structural Equation Models
Q)
QL
-t3
QL
•oo
FIGURE 4. ADF estimators-Type I error rate as a function of multivariateskewness
and multivariaterelative kurtosis.
do not deviate to any great degree from multivariate normality. This result
is consistent with theirassumptions.Thereis no evidence from this study to suggest that the elliptical estimators offer substantial improvement. The elliptical
estimators may be less sensitive to the effects of larger models, but this may
be due to the relatively smaller range of model sizes represented in the metaanalysis.
The resultsdo not discouragethe use of latentprojectiontechniquesas an alternative to normaltheorymethodswhen the data are eithercategoricalor censored.
The ADF methodis not recommended.Othertechniquesthatare distribution-free
may prove to be useful in futureresearch.However, any conclusions with respect
to these methodsis prematureat the presenttime because they arenot heavily represented in the meta-analysis.Also, the estimation methods in this class include
a variegated set of procedures.Furtherresearch should be undertakento make
127
Powell and Schafer
00
QLO
a>co
co
q)
e-
FIGURE5. Distribution-freeestimators other thanADF TypeI error rate as afunction
of multivariateskewness and multivariate relative kurtosis.
128
Robustnessfor StructuralEquationModels
Q)
ilt
FIGURE6. LatentprojectionestimatorsTypeI errorrateas afunctionof multivariate
skewnessandmultivariate
relativekurtosis.
comparisonsamongthese methodson the basis of theirrelativesensitivitiesto the
factors of sample size, non-normality,and model size.
In short,the currentstateof robustnessdoes not allow us to recommendthe use
of SEM when dataarenon-normal.The estimatorsthathave not been shown to be
sensitive to non-normalityarenot sufficientlywell reportedin the studyto recommend their use. When data do approximatemultivariatenormality,there is no
strongreason to preferany of the techniquesover the normaltheoryestimators.
129
Powell and Schafer
References
References marked with an asterisk indicate studies included in the meta-analysis.
*Babakus,E. (1985). Thesensitivityof maximumlikelihoodfactoranalysis given violations
of intervalscale and multivariatenormalityassumptions.Unpublisheddoctoraldissertation, Universityof Alabama.
*Benson,J., & Fleishman,J. (1994). The robustnessof maximumlikelihoodanddistributionfree estimatorsto non-normalityin confirmatoryfactoranalysis.Qualityand Quantity,28,
117-136.
Bollen, K. A. (1989). Structuralequationswith latent variables. New York:Wiley.
*Boomsma,A. (1983). On the robustnessofLISRELmaximumlikelihoodestimationagainst
small sample size and non-normality.Unpublisheddoctoral dissertation,University of
Groningen,Groningen,The Netherlands.
*Brown,R. L. (1989a). Using covariancemodeling for estimatingreliabilityon scales with
orderedpolytomousvariables.Educationaland PsychologicalMeasurement,49, 385-398.
*Brown, R. L. (1989b). Congeneric modeling of reliability using censored variables.
AppliedPsychological Measurement,13(2), 151-159.
*Browne, M. W. (1984). Asymptotically distribution-freemethods for the analysis of
covariance structures.British Journal of Mathematicaland Statistical Psychology, 37,
62-83.
*Chou,C. C., & Bentler,P. M. (1990). Model modificationin covariancestructuremodeling:
A comparisonamong likelihood ratio,Lagrangemultiplier,andWald tests. Multivariate
BehavioralResearch, 25(1), 115-136.
*Chou,C. C., Bentler,P. M., & Satorra,A. (1991). Scaled test statisticsandrobuststandard
errorsfor non-normaldatain covariancestructureanalysis:A Monte Carlostudy.British
Journal of Mathematicaland StatisticalPsychology, 44, 347-357.
*Curran,P. J., West, S. G., & Finch, J. (1995). The robustness of test statistics to nonnormalityand specification error in confirmatoryfactor analysis. Unpublishedmanuscript, University of California, Los Angeles.
*Dolan, C. V. (1994). Factoranalysis of variableswith 2, 3, 5 and 7 responsecategories:A
comparisonof categorical variable estimatorsusing simulated data. British Journal of
Mathematicaland Statistical Psychology, 47, 309-326.
Fuller,E. L., & Hemmerle,W. J. (1966). Robustnessof the maximumlikelihood procedure
in factoranalysis. Psychometrika,31, 252-266.
*Harlow,L. L. (1985). Behavior of some elliptical theoryestimatorswith non-normaldata
in a covariance structureframework:A Monte Carlo study. Unpublisheddoctoraldissertation,Universityof California,Los Angeles.
*Harlow,L. L., Chou, C. P., & Bentler,P. M. (June, 1986). Performanceof the chi-square
statisticwithML,ADF, and ellipticalestimatorsforcovariancestructures.Paperpresented
at the PsychometricSociety annualmeeting,Toronto.
Harwell,M. C. (1992). SummarizingMonteCarloresultsin methodologicalresearch.Journal of EducationalStatistics, 17(4), 297-293.
Harwell,M. C., Rubinstein,E. N., Hayes, W. S., & Olds, C. C. (1992). SummarizingMonte
Carloresultsin methodologicalresearch:The one- and two-factorfixed effects ANOVA
cases. Journal of EducationalStatistics, 17(4), 315-339.
Hedges, L., & Olkin, I. (1985). Statisticalmethodsformeta-analysis.San Diego: Academic
Press.
130
Robustnessfor StructuralEquationModels
Hedges,L., & Vevea, J. (1998). Fixed- andrandom-effectsmodels in meta-analysis.Psychological Methods,3(4), 486-504.
*Henly,S. J. (1991). Robustestimationwithnon-normaldata andfinite samplesin theanalysis ofcovariance structures.Unpublisheddoctoraldissertation,Universityof Minnesota.
*Hu, L., Bentler,P. M., & Kano,Y. (1992). Cantest statisticsin covariancestructureanalysis be trusted?Psychological Bulletin, 112(2), 351-362.
JOreskog,K. G. (1973). A general method for estimating a linear structuralequation system. In A. S. Goldbergerand 0. D. Duncan (Eds.), Structuralequation models in the
social sciences. New York: SeminarPress.
*Kaplan,D. (1991). The behaviourof three weighted least squaresestimatorsfor a structuredmeans analysis with non-normalLikertvariables.BritishJournal of Mathematical
and Statistical Psychology, 44, 333-346.
Keselman,J. C., Lix, L. M., & Keselman,H. J. (April, 1993). Theanalysis of repeatedmeasurements:A quantitativeresearch synthesis. Paperpresentedat the AnnualMeeting of
the AmericanEducationalResearchAssociation, Atlanta.
Mardia,K. V. (1970). Measures of multivariateskewness and kurtosiswith applications.
Biometrika,57, 519-530.
Muthen, B. (1984). A general structuralequationmodel with dichotomous,orderedcategorical, and continuouslatentvariableindicators.Psychometrika,49, 115-132.
Muthen, B. (July, 1985). TOBITfactor analysis. Paperpresentedat the FourthEuropean
Meeting of the PsychometricSociety, Cambridge,England.
Muthen,B. (1987). LISCOMP:Analysis of linear structuralequations with a comprehensive measurementmodel. Mooresville, IN: Scientific Software,Inc.
*Muthen,B. (1989). Multiplegroupstructuralmodelling with non-normalcontinuousvariables. BritishJournal of Mathematicaland StatisticalPsychology, 42, 55-62.
*Muthen, B., & Kaplan, D. (1985). A comparisonof some methodologies for the factor
analysis of non-normalLikertvariables.BritishJournal of Mathematicaland Statistical
Psychology, 83, 171-189.
*Muthen, B., & Kaplan, D. (1992). A comparisonof some methodologies for the factor
analysis of non-normalLikertvariables:A note on the size of the model. BritishJournal
of Mathematicaland Statistical Psychology, 45, 19-30.
*Potthast, M. J. (1991). A simulation of categorical variable methodology in confirmatoryfactor analysis. Unpublished doctoral dissertation,University of Maryland,College Park.
*Salomaa,H. (1990). Factor analysis of dichotomousdata. Unpublisheddoctoraldissertation, University of Turku,Helsinki, Finland.
*Satorra,A., & Bentler, P. M. (1988). Scaling correctionsfor statisticsin covariancestructureanalysis. UCLAStatisticalSeries, ReportNo. 2. Los Angeles: Universityof California, Departmentof Psychology.
*Satorra,A., & Bentler, P. M. (1990). Model conditions for asymptoticrobustnessin the
analysis of linear relations.ComputationalStatisticsand Data Analysis, 10, 235-249.
*Silvia, E. S. M. (1988). Effects of samplingerrorand model misspecificationon goodness
offit indicesfor structuralequationmodels.Unpublisheddoctoraldissertation,The Ohio
State University.
Vale, C. D., & Maurelli, V. A. (1983). Simulatingmultivariatenon-normaldistributions.
Psychometrika,48, 465-471.
*Waller, N. G., & Muthen, B. (1992). Genetic tobit factor analysis: Quantitativegenetic
modeling with censored data.Behavior Genetics, 22(3), 265-292.
131
Powell and Schafer
*Wendler,C. L. W. (1993). Factor analysis of dichotomousitems:a comparisonof tetrachoric andphi. Unpublisheddoctoraldissertation,Universityof California,Los Angeles.
ADF statisticsin covariancestruc*Yung, K., & Bentler,P. M. (1994). Bootstrap-corrected
tureanalysis. BritishJournal of Mathematicaland StatisticalPsychology,47, 63-84.
*Yuan, K., & Bentler,P. M. (1995). Mean and covariancestructureanalysis: Theoretical
andpracticalimprovements.UCLAStatisticalSeries, ReportNo. 180. Los Angeles: University of California,Departmentof Psychology.
Authors
DOUGLAS A. POWELLis a ConsultingStatisticianfor DataManagementServices of the
NationalCancerInstitute,FrederickCancerResearchandDevelopmentCenter,Frederick,
Maryland; [email protected]. His research specialties are meta-analysis, applied
statistics, andresearchdesign.
WILLIAMD. SCHAFERis Affiliated ProfessorEmeritusin the Departmentof Measurement Statistics and Evaluation, University of Maryland, College Park, Maryland;
[email protected] researchspecialtiesare appliedstatisticsand accountabilityin
eduction.
132