STRUCTURAL EQUATION MODELING, 14(1), 26–47
Copyright © 2007, Lawrence Erlbaum Associates, Inc.
Performance of Factor Mixture Models
as a Function of Model Size, Covariate
Effects, and Class-Specific Parameters
Gitta Lubke
University of Notre Dame
Bengt O. Muthén
University of California, Los Angeles
Factor mixture models are designed for the analysis of multivariate data obtained
from a population consisting of distinct latent classes. A common factor model is assumed to hold within each of the latent classes. Factor mixture modeling involves obtaining estimates of the model parameters, and may also be used to assign subjects to
their most likely latent class. This simulation study investigates aspects of model performance such as parameter coverage and correct class membership assignment and
focuses on covariate effects, model size, and class-specific versus class-invariant parameters. When fitting true models, parameter coverage is good for most parameters
even for the smallest class separation investigated in this study (0.5 SD between 2
classes). The same holds for convergence rates. Correct class assignment is unsatisfactory for the small class separation without covariates, but improves dramatically
with increasing separation, covariate effects, or both. Model performance is not influenced by the differences in model size investigated here. Class-specific parameters may improve some aspects of model performance but negatively affect other
aspects.
Factor mixture models combine latent class analysis and common factor analysis.
Factor mixture models are designed for data from possibly heterogenous populations consisting of several latent classes, and are an adequate choice if it is reasonable to assume that observed variables within each class can be modeled using a
common factor model. There are two types of latent variables in factor mixture
Correspondence should be addressed to Gitta Lubke, Department of Psychology, 118 Haggar Hall,
University of Notre Dame, Notre Dame, IN 46556. E-mail: [email protected]
PERFORMANCE OF FACTOR MIXTURE MODELS
27
models, the categorical latent class variable indicating the class membership of
each subject, and the continuous latent factor(s) representing the common content
of the observed variables. It is not observed to which class a subject belongs, hence
class membership is latent, and it is not observed how a subject scores on the factors underlying the observed variables, hence factors are latent.
Factor mixture modeling can be used to assign subjects to their most likely class
and to obtain estimates of the model parameters. Subject assignment can be the
primary objective of a study, for instance when the interest is to establish subpopulations representing different types of risk behavior. Detection of different
risk behavior classes can serve to investigate the effectiveness of a specific prevention program for members of a high-risk class. On the other hand, a study may aim
mainly at the comparison of classes with respect to model parameters such as factor means in growth mixture models, or class-specific effects of covariates. Factor
mixture models are used increasingly in empirical research, both for class assignment (e.g., Neuman et al., 1999; Rasmussen et al., 2002) and class comparisons
(e.g., Duncan, Duncan, & Stryker, 2000; McArdle & Epstein, 1987; B. O. Muthén
& Muthén, 2000; Nagin & Tremblay, 2001). The primary goal of this study is to assess the performance of these two aspects of factor mixture modeling and establish
the conditions under which applied researchers can expect factor mixture models
to perform properly. It is investigated to what extent covariate effects, model size,
or presence of class-specific model parameters such as class-specific factor loadings can influence model performance. Because separation between classes is
known to affect model performance, the study is carried out at different levels of
separation. Effects of class separation on model performance that are due to
covariate effects are distinguished from class separation effects that are due to differences between classes regarding factor means or intercepts of observed variables within class.
A large body of research has focused on establishing the correct number of
classes and on developing or assessing indexes that may serve to compare alternative
models (Bamber & Santen, 2000; Jedidi, Ramaswarmy, DeSarbo, & Wedel, 1996;
Nagin, 1999; see also Everitt & Hand, 1981; McLachlan & Peel, 2000, for in-depth
discussions, and Lubke & Neale, 2005, for some simulation results). In this study the
focus is not on finding the most adequate model for a given data set; rather, the performance of factor mixture models when true models are fitted to artificial data is considered. The objective is to quantify the proportion of subjects that are correctly assigned to their true class, and the coverage of true parameter values as a function of
increasing covariate effects and class separation. Because true models are fitted,
these proportions and average rates provide an upper bound of what might be expected in an empirical study in which fitted models are simplifications of the true
data generating process. The quantification of model performance is done for different factor mixture models under varying conditions. Because models are fitted to artificial data, deviations from true values can be computed.
28
LUBKE AND MUTHÉN
The data generating models are the latent profile model (Lazarsfeld & Henry,
1968)1 and factor mixture models with one, two, or three within-class factors and 8
or 12 observed variables. The models differ with respect to their size and complexity. The within-class structure of the latent profile model is relatively simple in that
observed variables within class are assumed to be independent conditional on class
membership (i.e., local independence), hence all covariances within class are zero.
The one-, two-, and three-factor models require the estimation of an increasingly
complex latent structure within class.
Class separation is measured in terms of the multivariate Mahalanobis distance (MD) between two classes. Although it is evident that increasing class separation will improve correct class assignment, it is not known how far classes
have to be separated to assign subjects to their true class with an acceptably high
probability, or to recover true model parameters such as factor mean differences
between classes. This study includes the evaluation of model performance for
smaller class separation than has been investigated so far in the context of factor
mixture analysis. Increasing effects of a continuous covariate that predicts class
membership are investigated. Class-predicting covariates increase class separation when considering a multivariate distance computed for covariates and observed within-class variables. To distinguish between class separation due to
covariate effects and class separation due to factor mean or intercept differences,
three multivariate MDs are computed, namely (a) using only the means and
covariances of the observed variables within class, (b) using the observed variables and the covariates jointly, or (c) using only the covariates. Larger effects of
covariates on class membership result in a larger MD when computed for observed variables and covariates jointly.2
Class-invariant versus specific factor covariances, and measurement invariance
versus noninvariance concern the restrictiveness of the within-class model. More
restrictive models require fewer parameters to be estimated, which may ease
model estimation. However, the less restrictive models have more class-specific
parameters, which may be advantageous to distinguish between classes.
The focus of this simulation study is showing what an empirical researcher can
expect regarding correct class assignment and parameter coverage under a variety
of conditions. Correct class assignment is assessed as the proportion of subjects assigned to the true class based on the highest posterior class probability (i.e., modal
assignment). Coverage of true parameter values is computed for a given number of
replications as the proportion of replications for which the 95% confidence interval of the estimated parameter covers the true parameter. We also evaluate whether
entropy (see, e.g., Muthén & Muthén, 2001) is a suitable indicator of correct class
1The latent profile model can be considered as a factor mixture model with zero factors, or with factors with zero variance.
2The MD between two classes that is used in this study equals M = (␮ - ␮ )t -1(␮ - ␮ ) .
1
2 ⌺
1
2
PERFORMANCE OF FACTOR MIXTURE MODELS
29
assignment. Entropy is closely related to average class probabilities, which may
provide an indication of how problematic or unproblematic class assignment is in a
given analysis. Intuitively, it seems that a researcher can be more confident about
the results of, say, a two-class model, if subjects are assigned to either one of the
classes with a probability close to unity than if class probabilities are close to .5.
Finally, convergence rates are presented for all models.
MODEL SPECIFICATION
Slightly different specifications of factor mixture models have been proposed in
the literature (Arminger, Stein, & Wittenberg, 1999; Dolan & van der Maas, 1998;
Jedidi, Jagpal, & DeSarbo, 1997; McLachlan & Peel, 2000; Vermunt & Magidson,
2002; Yung, 1997). Here, we largely follow the model description in Muthén and
Shedden (1999; for a less technical account, see Lubke & Muthén, 2005). Considered are factor mixture models with a single categorical latent variable and continuous outcomes within class.3
General Model Description
The factor mixture model consists of several different regressions. First, observed
continuous variables within class are regressed on the within-class factors and on
covariates. Second, the within-class factors are regressed on the latent class variable and on covariates. Third, the class variable is regressed on covariates. Class
membership can also function as a predictor of observed outcome variables, but
this option is not considered in this article.
Regression of observed variables on factors and covariates. Throughout, subscript i, where i = 1, … , I, is used as a subject index. Following commonly
used notation we denote the J = 1, … J observed continuous variables as Y, and the
scores of subject i on l = 1, … L underlying continuous factors as ␩il. Factors are
assumed to be multivariate normally distributed conditional on class membership
and covariates. The J × 1 vector ␯ contains the regression intercepts, ⌳y is the J × L
matrix of factor loadings, and εi are the errors of subject i. Errors are assumed to be
normally distributed with zero mean, and are assumed to have zero covariances
with factors or other errors. Covariates can be continuous or categorical, and are
3The factor mixture model that can be estimated in the current version of Mplus (L. K. Muthén &
Muthén, 2004) is more general in that it allows more class-specific parameters than the model described
in the next section, mixed categorical or continuous observed variables within class, and more than a
single latent class variable.
30
LUBKE AND MUTHÉN
denoted as X. The regression weights of Y on X are contained in ⌫y. The linear regression of Y on ␩ and X is expressed as
y i = ␯ + ⌳ y ␩i + ⌫ yx i + ε i.
(1)
Regression of factors on class and covariates. The latent class variable
is a multinomial variable. For k = 1, … , K latent classes, we have
ìï1 if subject i belongs to class k
cik = ïí
ïïî0
otherwise.
(2)
The factors are regressed on the class variable and on covariates X, which can
be expressed for subject i as
ηi = Ac i + ⌫ ηx i + ␨ i.
(3)
The residual factor scores of subject i, that is, the part of the factor that is not explained by class membership or by the covariates, is denoted as ␨i. These are
multivariate normally distributed with zero means within each class. ⌫η contains
the regression weights of the factors on the covariates. A has dimensions L × K and
contains possibly class-specific regression intercepts. If regression intercepts ␯ in
Equation 1 are estimated, one column of A has to be fixed, which is equivalent to
the restriction in multigroup analysis where the factor means of one of the groups
have to be fixed in case intercepts are estimated (Sörbom, 1974). Taking expectations, one can see that conditional on X, the other columns of A contain the factor
mean differences with respect to the arbitrarily chosen reference class.
Regression of the latent class variable on covariates. The third part is a
multinomial logistic regression where X predicts the log odds of the probability of
belonging to class k as compared to the probability of belonging to an arbitrarily
chosen reference class, say the Kth class. The third part of the model is denoted as
é P (cik = 1 | x i ) ù
ú = λ ck + ⌫ ck x i .
ln ê
êë P(ciK = 1 | x i ) úû
(4)
⌫ck contains the regression weights, and the regression intercept is denoted as λck.
Conditional on class membership and covariates X, observed variables Y are
multivariate normally distributed. Class membership is considered missing, and an
Expectation-Maximization (EM)-type algorithm can be used to estimate the
model conditional on covariates X. In the E step, the posterior probabilities of the
subjects of belonging to each of the K classes is computed given the parameter estimates. In the M step, estimates of the model parameters are updated given the pos-
PERFORMANCE OF FACTOR MIXTURE MODELS
31
terior class probabilities of the subjects. Approaches to estimate mixture models
are described in McLachlan and Peel (2000).
This general model encompasses several well-known models as submodels.
Fixing within-class factor variances to zero results in local independence of observed variables within class, which is the main characteristic of classic latent class
models. Fixing the number of classes to 1 results in common factor models. The
submodels used in this study are described in more detail in the next section.
SIMULATION STUDY DESIGN
The study has three parts. In Part 1, the effects of model size and model complexity
are investigated. Part 2 focuses on covariate effects and class separation, and Part 3
aims at evaluating the effects of different class-specific parameters. Throughout
the simulation, two-class models are considered, K = 2. There are no direct effects
of covariates on the observed variables or covariate effects on factor scores, consequently the matrices ⌫y and ⌫η contain zeros. The covariate effects investigated in
Part 2 concern the prediction of class membership. All parameter values used for
the data generation are shown in the Appendix.
Part 1: Model Size and Complexity
Four models are compared in Part 1, namely the latent profile model, and one-,
two-, and three-factor models. The latent profile model can be represented as a special case of the factor mixture model where residual factor scores ␨ (see Equation
3) have zero variance. As a result, the covariance matrix of observed variables Y
conditional on class membership equals the covariance matrix of the residuals ε
(see Equation 1). Because residuals are assumed to have zero covariances (discussed earlier), the conditional covariance matrix of observed variables is diagonal, and observed variables are independent given class. The latent profile model
used for data generation has eight observed variables. The means of the observed
variables are class specific, and are used to manipulate class separation. Residual
variances are class invariant.
The single-, two-, and three-factor models in this simulation differ with respect
to the number of observed variables. The single-factor and two-factor models have
8 observed indicators, whereas the three-factor model has 12 indicators. All factor
models in Part 1 are measurement invariant; that is, intercepts, loadings, and residual variances are class invariant (Lubke, Dolan, Kelderman, & Mellenbergh, 2003;
Meredith, 1993). In both classes, factor variances and covariances (if present)
equal unity and 0.5, respectively, which results in class-invariant covariance matrices of observed variables Y. The analysis of normal mixtures with class-invariant
covariance matrices may be less prone to problems related to multiple singularities
32
LUBKE AND MUTHÉN
of the likelihood. It is known that the likelihood surface of a normal mixture with
class-specific covariance matrices may have many singularities (McLachlan &
Peel, 2000). This can affect convergence rates and increases the risk of incorrect
solutions corresponding to a local instead of the global maximum. Model performance in Part 1 is compared to the analysis data with class-specific covariance matrices covered in Part 3.
Data are generated under these four models for two levels of class separation.
The MD when computed using the model implied within class mean vectors and
covariance matrices equals 1 and 1.5, respectively (MD is explained in more detail
in the next section). None of the models in Part 1 has covariate effects.
Part 2: Mahalanobis Distance, Class Separation,
and Covariate Effects
In Part 2, class separation and covariate effects are investigated in more detail. The
separation of the two classes is measured by the multivariate MD. The MD between two points, say P1 with coordinates (a1, b1) and P2 with coordinates (a2, b2),
where A and B are uncorrelated and have unit variance, is equal to the squared standardized Euclidian distance between the two points. The MD is a distance that not
only takes variances but also covariances between variables into account. Everything else held constant, the MD between two points decreases with increasing
covariation of the variables involved. Because the MD takes (co)variance(s) into
account, factor mean differences, intercept differences, and also the reliability of
observed variables can affect this distance. Figure 1 shows the relation among the
standardized factor mean difference, reliability, and the MD for a single-factor
model with five indicators. Compared to standardized factor mean differences,
commonly observed reliability variation (i.e., between 0.4 and 0.8) has only a limited impact on the MD. The MD is largely determined by the standardized factor
mean differences between the classes.
Note that an MD can be computed using model-implied means and covariances.
The MD when based on model-implied means and covariances is denoted as MDi,
which does not vary across MC replications for a given model, and therefore characterizes different types of data sets. An MD can also be computed using observed
means and covariances. This distance is denoted as MD0. Contrary to MDi, MD0
varies across data sets that are generated under a given model due to random variation of factor scores and residual scores. Hence, MD0 characterizes a single data
set, and can be used to predict aspects of model performance that vary across MC
replications for a given model (e.g., correct class assignment). Furthermore, MDs
can be computed (a) only for the observed within-class variables Y, (b) only for the
covariates X, or (c) for Y and X jointly.
The data-generating model in Part 2 is a single-factor/two-class model with
eight observed variables and four levels of intercept differences between classes.
PERFORMANCE OF FACTOR MIXTURE MODELS
FIGURE 1
33
Relation between standardized factor mean difference, reliability, and MD.
The MDi is computed for the observed variables Y are 0.5, 1.0, 1.5, and 2.0, respectively. A single normally distributed covariate predicts class membership. The four
levels of intercept differences are combined with four effect sizes of the covariate,
leading to 16 different types of data. The effect sizes of the covariate measured in
terms of MDi computed for the covariate alone are 0, .5, 1, and 1.5, respectively.
Part 3: Class-Specific Parameters
Part 3 focuses on class-specific parameters. Parameter coverage of class-specific
parameters is an important aspect of model performance, especially when investigating differences in the factor structure of a given test across latent classes. If intercepts, loadings, or residual variances differ across classes, measurement invariance is violated (Lubke & Dolan, 2003; Lubke et al., 2003; Meredith, 1993;
Widaman & Reise, 1997). Class-specific factor covariances do not violate measurement invariance, but detection of this type of class difference can be important
conceptually when characterizing the different classes. Note that class-specific
factor loadings, residual variances, and factor covariances lead to class-specific
covariance matrices of the observed variables Y, which may increase problems related to multiple singularities such as convergence on local maxima or nonconvergence. To minimize this risk, different sets of random starting values are
used (see later).
The data-generating model in Part 3 is a two-factor/two-class model. Four different types of data are generated: (a) data with class-specific intercepts, (b) data
with class-specific loadings, (c) data with class-specific residual variances, and (d)
34
LUBKE AND MUTHÉN
data with class-specific factor covariances. In the latter manipulation, the covariance between the two factors is close to zero in one of the classes. Data are generated under these four models with the same two levels of MDi as in Part 1. As in
Part 1, no covariates are included. In addition to fitting the true models, we also fit a
model with fixed factor loadings to the first type of data (class-specific intercepts).
Hence, the fitted factor structure of this model is similar to a simple linear growth
mixture model where the factor loadings are also fixed. Fixing parameters may increase model performance.
Data Generation and Measures of Model Performance
For all models, 100 data sets (i.e., Monte Carlo [MC] replications) are generated
using Equations 1 through 4.4 The number of subjects for each of the 100 replications is 300, with prior class probabilities for the two classes equaling 0.5.5 The
100 replications differ with respect to the random variables; that is, the subjects’
scores on the latent class variable, c, the residual factor scores, ␨, the regression residuals, ε, and the covariates X (if included). The model parameters corresponding
to the within-class model (i.e., the regression of observed variables on the factors,
and factors on the latent class variable, Equations 1 and 3) are fixed for each set of
100 MC replications. The specific parameter values used for the data generation
are listed in the Appendix.
There are four main measures of model performance in this simulation study.
The first measure is coverage of true parameter values, which is derived as follows.
For each individual MC replication, 95% confidence intervals are computed using
parameter estimates and their maximum likelihood standard errors. Our measure
of coverage is the proportion of MC replications for which the confidence interval
around the estimated value covers the true value. Throughout, coverage is reported
for the different parameter matrices separately. If a parameter matrix contains
more than a single parameter, average coverage for that matrix is reported. Second,
we consider the proportion of subjects that are assigned to their true class based on
their highest posterior class probability. Third, entropy is evaluated as a potential
indicator of the quality of class assignment. Entropy is closely related to average
class probabilities. Average class probabilities are computed by taking the mean of
the highest class probability of each subject across subjects. In our study, entropy
and average class probabilities correlated .95 and higher for each of the sets of 100
MC replications, hence only entropy is reported. Fourth, convergence rates are pre-
4As noted by an anonymous reviewer, larger numbers of MC replications would be preferable especially to obtain parameter coverage rates; however, the computation time of fitting mixture models imposes practical limits.
5Sample size effects and different class proportions are investigated in Lubke and Neale (2006).
PERFORMANCE OF FACTOR MIXTURE MODELS
35
sented. Some additional indicators of model performance are presented and explained in the Results section.
True models are fitted to the data throughout. Fifty sets of random starting values are provided for each fitted model. Ten iterations are computed for each of the
50 sets, and the 10 solutions with the highest likelihood values are then selected
and iterated until the convergence criterion is met.
Models with and without covariate effects are fitted in Part 2 to data generated
with different effect sizes of a single continuous covariate. Because model estimation is carried out conditional on the covariate, and because the covariate only predicts class membership, a model neglecting the covariate effect can also be considered a true model. Fitting a model without covariate effects to data generated with
covariates predicting class membership does not lead to biased parameter estimates. In this way, the effect of including covariates on the model performance
(e.g., parameter coverage, correct class assignment, average class probabilities,
entropy, and convergence) can be assessed. Fitting a model with a covariate effect
on class membership should result in better performance because the covariate increases the MD between classes.
The analyses are carried out using an extended version of the Runall utility in
combination with Mplus 3.12.6
RESULTS
In addition to the four main indicators of model performance, the number of Monte
Carlo replications for which one of the two fitted classes contained only two or less
subjects are reported. In these MC replications, the subjects of two classes of the
data-generating model are apparently collapsed into a single class; the second
class contains only an extremely small number of subjects. Note that throughout
this section, coverage and so on are computed for noncollapsed MC replications.
Part 1: Model Size and Complexity
The results of Part 1 are shown in Table 1. As explained earlier, parameter coverage
is measured as the proportion of MC replications for which the 95% confidence intervals of the estimated parameters cover the true parameters, and is reported for
the different parameter matrices separately. Note that 100 MC replications is a relatively small number when computing confidence intervals. Individual coverage
rates should therefore be used with some caution. The main focus here is on investigating trends when comparing model performance across different conditions.
6The
original Runall utility is available on the Mplus Web site (www.statmodel.com).
36
LUBKE AND MUTHÉN
TABLE 1
Part 1: Model Size
Coverage
Assign
Entropy
aLRT
Collapsed
Nonconverged
υ
Λ
ψ
α
Θ
αc
0
F1.d1
F1.d2
F2.d1
F2.d2
F3.d1
F3.d2
.94
.85
.70
.93
.73
.65
.37
1
14
0
.95
.86
.87
.94
.92
.75
.43
1
13
0
.94
.87
.72
.95
.74
.66
.37
1
13
0
.93
.89
.85
.96
.93
.75
.40
.98
6
0
.95
.85
.72
.94
.73
.66
.35
.98
12
0
.96
.89
.84
.94
.92
.75
.42
.99
8
0
LPM.d1
LPM.d2
.91
.96
.96
.87
.70
.45
.19
0
0
.96
.98
.85
.56
1
0
0
Note. F1 = single-factor model; F2 = two-factor model; F3 = three-factor model; LPM = latent
profile model; d1 and d2 refer to class separation MD = 1 and MD = 1.5. Parameter coverage is presented for intercepts of observed variables υ, factor loadings Λ, factor variances and covariances ψ, factor mean differences α, residuals Θ and the intercept of the latent class variable αc, which corresponds
to the estimation of the class proportions. Assign = proportion of correct class assignment; aLRT = adjusted likelihood ratio test; Collapsed = number of collapsed Monte Carlo (MC) replications as collapsed; Nonconverged = number of nonconverged MC replications.
Correct class assignment is computed as the proportion of subjects for whom the
highest posterior class probability was equal to the true class probability. In addition, Table 1 also shows the number of nonconverged and collapsed MC replications, entropy, and the adjusted likelihood ratio test (aLRT; Lo, Mendell, & Rubin,
2001).
When comparing the four different models, it is obvious from Table 1 that
model performance does not change with the increase in model size investigated
here. For the three different factor models the parameter coverage, correct assignment, entropy, aLRT, and numbers of nonconverged or collapsed MC replications
is almost identical. The latent profile model has slightly better entropy and correct
assignment.
As expected, class separation influences model performance. Parameter coverage of the factor mean and intercept differences is better for the larger distance
models, as are entropy and correct assignment.
Regarding the absolute values of model performance, parameter coverage is
above 90% for factor loadings, factor variances and covariances, and residual variances even for the smaller class separation. Factor mean differences are covered
less well, but approach 90% for larger separation. This result is important for researchers investigating factor mean differences across classes. Correct class assignment is 65% for small and 75% for larger separation. Hence, approximately
PERFORMANCE OF FACTOR MIXTURE MODELS
37
one third and one quarter of the subjects are misclassified for the two levels of class
separation studied in this part (i.e., factor mean differences of 1 and 1.5 SD, respectively). This classification error rate improves drastically with an increase in separation as shown in Part 2 next. Note that the aLRT performs very good except for
the small distance latent profile model, where based on the aLRT the necessity of a
second class would only be accepted in 19% of MC replications.
Part 2: Mahalanobis Distance, Class Separation,
and Covariate Effects
Part 2 addresses class separation. Effects of separation due to factor mean differences across classes and effects of separation due to mean differences between
classes of a single covariate are studied in a fully crossed design. This is done by
combining data that are separated by MDi based on observed variables within class
Y equaling 0.5, 1, 1.5, and 2 with covariate effects corresponding to MDi computed
for the covariate X of 0, 0.5, 1, and 1.5. Note that the observed variable and
covariate MDs are not strictly additive because covariates affecting class membership and observed variables within class are correlated, and an MD depends on the
correlation. The data-generating model is a single-factor/two-class model with
class-specific intercepts. Results can be compared to the first two columns of Table
1 featuring the same model but with class-invariant intercepts and factor mean differences.
Parameter coverage is acceptable even for the smallest class separation without
covariate effects. In fact, the coverage rates for class separation MDi = .05 are very
similar to results reported in the first column of Table 1. The results are not reported in a separate table because they do not increase significantly with increasing
separation or covariate effects, which indicates that they do not depend on the differences in class separation studied here. It is noteworthy that the coverage of the
intercepts is slightly better than the coverage for factor mean differences (see also
Table 3). The estimation of class proportions (i.e., parameter λc) increases from .33
to .73 with increasing separation without covariates and from .55 to .82 when including the largest covariate effects. Smaller covariate effects result in intermediate values.
Correct class assignment and entropy are reported in Table 2. Correct assignment increases monotonically with class separation due to intercept differences
and with increasing covariate effects. Entropy increases similarly, although increases in entropy and correct assignment are not related in a monotone way.
The results presented in Table 2 are averaged over MC replications and show
that intercept differences and covariate effects both contribute to the improvement
of correct class assignment and entropy. The following post-hoc analysis was conducted to evaluate whether equal size Y intercept differences and X mean differences as measured by the corresponding MDs are equally important predictors of
38
LUBKE AND MUTHÉN
TABLE 2
Part 2: Proportion Correct Class Assignment and Entropy
for the Single-Factor/Two-Class Model With Increasing Intercept
Differences and Covariate Effects
MDx = 0
d1
d2
d3
d4
MDx = .5
MDx = 1
MDx = 1.5
Assign
Entropy
Assign
Entropy
Assign
Entropy
Assign
Entropy
.58
.74
.85
.92
.41
.33
.54
.73
.61
.76
.86
.92
.39
.38
.56
.75
.68
.80
.88
.93
.45
.45
.62
.77
.75
.84
.90
.94
.55
.55
.69
.82
Note. d1–d4 correspond to MDi computed for ⌼ equaling 0.5, 1, 1.5, and 2, respectively. MDx is
the MD computed for ⌾.
correct assignment. Because the observed MDs vary over MC replications, it is
possible to select matched pairs of equal observed MDy and MDx, and to compare
the correct assignment proportion of the corresponding individual MC replications
using a paired t test. The t test was not significant, thereby showing no evidence
that either intercept differences or covariate mean differences have a stronger influence on correct assignment when holding the MD induced by the two types of
class separation constant.
The aLRT was not significant for a large proportion of the data sets with the
smallest intercept differences and no covariate effect (i.e., .96), indicating that the
second class is not necessary to model the data. This result improved dramatically
with increasing covariate effects or intercept differences. The aLRT was significant in all MC replications of a given combination of intercept and covariate effects when intercept differences or the covariate effect had an MD of 1, which mirrors the result of the single-factor model with factor mean differences studied in
Part 1. The number of collapsed MC replications is 24 for the smallest intercept
differences (MDi = 0.5), and decreases to 4 when adding the smallest covariate effect. All other combinations do not result in collapsing of the two true classes into a
single class. This is a dramatic difference from the single-factor model in Part 1,
which had 14 and 13 collapsed MC replications for MDi equaling 1 and 1.5, respectively. Although not the primary focus of Part 2, the result indicates that
class-specific intercepts improve this aspect of model performance (see also the results of Part 3). As in Part 1, there was no nonconvergence.
Part 3: Class-Specific Parameters
The data-generating model in this part is the two-factor/two-class model. Potential
effects of different class-specific parameters on model performance are investigated separately for intercepts, loadings, factor covariances, and residual vari-
39
PERFORMANCE OF FACTOR MIXTURE MODELS
TABLE 3
Part 3: Fixed and Model Specific Parameters
Λ Fixed
Coverage
Assign
Entropy
aLRT
Collapsed
Nonconverged
ψ1
Λ1
Θ1
α
ψ
ν2
Λ2
Θ2
ψ2
αc
d1
d2
.94
.94
Λ Cs
ν Cs
d1
d2
d1
d2
.74
.88
.88
.84
.91
.77
.91
.95
.94
.95
.95
.93
.96
.95
.95
.88
.94
.96
.76
.91
.93
.95
.96
.74
.32
1
0
0
.96
.85
.54
1
0
0
ψ Cs
.88
.70
.34
.94
0
10
.98
.78
.43
1
0
5
.73
.65
.55
.91
0
0
.94
.93
.91
.83
.58
.93
0
0
Θ Cs
d1
d2
d1
d2
.94
.95
.78
.86
.94
.96
.94
.93
.93
.91
.64
.86
.92
.93
.77
.85
.89
.92
.88
.87
.67
.33
1
0
0
.94
.97
.77
.40
.99
0
3
.77
.67
.40
.78
0
1
.89
.76
.46
.96
0
0
Note. d1 and d2 refer to class separation MD = 1 and MD = 1.5; CS = Class specific. Parameter
coverage is presented for intercepts of observed variables ν, factor loadings Λ, factor variances and
covariances ψ, factor mean differences, α, residuals Θ for both classes, and for the intercept of the latent class variable αc, which corresponds to the estimation of the class proportions. Assign = Proportion
of correct class assignment; aLRT = adjusted likelihood ratio test; Collapsed = number of collapsed
Monte Carlo (MC) replications; Nonconverged = number of nonconverged MC replications.
ances. In addition, the effect of having fixed factor loadings is investigated. Results
of Part 3 can be compared to the third and fourth columns of Table 1, which show
the results of the two-factor/two-class model with class-invariant parameters (invariant except for factor mean differences).
Fixing factor loadings has clear beneficial effects on the estimation of αc, the
parameter related to class proportions, on correct assignment, and on the number
of collapsed MC replications. The coverage of αc increases to .96 for both distances (from .74 and .93), the correct assignment increases to .74 and .85 (from .66
and .75), and there are no collapsed or nonconverged MC replications. Because the
factor structure of the two-factor/two-class model studied here is similar to a simple linear growth mixture model, similar results can be expected for the latter.
The model with class-specific loadings differs from the corresponding model in
Part 1 only with respect to the loadings in the second class. Again, there is an improvement in model performance, although the improvement is (not surprisingly)
less dramatic than for the model with fixed loadings. Estimates of factor mean differences have about the same coverage, as do the other within-class parameters.
The coverage of αc is improved for the small distance, which is accompanied by a
40
LUBKE AND MUTHÉN
slight increase in correct assignment. Entropy is not improved. Again, there are no
collapsed MC replications. Note, however, that for the small distance there are 10
nonconverged MC replications, and 5 for the larger separation.
Class-specific intercepts have slightly higher coverage rates than factor means.
Note that there are no collapsed MC replications. Entropy is higher than for the
corresponding model in Part 1, but this does not correspond with an improved correct assignment, at least not for the smaller separation. The aLRT seems to perform
slightly worse. Recall that the measures of model performance including the aLRT
are computed for the noncollapsed MC replications only.
Class-specific factor covariances are recovered well, as the coverage is .88, and
.94, respectively, which matches the coverage of the model in Part 1. Although αc
is covered slightly better, class assignment remains at .67 and .77. Entropy and
aLRT perform as in Part 1;, again, there are no collapsed MC replications.
Finally, class-specific residual variances are also recovered well; coverage does
not differ from the corresponding Part 1 results. There is no improvement regarding correct assignment, entropy, or the coverage of αc. Importantly, factor mean
differences are less well recovered if residual variances are class specific. The
aLRT performs worse for the smaller distance. There are no collapsed MC replications and one nonconverged MC replication.
DISCUSSION
Simulation studies are notorious for their limitations, and this study is no exception. The most important limiting factor 3in this study is the computation time
needed to fit large numbers of mixture models. Possible extensions of this study
may include (but are not limited to) an increase in the number of MC replications,
other types of within-class models, smaller class separation, more than two
classes, unequal class proportions, and varying sample sizes. A different study focusing on model selection when fitting correctly and incorrectly specified models
has shown that incorrect model choice due to decreasing class separation can at
least partially be compensated for with an increase in sample size (Lubke & Neale,
2006). Although this finding may extend to parameter coverage and correct class
assignment, it would need further investigation. Unequal class proportions did not
have a noticeable effect on model selection in that study; however, this may be different for parameter coverage especially if the number of subjects in the minority
class is very small. The choice of scenarios in this study is based on reviewing articles describing analyses of empirical data, which are mainly studies using growth
mixture models. Apart from the fact that loadings are usually fixed in growth models, class separation in most of those studies was larger than the separation investigated in this simulation. The general aim of this study is to obtain guidelines
PERFORMANCE OF FACTOR MIXTURE MODELS
41
concerning the performance of more general factor mixture models under less favorable circumstances.
The very general result across all three parts of this study is that parameter coverage is good even for small class separation, but that correct class assignment is
satisfactory only when classes are well separated. The study shows that the coverage of factor mean differences and αc is more sensitive to class separation than
other parameters, and decreases if variances (especially residual variances) are allowed to be class specific. Interestingly, the complexity of the within-class model
with respect to the factor structure, or the number of observed variables within
class, does not seem to greatly influence model performance. For the models evaluated in Part 1, there are virtually no differences between models.
In an empirical study, class separation regarding observed variables within
class is usually not under the control of the researcher. Although the coverage of
within-class parameters is generally good, factor mean or intercept differences,
which characterize the difference between classes researchers might be most interested in, are not covered satisfactorily at the smallest distance. In addition,
classification error rates when assigning subjects to their correct class are quite
high. As shown in Part 2, however, it is possible to improve coverage of factor
mean or intercept differences and correct assignment considerably by including
class-predicting covariates. Even if class separation with respect to the observed
variables within class is very small, small covariate effects can already reduce
classification errors. Our post-hoc comparison of class separation due to differences in Y and equal-sized class separation due to covariate effects showed no
indication that inclusion of class-prediction covariates cannot fully counterbalance lacking mean differences in Y. It is important to note, though, that both observed variables Y and covariates X are the basis for the model-based clustering
of subjects when fitting a factor mixture model with a class predicting covariate.
If the interest is in, say, examining potential subtypes of a disorder, including
covariates such as gender may lead to clusters of subjects that correspond to a
mix of subtype and gender differences. It depends therefore on the particular application whether inclusion of covariates makes sense on a conceptual level.
Having class-specific or class-invariant parameters is also usually not under the
control of the researcher. Coverage of class-specific parameters and other aspects
of model performance are especially important in the context of measurement
invariance (Meredith, 1993). Measurement invariance holds if differences between classes can be modeled with factor mean differences and factor variance and
covariance differences. In Part 1 we evaluated models that are measurement invariant across classes. The results were contrasted in Part 3 where models with different types of class-specific parameters were fitted. Most important, coverage of
class-specific parameters is comparable to the coverage of class-invariant parameters. Considering the fact that none of the models with class-specific parameters
resulted in collapsing of the true two classes into a single class, it is obvious that
42
LUBKE AND MUTHÉN
measurement-invariant models are more problematic to fit. Up to 14% of the replications in some settings of the invariant models in Part 1 resulted in collapsed
classes. In an empirical setting, in which a researcher usually has only a single data
set, inclusion of covariates may help overcome this problem.
It deserves attention that fixed factor loadings, which are common when fitting growth mixture models, result in an improvement of various aspects of
model performance. Parameter coverage of the remaining parameters is .94 and
above, and correct assignment is .74 and .85 when classes have a factor mean
difference of 1 and 1.5 SD, respectively. Fixing loadings reduces the number of
parameters involved in fitting the observed covariance structure. Hence, this
model improvement may be related to the finding that class-invariant as opposed
to class-specific residual variances increase the coverage of factor mean differences. The improvement of correct assignment when compared to a corresponding model with estimated factor loadings is not accompanied by an increase in
entropy (see later).
Given the variation found in the quality of class assignment in this study, an
interesting question is how to obtain a good measure of the quality of class assignment in practice when there is only a single data set, true class membership
is not known, and, as a result, correct assignment proportions cannot be computed. In this study, entropy and average class probabilities were highly correlated. Entropy seemed to vary with correct assignment, although not perfectly
so. This study seemed to indicate that entropy values below .60 are generally related to misclassifying approximately 20% or more of the subjects. Entropy values around .80 and above are related to at least 90% correct assignment. Although the relation between entropy and correct assignment is not monotone,
these values may serve as an indication of correct class assignment. Further research is needed to investigate cutoff values if the fitted model is a simplification
of the true data-generating process, and if the fitted model includes slight model
misspecifications.
The aLRT (Lo et al., 2001) performed very well for the models investigated in
this study. The performance deteriorated when class separation was only 0.5 SD
and no covariates were included to improve overall separation. Convergence rates
were generally above .95 and often equal to 1, and hence very satisfactory even for
small separation.
This study reveals that, in general, true factor loadings are recovered very well
even for small distances. Other aspects of model performance are less satisfactory
at the smaller distances investigated here. The study also shows that fixed factor
loadings can result in a considerable improvement of model performance. Therefore, it may be worth evaluating a two-step analysis where loadings are estimated
in a first step and fixed to the estimates in a second step. Although two-step procedures may induce other types of problems (for an interesting discussion see Ander-
PERFORMANCE OF FACTOR MIXTURE MODELS
43
son & Gerbing, 1988, 1992; Fornell & Yi, 1992a, 1992b), it may be a useful approach in an exploratory context when classes are not well separated and inclusion
of covariates is undesirable because the aim is to cluster subjects with respect to
observed variables within class.
ACKNOWLEDGMENTS
The research by Gitta Lubke was supported through a subcontract to grant
MH65322 by the National Institute of Mental Health (NIMH). The research of
Bengt O. Muthén was supported by NIMH and the National Institute on Drug
Abuse under grant No. MH40859, by NIMH under grants No. MH01259, No.
MH38725, and No. MH42968, and by the National Institute on Alcohol Abuse and
Alcoholism under grant K02 AA 00230.
REFERENCES
Anderson, J. C., & Gerbing, D. W. (1988). Structural equation modeling in practice: A review and recommended two-step approach. Psychological Bulletin, 103, 411–423.
Anderson, J. C., & Gerbing, D. W. (1992). Assumptions and comparative strengths of the two-step approach: Comment on Fornell and Yi. Sociological Methods and Research, 20, 321–333.
Arminger, G., Stein, P., & Wittenberg, J. (1999). Mixtures of conditional mean and covariance structure
models. Psychometrika, 64, 475–494.
Bamber, D., & Santen, v. J. P. H. (2000). How to assess a model’s testability and identifiability. Journal
of Mathematical Psychology, 44, 20–40.
Dolan, C. V., & van der Maas, H. L. J. (1998). Fitting multivariate normal finite mixtures subject to
structural equation modeling. Psychometrika, 63, 227–253.
Duncan, S. C., Duncan, T. E., & Strycker, L. A. (2000). Risk and protective factors influencing adolescent problem behavior: A multivariate latent growth curve analysis. Annals of Behavioral Medicine,
22, 103–109.
Everitt, B. S., & Hand, D. J. (1981). Finite mixture distributions. London: Chapman & Hall.
Fornell, C., & Yi, Y. (1992a). Assumptions of the two-step approach to latent variable modeling. Sociological Methods and Research, 20, 291–320.
Fornell, C., & Yi, Y. (1992b). Assumptions of the two-step approach: Reply to Anderson and Gerbing.
Sociological Methods and Research, 20, 334–339.
Jedidi, K., Jagpal, H. S., & DeSarbo, W. S. (1997). STEMM: A general finite mixture structural equation model. Journal of Classification, 14, 23–50.
Jedidi, K., Ramaswarmy, V., DeSarbo, W. S., & Wedel, M. (1996). On estimating finite mixtures of
multivariate regression and simultaneous equation models. Structural Equation Modeling, 3,
266–289.
Lazarsfeld, P. F., & Henry, N. W. (1968). Latent structure analysis. Boston: Houghton-Mifflin.
Lo, Y., Mendell, N., & Rubin, D. B. (2001). Testing the number of components in a normal mixture.
Biometrika, 88, 767–778.
44
LUBKE AND MUTHÉN
Lubke, G. H., & Dolan, C. V. (2003). Can unequal residual variances across subpopulations mask differences in residual means in the common factor model? Structural Equation Modeling, 10, 175–192.
Lubke, G. H., Dolan, C. V., Kelderman, H., & Mellenbergh, G. J. (2003). On the relationship between
sources of within- and between-group differences and measurement invariance in the context of the
common factor model. Intelligence, 173, 1–24.
Lubke, G. H., & Muthén, B. O. (2005). Investigating population heterogeneity with factor mixture
models. Psychological Methods, 10, 21–39.
Lubke, G. H., & Neale, M. C. (2006). Distinguishing between latent classes and continuous factors:
Resolution by maximum likelihood. Manuscript in press in Multivariate Behavioral Research.
McArdle, J. J., & Epstein, D. (1987). Latent growth curves within developmental structural equation
models. Child Development, 58, 110–133.
McLachlan, G. J., & Peel, D. (2000). Finite mixture models. New York: Wiley.
Meredith, W. (1993). Measurement invariance, factor analysis, and factorial invariance. Psychometrika, 58, 525–543.
Muthén, B. O., & Muthén, L. K. (2000). Integrating person-centered and variable-centered analysis:
Growth mixture modeling with latent trajectory classes: Alcoholism. Clinical and Experimental Research, 24, 882–891.
Muthén, B. O., & Shedden, K. (1999). Finite mixture modeling with mixture outcomes using the EM
algorithm. Biometrics, 55, 463–469.
Muthén, L. K., & Muthén, B. O. (2001). Mplus user’s guide. Los Angeles: Muthén & Muthén.
Muthén, L. K., & Muthén, B. O. (2004). Mplus version 3.01 [Computer program]. Los Angeles:
Muthén & Muthén.
Nagin, D. S. (1999). Analyzing developmental trajectories: A semi-parametric, group-based approach.
Psychological Methods, 4, 139–157.
Nagin, D. S., & Tremblay, R. E. (2001). Analyzing developmental trajectories of distinct but related behaviors: A group-based approach. Psychological Methods, 6, 18–34.
Neuman, R. J., Todd, R. D., Heath, A. C., Reich, W., Hudziak, J. J., Bucholz, K. K., et al. (1999). The
evaluation of ADHD typology in three constrasting samples: A latent class approach. Journal of the
American Academy of Child and Adolescent Psychiatry, 38, 25–33.
Rasmussen, E. R., Neuman, R. J., Heath, A. C., Levy, F., Hay, D. A., & Todd, R. D. (2002). Replication
of the latent class structure of Attention-Deficit/Hyperactivity Disorder (ADHD) subtypes in a sample of Australian twins. Journal of Child Psychology and Psychiatry, 43, 1018–1028.
Sörbom, D. (1974). A general method for studying differences in factor means and factor structure between groups. British Journal of Mathematical and Statistical Psychology, 27, 229–239.
Vermunt, J. K., & Magidson, J. (2002). Latent class cluster analysis. In J. A. Hagenaars & A. L.
McCutcheon (Eds.), Applied latent class analysis (pp. 89–106). Cambridge, UK: Cambridge University Press.
Widaman, K. F., & Reise, S. P. (1997). Exploring the measurement invariance of psychological instruments: Applications in the substance use domain. In K. J. Bryant, M. Windle, & S. G. West (Eds.),
The science of prevention: Methodological advances from alcohol and substance abuse research
(pp. 281–324). Washington, DC: American Psychological Association.
Yung, Y. F. (1997). Finite mixtures in confirmatory factor analysis models. Psychometrika, 62,
297–330.
APPENDIX
Here the parameter values used for the data generation are listed for each model.
The notation corresponds to Equations 1 through 4.
PERFORMANCE OF FACTOR MIXTURE MODELS
45
Part 1
Latent profile model.
Class-invariant parameters:
residual variances, diagonal of the matrix Θ ε = [.5.5.5.5.5.5.5.5]¢
Class-specific parameters:
regression intercepts ν = [00000000]¢
MD = 1, regression intercepts class 2 ν = [.25 -.17.4 -.5.25 -.17.4 -.5]¢
MD = 1.5, regression intercepts class 2 ν = [.45 -.35.6 -.67.45 -.35.6 -.67]¢
Single-factor model.
Class-invariant parameters:
factor loadings Λ y = [1.8.8.8.8.8.8.8]¢
factor variance Ψ = 1
regression intercepts ν = [00000000]¢
residual variances, diagonal of the matrix Θ ε = [.25.15.15.15.15.15.15.15.15]¢
Class-specific parameters:
factor mean class 1, first column of A = [0]
MD = 1, factor mean class 2, second column of A = [1.05]
MD = 1.5, factor mean class 2, second column of A = [1.55]
Two-factor model.
Class-invariant parameters:
é 1 .8 .8 .8 0 0 0 0 ù ¢
ú
factor loadings Λ y = ê
ê 0 0 0 0 1 .8 .8 .8ú
ë
û
é 1 .5ù
ú
factor covariance matrix Ψ = ê
ê.5 1 ú
ë
û
regression intercepts ν = [00000000]¢
residual variances, diagonal of the matrix Θ ε = [.25.15.15.15.25.15.15.15]¢
Class-specific parameters:
factor means class 1, first column of A = [00]¢
MD = 1, factor mean class 2, second column of A = [.945.945]¢
MD = 1.5, factor mean class 2, second column of A = [1.351.35]¢
Three-factor model.
Class-invariant parameters:
é 1 .8 .8 .8 0 0 0 0 0 0 0 0 ù ¢
ê
ú
factor loadings Λ y = êê 0 0 0 0 1 .8 .8 .8 0 0 0 0 úú
ê 0 0 0 0 0 0 0 0 1 .8 .8 .8ú
êë
úû
46
LUBKE AND MUTHÉN
é 1 .5 .5ù
ê
ú
factor covariance matrix Ψ = êê.5 1 .5úú
êê.5 .5 1 úú
ë
û
regression intercepts ν = [000000000000]¢
residual variances, diagonal of the matrix Θ ε = [.25.15.15.15.25.15.15.15.25
.15.15.15]¢
Class-specific parameters:
factor means class 1, first column of A = [000]¢
MD = 1, factor mean class 2, second column of A = [.86.86.86]¢
MD = 1.5, factor mean class 2, second column of A = [1.261.261.26]¢
Part 2
Data for Part 2 are generated using the same parameter values as listed in Part 1 for
the single-factor model, with the following exceptions.
Single-factor model with class-specific intercepts and covariate effects.
regression intercepts class 1 ν = [00000000]¢
regression intercepts class 2, MDy = 0.5, ν = [.5 -.1.15 -.075.5 -.1.15 -.075]¢
regression intercepts class 2, MDy = 1.0, ν = [.15 -.12.2 -.3.15 -.12.2 -.3]¢
regression intercepts class 2, MDy = 1.5, ν = [.2 -.2.36 -.4.2 -.2.36 -.4]¢
regression intercepts class 2, MDy = 2.0, ν = [.3 -.26.5 -.5.3 -.26.5 -.5]¢
covariate effect, MDx = 0.5, γ c = [.5]
covariate effect, MDx = 1, γ c = [1.2]
covariate effect, MDx = 1.5, γ c = [1.95]
Part 3
Data for Part 3 are generated using the same parameter values as listed in Part 1 for
the two-factor model, with the following exceptions.
Two-factor model with class-specific loadings.
é 1 .8 .6 .8 0 0 0 0 ù ¢
ú
factor loadings class 1 Λ y = ê
ê 0 0 0 0 1 .6 .8 .6ú
ë
û
é 1 .9 .6 .9 0 0 0 0 ù ¢
ú
factor loadings class 2 Λ y = ê
ê 0 0 0 0 1 .6 .9 .9ú
ë
û
PERFORMANCE OF FACTOR MIXTURE MODELS
47
Two-factor model with class-specific intercepts.
regression intercepts class 1 ν = [00000000]¢
regression intercepts class 2, MD = 1, ν = [.1.1.34 -.2.1.1.34 -.2]¢
regression intercepts class 2, MD = 1.5, ν = [.2.34.41-.35.2.34.41-.35]¢
Two-factor model with class-specific factor variances and covariances.
é 1 .5ù
ú
factor covariance matrix class 1 Ψ = ê
ê.5 1 ú
ë
û
é 1 .15ù
ú
factor covariance matrix class 1 Ψ = ê
ê.15 1 ú
ë
û
Two-factor model with class-specific residual variances.
residual variances class 1, diagonal of the matrix Θ ε = [.25.15.15.15.25.15.15.15]¢
residual variances class 2, diagonal of the matrix Θ ε = [.3.3.1.1.3.3.1.1]¢