IMPS2001, July 15-19,2001
Osaka, Japan
Use of SEM programs to
precisely measure scale
reliability
Yutaka Kano and Yukari Azuma
Osaka University
1
2
p
Reliability measure for
X
X
i
i 1
Reliability with possibly correlated errors
X i   i  i f  u i   ' 
α coefficient
p 

1

p 1










i

i
2

i  


V ( X ) 
i
V (X )



i 


2
 Cov(u , u )
i
i, j
j
3
An example
α 0.69
ρ’ 0.69
0.74
0.64
0.78
0.60
4
From the example
Coefficient alpha can be distorted
seriously by error correlations

e.g. Green-Hershberger (2000), Raykov (2001)
In the case, ρ’ has to be used to
correctly figure out the reliability
'









i
2

i

i  



i 


2
 Cov(u , u )
i
i, j
j
5
Problem
How can one identify error correlations?


The factor model allowing for (fully)
correlated errors is not identifiable, because
it contains too many parameters
A trivial solution would be
^
Cov(u , u )  s
i
j
ij
  ij (ˆ)
It does not work because

i j
^
Cov(u , u )  0
i
j
6
LM approach
Start from the factor model with no error
correlation
Perform the LM test for releasing a zero
covariance between errors using a SEM
program
EQS can perform it most easily and most
accurately
7
Real data analysis
A questionnaire on perception on physical
exercise
n=653, p=15, one-factor model
The data were collected by Dr. Oka
(Waseda U.)
8
Result_1
Best fitted model, with 7 correlated errors


χ2=250.375(df=83) (n=653)
GFI=0.950, CFI=0.952, RMSEA=0.056
9
Result_2
Estimates of reliability



α = 0.90
^
ρ’ = 0.90 by Cov(ui , u j )  sij   ij (ˆ)
ρ’ = 0.87 by LM test
Note that
 s

ˆ)  0.005


(

ij
ij
i j

^
Cov(u , u )  1.996
7covariances
i
j
10
Search for variables
Even though one factor model is fitted
well, inclusion of a variable with small true
variance can reduce reliability
There is no convenient way to select
variables for the composite scale to have
maximum reliability
11
Mathematically…
Let X i  i f  ui with V ( X i )  1 and ui uncorrelated. If
i 
 AB  A A 2  B 2  B
then dropping X j
where A 
,
A B
increases reliabilit y,
2
  , B  V (u ).
j
j i
j
j i
It is complicated
It will become more complicated if error
correlations are allowed
12
New program
A new program is being developed which


gives a list of reliability estimates for each
factor;
gives a list of predicted reliability estimates
when one variable is removed
Error correlations are allowed
13
Flowchart
No
Well fit?
DATA
Factor analysis
Yes
Decide composite
scale items
Free some
error covariances
to get good fit
Print reliability
End
14
Example, continued
ρ’ = 0.87 with 15 variables
15
Scale
developer
16
If V13 is
removed,
then…
17
Results
For one-factor model with uncorrelated errors,
the variable with the smallest factor loading is
least favorable.

If there is a variable whose deletion improves
reliability, then this is the variable.
For one-factor model with correlated errors, the
variable with the smallest factor loading is not
always least favorable.

While deletion of the variable does not improve
reliability, there may be other variables to be deleted
to improve reliability.
The example here is the case.
18
Summary
Correlated errors invalidate the coefficient alpha
and traditional one-factor based reliability.
LM test is useful to find error correlations.
Magnitude of factor loadings does not
necessarily provide accurate information on
indicator selection when correlated errors exist.
The forthcoming Web-based program will help
reliability analysis.