Language of Instruction- Supplemental Materials
Supplemental Materials to “The Internal/External Frame of Reference of
Academic Self-concept: Extension to a Foreign Language and the Role of
Language of Instruction”
In this document, we include detailed information on the construction of the
achievement measures used in the analysis of the study, as well as the complete
results for the measurement invariance analysis on the factor structure of academic
self-concept (ASC) and achievement across languages of instruction (LOI).
Part I
Construction of Achievement Measures
The data set provided two sets of achievement measures. One of the achievement
measures was a standardised achievement test taken by all the students (in July, when
they were still in Grade 6) prior to their entry into the first year of secondary
schooling (Grade 7, in September). The standardised achievement test was a highstake test, such that scores from this test were comparable across schools and
represented the achievement level of the students prior to their starting secondary
school. Scores from this test were used as a basis for the selection of students in the
highly competitive Hong Kong school system. In terms of the present investigation,
this test measure can appropriately serve as the predictor of ASC, as ASC is a
measure of one’s self-perceived academic ability.
The other measurement of academic achievement in the data set was the school
marks. The school marks were the overall grades from students’ end-of-year exams in
Grade 7. These exams served as final exams and were usually designed by the school
teachers themselves as tests of the content taught to the students at that particular
school. This measurement is not comparable across schools, but this was the measure
used to inform students of their academic rankings in the classrooms at the end of the
term or school year. Students were more likely to use this information to evaluate
their ASC by comparing their abilities with those of their classmates through rankings
of school marks made public by their teachers. The standardised test could also
provide students with a frame of reference, but it would be less salient in this aspect,
compared to the school marks. Indeed, students were never told their results on the
standardised achievement tests. In short then, although the school exams might not
reflect true ability as accurately as the standardised test in terms of an absolute
standard generalisable across students and schools, the class marks and school grades
were usually more highly related to ASC than the standardised test scores. Both
standardised test scores and school marks were useful and had their own advantages
in providing unique information for the prediction of self-concept measures.
As a result, a final scale derived from school marks, moderated by the standardised
test results so that the achievement measure was comparable across schools and
classrooms (see Marsh, Kong, & Hau, 2001 for more discussion). The procedure used
to obtain the respective measurement of achievement was as follows:
Firstly, the school marks were standardised within each class in order to obtain the z
scores, which represented the relative rankings of the students in terms of academic
ability within each class.
1
Language of Instruction- Supplemental Materials
Secondly, the standardised achievement scores were standardised across all students
in the data set; then an average z score was computed for each classroom (the sum of
the z scores in a class divided by the number of students). The average standardised
achievement served as the indicator of the ability level of this class. Each class was
also assigned a unique variance by calculating the variance of the standardised test
scores (z scores) of the students within each class. In this way, the dispersion of
student ability in each class was captured. There was one low-ability-band school
(one class, 40 students) whose students’ standardised test scores were not provided in
the data. For this school, scores of -0.7 and 0.9 were assigned as the average ability
and standard deviation respectively of the student ability scores for this school. These
values were based on local knowledge of pupil intake, the band in which the school
was classified and the previous academic performances of the students studying in
this school.
Thirdly, each student was assigned a value obtained through multiplying the school
mark z scores (obtained from the first step) by the standard deviation scores of each
class (obtained from the second step). This procedure preserved both the variation of
the ability within each classroom (from the standardised test) and the rankings within
each classroom (from the school mark).
Lastly, the class average z scores (obtained from Step 2) were added to the scores
obtained in Step 3. This way the school average abilities were preserved across
schools.
Part II
Multiple Group Analysis and Structures of the Academic Self-concept in Math,
English and Chinese, across Language of Instruction (Research Question 1)
Multiple Group Analysis
Multiple group analysis was used to assess the measurement invariance of the
academic self-concepts (ASC) and their structural relations with academic
achievement across groups of language of instruction (LOI; for a detailed explaination
of this method, see Byrne, 2004; Byrne, Shavelson, & Muthén, 1989).
The baseline model tested configural invariance where both groups had the same
model structural specifications - same number of factors with the same items for each
factor, but none of the parameters were set to be equivalent across the groups. Good
fit of the baseline model is a prerequisite for testing the metric invariance between
two LOI groups, where the factor loadings are constrained to be equal across groups.
In measurement invariance analysis, the restrictive model is preferred if the fit indices
are not significantly inferior compared to those of the less restrictive model, and it is
recommended that a few rather than a single model fit indices are used to evaluate the
models (Marsh, 2007). In terms of the root mean square error of approximation
(RMSEA), the change needs to be less than 0.015 (Chen, 2007). For the comparative
fit index (CFI), the change should be less than 0.01 (Chen, 2007; Cheung & Rensvold,
2001). We presented also the Tucker-Lewis index (TLI), the standardised root mean
square residual (SRMR) and Chi-square differences as overall tests for goodness of
fit, but it should be noted that the Chi-square difference is not recommended with
large sample sizes (Marsh, Hau, Balla, & Grayson, 1998).
2
Language of Instruction- Supplemental Materials
Structures of the Math, English and Chinese ASCs across LOI
Two competing CFA models of the ASCs were tested: a model with only first-order
factors and a model with higher-order factors for verbal ASCs and achievements.
Model TGS1 (Table S1 and Table S2, Figure S1A) posited a CFA model of three
first-order ASC factors and three single-indicator achievement factors in Math,
English and Chinese. In this model, the factor loadings for ASCs (not shown) were all
very high (0.68 to 0.87) and the fit indices were excellent (e.g. CFI = 0.993, TGS1,
Table S1).
The correlation between English and Chinese achievements (0.72) was especially
high compared to the achievement correlations between math and English (0.65) and
between Math and Chinese (0.63). The correlations among self-concepts were lower
than the correlations among achievements. However, the correlation between Chinese
self-concept and English self-concept was much higher (0.45) compared to the
correlations between self-concepts in math and English (0.19) and math and Chinese
(0.24). Such a high correlation in self-concepts and achievements for English and
Chinese indicates a substantial overlap in both perceived and actual abilities in
English and Chinese, in comparison to math.
Model TGS1 showed a high correlation between Chinese and English selfconcept and achievement factors, so a higher-order model was tested. In Model TGS2
(Figure S1 B, Table S1, Table S2), a higher-order verbal self-concept factor was
specified, with English and Chinese first-order factors as indicators of this higherorder factor. Similarly, English and Chinese achievements were posited as indicators
for a verbal achievement factor. Model TGS2 had reasonable fit indices. The firstorder factors were all well defined, with factor loadings ranging from 0.67 to 0.87.
The factor loadings for the second-order verbal self-concept factor were also fairly
high (0.54 for the English first-order self-concept factor and 0.83 for the Chinese firstorder self-concept factor), thus supporting the validity of the higher-order verbal selfconcept construct.
Model TGS2’s fit indices were only slightly inferior compared to the fit indices for
Model TGS1, which is not surprising, given that there was a fairly high correlation
between the first-order Chinese and English self-concept factors in Model TGS1.
Both models seemed to be plausible in terms of supporting the construct validity of
the ASC. It is worth noting that whilst the fit of Model TGS1 was slightly better than
that of Model TGS2, Model TGS2 was a more parsimonious (i.e. it had fewer
estimated parameters and more degrees of freedom) and elegant representation of the
construct structure. We subsequently returned to these models in the evaluation of
path coefficients among factors that were specified to represent the I/E model theory
(see the main text). For now, however, we evaluated whether the factor structure
varied systematically for English-LOI and Chinese-LOI students.
The ASC Constructs and the Effects of the LOI
In order to examine whether the factorial structures of ASCs and corresponding
achievements were invariant across LOI groups, multi-group invariance tests were
conducted. Two groups were specified: students from schools where classes were
taught using English as the LOI, and students from schools where classes were taught
3
Language of Instruction- Supplemental Materials
using Chinese as the LOI. Because the analytic interest was in the pattern of results
for domain-specific factors, the invariance tests were based on Model TGS1, which is
also the best-fitting model in the previous set of analyses.
Three models (Models MGS1_1 to MGS1_3, Table S1) were tested with increasingly
restrictive model constraints. Model MGS1_1 was the baseline model with no
parameter constraints imposed, so this was the model with the best fit among the three
models. In the baseline model, both groups had the same factor specifications, and
none of the parameters were set to be equivalent. The baseline model tested configural
invariance, which was a prerequisite for the subsequent invariance assessments.
Model MGS1_1 demonstrated excellent model fit to data and therefore met configural
invariance across the two groups.
In Model MGS1_2, the loading invariance of latent constructs was tested by
constraining the factor loadings to be the same across groups. Loading invariance
ensures equal meaning of latent constructs and thus is a prerequisite for comparing the
structural invariance of the model. The model fit was nearly identical to the fit in
Model MGS1_1, even though six more degrees of freedom were gained. This showed
good support for the measurement invariances of the ASCs across LOI groups.
Model MGS1_3 additionally imposed factor covariance invariance. The model fit
remained almost the same as in Model MGS1_2, with a gain of 15 degrees of
freedom, thus indicating strong invariance of factor covariances. This could be
regarded as possible support for the I/E model across the two groups, as the factor
covariances would represent the path coefficients in the SEM model with the same
ASC and achievement constructs.
In conclusion, the CFA Model described in TGS1 generalised well across the two LOI
student groups. The students from English-LOI schools and the students from
Chinese-LOI schools shared identical ASC structures and the questionnaires used to
measure ASCs represented the same meaning across students from both English-LOI
and Chinese-LOI classes.
4
Language of Instruction- Supplemental Materials
Supplemental References
Byrne, B. M. (2004). Testing for multigroup invariance using AMOS graphics: A road less
traveled. Structural Equation Modeling: A Multidisciplinary Journal, 11(2), 272-300.
Byrne, B. M., Shavelson, R. J., & Muthén, B. O. (1989). Testing for the equivalence of factor
covariance and mean structures: The issue of partial measurement invariance.
Psychological bulletin, 105(3), 456-466.
Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance.
Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 464-504.
Cheung, G. W., & Rensvold, R. B. (2001). The effects of model parsimony and sampling
error on the fit of structural equation models. Organizational Research Methods, 4(3),
236-264.
Marsh, H. W. (2007). Application of confirmatory factor analysis and structural equation
modeling in sport/exercise psychology. In G. Tenenbaum & R. C. Eklund (Eds.),
Handbook of on sport psychology (3 ed., pp. 774-798). New York: Wiley.
Marsh, H. W., Hau, K. T., Balla, J. R., & Grayson, D. (1998). Is more ever too much? The
number of indicators per factor in confirmatory factor analysis. Multivariate
Behavioral Research, 33(2), 181-220.
Marsh, H. W., Kong, C. K., & Hau, K. T. (2001). Extension of the internal/external frame of
reference model of self-concept formation: Importance of native and nonnative
languages for Chinese students. Journal of Educational Psychology, 93(3), 543-553.
5
Language of Instruction- Supplemental Materials
6
Table S1
Summary of Goodness of Fit for CFA Models in ASCs
Model
χ2
DF CFI
CFA Model of Self-Concepts
TLI
RMSEA SRMR Description
Figure S1 A: CFA Math, English, and Chinese,
TGS1
105.865 33 0.993 0.987 0.034
0.019 FO sc, FO ach
TGS2
370.999 38 0.969 0.947 0.067
0.040 Figure S1 B: CFA Math, HO vsc, HO vach
Multiple-group CFA Models in Math, English, and Chinese (TGS1, Figure S1 A)
CFA INV=none; Free=FL, FV, FC, Uniq., CU,
MGS1_1 159.325 66 0.990 0.980 0.038
0.022 Inter (FMns=0)
CFA INV=FL; Free=FV, FC, Uniq., CU, Inter
MGS1_2 174.820 72 0.989 0.980 0.038
0.024 (FMns=0)
CFA INV=FL, FC; Free=FV, Uniq., CU, Inter
MGS1_3 230.210 87 0.985 0.977 0.041
0.047 (FMns=0)
Note. Models labeled 'TG' were based on the total (single) group, whereas 'MG' refers to models with multiple groups. CHI = chisquare, DF = degrees of freedom, CFI = comparative fit index, TLI = Tucker-Lewis index, RMSEA = root mean square error of
approximation, SRMR = standardised root mean square residual, FO = First-order, HO = Higher-order, sc = self-concept, ach =
achievement, vsc = verbal self-concept, vach = verbal achievement. For multiple-group invariance models, INV = the sets of
parameters constrained to be invariant across the multiple groups: FL = Factor loadings, PC = Path coefficients, FV = Factor
variances, Uniq = Item uniquenesses, FC = Factor covariances, CU = Correlated uniquenesses, Inter = Item intercepts, FMn =
Factor means, PC- = Path coefficients that are hypothesised to be negative, PC+ = Path coefficients that are hypothesised to be
positive.
Language of Instruction- Supplemental Materials
Table S2
Correlations of ASCs based on First-order and Higher-order CFA models
Model TGS1: CFA Math, English, and Chinese, FO
sc, FO ach
Factors MSC
ESC
CSC
MACH EACH
CACH
MSC
1.00
ESC
0.19
1.00
CSC
0.24
0.45
1.00
MACH 0.39
0.11
0.011-ns 1.00
EACH
0.078ns 0.42
0.061ns 0.65
1.00
CACH
0.044ns 0.22
0.19
0.63
0.72
1.00
Model TGS2: CFA Math, HO vsc, HO
vach
Factors
MSC
VSC
MACH VACH
MSC
1.00
VSC
0.26
1.00
MACH 0.40
0.11
1.00
VACH
0.078ns 0.44
0.75
1.00
Note. FO = First-order, HO = Higher-order, MSC = Math self-concept, ESC = English selfconcept, CSC = Chinese self-concept, VSC = verbal self-concept, MACH = Math
achievement, EACH = English achievement, CACH = Chinese achievement, VACH = verbal
achievement. Model TGS1 posited a CFA model of first-order factors consisting of Math,
English and Chinese self-concepts and achievements. Model TGS2 posited a CFA model of
first-order Math self-concept and achievement factors and second-order verbal self-concept
and achievement factors incorporating English and Chinese first-order factors. ns = nonsignificant at the 0.05 level (all other parameter estimates are statistically significant at p <
.05).
7
Language of Instruction- Supplemental Materials
A: CFA Model of First-order ASCs
B: CFA Model of Higher-order Verbal ASC and
(TGS1)
Achievement (TGS2)
Figure S1. CFA models of self-concept and achievement factors. SC = Self-concept, Ach =
Achievement, Verb = Verbal, Vsc = Verbal self-concept, Vach = Verbal achievement. The
covariances for self-concept item residuals were correlated uniquenesses for the parallel worded
items.
8