Using Growth Models to Monitor School
Performance Over Time: Comparing NCE, Scale
and Scores on NRTs and SBTs
Pete Goldschmidt, Kilchan Choi, Felipe Martinez,
and John Novak
American Educational Research Association
Annual Meeting
March, 2008
Introduction
Using Growth Model Value Added estimates,
do inferences about school change
Examine the role of the metric
NCE vs Scale Scores on a Vertically
equated assessment.
Examine the role of switching Assessment
type NRT vs SBT
4
Summary Parameter Estimates Compared
Estimated Initial Status
Residual Initial Status
Estimated Growth
Value Added
5
Summary of Estimates Compared Using
Rank Order Correlations
Also compare school ranks based on the
residual Initial Status and Value Added
estimates
6
Summary of Results Describing SAT-9 Reading Achievement
25%
SAT-9 Rea d in g Ach ievemen t
50%
75%
N CE
SS
N CE
SS
N CE
SS
Sp ecia l Ed u ca tion ( 010 )
-0.47
-0.44
-0.47
-0.44
-0.47
-0.44
Low SES ( 020 )
-0.36
-0.4
-0.35
-0.4
-0.35
-0.39
LEP ( 030 )
-0.34
-0.35
-0.33
-0.34
-0.32
-0.33
Min ority ( 040 )
-0.48
-0.54
-0.48
-0.54
-0.48
-0.53
0.1
0.1
0.1
0.1
0.1
0.1
LAAMP Effect ( 001 )
0.03
0.04
0.02
0.03
0.02
0.02
Min ority ( 002 )
-0.01
-0.01
-0.01
-0.01
-0.01
-0.01
Low ( 003 )
0.13
0.1
0.17
0.15
0.2
0.17
0.07
0.64
0.07
0.63
0.07
0.63
0
-0.03
0
-0.03
0
-0.03
0.05
0.06
0.05
0.06
0.05
0.06
Mea n In itia l sta tu s (g 000 )
Stu d en t Pred ictors
Girl ( 050 )
Sch ool Pred ictors
Mea n Grow th (g 100 )
Stu d en t Pred ictors
Sp ecia l Ed u ca tion ( 110 )
Low SES ( 120 )
LEP ( 130 )
0.07
0.07
0.07
0.07
0.07
0.07
Min ority ( 140 )
-0.03
-0.02
-0.03
-0.02
-0.03
-0.02
0.01
0.01
0.01
0.01
0.01
0.01
LAAMP Effect ( 101 )
0.01
0.01
0.01
0.01
0.01
0.01
Min ority ( 102 )
0.11
0.14
0.12
0.15
0.12
0.16
Low ( 103 )
-0.08
-0.08
-0.08
-0.08
-0.08
-0.08
Girl ( 150 )
Sch ool Pred ictors
8
Correlations Between Value added estimates for NRT for
models without student covariates
Spearman Correlation
Sample
R25
R50
R75
Test Type
Initial Status Growth
Kendall (Tau) Correlation
Initial Status
Growth
Read
0.988
0.936
0.925
0.806
Math
0.987
0.963
0.925
0.863
Read
0.99
0.932
0.931
0.798
Math
0.988
0.964
0.929
0.87
Read
0.991
0.932
0.935
0.798
Math
0.989
0.964
0.932
0.871
9
Correlations Between Value added estimates for NRT for
models with student covariates
Spearman Correlation
Sample Test Type
R25
R50
R75
Initial Status Growth
Kendall (Tau) Correlation
Initial Status
Growth
Read
0.964
0.914
0.857
0.779
Math
0.975
0.955
0.887
0.849
Read
0.97
0.91
0.872
0.775
Math
0.978
0.956
0.898
0.857
Read
0.974
0.908
0.881
0.776
Math
0.981
0.955
0.905
0.857
13
Comparison of Relative Bias to the Effect Size of Growth
Efect Size of Grow th (scale scores)
0.62
-0.89
-0.88
Relative Bias in Growth
-0.87
-0.86
-0.85
-0.84
-0.83
-0.82
-0.81
-0.80
-0.79
0.63
0.64
0.65
0.66
0.67
0.68
0.69
0.70
0.71
0.72
0.73
Correlations between School Means by Year:
NRT1 and SBT2
Reading
Math
Year
2002
2003
2004
(SAT-CST) (CAT-CST) (CAT-CST)
0.971
0.967
0.960
0.949
0.955
0.887
1 NRT consists of SAT9 and CAT6
2 SBT is California Standards Test
Estimated effects of student characteristics in initial Status and Growth
for NRT and SBT
Reading
Mathematics
Mean Initial status (g000)
Girl (g010)
Special Education (g020)
Low SES (g030)
LEP (g040)
Minority (g050)
NRT
0.17
0.07
-0.70
-0.47
-0.14
-0.30
SBT
-0.13
0.15
-0.72
-0.60
-0.15
-0.31
NRT
0.14
-0.09
-0.71
-0.46
0.04
-0.30
SBT
0.07
-0.14
-0.74
-0.54
0.01
-0.37
Mean Growth (g100)
Girl (g110)
Special Education (g120)
Low SES (g130)
LEP (g140)
Minority (g150)
-0.26
0.09
0.07
-0.08
0.04
-0.01
0.13
0.01
0.05
0.00
0.06
0.02
-0.20
0.02
0.03
-0.03
0.02
0.01
-0.01
0.05
0.10
-0.01
0.06
0.07
Correlations among Value Added estimates based on:
NRT1 and SBT2
Spearman Correlation
Kendall (Tau) Correlation
Sample
Test Type Initial Status Growth Initial Status
Model 1 -Unconditional
Read
0.979
0.548
0.880
Growth
Math
0.966
0.793
0.830
Model 2 –
Read
0.954
0.740
0.781
Growth with Covariates
Math
0.933
0.808
0.772
1 NRT consists of SAT9 and CAT6
2 SBT is California Standards Test
Growth
0.340
0.627
0.534
0.659
Mathematics
School context and inferences
While individual student characteristics’ impact differ
depending on assessment used (though not metric) particularly for growth,
School enrollment characteristics have virtually no impact
inferences between NRT and SBT.
Mathematics
Reading
Mathematics
Relationship between missing scores and school
performance
3
Value Added
2
1
0
-1
-2
-3
0
10
20
30
40
50
School Mean Performance
60
70
80
Relationship between missing scores and school
performance
Performance based on
School Means
NRT
CST
Value Added
NRT
CST
* p < .05
Percent Missing
-0.42 *
-0.42 *
0.2
0.05
Summary – the scale
Using a relative scale for monitoring
individual achievement growth when the
assessment is vertically equated –
significantly under-estimates growth.
Using a relative scale for monitoring
school performance based on growth
when the assessment is vertically equated
– yield very consistent results to using an
absolute scale.
No patterns as to where deviations
may occur.
Summary – the assessment
Individual results between NRT and CST highly correlated in
each year.
Individual student characteristics affect relative performance
Attempting to become more egalitarian?
School results fairly consistent in Mathematics, but not in
Language Arts
School characteristics have virtually no impact on
changes in inferences or rankings of schools.
Summary – the method
Means highly correlated with student background
Means inversely correlated to misingness
VA added estimates based on individual growth
substantively less related to student background
VA added estimates based on individual growth
substantively less related to missingness