Creating Usable Knowledge to Improve
Teacher Education Programs Serving
Urban Public Schools
CENTER FOR RESEARCH ON TEACHING AND LEARNING
Department of Teaching and Learning
82 Washington Square East, Suite 700
New York, NY 10003 | 212 998 5872 | 212 995 3636 fax
www.steinhardt.nyu.edu/teachlearn/crtl
Creating Usable Knowledge to Improve Teacher Education
Creating Usable Knowledge to Improve Teacher Education
Programs Serving Urban Public Schools
A Paper Presented at the Annual Meeting of the American
Educational Research Association
San Diego, CA, April 2009
Barbara Hummel-Rossi, Robert Tobias, and Jane Ashdown
New York University
Jane Ashdown is now at Adelphi University
CRTL Research Paper Series
RP-0309-01
March 2009
Center for Research on Teaching and Learning
Department of Teaching and Learning
The Steinhardt School of Culture, Education, and Human Development
New York University
© Copyright 2009 by the Center for Research on Teaching and Learning
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
Creating Usable Knowledge to Improve Teacher Education
2
ABSTRACT
This paper reports the final phases of a multi-year investigation following teacher
education graduates (n = 3002) into the teaching profession using a longitudinal design
and value-added methods. The objective of this research was the development of an
accountability system that generates usable knowledge leading to teacher preparation
program improvements that positively impact the teaching effectiveness of program
graduates. Findings from the final phases revealed information concerning the
achievement of the teachers’ pupils, the teachers’ practice, and their career plans. Data
from all study phases are integrated to provide an overall evaluation of the teacher
education program and recommendations for developing and maintaining a teacher
education accountability system.
OBJECTIVES
This study reports on the final phases of the development of a teacher education
accountability system linking university-based teacher preparation with educationally
meaningful and measurable outcomes. A longitudinal design incorporating value-added
methods (VAM) was employed. As researchers from a large private university in New
York City (NYC), we are interested in outcomes pertinent to the quality of the local
public school system.
Phase I of this study investigated the feasibility of developing a database for
tracking teacher education graduates into the labor market (Tobias, 2005). Phase II
focused on descriptive characteristics of the teacher education graduates (n = 3002) and
the schools (n = 481) in which they were teaching (Hummel-Rossi, Tobias, Ashdown and
Smith, 2007). Phase III examined the relation between teacher education graduates and
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
Creating Usable Knowledge to Improve Teacher Education
3
their pupils’ academic performance using VAM, and Phase IV surveyed a sub-sample of
graduates about their practice. This paper presents analyses from these last two phases
and addresses the ultimate objective of the study to use the cumulative findings as a
knowledge base for improving the effectiveness of the university teacher education
programs.
PERSPECTIVE
The recent election of President Obama and the subsequent appointment of a new
Secretary of Education have renewed public attention on school reform and teacher
quality issues. The evidence from research demonstrating a strong teacher effect on pupil
achievement (Ballou, Sanders, & Wright, 2004; Rivkin, Hanushek, & Kain, 2000;
Wright, Horn, & Sanders, 1997) and a cumulative effect on pupil achievement of both
good and bad performing teachers (Mendro, 1998; Sanders, & Horn, 1998) serve as a
warrant for focusing on teacher quality in the pursuit of improved pupil performance.
Studies of teacher characteristics, such as degree level, content area preparation, licensure
exam performance, and years of teaching experience have been undertaken in an effort to
address the question of the variables demonstrating most effect on pupil achievement
(Boyd, Goldhaber, Lankford, & Wyckoff, 2007; Wayne, & Youngs, 2003; Rowan,
Correnti, & Miller, 2002; Goldhaber, & Brewer, 1997; Murnane, & Philips, 1981).
Findings concerning specific variables have been mixed. For example, a four-year study
using random assignment of teachers and pupils to classes revealed a significant effect
for teacher experience (more than three years teaching), but only for second grade
reading and third grade mathematics achievement gains (Nye, Konstantopoulos, &
Hedges, 2004).
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
Creating Usable Knowledge to Improve Teacher Education
4
Investigations into teacher and school quality have extended backwards into
examinations of the characteristics of teacher education programs as they relate to the
teaching effectiveness of the graduates of those programs. Such investigations have been
fueled in part by criticisms about the quality of university-based teacher preparation
(Levine, 2006). Thus, attention to empirical measures of success in terms of evidence of
pupil learning in teacher education research has spurred interest in value-added methods
(Noell & Burns, 2006; Wilson & Youngs, 2005).
Value-added methods have been adopted in the development of accountability
models that involve partnerships between state education departments and state university
systems (Lasley, Siedentop, & Yinger, 2006; Noell & Burns, 2006). VAM models also
are supported at schools of education through funding from the Carnegie Foundation
initiative, Teachers for a New Era (Fallon, 2006). Further examples involve
investigations of different pathways into teaching as they impact teacher quality and pupil
learning in specific labor markets, such as NYC (Lankford, Loeb, & Wyckoff, 2002;
Boyd, Grossman, Lankford, Loeb, & Wyckoff, 2006).
VAM are not new to research. In 1976, Bryk and Weisberg described VAM as an
alternative analytic approach to the feasibility problems associated with conducting
randomized experiments in educational evaluations. These authors were concerned with
questions about causal relations within the developmental processes at the core of
education. In evaluating the impact of an educational treatment or intervention, the
researchers argued that estimating expected growth for subjects under typical or “control”
conditions (the absence of the treatment or intervention under investigation) compared
with actual growth processes for subjects exposed to the intervention or treatment allows
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
Creating Usable Knowledge to Improve Teacher Education
5
for estimates of the value-added growth resulting from the intervention or treatment.
Bryk and Weisberg defined a treatment effect or value-added as “the mean difference for
a specified population between the actual post-test outcome and the predicted outcome on
the basis of natural maturation” (p.131-132).
Although VAM offer important opportunities to more fully understand the
relation between pupil achievement and teaching effectiveness, the application of these
methods presents significant measurement and technical challenges. For example,
vertically-equated tests of pupil achievement within a school system allowing pupil
performance to be linked to a common scale are largely absent from state achievement
testing systems (Martineau, 2006). Identifying an appropriate statistical model involves
many decisions including selecting variables, choosing a time frame, handling missing
data, and meeting model assumptions. An entire issue of the Journal of Educational and
Behavioral Statistics (Wainer, 2004) was devoted to discussing technical issues
associated with the use of VAM.
Noell and Burns (2006) observed that teacher education accountability efforts
face even greater hurdles than studies examining applications of value-added models to
the assessment of teacher or school effectiveness. Problems identified by Noell and
Burns include data availability and data management. Questions also have been raised
about relying on pupil test performance as the sole outcome of interest in assessing the
impact of teacher preparation (Cochran-Smith, 2006). With these complexities in mind,
Amrein-Beardsley (2008) called for guidelines for conducting VAM studies to be
established within the behavioral and social sciences.
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
Creating Usable Knowledge to Improve Teacher Education
6
RESEARCH QUESTIONS
The research questions addressed in Phases III and IV were:
Phase III: Using VAM methodologies, what is the impact of the graduates’
teacher education on their teaching practice as demonstrated by their pupils’
academic achievement?
Phase IV: How does a sub-sample of teacher education graduates evaluate their
teaching practice? What are the career plans of these graduates?
Summative: What do the results from the analyses of the above questions reveal
about the effectiveness of our teacher education programs?
METHOD
Participants
The Phase I feasibility study resulted in the creation of a database of 3,002 teacher
education graduates who completed a teacher certification program with a BS or MA
degree between 2001-2006. Of these 3,002 graduates, 1,490 taught in NYC schools
between 2001 and 2006. This database was created in a two-stage process whereby we
first matched graduates with New York State (NYS) Basic Education Data Survey
(BEDS) and then matched the resulting database with a database provided by the Human
Resources Department of the NYC Department of Education (DOE). This final database
was used to investigate Phases II, III and IV.
For the VAM study phase, the database developed in Phase I, updated to include a
few graduates from year 2007, was matched against pupil achievement data provided by
the NYC DOE. The New York State testing program (NYSTP) mandates English
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
Creating Usable Knowledge to Improve Teacher Education
7
language arts and mathematics standardized testing of all public school pupils in grades
three through eight. The match of recent graduates against the NYC DOE test data files
yielded VAM data for 191 cases.1 These were graduates from the university for the years
2001 thru 2007 who were teaching in the New York City public schools in the year 20062007 in grades 4 – 8 and who were teaching in elementary schools or teaching English
language arts (ELA) or mathematics in middle schools. In addition, it was stipulated that
each class that these graduates taught had to have at least five pupils with state ELA or
mathematics test scores. The matched cases included 29 (15.2%) who had received BS
degrees and 162 (84.8%) who had received MA degrees. This is a higher percentage of
MA graduates than the 67% that is typical for the university’s graduating classes. The
years of graduation for the matched cases ranged from 2001 to 2007, with the highest
percentage, 24.6%, having graduated in 2004 (see Table 1).
Table 1
Year of Graduation for VAM Study
Sample
Year
N
Percent
Cum. %
2001 29
15.2
15.2
2002 25
13.1
28.3
2003 24
12.6
40.8
2004 47
24.6
65.4
2005 31
16.2
81.7
2006 32
16.8
98.4
2007
3
1.6
100.0
Total 191
100.0
1
One issue that arises in these analyses concerns participants who taught classes at multiple grade levels.
In this study, separate VAM statistics were computed within grade levels for these participants, giving them
extra weight in the analysis. Specifically, a total of 36 cases, 18 elementary graduates and 18 secondary
graduates, were counted twice and five cases, one elementary and four secondary, were counted three
times. We frequently refer to the participants as cases to remind the reader of this distinction. This is a
methodological issue that will receive substantial attention in our continuing research.
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
8
Creating Usable Knowledge to Improve Teacher Education
Nearly two-thirds, 63.9% of the matched cases were graduates from elementary
teacher education programs and about one-third, 36.1%, graduated from secondary
teacher education programs. The ratio was different for BS and MA graduates. More BS
cases were graduates from elementary teacher education programs than were MA
graduates, 79.3% for the former versus 61.1% for the latter (see Table 2).
Table 2
Cross-Tabulation of Degree by Program of VAM Sample
Degree
BS
MA
Total
Count
% within
Degree
Count
% within
Degree
Count
% within
Degree
Program
Total
Elementary Secondary
23
6
29
79.3
99
20.7
63
100.0
162
61.1
122
38.9
69
100.0
191
63.9
36.1
100.0
The elementary and secondary teacher education graduates also differed in total years of
teaching experience. The secondary teacher education graduates had significantly more
total years of experience than did the elementary teacher education graduates (chisquare= 4.86, df=6, p<.0001). Indeed, a total of 37.6% of the secondary graduates had
five or more years of experience compared to 17.2% of the elementary graduates (see
Table 3).
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
9
Creating Usable Knowledge to Improve Teacher Education
Table 3
Cross-Tabulation of Years of Teaching Experience by Teacher Education Program
Experience
Less than 1
1
2
3
4
5-9
10 or more
Total
Count
% within
Program
Count
% within
Program
Count
% within
Program
Count
% within
Program
Count
% within
Program
Count
% within
Program
Count
% within
Program
Count
% within
Program
Program
Elementary
Secondary
24
2
Total
26
19.7%
2.9%
13.6%
23
11
34
18.9%
15.9%
17.8%
11
9
20
9.0%
13.0%
10.5%
19
14
33
15.6%
20.3%
17.3%
24
7
31
19.7%
10.1%
16.2%
20
19
39
16.4%
27.5%
20.4%
1
7
8
.8%
10.1%
4.2%
122
69
191
100.0%
100.0%
100.0%
Most of the participants taught in grades that were consistent with their teacher
education programs, but some did not. Of the 122 elementary program graduates, 11
were teaching secondary grades 7 or 8 and of the 69 secondary graduates, 3 were
teaching in grades 4 or 5 (see Table 4).
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
10
Creating Usable Knowledge to Improve Teacher Education
Table 4
Grades Levels Taught in 2006-2007 by Elementary and
Secondary Program Graduates
Program
Grade
Taught
4
5
6
7
8
Total
Elementary
53
44
14
9
2
122
Total
Secondary
2
1
16
32
18
69
55
45
30
41
20
191
University files contained data on license exam scores for teacher certification
and total grade point average for a subset of the cases. To be eligible for teacher
certification in New York State, applicants must pass certain exams authorized by the
State Education Department. These exams include the Liberal Arts and Sciences Test
(LAST), a test of general knowledge in the arts and sciences, and the Assessment of
Teaching Skills-Written exam (ATS-W), a written test of pedagogical knowledge, which
has separate forms for the elementary and secondary levels. These tests are scored on a
scale of 100-300, with 200 set as the passing cut-score. Our graduates’ average mean
scale scores for both the elementary and secondary teacher education programs far
exceeded the passing cut score. However, the elementary graduates showed higher mean
scale scores than did the secondary graduates; mean scale scores on the LAST were 273.5
(SD=15.9) for elementary teacher education graduates compared to 261.2 (SD=22.1) for
the secondary teacher education graduates; on the ATS-W mean scale scores were 272.1
(SD=11.8) for the elementary graduates and 263.6 (SD=11.8) for the secondary
graduates. The elementary graduates also had a slightly higher mean total grade point
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
Creating Usable Knowledge to Improve Teacher Education
11
average (GPA) than the secondary graduates, means of 3.80 (SD=0.22) and 3.77
(SD=0.33), respectively, a difference of about one-tenth of a standard deviation (see
Table 5).
Table 5
Mean State Certification Exam Scores and Mean Total GPA of
Elementary and Secondary Teacher Education Graduates
ATS-W
Total
Program
LAST Elementary Secondary
GPA
Elementary Mean 273.5
272.1
3.80
N
43
47
26
SD
15.9
11.7
.
0.22
Secondary Mean 261.2
263.6
3.77
N
16
22
37
SD
22.1
11.8
0.33
Total
Mean 270.2
272.1
263.6
3.78
N
59
47
22
63
SD
18.5
11.7
11.8
0.29
Notes: LAST is the Liberal Arts and Sciences Test for teacher certification.
ATS-W is the Assessment of Teaching Skills-Written exam.
These exams are scored on a scale of 100 – 300, with a minimum
score of 200 required for eligibility for teacher certification.
Instruments
Achievement tests. Pupils in NYS public school grades 3 - 8 participate in the
NYSTP in English language arts (ELA) and mathematics. These tests are aligned with
NYS learning standards. They have good internal consistency reliability; factor analysis
results support their construct validity; and the test blueprints show a thorough sampling
of NYS content and performance standards, thereby supporting the tests’ content
validities. The standardized test scores from these tests were the dependent variables and
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
Creating Usable Knowledge to Improve Teacher Education
12
baseline covariates in the VAM analysis. A full description of these tests is available at
www.emsc.nysed.gov/osa/pub/reports.shtml
Professional Practice. To investigate the professional practice of the teacher
education graduates, the teachers in Phase III were mailed a survey concerning their
career plans and teaching practice. Fifty-three teachers (27.7 %) returned the surveys.
Due to logistical problems in creating the database for this phase, the surveys were not
mailed until the end of May when the teachers faced numerous end-of-year
responsibilities. This timing probably resulted in the low response rate. The data on
these 53 teachers were used to investigate the research questions concerning teaching
practice and career plans.
Teachers were asked to complete the Teacher Demographic Form (TDF) and the
Domain Referenced Teacher Observation Self-Assessment (DRTO-RS). The TDF asks
for information not available in the university or NYS databases, including graduates’
future career plans. The DRTO-RS, developed for this study, is a teacher self-report
measure of four domains of teaching practice based on Danielson’s (1996) framework for
teaching. The DRTO-RS is part of a comprehensive evaluation system consistent with
the pedagogical philosophy of our teacher education programs. This 20-item instrument,
to which the teacher responds on a four-point behavioral descriptor scale, has good inter
rater agreement, alpha reliability, and construct validity.
Analysis
VAM were employed for Phase III of the study to explain the impact of the
graduates’ teacher education on their teaching as evidenced by their pupils’ achievement.
Analysis of the value-added effects of graduates was performed with data on 191
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
Creating Usable Knowledge to Improve Teacher Education
13
graduates using a value-added model developed by the NYC DOE (2008). The analysis
employed a four-level hierarchical model of students within classrooms within teachers
within schools as follows:
(1) yijkt = ujkt + Xijk,t-1 β +εijkt
(2) ujkt = δj + ZjktΠ +θjkt
(3) δj = Γ + vj
The corresponding four-level model is cross-nested for teachers who change schools and,
for simplicity, is omitted from the above notation. Covariates were used for
characteristics of individual pupils (X), classrooms (Z), and schools (not shown), which
are the average characteristics of the classrooms within schools. Characteristics of
teachers were controlled through covariation within the model and by comparing the
outcomes for each teacher to those with similar levels of experience using percentile
ranks. The metric for the dependent measure was pupils’ mean actual-versus-predicted
gain/loss on a proficiency scale of 1 to 4.5. A zero mean indicates an actual gain/loss
equal to predicted gain/loss. Empirical Bayes (shrinkage) estimators were used to adjust
the value-added teacher effects for the number of years of test data available for each
pupil in the analysis. Descriptive statistics were used to explore the phase IV questions.
Phase III and IV results are reported below.
RESULTS
VAM Findings
Using VAM methodologies, Phase III of the study addressed the impact of
graduates’ teacher education on their teaching practice as demonstrated by the academic
achievement of their pupils.
For the analysis of VAM performance, the NYC
Department of Education converted pupil ELA and math achievement test scores to a
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
14
Creating Usable Knowledge to Improve Teacher Education
proficiency-level scale, which ranged from 1 – 4.5. The scores of 2, 3, and 4 correspond
to the scale-score cut-points established by the New York State Education Department for
the corresponding proficiency levels 2, 3, and 4, with 3 signifying meeting the state
learning standards for grade. A score of 2.5 indicates performance that is mid-way
between the level 2 and level 3 cut-score, and so on. The VAM analysis compared the
actual mean gain in proficiency level achieved by the pupils for each graduate in each
grade taught to the predicted mean gain in proficiency level that was based on the VAM
equations for teachers with similar total teaching experience. As a point of reference, a
mean actual or predicted gain of zero indicates that the average pupil has made or was
predicted to have made one year’s growth in her/his grade.
Analyses of the data for the participants revealed that their pupils’ gains in ELA
and math exceeded their predicted gains based on the VAM models, thereby indicating
that the graduates had a positive VAM effect (see Tables 6 and 7).
The VAM effects for
the total participants were .03 (N=147, SD=.11) for ELA and .03 (N=129, SD=.17) for
math. Using Cohen’s effect size, the VAM effect (ES=0.27) for ELA can be considered
small and educationally meaningful while that for math (ES=0.17) is not considered
educationally meaningful. Another way to interpret these effects is to compare them to
the effects obtained by other teachers in the NYC public schools with similar total years
of experience. The median percentile ranks of the participants were 51 for ELA and 54
for mathematics, indicating that the VAM effects for the graduates were close to the
median as compared to NYC teachers with similar years of teaching experience.
Inspection of the results in Tables 6 and 7 for participants with different levels of
experience indicates that, for both ELA and mathematics, the participants at all
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
Creating Usable Knowledge to Improve Teacher Education
15
experience levels had positive VAM effects on their pupils, with the sole exception of
mathematics for teachers with three years of experience. The strongest effect sizes were
observed in ELA for teachers with two years experience (ES=0.58) and in mathematics
for teachers with one year of experience (ES=0.52), both effects considered moderate and
educationally meaningful.
Table 6
Mean Actual-Versus-Expected ELA Test Gains for Pupils of Teacher Education
Graduates and Their Percentile Rank Among All District Teachers With Similar
Years of Experience
Mean Gains in ELA Proficiency
Levels*
Years of
Statistic
Experience
Effect
Difference
Actual Predicted
(VAM)
Percentile Size
<1
M/Mdn
0.06
0.03
0.03
60
0.25
SD
0.18
0.14
0.11
N
24
24
24
1
M/Mdn
0.07
0.04
0.03
49
0.21
SD
0.19
0.13
0.15
N
28
28
28
2
M/Mdn
0.08
0.02
0.06
61
0.58
SD
0.11
0.12
0.10
N
19
19
19
3
M/Mdn
0.05
0.05
0.00
41
0.03
SD
0.18
0.12
0.12
N
24
24
24
4
M/Mdn
0.06
0.03
0.03
52
0.30
SD
0.09
0.09
0.10
N
24
24
24
5-9
M/Mdn
0.03
0.02
0.01
52
0.13
SD
0.15
0.14
0.08
N
28
28
28
Total
M/Mdn
0.06
0.03
0.03
51
0.27
SD
0.16
0.12
0.11
N
147
147
147
147
147
* Achievement scores are converted to a proficiency-level scale that ranges from 1
- 4.5 proficiency units. Gains indicate the mean difference between actual and
expected proficiency-level scores for similar students, in similar classes, and
similar schools.
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
Creating Usable Knowledge to Improve Teacher Education
16
Table 7
Mean Actual-Versus-Expected Math Test Gains for Pupils of Teacher Education
Graduates and Their Percentile Rank Among All District Teachers With Similar
Years of Experience
Mean Gains in Math Proficiency
Years of
Levels*
Statistic
Effect
Experience
Actual Predicted Difference (VAM) Percentile Size
<1
M/Mdn
0.04
0.02
0.02
45
0.16
SD
0.15
0.13
0.14
N
22
22
22
1
M/Mdn
0.22
0.12
0.10
69
0.52
SD
0.22
0.15
0.19
N
24
24
24
2
M/Mdn
0.12
0.05
0.07
58
0.37
SD
0.21
0.03
0.19
N
12
12
12
3
M/Mdn
0.04
0.10
-0.06
45
-0.38
SD
0.17
0.12
0.16
N
22
22
22
4
M/Mdn
0.09
0.08
0.01
49
0.08
SD
0.22
0.11
0.16
N
24
24
24
5-9
M/Mdn
0.08
0.06
0.02
64
0.13
SD
0.21
0.10
0.17
N
25
25
25
Total
M/Mdn
0.10
0.07
0.03
54
0.17
SD
0.21
0.12
0.17
N
129
129
129
129
129
* Achievement scores are converted to a proficiency-level scale that ranges from 1 4.5 proficiency units. Gains indicate the mean difference between actual and
expected proficiency-level scores for similar students, in similar classes, and similar
schools.
Correlates of VAM Performance
The relations between VAM effects and measured characteristics for the
participants were examined in several ways. First, VAM effects in ELA and mathematics
were plotted against years of total teaching experience (see Figures 1 and 2 for ELA and
math, respectively). In this analysis, years of experience as a variable was excluded from
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
Creating Usable Knowledge to Improve Teacher Education
17
the model, that is, leaving it uncontrolled in the model, so that its effects could be
observed in the graphs. The figures demonstrate a strong linear relation between VAM
effects and experience that is disrupted for participants in their third year. VAM effects
increase from performance below expectancy in their first year to performance that is
slightly greater than expectancy in their second year. After a sharp decrease in effects for
those in their third year, the pattern of improving performance resumes in the fourth year
and beyond. Of course, these data are cross-sectional and may be affected by cohort
effects. The fact that the decline in effects during the third year is observed in both ELA
and math is curious and warrants further investigation.
Mean of ELA 1-year Teachers VAD (Diff from Predicted)
Figure 1. Mean VAM Effect in ELA by Years of Teaching Experience
0.1000000
0.0500000
0.0000000
-0.0500000
-0.1000000
-0.1500000
Less than 1
1
2
3
4
5-9
Experience
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
Creating Usable Knowledge to Improve Teacher Education
18
Mean of MATH 1-year Teachers VAD (Diff from Predicted)
Figure 2. Mean VAM Effects in Math by Years of Teaching Experience
0.2000000
0.1000000
0.0000000
-0.1000000
-0.2000000
Less than 1
1
2
3
4
5-9
Experience
Next, in order to determine whether there was a relation between the level of
teacher education program and VAM effects, t-tests for independent samples were
applied to the differences between the mean VAM effects, using the similar experience
model, of elementary and secondary program graduates in both ELA and mathematics.
The difference was statistically significant for ELA, but not for mathematics. For ELA,
the mean VAM effect for elementary program graduates was .045 proficiency levels
higher than that for secondary program graduates; the means were .04 (SD=.12) for the
former and -.01(SD=.09) for the latter, t=2.27, df=145, p=.025.
Finally, in order to determine whether VAM effects were associated with any
other measured variables, Pearson product-moment correlation coefficients were
computed between VAM effects, controlling for experience, and Total GPA, ATS-W
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
Creating Usable Knowledge to Improve Teacher Education
19
exam scores, and LAST exam scores. None of these correlation coefficients were
statistically significant. However, using Cohen’s rubric, the correlation coefficients
between VAM effects and the ATS-W were small but possibly meaningful educationally,
r=.209, df=60, p=.108 for ELA and r=.245, df=48, p=.093 for math.
Teaching Practice
Graduates’ teaching practice was analyzed for Phase IV of this study using
completed self-assessments of their proficiency in the four domains (planning and
preparation, classroom environment, instruction, professional responsibilities) of the
Domain Referenced Teacher Observation Self-Assessment. Forty-nine of the 53
responding teachers provided useable returns; 46 of these teachers reported that their
highest degree was a master’s degree and 3 said that their highest degree was a bachelors
degree. Overall, graduates rated themselves as proficient (87.3%), as compared to initial
proficiency (0%), developing proficiency (6.1%), or advanced proficiency (10.2%).
Graduates reported themselves strongest in the domain of professional responsibilities
and weakest in the instruction domain (diff. = .43 SD). Within the professional
responsibilities domain, 87.8% of the graduates rated their relationships and interactions
with colleagues and parents as proficient or advanced proficient, and 87.8% indicated that
they were either proficient or advanced proficient in the sensitivity of their actions and
interactions with others in the community and to the cultural contexts of their teaching
situations. Teachers viewed their greatest strength as their knowledge of their
pedagogical content (89.8% proficient or advanced proficiency). In contrast, in the
instructional domain in which the graduates reported their greatest weaknesses, 59.2%
reported that they were proficient or advanced proficient in facilitating pupil-led
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
Creating Usable Knowledge to Improve Teacher Education
20
discussions, and 67.3% reported being proficient or advanced proficient in awareness of
what all their pupils were doing at all times and having the pupils monitor their own
behaviors. In a related area in the classroom environment domain, 63.2% of the
graduates said that they were proficient or advanced proficient in effectively and
efficiently transitioning from one part of a lesson to another part. Clearly, these areas of
greatest weakness are areas that benefit from teacher hands-on experience in the
classroom, but they also could receive greater attention in the student teaching
experience.
Career Plans
Analyses of graduates’ career plans were completed using data from the Teacher
Demographic Form (n = 53). The majority of the respondents (67.9%) intend to stay in
the same school with the same teaching assignment for the next academic year.
Seventeen percent expect to be in the same school, but with a different assignment, 3.8%
expect to be in another school in the same district, 7.5% expect to be in another school in
another school district, and 21.2% plan to temporarily or permanently leave teaching.
About half of the teachers who reported their birthplace (27/51 = 52.9%) grew up in New
York City (n = 13) or the suburbs of New York City (n = 14). All but one of the 53
teachers were teaching in their certified subject areas and all but one of the 53 teachers
were teaching in their certified grade areas. Forty teachers were teaching at the
elementary level, 12 were teaching at the middle school level, and one teacher taught prekindergarten. No high school teachers were sent surveys. It is encouraging that so many
teachers were teaching in their certified subject and grade areas. Although the sample
size is small, the number of teachers that plan to temporarily or permanently leave
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
Creating Usable Knowledge to Improve Teacher Education
21
teaching is discouraging and reflects the reality of urban public education today. The
National Commission on Teaching and America’s Future (Barnes, Crowe, & Schaefer,
2007) reports the national teacher turnover rate at 16.8%, but in urban schools it is over
20%.
DISCUSSION
What are the implications of the results reported above from Phase III and Phase
IV of the study with regard to the effectiveness of the university teacher education
programs experienced by the study participants? As Fallon (2006) has noted, evidence of
pupil learning should be the primary indicator of high quality teaching. Given concerns
about low academic achievement levels in urban school systems, it is encouraging that
the results reported here demonstrate that, overall, participating teacher education
graduates positively impacted pupil learning beyond predicted gains in achievement.
However, the features of the graduates’ teacher education experience that may have
contributed to this finding are more difficult to identify. For example, the VAM effect for
elementary program graduates was higher than for secondary program graduates in ELA.
A tentative interpretation could be that elementary program graduates gain more
experience in pedagogical skills in English Language Arts than their secondary education
peers. These findings were statistically significant for ELA, but not for mathematics. It
should be noted that the NYSED requires all teacher education programs to include six
credits of literacy course work, however, there is no specific credit requirement for
course work in mathematics.
The teacher characteristic years of experience in the profession was a correlate of
the VAM effect in ELA and in mathematics. Murnane and Philips (1981) posited several
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
Creating Usable Knowledge to Improve Teacher Education
22
factors as influencing the relation between experience and teaching effectiveness
including self-selection and learning-by-doing. The self-selection hypothesis suggests
that those who choose to remain in the profession differ in systematic ways from those
who choose to leave (p.94). It is a limitation of this study that we do not have information
about those graduates who chose to leave the profession or chose to teach in schools
other than in New York City. The learning-by-doing hypothesis suggests that teachers
gain experience as they spend more time on the complex tasks of teaching. This latter
hypothesis may have some explanatory power in relation to the program experiences of
the teacher education graduates. The university teacher education programs all place
emphasis on preparing educators who are reflective practitioners, that is, teachers who are
committed to a process of continuous improvement by reflecting on and learning from
their teaching experiences.
The university programs’ emphasis on reflective practice also may account for the
results from the self-assessment of teaching practice where a small sub-sample of
graduates rated themselves strongest in the domain of professional responsibilities.
Efficacy in interactions with colleagues and parents as well as sensitivity to the cultural
contexts of teaching may well be outcomes from programs that focus on examination of
and self-reflection about teaching practice. However, the weakness reported in the
domain of instruction, especially in facilitating discussions led by pupils, needs further
exploration, with potential implications for the student teaching experience.
With regard to developing the accountability system at the heart of this study, it
has required cooperation among multiple stakeholders (university program faculty,
students and graduates, the state education department, the local school system) operating
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
Creating Usable Knowledge to Improve Teacher Education
23
from different organizational perspectives, although sharing a common interest in
improving pupil achievement and teaching effectiveness. The authors engaged in this
multi-year study have varying disciplinary backgrounds including expertise in testing and
measurement, program evaluation, statistics, and clinical practice. These organizational
and intellectual collaborations speak to the complexity of the teacher education enterprise
and to the extensive resources needed to undertake a more sophisticated evaluation of
teacher preparation than previously has been attempted. In addition, a teacher education
accountability system can only be of value as a tool of program improvement where
robust feedback systems are in place to turn data analyses into usable knowledge and an
evidence base for program improvements. Creating and engaging in such feedback
systems has implications for the role of program faculty, clinical supervisors, and school
partners. Work in this area currently is underway as a further development of this
accountability system.
It is worth concluding with a reminder of the limitations of this study. Of note,
we were able to match only a relatively small number of cases from our original sample.
In addition, we do not know how the unmatched cases differ from the matched cases.
Noell and Burns’ (2006) caution about the challenges of data availability and
management in the application of VAM continues to hold true. However, it also should
be noted that the findings reported here depended on the considerable cooperation of staff
at the NYC Department of Education for which we are highly appreciative.
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
Creating Usable Knowledge to Improve Teacher Education
24
REFERENCES
Amrein-Beardsley, A. (2008). Methodological concerns about the education valueadded assessment system. Educational Researcher, 37, 65-75.
Ballou, D., Sanders, W., & Wright, S. (2004). Controlling for student background
variables in value-added assessment in education. Journal of Educational and
Behavioral Statistics, 29, 37-65.
Barnes, G., Crowe, E., & Schaefer, B. (2007). The cost of teacher turnover in five
school districts. Washington, DC: The National Commission on Teaching and
America’s Future.
Boyd, D., Goldhaber, D.. Lankford, H., & Wyckoff, J. (2007), The effect of certification
and preparation on teacher quality. The Future of Children, 17, 45-68.
Boyd, D., Grossman, P., Lankford, H., Loeb, S., & Wyckoff, J. (2006). How changes in
entry requirements alter the teacher workforce and affect student achievement.
Education Finance and Policy, 1(2), 176-216.
Bryk, S., & Weisberg, H. L. (1976). Value-added analysis: A dynamic approach to the
estimation of treatment effects. Journal of Educational Statistics, 1, 127-155.
Cochran-Smith, M. (2006). Thirty editorials later: Signing off as editor. Journal of
Teacher Education, 57, 95-101.
Danielson, C. (1996). Enhancing professional practice: A framework for teaching.
Alexandria, VA: Association for Supervision and Instruction.
Fallon, D. (2006). The buffalo on the chimneypiece: The value of evidence. Journal of
Teacher Education, 57, 139-154.
Goldhaber, D., & Brewer, D. J. (1997). Why don’t schools and teachers seem to matter?
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
Creating Usable Knowledge to Improve Teacher Education
25
Assessing the impact of unobservables on educational productivity. The Journal
of Human Resources, 32, 505-523.
Hummel-Rossi, B., Tobias, R., Ashdown, J., & Smith, A. (2007, April). Value-added
methods applied to an accountability model for teacher education. Paper
presented at the meeting of the American Educational Research Association,
Chicago, IL.
Lankford, H., Loeb, S., & Wyckoff, J. (2002). Teacher sorting and the plight of urban
schools: A descriptive analysis. Educational Evaluation and Policy Analysis, 24,
37-62.
Lasley, T. J., Seidentop, D., & Yinger, R. (2006). A systemic approach to enhancing
teacher quality: The Ohio model. Journal of Teacher Education, 57, 37-50.
Levine, A. (2006). Educating school teachers. The Education Schools Project..
Washington, DC
Martineau, J. A. (2006). Distorting value added: The use of longitudinal vertically
scaled student achievement data for growth-based, value-added accountability.
Journal of Educational and Behavioral Statistics, 31, 35-62.
Mendro, R. L. (1998). Student achievement and school and teacher accountability.
Journal of Personnel Evaluation in Education, 12, 257-267.
Murnane, R. & Philips, B. R. (1981). What do effective teachers of inner-city children
have in common? Social Science Research, 10, 83-100.
New York City Department of Education. (2008, June). Value-added technical report.
Report to VAM Technical Advisory Board. Draft. New York: NYC DOE.
Noell, G. H., & Burns, J. L. (2006). Value-added assessment of teacher preparation: An
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
Creating Usable Knowledge to Improve Teacher Education
26
illustration of emerging technology. Journal of Teacher Education, 57, 37-50.
Nye, B., Kostantopoulos, S., & Hedges, L. V. (2004). How large are teacher effects?
Educational Evaluation and Policy Analysis, 26, 237-257.
Rowan, B., Correnti, R., & Miller, R. J. (2002). What large-scale, survey research tells
us about teacher effects on student achievement: Insights from the Prospects
study of elementary schools. Teachers College Record, 104, 1525-1567.
Rivkin, S. G., Hanushek, E.A., & Kain, J.F. (2000). Teachers, schools and academic
achievement. Cambridge, MA: National Bureau of Economic research, NBER
Working Paper #W6691
Sanders, W. L., & Horn, B. (1998). Research findings from the Tennessee Value-Added
Assessment System (TVASS) database: Implications for educational evaluation
and research. Journal of Personnel Evaluation in Education, 12, 247-256.
Tobias, R. (2005, July). Results of the first follow-up study of graduates of the
Steinhardt School of Education’s teacher education programs: Graduates from the
class of 2001-2005 teaching in New York State. New York University: Center for
Research on Teaching and Learning, Research Report Series RR-0705-01.
Wainer, H. (Ed.). (2004). Journal of Educational and Behavioral Statistics, 29, 1-156.
Wayne, A. J., & Youngs, P. (2003). Teacher characteristics and student achievement
gains: A review. Review of educational Research, 73, 89-112.
Wilson, S. M., & Youngs, P. (2005). Research on accountability processes in teacher
education. In M. Cochran-Smith & K. Zeichner (Eds.), Studying teacher
education: The report of the AERA panel on research and teacher education.
Mahwah, NJ: Lawrence Erlbaum Associates.
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
Creating Usable Knowledge to Improve Teacher Education
27
Wright, S., Horn, S., & Sanders, W. (1997). Teacher and classroom context and effects
on student achievement: Implications for teacher evaluations. Journal of
Personnel Evaluation in Education, 11, 57-67.
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01
Creating Usable Knowledge to Improve Teacher Education
28
Author Note
Jane Ashdown is now at the City University of New York.
Barbara Hummel-Rossi, Department of Applied Psychology, New York
University.
Robert Tobias, Center for Research on Teaching and Learning, Department of
Teaching and Learning, New York University.
The authors are pleased to acknowledge Ognjen Simic, Research Scientist, Center
for Research on Teaching and Learning, for his assistance in data management and data
analysis for this study
_______________________________________________________________________
CRTL Paper Presentation Series RP-0309-01