Fourth Annual Report to the Teacher
Education Accreditation Council
(TEAC)
CENTER FOR RESEARCH ON TEACHING AND LEARNING
1
Fourth Annual Report to TEAC, February 2011
Fourth Annual Report to the Teacher Education
Accreditation Council
Robert Tobias, Director
Center for Research on Teaching and Learning
CRTL Accreditation Report
AR-0211-01
February 2011
Department of Teaching and Learning
The Steinhardt School of Culture, Education, and Human Development
New York University
Fourth Annual Report to TEAC, February 2011
CONTENTS
INTRODUCTION
Evidence Base
UPDATED PROGRAM OUTCOMES
DRSTOS-R
Follow-Up Surveys
New York State Teacher Certification Exam (NYSTCE)
Student Teacher End of Term Feedback Survey
Grade Point Averages
4
4
7
7
8
12
13
16
USE OF DATA FOR PROGRAM IMPROVEMENT
16
TEACHER EDUCATION RESEARCH BIBLIOGRAPHY
17
2
Fourth Annual Report to TEAC, February 2011
LIST OF TABLES
TABLE 1
Summary of measures, cohorts, and subjects for inquiry by claim
6
TABLE 2
Mean scores and percents meeting standards on the Domain Referenced Student Teacher
Observation Scale-Revised (DRSTOS-R) for Steinhardt teacher education students in
their final student teaching placement (fall 2009 –spring 2010)
8
TABLE 3
Comparison of self perceptions of Class of 2009 Steinhardt teacher education graduates
and Levine study sample as to how well their teacher education program prepared them
11
for teaching
TABLE 4
Mean scaled scores, effect sizes, and passing rates for Steinhardt teacher education
graduates on the New York State Teacher Certification Exams: Class of 2010
13
TABLE 5
Mean scores for claim scales on the End of Term Feedback Questionnaire Classes of
2009 and 2010
15
TABLE 6
Mean total GPA for BS and MA teacher education graduates: Class of 2010
16
APPENDIX A (TEAC APPENDIX E)
20
APPENDIX B
27
3
Fourth Annual Report to TEAC, February 2011
INTRODUCTION
Having received initial accreditation of our teacher education programs in January
2007, this is the fourth annual report submitted by NYU’s Steinhardt School of Culture,
Education, and Human Development to the Teacher Education Accreditation Council
(TEAC). This report focuses on our self-study activities for the 2009 – 10 academic year.
Prepared according to the specifications on TEAC’s web site, the report includes (1)
updates of most of the tables included in the results section of the first Inquiry Brief, (2)
an update of Appendix E, the evidence that the program self study relies upon, and (3)
tables that describe the databases that contain the evidence. In addition, the report
describes changes in the measures that are being used in the ongoing self-study of the
effectiveness of Steinhardt’s teacher education programs, the most recent results from the
analyses of data, and ways in which the data have been used to improve the programs.
The report also updates a bibliography of papers and reports on the research that NYU
faculty and research centers have performed on NYU’s teacher education programs.
THE EVIDENCE BASE
The third annual report documented several changes to the measures used to
construct the evidence base for self study since the initial inquiry brief. Additional
changes were made during the past academic year, which can be seen in the table that
summarizes the measures and the data that are being collected in the ongoing self study
(see Table 1.) First, we have continued to expand the assessment of our student teachers
developing pedagogical proficiency using the Domain Referenced Student Teacher
Observation Scale-Revised (DRSTOS-R). In 2009 – 10, we trained an additional 25
student teacher supervisors and assessed more than 350 student teachers in spring 2010,
passing the milestone of 300 for the first time. In addition, Steinhardt’s Center for
Research on Teaching and Learning (CRTL) issued individual program reports to the
faculty of Steinhardt’s 10 program certification areas, providing them with detailed
results to inform program evaluation and planning.
The third annual report also described the development of the Educational Beliefs
and Multicultural Awareness Survey (EBMAS), a new dispositions survey which
replaced the Educational Beliefs Questionnaire, which had been in use since 2004. Based
on continued investigation of the psychometric properties of EBMAS, CRTL altered its
scale structure by reorganizing the items into three scales rather than the initial four. This
was accomplished by combining most of the items from the Caring and Social Justice
scale and the Multicultural Awareness scale into a single scale labeled as Multicultural
Awareness (MCA). Factor analysis of a body of data from a larger sample than the
original and inspection of the items by faculty and staff supported the change in scale
4
Fourth Annual Report to TEAC, February 2011
structure (see Table 1). In addition to MCA, EBMAS measures two types of teacher
efficacy: General Teacher Efficacy (GTE), the belief in the power of teaching to educate
all students, and Personal Teacher Efficacy (PTE), the individual teacher’s belief in their
own ability to teach all children. The change in the scale structure of EBMAS
necessitated a modification in the alignment between measures and claims for the self
study. GTE and PTE were moved from claim 2, pedagogical knowledge to claim 3,
teaching skill and caring, which is a better conceptual fit. EBMAS Scale 3, Caring/Social
Justice no longer exists and was removed from the study’s measures. A paper on the
development and of EBMAS is being revised and EBMAS data in support of the claims
will be presented in the Inquiry Brief for continued accreditation.
The third annual report also described the expanded methods and procedures
CRTL is using for following up cohorts of NYU teacher education program graduates in
efforts to assess their post-graduates success in the profession. These methods include
electronic tracking of graduates using city and state human resource systems, value-added
modeling using pupil achievement test scores, and graduate follow-up surveys. Data
from these expanded methods will be reported in the next Inquiry Brief, with the
exception of some of the results from the one-year follow-up survey of graduates from
May 2009, which are included in this report.
CRTL continues to collect, analyze, and report data on self-study measures,
including scores on the New York State Teacher Certification Exams, student teacher self
reports of the contributions of their field supervisors and cooperating teachers to their
professional growth, and the grade point averages of graduates. All of these data are
being used by program directors to inform the continuous improvement of the programs.
Overall, these data indicate that NYU continues to meet its program claims with
substantial improvements in outcomes for some of the measures. However, discussions
of the data and their implications for our programs have led to consideration of changes
to the curriculum and greater emphasis on certain area, such as clinical practice. These
changes may require us to re-think some of the fundamental principles of our teacher
education program, including the goals which guide our claims. These discussions are
ongoing and will be reflected in our next Inquiry Brief.
5
Fourth Annual Report to TEAC, February 2011
Table 1
Summary of measures, cohorts, and subjects for inquiry by claim
(Updated January 2011 for 2012 Inquiry Brief)
Claims
Measures
Cohort
Subjects
Classes of 2007 – 10
Claim 1:
Subject
Matter
Knowledge
DRSTOS-R: Planning & Preparation
Score and Total Score
NYSTCE Exam Scores: Content
Specialty Tests
Student Teacher ETFQ
Classes of 2006 – 10
Most program
graduates
All tested students
Classes of 2006 – 10
All respondents
Content Core GPA; undergrad major
GPA
1 & 5 year Follow-Up Surveys
Program Exit Survey*
Classes of 2006 – 10
All BS graduates; all
incoming MA students
All respondents
All respondents
Claim 2:
Pedagogical
Knowledge
DRSTOS-R: Instruction and Classroom
Environment Scores and Total Score
NYSTCE Exam Scores:
--ATS-W Exam Score
Student Teacher ETFQ
Pedagogical Core GPA
1 & 5 year Follow-Up Surveys
Program Exit Survey*
Claim 3:
Teaching
Skill/
Caring
DRSTOS-R: Professional
Responsibilities & Classroom
Environment Domain Scores and Total
Score
Student Teacher ETFQ
EBMAS Scales 1 & 2: General
Teaching Efficacy & Personal
Teaching Efficacy**
Student Teaching GPA
1 & 5 year Follow-Up Surveys
Program Exit Survey*
Pupils’ value-added standardized
test scores***
Liberal Arts
Cross-Cutting
Themes
Multicultural
Awareness
Classes of 2007 – 09
Classes of 2006, 2008 –
10
Classes of 2007 – 10
Most program
graduates
Classes of 2006 – 10
All tested students
Classes of 2006 – 10
All respondents
Classes of 2006 – 10
All program graduates
Classes of 2007 – 09
Classes of 2006, 2008 –
10
Class of 2007 – 10
All respondents
All respondents
Most program
graduates
Classes of 2006 – 10
Class of 2010 – 11
All respondents
All respondents
Classes of 2006 – 10
Class of 2007 – 09
Classes of 2006,, 2008 –
10
Classes of 2004 – 2008
All program graduates
All respondents
All respondents
DRSTOS-R: Total Score
Classes of 2007 - 10
NYSTCE Exam Scores:
--Liberal Arts & Sciences
Classes of 2006 - 10
All program graduates
teaching grades 4 – 8 in
NYC public schools
Most program
graduates
All tested students
Liberal Arts Core GPA; Total undergrad
GPA
Classes of 2006 - 10
All BS graduates; all
incoming MA students
EBMAS: Scale 3: Multicultural
Awareness
Classes of 2010 – 11
All respondents
Note: Bold type signifies new measures that were not included in the original
Inquiry Brief.
6
Fourth Annual Report to TEAC, February 2011
* Replaces the Fast Track End of Program Questionnaire (FTEPQ)
** Replaces the Educational Beliefs Questionnaire (EBQ)
*** Also used as a measure of subject matter knowledge and pedagogical knowledge
UPDATED PROGRAM OUTCOMES
DRSTOS-R
Table 2 presents DRSTOS-R ratings for students in their final student teaching
placement for the period fall 2009 thru spring 2010 for 186 BS students and 373 MA
students. Similar data were reported in the first Inquiry Brief and in the third annual
report. With the exception of Planning and Preparation for BS students, the mean scores
for all domains and the total scale for both BS and MA students exceed 3.0, the value that
indicates “Entry Level Proficiency” on the four-point scale. Accordingly, the average
student in the final student teaching placement met the program’s proficiency standard.
However, the percentage of students meeting the proficiency standard fell short of the
program goal of 80% for nine of the ten measures; the lone exception was Professional
Responsibilities for MA students (85.5%). The results for MA students were stronger
than they were in the first Inquiry Brief, but lower than those reported in the third annual
report for students during the period fall 2007 – spring 2009. The MA students
performance far exceeded that of the BS students, with Total Score means of 3.25 (SD =
0.51) for the former and 3.07 (SD = 0.62) for the latter. The first Inquiry Report
highlighted the need for improved performance on DRSTOS-R by MA students which
has been realized in the results of both the third and fourth annual reports. However, the
decline in performance for the MA students relative to the last report needs to be
addressed, as does the lower scores for the BS students.
In the third annual report, the improved performance of the MA students was
linked to the a three-year partnership program with the NYC Department of Education
and the City University of New York that was designed to upgrade the MA programs
through improved field experiences and enhanced course curricula. That program ended
two years ago, which resulted in the loss of funds that supported resources, including
staff for the field component. At the same time, organizational changes in the New York
City public schools have created pressures on schools that may affect the time and
attention that cooperating teachers can devote to mentoring student teachers. Also, it is
difficult to measure the extent to which the changes are a statistical artifact of the rapid
expansion of DRSTOS-R training and administration. Steinhardt teacher education
faculty is exploring these and other issues in an attempt to understand the trends in
performance and ways to improve them.
7
Fourth Annual Report to TEAC, February 2011
Table 2
Mean scores and percents meeting standards on the Domain Referenced Student
Teacher Observation Scale-Revised (DRSTOS-R) for Steinhardt teacher education
students in their final student teaching placement (fall 2009 - spring 2010)
Claims
Scale Domain
Number
of Items
Number
of
Students
Mean
Score
(Scale: 1-4)
Standard
Deviation
% Meeting
Standards *
BS Students
1
Planning and Preparation
6
186
2.95
0.60
57.5%
3
Classroom Environment
7
186
3.11
0.53
60.2%
2
Instruction
5
186
3.04
0.56
60.2%
3
Professional Responsibilities
3
186
3.29
0.62
78.5%
Total Score
21
58
3.07
0.62
58.1%
1,2,3
Crosscutting
Theme
MA Students
1
Planning and Preparation
6
373
3.22
0.56
74.5%
3
Classroom Environment
7
373
3.22
0.54
70.2%
2
Instruction
5
373
3.20
0.54
71.8%
3
Professional Responsibilities
3
372
3.43
0.62
85.5%
Total Score
21
373
3.25
0.51
70.0%
1,2,3
Crosscutting
Theme
* The program goal is 80% of students with mean scores of at least 3.0
Note. Nominal scale values are as follows: (1) Not Yet Proficient; (2) Partially Proficient; (3) Entry Level
Proficient; (4) Proficient. The standard for program graduates is Entry Level Proficient (3).
Follow-Up Surveys
In spring 2010, 135 (36.5%) of January and May 2009 graduates responded to an
on-line survey constructed by CRTL in consultation with faculty from teaching and
learning and applied psychology and representatives from the NYC Department of
Education. The response rate is typical for on-line surveys and the respondents were
closely representative of the total population of graduates by degree and certification
program. Two-thirds of the respondents were teaching in their first year after graduation
with 32% in NYC public schools, 15% in NYC private schools, and 19% in schools
outside of NYC. The percentage obtaining employment in NYC public schools was 20
8
Fourth Annual Report to TEAC, February 2011
percentage points lower than for the Class of 2007 and attributable to a hiring freeze
imposed by the NYC Department of Education.
The survey asked the graduates to indicate the extent to which the Steinhardt
teacher education program had prepared them with the skills and knowledge to teach.
Their perceptions were retrospective coming 8 – 12 months after graduation. The
responses were selected on a four-point ordinal scale with the categories (4) Very Well,
(3) Moderately Well, (2) Somewhat Well, and (1) Not Well At All. Table 3 displays the
number and percent of Steinhardt graduates that responded Very Well or Moderately
Well to the 15 survey items and compares their responses on 11 of these items to the
responses of a national sample of teacher education graduates from Art Levine’s 2006
study of schools of education.
Overall Ratings:
Overall, the majority of the NYU graduates rated each of the 15 items Very Well
and Moderately Well combined. The highest rated items were Use different pedagogical
approaches (n=123, 92.5%), Understand how students learn (n=117, 86.6%), and
Impact your students’ ability to learn (n=116, 86.1%). More than 85% of the
respondents reported that they were Very Well or Moderately Well prepared with the
skills and knowledge important to teaching on these three items.
In addition, more than 75% of the respondents reported their teacher education
programs prepared them Very/Moderately Well in the following skill areas: Use student
performance assessment techniques (n=111, 82.9 %), Address needs of students from
diverse cultural backgrounds (n=111, 82.3%), Have a mastery of your subject area
(n=107, 79.9%), and Work collaboratively with teachers administrators and other
school personnel (n=102, 75.6%). Conversely, the lowest rated items were Work with
parents (n=56, 42.1%), Address needs of students with limited English proficiency
(n=63, 46.7%), Participate as a stakeholder in the community where you teach (n=76,
56.3%), and Address needs of students with disabilities (n=79, 59%).
Comparison with Levine Sample:
Comparisons of the responses of the NYU graduates to those of the sample of
respondents from the Levine study show that higher percentages of NYU graduates
reported they were Very/Moderately Well prepared than the Levine sample for nine of
the 11 important skill and knowledge areas. The differences reached statistical
significance, as determined by a single-sample difference of proportions Z test, for six
items as follows (Table 6B): Use different pedagogical approaches, 92.5% for NYU
graduates versus 74% for the Levine sample, Z=4.68, p<.05; Implement state/district
curriculum & standards, 76.1% versus 60.0%, respectively, Z=3.32 p<.05; Use student
performance assessment techniques, 82.8% versus 67.0%, respectively, Z=3.54, p<.05;
Address needs of students with limited English proficiency, 46.7% versus 27.0%,
respectively, Z=2.57, p<.05; Address needs of students from diverse cultures, 82.2%
versus 52.0%, respectively, Z=6.43, p<.05; and Integrate technology into the grade level
9
Fourth Annual Report to TEAC, February 2011
or subject taught, 66.7% versus 41.0%, respectively, Z=4.94, p<.05. While comparisons
with published data provide useful benchmarks, it is important that we establish our own
targets to track our progress over time. Discussions with faculty will be held for this
purpose.
Comparison to the Class of 2007 Results:
The program perceptions of the respondents from the Class of 2009 were
substantially more positive than 2007. The perceptions of the Class of 2009 were
significantly higher statistically than the Levine sample in six areas compared to four for
the Class of 2007. The perceptions of the Class of 2009 were more positive than 2007 in
Use different pedagogical approaches, Implement state/district curriculum standards,
and Use student performance assessment techniques. In all three areas, the perceptions
of the Class of 2009 were significantly more positive than the Levine sample while the
perceptions of the Class of 2007 were not. The latter two reflect the impact of faculty
initiatives to upgrade curriculum and methods courses in these areas. There were also
large increases in perceptions of preparation in Identify and use resources within the
community where you teach, Participate as a stakeholder in the community where you
teach, and Integrate technology into the grade level or subject taught. The gains in the
first two areas can be attributed to NYU’s growing emphasis on integrating student
teachers into the life of the school community through school partnerships.
One area that continues to receive low ratings is Work with parents. This topic
will receive continued attention in faculty discussions. In other survey questions,
graduates continued to voice a strong need for mentoring to assist them with induction
into the profession. Only about one-third were receiving mentoring from their
schools/districts and nearly two-thirds welcomed the offer of additional support from
NYU. In response, we have instituted an Early Career Support program that offers
ongoing assistance to NYU graduates. In addition, we are exploring the development of
program extensions that include continued training and mentoring beyond graduation.
10
Fourth Annual Report to TEAC, February 2011
Table 3
Comparison of self perceptions of Class of 2009 Steinhardt teacher education graduates
and Levine study sample as to how well their teacher education programs prepared them
for teaching
Did your program prepare you to:
Use different pedagogical approaches
Have a mastery of your subject area
Maintain order & discipline in the
classroom
Understand how students learn
Impact my students' ability to learn
Implement state/district curriculum &
standards
Use student performance assessment
techniques
Address needs of students with disabilities
Address needs of students with limited
English proficiency
Address needs of students from diverse
cultures
Work with parents
Work collaboratively with teachers,
administrators, & personnel
Identify & use resources within the
community where you teach
Participate as a stakeholder in the
community where you teach
Integrate technology into the grade level or
subject taught
N*
%*
N
%
N
%
N
%
N
%
N
%
N
%
N
%
N
%
N
%
N
%
N
%
N
%
N
%
N
%
NYU
grads
123
92.5
107
79.9
84
62.7
117
86.7
116
86.6
102
76.1
111
82.8
79
59.0
63
46.7
111
82.2
59
42.1
102
75.6
93
68.9
76
56.3
90
66.7
Levine
sample
Z difference
74.0
4.68 **
73.0
1.61
57.0
1.06
81.0
1.58
NA
NA
60.0
3.32 **
67.0
3.54 **
60.0
-1.82
27.0
2.57 **
52.0
6.43 **
43.0
-0.17
NA
NA
NA
NA
NA
NA
41.0
4.94 **
Source: First-Year Feedback Survey of spring 2009 NYU Teacher Education Graduates
NA = Skill/knowledge not included in the Levine study survey
* N and % that responded the program prepared them (4) Well or (3) Moderately to
teach on a four-point scale that included (2) Somewhat Well and (1) Not Well At All
** p < .05
11
Fourth Annual Report to TEAC, February 2011
New York State Teacher Certification Exams (NYSTCE)
Table 4 reports the scores of NYU teacher education graduates for the Class of
2010 on the NYSTCE exams that teacher candidates must pass in order to be eligible to
teach in New York State public schools. The table reports mean scaled scores, effect
sizes, which are defined as the number of standard deviations (SDs) above the passing
score of 220, and pass rates for the three sets of exams required of candidates. The
passing rates remain close to 100% for the Liberal Arts and Sciences Test and exactly
100% for the Assessment of Teaching Skills-Written (ATS-W) tests, similar to the values
reported in the initial Inquiry Brief and third annual report for the Classes of 2004 –
2009. The passing rates for the Content Specialty Tests (CSTs) remained around the
same as those reported in the third annual report, 96% for BS students and 97% for MA
students, which were higher than those in the first Inquiry Brief by around three
percentage points for BS students and four percentage for MA students. In addition, the
effect sizes for the CSTs remained higher than those reported in the Inquiry Brief by
about one-half SD for BS students and one-third SD for MA students. The effect sizes
for the ATS-W also remained higher than those in the first Inquiry Brief by 0.44 SD and
0.55 SD for BS and MA students, respectively. The Steinhardt scores for the Class of
2010 continued to be meaningfully higher than those that were released by the Teacher
Quality Research Consortium (TQRC), as reported in the third annual report. TQRC was
a pilot project funded by the New York State Education Department (NYSED) designed
to link data from the NYSED with data from institutions of higher education (IHE) in
NYS. The CST mean scaled scores of NYU graduates continued to exceed those of all
IHEs combined by 0.40 SD for BS students while they continued to be about equal for
MA students. As reported last year, NYU’s BS students study their content area subject
matter in NYU’s College of Arts and Sciences while MA students take these courses at
their respective undergraduate institutions. Accordingly, these findings continue to
reflect positively on the quality of NYU’s content major courses. Similarly, the
comparative advantage for NYU graduates on the ATS-W is evidence of the quality of
the Steinhardt School’s pedagogical courses and seminars. The improvement in CST
scores coincides with greater coordination and communication between Steinhardt and
the College of Arts and Sciences.
12
Fourth Annual Report to TEAC, February 2011
Table 4
Mean scaled scores, effect sizes, and passing rates for
Steinhardt teacher education graduates on New York
State Teacher Certification Exams (Class of 2010)
Statistic
BS
MA
Total
Liberal Arts & Sciences Test (LAST)
N Tested
85
227
312
Mean Scaled Score (MSS)
269.2 270.7 270.3
Standard Deviation (SD)
18.9
17.5
17.8
SDs Above Passing Score *
2.60
2.90
2.83
Percent Passing **
97.7% 98.7% 98.4%
Assessment of Teaching Skills-Written (ATS-W)
N Tested
88
236
324
Mean Scaled Score (MSS)
271.0 270.4 270.6
Standard Deviation (SD)
14.8
13.9
14.1
SDs Above Passing Score *
3.45
3.63
3.59
Percent Passing **
100% 100% 100%
Content Specialty Tests (CST) ***
N Tested
91
241
332
Mean Scaled Score (MSS)
255.2 254.4 254.6
Standard Deviation (SD)
21.6
19.8
20.3
SDs Above Passing Score *
1.63
1.74
1.70
Percent Passing **
95.6% 96.7% 96.4%
* Effect Size = (MSS - 220)/SD
** Passing score = 220 on a 100 - 300 scale
*** If student has taken multiple tests, data are based on
most recent exam
Student Teacher End-of-Term Feedback Surveys
NYU uses the on-line End-of-Term Feedback Questionnaire (ETFQ) to assess the
quality and effectiveness of each term’s student teaching experience. The survey asks
student teachers to rate the effectiveness of their cooperating teachers and field
supervisors in helping them to develop their expertise in specific aspects of teaching
using a five-point Likert scale ranging from (1) Very Poorly to (5) Very Well. The
program target is a mean score of 4.0 (Well) for each area. For subject matter
knowledge, student teachers are asked how well cooperating teachers and field
supervisors helped them to develop the content knowledge specific to their field and age
group; for pedagogical knowledge they were asked how well their mentors provided
13
Fourth Annual Report to TEAC, February 2011
assistance in furthering their organizational teaching skills; and for teaching skill and
caring they were asked about the assistance they received in enhancing their teaching
practice and developing classroom management skills.
Table 5 displays the mean ETFQ scores and effect sizes for the items related to
each of the three claims for the Classes of 2009 and 2010. The effect sizes represent the
difference between the criterion of 4.0 and the mean scores in standard deviation units.
The total means for 2009 and 2010 combined for all three claims meet or exceed the
criterion of 4.0 for both BS and MA students. The third annual report found that all three
means for MA students were higher than the respective values reported in the initial
Inquiry Brief for the classes of 2004 and 2005, which were below 4.0 for two of the three
claims, subject matter knowledge and teaching skill and caring. The means for MA
students in the Classes of 2009 and 2010 continue to be higher than those in the first
Inquiry Brief and show a general increasing trend over the decade.
14
Fourth Annual Report to TEAC, February 2011
Table 5
Mean scores for claim scales on the End of Term
Feedback Questionnaire
Classes of 2009 and 2010
Class of 2009
BS
MA
Class of 2010
BS
Totals
MA
BS
MA
Claim 1 Scale: Subject Matter Knowledge
N
Respondents
Mean Score
42
59
89
158
131
217
3.95
3.94
4.18
4.07
4.11
4.03
Standard
Deviation
(SD)
0.80
1.13
0.86
0.91
0.84
0.97
Effect Size
(ES)
-0.06
-0.05
0.21
0.08
0.13
0.03
Claim 2 Scale: Pedagogical Knowledge
N
Respondents
Mean Score
42
59
89
158
131
217
4.18
4.13
4.29
4.27
4.26
4.23
Standard
Deviation
(SD)
0.61
0.91
0.72
0.78
0.69
0.82
Effect Size
(ES)
0.29
0.14
0.40
0.34
0.38
0.28
Claim 3 Scale: Teaching Skill and Caring
N
Respondents
Mean Score
42
59
89
158
131
217
4.21
4.11
4.27
4.25
4.25
4.22
Standard
Deviation
(SD)
0.63
094
0.78
0.76
0.74
0.81
Effect Size
(ES)
0.33
0.12
0.34
0.33
0.34
0.27
Notes: Claim 1 scale items are Items 9 and 18; Claim 2 scale items are Items 7 and 15; and Claim 3 scale
items are Items 8, 11, 16, and 19. Items are measured on a 5-point Likert scale with scale values of (1)
“Very Poorly”, (2) “Poorly”, (3) “Average”, (4) “Well”, and (5) “Very Well”. The criterion for each claim is a
mean score of at least 4.0. The formula for effect size is as follows:
ES = (Mean Score – 4.0)/SD. Students left some items blank, resulting in different numbers of respondents
for the claim scales.
15
Fourth Annual Report to TEAC, February 2011
Grade Point Averages (GPAs)
Among the measures that NYU uses as evidence to assess its claims are GPAs for
subsets of courses related to each of the claims. The initial Inquiry Brief included
separate GPAs for the content-core courses to assess subject matter knowledge,
pedagogical-core courses to assess pedagogical knowledge, student teaching and student
teaching seminars to assess teaching skill and caring, and Morse Academic Plan courses
to assess the cross-cutting theme of liberal arts knowledge. This annual report includes
total mean GPAs for BS and MA students for the graduates of the classes of 2010 (see
Table 6.) Mean GPAs for this class were 3.51 (SD = 0.26) for BS students and 3.85 (SD
= 0.18) for MA students. The mean GPA for BS students is the about the same as the
total GPAs reported for BS and MA students in the Classes of 2006 – 2009 combined.
GPAs for the separate claims will be reported in the next Inquiry Brief.
Table
Mean total GPA for Steinhardt BS
and MA teacher education graduates:
Class of 2010
Mean
N
Std. Deviation
BS MA Total
3.51 3.85 3.77
113 377
490
.26
.18
.25
USE OF DATA FOR PROGRAM IMPROVEMENT
Steinhardt deans, chairs, program directors, and faculty continue to make
increased use of the performance data to inform program planning and improvement. As
described above, data are analyzed and reviewed at meetings of the Teacher Education
Council, the Teacher Education Working Group, and meetings of the teacher education
program directors. They are also reviewed by the Office of Clinical Studies to assess the
effectiveness of field services, including the quality of the environment for pre-service
training at specific school sites and the ability of individual cooperating teachers and field
supervisors to provide high-quality mentoring to student teachers. Overall trends in the
data are analyzed as well as disaggregated outcomes for specific certification areas.
Where possible, data are analyzed longitudinally and historically to identify trends over
time and comparisons are made to program standards and published data from other
institutions of higher education, as well as available normative data.
16
Fourth Annual Report to TEAC, February 2011
There has been a cultural shift in the use of data for program decision-making by
the Department of Teaching and Learning. Review of the data from the evidence base
has become a regular part of the monthly program directors’ meetings. The data are
disaggregated by program and used in a needs-assessment process to identify areas for
institutional reform. Reciprocally, perceptions of problems in curricula and field work
are validated and illuminated by analyses of the data, often leading to revisions and
expansion of data collection and analysis. The development of rapid feedback systems
has increased our capacity to identify and deal with problems quickly. This includes
quality-control systems that process data from course evaluations and student teacher
feedback questionnaires in ways that allow for rapid identification of problems and quick
corrective actions.
The third annual report documented a number of enhancements to Steinhardt’s
teacher education programs that were informed by evidence from the teacher education
database. This year, we have supplemented the quantitative database with information
from intensive qualitative studies. Dean Pat Wasley from the University of Washington
conducted an external review of the Department of Teaching and Learning’s programs
aimed at identifying areas of strength and weaknesses to guide programmatic
improvements. Dean Wasley presented her preliminary findings to the Department’s
cabinet and faculty at meetings during November and December 2010. In addition,
Professor Joe McDonald organized a “shadowing study”, in which Teaching and
Learning doctoral students paired up with teacher education students and documented the
student’s daily program experiences by attending their classes and clinical field
placements. The doctoral students will prepare reports that will be reviewed by faculty to
provide rich qualitative data to augment the evidence base informing program
improvements.
TEACHER EDUCATION RESEARCH BIBILIOGRAPHY
The third annual report included a list of research reports and papers on teacher
education prepared by NYU faculty and research centers. An updated list is presented
below.
Alter, M., Rust, F., Jeffery, J. & Keane, A. Closing the teacher preparation gap:
gathering evidence of student learning. A paper funded through a Petrie Foundation
grant, July 2008.
Hummel-Rossi, B., Tobias, R., & Ashdown, J. Creating usable to improve teacher
education programs serving urban public schools. A paper delivered at the annual
meeting of the AERA, San Diego, April 2009. CRTL research report series RR-0409-1
Hummel-Rossi, B. Tobias, R., Ashdown, J., & Smith, A. Teacher education’s
responsibility to its metropolitan constituents: A longitudinal value-added study. A paper
presented at the annual meeting of AERA, New York City, March 2008. CRTL research
report series RR-0308-1.
17
Fourth Annual Report to TEAC, February 2011
Jeffery, J. & Tobias, R. Circle of inquiry: partnership researchers’ perspectives on
school-university collaborative processes. A paper delivered at the annual meeting of the
AERA, San Diego, April 2009. CRTL research report series RR-0409-1.
Meier, J. & Crowe. Evaluation of partnership for teacher excellence. An evaluation
report prepared by Arete Consulting for the Petrie Foundation, April 2009.
Sirin, S. & Collins, B. Graduate teacher education at New York University: An
exploration of backgrounds, perceptions, and expectations. New York University: New
York, 2009.
Taub, A., Tobias, R. & Mayher, J. Inquiry brief: a self-study of NYU’s teacher education
programs. A report to TEAC for initial accreditation, November 2005.
Tian, J. & Tobias, R. Student evaluations of co-taught, off-site inquiries courses: fall
2009. CRTL program evaluation series report PE-2010-01.
Tobias, R. & Tian, J. Program exit survey of May 2010 NYU teacher education
graduates. CRTL program evaluation series report PE-1110-01, November 2010.
Tobias, R., Tian, J. & Saad, B. One-year follow survey of Steinhardt teacher education
graduates: Class of 2009. CRTL program evaluation series report PE-0410-01, April
2010.
Tobias, R., Hummel-Rossi, B., Ashdown, J., Simic, O. & Woo, K. Accountability in
teacher education: ecological analyses of VAM effects. A paper presented at the annual
meeting of the AERA, Denver, CO, May 2010. CRTL research report series RR-041001, April 2010.
Tobias, R. & Tian, J. Graduating students’ perceptions of the effectiveness of their
teacher education programs: Class of 2009. CRTL program evaluation series report PE0909-01, September 2009.
Tobias, R. & Tian, J. & Woo, K. First-year feedback survey of spring 2008 NYU teacher
education graduates. CRTL research report series RR-0909-1, September 2009.
Tobias, R., Woo, K. & Pignatosi, F. Are we developing high quality teachers and can we
prove it? A study of the validity and utility of a pre-service teacher assessment system.
A paper presented at the annual meeting of the AERA, San Diego, April 2009. CRTL
research report series RR-0409-2.
Tobias, R. & Bang, H. First-year feedback survey of spring 2007 NYU teacher education
graduates. CRTL research report series RR-0608-1, June 2008.
18
Fourth Annual Report to TEAC, February 2011
Tobias R., Fan, B., & Bang, H. Measuring the developing dispositions of pre-service and
beginning teachers. A paper presented at the annual meeting of AERA, New York City,
March 2008. CRTL research report series RR-0308-2.
Tobias, R. & Robin, J. The effects of multiple-perspective assessment upon student
teachers and their pupils. Final report for TNE mini-grant 3571-00/EOC 6820. CRTL
research report series RR-0707-1, July 2007.
Tobias, R. & Barrett, T. Graduating students’ perceptions of the effectiveness of their
teacher preparation programs: Steinhardt Graduating class of May 2006. CRTL research
paper series RR-0307-1, March 2007.
Tobias R. & Robin, J. Analysis of the Fast Track Learning Partners Survey, Summer
2007, CRTL research report series PE-0307-1, March 2007.
Tobias, R. Results of the first follow-up study on graduates of NYU’s Steinhardt School
of Education’s teacher education programs: graduates from the Classes of 2001 – January
2005 teaching in New York State. CRTL research report series RR-0705-01, July 2005.
Tobias, R. Analysis of student course reaction forms for Department of Teaching and
Learning courses during the period spring 2003 – summer 2004. CRTL research report
series RR-0904-1, September 2004.
Tobias, R. & Fan, B. Fast track evaluation report: analysis of student perceptions of the
effectiveness of the Fast Track summer session 2003. CRTL research report series PE1203-01, December 2003
19
Fourth Annual Report to TEAC, January 2011
________________________________________________________________________________________________
Appendix A
NYU Steinhardt’s Fourth Annual TEAC Report: Appendix E (TEAC Inquiry Brief): Evidence
January 2011
Type of Evidence
Note: items under each category
are examples. Program may
have more or different evidence
Available*
In the Brief
Reasons for including the results
in the Brief
(Location in Brief)
Not Available
Not in the Brief
Not for future use
For future use
Reasons for not
including the results in
Reasons for including in future Briefs Reasons for not including in future Briefs
the Brief
Grades
1. Student grades and grade
point averages
Content area GPA, pedagogical
course GPA, and student
teaching GPA continue to be
used as valid and reliable
measures of all three claims.
(File 1 in Appendix B.)**
Scores on standardized tests
2. Student scores on
standardized license or board
examinations
Scaled scores on the NYSTCE
content area and Assessment of
Teaching Skills exams continue
to be used as valid, reliable, and
sensitive measures of content
knowledge and pedagogical
knowledge, while scaled scores
on the Liberal Arts and Sciences
Test continue to be used to assess
the cross-cutting theme of
general academic knowledge.
(File 5 in Appendix B.)**
20
Fourth Annual Report to TEAC, January 2011
________________________________________________________________________________________________
3. Student scores on admission
tests of subject matter
knowledge for graduate study
Standardized admissions exams,
such as the SAT and GRE are
optional for admission to the
graduate teacher education program.
To the extent that these scores are
available, they will be reported as
ancillary data in future briefs.
4.Standardized scores and gains
of the program graduates own
students
Standardized test score gains for the
pupils of program graduates were not
available for NYU’s initial Inquiry
Brief. However, these data are
currently being collected and
analyzed for the pupils of program
graduates who are teaching in grades
4 thru 8 in New York City public
schools (the majority of program
graduates) and will be presented in
the next Inquiry Brief using valueadded analytic methods. NYU is
exploring the possibility of obtaining
and using standardized test scores for
the pupils of graduates who are
teaching high school students. The
data file was constructed in
December, 2008 and includes the
value-added test data of 150
graduates. It was augmented with
the data for 190 graduates in 2009.
(File 12 in Appendix B.) **
21
Fourth Annual Report to TEAC, January 2011
________________________________________________________________________________________________
Ratings
5. Ratings of portfolios of
academic and clinical
accomplishment
Portfolio data were not
included in the original
Brief and will not be
used in future Briefs.
Attempts to develop a
standard rubric to score
portfolios proved
unsuccessful due to
insufficient inter-rater
agreement.
6. Third-party rating of
program's students
7. Ratings of in-service,
clinical, and PDS teaching
NYU considered using third-party ratings
of program students but determined the
procedures to be not feasible logistically.
NYU did not have access to inservice ratings of program graduates
for the first Brief. We are in
discussion with the NYC Department
of Education (NYCDOE) to try to
obtain these ratings for future Briefs.
The success of these efforts will be
largely contingent upon the adoption
of new teacher performance
standards and rating systems by the
NYCDOE, which is currently being
negotiated.
22
Fourth Annual Report to TEAC, January 2011
________________________________________________________________________________________________
8. Ratings by cooperating
NYU continues to expand the
teacher and college/
use of the DRSTOS-R, a protocol
university supervisors, of
practice teachers' work samples used to assess the developing
proficiency of student teachers.
Data from the assessment of a
sample of DRSTOS-R ratings by
student-teacher supervisors were
submitted in the first Brief. Data
for a much larger sample will be
included in future Briefs. Plans
for augmenting the data with the
DRSTOS-R ratings of
cooperating teachers are
contingent upon NYCDOE
decisions about the adoption of
new performance standards and
assessments for teachers. (See
Item 7 above.) (File 2 in
Appendix B.)**
Rates
9. Rates of completion of
courses and program
These data were not included in the
initial Brief because the teacher
education programs were in
transition to conform to new state
certification requirements and
historical completion data were
deemed obsolete. Program
completion rates will be included in
the next Inquiry Brief. (File is under
construction.)
23
Fourth Annual Report to TEAC, January 2011
________________________________________________________________________________________________
10. Graduates' career retention
rates
11. Graduates' job placement
rates
12. Rates of graduates'
professional advanced study
13. Rates of graduates'
leadership roles
Some data on career retention
rates were included in the initial
Brief. NYU has expanded its
capability for tracking career data
for program graduates through
partnerships with the New York
State and New York City DOEs.
Accordingly, the next Inquiry
Brief will include richer and
more complete career retention
data. (File 9 in Appendix B.)**
Same as Item 10 above.
Since submission of the initial Brief,
NYU has initiated a follow-up study
to assess the professional success and
career progress of its teacher
education graduates. The follow-up
study includes a survey of graduates
who are working in NYC public
schools, the primary employer of
NYU graduates. These data will be
analyzed and reported in the next
Inquiry Brief.
See Item 12 above.
14. Rates of graduates'
professional service activities
See Item 12 above.
24
Fourth Annual Report to TEAC, January 2011
________________________________________________________________________________________________
Case studies and alumni competence
15. Evaluations of graduates by
their own pupils
NYU believes that the questionable
reliability and validity of these data render
the high resource expenditures required to
collect them unwarranted.
16. Alumni self-assessment of
their accomplishments
See Item 12 above.
17. Third-party professional
recognition of graduates (e.g.
NPTS)
See Item 12 above.
18. Employers' evaluations of
the program's graduates
See Item 7 above.
19. Graduates' authoring of
textbooks, curriculum materials,
etc.
See Item 12 above.
20. Graduates own pupils'
learning and accomplishment
See Item 4 above.
Other Data
21. Students’ self-ratings of
growth in pedagogical
knowledge and teaching skills.
NYU continues to collect,
analyze and report data from the
ETFQ, a questionnaire
administered to student teachers
at the end of each field
placement. The ETFQ asks
students to assess the extent to
which the field experience has
contributed to their professional
growth as teachers. The
questionnaire is being updated to
provide more valid data on
students’ professional growth.
Data from the updated ETFQ will
be reported in future Briefs. (File
7 in Appendix B.)**
25
Fourth Annual Report to TEAC, January 2011
________________________________________________________________________________________________
22. Students’ dispositions to
teaching.
23. Graduates ratings of the
effectiveness of NYU’s teacher
education programs.
Data from the Educational
Beliefs Questionnaire (EBQ)
were reported in the initial Brief
and continue to be collected.
CRTL has replaced the EBQ with
a new dispositions survey that it
designed and pilot tested in 2009.
The new survey is entitled the
Educational Beliefs and
Multicultural Awareness Survey
(EBMAS). The new survey
assesses more constructs than the
EBQ and has stronger
psychometric properties. The
next Inquiry Brief will contain
EBMAS data for the Classes of
2009 and 2010. (File 4 in
Appendix B.) **
The initial Brief reported results from
a survey of Fast Track graduate
students at the end of the program’s
summer component. NYU has
replaced this survey with two surveys
of program graduates: an exit survey
and a one-year follow-up survey.
These surveys assess the extent to
which graduates feel that the
program has prepared them to be
successful teachers. NYU faculty
believes the graduates are better
positioned to provide useful
information about the effectiveness
of the programs than the mid-course
Fast-Track students. Some results
from these surveys were used in the
Third Annual Report and will be
reported in more detail in the next
Inquiry Brief. (Files 8 & 10 in
Appendix B.)**
26
Fourth Annual Report to TEAC, January 2011
________________________________________________________________________________________________
** File numbers refer to the data files listed in Appendix B, Summary of CRTL Data Files.
Appendix B
NYU Steinhardt’s Fourth Annual Report: Summary of CRTL Data Files
January 2011
Data File
1. Teacher Education
graduate file
2. Domain-Referenced
Student Teacher
Observation Scale
(DRSTOS-R)
3. Educational Beliefs
Questionnaire (EBQ)
Description
Extent of Data
Descriptive data for all graduates of Steinhardt teacher
education programs, including date of graduation, degree,
certification area, GPA, identification data, and contact
information. This file can be linked to a course transcript
file.
Scores from this 21-item protocol for rating the teaching
performance of student teachers, which was adapted by
CRTL from the work of Charlotte Danielson. The
DRSTOS-R is designed to assess the ability of teacher
education students to understand and integrate their
classroom experiences and apply that learning to their own
practice as student teachers.
An assessment tool that provides insight into the beliefs
held by our pre-service teachers at various points in their
programs. Students respond to the survey’s 26 items using a
6-point Likert scale. The EBQ yields two scores: Teaching
Efficacy and Caring. The former is defined as belief in the
effectiveness of teaching to promote the learning of students
from diverse backgrounds while the latter taps concern for
the education of all children.
The database contains information on all
teacher education graduates from the classes
of 2001 – 20010.
The database contains the ratings of more
than 1,900 student teachers from the fall
2004 through the spring 2010 semesters.
The student teachers vary by degree and
certification programs, as well as early and
late placements.
The database contains the ratings of
approximately 1,600 students from the fall
2004 through the spring 2008 semesters.
The students vary by degree, certification
programs, and total credits earned at time of
administration. Many of the assessments
are longitudinally linked to permit the
assessment of within-person trends over
time.
27
Fourth Annual Report to TEAC, January 2011
________________________________________________________________________________________________
4. Educational Beliefs
and Multicultural
Awareness Survey
(EBMAS)
5. New York State
Teacher Certification
Exam Scores
(NYSTCE)
6. Student Course
Reaction Forms
(SCRF)
7. End of Term Student
Teacher Feedback
Questionnaires
(ETFQ)
EBMAS replaced the EBQ as a measure of teacher
education students’ dispositions toward teaching in 2009.
Students respond to 39 items using a 6-point Likert scale
along the dimension of agreement with dispositional
statements. EBMAS yields three scores: General Teacher
Efficacy (GTE), Personal Teacher Efficacy, and
Multicultural Awareness (MCA). It is administered at the
beginning and end of NYU studies for BS and MA students
and also in the junior year for BS students.
In order to receive New York State certification as a
teacher, graduates must pass the examinations administered
through the NYSTCE program. Tests include the Liberal
Arts and Science Test (LAST), the Assessment of Teaching
Skills-Written (ATS-W) and the Content Specialty Test for
the Core subjects they teach.
CRTL processes student course evaluations and maintains
an historical database of the quantitative data.
This is a questionnaire that was developed to elicit feedback
from teacher education students concerning the extent to
which they perceive that the student teaching experience
enhanced their professional knowledge and expertise. The
format includes a combination of forced-choice and openend items divided into three parts: feedback on the school,
cooperating teacher, and the supervisor. This questionnaire
is currently being revised.
The database contains the surveys of 683
students from the spring 2009 through the
fall 2010 semesters. The students vary by
degree, certification programs, and total
credits earned at time of administration.
The database contains the NYSTCE scores
for all NYU students who have taken the
exams from April 2005 through the most
recent test administration in June, 2010.
There are a total of nearly 10,000 exam
scores for approximately 4,800 students.
Data are maintained for all T&L courses
that submitted SCRFs beginning in fall 2002
through spring 2010. The database also has
course evaluation data for other Steinhardt
departments for spring 2007 through spring
2010. Data are available at both the course
level and student level.
The database contains the ratings of nearly
than 5,000 student teachers from the fall
2002 through the spring 2010 semesters.
The student teachers vary by degree and
certification programs, as well as early and
late placements
28
Fourth Annual Report to TEAC, January 2011
________________________________________________________________________________________________
8. End of Program Exit
Questionnaire
This is an on-line exit survey that is periodically
administered to all teacher education program graduates.
Questions focus on the perceptions of graduates as to the
extent to which the programs have prepared them to begin
teaching, the most and least effective of the program, and
their plans for the future.
The survey was administered in May 2006,
May 2008, and May 2009, and May 2010.
9. State/City Information
System Follow-Up
Tracking Data
CRTL matches files of graduates to state and city human
resources files to locate the schools in which graduates are
teaching. The matched file contains all of the variables
from the graduate file (1 above), as well as the following
information from the teacher information systems: school,
including county and district, number of years teaching,
number of years in same school, subjects taught and grades
taught. This file can also be linked to school report cards,
school progress reports, and learning environment survey
files to pick up school demographic and performance data.
10. One-year Graduate
Follow-Up Survey
Data from a follow-up survey that was sent to all of the
teacher education graduates from the Classes of 2007 –
2009. The survey asks graduates their current employment
status, the extent to which the NYU programs prepared
them for teaching, and the most and least effective aspects
of the teacher education programs.
CRTL has two separate follow-up files. The
first contains data on approximately 1,000
graduates from the classes of 2001 –
January 2005 who were employed in
schools in New York State, Connecticut or
Florida as of October 2004. The second has
approximately 1,500 graduates from the
classes of 2001 – May 2006 who were
employed in New York City schools as of
spring 2007; this file was updated for 2008
graduates in December 2008 and will be
updated again in January 2011 with
graduates from the Classes of 2009 and
2010..
Separate SPSS data files with survey data
for all respondents have been constructed
for each of the three survey administrations.
29
Fourth Annual Report to TEAC, January 2011
________________________________________________________________________________________________
11. Partnership Survey
Data files
This file contains data from several surveys administered in
connection with the evaluation of the Partnership Initiative.
Data for the following surveys are available:
• Summer 2006 Learning Partners
• Fall 2006 Learning Partners
• Summer 2007 Learning Partners
• Summer 2008 Learning Partners
• Fall 2006 Student Teachers
• Fall 2006 Host School Liaisons
• Fall 2006 NYU Liaisons
• Fall 2007Student Teachers
• Fall 2007Host School Liaisons
• Fall 2007NYU Liaisons
These data are no longer collected since the
Partnership Initiative has ended.
12. Graduate value-added
pupil achievement test
score file
Value-added modeling (VAM) effect scores for graduates
who teach English language arts (ELA) or math to students
in grades 4 – 8 in the New York City public schools. The
scores represent the mean difference between the actual and
predicted state standardized test scores of the pupils taught
by the graduates. The predicted scores are based on a fourlevel HLM model that controls for characteristics of
students, classes, teachers, and schools.
The file includes data for more than 250
Steinhardt teacher education graduates from
the Classes of 2006 – 2008.
30