Evaluation of the Professional Development Lab (PDL) Programs in
Community School District 20
2002 – 2003
CENTER FOR RESEARCH ON TEACHING AND LEARNING
Department of Teaching and Learning
82 Washington Square East, Suite 700
New York, NY 10003 | 212 998 5872 | 212 995 3636 fax
www.steinhardt.nyu.edu/teachlearn/crtl
Evaluation of the PDL Programs in Community School District 20
Evaluation of the Professional Development Lab (PDL) Programs in
Community School District 20
2002 – 2003
Robert J. Tobias, Director
Rosa L. Rivera-McCutchen, Graduate Assistant
CRTL External Evaluation Report Series
EE-0304-01
March 2004
Center for Research on Teaching and Learning
Department of Teaching and Learning
Steinhardt School of Culture, Education, and Human Development
New York University
© Copyright 2004 by the Center for Research on Teaching and Learning
CRTL Research Report Series, EE-0304-01
i
Evaluation of the PDL Programs in Community School District 20
ACKNOWLEDGEMENTS
The Center for Research on Teaching and Learning would like to thank a number of individuals
who were instrumental in the research and writing phases of this report. Ognjen Simic, Research
Scientist, conducted statistical analyses of the data. Laura Carroll and Emily Hole, graduate
student workers, assisted with data collection and entry. Alexandra Snyder, also a graduate
student worker, compiled the school profiles. Beth McDonald, Research Scientist, reviewed
early drafts of the report and provided valuable feedback.
Finally, this report would not have been possible without the cooperation of the PDL staff,
consultants and participants, including the New Teacher Facilitators and Resident Teachers.
CRTL Research Report Series, EE-0304-01
ii
Evaluation of the PDL Programs in Community School District 20
TABLE OF CONTENTS
List of Figures and Tables ........................................................................................................... v
Executive Summary ..................................................................................................................... vi
I. Introduction ............................................................................................................................. 1
II. The Programs .......................................................................................................................... 2
The Professional Development Laboratory (PDL) ............................................................... 2
PDL’s Programs in CSD 20................................................................................................... 2
PDL for New Teachers .................................................................................................... 2
PDL for Middle School Social Studies Teachers ............................................................ 6
Learning Through Practice Leadership Program............................................................. 7
Context................................................................................................................................... 8
III. The Evaluation ....................................................................................................................... 9
The Evaluators ....................................................................................................................... 9
Evaluation Questions ............................................................................................................. 9
Methods of Data Collection ................................................................................................... 10
New Teacher Staff Training Observations ...................................................................... 10
NTF and RT Questionnaires and Interviews ................................................................... 10
New Teacher Surveys ...................................................................................................... 11
Individual Performance Assessment Logs (IPAL’s) ....................................................... 11
Teacher Efficacy Scale .................................................................................................... 12
New Teacher Classroom Observations ........................................................................... 12
Social Studies Institute Feedback Questionnaire............................................................. 13
Teacher Retention Data.................................................................................................... 13
Standardized Achievement Test Scores........................................................................... 13
Assumptions and Limitations ................................................................................................ 14
IV. Findings ................................................................................................................................. 16
New Teacher Participants ...................................................................................................... 16
Social Studies Teacher Participants ....................................................................................... 17
Evaluation Question 1............................................................................................................ 18
Evaluation Question 2. .......................................................................................................... 19
PDAQ .............................................................................................................................. 19
IPAL’s .............................................................................................................................. 21
Interview Data on the Cycle Component......................................................................... 23
Evaluation Question 3A......................................................................................................... 23
Evaluation Question 3B ......................................................................................................... 27
Evaluation Question 4A......................................................................................................... 28
Evaluation Question 4B ......................................................................................................... 30
Evaluation Question 5A......................................................................................................... 31
Evaluation Question 5B ......................................................................................................... 32
Evaluation Question 6............................................................................................................ 33
Evaluation Question 7A......................................................................................................... 35
Evaluation Question 7B ......................................................................................................... 37
V. Conclusions and Recommendations ....................................................................................... 40
Conclusions............................................................................................................................ 40
CRTL Research Report Series, EE-0304-01
iii
Evaluation of the PDL Programs in Community School District 20
Evaluation Question 1...................................................................................................... 40
Evaluation Question 2. .................................................................................................... 40
Evaluation Question 3A................................................................................................... 41
Evaluation Question 3B ................................................................................................... 41
Evaluation Question 4A................................................................................................... 42
Evaluation Question 4B ................................................................................................... 42
Evaluation Question 5A................................................................................................... 42
Evaluation Question 5B ................................................................................................... 42
Evaluation Question 6...................................................................................................... 43
Evaluation Question 7A................................................................................................... 43
Evaluation Question 7B ................................................................................................... 44
Recommendations.................................................................................................................. 44
PDL for New Teachers and LTPLP................................................................................. 44
PDL for Middle School Social Studies Teachers ............................................................ 46
References .................................................................................................................................... 47
CRTL Research Report Series, EE-0304-01
iv
Evaluation of the PDL Programs in Community School District 20
LIST OF TABLES AND FIGURES
Figure 1. PDL for New Teachers Program Model....................................................................... 4
Figure 2. Three-year Implementation of NTF & TLI Components............................................. 5
Figure 3. Social Studies Institute Program Model ...................................................................... 7
Table 1. PDL for Middle School Social Studies Teachers Three-Year Model ........................... 8
Table 2. Distribution of New Teacher Program Interns by School ............................................. 16
Table 3. Descriptive Characteristics of New Teacher Program Interns ...................................... 17
Table 4. Subjects Taught by New Teacher Interns ..................................................................... 17
Table 5. Distribution of Social Studies Program Participants by School .................................... 18
Table 6. New Teachers’ Importance Ratings and Reports of PDL Program Support
on the Professional Development Activities Questionnaire .................................................. 21
Table 7. Mean Self Ratings on the Items of the Elements of Effective Classrooms
and Instruction ....................................................................................................................... 26
Table 8. Correlations Between New Teachers’ Self-Reported Degree of Professional
Skill and the Extent of Program Mentoring in Four Essential Areas of Effective
Classroom Teachers ............................................................................................................... 27
Table 9. Mean Self Ratings on the Teacher Efficacy Scale (New Teachers) .............................. 29
Table 10. Mean Self Ratings on the Teacher Efficacy Scale Pre-Test (S.S. Teachers)............... 30
Table 11. Mean Observation Ratings in the Domain Referenced Teacher Observation
Tool ........................................................................................................................................ 31
Table 12. Three-year Comparison of the Retention Rates of New Teachers in PDL
Cohort Schools and Comparison Schools.............................................................................. 34
Table 13. PDL for New Teachers: Means and SD for Pre and Post ELA Scale Scores
of Students of PDL-Served and Non-PDL Served Teachers ................................................ 36
Table 14. PDL for New Teachers: Summary of T-Tests for the Significance of the
Difference in ELA Gain Scores for Students of PDL Served and Non-PDL
Served Teachers ..................................................................................................................... 37
Table 15. PDL for Middle School Social Studies Teachers: Means and SD’s for
Social Studies and ELA Scale Scores of Students of PDL-Served and Non-PDL
Served Teachers ..................................................................................................................... 38
Table 16. PDL for Middle School Social Studies Teachers: Summary of Stepwise
Multiple Regression Analysis of the Effects of PDL on Grade 8 Students
Social Studies Scale Scores Controlling for 2002 ELA Scale Scores .................................. 39
CRTL Research Report Series, EE-0304-01
v
Evaluation of the PDL Programs in Community School District 20
EXECUTIVE SUMMARY
INTRODUCTION
This report presents the findings of an evaluation of the Professional Development
Laboratory (PDL) programs in New York City’s Community School District (CSD) 20 during
the 2002-2003 academic year. Two of these programs were designed to provide support for the
professional development of two groups of middle school teachers in the district: first-year
teachers who required mentoring by NYS state law and social studies teachers who wished to
enhance their instructional skills. A third program was designed to provide professional
development in the area of leadership for staff developers and exemplary teachers.
The evaluation was conducted by the Center for Research on Teaching and Learning
(CRTL) of the Department of Teaching and Learning in New York University’s Steinhardt
School of Education. The purpose of the evaluation was to document the implementation of the
PDL programs for middle school teachers in CSD 20 and to assess its impact upon four groups
of participants as follows:
•
•
•
•
First, the impact of PDL training upon the mentoring capacities of school-based
professional-development staff in CSD 20 middle schools;
Second, the effects of the program upon the professional competence, sense of efficacy,
and retention in the profession of new middle school teachers;
Third, the effects of the program upon the instructional skills and sense of efficacy of
selected middle school social studies teachers; and
Last, the indirect effects of the program upon the academic achievement of students
served by the programs’ teacher participants
THE PROGRAMS
PDL
PDL was instituted in 1989 and currently functions under the aegis of the Department of
Teaching and Learning’s Ruth Horowitz Center for Teacher Development at New York
University’s Steinhardt School of Education. Over the past fourteen years, PDL has
implemented a variety of professional development and training programs for teachers and
school administrators in 18 New York City school districts. At the heart of the PDL philosophy
is the belief that by opening classroom doors, teachers have the opportunity to reflect on their
teaching practice and analyze student work to develop instructional approaches, all while
collaborating with one another. The PDL model centers on the belief teachers learn best to
improve their practice when the learning takes place in actual classrooms during the school day.
PDL designed and implemented the three programs in CSD 20 that are described below.
CRTL Research Report Series, EE-0304-01
vi
Evaluation of the PDL Programs in Community School District 20
PDL for New Teachers Program
This program was a three-year induction model for first-year teachers that was designed
to build their professional competency in planning lessons, analyzing student academic growth
and student work, delivering appropriate instructional strategies, and using cooperative learning
strategies. The program also aimed to develop the capacity of CSD 20 to provide high-quality
mentoring for new teachers. Toward that end, PDL provided various training opportunities for
New Teacher Facilitators (NTF’s) and Resident Teachers (RT’s) who, in turn, mentored new
teachers. Another component of this was the PDL Cycle Program. The goal of this component
was to provide the new teachers with opportunities to improve their pedagogical skills by
participating in focused visits to RT classrooms. Additionally, new teachers were encouraged to
enroll in one of four Teacher Leadership Institutes, courses that were designed by PDL to assist
new teachers in developing effective teaching strategies and leadership capacity.
PDL for Middle School Social Studies Program
This program was designed to help CSD 20 social studies teachers develop professional
competency in planning lessons, understanding student development, analyzing student work,
developing appropriate instructional strategies, and integrating technology in the social studies
classroom. To achieve its goals, the program provided a series of training institutes for 20
middle school social studies teachers.
Learning Through Practice Leadership Program (LTPLP)
This program was designed to develop the leadership potential of selected district
teachers and staff developers whose classrooms exemplified standards-based teaching and
learning and best practices. All participants also played a role in the PDL for New Teachers
program. LTPLP consisted of seven full-day training sessions that focused on various leadership
strategies, differentiated instruction, multiple intelligences, using protocols, observation
techniques, and using data to guide instruction.
THE EVALUATION
A CRTL research team conducted the evaluation during the period September 2002
through June 2003. The evaluation used a variety of methods to collect qualitative and
quantitative data on the effects of the programs, including interviews, observations, self-report
questionnaires, affective surveys, internal program records, human resources data, and student
achievement test scores. Specific data collection methods and instruments included the
following:
•
•
NTF and RT Questionnaires and Interviews assessed the impact of PDL training they
received on their leadership skills and on their work with new teachers;
Participant Information Form collected basic demographic information from new
teachers;
CRTL Research Report Series, EE-0304-01
vii
Evaluation of the PDL Programs in Community School District 20
•
•
•
•
•
•
Professional Development Activities Questionnaire (PDAQ) asked teachers to
identify the support they received through the program in a number of professional
development areas, and to rate the level of importance of each;
Elements of Effective Classrooms and Instruction (EECI) asked new teachers to rate
their level of development as a teacher in twenty-seven areas, and the extent to which
they had received mentoring in those areas;
Teacher Efficacy Scale (TES) measured the teachers’ sense of professional efficacy
and its relationship to teacher behavior and student achievement;
Domain Referenced Teacher Observation (DRTO) protocol recorded ratings from
observations and post-observation conferences to measure the new and social studies
teachers’ proficiency in each of four domains of practice, including Planning and
Preparation, Classroom Environment, Professional Responsibility, and Classroom
Instruction;
Social Studies Feedback Questionnaire assessed social studies teachers’ ratings of
program professional development in seven areas that were identified in the
program’s literature, and the extent to which the training influenced their practice;
Teacher Retention data obtained from the NYC DOE.
Contextual Caveat
The evaluation was conducted in the context of the ongoing implementation of the
program in a school system that was undergoing major restructuring. Throughout the year, CSD
20 was in transformation as it prepared for the transition from a semi-independent school district
to incorporation in a larger region. The reorganization had an impact on the district and school
climate, leaving many personnel at the district and school level unsure of their job security. This
uncertainty was exacerbated by the announcement in February 2003 that CSD 20’s
superintendent would be resigning his position in the spring. The reorganization of the DOE also
led the district to rely heavily on NTF’s to fulfill additional administrative responsibilities which
were beyond the scope of their mentoring roles. These factors no doubt affected the
implementation of the program and its effects upon the participants in ways that could not be
measured.
FINDINGS AND CONCLUSIONS
The data collected in this evaluation were analyzed to address six questions concerning
the effects of the program. The findings and conclusions pertaining to these questions are
summarized below.
Evaluation Question 1: How well did PDL prepare NTF’s and RT’s for their
mentoring and modeling roles, respectively, in the new teacher program?
The findings of this report suggested that the PDL training provided to NTF’s was
effective and appropriate. NTF’s reported that the various training components aimed at
preparing them for their mentoring roles, including the Author Study Exchanges, weekly NTF
meetings, and the NTF Institutes, were useful to them. The LTPLP sessions, though not directly
affiliated with the PDL for New Teachers program, were also noted as highly useful training
CRTL Research Report Series, EE-0304-01
viii
Evaluation of the PDL Programs in Community School District 20
opportunities. Observations of the sessions suggested that NTF’s were given opportunities to
reflect on and practice learned skills. Data collected from the PDAQ questionnaire supported
this conclusion, indicating that NTF’s responded to the most important needs of their new
teacher interns.
Other data collected pointed to school-level contextual obstacles that had to be overcome
by the NTF’s in their work with their interns. NTF’s commented that in some cases their
mentoring schedules were interrupted by emergency coverages that were assigned to their
interns. Also, several of the NTF’s assumed other functions in the schools, and they noted that
the additional responsibilities often interfered with their mentoring responsibilities. The latter
may have been a manifestation of the resource demands attendant to DOE restructuring, a
contextual effect that may not generalize to other program implementations.
Evaluation Question 2: In the perception of new teachers, to what extent did
program services address their most important staff professional development and
support needs?
Evidence from the PDAQ indicated that new teachers perceived the PDL for New
Teachers program to be responsive to their most important professional development and support
needs, including motivating students, classroom management, lesson planning, and clarifying
instructional goals. The sample IPAL’s collected also suggested that NTF’s responded to the
needs of their interns. IPAL’s consistently pointed to support provided in the areas of classroom
management and lesson planning, two areas reflected in the PDAQ as important to the new
teachers.
Additionally, many of the teachers who participated in the Cycle Program reported that
they benefited from the program’s activities, particularly the opportunity to observe a veteran
teacher using instructional strategies that incorporated group work, manipulatives, and multiple
intelligences. Participants did note that the program could have been enhanced with a second
round of visits to the RT’s classrooms.
Overall, the evidence suggested that the PDL for New Teachers program had a structured
model for implementation, but also had sufficient flexibility to meet the individual professional
development needs of new teachers.
Evaluation Question 3A: To what extent did new teachers believe that the
program facilitated their professional growth in the competencies of effective
teachers?
Multiple analyses of the data collected from the EECI showed a relationship between the
areas in which new teachers reported receiving PDL support and the areas where teachers
believed themselves to be most effective. The data also showed that more program participants
felt competent and supported in the domain of Planning and Preparation, which included areas
such as content knowledge, assessment, and goal setting. The relationship between the teachers’
beliefs about their competencies and the support they received from PDL was also strongest in
this domain. On the other hand, a notably low number of respondents indicated high skill levels
CRTL Research Report Series, EE-0304-01
ix
Evaluation of the PDL Programs in Community School District 20
in the areas of gaining access to school resources for students and engaging families.
Correspondingly, a low number of respondents indicated high levels of support from PDL in
those same areas.
Evaluation Question 3B: To what extent did social studies teachers believe that the
program facilitated their professional growth in the competencies of effective teachers?
Participants in the PDL for Middle School Social Studies Teachers program believed the
program facilitated their professional growth as teachers. The Social Studies Feedback Survey
and PDL’s internal program feedback forms indicated that participants believed their ability to
integrate technology, primary source documents, literature, and cooperative learning strategies
into their classrooms was enhanced by this PDL program. Some participants did note, however,
that their ability to integrate technology in their practice was hampered due to their schools’ lack
of technological resources. This varied from school to school.
Evaluation Question 4A: Did new teachers served by the program show an
increase in their sense of efficacy as a teacher?
The evidence with respect to PDL’s impact on new teachers’ sense of efficacy was
inconclusive. There was no significant increase in the new teachers’ sense of efficacy, due in
part to the low return rate on the post-TES. Additionally, the literature regarding teacher
efficacy suggested that the construct of “general teacher efficacy” may be problematic and
difficult to measure. These findings highlight the need for more research on the measurement of
this construct, and teacher beliefs and attitudes in general.
Evaluation Question 4B: Did social studies teachers served by the program show an
increase in their sense of efficacy as a teacher?
The evidence with respect to PDL’s impact on the middle school social studies teachers’
sense of efficacy was similarly inconclusive. There was no significant increase in the social
studies teachers’ sense of efficacy, due in part to the low return rate on the post-TES.
Additionally, the cautionary literature cited above obtains here as well.
Evaluation Question 5A: To what extent did new teachers served by the program
demonstrate the professional competencies of effective classroom teachers?
On average, participants in the PDL for New Teachers program were either partially
proficient or were approaching proficiency in the professional competencies of effective
teachers. The data from the DRTO demonstrated a slight increase in the mean scores of the
sample of teachers over time. There was a notable increase in the mean scores of six areas. Four
of the areas were in the Planning and Preparation domain, and the remaining two were in the
Instruction and Classroom Environment domain, respectively. In general, the changes in mean
scores were in the expected direction. It is expected that with a larger sample and/or a longer
intervention, the difference in mean scores would have been greater.
CRTL Research Report Series, EE-0304-01
x
Evaluation of the PDL Programs in Community School District 20
Evaluation Question 5B: To what extent did social studies teachers served by the
program demonstrate the professional competencies of effective classroom
teachers?
The sample of social studies teachers who participated in the observation phase of the
research generally demonstrated at least “partial proficiency” in the professional competencies of
effective classroom teachers. In three of the four cases, scores generally remained constant or
increased slightly in the second observation. Conversely, in the fourth case, the teacher’s ratings
decreased in a number of the twenty-nine areas from “proficient” to “partially proficient.”
Evaluation Question 6: Were the retention rates of new teachers served by the
program better than new teachers in comparable schools that were not?
The analysis of new teacher retention data showed a trend toward a favorable effect of
PDL upon reducing the rate of new teachers leaving the New York City school system. For both
cohorts of PDL schools, the attrition rates showed a slight increase in the first year of service, but
far less than the increase in comparison schools. Further, there was a decline in system leavers
for the schools that were served by PDL for a second year. The data on new teachers who stayed
in the same school were more equivocal. Teacher retention is a complex issue and is affected by
many factors. Moreover, the evaluation of the program took place under the influence of the
confounding effects of new state regulations and the systemic restructuring of the New York City
public schools.
Evaluation Question 7A. Did students of new teachers served by the program
show gains in ELA achievement test scores that were equal to those for all CSD
20 middle school students?
The analysis of ELA achievement test score data indicates that PDL had a favorable
affect on students of PDL-served teachers in CSD 20. The mean gains on ELA achievement of
the PDL group were higher than those for the non-PDL group in three out of the four schools and
four out of the six grade-within-school comparisons. Mean gains for the two groups were similar
in the other two analyses. The mean gains of the PDL group were also higher for the combinedschools analyses. In general, PDL had statistically significant and educationally meaningful
effects on the achievement of students of new teachers who were served by the PDL for New
Teachers program.
Evaluation Question 7B. Were the State Intermediate Level Social Studies
Achievement Test (SILSSAT) scores of students of teachers served by the
program higher than those of students of non-PDL teachers in the same schools?
The analysis of SILSSAT scores indicates that PDL did not have a statistically significant
effect on scores beyond the influence of prior general achievement. The mean 2003 SILSSAT
scale scores tended to be higher for the non-PDL group than for the PDL group both within
schools and across the four schools combined. However, the mean 2002 ELA scale scores also
tended to be higher for the non-PDL group. Accordingly, the differences in SILSSAT means
CRTL Research Report Series, EE-0304-01
xi
Evaluation of the PDL Programs in Community School District 20
may have been attributable to differences in the general achievement of the two groups prior to
the program.
RECOMMENDATIONS
Based on the findings and conclusions of the evaluation, the following recommendations
are offered to enhance the effects of future implementations of these programs.
PDL for New Teachers
♦ Make principals and other supervisory staff more aware of PDL’s mission and goals through
by strengthening orientation and conducting ongoing updates.
♦ PDL and District staff should continue to work with school leadership to devise ways to
encourage and facilitate structured visits to classrooms.
♦ Help the NTF’s identify the impact of their work on students. All NTF’s who responded to the
CRTL questionnaire indicated that they believed their interns were positively impacting their
students, yet they often could not cite “concrete evidence.” Being able to cite specific
evidence would not only help to enhance the new teacher’s craft, but would also go a long way
toward helping the role of PDL and the NTF become more valued.
♦ Consider ways to schedule and deliver training so that NTF’s are out of their school buildings
less often. All of the NTF’s who responded to the questionnaire noted that the various training
components were valuable and reinforcing. The biggest drawback they noted, however, was
that the numerous training sessions took them out of the building and often interfered with
their mentoring schedule. This was exacerbated by the additional administrative
responsibilities placed on them by the District.
♦ Expand the PDL for New Teachers program emphasis to include explicit training on gaining
access to school resources for students and working with families, two areas that the literature
suggests are important features of effective instruction. The data collected on the EECI
indicated that teachers’ self-ratings of their professional skill and the emphasis of mentoring
particularly in these two areas were relatively low.
♦ PDL should consider expanding the pool of teachers eligible for The Cycle Program so more
cohorts of teachers, including new teachers who have already met the mentoring requirement,
can take advantage of this highly effective model. PDL should also consider beginning the
Cycle Program earlier in the year and expand it to include a visit by the Resident Teachers to
their Visiting Teachers’ schools.
While the above recommendations are aimed at strengthening the quality of
implementation of future replications of the program, it should be emphasized that there is
considerable evidence of the program’s impact upon the professional growth of new teachers and
the quality of teaching and learning in their classrooms. Particularly notable in this regard is
evidence suggesting that the program’s basic model is effective and should be retained as a
foundation.
CRTL Research Report Series, EE-0304-01
xii
Evaluation of the PDL Programs in Community School District 20
PDL for Middle School Social Studies Teachers
♦ PDL and District staff should ensure that participants in the PDL for Middle School Social
Studies program, and their students, have access to the technology resources that would allow
them to implement learned approaches.
♦ Evaluations of future replications should pay greater attention to the collection of more
complete data. This can be achieved by identifying the students served by program teachers
prospectively, i.e. at the beginning of the program, so that plans can be developed to capture
their achievement test scores later in the project year. These achievement test data should be
supplemented by other evidence of growth in student achievement, chief among which is the
structured analysis of student work.
IMPLICATIONS FOR RESEARCH AND EVALUATION
Along with higher education institutions across the country, the NYU Steinhardt School
of Education is in the process of developing evaluation and accountability systems for its teacher
education programs. Accountability through accreditation is a requirement under federal and
state legislation aimed at improving teacher education. The work conducted in this evaluation of
the Professional Development Laboratory has direct implications for the evaluation system the
Steinhardt School of Education is creating for its teacher education programs. This evaluation
gave CRTL the opportunity to pilot a variety of instruments, including the Teacher Efficacy
Scale, the Domain Referenced Teacher Observation protocol, the Elements of Effective
Classrooms and Instruction teacher questionnaire, and the Professional Development Activities
Questionnaire. Piloting these instruments gave CRTL valuable insight in furthering the creation
of an effective evaluation system for the NYU Steinhardt School of Education teacher education
programs
CRTL Research Report Series, EE-0304-01
xiii
Evaluation of the PDL Programs in Community School District 20
I. INTRODUCTION
This report presents the findings of an evaluation of the Professional
Development Laboratory (PDL) programs in New York City’s Community School
District (CSD) 20 during the 2002-2003 academic year. These programs were designed
to provide support for the professional development of two groups of middle school
teachers in the district: first-year teachers who required mentoring and social studies
teachers who wished to enhance their instructional skills. A third program was designed
to provide professional development in the area of leadership for staff developers and
exemplary teachers.
The evaluation was conducted by the Center for Research on Teaching and
Learning of the Department of Teaching and Learning in New York University’s
Steinhardt School of Education. The purpose of the evaluation was to document the
implementation of the PDL programs for middle school teachers in CSD 20 and to assess
its impact upon four groups of participants as follows:
•
•
•
•
First, the impact of PDL training upon the mentoring capacities of school-based
professional-development staff in CSD 20 middle schools;
Second, the effects of the program upon the professional competence, sense of
efficacy, and retention in the profession of new middle school teachers;
Third, the effects of the program upon the instructional skills and sense of
efficacy of selected middle school social studies teachers; and
Last, the indirect effects of the program upon the academic achievement of
students served by the programs’ teacher participants.
In the objectives of the evaluation stated above, direct effects refer to those that
are attributable to the work of PDL with the individuals who are exhibiting growth or
improvement. Indirect effects, on the other hand, are changes in individuals that are
mediated through the work of other individuals who impact upon them.
By focusing on these participant groups, the evaluation assesses the cascading
impact of the programs beginning with the training provided by PDL staff and
consultants to school-based professional developers, to the work of the latter with new
and experienced teachers, and ending with the effects of this work upon the academic
performance of students.
This evaluation report is organized in five sections. Following the introduction,
the second section describes the programs, including its sponsoring agency, PDL,
descriptive features of its context, CSD 20, and the rationale for the program. The third
section describes the evaluation, including the evaluation agency, the methods, and
assumptions and limitations. The fourth section describes the major evaluation findings,
and the last section presents conclusions and recommendations based on the findings.
CRTL Research Report Series, RR-0304-01
1
Evaluation of the PDL Programs in Community School District 20
II. THE PROGRAMS
The Professional Development Laboratory (PDL)
PDL designed and implemented the professional development programs that were
the focus of this evaluation. PDL was instituted in 1989 and currently functions under
the aegis of the Department of Teaching and Learning’s Ruth Horowitz Center for
Teacher Development at New York University’s Steinhardt School of Education.
Additional PDL partners include the New York City Department of Education (DOE),
the UFT, and the business sector. Over the past fourteen years, PDL has worked with 15
New York City school districts. Currently, PDL runs programs in Brooklyn’s
Community School Districts (CSD) 18 and 20. Over this period, PDL has implemented a
variety of professional development and training programs for teachers and school
administrators. The breadth of this work is best described by the goals stated in PDL’s
awareness literature:
•
•
•
•
•
•
•
•
Developing leadership capacity in teachers and administrators for the purpose of
improving student learning.
Preparing facilitators to convene groups of educators and school teams to examine
student work and analyze student data in order to improve student performance.
Establishing networks to share practice across school districts.
Integrating technology into the teaching and learning process.
Evaluating the impact of professional development on student learning.
Collaborating with other school change initiatives.
Providing access to expertise in the field of education research.
Incorporating current educational research into program design.
At the heart of the PDL philosophy is the belief that by opening classroom doors,
teachers have the opportunity to reflect on their teaching practice and analyze student
work to develop instructional approaches, all while collaborating with one another. The
PDL model centers on the belief teachers learn best to improve their practice when the
learning takes place in actual classrooms during the school day.
PDL’s Programs in CSD 20
The focus of this evaluation is on three of PDL’s programs in CSD 20: PDL for
New Teachers, PDL for Middle School Social Studies Teachers and the Learning
Through Practice Leadership Program (LTPLP). Although these three programs are
discrete, all focus on the same ultimate goal—to improve teaching and learning by
helping to build the professional competency of teachers.
PDL for New Teachers
The first year of teaching is extraordinarily overwhelming for teachers and
involves a number of phases, starting with anticipation, then survival, disillusionment,
CRTL Research Report Series, RR-0304-01
2
Evaluation of the PDL Programs in Community School District 20
rejuvenation, and reflection (Moir, 1990). Research shows mentoring is an essential
component of support and retention of new teachers as they go through these phases
(Moir & Bloom, 2003; Black, 2001; Mauer & Zimmerman, 2000). Highly qualified
mentors are critical elements in any teacher induction program, and cannot be easily
replaced. However, the literature cautions that poorly trained mentors can hinder the new
teacher’s induction and increase attrition (Moir & Bloom, 2003). Quality mentorship
programs involve ongoing mentor training, with opportunities for building knowledge,
reflection, and problem-solving as mentors work with their interns (Moir & Gless, 2003;
Feiman-Nemser, 2003). States with rigorous and high-quality mentor training programs
have a marked decrease in teacher attrition (Darling-Hammond, 2003). However, in
order to be successful, mentoring and professional development programs require a
financial and logistical “buy-in” on the part of the school administration and their district.
PDL for New Teachers was a three-year model designed to build the capacity of
CSD 20 to provide the high-quality mentoring that research has found to be integral to
the professional growth and retention of new teachers. The program was designed to
build professional competency in planning lessons, analyzing student academic growth
and student work, delivering appropriate instructional strategies, and using cooperative
learning. In addition, the program aimed to integrate new teachers into the school
community, assist them in achieving certification, and increase their retention in the
profession. The program aimed to achieve its goals by linking the new teachers to New
Teacher Facilitators (NTF), who were trained to serve as mentors by both PDL staff and
PDL-trained district staff.
The program was organized around a comprehensive array of training
components that were designed to ensure that district staff, responsible for mentoring the
new teachers, engaged in collaborative professional activities that enhanced their
mentoring skills. Figure 1 displays the program model as depicted by the organization of
the program components. The NTF’s were integral to the success of the program, since
they directly provided mentoring to the new teachers and coordinated the delivery of
program resources as well. As Figure 1 shows, the program model trained NTF’s
through three components: NTF Weekly Meetings/Author Study Exchanges, the 10-day
NTF Institute, and the LTPLP.
The weekly meetings were facilitated by PDL staff and were conducted at a PDL
Lab in one of the middle schools or the district office, and were attended by the five firstyear NTF’s. From November through May, the weekly meetings were replaced with an
Author Study Exchange that was attended by all NTF’s. During the Author Study
Exchanges, also facilitated by PDL staff, NTF’s read and discussed the current research
literature on topics such as effective coaching and feedback strategies, analyzing student
work, and differentiated instruction.
The seven Author Study Exchanges were half-day training sessions at the district
office. Figure 2 displays the program’s designed three-year implementation scheme, with
two cohorts of new teachers receiving support from NTF’s. For each year of
implementation, the NTF’s provide formal mentoring support to the new teachers, and
CRTL Research Report Series, RR-0304-01
3
Evaluation of the PDL Programs in Community School District 20
informal support to returning teachers. The model also displays the additional support
provided to new teachers through the Teacher Leadership Institute (TLI) courses,
commensurate with their level of need.
FIGURE 1
PDL FOR NEW TEACHERS PROGRAM MODEL
NTF Weekly
Meetings/Author
Studies
NTF Institutes
LTPLP
NTF
♦ Weekly Observations &
Conferences
♦ Schedules NT
observations of veteran
staff
♦ Models effective practice
♦ Provides access to
resources
TLI
♦ Monthly course taught
by NTF
♦ Focus on INTASC
standards
New
Teachers
Students
RT
♦ Models effective practice
♦ Conducts post-observation
conferences
The NTF Institutes involved ten full days of training, on NYU’s campus, focused
on building the leadership capacity of the NTF’s, as well as developing a professional
community to share experiences and knowledge with one another. Sessions focused on
effective mentoring strategies, differentiated instruction, linking the INTASC standards1
to practice, and using protocols to assess the progress of the NTF’s work with new
teachers. NTF’s were expected to integrate the strategies learned during the Institutes into
their mentoring practice.
1
INTASC stands for Interstate New Teachers Assessment and Support Consortium.
CRTL Research Report Series, RR-0304-01
4
Evaluation of the PDL Programs in Community School District 20
FIGURE 2
THREE-YEAR IMPLEMENTATION OF NTF & TLI COMPONENTS
Year One (2001-2002)
Yr 1 NTF’s
Yr 1 New
Teachers
TLI 1
Year Two (2002-2003)
Yr 2 New
Teachers
TLI 1
Yr 1 New
Teachers
TLI 2
Yr 1& 2
NTF’s
Year Three (2003-2004)
Yr 2 New
Teachers
TLI 2
Yr 1& 2
NTF’s
Yr 1 New
Teachers
Another training opportunity that both NTF’s and some RT’s received was the
LTPLP. While this program is not directly linked to the PDL for New Teachers program,
it provided training for those leaders involved with the program in the form of seven
training days at the district focused on various leadership strategies, differentiated
instruction, multiple intelligences, using protocols, observation techniques, and using
Department of Education data to guide instruction.
Another component of the PDL for New Teacher program was the PDL Cycle
Program. The goal of this component was to provide the new teachers with opportunities
to improve their pedagogical skills while working with Resident Teachers (RT),
experienced teachers who were recognized by the district for their professional expertise
and exemplary classroom practice. New teachers earned new teacher credit for their
participation in the Cycle Program.
CRTL Research Report Series, RR-0304-01
5
Evaluation of the PDL Programs in Community School District 20
New teachers were also encouraged to enroll in the PDL-designed TLI, where
they could earn, NYU graduate credit, NYC Department of Education “G” credit, or new
teacher credit. There were three basic TLI’s (TLI 1) offered for first year teachers, and
one advanced TLI (TLI II) intended for second year teachers. All courses were offered at
three of CSD 20’s middle schools, in fifteen two-hour sessions over the 2002-2003
academic year. The goals of the basic TLI programs were stated in the course outline as
to:
•
•
•
Assist the new teacher in developing the requisite skills to become an effective
classroom teacher;
Enhance the new teacher’s ability to assume a leadership role as a learner and
teacher; and
Enhance the new teacher’s ability to use collaboration with colleagues to analyze
and solve problems.
The goals of TLI II incorporated and expanded those of the basic TLI course:
•
•
•
•
•
Support and coach teachers in providing meaningful and sustained assistance to
improve teaching and learning;
Stimulate discussion and dialogue in order to deepen the understanding of how
students think and learn;
Actively design programs of instruction to increase student engagement in
learning;
Utilize ongoing assessments, rubrics and portfolios to diagnose and assess student
performance; and
Provide teachers with resources to select goals, determine improvement methods
and monitor student progress.
PDL for Middle School Social Studies Teachers
The research literature on effective professional development indicates that
traditional in-service models of professional development are generally unsuccessful
(Fickel, 2002; Corcoran, 1995). Professional development programs that have been
shown to have a positive impact on teachers are sustained over a substantial period of
time, involve uninterrupted day-long commitments with release time for staff, and
opportunities to use and reflect on new knowledge and skills (Fickel, 2002; Guskey,
2002; Guskey, 1998).
The PDL for Middle School Social Studies Teachers program was a researchbased program designed to help CSD 20 social studies teachers develop professional
competency in planning lessons, understanding student development and analyzing
student work, developing appropriate instructional strategies, and integrating technology
in the social studies classroom. Figure 3 displays the program model as depicted by the
organization of program components. To achieve its goals, the program provided a series
of Social Studies Institutes for middle school social studies teachers.
CRTL Research Report Series, RR-0304-01
6
Evaluation of the PDL Programs in Community School District 20
Designed to be implemented over a period of three years, the program also aimed
to develop the leadership potential of middle school social studies teachers by enhancing
their facilitation and communication skills and teaching them the principles of mentoring.
Year two of implementation was designed to train Cohort 1 participants in facilitation
and mentoring principles, so that they could serve as Resident Teachers for Cohort 2. At
the same time, Cohort 2 social studies teachers would receive training in the SSI. Finally,
the third year of implementation was designed so that Cohort 1 would continue to receive
leadership training and Cohort 2 would begin leadership training. Both Cohorts 1 and 2
would open their classrooms as Resident Teachers for Cohort 3, who would also receive
training in the Social Studies Institute (SSI). (See Table 1.)
FIGURE 3
SOCIAL STUDIES INSTITUTE PROGRAM MODEL
Social Studies Institutes
♦ Integrating technology
♦ Finding & using primary
documents
♦ Integrating literature
♦ Using cooperative learning
strategies
Cohort 1 Social
Studies Teachers
Students
Learning Through Practice Leadership Program
The Learning Through Practice Leadership Program (LTPLP) was designed to
develop the leadership potential of selected district teachers and staff developers whose
classrooms exemplified standards-based teaching and learning and best practices, through
ongoing training (Refer to Figure 1). The CSD 20 leadership selected eleven participants
for this program, all of whom also played a role in the PDL for New Teachers program
(all eight NTF’s and five RT’s). The program consisted of seven full-day training
sessions that focused on various leadership strategies, differentiated instruction, multiple
intelligences, using protocols, observation techniques, and using Department of
Education (DOE) data and the GROW Report (DOE diagnostic skills analysis) to guide
instruction.
CRTL Research Report Series, RR-0304-01
7
Evaluation of the PDL Programs in Community School District 20
TABLE 1.
PDL FOR MIDDLE SCHOOL SOCIAL STUDIES TEACHERS THREE-YEAR
MODEL
Cohort 1
Cohort 2
Cohort 3
Year 1
(2002-2003)
SSI
N/A
N/A
Year 2
(2003-2004)
Year 3
(2004-2005)
Mentoring and Residency
Training (10 Days)
Mentoring & Residency
Training (5 Days)
SSI
Cohort 1 Resident Teachers
Mentoring & Residency
Training (10 Days)
N/A
SSI
Cohort 1 & 2 Resident
Teachers
Context
The PDL programs served teachers in nine middle schools located in CSD 20,
which encompasses the neighborhoods of Bay Ridge, Bensonhurst, Dyker Heights,
Kensington, Flatbush, and Bath Beach in Brooklyn, New York. The district’s population
is varied linguistically and ethnically with over 60 languages spoken in the homes of the
students in those neighborhoods. Although diverse, the ethnic distribution of CSD 20
students differs from that for the city overall. Based on the 2001 – 2002 Annual District
Report (NYC Department of Education, 2003), almost 37 percent of CSD 20’s students
are White, 5 percent Black, 25 percent Hispanic, and 33 percent Asian/Pacific Islander,
compared to 15 percent, 33 percent, 39 percent and 12 percent for the city overall.
Almost 13 percent of the students arrived in the United States within the last three years,
compared to 7 percent for the city, and more than 20 percent are limited in English
proficiency, compared to 13 percent for the city. More than 70 percent are eligible for
free lunch, which is about the same as the 73 percent citywide. The teaching staff in
CSD 20 tends to be slightly more experienced and educated than the faculty citywide.
Nevertheless, 14 percent of the teachers were provisionally appointed, 45 percent had less
than 5 years total teaching experience, and 34 percent had no more than two years
teaching experience in their schools. The student population is large, with almost thirtythousand students distributed among the district’s 30 schools, and the schools tend to be
crowded, serving 107.5 percent of capacity, compared to 91.9 percent citywide. CSD 20
has consistently outperformed the city overall on standardized tests in English Language
Arts (ELA) and mathematics. For the three years between 2000 and 2002, approximately
half of CSD 20 students in grades 3 thru 8 met grade-level performance standards on
state and city tests in ELA and mathematics, compared to less than 40 percent for the city
overall. However, there has been a steep decline in test performance—steeper than that
for the city overall—between grade 4 and grade 8. In 2002, 59 percent of CSD 20 4th
graders, but only 34 percent of 8th graders, met state standards on the state ELA test.
CSD 20’s 4th grade performance was 12 percentage points higher than the city’s, but CSD
20 held only a 3 percentage point advantage in grade 8. A similar pattern was displayed
in mathematics with the percentage of students meeting state standards declining from 73
percent in grade 4 to 41 percent in grade 8, and a 21 percentage point advantage over the
city in grade 4 shrinking to 11 percentage points in grade 8. The declining standardized
CRTL Research Report Series, RR-0304-01
8
Evaluation of the PDL Programs in Community School District 20
test performance between grades 4 and 8 signaled a need to consider ways to bolster
teaching and learning in CSD 20’s middle schools. Profiles of each of the nine middle
schools in CSD 20 are presented in Appendix A.
Another contextual feature that may have affected the implementation of PDL
programs during the 2002-03 academic year was the reorganization of the DOE.
Throughout the year, CSD 20 was undergoing transformation as it prepared for the
transition from a semi-independent school district to incorporation in a larger region. The
reorganization had an impact on the district and school climate, leaving many personnel
at the district and school level unsure of their job security. This uncertainty was
exacerbated by the announcement in February 2003 that CSD 20’s superintendent would
be resigning his position in the spring. The reorganization of the DOE also led the
district to rely heavily on NTF’s to fulfill additional administrative responsibilities which
were beyond the scope of their mentoring roles.
III. THE EVALUATION
The Evaluators
The evaluation was conducted by the Center for Research on Teaching and
Learning (CRTL). CRTL was organized in February, 2002 and functions within the
organization of the Department of Teaching and Learning of New York University’s
Steinhardt School of Education. CRTL’s mission is to support the department in the
development of teacher education and professional development programs that prepare
highly qualified educators to work with diverse student populations. CRTL fulfills its
mission by conducting evaluations of the department’s pre-service and in-service
programs, establishing systems of accountability, and supporting research done by
department faculty and students. Five CRTL staff members conducted this evaluation as
follows: The director developed the evaluation design and oversaw all aspects of its
implementation; a CRTL graduate research assistant served as the principal investigator
with responsibility for designing the instruments, scheduling and conducting site visits
and interviews, administering the instruments, and collecting all evaluation data; two
CRTL graduate student workers coded, key entered, and prepared data for analysis; and a
CRTL research scientist conducted all data analyses. CRTL staff members were
supported by two adjunct faculty members from the Department of Teaching and
Learning who conducted structured observations in the classrooms of a sample of
teachers served by the program.
Evaluation Questions
The evaluation was designed to track the flow of services across the program
models, as depicted in Figures 1 and 2 above, documenting program implementation and
assessing its impact for each model component.
In this way, for the PDL for New Teachers program, the evaluation traced and analyzed
the path of program effects as they rippled through the model from the training of NTF’s
and RT’s by program and district staff, through the mentoring of new teachers by NTF’s
CRTL Research Report Series, RR-0304-01
9
Evaluation of the PDL Programs in Community School District 20
and the modeling of teaching techniques by RT’s, to the impact of new teacher training
upon the teaching and learning process and, consequently, student achievement. A
similar design was used to document and assess the effects of the PDL Program for
Middle School Social Studies Teachers. The evaluation was designed to address the
following questions about the effectiveness of the PDL program models at CSD 20:
1. How well did PDL prepare NTF’s and RT’s for their mentoring and modeling
roles, respectively, in the new teacher program?
2. In the perception of new teachers, to what extent did program services address
their most important staff professional development and support needs?
3. To what extent did new teachers believe that the program facilitated their
professional growth in the competencies of effective teachers?
4. Did new teachers served by the program show an increase in their sense of
efficacy as teachers?
5. To what extent did new teachers served by the program demonstrate the
professional competencies of effective classroom teachers?
6. Were the retention rates of new teachers served by the program better than new
teachers in comparable schools that were not?
7. Did students of new teachers served by the program show gains in ELA
achievement test scores that were equal to those for all CSD 20 middle school
students?
The evaluation also addressed Evaluation Questions 3, 4, 5, and 7 for the teachers
who participated in the PDL for Middle School Social Studies Teachers program, with
performance on the grade 8 state examination in social studies added as data for
Evaluation Question 7.
Methods of Data Collection
The evaluation used a variety of methods to collect qualitative and quantitative
data on the programs’ components, including interviews, observations, self-report
questionnaires, affective surveys, human resources data, and student achievement test
scores. The data that were collected are described below.
New Teacher Staff Training Observations
CRTL staff conducted several observations of training sessions held for NTF’s
and RT’s. CRTL staff attended three of the five LTPLP training sessions and two NTF
Author Study Exchanges to collect ethnographic data. These data were used to address
Evaluation Question 1.
NTF and RT Questionnaires and Interviews
All participants in the LTPLP program—NTF’s and RT’s— were asked to fill out
a questionnaire which asked them to discuss the impact of all the PDL training they
received on their leadership skills and on their work with new teachers. Four of the
CRTL Research Report Series, RR-0304-01
10
Evaluation of the PDL Programs in Community School District 20
participants—three NTF’s and one RT—also agreed to participate in a focus group
regarding the same. However, due to scheduling conflicts, individual interviews were
conducted with each of the four volunteers at their respective schools. Questions were
taken from the questionnaire and expanded with follow-up questions. The four
volunteers were compensated for their participation in the interview. The data from these
questionnaires and interviews were used to assess how well PDL prepared the NTF’s and
RT’s to work with new teachers as mentors and models, respectively (Evaluation
Question 1).
New Teacher Surveys
In May, NTF’s were asked to distribute three surveys to their interns—the term
used to describe the new teacher facilitators’ mentoring relationship with the new
teachers. The first was a Participation Information Form that asked new teachers to
provide basic demographic information that would be used for descriptive analysis. The
second, the Professional Development Activities Questionnaire, asked teachers to identify
the support they received through the program in a number of professional development
areas, and to rate the level of importance of each. The list of support areas was
developed by CRTL evaluators to reflect the key professional issues that concern new
teachers in their first year of teaching. To develop this survey, CRTL evaluators
interviewed new teachers in selected non-program public schools in New York City.
CRTL evaluators compiled a list of areas of concern that were mentioned during the
interviews. The list was culled to a final set of issues that represented the most
frequently-mentioned areas of concern. This survey was used to assess the extent to
which new teachers perceived that the program addressed their most important
professional development needs (Evaluation Question 2). The third survey, the Elements
of Effective Classrooms and Instruction, asked teachers to rate their current level of
development as a teacher in twenty-seven areas, and the extent to which they had
received mentoring in those areas. The elements were based on the characteristics of
effective teachers as presented in the work of Charlotte Danielson (1996). The twentyseven areas were organized into four distinct domains—Planning and Preparation,
Classroom Environment, Professional Responsibility, and Classroom Instruction. The
data from the third survey were used to assess the extent to which the new teachers
showed developing expertise in the areas in which they received program support
(Evaluation Question 3). All of the above surveys were individually addressed in manila
envelopes, and were returned directly to CRTL via business-reply envelopes. Some of
these surveys were mailed directly to the teachers at their schools, while others were
delivered to the teachers by their NTF’s.
Individual Performance Assessment Logs (IPAL’s)
The IPAL’s are internal program documents used by the NTF’s to track the
weekly progress of their interns. The form is used to specify areas of strength and
weakness, and calls for the NTF and the intern to identify subsequent activities to support
the new teacher’s growth. The form is also used to document the new teacher’s
fulfillment of the NYS mentoring requirement.
CRTL Research Report Series, RR-0304-01
11
Evaluation of the PDL Programs in Community School District 20
Teacher Efficacy Scale
New teachers’ sense of their effectiveness as teachers (Evaluation Question 4)
was assessed with the Teacher Efficacy Scale (Gibson and Dembo, 1984). The TES was
developed to measure a teacher’s sense of efficacy and its relationship to teacher
behavior and student achievement. The original scale consisted of thirty items rated
on a six point Likert-type scale. However, a factor analysis indicated that two substantial
factors emerged from only sixteen of the items. The first factor was labeled personal
teacher efficacy because the items relate to the individual teacher’s belief that her actions
have an impact on student behavior and learning. The second factor, or teacher efficacy,
relates to the belief that any teacher’s ability to influence student behavior and/or learning
is limited by external elements such as family and home environment. In fall 2002, each
NTF was given copies of the TES to distribute to all of their interns. NTF’s were also
given postage-paid reply envelopes for teachers to return the surveys directly to CRTL.
In spring 2003, NTF’s were provided with the names of teachers who had returned the
TES and were asked to follow up with those teachers who had not yet returned them.
NTF’s were provided with additional surveys and postage-paid envelopes. Teachers who
filled out the TES prior to March were mailed a post-test TES in May. The TES was also
redistributed to those teachers who failed to complete it during the first round of
distribution. The TES was also administered pre and post to all teachers in the PDL
Middle Schools Social Studies Program.
New Teacher Classroom Observations
New teachers were recruited by their NTF’s to participate in the observation
phase of the evaluation. Two (one pre and the other a follow-up) observations were
scheduled with all 13 volunteers. The pre-observation was conducted on all 13; due to
scheduling conflicts, only ten follow-up observations were conducted. Two NYU
adjunct faculty members, who serve as student teacher supervisors, were hired as
consultants to conduct the observations, using the Domain Referenced Teacher
Observation (DRTO) protocol (adapted from Danielson, 1996). The consultants were
trained by a CRTL staff member who underwent one full day of training with the
DRTO’s creator, Charlotte Danielson. The consultants were trained over a period of
three days, during which they viewed un-narrated videos depicting classroom instruction.
After viewing each video, the consultants used the DRTO rubric to individually rate the
instruction. Areas of discrepancy were discussed and resolved before moving to the next
video. Training proceeded until the raters agreed on 90% of the ratings. During the
course of training, it became evident that the ratings of one of the consultants who was
originally recruited to conduct observations, was consistently in the outlier range.
Accordingly, this consultant was not employed in the operational study.
The protocol consisted of twenty-nine items organized into four domains:
Planning and Preparation, Classroom Environment, Professional Responsibility, and
Classroom Instruction. The consultants rated the teachers’ proficiency in each area.
Twenty-seven items on the DRTO were matched with the items on the aforementioned
CRTL Research Report Series, RR-0304-01
12
Evaluation of the PDL Programs in Community School District 20
Elements of Effective Classrooms and Instruction. At the time of the first observation,
teachers were asked to collect three samples of typical assignments from three students
who varied in their level of work, with one approaching state standards, one meeting the
standards, and the other exceeding them. Participants in this phase of the research were
compensated for each observation they completed. Two of the observed teachers also
participated in the PDL Middle School Social Studies Teachers program. Data from the
observations were used to assess the extent to which new teachers served by the program
demonstrated the professional competencies of effective classroom teachers (Evaluation
Question 5).
Social Studies Institute Feedback Questionnaire
The SSI Feedback Questionnaire consisted of ten items designed to determine if
the goals of the program, as stated by the Institute’s facilitators, had been met. First,
teachers were asked to rate the extent of professional development in seven areas that
were identified in the program’s literature. For those same areas, teachers were also
asked to rate the extent of the influence of the professional development on their practice.
The questionnaire also included three open-ended questions designed to elicit specific
examples of the skills participants learned in the program. (Evaluation Question 3B)
Teacher Retention Data
Data on teacher retention were obtained from the Division of Assessment and
Accountability (DAA) of the New York City Department of Education (NYC DOE) and
project staff. The DAA data files were extracted from the NYC DOE’s human resources
electronic record system and contained information on all teachers in the New York City
public schools for fall 2000, fall 2001, and fall 2002. For each year, the files contained
information on each teacher’s school, subjects taught, number of years of experience in
their current schools, and total years of experience in education. Using encrypted
identification numbers, the files for the three years were matched. Using SPSS
programming, flags were applied to each matched teacher record to indicate their status
in 2001 and 2002 as follows: continued employment in the same school, continued
employment in NYC DOE but in a different school, and left NYCDOE. Project staff
determined the retention status of the new teachers served by PDL in 2002-2003 through
telephone interviews with the staff of CSD 20 middle schools. These data were sorted
into the same three status categories.
Standardized Achievement Test Scores
To address Evaluation Question 7, the standardized achievement test scores of
students served by the PDL-trained teachers were compared to those of students served
by more experienced teachers in the same schools. The Division of Assessment and
Accountability prepared two electronic files of standardized test files for CSD 20
students. The first file contained the matched 2002 and 2003 state and city English
language arts (ELA) test scores for every CSD 20 student on register in grades 6 – 8 in
2003. This file was used to compare the gains in ELA scale scores obtained by students
CRTL Research Report Series, RR-0304-01
13
Evaluation of the PDL Programs in Community School District 20
of teachers in the PDL for New Teachers program with the scale-score gains of students
in the same schools who were not served by the program. The second file contained the
2003 Grade 8 State Social Studies Examination scores for all CSD 20 students matched
with each student’s 2002 ELA test scores. The social studies test scale scores for
students of teachers served by the PDL for Middle Schools Social Studies Teachers
program were compared to the scores of students of other social studies teachers in the
same schools. ELA scores were used as a covariate in this analysis to control for preprogram differences in the general achievement levels of the two groups of students.
Assumptions and Limitations
The evaluation was conducted in the context of the ongoing implementation of the
program in its natural context using a variety of measurement techniques, including
interviews, observations, record reviews, questionnaires, surveys, and student test data.
The evaluators have no reason to believe that the normal assumptions of naturalistic
research have been violated in this study. Specifically, the evaluation assumes that:
•
•
•
the responses of the subjects to all instruments were candid and reflected their
actual beliefs and understandings;
the activities observed by the evaluators were representative of normal program
implementation and were not contrived for the evaluation; and
the data used to describe the program and assess its effects are based, in whole or
in part, on the activities of the PDL program.
In addition to the general assumptions that apply to naturalistic research, the
reader is cautioned to be mindful of certain limitations that are specific to the
circumstances of this evaluation.
•
•
•
•
The instruments were developed or adapted specifically for the purpose of this
evaluation. As such, there was limited time to pilot these instruments and assess
their validity and reliability.
The period of time between pre- and post-administration of the instruments was
relatively brief. Ideally, we would have administered pre-tests prior to or closer
to the start of the intervention.
While the return rate on some of the surveys was reasonably high, we would have
preferred a greater response rate on others. We also would have liked a more
consistent response from the participants on all surveys, particularly those
teachers who took part in the observation phase of the research.
Student work collected from teachers was not consistent with some handing in
tests and others projects. Also in some cases teachers only submitted work for
one student, not three. In short, we were unable to use the student work in any
substantive way. Better planning in the collection and assessment of student work
in ways that would permit the teachers to better understand how to use the work
for evaluation and instruction should be explored in future evaluations.
CRTL Research Report Series, RR-0304-01
14
Evaluation of the PDL Programs in Community School District 20
•
•
The evaluation focused on the effects of one year of program implementation.
The true magnitude of the effects of this program can only be assessed in
longitudinal follow-up, which is beyond the scope of the evaluation.
Standardized test data could not be matched to all of the teachers served by the
programs. The evaluators believe the resultant attrition of subjects was random
and did not substantially bias the internal validity of the analyses for Evaluation
Question 7.
Although the above assumptions and limitations warrant some caution, the
evaluators believe that the findings from this evaluation provide a fair and reasonable
assessment of the program and its effects.
CRTL Research Report Series, RR-0304-01
15
Evaluation of the PDL Programs in Community School District 20
IV. FINDINGS
This section describes the results of the 2002-03 PDL evaluation. The section
begins with a description of the new teachers who were targeted for mentoring and
professional support by the PDL for New Teachers program. The section continues with
the presentation of the findings organized by the study’s evaluation questions.
New Teacher Participants
The PDL for New Teachers program served a total of 67 new teachers in CSD 20
during the 2002 – 2003 school year. There were large differences in the numbers of new
teachers served among the district’s middle schools. The numbers ranged from as few as
2 (3.0% of all participants) new teachers in two of the schools to as many as 14 (20.9%)
in one school. (See Table 1.) Since each school was served by one NTF, there was a
wide range in caseload among the program’s NTF’s. It should be pointed out that the
several NTF’s assumed other functions in the school such as reading specialists, assistant
principal, staff developer, etc.
Consistent with the program’s proposed target population, the new teacher interns
were young and inexperienced. Their median age was 28, and their median total
experience in education was 2 years. All were in their first year in the school in which
they were served. About one quarter had earned their master’s degrees, and 67% had
only a bachelor’s degree. (See Table 2.)
TABLE 2
DISTRIBUTION OF NEW TEACHER PROGRAM INTERNS BY SCHOOL
School
N of New Teachers
% of Participants
A
2
3.0%
B
10
14.9%
C
6
9.0%
D
11
16.4%
E
9
13.4%
F
14
20.9%
G
2
3.0%
H
13
19.4%
TOTAL
67
100%
CRTL Research Report Series, RR-0304-01
16
Evaluation of the PDL Programs in Community School District 20
TABLE 3
DESCRIPTIVE CHARACTERISTICS OF NEW TEACHER PROGRAM
INTERNS
Characteristic
Median Age
Median Yrs Teaching Anywhere
Median Yrs in Current School
% Bachelor's Degree
% Master's Degree
Median/%
28 Years
2 Years
1 Year
67.0%
24.7%
The new teachers taught a wide range of subjects in the 2002 – 2003 school year.
The most frequently taught subjects were math and science and English/literature, 25.9 %
of respondents each. (See Table 3.)
TABLE 4
SUBJECTS TAUGHT BY NEW TEACHER INTERNS
Subjects Taught
N
% of Respondents
Math/Science
7
25.9%
Social Sciences
3
11.1%
Arts/Electives
5
18.5%
English/Literature
7
25.9%
Special Education/ Bilingual
4
14.8%
Other
1
3.7%
TOTAL
27
99.9%
Social Studies Teacher Participants
Twenty Social Studies teachers from eight of nine middle schools in CSD 20 were
participants in the 2002-2003 PDL for Middle School Social Studies Teachers program2.
There were large differences in the number of years of experience among program
participants, ranging from 1 to 22 years, with a mean of 6.6 years and a median of 4
years. (See Table 5.) Five of the participating schools had 3 participants, and one school
had 1.
2
One participant withdrew from the program due to an injury she sustained in the line of duty.
CRTL Research Report Series, RR-0304-01
17
Evaluation of the PDL Programs in Community School District 20
TABLE 5
DISTRIBUTION OF SOCIAL STUDIES PROGRAM PARTICIPANTS BY
SCHOOL
School
A
B
C
D
E
F
G
H
Total
N of SS
Teachers
1
3
3
3
3
3
2
2
20
% of
Participants
5%
15%
15%
15%
15%
15%
10%
10%
100%
Evaluation Question 1. How well did PDL prepare NTF’s and RT’s for
their mentoring and modeling roles, respectively, in the new teacher
program?
During the 2002-2003 academic year, there were a total of eight NTF’s in eight of
nine middle schools in CSD 20, with a range of two to fourteen interns assigned to each.
Three of the NTF’s were selected and trained in the fall of 2001, and were reappointed in
2002-2003. The remaining five NTF’s were selected in September and October, with
two selected in late November and December 2002.
Topics covered during the weekly meetings included the stages of new teacher
development, understanding the role of the mentor, using the INTASC standards to
formulate instructional strategies, using the IPAL's, learning and practicing mentoring
strategies, and using protocols to facilitate meetings.
The data from interviews, internal evaluation forms and a CRTL questionnaire
suggested that NTF’s believed the training they received from the NTF Institutes, the
weekly NTF meetings, and the Author Study Exchanges enhanced their leadership
abilities, with respect to their listening, communication and facilitation skills. A number
of NTF’s commented on the usefulness of the various protocols that were used during the
training sessions to facilitate their own meetings with interns. The first year NTF’s who
participated in the weekly meetings also suggested that their understanding of new
teacher development increased as a result of the extensive work they did with the
INTASC standards.
While the data indicates that PDL training effectively prepared the NTF’s to
mentor their new teacher interns, some data suggest obstacles at the schools that may
have made their job more difficult. One concern was that while CSD 20 was committed
to the goals of PDL, several NTF’s believed that this commitment was not always shared
by the school principals. NTF’s commented that in some cases, new teachers were
CRTL Research Report Series, RR-0304-01
18
Evaluation of the PDL Programs in Community School District 20
assigned emergency coverages that often interfered with the mentoring schedule. In at
least one school, the NTF was also assigned coverages. Additionally, the NTF’s reported
that they were often pulled away from their mentoring responsibilities in order to assist
with grading and/or other administrative duties. This was especially true of the six
NTF’s who were district reading specialists. Several NTF’s indicated that their presence
in the school was not formally legitimized by their principals, through introduction or
other measures. Some NTF’s reported that veteran teachers felt threatened by the NTF,
and did not allow them and new teachers to visit their classrooms. At least one NTF
indicated that an antagonistic relationship between teachers and school administrators led
both new and veteran teachers to be wary of her position in the school.
Observations of training sessions also suggested that the NTF’s were provided
with valuable opportunities to learn and practice leadership skills. During one LTPLP
session, NTF’s and RT’s were divided into two groups with each group visiting one of
the participating PDL CSD 20 middle schools. Each group observed a teacher
conducting a lesson using a balanced literacy strategy. After the observations, the session
focused on strategies for both “warm” and “cool” feedback. As the participants practiced
the strategies, a PDL facilitator interjected with suggestions for improving the strategies,
such as using proper nouns rather than pronouns, and making eye contact.
The observations also suggested that the NTF’s were encouraged to transform
theories of mentoring and leadership into practical strategies. This process occurred
through first individual, then group reflection, culminating in the creation of a formal
plan of action. For example, during one Author Study, the teachers were asked to choose
one of the instructional strategies outlined in the book that they preferred. They were
asked to share their choices in small groups. Then, the groups were asked to choose one
of the instructional strategies discussed and create an instructional objective and
strategies they could use with one of their interns.
All NTF’s felt that the various modes of PDL training gave them the vocabulary
and strategies to help them with their interns. They indicated that while test scores might
not have indicated that their work with interns positively impacted students, their efforts
to assist their new teachers in becoming competent and reflective practitioners had a
positive impact on student performance. As one NTF stated, “I don’t have any concrete
evidence [of PDL’s impact on student achievement] but I can say that the students do
well if the teacher is prepared and confident in his/her craft.” Another NTF indicated that
at least one of her intern’s levels of expectations for her students had increased; this had a
positive impact on her students’ performance.
Evaluation Question 2. In the perception of new teachers, to what extent
did program services address their most important professional
development and support needs?
PDAQ
CRTL Research Report Series, RR-0304-01
19
Evaluation of the PDL Programs in Community School District 20
In order to determine the extent to which the new teachers perceived the program
as meeting their professional development needs, all new teachers were asked to
complete the Professional Development Activities Questionnaire (PDAQ). Respondents
were asked to answer two sets of questions on the PDAQ. First, respondents used a fivepoint Likert-type scale to rate how important it was that they, as new teachers, receive
support in 26 areas of professional development. The scale ranged form “Not Important
at All” to “Very Important”. The second set of questions asked them to check off each
area in which they received support during their first year through the program. Results
of the questionnaire for the 32 teachers who responded are displayed in Table 4. Table 4
shows the percentage of respondents rating each area “Very Important” or “Important”
and the percentage who indicated they received support in those areas from the program.
(For purposes of this discussion these ratings are combined under the label important.)
Overall, all of the areas were rated important by more than half of the
respondents. This is to be expected since the list was constructed through interviews with
new teachers concerning areas of importance to them for first-year professional
development. However, there were considerable differences in the ratings among the
areas. All respondents rated motivating students as important to them and over 93%
percent gave the same rating to classroom management. These data corresponded with
data retrieved from the sample of IPAL’s that indicated NTF’s concentrated their
mentoring in these two areas. (See discussion of IPAL data below.) In addition, 90.6%
of the respondents gave ratings of important to dealing with diverse student needs. In at
least two cases, the sample IPAL’s suggest that this area was being met by the NTF’s.
Overall, 10 areas were rated important by at least 80% of respondents. Conversely, 5
areas, excluding “other”, were rated important by less than 65% of the respondents.
Among these, was classroom and/or school inter-visitation (58%), a significant feature of
the PDL for New Teachers program. When the percentage of respondents who reported
that they received program support was averaged separately for the 10 areas with the
highest percentage of important ratings and the five areas with the lowest percentage of
important ratings, the mean support percentage was meaningfully higher for the former
than the latter. The mean support percentage was 22.0% for the most important areas
versus 14.2% for the least important areas. These findings suggest that the new teacher
interns perceived that the PDL program was focused on their most important needs.
Moreover, each area showed at least some respondents reporting that they received
program services. This suggests that the program addressed a wide range of new teacher
needs and that the services were tailored to the needs of the individual teacher.
CRTL Research Report Series, RR-0304-01
20
Evaluation of the PDL Programs in Community School District 20
TABLE 6
NEW TEACHERS’ IMPORTANCE RATINGS AND REPORTS OF PDL
PROGRAM SUPPORT ON THE PROFESSIONAL DEVELOPMENT
ACTIVITIES QUESTIONNAIRE
Areas of Importance
1. Motivating students
2. Classroom management
3. Dealing with diverse learning needs
4. Certification
5. Curriculum materials for content area/grade level
6. Clarification of departmental/school-wide instructional goals
7. Observations with feedback
8. Support in the content area
9. Orientation (including payroll, your rights as a teacher, etc)
10. Modeling of classroom practice in a classroom
11. Mentoring
12. Individual meetings to discuss classroom practice
13. Lesson planning/design
14. A collegial environment
15. Support with curriculum development
16. Navigating the Department of Education
17. Approaches to assessing student performance
18. Materials (paper, markers, rulers, etc)
19. Forums for discussing issues/ideas
20. Time management
21. Setting up your classroom
22. Building trust among colleagues
23. Inter-visitations to other classrooms/schools
24. Avoiding isolation
25. Other
26. Support groups
Percent of New Teachers
(N = 32):
Rating Very
Receiving
Important/
Program
Important
Support
100.0
14.5
93.8
20.3
90.6
21.7
87.5
18.8
87.5
26.1
83.9
21.7
83.9
27.5
83.3
23.2
81.3
21.7
80.0
24.6
75.0
29.0
75.0
31.9
75.0
24.6
75.0
13.0
71.9
26.1
68.8
15.9
68.8
21.7
68.8
15.9
68.8
13.0
65.6
17.4
63.3
11.6
62.5
14.5
58.1
23.2
56.3
13.0
55.6
5.8
51.7
8.7
IPAL’s
Each week, NTF’s were expected to observe their assigned interns and meet with
them afterwards to discuss their observations. The outcomes of the meetings were
recorded on IPAL’s and transmitted to PDL staff who turned them over to DOE
personnel, who in turn awarded the new teachers with credit towards fulfilling their
mentoring requirements. Whenever possible, NTF’s were also expected to take their
interns to observe experienced teachers in their school. Effective strategies and
CRTL Research Report Series, RR-0304-01
21
Evaluation of the PDL Programs in Community School District 20
techniques used in those lessons were discussed and incorporated into the interns’
planning.
A review of a sample of IPAL’s revealed a number of common themes. NTF’s
noted that the new teachers were concerned about planning and classroom management.
With respect to planning, teacher concerns and questions referred to curriculum mapping,
pacing the lesson to fit within the class period, and incorporating instructional strategies
(group work, using manipulatives, etc.) that motivated students. Many of the concerns
surrounding classroom management centered on getting students on task and dealing with
students who were disruptive. These themes are consistent with the new teachers’
importance ratings on the PDAQ (See Table 4), where over 93% of teachers indicated
that motivating students and classroom management were important or very important.
The IPAL’s suggest a number of strategies taken by the NTF to assist the new
teachers. Most often, the IPAL suggested that the NTF’s helped the teacher plan lessons,
or they coordinated meetings between the new teacher, the content area staff developer,
and themselves. This was especially evident in the area of curriculum mapping. NTF’s
also team taught lessons that incorporated group activities to demonstrate effective group
management and facilitation. In many cases, the NTF’s scripted new teachers’ lessons to
help them with pacing. These themes were also reflected in the new teachers’ importance
ratings on the PDAQ. 75% of new teachers reported that lesson planning was important
or very important, while 80% indicated that modeling practices was important (See Table
4).
When classroom management was a concern, NTF’s generally observed how the
teacher’s movement around the classroom and instructional strategies impacted student
behavior. The IPAL’s indicate that suggestions for managing classroom behavior often
centered on modification of instructional strategies, such as incorporating more studentcentered and group activities. NTF’s also made suggestions that new teachers make a
more concerted effort to call on withdrawn or misbehaving students, to draw them into
the lesson. In only one of the IPAL’s, the NTF instructed the new teacher to use a
disciplinary approach in managing the behavior of some students. The NTF’s emphasis
on classroom management corresponds to the new teachers’ importance ratings on the
PDAQ, where 93% of the new teachers reported that classroom management was
important or very important (Table 4).
Another common area of concern for the new teachers was preparing their
students for the NYS and NYC examinations. While some teachers in the sample IPAL’s
expressed these concerns in the fall of 2002, this was a consistent area of concern for
most of the teachers in the spring, coincident with the onset of the state and city testing
programs. NTF’s provided teachers with copies of previous state and city examinations,
and worked with the new teachers to identify major concepts that appeared on the exams
frequently. Additionally, two of the NTF’s in the sample collaborated with their new
teachers to use the GROW Report to plan instructional strategies that addressed the
different needs of their students.
CRTL Research Report Series, RR-0304-01
22
Evaluation of the PDL Programs in Community School District 20
Interview Data on the Cycle Component
In the fall, the NTF’s were asked to identify new teachers who might benefit
from participating in the Cycle Program. Seven new teachers were identified and
matched with Resident Teachers (RT), who were trained and certified by PDL and CSD
20 as exemplary role models. Wherever possible, new teachers and RT’s were matched
by discipline. The new teachers met with the RT’s, the NTF’s, and the Cycle Program
facilitator during two after school sessions to discuss the program goals and objectives, as
well as the goals of the new teachers. During one of two periods in January and
February, the new teachers visited the RT’s schools over a period of three days. Prior to
the visits, the Cycle Program PDL facilitator, RT’s, and new teachers met individually to
discuss goals and focus on areas that meet the needs of the new teacher. During the first
two days of the visit, the RT modeled pedagogical skills that addressed the goals set
earlier, and the facilitator led a pre and post-conference where the new teacher had the
opportunity to discuss the lesson with the RT. On the third day the new teachers used the
skills learned in the Cycle Program to teach all of the RT’s classes. The new teachers
were expected to return to their home schools and use the newly learned skills with their
own students.
Internal program feedback suggested that most of the new teachers found the
Cycle Program to be beneficial. A number of participants commented that the
opportunity to visit a veteran teacher in his/her own environment was helpful in
demonstrating how strategies, such as group work, using manipulatives, and multiple
intelligences, might work in a classroom. One concern that was raised was that while the
strategies worked for the RT’s, classroom management issues that new teachers were
facing might interfere with the application of these strategies in the classroom. One new
teacher suggested that having the RT visit the classroom to provide feedback on learned
strategies might be useful in overcoming these challenges. One of the RT’s commented
that it would have been beneficial if they had the opportunity to meet with the new
teachers a second time. However, this same RT indicated that frequent disruption in the
VT’s and RT’s teaching schedule, while positive with respect to new teacher induction
and reflective practice, was too costly for the school to support. The RT also stated that
the new teacher’s students would be negatively impacted by the frequent absence
required by an additional Cycle Component.
Evaluation Question 3A. To what extent did new teachers believe that
the program facilitated their professional growth in the competencies of
effective teachers?
The Elements of Effective Classrooms and Instruction (EECI) was used to assess
new teacher perceptions of the extent to which the program facilitated their professional
growth in the first year of teaching in their schools. The EECI was comprised of 27
elements associated with effective classroom teachers, clustered in four areas: Planning
and Preparation, the Classroom Environment, Instruction, and Professional
Responsibilities. (See Appendix B.) The new teachers were asked to rate each item with
respect to two different dimensions. First, the teachers rated their degree of development
CRTL Research Report Series, RR-0304-01
23
Evaluation of the PDL Programs in Community School District 20
in each element using a 5-point Likert-type scale, ranging from (1) Minimal to (5)
Advanced. Second, they rated the emphasis project mentoring placed on each element
using a 5-point Likert-type scale, ranging from (1) No Emphasis to (5) Very Heavy
Emphasis. Results of the questionnaire for the teachers who responded are displayed in
Table 6. The number of responses for each item varies because all respondents did not
respond to every item. Table 6 shows the items displayed according to the total
percentage of respondents who rated the items at level 4 or 5 for each of the 27 effective
elements. These items are displayed in rank order and are organized by domain. The
means and standard deviations for each item are also shown. Additionally, the table
displays the percentage of respondents who rated the emphasis of program mentoring at
levels 4 or 5, as well as the means and standard deviations for each item.
Of the four domains, respondents indicated that they felt most prepared in the area
of Planning and Preparation, with 56.2% of respondents indicating a level of professional
skill at 4 or 5. Overall, the mean score for program emphasis at level 4 or 5 in this
domain was 55.9%. The domain with the second highest ratings was Classroom
Environment, with a mean score in professional skill at 53.7%, and a mean score in
mentoring emphasis at 57.8%. The mean ratings for professional skill level in each of the
two remaining domains (Instruction and Professional Responsibilities) were each below
50%, and the mean ratings for mentoring emphasis were each below 52%.
In the domain of Planning and Preparation, 60% of respondents rated their skill
level at 4 or 5 in five of the fourteen areas. Those items related to assessment, content,
multicultural and gender issues, and creating appropriate goals. In these same areas, over
65% of the respondents rated the emphasis of the PDL program mentoring at levels 4 or
5. In comparison, only 28.1% of the respondents indicated that they were at a level 4 or 5
in “gaining access to school resources for students.” Similarly, 28.2% indicated that
program emphasis in this area was at level 4 or 5. This indicates a strong relationship
between program emphasis and the level of professional skill teachers reported.
In the Classroom Environment domain, an average of 60.4% of the respondents
rated their level of professional skill at 4 or 5 in three of the six areas, which deal
primarily with managing student behavior and engagement. In these same areas, the
mean emphasis of mentoring received from PDL was 61.4%. In the remaining three
areas, which relate primarily to organization and classroom routines, the mean rating for
level of professional skill at level 4 or 5 was 46.9%. The mean emphasis of PDL training
in these areas was 54.2%. This indicates a moderate relationship between the PDL
program emphasis and the teachers’ self-reported level of professional skill.
No greater than 59% of the respondents rated their professional skill level at 4 or
5 in any of the four items in the domain of Instruction. The same is true for respondents’
ratings of mentoring in the four areas. The four items deal primarily with instructional
strategies that foster student contributions in classroom discussions, group work, and the
learning process. The mean ratings for professional skill and mentoring emphasis in all
four areas were 48.5% and 51.2%, respectively.
CRTL Research Report Series, RR-0304-01
24
Evaluation of the PDL Programs in Community School District 20
In the last domain, Professional Responsibilities, at least 50% of the respondents
rated their professional skill level at 4 or 5 in those areas pertaining to being reflective
and working with other teachers (64.5% and 50%, respectively). For those same areas,
the mean rating of program mentoring emphasis at levels 4 or 5 was 54.8% and 46.9%,
respectively. In the area of engaging families, only 15.6% of the respondents rated their
level of professional skill at 4 or 5. 21.9% rated the program emphasis in this area at
level 4 or 5.
To further explore the relationship between program mentoring and the selfreported professional development of program-served new teachers, evaluators applied
correlation analysis to the EECI data. Table 7 displays a summary of the correlation
analysis between the teachers’ ratings of their level of professional growth in the four
areas of professional development measured by the EECI and the extent of PDL
mentoring they reported receiving in each of those areas. The Pearson product moment
correlation coefficients for all four areas were moderate to moderately high and all were
statistically significant beyond p < .01 for two-tailed tests with 31 degrees of freedom.
Professional responsibility was the area that showed the strongest relationship between
professional growth and extent of mentoring with a coefficient of determination equal to
.483, meaning that nearly one-half of the variance in professional growth was shared by
the extent of mentoring provided by PDL. The extent of the relationship in the other
areas was more modest, with coefficients of determination indicating between one-fifth
and one-quarter shared variance.
Thus, findings from multiple analyses of EECI data suggest that in those areas
where teachers believed their skill level to be advanced, or approaching advanced, they
generally reported PDL support to be strongest. Moreover, even in those areas where
fewer teachers indicated having a high level of professional skill, a comparable
percentage of teachers indicated that they had received heavy or very heavy emphasis of
mentoring. Similar to the PDAQ findings, the results of the EECI also demonstrate the
flexibility of the program model as it adjusts to the needs of the individual new teacher.
CRTL Research Report Series, RR-0304-01
25
Evaluation of the PDL Programs in Community School District 20
TABLE 7
MEAN SELF RATINGS ON THE ITEMS OF THE ELEMENTS OF EFFECTIVE CLASSROOMS AND INSTRUCTION
Level of Professional Skill
Cluster
Planning and Preparation
Emphasis of Mentoring
N
Mean
S.D.
%
Levels 4
&5
Effective Elements
N
Mean
S.D.
% Levels
4&5
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Use of a Variety of Assessment Methods
Knowledge of Content Areas
Establishment of Goals in Level of Expectations, Clarity and Suitability
Multicultural and Gender Issues
Classroom Assessments Aligned w/ Instructional Goals
Lesson Plans
State Content Standards
Different Approaches to Learning
Ability to Adjust Instruction
Classroom Assessment Criteria and Standards
Pedagogical Practices
Establishment of Goals for Different Types of Learning
Knowledge of All Resources Available through School
Knowledge of Gaining Access to School Resources for Students
32
32
32
32
32
32
32
32
32
32
31
32
32
32
3.84
3.88
3.81
3.84
3.69
3.69
3.84
3.69
3.78
3.63
3.47
3.56
3.28
2.81
0.81
0.79
0.90
0.85
0.69
0.93
0.88
0.69
0.83
0.91
1.07
0.91
1.22
1.25
71.9
68.8
65.6
62.5
62.5
59.4
59.4
56.3
53.1
53.1
51.6
50.0
43.8
28.1
32
31
32
32
32
32
32
32
32
32
31
32
32
32
3.81
3.19
3.69
3.31
3.72
3.5
3.53
3.66
3.66
3.69
3.16
3.53
3.16
2.81
1.12
1.05
1.18
1.33
1.02
1.16
1.08
1.24
1.12
1.15
1.19
1.14
1.22
1.09
68.8
35.5
65.6
43.8
65.7
56.2
59.4
65.7
59.4
68.8
65.5
56.3
43.8
28.2
15
16
17
18
19
20
Being Alert to Student Behavior
Polite and Respectful Classroom Environment
Students Generally Engaged
Smooth Transitions
Routines for Handling Materials and Supplies
Standards of Conduct are Clear
32
32
32
32
32
32
3.88
3.84
3.75
3.44
3.5
3.55
0.87
1.08
0.95
1.01
1.08
1.16
62.5
59.4
59.4
46.9
46.9
46.9
32
32
32
32
32
32
3.72
3.37
3.63
3.47
3.44
3.56
1.2
1.39
1.24
1.29
1.32
1.27
68.7
50
65.6
59.4
46.9
56.2
21
22
23
24
High Quality Questions
Creation of Appropriate Instructional Groups
Ability to Have Students Assume Responsibility for Discussions
Creation of a System for Maintaining Info on Student Progress
32
32
32
32
3.88
3.56
3.47
3.25
0.98
0.98
0.98
1.08
59.4
53.1
46.9
34.4
32
32
32
31
3.67
3.44
3.41
3.16
1.19
1.11
1.21
1.24
59.4
53.1
46.9
45.2
25
26
27
Being a Reflective Practitioner
Ability to Work as Part of a Team w/ other Teachers
Ability to Engage Families
31
32
32
3.81
3.47
2.56
0.79
1.19
1.16
64.5
50.0
15.6
31
32
32
3.42
3.03
2.72
1.23
1.38
1.25
54.8
46.9
21.9
Item#
Classroom Environment
Instruction
Prof Responsibilities
CRTL Research Report Series, RR-0304-01
26
Evaluation of the PDL Programs in Community School District 20
TABLE 8
CORRELATIONS BETWEEN NEW TEACHERS’ SELF-REPORTED DEGREE OF
PROFESSIONAL SKILL AND THE EXTENT OF PROGRAM MENTORING IN FOUR
ESSENTIAL AREAS OF EFFECTIVE CLASSROOM TEACHERS
N
Level of
Significance
.439
r2
Coefficient of
Determination
.193
32
.012
Classroom
Environment
.471
.222
32
.006
Professional
Responsibility
.695
.483
32
.001
Classroom
Instruction
.484
.234
32
.005
Area of
Professional
Development
Planning and
Preparation
Correlation
Coefficient
Evaluation Question 3B. To what extent did the social studies teachers believe
that the program facilitated their professional growth in the competencies of
effective teachers?
CSD 20 leadership, in collaboration with the eight participating middle school principals,
selected twenty teachers to participate in the PDL for Middle School Social Studies Teachers
program. Participants were given release time to attend eight SSI’s, facilitated by PDL and
District staff. During the SSI’s, two facilitators modeled instructional strategies that
incorporated literature and literacy, cooperative learning, and simulation activities. They also
used protocols to look at the participants’ student work. The sessions also focused on the use of
technology as both a planning and instructional tool.
A review of a CRTL questionnaire, as well as PDL’s internal program feedback forms,
suggests that participants believed this program was useful to them in a number of areas,
particularly in integrating the use of technology in the classroom as a learning tool and as a
resource for students. Teachers indicated that the extent of professional development in all of the
program areas, including developing professional competency in planning lessons, understanding
student development and analyzing student work, developing appropriate instructional strategies,
and integrating technology in the social studies classroom, was at least moderate, with several
indicating that the emphasis was large. Teachers commented that the SSI’s gave them the skills
and the opportunities to access resources, such as primary source documents and activities on the
internet. Teachers found this skill particularly useful in integrating the use of document-based
questions in their instruction. Teachers also noted their increased confidence in using literature
CRTL Research Report Series, RR-0304-01
27
Evaluation of the PDL Programs in Community School District 20
in their classrooms, as well as sharing ideas and materials with their colleagues. Most
participants also commented that the time given to them for planning was extremely useful. All
respondents indicated that the professional development had at least a moderate influence on
their practice, with one participant indicating a large influence in all but one of the areas.
Many of the participants’ perceptions of the SSI’s are best demonstrated through one
teacher’s comment about the program: “I feel it gave me a much needed boost in the
classroom—the mundane was replaced by a new outlook on teaching.”
Evaluation Question 4A. Did new teachers served by the program show an
increase in their sense of efficacy as a teacher?
Items on the TES that related to the Personal Teacher Efficacy (PTE) factor dealt
primarily with the individual teacher’s impact on his/her students. They imply that if the
respondent makes a concerted effort, it will influence the academic achievement of his/her
students. The seven items relating to the General Teacher Efficacy (GTE) factor imply that,
regardless of any teacher’s effort, home environment and parental/family background dictate the
academic achievement and behavior of students.
The statements are generally worded to demonstrate the limited impact that any teacher
in general may have on students. The mean score of those items on the TES pre-test (see Table
8) that related to PTE were all above level 4 (agree slightly more than disagree).
Four of the items had a mean score above 4.6, with item five showing the highest mean
score of 4.75. Conversely, only three of the items related to GTE had a mean score over 4.0.
There were no statistically significant differences in mean pre- and post-test scores for PTE or
GTE.
CRTL Research Report Series, RR-0304-01
28
Evaluation of the PDL Programs in Community School District 20
TABLE 9
MEAN SELF RATINGS ON THE TEACHER EFFICACY SCALE (NEW TEACHERS)
Item #
Pre Test
Personal Teacher Efficacy
N
Mean
S.D.
1
5
6
7
9
10
12
13
15
48
48
48
48
48
47
48
47
48
4.61
4.75
4.23
4.73
4.48
4.49
4.48
4.66
4.13
1.26
1.14
1.13
1.05
1.11
1.02
1.01
1.34
1.10
2
3
4
8
11
14
16
46
48
48
47
47
48
46
3.15
2.92
4.00
3.32
4.72
4.13
3.96
1.59
1.51
1.60
1.53
1.21
1.39
1.60
1
5
6
7
9
10
12
13
15
14
14
14
14
14
14
14
13
13
4.86
4.86
4.36
5.00
4.93
4.86
4.43
4.85
4.77
0.66
1.23
1.15
1.30
0.83
1.10
1.50
1.07
0.83
2
3
4
8
11
14
16
14
14
14
14
14
13
13
3.93
2.86
4.86
3.86
5.29
3.69
4.84
1.14
1.23
1.46
1.75
0.91
1.80
0.90
General Teacher Efficacy
Post Test
Personal Teacher Efficacy
General Teacher Efficacy
CRTL Research Report Series, RR-0304-01
29
Evaluation of the PDL Programs in Community School District 20
Evaluation Question 4B. Did Social Studies teachers served by the program
show an increase in their sense of efficacy as a teacher?
The TES pre-test was administered to participants of the PDL for Middle School Social
Studies program during their December 2002 meeting. Table 9 displays the mean scores and
standard deviations for each item. The items are grouped by the two factors measured on the
TES, personal teacher efficacy (PTE), and general teacher efficacy (GTE). The average of the
mean scores of those items related to PTE is 4.07 (agree slightly more than disagree).
Conversely, the average of the mean scores of the items related to GTE is 3.31 (disagree slightly
more than agree). In two of the items related to GTE, the mean scores were between 2.21 and
2.63, or moderately disagree.
Four of the nineteen program participants who completed the TES pre-test completed the
post-test. While no statistically significant findings emerged from the matched pre and post-tests
of the four participants, on average the participants did show at least a 1 point increase in their
sense of efficacy scores on four of the items. On items 4 and 11, which relate to GTE, the
participants’ efficacy scores increased by 1.25 and 1.00. Participants showed an increase of 1.25
on item 7, and 1.50 on item 15, both of which measured PTE.
TABLE 10
MEAN SELF-RATINGS ON THE TEACHER EFFICACY SCALE PRE-TEST
(S.S. TEACHERS)
Item #
Personal Teacher Efficacy
N
Mean
S.D.
1
5
6
7
9
10
12
13
15
18
19
19
19
19
19
19
19
18
3.28
4.79
3.37
4.37
3.74
3.84
3.95
5.00
4.33
1.23
0.79
1.01
1.38
1.24
1.17
0.97
0.88
1.09
2
3
4
8
11
14
16
18
19
19
19
19
19
19
4.06
2.63
4.16
3.63
2.21
3.89
2.58
1.21
1.61
1.46
1.38
0.98
1.20
0.96
General Teacher Efficacy
CRTL Research Report Series, RR-0304-01
30
Evaluation of the PDL Programs in Community School District 20
Evaluation Question 5A. To what extent did new teachers served by the
program demonstrate the professional competencies of effective classroom
teachers?
The DRTO was used by CRTL consultants who conducted structured observations in the
classrooms of samples of teachers served by the program. The DRTO was comprised of 29
elements associated with effective classroom teachers, clustered in four areas: Planning and
Preparation, the Classroom Environment, Instruction, and Professional Responsibilities. (See
Appendix B.) The observers rated the teachers on a continuum, which included Not Proficient,
Partially Proficient, Approaching Proficient, Proficient, and Distinguished. Table 10 shows the
mean scores of the observations broken down into the four domains, with observations 1 and 2
displayed separately. The standard deviations are also shown. The analysis employed pair-wise
deletion, therefore only those cases with pre and post observations were included. This explains
why the N varies from one domain to the next.
Table 10 shows that on average, teachers were rated as “partially proficient” in the
domains of Planning and Preparation and Classroom Environment, and as “approaching
proficient” in the domains of Instruction and Professional Responsibilities during the first
observation. During the second observation, the mean scores increased slightly in all four
domains.
TABLE 11
MEAN OBSERVATION RATINGS ON THE DOMAIN REFERENCED TEACHER
OBSERVATION TOOL
DOMAIN
Planning and Preparation
Classroom Environment
Instruction
Professional Responsibilities
N
11
11
10
10
Observation 1
Mean
S.D.
2.89
0.64
2.90
0.72
3.08
0.78
3.07
0.72
N
11
11
10
10
Observation 2
Mean
S.D.
3.00
1.01
3.06
0.99
3.16
0.98
3.17
0.85
Table 10 shows that on average, teachers were rated as “partially proficient” in the
domains of Planning and Preparation and Classroom Environment, and as “approaching
proficient” in the domains of Instruction and Professional Responsibilities during the first
observation. During the second observation, the mean scores increased slightly in all four
domains.
There were six items on the DRTO where the mean scores increased notably between the
first and second observations. Four of the items (5, 9, 10, 11), were in the Planning and
Preparation domain. In item 5, which pertained to teacher’s knowledge of students’ varied
approaches to learning, the mean scores increased .57 points on the second observation from
2.45. Item 9, which was concerned with selecting balanced instructional goals, increased .55
CRTL Research Report Series, RR-0304-01
31
Evaluation of the PDL Programs in Community School District 20
points on the second observation from a mean score of 2.50 on the first observation. The mean
score on item 10, which related to finding resources for teaching, increased from 2.73 on the first
observation to 3.27 on the second. Item 11 dealt with teachers’ knowledge of resources available
for students. The mean observation scores on item 11 increased .30 points from 2.90. Item 21,
which showed an increase of .30 points from an original mean score of 3.00, referred to
monitoring student behavior. The item was in the Classroom Environment domain. The last
area that showed a notable increase in mean scores was item 25, “grouping of students.” This
item, under the Instruction domain, increased from 3.31 in the first observation to 3.75 in the
second. Four of these six items showed sampled teachers moving from a classification of
“partially proficient” to “approaching proficient” in the professional competencies of effective
teachers.
The mean scores increased slightly during the second observation, but not significantly.
Overall, the changes in mean scores were in the expected direction (positive), and it is likely that
with a larger sample, the difference might have been greater. Similarly, the results suggest that
conducting the second observation after a longer period of intervention might also have yielded
greater differences in the mean scores.
Evaluation Question 5B. To what extent did social studies teachers served by
the program demonstrate the professional competencies of effective classroom
teachers?
The sample of social studies teachers with both pre and post-observations was too small
(4 cases) for statistical analysis. However, a visual inspection of the data collected on the DRTO
(See Appendix B) shows that in three of four cases the scores generally remained constant or
increased. One teacher’s ratings increased from the level of “approaching proficient” to
“proficient” in seven areas, including knowledge of students’ approaches to learning and skills,
selecting instructional goals, discussion techniques, and instructional groups. Another teacher’s
ratings increased by .5 in four areas, and by 1 point in another four areas. In three of those areas
where the teacher increased most, the ratings went from “partially proficient” to “approaching
proficient.” Those areas included knowledge of content, and knowledge of students’ approaches
to learning and skills. Conversely, another teacher’s ratings decreased in thirteen of the twentynine areas. In some areas, the teacher’s ratings decreased two points from “proficient” to
“partially proficient.”
The results of the DRTO for the sample of Social Studies teachers suggest that there were
a number of limitations in data collection. The sample of teachers with both pre and postobservation data was extremely low. A larger sample, as well as a longer period of time between
the two observations may have yielded more substantive data.
CRTL Research Report Series, RR-0304-01
32
Evaluation of the PDL Programs in Community School District 20
Evaluation Question 6. Were the retention rates of new teachers served by the
program better than new teachers in comparable schools that were not?
The impact of PDL on the retention rates of new teachers was looked at in two ways.
First, the new-teacher retention rates for PDL schools were compared for the years prior to
program service and during program service. Second, the changes in retention rates for PDL
schools were compared to those for comparison schools during the years that the former were
being served by the program. The comparison schools were all middle schools in two
neighboring community school districts that served populations that were demographically
similar to CSD 20.
For purposes of this analysis, the PDL schools were divided into two cohorts. Cohort 1
included the three CSD 20 schools that began PDL in the 2001 – 2002 school year. Cohort 2
included the five CSD 20 schools that were first served in 2002 – 2003. Using the human
resources data files provided by the NYC DOE, the evaluators identified all teachers in each
cohort who were new to the profession and required mentoring as of September 2000 and
September 2001, separately. A new teacher was operationally defined as holding a Preparatory
Provisional Teaching license (PPT) and having less than one year of total teaching experience.
The evaluators identified new teachers meeting the same criteria in the comparison schools for
the same time periods. Next, the status of the identified new teachers was determined one year
later; that is September 2001 for the September 2000 new teachers and September 2002 for the
September 2001 new teachers. There were three status categories for this one year follow-up:
teachers who stayed in the same school, teachers who stayed in the NYC DOE but transferred to
a new school, and those who left the NYC DOE. The data collected by project staff for PDLserved new teachers in 2002 – 2003 were similarly categorized. These data were not available
for the comparison schools at the time this analysis was conducted.
Table 12 summarizes the three-year analysis of new teacher retention data for the two
PDL cohorts and the comparison schools. To facilitate the interpretation of the data in Table 12,
the following explanation is offered. The first three cells in row 1 give the status in September
2001 of teachers who were new to Cohort 1 schools in September 2000. The next three cells in
the row give the status in September 2002 of teachers who were new to these schools in
September 2001. Of the new teachers in Cohort 1 schools in September 2000, 19 (86.4%) stayed
in the same school in September 2001, and 3 (13.6%) left the NYC DOE completely. None of
these new teachers transferred to other schools in the NYC DOE. Analyzed this way, the
retention data indicate some general trends in teacher retention for all schools in the analysis
including a trend for Cohort 1 that is suggestive of a positive impact of PDL.
Table 12 shows an overall increase in the percentage of new teachers leaving the DOE
system for all three groups between the periods September 2000–September 2001 and September
2001–September 2002. However, the three groups differed in the magnitude of this increase.
The percentage of teachers leaving the NYC DOE for Cohort 1 increased by 1.2 percentage
points for the September 2001 new teachers (14.8%), those who were served by PDL, compared
to the September 2000 new teachers (13.6%) who were not served. This increase was far less
CRTL Research Report Series, RR-0304-01
33
Evaluation of the PDL Programs in Community School District 20
than the 8 percentage-point increase for the Comparison Schools for the same period (from
13.8% to 21.8%). Cohort 2, whose new teachers were not served in either of these years, showed
the largest increase (16.2%) in attrition rate for this period (from 3.8% to 20.0%). For the
September 2002 new teachers, Cohort 1 schools, in their second year of PDL, showed a decline
of 5.7 percentage points in system leavers, while Cohort 2, in its first year of PDL service,
showed a small increase of 2.7 percentage points. Thus, these data show evidence of a favorable
effect of PDL upon new teacher attrition rates for Cohort 1 over two years of PDL service.
Program effects for Cohort 2 cannot be assessed until Comparison School data become available
for September 2002-September 2003.
TABLE 12
THREE-YEAR COMPARISON OF THE RETENTION RATES OF NEW TEACHERS
IN PDL COHORT SCHOOLS AND COMPARISON SCHOOLS
Gray shaded areas indicate PDL program intervention.
Cohort 1
Schools
Cohort 2
Schools
Comparison
Schools
Sept. 2000 - Sept. 20013
Stayed in
Same
Stayed
Left DOE
School
in DOE
System
% (N)
% (N)
% (N)
Sept. 2001 - Sept. 2002
Stayed in
Same
Stayed
Left DOE
School
in DOE
System
% (N)
% (N)
% (N)
Sept. 2002 - Sept. 20034
Stayed in
Same
Stayed in Left DOE
School
DOE
System
% (N)
% (N)
% (N)
86.4 (19)
0 (0)
13.6 (3)
77.8 (21)
7.4 (2)
14.8 (4)
77.3 (17)
13.6 (3)
9.1 (2)
88.5 (23)
7.7 (2)
3.8 (1)
73.7 (22)
6.7 (2)
20.0 (6)
72.7 (16)
4.6 (1)
22.7 (5)
75.4 (49)
10.8 (7)
13.8 (9)
67.3 (37)
10.9 (6)
21.8 (12)
N/A
N/A
N/A
Cohort 1 data for new teachers staying in the same school are less clear. Cohort 1
showed an 8.6 percentage-point decline in new teachers staying in the same school for the
September 2001 new teachers compared to September 2000, about the same as the 8.1
percentage-point decline for the Comparison Schools. For Cohort 1, the percentage staying in
the same school remained about the same for September 2002. Cohort 2 schools showed a
decline in the percentage staying in the same school of 14.8 percentage points for the September
2001 new teachers even though PDL didn’t begin until the next year, during which Cohort 2
showed a 1.0 percentage point decline. Thus, the retention data suggest a positive effect of PDL
on retaining new teachers in the NYC DOE system, but there is no evidence of an effect upon
retention in the same schools.
3
4
September 2000 – September 2002 data obtained from the DOE
September 2002 – September 2003 data obtained from PDL staff
CRTL Research Report Series, RR-0304-01
34
Evaluation of the PDL Programs in Community School District 20
Evaluation Question 7A. Did students of new teachers served by the program
show gains in ELA achievement test scores that were equal to those for all
CSD 20 middle school students?
The academic progress of New York City public school students is assessed annually
through the administration of standardized tests in ELA and mathematics in grades 3 - 8. Test
performance is reported through scale scores that range from approximately 450 to 800 and
performance levels from 1 to 4, with Level 3 signifying that students have met the learning
standards for their grade. The scale scores for each grade are vertically aligned, meaning that
performance across grades can be tracked by analyzing a student’s scale score gains from one
year to the next. Students are expected to show scale-score gains from one grade to the next,
reflecting increases in their mastery of skills and their depth of understanding of the subject
matter from one year to the next. Because of their utility for tracking student progress, scale
scores were used in the analysis of test data to address Evaluation Question 7A.
The PDL for New Teachers program is designed to upgrade the skills of novice teachers,
thereby enhancing teaching and learning in their classrooms. One way to measure the
effectiveness of teaching and learning is through the standardized achievement test scores of the
students. A common concern for the students of new teachers is that their academic progress
may be interrupted as the new teachers learn their craft. By providing support to strengthen the
skills of new teachers, PDL has the potential to support the continuity of learning for students in
their classes. One manifestation of this continuity would be continued progress in standardized
test scores. Accordingly, the evaluators tested the hypothesis that the gains in mean ELA scale
scores for students served by PDL-trained new teachers would not differ significantly from the
mean gains of students served by other teachers, i.e. more experienced teachers, in the same
schools. In order to test this hypothesis, the evaluators needed ELA test scores for two years—
the pre-project year 2002 and the project year 2003—for each student served by a PDL-trained
teacher and for students served by other teachers in the same schools. The evaluators obtained
these data for N = 12 PDL-trained teachers in four middle schools served by the program. ELA
test score data describing the gains in performance of the students of PDL teachers and those of
non-PDL teachers in the same school are presented in Table 13. Means and standard deviations
are presented by grade within school and by grade across schools combined. Analyses could not
be performed for students in Grade 8 because a scoring error by the test publisher nullified grade
7 scores for 2002. The mean gains of the PDL group were higher than those for the non-PDL
group in three out of the four schools and four out of the six grade-within-school comparisons.
Mean gains for the two groups were similar in the other two analyses. For one school,
School H, the PDL group had higher mean gains for both Grade 6 and Grade 7. For most
analyses, the PDL group had substantially higher mean scale-score gains than the non-PDL
group; for grade 7 in School A the mean gain for the PDL group was nearly 10 scale scores
higher than the non-PDL group, 20.84 versus 10.98, respectively. The mean gains of the PDL
group were also higher for the combined-schools analyses. The mean gains in Grade 6 were 6.35
CRTL Research Report Series, RR-0304-01
35
Evaluation of the PDL Programs in Community School District 20
for PDL compared to 3.03 for non-PDL, and in Grade 7 they were 20.98 for PDL 3.03 compared
to 15.80 for non-PDL.
TABLE 13
PDL FOR NEW TEACHERS
MEANS AND SD FOR PRE AND POST ELA SCALE SCORES OF STUDENTS OF PDLSERVED AND NON-PDL SERVED TEACHERS
ELA Scale Scores
Pre (2002)
Post (2003)
Mean
SD
Mean
SD
697.04 34.93
717.88
38.58
664.90 31.04
670.74
29.93
School
A
Group
PDL
Non-PDL
N Students
24
151
B
PDL
Non-PDL
66
901
667.97
672.06
15.61
38.3
669.21
682.37
18.56
40.56
C
PDL
Non-PDL
49
593
667.96
649.01
32.13
30.46
682.35
657.77
28.51
30.39
H
PDL
Non-PDL
128
553
635.83
641.02
26.04
31.59
649.52
652.39
25.96
31.52
All Above Schools
Combined
PDL
Non-PDL
267
2619
655.17
657.54
32.92
36.69
666.55
667.39
33.26
37.53
T-tests of the statistical significance of the differences in mean scale-score gains of the
PDL and non-PDL groups are summarized in Table 14. Three grade-within-school analyses
showed statistically significant differences (p < .05) in favor of the PDL group: Grade 7 in
School B, t = 2.03, df = 457, p = .043; Grade 6 in School H, t = 1.99, df = 358, p = .047; and
Grade 7 in School H, t = 1.99, df = 317, p = .047. None of the grade-within-school tests of
significance showed significant differences in favor of the non-PDL group. Table 14 also
displays the Effect Size for the mean differences. Unlike tests of statistical significance, Effect
Sizes are not affected by the size of the samples being compared. The magnitude of Effect Size
is interpreted using a rubric for educational meaningfulness. Effect Sizes of .20 - .49 indicate
differences that have small educational importance, .50 - .69 have moderate educational
importance, and .70 and higher indicate large education importance. Using this rubric, four of
the mean differences suggested that PDL had educationally meaningful effects upon gains in
ELA test scores; two of these effects were small—Grade 7 in School B and Grade 6 in School H,
E.S. = .29 and E.S. = .22, respectively— and two were moderate—Grade 7 in School A and
Grade 7 in School H, E.S. = .35 and E.S. = .38, respectively. The mean difference in Grade 7 for
all schools combined showed an effect of PDL on ELA gain scores that was both statistically
significant (t = 2.37, df = 1,191, p = .018) and educationally meaningful to a small degree (E.S. =
.21).
CRTL Research Report Series, RR-0304-01
36
Evaluation of the PDL Programs in Community School District 20
TABLE 14
PDL FOR NEW TEACHERS
SUMMARY OF T-TESTS FOR THE SIGNIFICANCE OF THE DIFFERENCE IN ELA
GAIN SCORES FOR STUDENTS OF PDL SERVED AND NON-PDL SERVED
TEACHERS
ELA Scale Score Gains
Mean
Mean
SD
Difference
20.83 42.88
14.99
5.84 23.63
School
A
Group
PDL
Non-PDL
N
24
151
t
df
B
PDL
Non-PDL
66
901
1.24
10.31
16.41
27.47
C
PDL
Non-PDL
49
593
14.39
8.76
H
PDL
Non-PDL
128
553
All Above Schools
Combined
PDL
Non-PDL
267
2198
p
2.53
173
0.012
-9.07
-2.65
965
0.008
22.88
22.97
5.63
1.65
640
0.100
13.69
11.36
20.93
24.79
2.33
0.98
679
0.326
11.38
9.85
23.79
25.41
1.53
0.94
2463
0.350
Evaluation Question 7B. Were the State Intermediate Level Social Studies
Achievement Test (SILSSAT) scores of students of teachers served by the
program higher than those of students of non-PDL teachers in the same
schools?
All middle school students in New York State public schools are administered the
SILSSAT once, usually in eighth grade but sometimes in seventh. SILSSAT assesses each
student’s level of mastery of the state social studies learning standards at the intermediate level.
Test scores are reported in scale scores and performance levels. The analysis used scale scores to
increase its sensitivity for detecting program effects.
The analysis compared the mean 2003 SILSSAT scale scores of students served by PDLtrained social studies teachers and those of non-PDL trained social studies teachers in the same
schools. To control for prior differences in achievement, differences in 2002 ELA scale scores
between the two groups were partially controlled through covariance. Test data were obtained
for (N = 5) PDL teachers in four CSD 20 middle schools. Table 15 displays the test score means
and standard deviations for each group by school and for the four schools combined. The mean
2003 SILSSAT scale scores tended to be higher for the non-PDL group than for the PDL group
both within schools and across the four schools combined. However, the mean 2002 ELA scale
scores also tended to be higher for the non-PDL group. Accordingly, the differences in
CRTL Research Report Series, RR-0304-01
37
Evaluation of the PDL Programs in Community School District 20
SILSSAT means may have been attributable to differences in the general achievement of the two
groups prior to the program.
TABLE 15.
PDL FOR MIDDLE SCHOOL SOCIAL STUDIES TEACHERS MEANS AND SD’S FOR
SOCIAL STUDIES AND ELA SCALE SCORES OF STUDENTS OF PDL-SERVED AND
NON-PDL SERVED TEACHERS
N
2003 Soc Stud
Mean
SD
60.10
7.54
68.09 15.34
2002 ELA
Mean
SD
688.08
15.70
707.42
41.12
School
B
Group
PDL
Non-PDL
D
PDL
Non-PDL
18
209
46.22
52.91
7.18
9.30
675.89
685.26
13.56
22.57
E
PDL
Non-PDL
10
321
57.50
59.20
9.36
12.91
679.40
692.63
16.96
26.20
H
PDL
Non-PDL
26
231
53.65
52.65
5.04
9.54
680.42
677.39
11.07
24.72
All Above Schools
Combined
PDL
Non-PDL
113
1175
56.18
59.93
8.70
14.29
683.61
693.54
15.17
33.52
59
414
Table 16 displays a summary of the multiple regression analysis for the indirect effects of PDLtraining upon SILSSAT scores, controlling for pre-program differences in ELA test scores. The
R-square change for PDL was small for all analyses and not statistically significant for the
combined analysis and for three of the four schools. Thus, PDL did not have a statistically
significant effect upon SILSSAT scores beyond the influence of prior general achievement.
CRTL Research Report Series, RR-0304-01
38
Evaluation of the PDL Programs in Community School District 20
TABLE 16
PDL FOR MIDDLE SCHOOL SOCIAL STUDIES TEACHERS
SUMMARY OF STEPWISE MULTIPLE REGRESSION ANALYSIS FOR THE
EFFECTS OF PDL ON GRADE 8 STUDENTS SOCIAL STUDIES SCALE SCORES
CONTROLLING FOR 2002 ELA SCALE SCORES
School
B
Step
1
2
Variables
Entered
ELA
PDL
R
0.839
0.840
Adjusted R
Square
0.703
0.704
R Square
Change
0.703
0.002
F
Change
1116.35
2.90
df
1,471
1,470
p
<.0001
0.089
D
1
2
ELA
PDL
0.711
0.720
0.504
0.514
0.506
0.013
230.26
5.99
1,225
1,224
<.0001
0.015
E
1
2
ELA
PDL
0.800
0.801
0.638
0.639
0.639
0.002
583.45
2.02
1,329
1,328
<.0001
0.156
H
1
2
ELA
PDL
0.644
0.644
0.413
0.410
0.415
0.000
180.91
0.03
1,255
1,254
<.0001
0.862
All Above Schools
Combined
1
2
ELA
PDL
0.811
0.811
0.658
0.657
0.658
0.000
2471.85
0.14
1,286
1,285
<.0001
0.712
CRTL Research Report Series, RR-0304-01
39
Evaluation of the PDL Programs in Community School District 20
V. CONCLUSIONS AND RECOMMENDATIONS
This section presents the conclusions of the evaluation in the form of answers to the
questions that it was designed to address. These conclusions are based on the analyses of the
wide range of data that has been collected on the implementation of the PDL program in CSD 20
during the 2002 – 2003 school year and its effects upon program participants, including the
implementers and the recipients of program services. Following the conclusions,
recommendations are offered that are aimed at enhancing future program implementation and its
effects.
Conclusions
Evaluation Question 1: How well did PDL prepare NTF’s and RT’s for their
mentoring and modeling roles, respectively, in the new teacher program?
The findings of this report suggest that the PDL training provided to NTF’s was effective
and appropriate. NTF’s reported that the various training components aimed at preparing them
for their mentoring roles, including the Author Study Exchanges, weekly NTF meetings, and the
NTF Institutes, were useful to them. The LTPLP sessions, though not directly affiliated with the
PDL for New Teachers program, were also noted as highly useful training opportunities.
Observations of the sessions suggested that NTF’s were given opportunities to reflect on and
practice learned skills. NTF’s characterized the varied training opportunities offered to them as
reinforcing and enhancing newly learned skills. They did not perceive them to be repetitive in
any way. Data collected from the PDAQ questionnaire support this conclusion, indicating that
NTF’s responded to the most important needs of their new teacher interns. This suggests that
PDL effectively prepared NTF’s for their roles as mentors. The sample IPAL’s collected also
indicate that NTF’s assisted their interns using techniques learned during PDL training sessions,
including instructional strategies that focused on the needs of the students, mapping and
diagramming lessons, and test preparation.
Other data collected pointed to school-level contextual obstacles that had to be overcome
by the NTF’s in their work with their interns. NTF’s commented that in some cases their
mentoring schedules were interrupted by emergency coverages that were assigned to their
interns. Also, several of the NTF’s assumed other functions in the schools, and they noted that
the additional responsibilities often interfered with their mentoring responsibilities. Many of the
NTF’s also indicated that they were not formally introduced to the staff, making it difficult for
them to encourage the participation of veteran teachers in their intern’s training.
Evaluation Question 2: In the perception of new teachers, to what extent did program
services address their most important staff professional development and support
needs?
CRTL Research Report Series, RR-0304-01
40
Evaluation of the PDL Programs in Community School District 20
Evidence from the PDAQ indicates that new teachers perceived the PDL for New
Teachers program to be responsive to their most important professional development and support
needs, including motivating students, classroom management, lesson planning, and clarifying
instructional goals. The sample IPAL’s collected also suggest that NTF’s responded to the needs
of their interns. IPAL’s consistently pointed to support provided in the areas of classroom
management and lesson planning, two areas reflected in the PDAQ as important to the new
teachers.
Additionally, many of the teachers who participated in the Cycle Program reported that
they benefited from the program’s activities, particularly the opportunity to observe a veteran
teacher using instructional strategies that incorporated group work, manipulatives, and multiple
intelligences. Participants did note that the program could have been enhanced with a second
round of visits to the RT’s classrooms. Some participants indicated that RT’s should have been
given the opportunity to visit the VT’s to observe their implementation of strategies learned
during the Cycle Program. However, one participant did note that the time commitment for the
Cycle Program was extensive, and any additional components would interfere with both the RT’s
and VT’s students learning.
Overall, the evidence suggests that the PDL for New Teachers program had a structured
model for implementation, but also had sufficient flexibility to meet the individual professional
development needs of new teachers.
Evaluation Question 3A: To what extent did new teachers believe that the program
facilitated their professional growth in the competencies of effective teachers?
Multiple analyses of the data collected from the EECI suggest that there was a
relationship between PDL support and those areas where teachers believed themselves to be
most effective. The data also suggests that more program participants felt competent and
supported in the domain of Planning and Preparation, which included areas such as content
knowledge, assessment, and goal setting. The relationship between the teachers’ beliefs about
their competencies and the support they received from PDL was also strongest in this domain.
The evidence suggests that on average, fewer teachers believed themselves to be highly
competent in the domains of Classroom Environment, Instruction, and Professional
Responsibilities. Additionally, a notably low number of respondents indicated high skill levels
in the areas of gaining access to school resources for students and engaging families.
Correspondingly, a low number of respondents indicated high levels of support from PDL in
those same areas.
Evaluation Question 3B: To what extent did social studies teachers believe that the
program facilitated their professional growth in the competencies of effective teachers?
The evidence collected suggests that participants in the PDL for Middle School Social
Studies Teachers program believed the program facilitated their professional growth as teachers.
The Social Studies Feedback Survey and PDL’s internal program feedback forms indicate that
CRTL Research Report Series, RR-0304-01
41
Evaluation of the PDL Programs in Community School District 20
participants believed their ability to integrate technology, primary source documents, literature,
and cooperative learning strategies into their classrooms was enhanced by this PDL program.
Some participants did note, however, that their ability to integrate technology in their practice
was hampered due to their schools’ lack of technological resources. This varied from school to
school.
Evaluation Question 4A: Did new teachers served by the program show an increase in
their sense of efficacy as a teacher?
The evidence with respect to PDL’s impact on new teachers’ sense of efficacy was
inconclusive. There was no significant increase in the new teachers’ sense of efficacy, due in
part to the low return rate on the post-TES. Additionally, the literature regarding teacher
efficacy suggest that the construct of “general teacher efficacy” may be problematic and difficult
to measure.
Evaluation Question 4B: Did social studies teachers served by the program show an
increase in their sense of efficacy as a teacher?
The evidence with respect to PDL’s impact on the middle school social studies teachers’
sense of efficacy was similarly inconclusive. There was no significant increase in the social
studies teachers’ sense of efficacy, due in part to the low return rate on the post-TES.
Additionally, the cautionary literature cited above obtains here as well.
Evaluation Question 5A: To what extent did new teachers served by the program
demonstrate the professional competencies of effective classroom teachers?
The evidence collected for this evaluation suggests that, on average, participants in the
PDL for New Teachers program were either partially proficient or were approaching proficiency
in the professional competencies of effective teachers. The data from the DRTO demonstrated a
slight increase in the mean scores of the sample of teachers over time. There was a notable
increase in the mean scores of six areas. Four of the areas were in the Planning and Preparation
domain, and the remaining two were in the Instruction and Classroom Environment domain,
respectively. Four of these six items showed sample teachers moving from a classification of
“partially proficient” to “approaching proficiency” in the professional competencies of effective
teachers. In general, the changes in mean scores were in the expected direction. It is expected
that with a larger sample and/or a longer intervention, the difference in mean scores would have
been greater.
Evaluation Question 5B: To what extent did social studies teachers served by the
program demonstrate the professional competencies of effective classroom teachers?
The data collected revealed that the sample of social studies teachers who participated in
the observation phase of the research generally demonstrated at least “partial proficiency” in the
professional competencies of effective classroom teachers. In three of the four cases, scores
CRTL Research Report Series, RR-0304-01
42
Evaluation of the PDL Programs in Community School District 20
generally remained constant or increased slightly in the second observation. Conversely, in the
fourth case, the teacher’s ratings decreased in a number of the twenty-nine areas from
“proficient” to “partially proficient.”
Evaluation Question 6: Were the retention rates of new teachers served by the program
better than new teachers in comparable schools that were not?
The analysis of new teacher retention data showed a trend toward a favorable effect of
PDL upon reducing the rate of new teachers leaving the New York City school system. For both
cohorts of PDL schools, the attrition rates showed a slight increase in the first year of service, but
far less than the increase in comparison schools. Further, there was a decline in system leavers
for the schools that were served by PDL for a second year. The data on new teachers who stayed
in the same school were more equivocal. Teacher retention is a complex issue and is affected by
many factors. Moreover, the evaluation of the program took place under the influence of the
confounding effects of new state regulations and the systemic restructuring of the New York City
public schools. The complex factors that affect new teacher retention are exemplified by the
stories of 44 new CSD 20 teachers who required mentoring by PDL in 2002 – 2003. Data
collected internally by PDL staff showed that 33 teachers returned to their schools. Of the 11
who did not return, 4 continued to teach in the NYC DOE system, with 2 of the 4 teaching in
another of the PDL schools in CSD 20. Two of the teachers did not receive their certification,
and consequently had to leave the system, 1 left teaching to pursue a Master’s degree, another
left to pursue medical credentials, and another had lost her position due to cut-backs. It is
evident that many teachers left for reasons that could not be changed by professional
development. Within the limitations of these complexities, the data suggest that PDL has had a
valuable effect on the professional growth of teachers in CSD 20 and the probability that they
will remain in the New York City schools.
Evaluation Question 7A. Did students of new teachers served by the program show
gains in ELA achievement test scores that were equal to those for all CSD 20 middle
school students?
The analysis of ELA achievement test score data indicates that PDL had a favorable
affect on students of PDL-served teachers in CSD 20. The mean gains on ELA achievement of
the PDL group were higher than those for the non-PDL group in three out of the four schools and
four out of the six grade-within-school comparisons. Mean gains for the two groups were similar
in the other two analyses. The mean gains of the PDL group were also higher for the combinedschools analyses. The mean gains in Grade 6 were 6.35 for PDL compared to 3.03 for non-PDL,
and in Grade 7 they were 20.98 for PDL 3.03 compared to 15.80 for non-PDL. In general, PDL
had statistically significant and educationally meaningful effects on the achievement of students
of new teachers who were served by the PDL for New Teachers program.
CRTL Research Report Series, RR-0304-01
43
Evaluation of the PDL Programs in Community School District 20
Evaluation Question 7B. Were the State Intermediate Level Social Studies
Achievement Test (SILSSAT) scores of students of teachers served by the program
higher than those of students of non-PDL teachers in the same schools?
The analysis of SILSSAT scores indicate that PDL did not have a statistically significant
effect on scores beyond the influence of prior general achievement.
The mean 2003 SILSSAT scale scores tended to be higher for the non-PDL group than for the
PDL group both within schools and across the four schools combined. However, the mean 2002
ELA scale scores also tended to be higher for the non-PDL group. Accordingly, the differences
in SILSSAT means may have been attributable to differences in the general achievement of the
two groups prior to the program.
Recommendations
PDL for New Teachers and LTPLP
1. PDL and District staff should consider ways to make principals and other supervisory staff
more aware of PDL’s mission and goals, as well as the program components.
Support from the school leadership is one area that NTF’s perceived was lacking.
Successful professional development programs must have strong support from both the school
and the district leadership. An orientation and ongoing updates of the program’s progress would
facilitate the NTF’s transition into his/her role, which is particularly important for those NTF’s
that are new to the school community. In addition, it makes it clear to all parties involved that
the professional development of new teachers is a priority. The PDL and District leadership
should also communicate to the school leadership that every effort should be made to ensure that
new teachers do not receive emergency coverages or additional responsibilities that interfere
with the mentoring schedule. The same efforts should be made with respect to NTF’s in regard
to the mentoring and training schedule.
2. PDL and District staff should work with school leadership to devise ways to encourage and
facilitate structured visits to classrooms.
Some NTF’s reported that veteran staff members felt threatened by the NTF’s, and were
reluctant to allow them or new teachers to visit their classrooms. School leadership should
encourage structured visits to classrooms and conferences among veteran and new staff, without
fear of reprisal. This will assist the NTF’s in providing new teachers with models of successful
teaching strategies, as well as legitimize their work.
3. NTF’s need additional training in helping new teachers assess the impact of their
instructional approaches on their students.
All NTF’s who responded to the CRTL questionnaire indicated that they believed their
interns were positively impacting their students, yet they often could not cite “concrete
CRTL Research Report Series, RR-0304-01
44
Evaluation of the PDL Programs in Community School District 20
evidence.” Being able to cite specific evidence would not only help to enhance the new
teacher’s craft, but would also go a long way toward helping the role of PDL and the NTF
become more valued.
4. PDL should consider ways to schedule and deliver training so that NTF’s are out of the
building less often—no more than once a week, if possible.
All of the NTF’s who responded to the questionnaire noted that the various training
components were valuable and reinforcing. The biggest drawback they noted, however, was that
the numerous training sessions took them out of the building and often interfered with their
mentoring schedule.
5. PDL should consider expanding its programmatic goals and focus to include more explicit
training in knowledge of gaining access to school resources for students and working with
families.
PDL for New Teachers was responsive to the most important needs articulated by new
teachers. However, the data collected on the EECI indicated that some of the teachers’ selfratings of their professional skill and the emphasis of mentoring in some areas were relatively
low, in particular knowledge of gaining access to school resources for students and working with
families.
6. PDL should consider expanding pool of teachers eligible for The Cycle Program so more
cohorts of teachers, including new teachers who have already met the mentoring requirement,
can take advantage of this highly effective model.
The data suggest that those new teachers who participated in the Cycle Program found it
to be very beneficial. An additional component whereby the RT visits and works with the VT at
his/her home school might be added. This will encourage the VT’s to practice the skills they
learned during the initial phase of the Cycle Program. Additionally, RT’s can help the VT
modify the approaches based on the needs of the VT’s students. The Cycle Program should also
begin prior to the spring semester to avoid disrupting the classroom schedules in a concentrated
period of time.
While the above recommendations are aimed at strengthening the quality of
implementation of future replications of the program, it should be emphasized that there is
considerable evidence of the program’s positive impact upon the professional growth of new
teachers and the quality of teaching and learning in their classrooms. Particularly notable in this
regard is evidence suggesting that the program’s basic model is effective and should be retained
as a foundation.
CRTL Research Report Series, RR-0304-01
45
Evaluation of the PDL Programs in Community School District 20
PDL for Middle School Social Studies Teachers
Most of the participants in the PDL for Middle School Social Studies Teachers program
noted that learning to use technology as a resource was a positive experience. However, some
noted that it was difficult to take advantage of this new skill because their schools lacked the
appropriate resources. While this is not a PDL program flaw, participants would most benefit if
the District and the schools ensured that all participants in the program, and their students, have
access to technology resources that allow them to implement the approaches they learned.
Additionally, to better assess the individual effects of the PDL for Middle School
Social Studies program on student achievement in Social Studies, it is recommended that
evaluations of future replications pay greater attention to the collection of more complete data.
This can be achieved by identifying the students served by program teachers prospectively, i.e. at
the beginning of the program, so that plans can be developed to capture their achievement test
scores later in the project year. In addition, these achievement test data should be supplemented
by other evidence of growth in student achievement, chief among which is the structured
analysis of student work.
CRTL Research Report Series, RR-0304-01
46
Evaluation of the PDL Programs in Community School District 20
REFERENCES
Archer, J. (1999). Mentoring New Teachers. Education Week, 18(27), 1.
Black, S. (2001). A Lifeboat for New Teachers. The American School Board Journal, 188(9),
46-48. [online] http://www.asbj.com/2001/09/0901research.htm June 17, 2003
Feiman-Nemser, S. 2003. What New Teachers Need to Learn. Educational Leadership. 60(8),
58-60. [online] www.ascd.org/members/ed_lead/200305/feimannemser.html May 30, 2003
Fickel, L. H. (fall 2002). Quality Professional Development: Suggestions about Process and
Content. The Educational Forum, 67, p47-54
Gibson, S. & Dembo, M.H. (1984). Teacher Efficacy: A Construct Validation. Journal of
Educational Psychology, 76(4), 569-582.
Guskey, T.R. (2002). Does It Make a Difference? Evaluating Professional Development.
Educational Leadership, 59(6), 45-51.
Guskey, T.R. (Aug 1998). Making Time to Train Your Staff. The School Administrator Web
Edition. [online] http://www.aasa.org/publications/sa/1998_08/focGuskey.htm May 30, 2003.
Halford, J.M. (1998). Easing the Way for New Teachers. Educational Leadership, 55, 33-36.
Mauer, E., & Zimmerman, E. (2000). Mentoring New Teachers. Principal, 79(3), 26-28.
Moir, E. (1990). Phases of First-Year Teaching. [online]
http://www.newteachercenter.org/article3.html June 6, 2003.
Moir, E. & Gless, J. (2003) Quality Induction: An Investment in Teachers [online]
http://newteachercenter.org/article-CCETQInd.html June 20, 2003.
McDonald, J.P. (2002). Teachers Studying Student Work. Phi Delta Kappan, 84(2), 120-127.
Moir, E. & Bloom, G. 2003. Fostering Leadership Through Mentoring. Educational Leadership.
60(8), 58-60.[online] http://www.ascd.org/members/ed_lead/200305/moir.html May 30, 2003
O’Shea, M.R. (2002). Teaching to Standards. Leadership, 31(3), 22-23.
CRTL Research Report Series, RR-0304-01
47