Third Annual Report to the Teacher Education Accreditation Council CENTER FOR RESEARCH ON TEACHING AND LEARNING Department of Teaching and Learning 82 Washington Square East, Suite 700 New York, NY 10003 | 212 998 5872 | 212 995 3636 fax www.steinhardt.nyu.edu/teachlearn/crtl Third Annual Report to the Teacher Education Accreditation Council Robert Tobias, Director CRTL Program Evaluation Series PE-0110-01 January 2010 Center for Research on Teaching and Learning Department of Teaching and Learning The Steinhardt School of Culture, Education, and Human Development New York University © Copyright 2010 by the Center for Research on Teaching and Learning ABSTRACT This is the third annual report submitted by NYU’s Steinhardt School of Culture, Education, and Human Development to the Teacher Education Accreditation Council (TEAC). It includes (1) updates of most of the tables included in the results section of the Inquiry Brief, (2) an update of Appendix E, the evidence that the program self study relies upon, and (3) tables that describe the databases that contain the evidence. For the DRSTOS-R ratings, the mean scores for all four domains and the total scale for both BS and MA students all exceed 3.0, the value that indicates “Entry Level Proficiency” on the four-point scale. Follow-Up Surveys show that eighty percent of the respondents were teaching in their first year after graduation with 52% in NYC public schools, 5% in NYC private schools, and 23% in schools outside of NYC. New York State Teacher Certification Exam (NYSTCE) results show that the passing rates remain around 100% for the Liberal Arts and Sciences Test and the Assessment of Teaching Skills-Written (ATS-W) tests, though, the passing rates for the Content Specialty Tests (CSTs) during the total period increased by 2.7 percentage points for BS students and 3.8 percentage points for MA students For the End of Term Feedback Questionnaire, the total means meet or exceed the criterion of 4.0 for both BS and MA students, while mean total GPAs saw little variation from past years. 2 CONTENTS Abstract ........................................................................................................................... 2 List of Tables .................................................................................................................. 4 Introduction..................................................................................................................... 5 Evidence Base.............................................................................................................. 5 Updated Program Outcomes ......................................................................................... 8 DRSTOS-R .................................................................................................................. 8 Follow-Up Surveys ...................................................................................................... 9 New York State Teacher Certification Exam (NYSTCE) ...........................................12 Student Teacher End of Term Feedback Survey .........................................................14 Grade Point Averages ..................................................................................................16 Use of Data for Program Improvement ........................................................................17 Teacher Education Research Bibliography..................................................................19 3 LIST OF TABLES Table 1. Summary of Measures Cohorts and Subjects for Inquiry by Claim ......................... 7 Table 2. Mean Scores and Percents Meeting Standards on the Domain Referenced Student Teacher Observation Scale-Revised (DRSTOS-R) for Late-Placement Teacher Education Students ............................................................................................................................ 9 Table 3. Comparison of Self Perceptions of May 2007 Steinhardt Graduates and Levine Study Sample as to How Well Their Teacher Education Program Prepared Them with Important Skills and Knowledge for Teaching ............................................................................................... 11 Table 4. Mean Scaled Scores, Pass Rates, and Effect Sizes for Steinhardt Teacher Education Graduates on the New York State Teacher Certification Exams: Classes of 2006-2009 ........... 13 Table 5. Mean Scores for Claim Scales on the End of Term Feedback Questionnaire Classes of 2006 and 2007 ................................................................................................................... 15 Table 6. Mean Total NYU GPA for BS and MA Graduates by Class (Classes of 2006September 2009) ................................................................................................................ 16 APPENDIX A (TEAC APPENDIX E) ............................................................................. 21 APPENDIX B ..................................................................................................................... 28 4 INTRODUCTION Having received initial accreditation of our teacher education programs in January 2007, this is the third annual report submitted by NYU’s Steinhardt School of Culture, Education, and Human Development to the Teacher Education Accreditation Council (TEAC). Prepared according to the specifications on TEAC’s web site, the report includes (1) updates of most of the tables included in the results section of the Inquiry Brief, (2) an update of Appendix E, the evidence that the program self study relies upon, and (3) tables that describe the databases that contain the evidence. In addition, the report describes changes in the measures that are being used in the ongoing self-study of the effectiveness of Steinhardt’s teacher education programs, the most recent results from the analyses of data, and ways in which the data have been used to improve the programs. The report also includes a bibliography of papers and reports on the research that NYU faculty and research centers have performed on NYU’s teacher education programs. THE EVIDENCE BASE There have been several changes to the evidence base since the initial inquiry brief, which can be seen in the table that summarizes the measures and the data that are being collected in the ongoing self study (see Table 1.) First, we have substantially expanded the assessment of our student teachers developing pedagogical proficiency using the Domain Referenced Student Teacher Observation Scale-Revised (DRSTOS-R). We have trained more than 50 student teacher supervisors resulting in a more than fivefold increase in the number of student teachers assessed each semester, from 40 to about 220 student teachers. At the same time, DRSTOS-R data have been used to help guide the training process through incorporation into the conversations between field supervisors and student teachers concerning the latter’s progress. In addition, the data are increasingly used by program directors to inform program planning. Finally, Steinhardt’s Center for Research on Teaching and Learning (CRTL) has continued to do research on the validity and reliability of DRSTOS-R data, including studies on its criterion validity using student work samples and value-added standardized test scores. This report presents data summarizing the results of the administration of DRSTOS-R to lateplacement student teachers, i.e. those who are approaching graduation, for the fall 2007 – spring 2009 semesters (see Table 2). Second, during 2009 CRTL developed a new dispositions survey to replace the Educational Beliefs Questionnaire, which had been in use since 2004. The new survey, entitled the Educational Beliefs and Multicultural Awareness Survey, measures four constructs, adding multicultural awareness and personal teaching efficacy to the two constructs that were measured by the EBQ, general teaching efficacy and caring/social justice. EBMAS was pilot tested on 107 Fast Track program graduate teacher education students in summer 2009 and first administered to all incoming undergraduates in fall 2009. Preliminary analysis of the psychometric properties of EBMAS showed that it had stronger factorial validity and higher internal consistency reliability than the EBQ. These 5 data will be presented in a report which is currently being prepared. EBMAS data will be reported in the fourth annual report and in the Inquiry Brief for continued accreditation. Third, CRTL has expanded methods and procedures for following up cohorts of NYU teacher education program graduates in efforts to assess their post-graduates success in the profession. Students are surveyed at three points in time: an exit survey is administered to all students at program completion; a one-year follow-up survey is administered to all graduates from each BS and MA class; and a five-year follow-up survey is currently being administered to graduates of the Class of 2005. The surveys ask graduates about their employment as teachers , the extent to which their NYU education provided them with the component skills and knowledge to succeed in teaching, the specific aspects of the program that were most and least helpful to their development as teachers, any honors or promotions they had earned, and their plans for the future. In order to provide normative data, many of the survey questions were drawn from Arthur Levine’s study of schools of education and the responses of NYU graduates are compared to the frequency distributions of responses of graduates in the Levine study. This report presents some of these results from the one-year follow-up survey of graduates from the Class of 2007 (See Table 3.) Graduates are also tracked by data matches to the human resources personnel files of the New York City and New York State Departments of Education. These matches are used to determine the numbers and percentages of graduates who teach in New York City and New York State public schools, the subjects that they teach, the demographic characteristics of the student bodies, and the graduates’ rates of retention, attrition, and school and district transfer. These data on employment in New York are supplemented with data on graduates teaching out of state that are obtained through the graduate surveys mentioned above. Finally, the effectiveness of graduates who teach in New York City is being assessed through the impact they have on their pupils’ standardized NYS achievement test score data that CRTL is obtaining from the NYC Department of Education. Valueadded modeling statistical methods are being used to determine the extent to which the scores of the graduates’ pupils exceed or fall short of expected performance scores that are based on the demographic characteristics and previous test score performance of the pupils. The results for NYU graduates are compared to those for other NYC teachers with similar years of teaching experience, teaching the same subjects and grades. These data will be reported in the fourth annual report and the next Inquiry Brief. CRTL continues to collect, analyze, and report data on self-study measures, including scores on the New York State Teacher Certification Exams (see Table 4), student teacher self reports of the contributions of their field supervisors and cooperating teachers to their professional growth (see Table 5), and the grade point averages of graduates (see Table 6). All of these data are being used by program directors to inform the continuous improvement of the programs. Overall, these data indicate that NYU continues to meet its program claims with substantial improvements in outcomes for some of the measures. These results are described in the next section. 6 Table 1 Summary of Measures, Cohorts, and Subjects for Inquiry by Claim (Updated January 2010 for 2012 Inquiry Brief) Claims Measures Cohort Subjects Classes of 2007 – 11 Claim 1: Subject Matter Knowledge DRSTOS-R: Planning & Preparation Score and Total Score NYSTCE Exam Scores: Content Specialty Tests Student Teacher ETFQ Classes of 2006 – 11 Most program graduates All tested students Classes of 2006 – 11 All respondents Content Core GPA; undergrad major GPA 1 & 5 year Follow-Up Surveys Program Exit Survey* Classes of 2006 – 11 All BS graduates; all incoming MA students All respondents All program graduates Claim 2: Pedagogical Knowledge DRSTOS-R: Instruction and Classroom Environment Scores and Total Score NYSTCE Exam Scores: --ATS-W Exam Score Student Teacher ETFQ EBMAS Scales 1 & 2: Personal & General Teaching Efficacy** Pedagogical Core GPA 1 & 5 year Follow-Up Surveys Program Exit Survey* Claim 3: Teaching Skill/ Caring DRSTOS-R: Professional Responsibilities & Classroom Environment Domain Scores and Total Score Student Teacher ETFQ EBMAS Scale 3: Caring/Social Justice** Student Teaching GPA 1 & 5 year Follow-Up Surveys Program Exit Survey* Pupils’ value-added standardized test scores*** Liberal Arts Cross-Cutting Themes Multicultural Awareness Classes of 2007 – 11 Classes of 2006, 2008 – 11 Classes of 2007 – 11 Most program graduates Classes of 2006 – 11 All tested students Classes of 2006 – 11 All respondents Class of 2010 – 11 All respondents Classes of 2006 – 11 Classes of 2007 – 11 Classes of 2006, 2008 – 11 Class of 2007 – 11 All program graduates All respondents All respondents Most program graduates Classes of 2006 – 11 Classes of 2010 – 11 All respondents All respondents Classes of 2006 – 11 Class of 2007 – 11 Classes of 2006,, 2008 – 11 Classes of 2004 – 2010 All program graduates All respondents All program graduates DRSTOS-R: Total Score Class of ‘05 NYSTCE Exam Scores: --Liberal Arts & Sciences Classes of ’04 & ‘05 All program graduates teaching grades 4 – 8 in NYC public schools Most program graduates All tested students Liberal Arts Core GPA; Total undergrad GPA Classes of ’04 & ‘05 All BS graduates; all incoming MA students EBMAS: Scale 4: Multicultural awareness Classes of 2010 – 11 All program graduates Note: Bold type signifies new measures that were not included in the original Inquiry Brief. * Replaces the Fast Track End of Program Questionnaire (FTEPQ) ** Replaces the Educational Beliefs Questionnaire (EBQ) *** Also used as a measure of subject matter knowledge and pedagogical knowledge 7 UPDATED PROGRAM OUTCOMES DRSTOS-R Table 2 presents DRSTOS-R ratings for students in their final student teaching placement for the two-year period fall 2007 thru spring 2009 for 58 BS students and 208 MA students. Similar data were reported in the first Inquiry Brief. The mean scores for all four domains and the total scale for both BS and MA students all exceed 3.0, the value that indicates “Entry Level Proficiency” on the four-point scale. All of the means exceed those reported in the first Inquiry Brief by .10 - .35 proficiency-scale points. The gains are especially noteworthy for the MA students. The first Inquiry Report highlighted the need for improved performance on DRSTOS-R by MA students, since their performance was below the program target criterion of at least 80% attaining a mean score of 3.0 in all four domains and the total scale score. As seen in Table 2, the gains by the new cohort of MA students enabled them to meet the program criterion for all five scores. Score trends for BS students are more ambiguous. Although they show higher mean scores than they the Inquiry Brief cohort, they met the program criterion on only one of the five scores, Professional Responsibilities, compared to four out of five in the first Inquiry Brief. The improved performance of the MA students can be linked to the strengthening of the programs through a three-year partnership with the NYC Department of Education and the City University of New York that was designed to upgrade the MA programs through improved field experiences and enhanced course curricula. Disaggregations of the data by program show support this inference, since the greatest gains occurred in the programs that were targeted by the partnership initiative—science, mathematics, and teaching English as a second language. 8 Table 2 Mean Scores and Percents Meeting Standards on the Domain Referenced Student Teacher Observation Scale-Revised (DRSTOS-R) for Late-Placement Teacher Education Students Number of Items Years Claims Scale Domain Fall 07/ Spring 09 1 3 2 3 1,2,3 Crosscutting Theme Planning and Preparation Classroom Environment Instruction Professional Responsibilities 6 7 5 3 Total Score 21 Mean Score (Scale: 1-4) Standard Deviation % Meeting Standards (Mean=>3.0) 58 58 58 57 3.27 3.31 3.23 3.46 .66 .61 .64 .68 77.6% 74.1% 75.9% 87.7% 58 3.30 .61 75.9% Number of Students BS Students MA Students Fall 07/ Spring 09 1 3 2 3 1,2,3 Crosscutting theme Planning and Preparation Classroom Environment Instruction Professional Responsibilities 6 7 5 3 208 208 208 208 3.31 3.36 3.30 3.57 .48 .46 .46 .50 84.1% 82.2% 79.8% 90.9% Total Score 21 208 3.36 .43 80.3% Notes. Nominal scale values are as follows: (1) Not Yet Proficient; (2) Partially Proficient; (3) Entry Level Proficient; (4) Proficient. The standard for program graduates is Entry Level Proficient (3). Follow-Up Surveys In spring 2008, 95 (30%) of May 2007 graduates responded to an on-line survey constructed by CRTL in consultation with faculty from teaching and learning and applied psychology and representatives from the NYC Department of Education. The response rate is typical for on-line surveys and the respondents were closely representative of the total population of graduates by degree and certification program. Eighty percent of the respondents were teaching in their first year after graduation with 52% in NYC public schools (the same statistic obtained through CRTL electronic tracking studies using NYC human resources files), 5% in NYC private schools, and 23% in schools outside of NYC. Table 3 compares respondents’ perceptions of how well NYU teacher education programs prepared them for teaching in 11 important skills and knowledge areas to the responses of a national sample of teacher education graduates from Art Levine’s 2006 study of schools of education. (The table also shows results for four items that were not included in Levine’s study.) The pattern of responses for the NYU graduates closely tracks that for Levine’s sample. However, higher percentages of NYU graduates reported feeling Very Well to Moderately Well prepared in nine of the 11 areas, with differences 9 that were statistically significant in four areas: Maintain order and discipline; address needs of ELLs; address needs of students from diverse cultures; and integrate technology into subject and grade level taught. The programs received especially high ratings in specific teaching skills and knowledge, including understanding how students learn, using different pedagogical approaches, and impacting students’ (i.e., pupils’) ability to learn. Graduates felt they were less prepared to work with parents, participate as a stakeholder in the communities in which they teach, and to address the needs of students with limited English proficiency, even though they felt significantly better prepared in the latter than Levine’s sample. These results further indicate the need to continue to forge partnerships with schools in order to strengthen the field experience by promoting the inclusion of student teachers in the school community. Indeed, respondents perceived student teaching to be the most valuable component of the teacher education program and commented that Inquiries, methods courses, and student teaching seminars were the most helpful courses in preparing them for teaching. They also evinced a strong need for mentoring to assist them with induction into the profession. Only about one-third were receiving mentoring from their schools/districts and nearly two-thirds welcomed the offer of additional support from NYU. This has important implications for planning program extensions that include continued training beyond graduation. 10 Table 3 Comparison of Self Perceptions of May 2007 Steinhardt Graduates and Levine Study Sample as to How Well Their Teacher Education Programs Prepared Them with Important Skills and Knowledge for Teaching 1 2 3 Somewhat Well 17 17.9 20 21.1 20 21.1 14 14.7 19 20 28 29.5 18 18.9 25 26.3 34 4 Not Well at All 2 2.1 7 7.4 7 7.4 2 2.1 3 3.2 8 8.4 13 13.7 14 14.7 25 N % N % N % N % N % N % N % N % N Very Well 33 34.7 29 30.5 29 30.5 39 41.1 33 34.7 31 32.6 31 32.6 21 22.1 15 Moderately Well 43 45.3 39 41.1 39 41.1 40 42.1 40 42.1 28 29.5 33 34.7 35 36.8 21 NYU grads 76 80.0 68 71.6 68 71.6 79 83.2 73 76.8 59 62.1 64 67.3 56 58.9 36 Levine sample Z difference Sig. 74.0 1.33 NS 73.0 -0.31 NS 57.0 2.87 <.01 81.0 0.55 NS NA NA 60.0 0.42 NS 67.0 0.06 NS 60.0 -0.22 NS % N % N % N 15.8 39 41.1 18 18.9 29 22.1 31 32.6 24 25.3 36 35.8 16 16.8 28 29.5 21 26.3 9 9.5 25 26.3 9 37.9 70 73.7 42 44.2 65 27.0 2.39 <.01 52.0 4.23 <.01 43.0 0.24 NS % N 30.5 19 37.9 35 22.1 23 9.5 18 68.4 54 NA NA % N 20.0 14 36.8 26 24.2 32 18.9 23 56.8 40 NA NA % N % 14.7 23 24.2 27.4 30 31.6 33.7 22 23.2 24.2 20 21.1 42.1 53 55.8 NA NA 41.0 2.93 Did your program prepare you to: Use different pedagogical approaches Have a mastery of your subject area Maintain order & discipline in the classroom Understand how students learn Impact my students' ability to learn Implement state/district curriculum & standards Use student performance assessment techniques Address needs of students with disabilities Address needs of students with limited English proficiency Address needs of students from diverse cultures Work with parents Work collaboratively with teachers, admins., & personnel Identify & use resources within the community where you teach Participate as a stakeholder in the community where you teach Integrate technology into the grade level or subject taught Sum 1&2 <.01 Source: First-Year Feedback Survey of spring 2007 NYU Teacher Education Graduates NA = Skill/knowledge not included in the Levine study survey NS = Not statistically significant 11 New York State Teacher Certification Exams (NYSTCE) Table 4 reports the scores of NYU teacher education graduates for the classes of 2006 – 2009 on the NYSTCE exams that teacher candidates must pass in order to be eligible to teach in New York State public schools. The table reports mean scaled scores, effect sizes, which are defined as the number of standard deviations (SDs) above the passing score of 220, and pass rates for the three sets of exams required of candidates. The passing rates remain around 100% for the Liberal Arts and Sciences Test and the Assessment of Teaching Skills-Written (ATS-W) tests, similar to the values reported in the initial Inquiry Brief for the Classes of 2004 and 2005. However, the passing rates for the Content Specialty Tests (CSTs) during the total period increased by 2.7 percentage points for BS students and 3.8 percentage points for MA students. Moreover, the effect sizes for the CSTs increased by one-half SD for BS students and one-third SD for MA students. In addition, the effect sizes for the ATS-W increased by 0.44 SD and 0.55 SD for BS and MA students, respectively. New data released by the Teacher Quality Research Consortium, a pilot project funded by the New York State Education Department (NYSED) designed to link data from the NYSED with data from institutions of higher education (IHE) in NYS, indicate that the CST mean scaled scores of NYU graduates exceed those of all IHEs combined by 0.40 SD for BS students; they are about equal for MA students. Since NYU’s BS students study their content area subject matter in NYU’s College of Arts and Sciences while MA students take these courses at their respective undergraduate institutions, these findings reflect positively on the quality of NYU’s content major courses. Similarly, the comparative advantage for NYU graduates on the ATS-W is evidence of the quality of the Steinhardt School’s pedagogical courses and seminars. Disaggregated analysis of the CST data by subject area indicate that NYU BS math graduates showed especially high scores on the math CST compared to their state-wide peers, with a mean scaled score 0.75 SD higher. Effect sizes for state-wide comparisons in English and Social Studies were also positive but much smaller at 0.20 SD for both. The improvement in CST scores is partially attributable to greater coordination and communication between Steinhardt and the College of Arts and Sciences. The Deans and program directors of the two schools meet and confer about policy and programs at the Teacher Education Council and faculty of the two schools take actions to implement policy at the Teacher Education Working Group. The particularly strong gains in math are evidence of the effectiveness of recent programs aimed at upgrading math education including the Partnership for Teaching Excellence and the Gateways program. 12 Table 4 Mean Scaled Scores, Pass Rates, and Effect Sizes for Steinhardt Teacher Education Graduates on the New York State Teacher Certification Exams: Classes of 2006-2009 Class of 2006 BS MA Class of 2007 BS MA Class of 2008 BS MA Class of 2009 BS Totals MA BS MA Liberal Arts & Sciences Test (LAST) N Tested 92 254 78 311 89 312 89 302 348 1179 Mean Scaled Score 268.7 270.5 268.1 267.4 268.1 268.0 268.0 270.7 267.7 269.2 15.1 18.0 16.0 17.6 18.7 17.5 33.3 16.0 22.1 17.3 3.23 2.80 3.01 2.69 2,47 2.78 1.44 3.17 2.54 2.86 100% 99.6% 100.0% 99.0% 98.9% 99.7% 98.9% 100.0% 99.4% 99.6% Standard Deviation SDs Above Passing * Percent Passing Assessment of Teaching Skills: Written (ATS-W) N Tested 92 254 78 311 89 312 83 322 337 1262 Mean Scaled Score 267.3 268.0 266.7 269.1 269.8 268.3 272.9 268.0 269.0 268.4 13.0 14.8 15.3 15.2 13.4 15.7 13.4 13.6 13.9 14.9 3.64 3.23 3.05 3.23 3.73 3.07 3.93 3.56 3.59 3.27 100.0% 99.6% 100.0% 100.0% 100.0% 99.7% 100.0% 100.0% 100.0% 99.8% N Tested 81 316 75 351 86 320 77 305 319 1292 Mean Scaled Score 249.4 251.5 244.1 255.5 250.3 256.0 254.9 257.6 249.7 255.1 19.2 21.5 34.0 19.6 17.6 19.4 16.3 21.6 22.9 20.6 1.53 1.46 0.71 1.81 1.72 1,85 2.14 1.74 1.52 1.72 92.6% 95.6% 93.3% 97.7% 98.8% 9.1% 100.0% 97.1% 96.2% 97.4% Standard Deviation SDs Above Passing* Percent Passing Content Specialty Tests Standard Deviation SDs Above Passing* Percent Passing *Effect Sizes 13 Student Teacher End-of-Term Feedback Surveys NYU uses the on-line End-of-Term Feedback Questionnaire (ETFQ) to assess the quality and effectiveness of each term’s student teaching experience. The survey asks student teachers to rate the effectiveness of their cooperating teachers and field supervisors in helping them to develop their expertise in specific aspects of teaching using a five-point Likert scale ranging from (1) Very Poorly to (5) Very Well. The program target is a mean score of 4.0 for each area. For subject matter knowledge, student teachers are asked how well cooperating teachers and field supervisors helped them to develop the content knowledge specific to their field and age group; for pedagogical knowledge they were asked how well their mentors’ provided assistance in furthering their organizational teaching skills; and for teaching skill and caring they were asked about the assistance they received in enhancing their teaching practice and developing classroom management skills. Table 5 displays the mean ETFQ scores and effect sizes for the items related to each of the three claims. The effect sizes represent the difference between the criterion of 4.0 and the mean scores in standard deviation units. The total means for all three claims meet or exceed the criterion of 4.0 for both BS and MA students. All three means for MA students are higher than the respective values reported in the initial Inquiry Brief for the classes of 2004 and 2005, which were below 4.0 for two of the three claims, subject matter knowledge and teaching skill and caring. All three effect sizes for MA students increased, with gains of 0.12 SD. for both subject matter knowledge and pedagogical knowledge and 0.23 SD for teaching skill and caring. Moreover, mean ratings for MA students were higher in 2007 than in 2006, gains that coincided with enhancements in the field component that accompanied the implementation of the Partnership for Teaching Excellence. 14 Table 5 Mean Scores for Claim Scales on the End of Term Feedback Questionnaire Classes of 2006 and 2007 Class of 2006 BS MA Class of 2007 BS Totals MA BS MA Claim 1 Scale: Subject Matter Knowledge N Respondents Mean Score 150 192 150 215 300 407 3.93 3.96 4.11 4.03 4.02 4.00 Standard Deviation (SD) 0.84 0.86 0.75 0.81 0.80 0.83 Effect Size (ES) -0.08 -0.05 0.14 0.03 0.03 0.00 Claim 2 Scale: Pedagogical Knowledge N Respondents Mean Score 150 192 151 216 301 408 4.04 4.07 4.26 4.17 4.15 4.12 Standard Deviation (SD) 0.82 0.79 0.68 0.76 0.75 0.77 Effect Size (ES) -0.05 0.09 0.38 0.22 0.20 0.16 Claim 3 Scale: Teaching Skill and Caring N Respondents Mean Score 150 192 151 216 301 408 4.09 4.06 4.22 4.18 4.16 4.12 Standard Deviation (SD) 0.71 0.8 0.7 0.72 0.70 0.76 Effect Size (ES) 0.13 0.08 0.31 0.24 0.22 0.16 Notes: Claim 1 scale items are Items 9 and 18; Claim 2 scale items are Items 7 and 15; and Claim 3 scale items are Items 8, 11, 16, and 19. Items are measured on a 5-point Likert scale with scale values of (1) “Very Poorly”, (2) “Poorly”, (3) “Average”, (4) “Well”, and (5) “Very Well”. The criterion for each claim is a mean score of at least 4.0. The formula for effect size is as follows: ES = (Mean Score – 4.0)/SD. Students left some items blank, resulting in different numbers of respondents for the claim scales. 15 Grade Point Averages (GPAs) Among the measures that NYU uses as evidence to assess its claims are GPAs for subsets of courses related to each of the claims. The initial Inquiry Brief included separate GPAs for the content cores courses to assess subject matter knowledge, pedagogical core courses to assess pedagogical knowledge, student teaching and student teaching seminars to assess teaching skill and caring, and Morse Academic Plan courses to assess the cross-cutting theme of liberal arts knowledge. This annual report includes total mean GPAs for BS and MA students for the graduates of the classes of 2006 – September 2010 (see Table 6.) Mean GPAs for these classes combined are 3.51 (SD = 0.29) for BS students and 3.79 (SD = 0.25) for MA students with little year-to-year variation in means and dispersion. CRTL is currently performing the data management work required to prepare these data for disaggregation and analysis by claim. These results will be presented in the fourth annual report. Table 6 Mean Total NYU GPA for BS and MA Graduates by Class (Classes of 2006 – September 2009) Class Degree Program BS MA Total 2006 3.48 131 2007 3.53 117 2008 3.51 126 2009 3.53 133 2010 * 3.33 5 Total 3.51 512 0.28 0.33 0.30 0.27 0.28 0.29 Mean N Std. Deviation 3.85 451 3.86 513 3.86 490 3.85 513 3.84 93 3.86 2060 0.17 0.17 0.23 0.15 0.20 0.18 Mean 3.77 3.80 3.79 3.78 3.81 3.79 N 582 630 616 646 98 2572 Std. Deviation 0.25 0.25 0.28 0.22 0.23 0.25 Mean N Std. Deviation * September 2009 graduates 16 USE OF DATA FOR PROGRAM IMPROVEMENT Steinhardt deans, chairs, program directors, and faculty have been making increased use of the performance data to inform program planning and improvement. As described above, data are analyzed and reviewed at meetings of the Teacher Education Council, the Teacher Education Working Group, and meetings of the teacher education program directors. They are also reviewed by the Office of Field Services to assess the effectiveness of field services, including the quality of the environment for pre-service training at specific school sites and the ability of individual cooperating teachers and field supervisors to provide high-quality mentoring to student teachers. Overall trends in the data are analyzed as well as disaggregated outcomes for specific certification areas. Where possible, data are analyzed longitudinally and historically to identify trends over time and comparisons are made to program standards and published data from other institutions of higher education, as well as available normative data. There has been a cultural shift in the use of data for program decision-making by the Department of Teaching and Learning. Review of the data from the evidence base has become a regular part of the monthly program directors’ meetings. The data are disaggregated by program and used in a needs-assessment process to identify areas for institutional reform. Reciprocally, perceptions of problems in curricula and field work are validated and illuminated by analyses of the data, often leading to revisions and expansion of data collection and analysis. The development of rapid feedback systems has increased our capacity to identify and deal with problems quickly. This includes quality-control systems that process data from course evaluations and student teacher feedback questionnaires in ways that allow for rapid identification of problems and quick corrective actions. In the past few years, this process of data review and planning has resulted in many actions that have changed Steinhardt’s teacher education programs, several of which are already yielding improved outcomes as described in the section above. Some of these key actions are summarized below. • • In recognition of the need to strengthen the content knowledge of teacher education students, structures and processes have been developed to foster closer collaboration and communication between Steinhardt and the College of Arts and Sciences. As a result, curricula have been modified in the areas of social studies and science and courses in these areas are now co-taught by Steinhardt and Arts and Sciences faculty. Increases in NYSTCE CST scores may be tied to these changes. In order to strengthen the field component, partnerships were forged between NYU and New York City public schools. These partnerships are designed to be of mutual benefit to the public schools and NYU’s teacher education program. For NYU’s student teachers, the partnerships have changed the focus of the field experience from isolated mentorship by a single cooperating teacher to greater integration of student teachers in the school community. Surveys used in the evaluation of these partnerships have found that student teachers in partnership 17 • • • • • schools report a broader and richer range of field experiences, including participation in professional development activities, parent events, and school functions, as well as inter-class visitations. These experiences may be linked to the higher ETFQ ratings described in the section above. To address the need to better connect theory to practice, the foundational core course for teacher education students, Inquiries, was moved off campus and is currently conducted in partnership schools and co-taught by Steinhardt faculty and public school teachers. On-site course delivery provides students opportunities to experience and observe the application of educational principles and theories in situ. Analysis of recent course evaluation data provides some evidence in support of the perceived effectiveness of this model. To address the need for increased coherence and consistency between courses and field work that merged from analyses of program exit surveys, the BS dual certification program in childhood and special education is piloting a new model of student teaching for juniors in four cohorts at four public schools during the 2009-10 academic year. Student teachers in the pilot spend full days in their schools supervised by a methods instructor who is responsible for the full cohort in the school. This pilot is being evaluated and, if successful, will be expanded in the next academic year. To increase the educational relevance of the course in adolescent development, a need identified through course evaluations, the curriculum was redesigned. Comparative analyses of course evaluation data are being conducted to assess the success of the redesigned course. To address the need for more instruction in assessment, especially in the area of literacy at the secondary level, a team of faculty has undertaken the adoption and adaptation of on-line modules that demonstrate assessment techniques and ways to identify and teach pupils with problems in literacy. These modules are also designed to ensure that there is consistency across the programs in the coverage of the topics of assessment and literacy. In response to the need for mentorship during induction that was identified through one-year follow-up surveys, the Department of Teaching and Learning is designing a program that incorporates post-graduate, in-service mentorship into its pre-service teacher education programs. This program, tentatively entitled InTouch, will include a planned mentorship component during the graduates’ induction year as part of its curriculum. NYU is using its evidence base to evaluate the impact of the above program reforms and to continue to identify areas in need of improvement. 18 TEACHER EDUCATION RESEARCH BIBILIOGRAPHY Consistent with its culture of self inquiry, Steinhardt research centers, faculty, and invited external researchers conduct ongoing research on Steinhardt’s teacher education programs. Below is a list of reports and papers produced form this research. Alter, M., Rust, F., Jeffery, J. & Keane, A. Closing the teacher preparation gap: gathering evidence of student learning. A paper funded through a Petrie Foundation grant, July 2008. Hummel-Rossi, B., Tobias, R., & Ashdown, J. Creating usable to improve teacher education programs serving urban public schools. A paper delivered at the annual meeting of the AERA, San Diego, April 2009. CRTL research report series RR-0409-1 Hummel-Rossi, B. Tobias, R., Ashdown, J., & Smith, A. Teacher education’s responsibility to its metropolitan constituents: A longitudinal value-added study. A paper presented at the annual meeting of AERA, New York City, March 2008. CRTL research report series RR-0308-1. Jeffery, J. & Tobias, R. Circle of inquiry: partnership researchers’ perspectives on school-university collaborative processes. A paper delivered at the annual meeting of the AERA, San Diego, April 2009. CRTL research report series RR-0409-1. Meier, J. & Crowe. Evaluation of partnership for teacher excellence. An evaluation report prepared by Arete Consulting for the Petrie Foundation, April 2009. Sirin, S. & Collins, B. Graduate teacher education at New York University: An exploration of backgrounds, perceptions, and expectations. New York University: New York, 2009. Taub, A., Tobias, R. & Mayher, J. Inquiry brief: a self-study of NYU’s teacher education programs. A report to TEAC for initial accreditation, November 2005. Tobias, R. & Tian, J. & Woo, K. First-year feedback survey of spring 2008 NYU teacher education graduates. CRTL research report series RR-0909-1, September 2009. Tobias, R., Woo, K. & Pignatosi, F. Are we developing high quality teachers and can we prove it? A study of the validity and utility of a pre-service teacher assessment system. A paper presented at the annual meeting of the AERA, San Diego, April 2009. CRTL research report series RR-0409-2. Tobias, R. & Bang, H. First-year feedback survey of spring 2007 NYU teacher education graduates. CRTL research report series RR-0608-1, June 2008. 19 Tobias R., Fan, B., & Bang, H. Measuring the developing dispositions of pre-service and beginning teachers. A paper presented at the annual meeting of AERA, New York City, March 2008. CRTL research report series RR-0308-2. Tobias, R. & Robin, J. The effects of multiple-perspective assessment upon student teachers and their pupils. Final report for TNE mini-grant 3571-00/EOC 6820. CRTL research report series RR-0707-1, July 2007. Tobias, R. & Barrett, T. Graduating students’ perceptions of the effectiveness of their teacher preparation programs: Steinhardt Graduating class of May 2006. CRTL research paper series RR-0307-1, March 2007. Tobias R. & Robin, J. Analysis of the Fast Track Learning Partners Survey, Summer 2007, CRTL research report series PE-0307-1, March 2007. Tobias, R. Results of the first follow-up study on graduates of NYU’s Steinhardt School of Education’s teacher education programs: graduates from the Classes of 2001 – January 2005 teaching in New York State. CRTL research report series RR-0705-01, July 2005. Tobias, R. Analysis of student course reaction forms for Department of Teaching and Learning courses during the period spring 2003 – summer 2004. CRTL research report series RR-0904-1, September 2004. Tobias, R. & Fan, B. Fast track evaluation report: analysis of student perceptions of the effectiveness of the Fast Track summer session 2003. CRTL research report series PE1203-01, December 2003 20 Third Annual Report to TEAC, January 2010 ________________________________________________________________________________________________ Appendix A NYU Steinhardt’s Third Annual TEAC Report: Appendix E (TEAC Inquiry Brief): Evidence January 2010 Type of Evidence Note: items under each category are examples. Program may have more or different evidence Available* In the Brief Reasons for including the results in the Brief (Location in Brief) Not Available Not in the Brief Not for future use For future use Reasons for not including the results in Reasons for including in future Briefs Reasons for not including in future Briefs the Brief Grades 1. Student grades and grade point averages Content area GPA, pedagogical course GPA, and student teaching GPA continue to be used as valid and reliable measures of all three claims. (File 1 in Appendix B.)** Scores on standardized tests 2. Student scores on standardized license or board examinations Scaled scores on the NYSTCE content area and Assessment of Teaching Skills exams continue to be used as valid, reliable, and sensitive measures of content knowledge and pedagogical knowledge, while scaled scores on the Liberal Arts and Sciences Test continue to be used to assess the cross-cutting theme of general academic knowledge. (File 4 in Appendix B.)** 3. Student scores on admission CRTL Program Evaluation Series, PE-0110-01 Standardized admissions exams, 21 Third Annual Report to TEAC, January 2010 ________________________________________________________________________________________________ tests of subject matter knowledge for graduate study such as the SAT and GRE are optional for admission to the graduate teacher education program. To the extent that these scores are available, they will be reported as ancillary data in future briefs. 4.Standardized scores and gains of the program graduates own students Standardized test score gains for the pupils of program graduates were not available for NYU’s initial Inquiry Brief. However, these data are currently being collected and analyzed for the pupils of program graduates who are teaching in grades 4 thru 8 in New York City public schools (the majority of program graduates) and will be presented in the Fourth Annual Report and next Inquiry Brief using value-added analytic methods. NYU is exploring the possibility of obtaining and using standardized test scores for the pupils of graduates who are teaching high school students. The data file was constructed in December, 2008 and includes the value-added test data of 150 graduates. It was augmented with the data for 190 graduates in 2009. CRTL Program Evaluation Series, PE-0110-01 22 Third Annual Report to TEAC, January 2010 ________________________________________________________________________________________________ Ratings 5. Ratings of portfolios of academic and clinical accomplishment Portfolio data were not included in the original Brief and will not be used in future Briefs. Attempts to develop a standard rubric to score portfolios proved unsuccessful due to insufficient inter-rater agreement. 6. Third-party rating of program's students 7. Ratings of in-service, clinical, and PDS teaching CRTL Program Evaluation Series, PE-0110-01 NYU considered using third-party ratings of program students but determined the procedures to be not feasible logistically. NYU did not have access to inservice ratings of program graduates for the first Brief. We are in discussion with the NYC Department of Education (NYCDOE) to try to obtain these ratings for future Briefs. The success of these efforts will be largely contingent upon the adoption of new teacher performance standards and rating systems by the NYCDOE, which is currently being negotiated. 23 Third Annual Report to TEAC, January 2010 ________________________________________________________________________________________________ 8. Ratings by cooperating NYU continues to expand the teacher and college/ use of the DRSTOS-R, a protocol university supervisors, of practice teachers' work samples used to assess the developing proficiency of student teachers. Data from the assessment of a sample of DRSTOS-R ratings by student-teacher supervisors were submitted in the first Brief. Data for a much larger sample will be included in future Briefs. Plans for augmenting the data with the DRSTOS-R ratings of cooperating teachers are contingent upon NYCDOE decisions about the adoption of new performance standards and assessments for teachers. (See Item 7 above.) (File 2 in Appendix B.)** Rates 9. Rates of completion of courses and program CRTL Program Evaluation Series, PE-0110-01 These data were not included in the initial Brief because the teacher education programs were in transition to conform to new state certification requirements and historical completion data were deemed obsolete. Program completion rates will be included in the Fourth Annual Report and next Inquiry Brief. (File is under construction.) 24 Third Annual Report to TEAC, January 2010 ________________________________________________________________________________________________ 10. Graduates' career retention rates 11. Graduates' job placement rates Some data on career retention rates were included in the initial Brief. NYU has expanded its capability for tracking career data for program graduates through partnerships with the New York State and New York City DOEs. Accordingly, the Fourth Annual Report and future Inquiry Briefs will include richer and more complete career retention data. (File 8 in Appendix B.)** Same as Item 10 above. 12. Rates of graduates' professional advanced study 13. Rates of graduates' leadership roles Since submission of the initial Brief, NYU has initiated a follow-up study to assess the professional success and career progress of its teacher education graduates. The follow-up study includes a survey of graduates who are working in NYC public schools, the primary employer of NYU graduates. These data will be analyzed and reported in the Fourth Annual Report and next Inquiry Brief. See Item 12 above. 14. Rates of graduates' professional service activities See Item 12 above. CRTL Program Evaluation Series, PE-0110-01 25 Third Annual Report to TEAC, January 2010 ________________________________________________________________________________________________ Case studies and alumni competence 15. Evaluations of graduates by their own pupils NYU believes that the questionable reliability and validity of these data render the high resource expenditures required to collect them unwarranted. 16. Alumni self-assessment of their accomplishments See Item 12 above. 17. Third-party professional recognition of graduates (e.g. NPTS) See Item 12 above. 18. Employers' evaluations of the program's graduates See Item 7 above. 19. Graduates' authoring of textbooks, curriculum materials, etc. See Item 12 above. 20. Graduates own pupils' learning and accomplishment See Item 4 above. Other Data 21. Students’ self-ratings of growth in pedagogical knowledge and teaching skills. NYU continues to collect, analyze and report data from the ETFQ, a questionnaire administered to student teachers at the end of each field placement. The ETFQ asks students to assess the extent to which the field experience has contributed to their professional growth as teachers. The questionnaire is being updated to provide more valid data on students’ professional growth. Data from the updated ETFQ will be reported in future Briefs. (File 6 in Appendix B.)** CRTL Program Evaluation Series, PE-0110-01 26 Third Annual Report to TEAC, January 2010 ________________________________________________________________________________________________ 22. Students’ dispositions to teaching. Data from the Educational Beliefs Questionnaire (EBQ) were reported in the initial Brief and continue to be collected. CRTL has replaced the EBQ with a new dispositions survey that it designed and pilot tested in 2009. The new survey is entitled the Educational Beliefs and Multicultural Awareness Survey (EBMAS). The new survey assesses more constructs than the EBQ and has stronger psychometric properties. The next Inquiry Brief will contain EBMAS data for the Classes of 2010 and 2011. 23. Graduates ratings of the effectiveness of NYU’s teacher education programs. The initial Brief reported results from a survey of Fast Track graduate students at the end of the program’s summer component. NYU has replaced this survey with two surveys of program graduates: an exit survey and a one-year follow-up survey. These surveys assess the extent to which graduates feel that the program has prepared them to be successful teachers. NYU faculty believes the graduates are better positioned to provide useful information about the effectiveness of the programs than the mid-course Fast-Track students. Some results from these surveys were used in the Third Annual Report and will be reported in more detail in the Fourth Annual Report and future Inquiry Briefs. (File 10 in Appendix B.)** ** File numbers refer to the data files listed in Appendix B, Summary of CRTL Data Files. CRTL Program Evaluation Series, PE-0110-01 27 Third Annual Report to TEAC, January 2010 ________________________________________________________________________________________________ Appendix B NYU Steinhardt’s Second Annual Report: Summary of CRTL Data Files January 20010 Data File 1. Teacher Education graduate file 2. Domain-Referenced Student Teacher Observation Scale (DRSTOS-R) 3. Educational Beliefs Questionnaire (EBQ) Description Extent of Data Descriptive data for all graduates of Steinhardt teacher education programs, including date of graduation, degree, certification area, GPA, identification data, and contact information. This file can be linked to a course transcript file. Scores from this 21-item protocol for rating the teaching performance of student teachers, which was adapted by CRTL from the work of Charlotte Danielson. The DRSTOS-R is designed to assess the ability of teacher education students to understand and integrate their classroom experiences and apply that learning to their own practice as student teachers. An assessment tool that provides insight into the beliefs held by our pre-service teachers at various points in their programs. Students respond to the survey’s 26 items using a 6-point Likert scale. The EBQ yields two scores: Teaching Efficacy and Caring. The former is defined as belief in the effectiveness of teaching to promote the learning of students from diverse backgrounds while the latter taps concern for the education of all children. The database contains information on all teacher education graduates from the classes of 2001 – 2009. CRTL Program Evaluation Series, PE-0110-01 The database contains the ratings of 998 student teachers from the fall 2004 through the spring 2009 semesters. The student teachers vary by degree and certification programs, as well as early and late placements. The database contains the ratings of approximately 1,600 students from the fall 2004 through the spring 2008 semesters. The students vary by degree, certification programs, and total credits earned at time of administration. Many of the assessments are longitudinally linked to permit the assessment of within-person trends over time. 28 Third Annual Report to TEAC, January 2010 ________________________________________________________________________________________________ 4. New York State Teacher Certification Exam Scores (NYSTCE) In order to receive New York State certification as a teacher, graduates must pass the examinations administered through the NYSTCE program. Tests include the Liberal Arts and Science Test (LAST), the Assessment of Teaching Skills-Written (ATS-W) and the Content Specialty Test for the Core subjects they teach. The database contains the NYSTCE scores for all NYU students who have taken the exams from April 2005 through the most recent test administration in June, 2009. There are a total of nearly 8,800 exam scores for approximately 4,500 students. 5. Student Course Reaction Forms (SCRF) CRTL processes student course evaluations and maintains an historical database of the quantitative data. 6. End of Term Student Teacher Feedback Questionnaires (ETFQ) This is a questionnaire that was developed to elicit feedback from teacher education students concerning the extent to which they perceive that the student teaching experience enhanced their professional knowledge and expertise. The format includes a combination of forced-choice and openend items divided into three parts: feedback on the school, cooperating teacher, and the supervisor. This questionnaire is currently being revised. This is an on-line exit survey that is periodically administered to all teacher education program graduates. Questions focus on the perceptions of graduates as to the extent to which the programs have prepared them to begin teaching, the most and least effective of the program, and their plans for the future. Data are maintained for all T&L courses that submitted SCRFs beginning in fall 2002 through fall 2009. The database also has course evaluation data for other Steinhardt departments for spring 2007 through fall 2009. Data are available at both the course level and student level. The database contains the ratings of nearly than 4,500 student teachers from the fall 2002 through the spring 2009 semesters. The student teachers vary by degree and certification programs, as well as early and late placements 7. End of Program Questionnaire CRTL Program Evaluation Series, PE-0110-01 The survey was administered in May 2006, May 2008, and May 2009. The next survey will be administered in January 2010. 29 Third Annual Report to TEAC, January 2010 ________________________________________________________________________________________________ 8. State/City Information System Follow-Up Tracking Data 8. Partnership Survey Data files CRTL matches files of graduates to state and city human resources files to locate the schools in which graduates are teaching. The matched file contains all of the variables from the graduate file (1 above), as well as the following information from the teacher information systems: school, including county and district, number of years teaching, number of years in same school, subjects taught and grades taught. This file can also be linked to school report cards, school progress reports, and learning environment survey files to pick up school demographic and performance data. This file contains data from several surveys administered in connection with the evaluation of the Partnership Initiative. CRTL Program Evaluation Series, PE-0110-01 CRTL has two separate follow-up files. The first contains data on approximately 1,000 graduates from the classes of 2001 – January 2005 who were employed in schools in New York State, Connecticut or Florida as of October 2004. The second has approximately 1,500 graduates from the classes of 2001 – May 2006 who were employed in New York City schools as of spring 2007; this file was updated for 2008 graduates in December 2008 and will be updated again in January 2010 with graduates from May 2009. Data for the following surveys are available: • Summer 2006 Learning Partners • Fall 2006 Learning Partners • Summer 2007 Learning Partners • Summer 2008 Learning Partners • Fall 2006 Student Teachers • Fall 2006 Host School Liaisons • Fall 2006 NYU Liaisons • Fall 2007Student Teachers • Fall 2007Host School Liaisons • Fall 2007NYU Liaisons These data are no longer collected since the Partnership Initiative has ended. 30 Third Annual Report to TEAC, January 2010 ________________________________________________________________________________________________ 9. Graduate Follow-Up Survey Data from a follow-up survey that was sent to all of the teacher education graduates from the Class of 2007. The survey asks graduates their current employment status, the extent to which the NYU programs prepared them for teaching, and the most and least effective aspects of the teacher education programs. A revised survey will be sent to the graduates of spring 2008 in January 2009. CRTL Program Evaluation Series, PE-0110-01 An SPSS data file with data for 95 (30 percent response rate) respondents from the spring 2007graduating class was constructed and analyzed in spring 2008. New data were collected for the graduates of 2008 and 2009. 31
© Copyright 2026 Paperzz