Evaluating the Impact of Title I, Part A in Colorado Prepared by Colorado Department of Education Office of Federal Program Administration In conjunction with OMNI Institute Background of Title I, Part A SEC. 1001. DECLARATION OF POLICY AND STATEMENT OF PURPOSE. (a)(1) “The Congress declares it to be the policy of the United States that a high­quality education for all individuals and a fair and equal opportunity to obtain that education are a societal good, are a moral imperative, and improve the life of every individual, because the quality of our lives ultimately depends on the quality of the lives of others." ~ Elementary and Secondary Education Act. The Title I, Part A program was enacted in 1965 as part of the Elementary and Secondary Education Act (ESEA). The program provides funds for system‐wide supports and resources for schools to improve learning for students at risk of educational failure. The program's central objective is to ensure that all children reach challenging standards. Title I, Part A provides supplemental resources to those schools and students who have farthest to go in achieving these standards. Title IA program funding is based on the poverty rates of districts nationwide, as measured by U.S. Census estimates1. A 1993 report, Reinventing Chapter 1: The Current Chapter 1 Program and New Directions (USDE, 1993), maintained that Title I programs reinforced low expectations of the students they served by providing remedial instruction and holding them to lower academic standards than other students. This report concluded that in order for the Title I program to effectively support all students in meeting challenging standards, fundamental change was required. With little to show in terms of improved outcomes for students in high poverty schools, reauthorizations of ESEA in 1994 (Improving America’s Schools Act (IASA)) and 2001 (No Child Left Behind (NCLB) Act) were designed to ramp up the accountability that accompanied the supplemental federal funds. Under NCLB, additional requirements were mandated for Title I schools that failed to meet Adequate Yearly Progress (AYP) goals. These sanctions include providing parents with the Public School Choice option to transfer their students to higher performing schools and Supplemental Educational Services for low‐income students. National Evaluation of Title I, Part A Since its inception, numerous researchers have analyzed the effectiveness of Title IA on improving student academic performance and closing the achievement gap between Title IA students (i.e., disadvantaged groups based on minority, English proficiency, and socioeconomic status) and non‐
Title IA students. It remains uncertain, however, whether the changes in IASA and NCLB have strengthened the education system for at‐risk students. Some studies have produced unfavorable reports of Title IA’s impact on student performance (e.g., van der Klaauw, 2005), while others have found that Title IA increased academic performance of students that otherwise would have shown decreased performance (Borgman & D’Agostino, 1996). Additionally, some researchers have found a positive relationship between per pupil spending and student performance (Krueger, 1998) while others have not (Hanushek, 1998).
1 National appropriations for Title I, Part A, measured in constant 2007 dollars, have grown from $5.7 billion in fiscal year (FY) 1966 to $9.5 billion in FY 2000 and $12.8 billion in FY 2007. Title IA Evaluation May 2011 Page 2 of 11 Kosters and Mast (2003) concluded that although Title I does appear to have an impact on student achievement based on some studies with state assessments, the reported growth in achievement is insufficient for closing the achievement gaps based on National Assessment of Educational Progress (NAEP) results. A study conducted by van der Klaauw (2005) found little support for the overall effect of Title IA in New York City Schools. However, changes to Title IA over the years produced better results than were seen in earlier years (i.e., after 2001 compared to 1993 and 1997), implying that the program’s evolution made it more effective in increasing student achievement. The 2007 National Assessment of Title IA, Final Report, posed the following questions and subsequent answers: Q. Are students whom Title I is intended to benefit (including low‐income students, racial/ethnic minorities, limited language proficient students, migrant students, and students with disabilities) making progress toward meeting state academic achievement standards in reading and mathematics? A. In a majority of the states that had three‐year trend data available from 2002‐03 to 2004‐05, the percentage of students achieving at or above the state’s proficient level rose for most student groups, but these increases were often small. Q. Are students, especially disadvantaged, showing achievement gains on the National Assessment of Educational Progress (NAEP)? A. Recent trends on the main NAEP assessment showed gains for 4th‐grade students overall in reading, mathematics, and science, and for minority students and those in high‐poverty schools, but trends for middle and high school students were mixed. Q. Are achievement gaps between disadvantaged students and other students closing over time? A. State assessments and NAEP both provide some indications that achievement gaps between disadvantaged students and other students may be narrowing, but recent changes are small. Colorado Performance and Title I Funding Trends Historical Achievement in Colorado Over the past five years, Colorado’s reading achievement has remained fairly flat. Given that reading is the content area primarily supported through Title IA, understanding the impact of these funds on student achievement in Title IA schools is critical. The program’s purpose is to significantly accelerate the academic achievement of the lowest achieving students. However, the percent of students remaining in the unsatisfactory category in reading has remained relatively stable across the years. As an example of this trend, the results of 4th grade reading scores are presented in Table 1. (For reports of CSAP reading proficiency levels across grades 3 ‐ 10 for these years, see http://www.cde.state.co.us/cdeassess/documents/csap/csap_summary.html and for overall state performance since 2008 see the performance tab in https://edx.cde.state.co.us/SchoolView/DataCenter/reports.jspx?_afrWindowMode=0&_afrLoop=2
191732215991159&_adf.ctrl‐state=9fub3avp8_9).
Title IA Evaluation May 2011 Page 3 of 11 Table 1. Fourth Grade Performance on CSAP Reading 2006 2007 2008 2009 2010 Percent Unsatisfactory 11% 14% 11% 13% 11% Percent Partially Proficient, Proficient, and Advanced 89% 86% 89% 87% 89% Title I, Part A Funding in Colorado Colorado has seen a substantial increase in the amount of Title I, Part A funds since 2007, not including the temporary surge of funds from the American Recovery and Reinvestment Act in 2009 (See Table 2, below). Table 2. Colorado Title I, Part A Funds for Distribution to Districts2 2007 2008 2009 2010 2011 $120,996,141 $116,203,265 $127,109,641 $145,557,651 $148,029,803 Colorado Evaluation of Title I, Part A Given the historical backdrop and the conflicting national evaluation findings on the effectiveness of Title IA (e.g., Kosters & Mast, 2003; Borman & D’Agostino, 1995, compared to van der Klaauw, 2005; Riddle 1996), it became imperative for the Colorado Department of Education (CDE) to examine the impact of Title I dollars in all Colorado school districts and conduct its own evaluation of the program within the state. The purpose of CDE’s evaluation of Title IA has been to answer two key questions: 1. Are Title I funds effective in increasing the academic achievement of students for whom Title IA funds are intended (i.e., disadvantaged students with the lowest performance)? and 2. Do per pupil allocations correlate with academic achievement? In 2008, CDE hired an external evaluator, OMNI Institute (OMNI), to conduct a summative evaluation of the impact of Title I, Part A funds on student achievement. CDE and OMNI collaborated to develop a rigorous evaluation of student academic growth across three years (2006 – 2009). The evaluation used Colorado State Assessment Program (CSAP) scale scores3, rather than proficiency levels, to plot the level of growth across multiple years for groups of students, specifically those that should have been most impacted by Title IA (i.e., the lowest performing students). As a next step, analyses were conducted to ascertain any relationships between per 2 This table only includes regular annual Title I, Part A allocations. Colorado’s evaluation of IA examined the effects of regular IA funds over the course of several years. Funds from ARRA were not included because of the temporary nature of funding and the additional federal requirements and stipulations for the use of funds. 3 CSAP Scale Scores are calculated based on students’ raw scores for the purposes of comparing performance across the same content area and same grade. The continuous scale score for CSAP reading for each grade can be found at http://www.cde.state.co.us/cdeassess/documents/csap/manuals/ContScaleRdg.pdf. Scale scores provide much more detailed information than just using proficiency levels. Title IA Evaluation May 2011 Page 4 of 11 pupil allocations at each school and student academic achievement using the Colorado Growth Model (CGM) data (for information about the CGM, please see http://www.schoolview.org/ColoradoGrowthModel2.asp). CGM was used in the second set of analyses for multiple reasons. In order to compare schools’ performance to the per pupil allocation, it was necessary to aggregate performance scores across grades, which is inappropriate to do with CSAP scale scores. Growth medians, on the other hand, can be calculated for a subset of students within schools. Therefore, the CGM Median scores made it feasible to continue to focus on the performance of unsatisfactory students within schools that received varying amounts of IA allocations. Comparing the percent of students that remained unsatisfactory across the years would not have detected changes in the performance of students in that category as well as the CGM. Furthermore, using CGM median scores made it possible to track the change in each student’s Title I status and corresponding PPA for each year of the analyses. A third set of analyses was conducted with a subset of students who were English Language Learners. The goal was to examine whether evaluation results were similar for one specific group of disadvantaged students targeted by Title IA funding. Methods and Results The results presented here provide a brief summary of the full evaluation conducted by OMNI. A full report is available upon request. 1. Hierarchical Linear Models Using CSAP Scale Scores This portion of the evaluation addressed the question below. Scale scores on the reading CSAP were used as the measure of academic achievement. -
On average, did students served by a Title IA program, schoolwide or targeted assistance, show accelerated growth in achievement compared to students not served by a Title IA program? Students who were in 3rd grade in 2006 and scored in the unsatisfactory4 proficiency level in reading in 2006 were included in analyses. Students scoring in the unsatisfactory proficiency level were selected because they were showing significant gaps in the knowledge and skills needed to meet Colorado’s reading standards. Students in 3rd grade were chosen in order to examine changes in achievement over the elementary school years (3rd through 6th grade). The final sample size was 5,490 students. The following students were excluded from analyses:  Students who skipped or repeated a grade,  Students coded as in a Board of Cooperative Education Services (BOCES) school rather than a district,  Students in correctional facilities,  Students who took CSAPA, and  Students who took the CSAP reading assessment in Spanish. 4 Because Title IA funds are intended to improve the academic achievement of the lowest performing students, only students who scored unsatisfactory in 2006 were used in the HLM analyses. The overall performance of Title IA schools is very different than the results presented here. The evaluation results presented herein are only intended to answer the specific evaluation questions presented, not rate the overall performance of Title IA schools in general. Title IA Evaluation May 2011 Page 5 of 11 Students were classified according to their Title I status in 20065: All students in a school providing schoolwide programming were grouped as schoolwide (SW; n=2,847); students provided services in a Title I targeted assistance program were grouped as targeted assistance (TA; 235); and students who were not in a schoolwide school and did not receive targeted assistance funding through Title I were grouped as not served (NS; n=2,408)6. Table 3. Demographics of Students Included in the Analyses Student Title I Status in 2006 N # of Low Income Students % of Students Low Income 2,847 2,422 85.1 235 160 68.1 Subtotal: Title I students 3,082 2,582 83.8 Not Served 2,408 1,234 51.2 Total: All students 5,490 3,816 69.5 Schoolwide Targeted Assistance Hierarchical linear modeling (HLM; Raudenbush & Bryk, 2002) was used to answer the evaluation question. HLM is a statistical technique that accommodates a hierarchical data structure. For this analysis, a three‐level structure was modeled: time (year) was nested in students and students were nested in schools (district level variation was not modeled). Results from the analysis provide information about any initial gaps in reading achievement between SW, TA, and NS students as a group, as well as whether there were differences in student learning rates, on average, as a function of Title I status. Achievement Gap Results:  SW students scored significantly lower than TA and NS students in 2006. TA and NS students scored similarly to each other. Thus, there was an initial achievement gap between students in schoolwide programs and their TA and NS counterparts. Growth Trajectory Results:  There were no significant differences in academic growth trajectories as a function of students’ Title I status. On average, students grew at a similar rate regardless of whether they were in a schoolwide program, received targeted assistance funds, or were not served by Title IA programs. Thus, the achievement gap remained stable over the three year period. 5 These analyses are limited in that changes in categories from year to year (e.g., went from a schoolwide to a targeted assistance school) could not be captured because status in 2006 was used to represent status in all years. It was not feasible to capture such annual changes within the HLM structure used. However, an exploratory set of analyses was conducted to assess whether there were any significant associations between length of time in a schoolwide or targeted assistance program and academic growth trajectories; no significant associations were detected. 6 This grouping of students was logical for addressing the evaluation question being investigated. It is acknowledged that the programming received for each group might vary greatly, including interventions other than Title IA services being implemented for the “not served” students. Title IA Evaluation May 2011 Page 6 of 11 Figure 1 displays results from the HLM analyses. The solid horizontal lines in the graph (Unsat‐PP) represent the score at which a student’s proficiency level changes from unsatisfactory to partially proficient for that grade. The dashed horizontal lines in the graph (PP‐Prof) represent the score at which a student’s proficiency level changes from partially proficient to proficient. As shown in the graph, the average score for each group of students remained in the unsatisfactory proficiency level in each year. Figure 1: Academic Growth Trajectories in Reading for 3rd Grade Students Scoring in the Unsatisfactory Proficiency Level in 2006 as a Function of Title I Status An additional analysis was conducted with English Language Learners who had limited English proficiency (LEP) or were not English proficient (NEP) and were in 3rd grade and scored unsatisfactory in 2006 (n=1,860). Results indicated a similar pattern of findings. LEP/NEP students in SW schools had significantly lower initial scale scores compared to LEP/NEP students in TA and NS schools. In addition, there were no significant differences in the rate of growth as a function of Title I status for English Language Learners. Title IA Evaluation May 2011 Page 7 of 11 2. Per Pupil Allocation The evaluation next addressed the question below. For reasons previously stated, student reading growth percentile data from Colorado’s Growth Model, instead of scale scores, were used as the indicator of achievement. What were the differences, if any, in growth in achievement for students receiving Title IA services as a function of the amount of per pupil allocation? Students served by Title IA funds through schoolwide or targeted assistance programs in 2006‐07, 2007‐08, or 2008‐09 were selected for analyses. For each year, students were selected for inclusion in the analysis if (1) they received Title IA funded services in that year (school‐wide or targeted assistance), (2) they had student growth percentile data, and (3) they scored in the unsatisfactory proficiency level in the prior year. Students in 4th through 10th grade were included (growth percentile data are calculated for students who have at least two years of CSAP data). Separate analyses were conducted that examined student growth percentile data in each of the three academic years. Students who scored in the unsatisfactory proficiency level7 in the prior year were included in each set of analyses. Analyses were conducted by combining growth percentiles across grades to arrive at a median growth percentile. In each year included in the analyses, categories of per pupil allocations (PPA) were created to classify students receiving low (less than $500), medium ($500 to $900), and high (greater than $900) amounts. The amounts selected for the three categories were based on analysis of histograms of per pupil allocations. Table 4 presents information on sample sizes for students included in analyses. In each year, slightly more than half of students served by Title IA were enrolled in programs which received low PPA amounts. -
Table 4. Number and percentage of students included in analyses by year and PPA category PPA Low (1­500) PPA Medium (501­900) PPA High (901 +) Total Year n % n % n % n % 2007 5380 54.8% 3360 34.2% 1086 11.1% 9826 100% 2008 6864 53.4% 3408 26.5% 2587 20.1% 12859 100% 2009 6269 53.8% 3647 31.3% 1735 14.9% 11651 100% The median growth percentile was calculated for students in each funding category to summarize the growth of students in each group. A median of 50 represents typical or average growth for a group of students. Figure 2 displays the median growth percentiles for students in reading by year and PPA amount. 7 These analyses were also focused on students who scored unsatisfactory on CSAP because IA funds are targeted for these students. Title IA Evaluation May 2011 Page 8 of 11 Figure 2: Median Growth Percentiles in Reading by Year and Per Pupil Allocation (PPA) for Title I Students Scoring in the Unsatisfactory Proficiency Level in the Prior Year Results indicate that for students scoring unsatisfactory in the prior year, students in schools with high PPA ($900 per pupil or higher) consistently had the highest median growth percentiles. In each year, the median growth percentile for students in schools with high PPA was above the 50th percentile. A second analysis for English Language Learners scoring unsatisfactory in the prior year was also conducted. Results indicated the same pattern of findings: English Language Learners who were in schools with high PPA had the highest median growth percentiles, all above the 50th percentile. Evaluation Concerns and Limitations Title I, Part A provides federal funds for supplemental services to school districts. It is difficult to measure the impact of these funds because districts have varying levels of local funding and programming. Changes in student performance studied in this evaluation do not control for the state or locally‐funded educational practices and strategies used by each district. Nonetheless, the trends across the state align with national trends and should not be ignored. Furthermore, the per pupil allocation analysis compared students in Title IA schools, which have similar demographics, to each other, thereby increasing the likelihood that differences in performance could be attributed to the dollar allocation rather than specific characteristics of the schools. Title IA Evaluation May 2011 Page 9 of 11 HLM growth trajectories are calculated to represent the average growth of each group; therefore, it is acknowledged that some students had higher rates of growth than the trajectory line just as some had lower rates of growth. CDE plans to investigate the Title IA‐funded strategies used by the schools that have students in the former category (higher performing than the average trajectory lines). Other limitations and concerns are acknowledged and noted in footnotes in the relevant sections of this report. Summary of Evaluation Results and Next Steps Colorado Title IA students’ growth trajectories were parallel with the growth trajectories of students not served by Title IA. Although Colorado Title IA students have not demonstrated the accelerated rate of growth intended by the program, maintaining a steady rate of growth, even if that does not result in higher proficiency levels, is a noteworthy accomplishment for Colorado’s Title IA students, schools, and districts. In fact, a meta‐analysis of 17 Title I evaluations conducted over 30 years provides evidence that although Title I students might not be surpassing non‐Title I students, the growth trajectories of Title I students would likely be lower than that of their non‐
Title I counterparts absent those Title I funds (Borman & D’Agostino, 1995). Nonetheless, the achievement gaps are not being closed as intended for Title IA students (Riddle, 1996). It is imperative that CDE investigate and provide guidance on the best practices that result in higher rates of acceleration for Colorado’s lowest performing students. The per pupil allocation comparison study revealed that, on average, schools that allocated more per pupil (more than $900 per student) had higher median growth percentiles. This finding supports the notion that diluting Title IA funds reduces the program’s effectiveness in increasing student performance. CDE encourages districts to evaluate their use of Title IA funds and rigorously examine the relationship between per pupil allocations and student performance. Districts are encouraged to share the results of such analyses with CDE. In addition, CDE will investigate the practices used by districts that have medium or low per pupil allocations but demonstrate a high rate of growth. The practices of schools with high per pupil allocation, but low student growth, will also be further examined. CDE invites districts that have data demonstrating accelerated growth of Title IA students to share such data analyses results. CDE will continue in its effort to capture the impact of Title IA funds in Colorado and document the strategies that are resulting in the best student outcomes. The Title IA statute gives tremendous flexibility to districts in the array of allowable activities. It is CDE’s goal to share with districts the strategies that are determined to be most effective in increasing student academic achievement, so that future Title IA funds can be directed more effectively. In addition to seeking input from and fostering collaborations with districts as delineated above, CDE and OMNI will continue to evaluate Title IA at the state level. The next phase of the evaluation will focus on selecting and evaluating ten of the most effective Title IA schools. The strategies and practices of those schools will be showcased for the benefit of schools searching for guidance on best practices. Title IA Evaluation May 2011 Page 10 of 11 References Borman, G.D. & D’Agostino, J.V. (1996). Title 1 and Student Achievement: A Meta­Analysis of Federal Evaluation Results, Educational Evaluation and Policy Analysis 18(4), 309‐326. Hanushek, E.A. (1998). Conclusions and Controversies about the Effectivenesss of School Resources, Economic Policy Review 4(1), New York: Federal Reserve Bank of New York, pp. 11‐27. Kosters, M. H. & Mast, B. D. (2003). Closing the Education Achievement Gap: Is Title I Working? In “Closing the Achievement Gap.” Ed. M. H. Kosters, the AEI Press, Publishers for the American Enterprise Institute, Washington, D.C. Krueger, A.B., (1998). Reassessing the View that American Schools are Broken, Economic Policy Review 4(1), New York: Federal Reserve Bank of New York, pp. 29‐43. Raudenbush, S.W., & Bryk, A.S. (2002). Hierarchical Linear Models: Applications and Data Analysis Methods (2nd ed.). Thousand Oaks, CA: Sage. Riddle, Wayne C. (1996). Title I, Education for the Disadvantaged: Perspectives on Studies of Its Achievement Effects. In “CRS Report for Congress,” 96‐82 EPW. Washington, D.C. Congressional Research Service. van der Klaauw, W. (2005). Breaking the Link Between Poverty and Low Student Achievement: An Evaluation of Title I, Journal of Econometrics, 2008, 142 (2), 731‐756. U.S. Department of Education, Office of Policy and Planning, Planning and Evaluation Service (1993). Reinventing Chapter 1: The Current Chapter 1 Program and New Directions. Title IA Evaluation May 2011 Page 11 of 11