School Turnaround in
Massachusetts
The Impact of SIG Funded School Redesign
Grants
Christina LiCalsi, PhD
Dionisio Garcia Piriz
XXXX_MM/YY
Background & Policy Context
2
Policy Context - National
• Many children attend low-performing schools—including
schools that fail students even after years of reform
• School Improvement Grants (SIGs) are one policy tool
that addresses chronically underperforming schools
• SIGs are federal grants made to states who in turn
distribute them to local educational agencies (LEAs) in
support of turnaround efforts in persistently
underperforming schools.
AMERICAN INSTITUTES FOR RESEARCH
3
Policy Context - Massachusetts
• School Redesign Grant (SRG)
•
Competitive
•
Awarded in three year increments
•
Help schools meet students’ needs through funding improvement strategies
–
Increased time for student learning
–
Professional development for educators
–
Academic enrichment programs for students
–
Etc.
AMERICAN INSTITUTES FOR RESEARCH
4
Study Overview
5
Study Context
• The Massachusetts Board of Elementary and Secondary
Education (ESE) commissioned an external mixedmethods study of the impact of SRGs on student
achievement and how schools use SRG funds.
• RQ: Is there an impact of being in a school that receives
an SRG on student ELA and mathematics achievement?
– variation in impacts by student subgroup, grade, and district.
AMERICAN INSTITUTES FOR RESEARCH
6
Study Overview
•
•
•
•
Comparative Interrupted Time Series
Student math and English achievement
1 through 3 years after SRG receipt
Positive impact on both math and English for all years
– Across districts and most grade levels
– Particularly large improvements for ELL and FRPL subgroups
AMERICAN INSTITUTES FOR RESEARCH
7
Research Design and Sample
8
Comparative Interrupted Time Series Design
(CITS)
• One of the strongest quasi-experimental designs when a
comparison series can be constructed.
• Compares the outcomes of a treatment group with a
comparison group after a treatment occurs, relative to
baseline trends prior to program implementation
• Appropriate when an abrupt policy change occurs – such
as SRG grant receipt - and where multiple years of preand post-intervention data are available
AMERICAN INSTITUTES FOR RESEARCH
9
Sample
• Students with ELA or mathematics test scores in the 200607 through 2014-15 school years in the following schools:
– Schools that received an SRG for the first time between 2010-11 and 201415 (47 schools over 5 cohorts across 10 districts)
– Traditional schools in the same districts serving the same grade ranges as
the SRG schools, but that never received an SRG or had not received it in
the time period being evaluated
– Schools chosen using additional matching strategies for robustness checks
(e.g. only other level 3 and level 4 schools, propensity score matched
comparison schools)
AMERICAN INSTITUTES FOR RESEARCH
10
Analytic Strategy
11
Analytic Strategy
• Multilevel, multiple baseline, CITS model allowing for
differing pre-treatment time trends by treatment status
• Impact of SRG on student math and ELA achievement
one, two, and three years after receipt
• District fixed effects, and time and school random effects
• Student-level characteristics (race, English language
learner status, free- or reduced-price lunch status, and
special education status) and grade level fixed effects
AMERICAN INSTITUTES FOR RESEARCH
12
Analytic Strategy
• ELA and mathematics standardized student test scores
(grades 3-8, 10)
– 2006-07 through 2013-14 raw MCAS scores standardized within year and
grade
– 2014-15 school year PARCC and MCAS theta scores standardized within
grade (grades 3-8); raw MCAS scores standardized (grade 10)
• Subgroup analyses to examine impacts for district and
grade subgroups
• Interactions to examine differences in impacts for special
student populations
AMERICAN INSTITUTES FOR RESEARCH
13
Findings
14
Mean Standardized ELA Score for Schools Servings Grades
3-8 by School Year and SRG Cohort
AMERICAN INSTITUTES FOR RESEARCH
15
Mean Standardized Mathematics Score for Schools Servings
Grades 3-8 by School Year and SRG Cohort
AMERICAN INSTITUTES FOR RESEARCH
16
Mean Standardized ELA Score for Schools Servings Grade
10 by School Year and SRG Cohort
AMERICAN INSTITUTES FOR RESEARCH
17
Mean Standardized Mathematics Score for Schools Servings
Grade 10 by School Year and SRG Cohort
AMERICAN INSTITUTES FOR RESEARCH
18
ELA Achievement Score Effect Sizes by Years After First
SRG Receipt
AMERICAN INSTITUTES FOR RESEARCH
19
Mathematics Achievement Score Effect Sizes by Years After
First SRG Receipt
AMERICAN INSTITUTES FOR RESEARCH
20
ELA Achievement Score Effect Sizes by District and Years
After First SRG Receipt
AMERICAN INSTITUTES FOR RESEARCH
21
Mathematics Achievement Score Effect Sizes by District and
Years After First SRG Receipt
AMERICAN INSTITUTES FOR RESEARCH
22
ELA Achievement Score Effect Sizes by Grade Range and
Years After First SRG Receipt
AMERICAN INSTITUTES FOR RESEARCH
23
Mathematics Achievement Score Effect Sizes by Grade
Range and Years After First SRG Receipt
AMERICAN INSTITUTES FOR RESEARCH
24
ELA Achievement Score Effect Sizes by English Language
Learner (ELL) Status and Years After First SRG Receipt
AMERICAN INSTITUTES FOR RESEARCH
25
Mathematics Achievement Score Effect Sizes by English
Language Learner (ELL) Status and Years After First SRG
Receipt
AMERICAN INSTITUTES FOR RESEARCH
26
ELA Achievement Score Effect Sizes by Free/Reduced Price
Lunch (FRPL) Status and Years After First SRG Receipt
AMERICAN INSTITUTES FOR RESEARCH
27
Mathematics Achievement Score Effect Sizes by
Free/Reduced Price Lunch (FRPL) Status and Years After
First SRG Receipt
AMERICAN INSTITUTES FOR RESEARCH
28
Concluding Remarks and
Discussion
29
Findings overview
• SRGs have a significant impact on student performance in
both ELA and mathematics, one, two, and three years
after grant receipt
•
similar across districts
• Statistically significant impacts on ELA and mathematics
in elementary and middle grades
•
ELA effect for students in grade 10 is not statistically significant
•
Mathematics effect for students in grade 10 is smaller but statistically
significant
• Impact of SRGs on ELL and FRPL eligible students is
particularly large
AMERICAN INSTITUTES FOR RESEARCH
30
Conclusion
Despite largely mixed reports about the impact of SIGs on
school improvement nationwide, Massachusetts has
experienced proven success with significant impacts on
student achievement in both mathematics and English
language arts for schools one, two, and three years after
grant receipt
AMERICAN INSTITUTES FOR RESEARCH
31
Thank you!
[email protected]
32
Model Specification
𝑌𝑖𝑡𝑗 = 𝛽0+ 𝛽1𝐺𝑟𝑎𝑛𝑡𝑗 + 𝛽2𝑇𝑖𝑚𝑒𝑡 + 𝛽3(𝐺𝑟𝑎𝑛𝑡𝑗 x 𝑇𝑖𝑚𝑒𝑡) + 𝛽4𝑃𝑌1𝑡𝑗 +
𝛽5𝑃𝑌2𝑡𝑗 + 𝛽6𝑃𝑌3𝑡𝑗 + 𝛽7𝑃𝑌4𝑡𝑗 + 𝛽8𝑃𝑌5𝑡𝑗 + 𝛽9𝑌11𝑡 + 𝛽10𝑌12𝑡 +
𝛽11𝑌13𝑡 + 𝛽12𝑌14𝑡 + 𝛽13𝑌15𝑡 + 𝐷𝑖𝑠𝑡𝑟𝑖𝑐𝑡𝑗 + 𝐻𝑆𝑖𝑡𝑗 + 𝑋𝑖𝑡𝑗 + 𝑣𝑗 +𝑢𝑡𝑗
+ 𝑒𝑖𝑡𝑗
AMERICAN INSTITUTES FOR RESEARCH
33