Performance Based Incentives for
Learning in the Mexican Classroom
Brian Fuller, MPA, Foundation Escalera
Victor Steenbergen, MPA Candidate, London School of Economics
Agenda
1. Star Program in Context
2. Theoretical Overview
3. Evaluation Strategy
4. General Treatment Effects on Test Scores
5. Indirect Treatment Effects (Teachers and Parents)
The Star Program
• Incentives program aimed at marginalized, rural middle
school students (ages 12-15) in Chiapas, Mexico.
• RCT with 147 schools participating (76 treatment, 71
control)
• In total 7852 students (4011 treatment, 3841 control)
The Star Program
• Chiapas, Mexico
o 37% of workers earn less than monthly minimum wage
o Ranks last nationally for computer ownership (7.2%),
landlines (12.6%)
o 49% rural residents
• Indigenous Peoples
o
o
o
39% cannot read or write
83% work in agriculture
Avg. income is 32% of avg non-indigenous
• Telesecundaria Middle Schools
o
o
Distance learning expands coverage -> many first-time learners
95% and 89% in lowest two categories in reading and math
• (21% and 19%, OECD)
The Star Program
Three times a year, each grade selected:
A. Most Improved Student (1 Female, 1 Male)
o
Increases in general GPA
o
Targets middle and low-achieving students
B.
A+ Lottery Winners (1 Female, 1 Male)
o
Individual subject grades = tickets
o
Targets high achieving students
The Star Program
Awards
• First and second grade students:
o material incentives (books, MP3-players, laptops)
o increases likelihood students keeping their prices
• Third grade students:
o financial incentives (scholarships up to M$2000).
o grants might involve parents in children’s learning.
• All winners:
o Certificates of achievement
o Public recognition at ceremony
Agenda
1. Star Program in Context
2. Theoretical Overview
3. Evaluation Strategy
4. General Treatment Effects on Test Scores
5. Indirect Treatment Effects (Teachers and Parents)
Theoretical Overview
• Incentive programs work when students lack sufficient
motivation to learn (Fryer, 2010)
o They do not ‘enjoy’ learning much
o Their concerns are short-term driven
o They lack knowledge of future benefits to education.
• Primary impact: financial reward for learning might
incentivise student effort to win the prize.
Theoretical Overview
• Student performance requires complementary support
from parents and teachers (Fryer, 2010)
• Secondary impact: incentive program might enhance
parent and teacher involvement (Kremer et al, 2004).
o Parents: family also benefit from the incentive program (e.g. scholarship)
o Teachers: community informally sanctions absent teachers and rewards winners.
o Teachers: increased student motivation leads to greater teacher effort.
Conceptual Framework
Student Extrinsic Motivation
Increase student
effort to win prize
INCENTIVE
PROGRAM
OVERALL
STUDENT
ACHIEVEMENT
Teacher Effort
Improve teaching and
parental support
Parental Involvement
Agenda
1. Star Program in Context
2. Theoretical Overview
3. Evaluation Strategy
4. General Treatment Effects on Test Scores
5. Indirect Treatment Effects (Teachers and Parents)
Evaluation Strategy
• Three groups of control variables
o Student characteristics
o Teacher and director variables
o School and regional level characteristics
• Randomization check using two-tailed t-tests
• OLS regression with treatment status as dependent variable
• The trial was well randomized; no significant differences for variables
hypothesized most closely associated with student achievement.
Evaluation Strategy
• General treatment effects: Generalized Least Squares regression
with random errors clustered at the school level.
• Effects across student ability: Quantile regression at different points
in conditional distribution of test scores, clustered on school level
using bootstrapping (Chen, Wei and Parzen, 2003).
• Indirect treatment effects: Ordered logit regression with clustered
standard errors on school level.
Agenda
1. Star Program in Context
2. Theoretical Overview
3. Evaluation Strategy
4. General Treatment Effects on Test Scores
5. Indirect Treatment Effects (Teachers and Parents)
General Treatment Effects
• General impact between 0.217 and 0.259 standard deviations.
• Impacts are generally significant at the 5% level.
• Slightly higher impact for males than females.
Overall
Boys
Girls
Combined
Mathematics
Reading
0.237**
0.238***
0.182**
(0.0945)
(0.0851)
(0.0907)
0.259**
0.244**
0.213*
(0.109)
(0.0975)
(0.110)
0.217**
0.233**
0.154
(0.103)
(0.0924)
(0.106)
Treatment Effects per Grade
Treatment Impact
Grade 1
• Only significant impact for women in mathematics
(0.181, significant at 10% level)
Grade 2
• Only significant impact for men in mathematics
(0.421, significant at 1% level)
Grade 3
• Impact for both men and women
• Impact in mathematics and reading (next table)
Treatment Effects Grade 3
• All impacts are statistically significant at conventional levels.
• Impact size between 0.200 and 0.346 standard deviations.
• Larger impact for reading than mathematics.
Overall
Boys
Girls
Combined
Mathematics
Reading
0.309***
0.246**
0.301***
(0.111)
(0.0978)
(0.111)
0.200*
0.270***
0.224**
(0.105)
(0.0879)
(0.0960)
0.391***
0.334***
0.346**
(0.151)
(0.116)
(0.153)
Impact across Student Ability
• Do incentive awards exclusive benefit previously high-performing
students? (Leuven et al, 2010).
Quantile
0.1
Quantile
0.25
Quantile
0.5
Quantile
0.75
Quantile
0.9
Combined
Mathematics
Reading
0.171**
0.244**
0.194*
(0.0749)
(0.103)
(0.107)
0.229
0.320**
0.139
(0.141)
(0.139)
(0.111)
0.278*
0.292**
0.169
(0.168)
(0.149)
(0.156)
0.284
0.244*
0.270
(0.191)
(0.130)
(0.167)
0.166
0.244*
0.210
(0.172)
(0.135)
(0.333)
Agenda
1. Star Program in Context
2. Theoretical Overview
3. Evaluation Strategy
4. General Treatment Effects on Test Scores
5. Indirect Treatment Effects (Teachers and Parents)
Treatment Effects on Teachers
Teacher attendance
Teacher was absent 3 days or
less in the last 4 weeks
Treatment
Control
Impact
90.3%
75.6%
14.68***
Treatment
Control
Impact
89.0%
78.3%
10.65*
Teacher effort
Teacher always or almost
always asks students if they
understand topic
Treatment Effects on Parents
Parental Involvement
Parents have attended a
school meeting this year
Treatment
Control
Impact
93.5%
88.0%
5.57*
Conclusion
1. Incentive program had a sizeable, significant and robust impact on
student performance, raising overall test scores by 0.237 standard deviations.
• Comparable impact to other education programs.
2. Overall impact for mathematics > reading
3. Overall impact for males > females
Conclusion
4. Impact for financial incentives > material incentives
• Relevance of prize
5. Impact for low, average and high-performing students
• Successful mix of incentives (‘most improved’ and ‘high grades’)
6. Additional treatment effects for teacher effort and parental support.
Questions
Brian Fuller
[email protected]