Estimating Impacts on Program-Related
Subgroups Using Propensity Score Matching:
Early College High Schools Study
Fatih Unlu and Ryoko Yamaguchi, Abt Associates, Larry Bernstein,
RTI, and Julie Edmunds, SERVE
SREE 2010 Conference
March 5, 2010
Acknowledgements
• Current paper part of the experimental evaluation of Early
College High Schools in North Carolina:
– Partnership between SERVE, NCDPI, NC New Schools
Project, Duke University, UNCG, Abt Associates, and RTI
International
– Funded through IES grant
• Special thanks:
– Bill Rhodes (Abt Associates)
– Mark Lipsey (Vanderbilt University)
Abt Associates Inc.
2
Overview
• Conceptual framework: program-related subgroups
• Background info on Early College High Schools study
• Research Questions
• Analytical and Empirical Framework
• Results
• Summary
Abt Associates Inc.
3
Motivation and Conceptual Framework:
Program-Related Subgroups
• Experimental evaluation of Early College High Schools (ECHS) in NC:
– Taking and passing Algebra I is an important outcome
– Take-up rate: 96% ECHS vs. 69% control
– Pass rate (overall): 74% ECHS vs. 58% control
– Pass rate (takers only): 77% ECHS vs. 84% control
• Program-Related (PR) Subgroups in RCTs: (Angrist et al., 1996):
– Always-takers: Take-up program regardless of assignment status
• Algebra I takers in the control group and their ECHS counterparts
– Compliers: Take-up program only if assigned to treatment
• ECHS students induced to take Algebra I by the program and their
control counterparts
– Never-takers: Never take-up program
– Defiers: Take-up program only if assigned to control
Abt Associates Inc.
4
Conceptual Framework: Program-Related
Subgroups
• Useful insights from estimating impacts on PR subgroups
• Problem with analyzing PR subgroups:
– PR subgroups are endogenous
– PR subgroups are not always observable.
• Solution: Use quasi-experimental methods to identify the
groups and effects:
– Instrumental Variables Analysis: LATE (e.g., Bloom
adjustment)
– Propensity Score Matching (Schochet & Burghardt, 2007;
Peck, 2004)
Abt Associates Inc.
5
Experimental Evaluation of ECHS in NC
• Early College High Schools (ECHS) Initiative in NC:
– Funded by NC Legislature with support from Gates Foundation
– Small high schools, located on college campuses,
– Goal: Provide traditionally underrepresented students with HS diploma
& college credit by placing them on a college-prep. track of study.
• Experimental Evaluation: 4 year RCT funded by IES. Students
attending ECHS in our study are randomly drawn from a pool of
applicants (treatment group).
– Unsuccessful applicants attend regular HS (control group)
• Target sample: 34 cohorts in 20 sites; ~3,000 students. Currently
available sample: 8 cohorts in 6 sites; ~700 9th graders.
Abt Associates Inc.
6
Experimental Evaluation of ECHS in NC (2)
• Outcome measures:
– College prep course-taking and passing (Administrative data)
– Students’ behaviors, attitudes, and high school experiences
(Surveys)
• Current paper: Focus on impacts on passing Algebra I - overall
and by program-related subgroups.
• Why Algebra I?
– It would be hard for students who don’t pass Algebra I by end of
9th grade to complete college-prep course of study.
– Higher level math courses not required for graduation, regular HS
Source:
may steer students away from them.
Abt Associates Inc.
Bureau of Labor Statistics
7
Research Questions
1. What is the overall impact of ECHS on passing Algebra I
in the 9th grade?
– Can be through two channels: (1) improved instruction
and curriculum and (2) improved access and support inducing students to take Algebra I
2. What is the impact of ECHS on “always-takers” in the 9th
grade?
– Pertains to the “improved instruction” channel
3. What is the impact of ECHS on “compliers” in the 9th
grade?
Source: Bureau
of Labor Statistics
– Pertains to the “improved access”
channel
Abt Associates Inc.
8
9th Grade Algebra I Taking
ECHS
P(A’’)=0.69
Control
A’’
A’
P(A’)=0.69
P’’=0.96
P(B’’)=0.27
B’’
B’
P(B’)=0.27
P(C’’)=0.04
C’’
C’
P(C’)=0.04
A’’/A’: Always-takers
B’’/B’: Compliers
C’’/C’: Never-takers
P(G): Proportion of students in Group G
Y(G): Pass-rate in Group G
Abt Associates Inc.
9
Research Question 1
• Overall Impact on Algebra I pass rate:
∆= Y(ECHS) – Y(Control)
• Y(ECHS) = P(A)Y(A’’) + P(B)Y(B’’) + P(C)Y(C’’)
• Y(Control) = P(A)Y(A’) + P(B)Y(B’) + P(C)Y(C’)
• Note: Y(C’’) = Y(B’) = Y(C’) = 0.
Therefore:
∆ = P(A)Y(A’’) + P(B)Y(B’’) - P(A)Y(A’)
Source: Bureau of Labor Statistics
= P(A) [Y(A’’) - Y(A’)] + P(B)Y(B’’)
Abt Associates Inc.
10
Research Question 2
• Impact on Algebra I pass rate for always-takers:
∆A= Y(A’’) – Y(A’)
• Problem: Always-takers in control (A’) known but
always-takers in ECHS (A’’) not observed
• Solution: Match Algebra I takers in control (A’) with
Algebra I takers in treatment (A’’ & B’’)  matched
ECHS students: A’’
• Once A’’ is found, then ∆A= Y(A’’)
– Y(A’)
Source: Bureau of Labor Statistics
Abt Associates Inc.
11
Research Question 3
•
Impact on Algebra I pass-rate for compliers:
∆B = Y(B’’) – Y(B’) = Y(B’’)
•
Recall:
∆ = P(A) [Y(A’’) - Y(A’)] + P(B)Y(B’’)
∆A
∆B
•
∆ = P(A) ∆A + P(B) ∆B  ∆B = [∆ - P(A) ∆A ]/P(B)
•
So, once ∆ and ∆A are estimated, we can estimate ∆B
Source: Bureau of Labor Statistics
Abt Associates Inc.
12
Finding Always-takers in ECHS
• Propensity Score: Probability of Algebra I taking
• Step 1- Model Algebra I taking in the control group:
– Use logistic regression;
– Algebra I taking  dependent var
– student characteristics (gender, race, 8th grade test scores,
etc.) independent vars.
• Passing 8th grade math is a good predictor
SLIDE
• Step 2 – Predict propensity scores in ECHS & Control:
Source:
Bureau of
Statistics
– Use the estimated logistic model
in Step
1Labor
to predict
SLIDE
probability of Algebra I taking in ECHS and Control
Abt Associates Inc.
13
Finding Always-takers in ECHS (2)
Step 3 – Implement matching:
– One-to-one matching
– Each control student in A’(always-taker) is matched with
one of the algebra-takers in ECHS with the closest
propensity score. SLIDE
• Step 4 – Check the quality of matches:
– Test whether matching characteristics are balanced across
control and matched ECHS.
SLIDE
Abt Associates Inc.
Results
GROUP
Proportion
Overall
9th Grade Algebra I Passing
ECHS
Control
Impact
Significant?
100%
74%
58%
16%
Yes
Alwaystakers
69%
85%
83%
2%
No
Compliers
27%
56%
0%
56%
Yes
Nevertakers
4%
0%
0%
-
-
Source: Bureau of Labor Statistics
Abt Associates Inc.
15
Summary
• Overall impact on 9th grade Algebra I pass rate. Impact
on always-takers is small & not statistically significant
whereas impact on compliers is large & statistically
significant
– Suggests impact is through “improved access” channel.
• Specification/sensitivity tests and extensions:
– Other matching methods
– Other course-taking outcomes
• Framework is applicable to other contexts
Abt Associates Inc.
16
Contact Information
Fatih Unlu
[email protected]
Ryoko Yamaguchi
[email protected]
Larry Bernstein
[email protected]
Julie Edmunds
[email protected]
Abt Associates Inc.
17
Abt Associates Inc.
18
Additional Slide 1: Modelling Algebra Taking
Logistic Regression Modeling Algebra I Taking in the Control Group
Odds
Ratio
Std. Err.
Z
P|z|
95% Confidence
Interval
African American
1.68
0.74
1.17
0.24
0.71
3.98
Hispanic
2.18
2.34
0.73
0.47
0.27
17.79
Male
0.49
0.18
-1.95
0.05
0.23
1.00
First Generation
College Bound
0.46
0.19
-1.87
0.06
0.20
1.04
Free/Reduced Priced
Lunch Eligible
1.11
0.53
0.22
0.83
0.43
2.85
Disabled
0.10
0.10
-2.35
0.02
0.02
0.68
Retained in 7th grade
or earlier
0.04
0.03
-4.40
0.00
0.01
0.16
8th grade Math Score
4.57
1.37
5.06
0.00
2.54
8.24
8th grade Reading
Score
0.94
0.24
-0.25
0.80
0.57
1.55
Covariates Imputed
0.06
0.06
-2.83
0.01
0.01
0.42
Variable
BACK
Abt Associates Inc.
Additional Slide 2: Estimated Propensity Scores
(Before Matching)
Histogram of the Propensity Score Before Matching
Alg I Takers - ECHS
0
.1
Fraction
.2
.3
Alg I Takers - Control
0
.1
.2
.3
.4
.5
.6
.7
.8
.9
1
0
.1
Propensity Score
Graphs by Student accepted in ECHS
BACK
Abt Associates Inc.
.2
.3
.4
.5
.6
.7
.8
.9
1
Additional Slide 3: Estimated Propensity Scores
(After Matching)
Histogram of the Propensity Score After Matching
Alg I Takers - ECHS
0
.1
Fraction
.2
.3
Alg I Takers - Control
0
.1
.2
.3
.4
.5
.6
.7
.8
.9
1
0
.1
Propensity Score
Graphs by Student accepted in ECHS
BACK
Abt Associates Inc.
.2
.3
.4
.5
.6
.7
.8
.9
1
Additional Slide 4: Propensity Scores of
Matched Pairs
0
.2
.4
.6
Propensity Score
.8
1
Propensity Score of Matched Pairs
0
50
100
Pairs
Control
Matched Treatment
BACK
Abt Associates Inc.
Matched Pair Prop. Scr Difference
150
Additional Slide 5: Testing Balance of Matching
Balance in Matching Characteristics
Control
Mean
Matched
ECHS Mean
Difference
Standardized
Difference
% African American
28.98%
24.77%
4.21%
0.09
% Hispanic
4.54%
6.35%
1.81%
0.08
% Male
31.30%
37.83%
6.53%
0.14
%First Generation College Bound
45.16%
43.39%
1.77%
0.04
% Free/Reduced Priced Lunch
Eligible
43.15%
48.70%
5.55%
0.11
% Disabled
1.87%
2.36%
0.49%
0.03
% Gifted
5.85%
7.82%
1.96%
0.08
% Retained in 8th grade or earlier
0.00%
0.63%
0.63%
0.11
8th grade Math Score
0.24
0.34
0.10
0.12
8th grade Reading Score
0.13
0.16
0.03
0.03
Variable
BACK
Abt Associates Inc.