Using A Regression Discontinuity Design
(RDD) to Measure Educational
Effectiveness:
Howard S. Bloom
MDRC
12-11-02
[email protected]
This Talk Will:








introduce the history and logic of RDD,
consider conditions for its internal validity,
considers its sample size requirements,
consider its dependence on functional form,
illustrate some specification tests for it,
consider limits to its external validity,
consider how to deal with noncompliance,
describe an application.
RDD History
In the beginning there was Thislethwaite
and Campbell (1960)
 This was followed by a flurry of
applications to Title I (Trochim, 1984)
 Only a few economists were involved
initially (Goldberger, 1972)
 Then RDD went into hibernation
 It recently experienced a renaissance among
economists (e.g. Hahn, Todd and van der
Klaauw, 2001; Jacob and Lefgren, 2002)
 Tom Cook has written about this story

RDD Logic

Selection on an observable (a rating)

A tie-breaking experiment

Modeling close to the cut-point

Modeling the full distribution of ratings
RDD As a Tie-Breaking
Experiment
Outcome
Measure
*
Rating
RDD As A Linear Regression
RDD With An Incorrect Functional Form
Outcome
Measure
Rating
Conditions for Internal Validity
The outcome-by-rating regression is a
continuous function (absent treatment).
 The cut-point is determined independently
of knowledge about ratings.
 Ratings are determined independently of
knowledge about the cut-point.
 The functional form of the outcome-byrating regression is specified properly.

RDD Statistical Model
Yi     0Ti   1Ri  ei
where:
Yi = outcome for subject i,
Ti = one for subjects in the treatment group
and zero otherwise,
Ri = rating for subject i,
ei = random error term for subject i, which is
independently and identically distributed
Variance of the Impact Estimator
2
(
1

s
R
1)
VAR(  0) 
2
P(1  P)n(1  R2)
^
2
s2 = variance of mean outcomes across subjects in the
treatment group or comparison group
R12 = square of the correlation between outcomes and
ratings within the treatment and comparison group
R22 = square of the correlation between treatment status and
the rating
P = proportion of subjects in the treatment group,
N = total number of subjects
Sample Size Implications

Because of the substantial multi-collinearity
that exists between its rating variable and
treatment indicator, an RDD requires 3 to 4
times as many sample members as a
corresponding randomized experiment
Specification Tests




Using the RDD to compare baseline
characteristics of the treatment and comparison
groups
Re-estimating impacts and sequentially deleting
subjects with the highest and lowest ratings
Re-estimating impacts and adding:
 a treatment status/rating interaction
 a quadratic rating term
 interacting the quadratic with treatment status
Using non-parametric estimation
External Validity
Estimating impacts at the cut-point
 Extrapolating impacts beyond the cut-point
with a simple linear model
 Estimating varying impacts beyond the cutpoint with more complex functional forms

Dealing With Noncompliance





Sharp and fuzzy RDDs
No-shows and crossovers
The effect of intent to treat (ITT)
The local average treatment effect (LATE)
The effect of treatment on the treated (TOT)
^
^
ITT   0
^
^
LATE 
0
rT  r C
Where rT and rC = the proportion of the treatment and
control groups receiving treatment, respectively
Application of RDD
To Reading First




Reading First (RF) is a cornerstone of No Child
Left Behind
RF resources are allocated purposefully to schools
that need it most and will benefit most
Some districts allocated RF resources based on
quantitative indicators
We chose a sample of 251 schools near the cutpoints for 17 such districts and 1 state
References
Cook, T. D. (in press) “Waiting for Life to Arrive: A History of the
Regression-discontinuity Design in Psychology, Statistics and Economics” Journal of
Econometrics.
Goldberger, A. S. (1972) “Selection Bias in Evaluating Treatment Effects:
Some Formal Illustrations” (Discussion Paper 129-72, Madison WI: University of
Wisconsin, Institute for Research on Poverty, June).
Hahn, H., P. Todd and W. van der Klaauw (2001) “Identification and
Estimation of Treatment Effects with a Regression-Discontinuity Design”
Econometrica, 69(3): 201 – 209.
Jacob, B. and L. Lefgren (2004) “Remedial Education and Student
Achievement: A Regression-Discontinuity Analysis” Review of Economics and
Statistics, LXXXVI.1: 226 -244.
Thistlethwaite, D. L. and D. T. Campbell (1960) “Regression Discontinuity
Analysis: An Alternative to the Ex Post Facto Experiment” Journal of Educational
Psychology, 51(6): 309 – 317.
Trochim, W. M. K. (1984) Research Designs for Program Evaluation: The
Regression-Discontinuity Approach (Newbury Park, CA: Sage Publications).