Introduction
The experimental ideal
Potential drawbacks
Applied Microeconometrics I
Lecture 2: Randomized controlled trials
Manuel Bagues
Aalto University
November 2 2016
Lecture Slides
1/ 24
Applied Microeconometrics I
Introduction
The experimental ideal
Potential drawbacks
Notes:
Deadline for problem sets has been extended!
Available on Wednesdays, due on Mondays Tuesdays at 22:00.
Recommended reading for week 1
Introduction Angrist & Pischke 2009
2/ 24
Applied Microeconometrics I
Introduction
The experimental ideal
Potential drawbacks
Correlation does not imply causation
Two important (well-known) points:
1
2
Correlation does not imply causation
y can cause x even if x takes place before y
3/ 24
Applied Microeconometrics I
Introduction
The experimental ideal
Potential drawbacks
Correlation does not imply causation
Cor(x,y) 6= 0
1
2
3
x implies y
y implies x
z implies x and y
Simple examples:
People that sleep less tend to live longer
Red cars are more likely to get involved in accidents
Countries that eat more chocolate receive more Nobel prizes
Messerli 2012
4/ 24
Applied Microeconometrics I
Introduction
The experimental ideal
Potential drawbacks
5/ 24
Applied Microeconometrics I
Introduction
The experimental ideal
Potential drawbacks
Potentially confusing examples:
Students living in households where there are more books tend to
perform better in PISA evaluations
GDP growth and public debt
Women in boards
GDP growth and R&D expenditure
CSR/marketing/R&D expenditure and firms’ profits
6/ 24
Applied Microeconometrics I
Introduction
The experimental ideal
Potential drawbacks
y can cause x even if x takes place before y
Example (i): rain
When many people carry their umbrellas in the morning usually it
rains in the afternoon
Not a bad idea to use this fact to predict rain...
... however, subsidizing umbrellas is not a great policy to increase
rain
Example (ii): Stock exchange
Stock market usually goes down before an extremist party wins
an election
7/ 24
Applied Microeconometrics I
Introduction
The experimental ideal
Potential drawbacks
8/ 24
Applied Microeconometrics I
Introduction
The experimental ideal
Potential drawbacks
Food for thought:
What is your question of interest? (For instance, for your master
thesis research proposal)
Is this a causal relationship?
9/ 24
Applied Microeconometrics I
Introduction
The experimental ideal
Potential drawbacks
How can we estimate a causal relationship?
The “Scientific” solution vs. the “Statistical” solution
In the physical sciences one can often answer this type of
question by running a ‘scientific’ experiment.
Example: Galileo’s Leaning Tower of Pisa experiment
Aristotle’s theory of gravity: objects fall at speed relative to their
mass
Throw two balls of different weight from the tower (Pisa or Deft?)
10/ 24
Applied Microeconometrics I
Introduction
The experimental ideal
Potential drawbacks
Leaning Tower of Pisa
11/ 24
Applied Microeconometrics I
Introduction
The experimental ideal
Potential drawbacks
Properties:
Temporal stability: the response does not change if the time when
a treatment is applied is varied slightly.
Causal transience: the response of one treatment is not affected
by prior exposure of the unit to the other treatment.
Unit homogeneity: units are homogeneous with respect to the
treatments and responses.
Unfortunately in Social Sciences none of these assumptions is
plausible. Instead we use the statistical solution.
12/ 24
Applied Microeconometrics I
Introduction
The experimental ideal
Potential drawbacks
Notation
Let’s think of a treatment as a binary random variable
Di = {0, 1}
And potential outcomes (counterfactuals):
Y1i if Di = 1
Yi =
Y0i if Di = 0
The causal effect of treatment for an individal i is Y1i − Y0i
Unfortunately, for i, we only observe Y1i if Di = 1 and Y0i if
Di = 0.
13/ 24
Applied Microeconometrics I
Introduction
The experimental ideal
Potential drawbacks
Observed difference in outcomes versus the causal effect of
the treatment
A naive comparison of averages does not tell us what we want to
know:
E[Y1i |Di = 1] − E[Y0i |Di = 0] =
E[Y − Y |D = 1]
| 1i {z0i i
}
Average effect of the treatment on treated
+ E[Y0i |Di = 1] − E[Y0i |Di = 0]
|
{z
}
Selection bias
The difference in the observed outcomes (between the treated
and the non treated) is equal to the average treatment effect on
the treated if there is no selection effect:
If E[Y0i |Di = 1] = E[Y0i |Di = 0]
then E[Y1i |Di = 1] − E[Y0i |Di = 0] = E[Y1i − Y0i |Di = 1]
14/ 24
Applied Microeconometrics I
Introduction
The experimental ideal
Potential drawbacks
Note that
Average treatment effect on the treated: E[Y1i − Y0i |Di = 1]
Different from the population average treatment effect:
E[Y1i − Y0i ]
They need not to be the same and the distinction is sometimes
important (also for policy makers!)
They will be the same only if, in addition to:
E[Y0i |Di = 1] = E[Y0i |Di = 0]
we assume:
E[Y1i |Di = 1] = E[Y1i |Di = 0]
So that:
E[Y1i |Di = 1] − E[Y0i |Di = 1] =
= E[Y1i |Di = 0] − E[Y0i |Di = 0]
(i.e.: the effect of treatment is the same for everybody)
15/ 24
Applied Microeconometrics I
Introduction
The experimental ideal
Potential drawbacks
Random assignment
We want to understand what would have happened to the treated
in the absence of treatment and thus overcome the selection
problem...
Solution → Random assignment
16/ 24
Applied Microeconometrics I
Introduction
The experimental ideal
Potential drawbacks
Random assignment
17/ 24
Applied Microeconometrics I
Introduction
The experimental ideal
Potential drawbacks
Random assignment
Random assignment solves the selection problem since it makes
Di independent of potential outcomes, hence:
E[Y1i |Di = 1] = E[Y1i |Di = 0] = E[Y1i ]
E[Y0i |Di = 1] = E[Y0i |Di = 0] = E[Y0i ]
Therefore, a comparison of the treatment and control group
outcomes provides information about the causal impact of the
average treatment effect on the treated (ATET):
E[Y1i |Di = 1] − E[Y0i |Di = 0] =
= E[Y1i |Di = 1] − E[Y0i |Di = 1] =
= E[Y1i − Y0i |Di = 1]
18/ 24
Applied Microeconometrics I
Introduction
The experimental ideal
Potential drawbacks
Random assignment
and the average treatment effect on the non-treated:
E[Y1i |Di = 1] − E[Y0i |Di = 0] =
= E[Y1i |Di = 0] − E[Y0i |Di = 0] =
= E[Y1i − Y0i |Di = 0]
and the population average treatment effect (ATE):
E[Y1i − Y0i |Di = 1] = E[Y1i − Y0i |Di = 0] = E[Y1i − Y0i ]
Note: With other empirical methods very often the ATET will not be
equal to the ATE.
19/ 24
Applied Microeconometrics I
Introduction
The experimental ideal
Potential drawbacks
History
Long history of randomized field trials
Earliest example: James Lind scurvy experiment?
Early experiments had many flaws due to lack of experience in
designing experiments and in data analysis.
During the 1980s the US federal government started to
encourage states to use experimentation, eventually becoming
almost mandatory.
In spite of these developments, randomization encountered
resistance from many US states on ethical grounds (even more so
in other countries).
Overtime, the number of randomized experiments published in
top journals has increased dramatically (Hamermesh 2012).
20/ 24
Applied Microeconometrics I
Introduction
The experimental ideal
Potential drawbacks
Types of random experiments
Lab experiments
The effect of feedback on relative performance (Azmat and
Iriberri 2010)
Field experiments
The effect of feedback on relative performance (Azmat, Bagues,
Cabrales and Iriberri 2016)
21/ 24
Applied Microeconometrics I
Introduction
The experimental ideal
Potential drawbacks
Several important points:
1 Ideal experiment helps to formulate causal question precisely
Does applicants’ race affect hiring decisions?
Bertrand and Mullainathan 2003, Doleac and Stein 2013
2
Questions that cannot be answered by any experiment are
fundamentally unidentified
The effect of start age on first grade test scores
Start age = age - time in school
3
Ideally we would like not only to evaluate a policy but also to
test a economic theory.
Example: Fryer, Levitt, List and Sadoff 2012
22/ 24
Applied Microeconometrics I
Introduction
The experimental ideal
Potential drawbacks
Experiments provide a very transparent and simple empirical strategy
and they solve the selection bias. However, there are a number of
potential problems:
1 Problems of implementation
Compliance and attrition
Cost, political issues (policy makers need to acknowledge
ignorance),...
2
Ethical issues
Examples: STAR, babies...
Note that the ethical argument is not obvious when (i) the
treatment cannot be applied to everybody (maybe due to some
budget constraints) and (ii) the optimal assignment rule is
unknown.
3
Hawthorne effect
The Illumination Experiment (Landsberger 1950, Levitt and List
2011)
Audit study in France (Behaghel et al. 2015)
23/ 24
Applied Microeconometrics I
Introduction
The experimental ideal
Potential drawbacks
4
General equilibrium issues
Social feedback: The potential outcome representation is
predicated on the assumption that the effect of treatment is
independent of how many individuals receive treatment
So the framework is generally not well suited to the evaluation of
system-wide reforms which are intended to have substantial
equilibrium effects.
Interesting exception: Will a public program of job placement
assistance to young unemployed workers improve their labor
market outcomes? (Crepon et al. 2012)
5
External validity vs Internal validity
Problem also with other identification strategies
Structural models
24/ 24
Applied Microeconometrics I