HPAM 8400, Selection bias handout
1. What is a potential outcome?
Describes what the outcome for a subject would have been with or without
the treatment. Importantly, potential outcomes are stable, meaning that
they do not vary with treatment assignment. In other words, what the outcome for subject i would have been in the absence of treatment, is not affected
by whether subject i did or did not actually receive treatment. Potential outcomes often represent an unobserved counterfactual – a hypothetical outcome
that we are unable to observe, but that is essential for understanding policy
impacts.
2. How are potential outcomes notated?
Yi (1), Yi (0)
(1)
3. What is a treatment effect (describe it in words)?
The change in outcomes due to the treatment. This is a formal way of
describing the impact of a policy or intervention.
4. Why are potential outcomes so important for scientific program evaluation?
Any claim about treatment effects depends on the idea of a fixed outcome for
the same unit with and without exposure to the program. In other words, we
aim to know how much better things were with the program, compared to
what they would have been without it. Likewise, when considering extending
a program to a new population, we do so based on the claim that outcomes
would be better with the program than without.
5. How is the treatment effect defined for an individual subject (describe
and show the mathematical notation)?
The difference between the potential outcome with treatment, and the potential outcome without treatment for that subject. What the outcome would
have been if that subject had received treatment, compared to what the
outcome would have been if that subject had not received treatment.
τi = Yi (1) − Yi (0)
1
(2)
6. Why is it usually not feasible to measure the individual treatment effect?
Because typically the potential outcomes with and without treatment cannot be observed for each individual. Either by applying treatment, or simply
by allowing time to pass without applying treatment, we have effectively
changed the subject, and therefore made this within-subject comparison impossible.
7. Since the individual treatment effect can’t be measured directly, what
must the researcher do instead(describe and show the notation)? Hint:
If the researcher cannot compare individuals to themselves, what comparison must she make instead?
Compare the outomce in the treated group to the outcome in the untreated
group.
E[Yi (1) | Di = 1] − E[Yi (0) | Di = 0]
(3)
8. What is the result of this comparison called?
The Average Treatment Effect, or ATE.
9. In order for this comparison to be a valid measure of the treatment
effect, what must be true (describe and show notation)? Hint: Answer
by comparing potential outcomes in the treated and untreated groups.
What must be true about these?
The two groups must have the same potential outcomes (on average). If
you had switched treatment assignment, the difference in outcomes would
still be the same. The two groups can’t be different for some reason other
than receiving treatment. In short, the two groups must be equivalent in
expectation.
E[Yi (1) | Di = 1] = E[Yi (1) | Di = 0]
(4)
E[Yi (0) | Di = 0] = E[Yi (0) | Di = 1]
(5)
10. If this condition fails (i.e., if the two groups do not have the same
potential outcomes) then what is measured by this comparison?
2
The average effect of treatment on the treated:
E[Yi (1) | Di = 1] − E[Yi (0) | Di = 1] + E[Yi (0) | Di = 1] − E[Yi (0) | Di = 0]
(6)
plus selection bias:
E[Yi (1) | Di = 1] − E[Yi (0) | Di = 1] + E[Yi (0) | Di = 1] − E[Yi (0) | Di = 0]
(7)
11. What is selection bias?
Any factor that makes some subjects more likely to receive treatment and
affects those subjects’ potential outcomes (see Figure 1).
12. How can the researcher know whether or not selection bias is present?
With randomization, we know equivalence holds on average. We know this
because D is determined independently of potential outcomes, so that the
treatment and control groups are both random sample of the entire pool of
subjects. Without randomization, we can’t know, because there could be
imbalance on unobservables.
3
Outcome
Selection
factor
Treatment
Figure 1: Graphical depiction of selection bias.
4