Non-Experimental Methods
AADAPT Workshop South Asia
Goa, December 17-21, 2009
Aim: We want to isolate the causal effect of our
interventions on our outcomes of interest
 Use rigorous evaluation methods to answer our
operational questions
 Randomizing the assignment to treatment is the “gold
standard” methodology (simple, precise, cheap)
 What if we really, really (really??) cannot use it?!
>> Where it makes sense, resort to non-experimental
methods

Can we find a plausible counterfactual?
 Natural experiment?

Every non-experimental method is associated
with a set of assumptions
 The stronger the assumptions, the more doubtful
our measure of the causal effect
 Question our assumptions
▪ Reality check,
resort to common sense!
3
 Principal Objective
▪ Increase firm productivity and sales
 Intervention
▪ Matching grants distribution
▪ Non-random assignment
 Target group
▪ SMEs with 1-10 employees
 Main result indicator
▪ Sales
4
Control Group
Treatment Group
14
(+) Impact of the
program
12
10
(+) Impact of external
factors
8
6
4
2
0
Before
After
5
Control Group
Treatment Group
14
(+) BIASED Measure of the
program impact
12
10
8
6
4
2
0
Before
After
“Before-After” doesn’t deliver results we can believe in!
6
Comparison Group
Treatment Group
« Before»
difference btwn
participants and
nonparticipants
14
12
10
8
« After »
difference btwn
participants and
non-participants
6
4
2
0
Before
After
>> What’s the impact of our intervention?
7
Counterfactual:
2 Formulations that say the same thing
1.
Non-participants’ sales after the intervention,
accounting for the “before” difference between
participants/nonparticipants (the initial gap between
groups)
2.
Participants’ sales before the intervention, accounting
for the “before/after” difference for nonparticipants (the
influence of external factors)

1 and 2 are equivalent
8
Underlying assumption:
Without the intervention, sales for participants and
non participants’ would have followed the same trend
>> Graphic intuition coming…
Average sales
(1000s)
2007
2008
Difference
(2007-2008)
Participants (P)
1.3
1.9
0.6
Non-participants (NP)
0.6
1.4
0.8
Difference (P-NP)
0.7
0.5
-0.2
10
Average sales
(1000s)
2007
2008
Difference
(2007-2008)
Participants (P)
1.3
1.9
0.6
Non-participants (NP)
0.6
1.4
0.8
Difference (P-NP)
0.7
0.5
-0.2
11
Impact = (P2008-P2007) -(NP2008-NP2007)
= 0.6 – 0.8 = -0.2
2
P2008-P2007=0.6
1.5
1
NP2008-NP2007=0.8
0.5
0
2007
Participants
2008
Non-Participants
12
Impact = (P-NP)2008-(P-NP)2007
= 0.5 - 0.7 = -0.2
2
P-NP2008=0.5
1.5
P-NP2007
1 =0.7
0.5
0
2007
Participants
2008
Non-Participants
13
Impact=-0.2
2
1.5
1
0.5
0
2007
Participants
2008
Non-Participants

Negative Impact:
 Very counter-intuitive: Increased input use should not
decrease sales once external factors are accounted
for!

Assumption of same trend very strong
 2 groups were, in 2007, producing at very different
levels
➤ Question the underlying assumption of same trend!
➤When possible, test assumption of same trend with
data from previous years
2.5
2
1.5
participants
1
non-participants
0.5
0
2006
2007
2008
>> Reject counterfactual assumption of same trends !
Average Sales
(1000s)
2007
2008
Difference
(2007-2008)
Participants (P)
1.5
2.1
0.6
Non-participants (NP)
0.5
0.7
0.2
Difference (P-NP)
1.0
1.4
0.4
17
Impact = (P2008-P2007) -(NP2008-NP2007)
= 0.6 – 0.2 = + 0.4
2.5
2
P08-P07=0.6
1.5
participants
non-participants
1
NP08-NP07=0.2
0.5
0
2007
2008
18
2.5
2
Impact = +0.4
1.5
participants
non-participants
1
0.5
0
2007
2008

Positive Impact:
 More intuitive

Is the assumption of same trend reasonable?
➤ Still need to question the counterfactual
assumption of same trends !
➤Use data from previous years
2.5
2
1.5
participants
non-participants
1
0.5
0
2006
2007
2008
>>Seems reasonable to accept counterfactual
assumption of same trend ?!

Assuming same trend is often problematic
 No data to test the assumption
 Even if trends are similar the previous year…
▪ Where they always similar (or are we lucky)?
▪ More importantly, will they always be similar?
▪ Example: Other project intervenes in our
nonparticipant firms…

What to do?
>> Be descriptive!
 Check similarity in observable characteristics
▪ If not similar along observables, chances are trends
will differ in unpredictable ways
>> Still, we cannot check what we cannot see… And
unobservable characteristics might matter more
than observable (ability, motivation, patience, etc)
Match participants with non-participants on the basis of
observable characteristics
Counterfactual:

Matched comparison group
 Each program participant is paired with one or more
similar non-participant(s) based on observable
characteristics
>> On average, participants and nonparticipants share the
same observable characteristics (by construction)
 Estimate the effect of our intervention by using
difference-in-differences
24
Underlying counterfactual assumptions

After matching, there are no differences between
participants and nonparticipants in terms of
unobservable characteristics
AND/OR

Unobservable characteristics do not affect the
assignment to the treatment, nor the outcomes of
interest

Design a control group by establishing close
matches in terms of observable characteristics
 Carefully select variables along which to match
participants to their control group
 So that we only retain
▪ Treatment Group: Participants that could find a match
▪ Comparison Group: Non-participants similar enough to
the participants
>> We trim out a portion of our treatment group!

In most cases, we cannot match everyone
 Need to understand who is left out

Example
Matched
Individuals
Portion of treatment
group trimmed out
Nonparticipants
Participants
Score
Wealth

Advantage of the matching method
 Does not require randomization
28

Disadvantages:
 Underlying counterfactual assumption is not
plausible in all contexts, hard to test
▪ Use common sense, be descriptive
 Requires very high quality data:
▪ Need to control for all factors that influence program
placement/outcome of choice
 Requires significantly large sample size to
generate comparison group
 Cannot always match everyone…
29
Randomized-Controlled-Trials require minimal
assumptions and procure intuitive estimates
(sample means!)
 Non-experimental methods require assumptions
that must be carefully tested

 More data-intensive
 Not always testable

Get creative:
 Mix-and-match types of methods!
 Address relevant questions with relevant techniques
30