Steps in Implementing an
Impact Evaluation
Victor Orozco, Development IMpact Evaluation
Initiative (DIME)
Steps
Build
capacity
Feed results
into policy
Set learning
agenda
Design
impact
evaluation
Analyze
data
Collect
follow up
data
Plan for IE
implementa
tion
Roll out
intervention
Conduct
baseline
Step 1. Build capacity for IE

Objectives:
 Become informed consumers of impact evaluation
 Set the learning agenda
 Use it as an internal management tool to improve
program over time

How
 Training
 Learning by doing
Step 2: Set learning agenda

Objective:
 Get answers to relevant policy and operational questions

How?
 Dialectic discussion involving key policy makers and
program managers
 Technical facilitation to structure framework of analysis
 Focus on few critical policy (what) and operational (how
to) questions
 Discuss agenda with authorizing environment and
constituencies
Cont. 2: Questions

Operational: design-choices of program
 Institutional arrangements, Delivery mechanisms, Packages,





Pricing/incentive
Management purpose
Use random trials to test alternatives
Measure effects on short term outcomes (months)
▪ take up rates, use, adoption
Scale up better implementation modalities
Policy: effectiveness of program
 Accountability purpose
 Use random assignment or next best method
 Measure effects medium to long term
 Scale up/down, negotiate budget, inform
Step 3: Design IE

Exploit opportunities:
 Will roll-out take time?
 Is the budget allocated insufficient to cover everyone?
 Are there quantitative eligibility rules?
 If the program has universal access, does it have imperfect take-up?

Set scale:
 Pilot to try out an intervention
 Large scale w. representative sample: more costly, externally valid
 Large scale with purposeful sample: less costly, indicative

Do power calculation to determine minimum
sample size
Cont. Step 3

Select “best” method for each of your
questions
 Feasible
 Requires least assumptions

Ethics
 Not to deny access to something for which there
is irrefutable evidence
 Test interventions before scale up when you have
no solid evidence
Step 4: Planning
implementation

Budget cost items
▪
▪
▪
▪

Staff time (PROJECT FUNDS) and training (DIME)
Analytical services and field coordination (DIME)
Data collection (PROJECT FUNDS)
Discussions and dissemination (shared)
Timeline
▪ Use it to organize activities, responsibilities and work backwards to
know when to start

Team
▪ Government (program manager, economist/statistician); WB
Project team (Task manager or substitute); Research team (Lead
researcher, co-researchers, field coordinator); Data collection
agency
Step 5: Assignment to treatment
and control

The smallest unit of assignment is the unit of
intervention
 Training and Credit: individuals and groups
 Municipal registration system: municipality

Create listing of treatment units assigned to
the intervention and control units that are not

Explain assignment to responsible parties to
avoid contamination
Step 6: Baseline data

Quality assurance : IE team (not data
collection agency) to
 Design questionnaire and sample
 Define terms of reference for data collection
agency
 Train enumerators
 Conduct pilot
 Supervise data collection

Do not collect data before your design is
ready and agreed
Cont. Step 6: Baseline data

Contract data collection agency
 Bureau of Statistics: Integrate with existing data
 Ministry concerned: Ministry of Agriculture/Water
Resources/Rural Development
 Private agency


Analyze baseline data a feed back into
program and evaluation design if needed
Check for balance between treatment and
control group: do they have similar average
characteristics?
Step 7: Roll out intervention



Conduct intensive monitoring of roll-out to
ensure evaluation is not compromised
What if treatment and control receive the
intervention?
What if all the control group receive some
other intervention?
Step 8: Follow-up data


Collect follow-up data with the same sample
and questionnaire as baseline data
At appropriate intervals
Step 9: Estimate program effects

Randomization: compare average outcomes for
treatment and control group

Other methods: Use relevant econometric analysis ,
test assumptions, check robustness

Are the effects statistically significant?
 Basic statistical test tells whether differences are due to the program
or to noisy data

Are they significant in real terms?
 If a program is costly and its effects are small, may not be worthwhile

Are they sustainable?
 Is the trajectory of results sustained?
Step 10: Discuss, Disseminate and
Feedback into policy







Are you thinking about this only now?
Discuss what are the policy implications of the results
What actions should be taken
How to present them to higher ups to justify
changes/budget/scale up?
Talk to policy-maker and disseminate to wider audience
If no one knows about it, it won’t make a difference
Make sure the information gets into the right policy
discussions
 Real time discussions
 Workshops
 Reports
 Policy briefs
Final step: Iterate

What do you need to learn next?