Managing Impact Evaluations: Time, Budget
and Management
Francis Rathinam
Evaluation Specialist, 3ie
[email protected]
Workshop on “Introduction to Impact Evaluation: Challenges and Opportunities
for Policymaking” 3ie-OPM, Kampala,16-17 June, 2016
Step 1: Assess the evaluability of the
programme
• Is it plausible to expect impact?
o Assess the intervention logic or implementation fidelity
• Is it feasible to measure attributable impact?
o Is it possible to construct a control group?
o Baseline data
o Contamination
o Complexity of project design
• Is the timing right?
Step 2: Develop a comprehensive ToR
•
Programme background – A brief description of the programme
•
Objectives – Why this evaluation
•
Intended beneficiaries of the report (the programme team and who else?)
•
Scope of work
•
Activities and responsibilities
•
Deliverables – What should be delivered
•
Schedule of activities – State the expected dates for delivery of reports
•
Qualifications – Profile and level of experience of the research team
•
Proposal/application assessment criteria
•
Payment schedule – link it to the reporting requirement
•
Risks management
Step 3: Put together an evaluation team
• The evaluation team should ideally
include at least one sector specialist,
evaluation specialist, statistician,
qualitative researcher and a survey
team
• More in-country researchers the better
it is
Step 4: Develop, refine and agree on the
design
• Bring the IA staff at different level on board
• This will help ensure that programme implementation adheres to the study design
and in feeding back any changes in the programme to the research team
• Agree on the programme TOC; define clear questions and indicators
• Agree on Methodology:
• Ex ante designs with baseline (building evaluation into programme design)
• There is no one-size-fits-all. Context, time and budget constraints dictate the most
feasible design
• Agree on a timeline for evaluation with clear milestones based on the
programme timeline
• Ask for a pre-analysis plan
Step 4: Develop, refine and agree on the
design…
• Provide routine feedback, especially on
the questionnaires, and baseline and
final draft reports
•
Help contextualise survey instruments
(what to ask and who to ask)
• Participate in interpreting findings and
results.
• Ensure management and implementing
team feed back
Step 5: After the Study Completion
• Engaging with other key stakeholders
around the study to build interest and
discussion on the study findings
• Co-present study with the researchers to
ensure that the IA perspective is
mainstreamed in the dialogue around the
study.
• Take the results back to the community /
beneficiaries
• Summarise the findings for upward
accountability and for further management
decision
Managing Expectations
• What you want
• What researchers want
Role of an IE Manager…
And finally be prepared for delays
Managing IE Budget
Some study costs
IEG BINP study US$40,000-60,000
IEG Ghana education study
US$500,000
Average 3ie study US$420,000+
Average 3ie study in Africa with two
rounds of surveys US$500,000+
A Key Cost Driver
Costs largely driven by
large survey so
additional rounds
increase costs (marginal
costs of increasing
sample size are not so
great)
Some Timelines
Ex post 12-18 months
Ex ante:
• Lead time for survey design 3-6 months
• Time gap between baseline and end line data
collection determined by the outcome variable of
interest
• Report writing and consultation 3-6 months
• Policy uptake
Need Help on Available Evidence?
Databases:
• 3ie IE repository
• 3ie Registry for International Development IEs
(RIDIE)
• JPAL repository
• PREM Impact Site (World Bank)
• Multilateral Banks
Useful 3ie Sources
Social Media
Communication
Briefs and reports
Website
Events
Newsletters
Mailing Lists
Thank you