Session 2
Theory of Change and
Measurement
Impact Evaluation Workshop,
Lilongwe, Dec. 14th, 2011
Martin Abel
1
Workshop Schedule – Day 1
• Session 1: Introduction to impact evaluations
• Session 2: Theory of change and measurement
• Session 3: Group work: Theory of Change
•
Session 4: Randomized Evaluation Design
•
Session 5: Group work: Evaluation Design
2
Recap: Measuring Impact (I)
Primary Outcome
Intervention
Impact
Time
3
Outline
• Theory of Change (ToC)
– Definition
– Case Study 1: Village quotas in India
– Mapping from ToC to indicators
– Case Study 2: Primary education in Madagascar
• Data collection
– When to collect data?
– Data collection instruments
4
Theory of Change (ToC)
• Definition:
– A theory of change is a description of how an
intervention is supposed to deliver the desired
results.
• ToC:
– sets out the causal logic of how and why a
particular project, program, or policy will reach its
intended outcomes (hypothesis).
5
Back to Chocolate
• What is the chain of causality from the programme
(chocolate) to the outcome (test score)?
• Do you think those who receive chocolates will
perform better or worse? Why?
6
The ToC takes us beyond the BLACK BOX
Intervention
Intervention
Chocolates given to randomly
selected participants
Inputs
Participants eat chocolates
Outputs
People
Participants are more motivated
like
Black Box Participants are distracted
chocolate
Assumptions
More focused in class
Less focused in class
Intermediary
outcomes
Positive effect on
learning outcomes
Negative effect on
learning outcomes
Outcomes
Black Box
 If we don’t see an effect: where was the causal chain interrupted?
Case Study 1
Quotas in Village Councils in India
Duflo et al. 2008. “Women as policy makers – evidence
from a randomized policy experiment in India”.
8
Case Study Example
Quotas in Village Councils in India
• 1993 Constitutional Amendment in India.
• Main characteristics of quota policy
– Reservation of leadership to women
– Implemented in randomly selected 1/3 of villages
• Main goals of Village Council
– Decentralise decision making, especially public goods
“Every village has to become a self-sufficient republic” (Ghandi)
– Give villagers more say
• Main goals of quota
 Why may these goals not be achieved?
9
Village Quotas: A Possible Theory of Change
Low investment in education and health at village level
Quotas
More female
council leaders
Women are more
empowered
Needs
assessment
Intervention
Output
indicators
Intermediary
outcomes
Different public goods
Different health and education
outcomes?
Outcomes
10
Village Quotas: A Possible Theory of Change
Low investment in education and health at village level
Quotas
Needs
assessment
Intervention
More female
council leaders
Assumptions
Output
indicators
Village leader has
decision power
Women have
different
preferences
Women are more
empowered
Different public goods
Different health and education
outcomes?
Public goods
reflect
women’s
preferences
Intermediary
outcomes
Outcomes
11
ToC to Indicators
• Once the chain of Theory of Change has been
drawn up:
 what data should we collect at every step of
the way to discriminate between different
plausible chains of causality?
12
Village quotes in India
Theory of Change
Low investment in education and health at village level
Quotas
Needs
assessment
Intervention
More female
council leaders
Output
indicators
Women are
empowered
Intermediary
outcomes
Different public goods
Different health and education
outcomes?
Outcomes
13
Village quotas in India
ToC → Indicators
Low investment in education and health at village level
Quotas
Indicators
More female
council leaders
Gender of leader
Women are
empowered
Female attendance in
council meetings; Time
women speak
Different public goods
Budget allocation; Type of
public goods available
Different health & education
outcomes?
Literacy levels, BMI of
children
14
Village quotas in India
ToC → Indicators → Results
Low investment in education and health at village level
Quotas
Indicators

Results

More female
council leaders
Gender of leader
More female leaders
Women are
empowered
Female attendance in
council meetings; Time
women speak
Female attendance 
Active participation
Different public goods
Budget allocation; Type of
public goods available
Allocations shift;
More investment in
roads, drinking water
Different health & education
outcomes?
Literacy levels, BMI of
children
Not yet measured


15
?
Example 2: Cell Phone Usage and
Pedestrian Accidents
• The ToC can help us to identify potential
interventions
• Example: Imagine, you are the mayor of Lilongwe
– In the last 5 years, there was a large increase of both
cell phone usage and number of traffic accidents
involving pedestrians
 What policies could tackle this problem?
16
Case Study 2
Primary Education Management in Madagascar
Nguyen & Lassibile, 2008. “Improving management in education:
evidence from a randomized experiment in Madagascar”. 17
Primary education management in Madagascar
Context
• Improved access to education but quality remains
poor
– Pass rate of 63% among Grade 5s
– Poor teacher performance identified as key problem
• Madagascar Ministry of Education implemented
programme to see whether top-down and/or
bottom-up approach is effective in improving
teacher performance
18
Primary education management in Madagascar
Context
• Two approaches:
– Top-down approach: provide district and subdistrict
administrators with information and tools for
supervision
– Bottom-up approach: encourage parental involvement
through provision of information and accountability
meetings, as well as provide schools with teaching tools
 Which method is most effective at increasing
teacher effort and learning outcomes?
19
Primary education management in Madagascar
Theory of Change: Top-Down
Poor learning levels in primary school
Needs
assessment
Top-down monitoring
programme
Intervention
Officials receive tools and information
Output
indicators
Intensity and frequency of
monitoring increases
Intermediary
outcomes
Teacher performance
improves
Learning outcomes improve
Primary
outcome
20
20
Primary education management in Madagascar
Theory of Change: Top-Down
Poor learning levels in primary school
Top-down monitoring
programme
Needs
assessment
Intervention
Assumptions
Officials receive tools and information
Use of tools
increases
monitoring
Increased monitoring
causes increased
teacher effort
Output
indicators
Intensity and frequency of
monitoring increases
Intermediary
outcomes
Teacher performance
improves
Learning outcomes improve
Primary
outcome
Primary education management in Madagascar
ToC → Indicators
Poor learning levels in primary school
Top-down monitoring
programme
Indicators
Officials receive tools & information
Self-reported receipt and
usage rates
Intensity and frequency of
monitoring increases
No. of visits to schools,
allocation of time & budget
Teacher performance
improves
Attendance, lesson plans,
frequency & quality of
evaluations
Learning outcomes improve
Student attendance, test
scores
22
Primary education management in Madagascar
ToC → Indicators → Results
Poor learning levels in primary school
Top-down monitoring
programme

Indicators
Results

Officials receive tools & information
Self-reported receipt and
usage rates
Tools were received,
tools were used
Intensity and frequency of
monitoring increases
No. of visits to schools,
allocation of time & budget
Schools not visited
more often, allocations
unchanged
Teacher performance
improves
Attendance, lesson plans,
frequency & quality of
evaluations
Teacher behaviour
entirely unaffected
Learning outcomes improve
Student attendance, test
scores
Test scores unchanged
23



Primary education management in Madagascar
Theory of Change: Bottom-Up
Poor learning levels in primary school
Community-based monitoring and
accountability programme
Parents attend
meetings
Teaching tools
delivered to schools
Parents hold schools
accountable
Teachers use of
tools
Teacher performance improves
Learning outcomes
improve
Needs
assessment
Intervention
Output
indicators
Intermediary
outcomes
Primary
24
outcome
24
Primary education management in Madagascar
Theory of Change: Bottom-Up
Poor learning levels in primary school
Needs
assessment
Community-based monitoring and
accountability programme
Intervention
Assumptions
Parents are
interested in
Parents attend
meetings
Teaching tools
delivered to schools
Parents hold schools
accountable
Teachers use of tools
Holding schools
accountable
influences
teacher effort
Teacher performance improves
Learning outcomes
improve
Tools make
teaching
more
effective
Output
indicators
Intermediary
outcomes
Primary
25
outcome
25
Primary education management in Madagascar
ToC → Indicators
Poor learning levels in primary school
Community-based monitoring and
accountability programme
Indicators
Parents attend
meetings
Tools delivered
to schools
No. of participants at
meetings, tools present
Parents hold
schools
accountable
Teachers make
use of tools
Participation of parents,
self-reported usage of
tools
Teacher performance improves
Attendance, lesson
plans, frequency &
quality of evaluations
Learning outcomes improve
Student attendance, test
scores
26
Primary education management in Madagascar
ToC → Indicators → Results
Poor learning levels in primary school
Community-based monitoring and
accountability programme
Indicators
Parents attend
meetings
Tools delivered
to schools
No. of participants at
meetings, tools present
Parents hold
schools
accountable
Schools make
use of tools
Participation of parents,
self-reported usage of
tools
Teacher performance improves
Attendance, lesson
plans, frequency &
quality of evaluations
Learning outcomes improve
Student attendance, test
scores

Results

Attendance at meetings
good, tools received


Attendance unchanged,
lesson plan & eval ↑
Attendance and test

scores ↑
Communication with
parents unchanged,
schools used tools
27
Theory of Change - Summary
• ToC is useful to help define research questions /
hypotheses
– Both for experimental and non-experimental research
• ToC important for identifying all outputs,
outcomes and assumptions we need to measure
• We learn lessons about underlying mechanism
which we can generalize to other programs
– “Projects can rarely be replicated, though the
mechanism underlying success or failure will often be
replicable and transportable.” (Deaton)
• Different chains of causality can have different
policy implications
28
Outline
• Theory of Change (ToC)
– Definition
– Case Study 1: Village quotas in India
– Mapping from ToC to indicators
– Case Study 2: Primary education in Madagascar
• Data collection
– When to collect data?
– Data collection instruments
29
When to collect data?
Not in
evaluation
Target
Population
Evaluation
Sample
Baseline
Survey
Assignment
Treatment
group
Control group
Endline
Survey
Measure
Impact
• Baseline Survey: before the programme is implemented
30
When to collect data?
Not in
evaluation
Target
Population
Evaluation
Sample
Baseline
Survey
Assignment
Treatment
group
Control group
Endline
Survey
Measure
Impact
• Measure inputs and outputs specified in ToC (Process Evaluation)
– Ex: Tutoring program for poor students
• How many tutoring sessions are held? How long is each session?
How many students attend? Is the appropriate teaching material
available?
• RCT compliance: Do people in treatment group receive the
program, and those in control not?
31
When to collect data?
Not in
evaluation
Target
Population
Evaluation
Sample
Baseline
Survey
Assignment
Treatment
group
Control group
Endline
Survey
Measure
Impact
• Endline Survey: after treatment (timing depends on intervention)
– Multiple follow-up surveys possible (medium vs. long-run effects)
• Eg. Village quotas in India – does the perception of women
change over time?
32
Data collection instruments or methods
1.
2.
3.
4.
Administrative data
Focus groups
Survey questionnaires
Non-standard instruments
33
Instruments (I)
1. Administrative data
– Cheap if routinely collected
– May not be accurate, specific to population of interest, or
collect all necessary characteristics
• Case Study 1 - Village quotas in India: 1991 census was used
to draw the sample and check that the treatment and control
groups were balanced
• Case Study 2 - Primary education management in
Madagascar: Scores from the national Grade 5 exam used to
measure learning outcomes
34
Instruments (II)
2. Focus group discussions
– Interview with more than one individual at a time
– Can be cheap way to test feasibility, exhaustibility etc
– Useful for understanding local context
3. Survey questionnaires
– Captures information reported by individual
– Can be cross sectional or panel
• Case Study 2 - Primary education management in
Madagascar: school questionnaires gathered information on
teacher behaviour, usage of tools etc
35
Instruments (III)
• Non-standard instruments:
– Some information can’t easily be collected through standard
instruments
– Eg. Corruption, teacher attendance, political participation, service
delivery, community trust, female empowerment
•
Random spot checks can be useful
– Can make use of unscheduled data collection by:
• Visiting construction sites to measure progress of public goods
delivery
• Measuring teacher and student absenteeism
• Visiting police stations incognito to measure discrimination or
corruption
36
Instruments (IV)
Implicit Association Tests (IAT) – Ex: Measuring Discrimination
• Perceptions of marginalised groups is often interesting to
researchers, but difficult to measure with explicit questions
“A doctor and his son have an accident and the
doctor is killed. The boy is rushed
Crime to surgery, and
the surgeon says, “this is my son.” What is going
on?”
A) The doctor’s wife was unfaithful
University
B) The surgeon is the boy’s mother
Prison
Education
Group Work
Case study:
• Identify Theory of Change
• Identify instruments and outcomes to test the ToC
Your organization’s program:
• Discuss Theory of Change
• What are (intermediary) outcomes and how would you
measure them?
Group Work
Group 1: Participation, Accountability and Transparency
Group Leader: Cristobal Marshall
Organizations: MEJN - Liu Lathu
CCJP
PACENET
Group 2: Participation, Accountability and Transparency
Group Leader: Clare Hofmeyr
Organizations: ACCU
TRAC.fm
MISA
Group 3: Participation, Accountability and Transparency
Group Leader: Martin Abel
Organizations: Church & Society
CHRR
Trocaire
Group 4: Economic Governance / Livelihoods
Group Leader: Thomas Coen
Organizations: MEJN
Local Development Fund
BMAU
Group 5: Agriculture / Livelihood
Group Leader: Michael Roscitt
Organizations: CISANET
ACODE
Oxfam
Group 7: Health
Group Leader: Willa Brown
Organizations: Norwegian Church Aid
Action Aid Malawi
Danish Church Aid
Group 6: Women and Children
Group Leader: Niall Keleher
Organizations: PLAN
CARE
CISCBE
Annex
40
Characteristics of good indicators (I)
I. Measurability: a measurable indicator is
observable, feasible, and detectable
– Observability: indicators must be observed in the
real world.
– Feasibility: indicators must be politically, ethically
and financially measurable
– Detectability: study must have instrument and
statistical power to measure indicator
41
Characteristics of good indicators (II)
II. Precision: the more exhaustive and exclusive
the indicator, the more precise it is
– Exhaustive: most outcomes can be measured by
more than one indicator.
– Exclusive: indicators is affected by the outcome of
interest and nothing else
42
Characteristics of good indicators (III)
III. Reliability: indicators are reliable when they
are hard to counterfeit
– Make sure answer to question is not easily
forgettable
– Deliberate misreporting: try to avoid providing any
incentive for people to lie in their answer
Use proxy estimators that measures the direct
consequence of the same outcome
Remove incentive of respondent to lie by not revealing
which is the ‘desirable’ answer
43