Impact Pathways:
An Approach for Understanding, Fostering
and Evaluating Research-for-Development
Outcomes
Boru Douthwaite, Technology Policy Analyst
Sophie Alvarez, Consultant
Simon Cook, Leader BFPs
Rick Davies, M&E Specialist,
Pamela George, CPWF Program Manager,
John Howell, M&E Specialist,
Ronald Mackay, Professor Emeritus,
Jorge Rubiano, National University of Colombia
CIAT Seminar, 1st November 2006
Impact Pathways Matter
How change happens
• “Improvements in poverty alleviation, food
security and the state of natural resources
result from dynamic, interactive, non-linear,
and generally uncertain processes of
innovation.”
EIARD, 2003
EIARD represents a group of European donors
15 EU Countries plus Norway and Switzerland
Impact Pathways Approach
• People plan and implement projects on the basis of their
change models - their implicit theories about how the
world works
• If you can improve these theories you can improve the
practice, making impact more likely
• Impact Pathways Approach – A participatory approach for:
1. Making practitioners’ theories explicit about how they will achieve
adoption and impact (impact pathways, program theory);
2. Improving these theories
3. Using those models / frameworks for M&E and impact assessment
4. As a result, contributing to project and program “adaptive
management” and thus likelihood of impact
History and Current Work
• Past Work in Nigeria on Striga
– Douthwaite, B., T. Kuby, E. van de Fliert and S. Schulz. 2003. Impact Pathway
Evaluation: An approach for achieving and attributing impact in complex
systems. Agricultural Systems 78 pp243-265
– Douthwaite, B., Schulz, S., Olanrewaju, A., Ellis-Jones, J. 2006. Impact pathway
evaluation of an integrated Striga hermonthica control project in Northern
Nigeria. Agricultural Systems. Published on-line
• Current Work (since Oct. 2005)
– CPWF-supported, CIAT-led impact assessment project in 9 river basins ($900,000)
– Douthwaite, B., Alvarez, B.S., Cook, S., Davies, R., George, P., Howell, J and
Mackay, R. 2006. The Impact Pathways Approach: A Practical Application of
Program Theory in Research-for-Development. For submission to an
Evaluation Journal
• Future Work
– EU-funded, Wageningen-led “eco-system approach for co-innovation of farm
livelihoods” project (Euro 1.8 million with 6 PhDs and 4 PostDocs)
– Phase II of Knowledge Sharing for Research Project (with Simone Staiger)
– PRGA INIS Project (CIAT and CIP led)
• See www.impactpathways.pbwiki.com
Logic model
>---Results-orientated perspective---->
Impact pathways – two
conceptualizations….
<-----the full picture---->
Network maps
>----------Actor-orientated perspective--------->
Impact Pathways
Two complementary conceptualizations of a project,
a program or an organization’s impact pathways:
1. A visual description of the causal chain of events and
outcomes that link outputs to the goal (logic model);
and
2. Network maps that show the evolving relationships
necessary to achieve the goal
•
Implementing organizations; stakeholders; ultimate
beneficiaries
Shows the project rationale; its logic
Foundation of ex-ante (and ex-post) impact
assessment
Foundations of the IP Approach
• Synthesis of concepts and tools from:
– Program Evaluation
• Renger and Titcomb (2002) – problem trees
• Chen (2005) – program theory
• Mayne (2004) - performance stories
– Innovation histories
• Douthwaite and Ashby, 2005
– Appreciative Inquiry
• Whitney and Trosten-Bloom, 2003
– Social network analysis
• Cross and Parker, 2004
The Process of Developing Impact Pathways
OUTPUTS OF IMPACT PATHWAYS (IP) WORKSHOP
PRODUCTS PRODUCED AFTER WORKSHOP
(Draft produced before workshop)
Developing a results-orientated
view of a project's IP
Project Problem Tree
Project Objective Tree
What the
project will
produce
Developing an actor-orientated
view of a project's IP
How project
goes from
outputs to goals
Necessary
relationships
to produce
the OUTPUTS
Outputs
Vision
Helps understand
project rationale
What needs to change
Where project is
going- Goal
Project Timeline
Extrapolation
Domain Analysis
IP logic model
Two descriptions
of the project's
impact pathways
Iterative
process
"Now" network
map
"Future"
network map
Necessary
relationships
to achieve
the VISION
Scenario
Analysis
Network maps
(Relationship by
relationship)
Impact
Narrative
Participatory
Development of
Impact Pathways
Example of a Problem Tree
Turning a problem tree into an objective
tree
PROJECT OUTPUT
4th LEVEL
Feeling of Ownership
3rd LEVEL
System of
maintenance in place
2nd LEVEL
1st LEVEL
Reservoirs/Small
dams ensembles
well maintained
Good tools for
planning, development
and support of
sustainable use of
reservoirs
Sufficient
Water
Good coordination of
reservoirs as a
system
Project Goal
Good use of existing
local reservoirs
Project 46: Small Multi-purpose Reservoir Ensemble Planning
Objectives Tree
Reservoir
ensembles
functioning
optimally
Livelihoods
secured
The Process of Developing Impact Pathways
– The Workshop
OUTPUTS OF IMPACT PATHWAYS (IP) WORKSHOP
PRODUCTS PRODUCED AFTER WORKSHOP
(Draft produced before workshop)
Developing a results-orientated
view of a project's IP
Project Problem Tree
Project Objective Tree
What the
project will
produce
Developing an actor-orientated
view of a project's IP
How project
goes from
outputs to goals
Necessary
relationships
to produce
the OUTPUTS
Outputs
Vision
Helps understand
project rationale
What needs to change
Where project is
going- Goal
Project Timeline
Extrapolation
Domain Analysis
IP logic model
Two descriptions
of the project's
impact pathways
Iterative
process
"Now" network
map
"Future"
network map
Necessary
relationships
to achieve
the VISION
Scenario
Analysis
Network maps
(Relationship by
relationship)
Impact
Narrative
Level of influence of Project
High
C
O
N
T
R
O
L
Low
Research
activity
Output
target
Output
Outcome
Impact
Scaling Out and Scaling Up
• Scaling up - an
institutional expansion,
from adopters and their
grassroots organizations
to policy makers, donors,
development institutions
• Scaling out - spread of a
project outputs (i.e., a
new technology, a new
strategy, etc.) from farmer
to farmer, community to
community, within the
same stakeholder groups
Develop a vision of project success two
years after the project ends
• Work in project groups
• Take 5 minutes to individually answer the question
Workshop
– You wake up 2 years after your project has ended. Your project
has been a success and is well on its way to achieving its goal.
Describe what this success looks like to a journalist:
Keep it realistic
• What is happening differently now?
• Who is doing what differently?
• What have been the changes in the lives of the people using the
project outputs, and who they interact with?
• How are project outputs disseminating (scaling-out)?
• What political support is nurturing this spread (scaling-up)? How did
that happen?
• Discuss and develop a common vision
Example
of a Vision
The Process of Developing Impact Pathways
– The Workshop
OUTPUTS OF IMPACT PATHWAYS (IP) WORKSHOP
PRODUCTS PRODUCED AFTER WORKSHOP
(Draft produced before workshop)
Developing a results-orientated
view of a project's IP
Project Problem Tree
Project Objective Tree
What the
project will
produce
Developing an actor-orientated
view of a project's IP
How project
goes from
outputs to goals
Necessary
relationships
to produce
the OUTPUTS
Outputs
Vision
Helps understand
project rationale
What needs to change
Where project is
going- Goal
Project Timeline
Extrapolation
Domain Analysis
IP logic model
Two descriptions
of the project's
impact pathways
Iterative
process
"Now" network
map
"Future"
network map
Necessary
relationships
to achieve
the VISION
Scenario
Analysis
Network maps
(Relationship by
relationship)
Impact
Narrative
Develop a project timeline from when your
project started until 2 years after it will end
Workshop
• Build a timeline of activities, outputs and
outcomes that take you from the beginning of the
project to achieving the vision
• It is a story of adoption of project outputs
(scaling-out) and the political support that helps
it along (scaling-up)
Example of a Timeline
The Process of Developing Impact Pathways
– The Workshop
OUTPUTS OF IMPACT PATHWAYS (IP) WORKSHOP
PRODUCTS PRODUCED AFTER WORKSHOP
(Draft produced before workshop)
Developing a results-orientated
view of a project's IP
Project Problem Tree
Project Objective Tree
What the
project will
produce
Developing an actor-orientated
view of a project's IP
How project
goes from
outputs to goals
Necessary
relationships
to produce
the OUTPUTS
Outputs
Vision
Helps understand
project rationale
What needs to change
Where project is
going- Goal
Project Timeline
Extrapolation
Domain Analysis
IP logic model
Two descriptions
of the project's
impact pathways
Iterative
process
"Now" network
map
"Future"
network map
Necessary
relationships
to achieve
the VISION
Scenario
Analysis
Network maps
(Relationship by
relationship)
Impact
Narrative
Family ties
Friendship ties
Workplace ties
Today’s tasks…..
Workshop
1. Identify relevant actors & relationships
2. Develop network diagrams for
•
•
Your project now
Residual network 2 years after project has finished
3. Identify key extension (scaling out) and political
support (scaling up) linkages
4. Identify differences between the two networks
and discuss implications
Differences between the MUS
project’s maps
Description
1 Research
2 Out-scaling,
Up scaling
During project period Within 2 yrs after
project period
Led by implementing
Led by stakeholders with
organizations
minimum role played by
implementing
organizations.
Co-development of
Led by: Agriculture
project outputs
Department, Irrigation
involving implementing Department, Water users
orgs and key
federations & private
stakeholders
sector
The Process of Developing Impact Pathways
– After the Workshop
OUTPUTS OF IMPACT PATHWAYS (IP) WORKSHOP
PRODUCTS PRODUCED AFTER WORKSHOP
(Draft produced before workshop)
Developing a results-orientated
view of a project's IP
Project Problem Tree
Project Objective Tree
What the
project will
produce
Developing an actor-orientated
view of a project's IP
How project
goes from
outputs to goals
Necessary
relationships
to produce
the OUTPUTS
Outputs
Vision
Helps understand
project rationale
What needs to change
Where project is
going- Goal
Project Timeline
Extrapolation
Domain Analysis
IP logic model
Two descriptions
of the project's
impact pathways
Iterative
process
"Now" network
map
"Future"
network map
Necessary
relationships
to achieve
the VISION
Scenario
Analysis
Network maps
(Relationship by
relationship)
Impact
Narrative
Impact Narrative
• Text description of the project impact pathways
• Achieves the integration between the logic and
network models
• Helps with colligation (tracing of logical steps,
Roberts, 1996), making hidden assumptions
explicit
• Helps with the plausibility of ex-ante impact
assessment
Project Outputs
(boxes 1-10)
Drought probability map
Crop production guides
or manuals for MoFA
2
Produced before
project end
Drought tolerant
varieties developed
3
Best-bet soil and
water conservation
and management
4 options manuals
6
Manuals on fish
culture in dugouts
and dugout
maintenance
Farmers using
drought
probability map
and drought
tolerant varieties,
stop using
traditional ones
Changes in
farmers'
attitudes and
perceptions
Improved
knowledge of
farmers at
pilot sites
5
Iterations of
learning
cycle
Adoption of
technologies
and changes
in practice
Farmers
modify and
innovate
Farmers plant to
avoid crop loss due
to draught,
majority have
intensified cropping
systems
7
10
Farmers routinely
generate organic
matter , e.g.
composting and cover
cropping
14
from
OP-VISION
key
15
from ObjT
18
National variety
release committee
responds promptly
to release varieties
20
Majority of communities in
Northern Ghana have
constructed and are using
domestic water harvesting
systems
12
13
Formal structures for
governance of
community water
resources operational
(?)
22
High labour
productivity
25
More water
available for
domestic needs
Adequate water
supply for dry
season
agriculture 26
Improved
food
security
High land and
water
productivity
21
Wider adoption
of project
outputs beyond
pilot sites
27
28
Changes to housing structure to
meet water harvesting needs
Adoption of project
outputs by MOFA
for extension after
project finishes
Communities have
knowledge of low-cost
domestic waterharvesting systems
Assumption
More time for
income
generating
activities for
women
24
Soil and water
conservation improved
in farmlands in N.
Ghana
Improved utility of
community dugouts
Scaling up
Dugouts enhanced to
retain water
from
Workshop
Improved
income for
women in rural
households
17
19
11
from TL
Project Goals
and Vision 2011
Improved cropping
systems in Northern
Ghana
Improved soil fertility
16
Manuals on
appropriate water
harvesting
systems
Farmers using
appropriate tillage
methods to conserve
soil moisture
Community dugouts efficiently utilized
for fish production
Communities trained on
efficient fish production
techniques
8
9
Scaling Out
1
pilot site testing and
demonstrations of these
4 outputs (1-4)
IP Logic Model (Draft 1)
Effective management
of community water
resources
(?)
30
23
29
Reduction in water related
diseases (?)
IP Logic Model (Final)
Project Activities
carried out in Pilot
Sites with boundary
partners and
ultimate
beneficiaries
1
Crop Related Outputs
Crop Related Outcomes
3
Drought probability
map
Crop production
guides or manuals for
MoFA
Scaling
Out
Best-bet soil and water
conservation and
management options
manuals
Changes in
stakeholders
attitudes and
perceptions
2
Improved cropping
systems in Northern
Ghana
Higher crop
yields
8
5
Farmers plant to
avoid crop loss due
to draught, majority
have intensified
cropping systems
Drought tolerant
varieties developed
Improved
knowledge of
stakeholders
at pilot sites
Farmers using
appropriate
tillage methods
to conserve
soil moisture
Farmers using
drought probability
map and drought
tolerant varieties
Farmers routinely
generate organic
matter , e.g.
composting and
cover cropping
Soil and water
conservation improved
in farmlands in N.
Ghana
Scaling Up
Adoption of
National variety release
committee releases
varieties
Iterations of technologies
learning
and changes
cycle
in practice
Wider adoption of project outputs beyond
pilot sites
7
Adoption of project outputs by MoFA for
extension after project finishes
Stakeholders
modify and
innovate
Communities
trained on
efficient fish
production
techniques
Manuals on fish culture in
dugouts and dugout
maintenance
Manuals on
appropriate water
harvesting systems
11 up
Scaling
Methods developed to
institutionalize dialogue
about water use among
multiple users
Water Related Outputs
4
Scaling
Out
Dugouts
enhanced to
retain water
Communities
have knowledge
of low-cost
domestic waterharvesting
systems
Improved soil
fertility
More time for
income
generating
activities for
women
More water
available for
domestic needs
Adequate water
supply for dry
season
agriculture
Reduction in water
related diseases
Community
dugouts
efficiently
utilized for fish
production
Changes to
housing
structure to
meet water
harvesting
needs
Water Users
Associations formed
and strengthened
Improved utility of
community dugouts
10
Majority of
communities in
Northern Ghana have
constructed and are
using domestic water
harvesting systems
Effective management
of community water
resources
6
Water Related Outcomes
9
Project Goals
Improved
income for rural
households
High labour
productivity
Improved food
security and
rural
livelihoods
High land and
water
productivity
11
Network Maps
Separate relationships
Final maps based on answers
The theory behind the IP approach
=
Impact Pathways
Program Theory
(Chen, 2005)
Normative Theory
(What is expected - project
milestones, etc.)
Causative Theory
(Explanations of causation)
Explicit Theory
(Theories as explained
to others)
Implicit Theory
(Personal theories
of action)
Greater congruence
increases project
effectiveness
(Argyris, 1980;
Patton, 1997)
“Stakeholders' implicit theories are not likely to be systematically and explicitly
articulated, and so it is up to evaluators to help stakeholders elaborate their ideas.”
(Chen, 2005, p. 14)
Impact of IPs 1
•
From Workshops
–
–
–
–
–
–
–
“I will use Impact Pathways in future design of
projects”
“The dynamics of the networks is useful to envision
the future”
“It helps show gaps”
“It is good for planning”
“It helps explain impact of my project”
“Constructing impact pathways should not be oneshot”
“The impact pathways should be a living document”
Impact of IPs 2
•
Significant Change Stories resulting the Volta IP
Workshop
1. Locally-organized IP Workshop to build basin
program
2. Exploiting an opportunity for political lobbying
3. Network concepts hybridized with influence mapping
to become main PhD research methodology
4. Problem and objective trees used to explain project to
primary stakeholders
Impact of IPs 3
•
Science Council review of CPWF 2007 –
2009 Medium Term Plan
–
‘The CPWF has introduced the use of
“objective trees” at the MTP project and CP
level, a useful and innovative complement to
the MTP logframe. In addition to providing a
useful overview, the process of preparing
these flow charts has clearly helped the CP
provide the necessary focus, clarity and
cohesion that now exists in the research plans
at all levels.’
Impact Pathways Evaluation
• Monitoring and evaluating progress along impact
pathways
– Regularly updating objective tree, timeline and network
maps
– Most Significant Change to pick up unexpected
consequences
– Provides the information needed for “adaptive
management”
• Impact Pathways Evaluation = Action research
– Is publishable; raises the status of M&E
IPs and ex-post Impact Assessment
• Ex-post impact assessment should identify
and describe (EIARD, 2003):
– The concept or model of innovation;
– The logic model underlying a project or
program
– In other words, the normative and causative
program theory
Research Questions
• What types of network should R4D programs
attempt to foster to achieve impact?
• Food security, poverty alleviation, improved health,
environmental security
• How does improving the congruence between
implicit and explicit stakeholder theories improve
research-for-development effectiveness?
• Are impact pathways generalizable?
– Can one project’s impact pathways help planning and
implementation of others?
IPs – the elevator conversation
• What do you do?
– Work to mainstream an impact orientation in agricultural R4D
projects and programs
• What is impact orientation?
– That the people implementing projects are clearer about how their
research will make a difference, and take responsibility that it does
• How do you do that?
– Work with practitioners to make explicit their impact pathways
– Carry out research to understand which impact pathways work,
when
•
•
•
Summary
Change Models matter: People plan and implement
projects on the basis of their change models - their
implicit theories about how the world works
If you can improve these theories you can improve the
practice, making impact more likely
The Impact Pathways Approach – A participatory
approach for:
1.
Making practitioners’ theories explicit about how they will
achieve adoption and impact (impact pathways);
2.
Improving these theories
3.
Using these models / frameworks for M&E and impact
assessment
•
4.
•
IP Evaluation = Action research
As a result, contributing to project and program “adaptive
management” and thus likelihood of impact
A work in progress, including research on network
structures and impact pathways
Impact Pathways Matter