CCSL Learning Brief No.9 ••
September 2014
Lessons in theory of
change: monitoring,
learning and evaluating
Knowledge to Action
Key messages
•Reflection on the theory of change (TOC) is just as important as measuring
indicators of progress in a monitoring, learning and evaluation (ML&E) plan.
Authors ••
Christine C. Jost1,2
Patti Kristjanson
1,2
Joost Vervoort1,3
Sophie Alvarez1,4
•Developing a brief outcome story for your impact pathway helps you to
think through the necessary changes in next-user behaviours, why those
changes are important, and what results those changes can generate.
•Outcome indicators provide clear evidence of progress towards
development outcomes.
Nafisa Ferdous1,2
Wiebke Förch1,5
1 CGIAR Research Program on Climate Change,
Agriculture and Food Security (CCAFS)
2 World Agroforestry Centre (ICRAF), Nairobi, Kenya
3 Environmental Change Institute, University of
Oxford, South Parks Road, OX1 3QY Oxford,
United Kingdom
4 International Centre for Tropical Agriculture (CIAT)
5 International Livestock Research Institute (ILRI),
Nairobi, Kenya
About Us ••
The CGIAR Research Program on Climate
Change, Agriculture and Food Security (CCAFS)
brings together the world’s best researchers
in agricultural science, development research,
climate science and Earth System science,
to identify and address the most important
interactions, synergies and tradeoffs between
climate change, agriculture and food security.
CCAFS is a strategic parternship of CGIAR and
Future Earth, led by the International Center for
Tropical Agriculture (CIAT). www.ccafs.cgiar.org
Impact pathway development
for theme Knowledge to Action
In 2010 the research theme on
Knowledge to Action (K2A) at CGIAR
Research Program on Climate Change,
Agriculture and Food Security (CCAFS)
developed a plan of work, using a
log frame structure. Our objective
was to explore and jointly apply
approaches and methods that enhance
K2A linkages with a wide range of
partners at local, regional and global
levels. Since then, the K2A theme has
supported a variety of projects with
the potential to catalyse action from
research-generated knowledge. These
projects were cutting edge; high risk
but with potential for real impact
should they succeed.
The success rate of individual projects
was substantial.1 What we found as
a research theme, however, was that
mechanisms were not in place to scaleout from these initial, project-based
successes. We attributed this to the
absence of a clear pathway to impact.
Impact pathways were inherent in our
log frame approach. However, explicit
“result chains” connecting research
products to development outcomes
and monitoring and evaluation (M&E)
strategies were missing. In 2013, we
converted our log frame into an impact
pathway with accompanying theory of
change (TOC). This allowed us to shift
from a project orientation to a focus
on behavioral outcomes.
Our log frame had one objective and
four outputs. For our TOC we stated
each of these as explicit outcomes
with specific and measurable changes
in next-user knowledge, attitude,
skills and practice. Each output was
converted into an impact pathway.
CCAFS requires its research themes
to achieve and report at least one
outcome per year. Searching for
ways to be innovative and clarify our
outcomes-thinking process, we decided
to summarize each of the specific
stories we wanted to tell per behavioral
outcome. Our overall Knowledge to
Action outcome story became:
CCSL Learning Brief No. 9 •• September 2014
Lessons in theory of change: monitoring, learning and evaluating Knowledge to Action
By 2016, decision-makers are
enacting food security and
climate resilience strategies and
policies favoring highly vulnerable
groups so as to enable innovative
smallholder farmer adaptation and
mitigation in 20 CCAFS countries,
5 CCAFS regions and in key global
processes.
The story clearly articulates necessary
changes in next-user behaviors, why
those changes are important, and what
impact those changes should enable.
The story is accompanied by four
outcome stories representing each of
our impact pathways. For example, our
outcome story for the CCAFS regional
socio-economic scenarios program2,
completed in 2013, was:
regions, economic communities
and international organizations
are collaborating to invest in this
process for creating an enabling
policy environment.
By the end of 2013, national
and regional stakeholders in
the five CCAFS regions are
focusing on developing more
appropriate adaptation and
mitigation strategies and policies
using forward-looking climate
change focused socio-economic
scenarios. In at least two of our
The Knowledge to Action impact
pathways also contain major output
groups (MOGs)3, major research actions
(MRAs)4 and specific projects. Thus,
the CCAFS Scenarios Program impact
pathway contained the elements
shown in Figure 1.
Figure 1.
Policies and Institutions IDO
Knowledge to Action
Outcome
CCAFS Scenarios
Program Outcome
MOG: By 2013, regional scenarios finalized in a process that includes developing
capacity in regional agencies and other stakeholders
MRA for Engagement through Scenarios
(Society for International Development):
Engagement of the private, public,
research, non-governmental and
international sectors leading to
joint climate change and agriculture
strategy development
MRA: for Scenario Development
(Oxford University): Development
of regional scenarios in East and
West Africa and South Asia through
a series of participatory workshops
and modeling
Project:
Participatory
socio-economic
scenarios
development
in the five CCAFS
regions
Project:
Quantification
of regional socioeconomic
scenarios
using IMPACT,
GLOBICOM and
climate change
models
Page 2
Project: Training
of scenarios
champions
in East Africa
Project: Support
to scenarios
champions to
influence the
integration
of smallholder
farmers into
policy
CCSL Learning Brief No. 9 •• September 2014
Lessons in theory of change: monitoring, learning and evaluating Knowledge to Action
Identifying indicators5
Initially, we brainstormed monitoring
and evaluation indicators based on
what would be the most holistic
and convincing evidence of progress
along our impact pathways, and
achievement of our outcomes. We
identified progress6, change7 and
impact8 indicators, paying particular
attention to change and impact
indicators that would provide
evidence of contribution to the CCAFS
intermediate development outcomes
(IDO)9.
We found, however, that these higherlevel indicators cut across themes
and regions in CCAFS, and had to be
arrived at through a time-consuming
consultative process involving all
CCAFS stakeholders (e.g. Flagship
leaders, Regional leaders, Coordinating
Unit, etc.). So we decided to work
with the main partners in each
impact pathway to identify indicators
and develop an outcome-focused
monitoring, learning and evaluation
(ML&E) plan, specific to their
pathways. We assume that these
indicators will also provide evidence of
progress towards and achievement of
our overall outcome story.
evaluation system. Several meetings
were held with the scenarios team to
develop the system with the program’s
research question in mind. In 2010
we started with the double question:
“Can a participatory approach be
used to develop regional socioeconomic scenarios for use in climate
change and agriculture planning, and
if developed will stakeholders use
them?” The impact pathway ended
in 2013 with an external evaluation10
using the ML&E indicators, providing
the results in Table 1.
An example is the CCAFS Scenarios
program monitoring, learning and
Table 1.
Indicator type Impact pathway Indicator
linkage
Results
Change
Milestone to
outcome story
Number of partner organizations
that are participating in the
scenarios process by region
240 organizations (governments, regional
economic bodies, private investors, farmers’
organizations, CSOs and NGOs, academia and
the media) have participated in regional scenarios
processes in the five CCAFs regions.
Progress
Activity to
milestone
Number and type of participants
in each workshop
The 240 organizations participating in the scenarios
processes were represented by 361 individuals
ranging from mid-level to senior individuals in their
organizations.
Change
Milestone to
outcome story
Increasing demand for scenarios
process: number of actors that
have requested or are supporting
the scenarios process in the
CCAFS regions
4 global partners (FAO, UNEP WCMC, Oxfam
Great Britain and GFAR) are supporting scenarios
processes. In the regions, 15 regional and national
organizations (regional economic bodies like
ECOWAS, EAC, ASEAN and SICA, governments,
civil society organizations and NGOs) have provided
active support for scenarios processes.
Change
Milestone to
outcome story
Funds provided for scenarios
process by partner organizations
in CCAFS regions
Global and regional partners have provided
660.000 USD (and an additional estimated 80.000
USD for smaller meetings) to the development of
scenarios and their use for policy and investment
guidance.
Progress
Activity to
milestone
Increasing use of scenarios in
CCAFS regions; number of
champions emerging; number
and types of partners engaged
in forward planning for climate
change adaptation and mitigation
81 impact pathways to guide policy and investment
were proposed to be taken forward by participants
across the five regions, of which 24 have been
initiated so far.
Progress
Activity to
milestone
Number of champion success
stories
The evaluation report refers to its regional
outcomes where many success stories are
underway. The report highlights 5.
Page 3
CCSL Learning Brief No. 9 •• September 2014
Lessons in theory of change: monitoring, learning and evaluating Knowledge to Action
Table 1. (continued)
Deliverable
Deliverable
Number of views, forwards and
tweets of scenarios blog
Scenarios pages all together (including documents,
landing page etc.) were visited 11,073 times
since the inception of the programme in 2010, of
which 4,299 visits were in 2013. Overall, the 16
blogs about CCAFS scenarios were visited 4418
times (since January 2010), of which 1821 visits
were to the seven blogs published in 2013There
have been 18 reposts of scenarios blogs/reports/
papers including on DFID, UN main page, FAO,
GFAR, Reuters, Al Jazeera and in regional/national
newspapers. Videos on the scenarios processes had
1025 views.
Change
Outcome story
to overall theme
outcome
New institutional arrangements
resulting from scenario
champions’ work
CCAFS has been included as part of the ECOWAS
task force, thus internalizing CCAFS in the ECOWAS
decision making process. In Central America the
‘the strengthening of the relations of the Executive
Secretary of CCAD with other public and private
entities in the region’ was highlighted by CCAD.
Table 2.
Type
Indicator
Process
(i) increasing demand for scenarios process (regions, globally), (ii) Increasing use of scenarios outputs or
methodology in CCAFS regions
Process
Increase engagement in forward planning for climate change adaptation and mitigation
Process
Increasing capacity for scenarios champions to generate successful policy results
Process
Increasing public interest in the CCAFS scenarios program
Impact
Proportion of CCAFS Scenarios Program budget met by partner organizations
Impact
Increasing number and diversity of partners engaged in the CCAFS Scenarios program
Impact
Increasing number of agriculture and food security policies, plans and investments that address climate change
adaptation and/or mitigation
Next steps
By including a process of reflection
in our ML&E, we discovered that
explorative scenarios and normative
back-casting methods require
facilitation and capacity building over
the longer-term11, because making
these processes impactful requires
more than a single intervention or
workshop. Based on our positive
initial results, in 2014 we started a
second phase of research for the
CCAFS Scenarios program. Based on
the new question, “Can the CCAFS
socio-economic scenarios be used by
partners to improve decision-making
for better food security, livelihoods and
environments in the CCAFS regions?,”
we have designed a two-year research
program of bounded, partner-led
policy case studies in five regions,
using pre-defined policy objectives and
consensus indicators.
In addition to the indicators we are
Page 4
using for the case studies – policies
under revision, improved capacity
to plan from a systems perspective,
credible planning, integrated
perspectives and collaborative action,
partner constellations, and changes in
decision-making – we will monitor the
indicators outlined in Table 2 along the
impact pathway.
Our theory of change for this
research assumes that policy and
institutional change occur because of
the availability of objective, science-
CCSL Learning Brief No. 9 •• September 2014
Lessons in theory of change: monitoring, learning and evaluating Knowledge to Action
based evidence and the ability of key
stakeholders to understand and use
that evidence, but more importantly
because these stakeholders are
capable of advocacy from the
top down and bottom up. The
Coordinating Unit and Themes of
CCAFS have a strong focus on policy
change at the top. Therefore, a gap
exists in identifying and supporting
ways to influence strategies, planning
and process from the bottom.
We need to support the capacity of
stakeholders in the CCAFS regions
to use the CCAFS socio-economic
scenarios, combined with back-casting
planning approaches as tools for
guiding decision-making. In addition
to monitoring indicators of progress
and behavioral change, we also need
to review our theory of change and
impact pathway annually by asking
ourselves, “Given the experience of
the past year, do these assumptions
continue to hold true?” and “Is our
strategy to support policy processes
from the bottom-up enabling progress
along our impact pathway?”
Lessons learnt
Achieving sustainable food security
in a world of growing populations,
changing diets and climate change
requires rapid shifts in food systems at
a global scale involving billions of poor
and vulnerable people. Changes at
this scale require new and innovative
approaches to climate change and
agriculture research, information
dissemination, networking and
behavior change. Changes in nextuser knowledge, attitudes, skills and
practices will have to happen to enable
smallholder farmers to adopt new
climate-resilient agricultural practices,
which mandates timely, high impact
research with stakeholders. This is the
overall Knowledge to Action theory of
change.
We have learned through the process
of developing a way to monitor our
impact pathways that annual reflection
on the TOC is just as important as
measuring indicators of progress
and change. On the outcome story
we wanted to tell challenged us to
Page 5
review the evidence being generated
by our projects, and to redesign our
portfolio by revising our research
questions. Outcome indicators
that measure changes in next-user
behavior provide much clearer
evidence of progress towards impact
than do output indicators. Thinking
about the potential contribution of
a collaborative research program’s
(CRP) current work and future impact
pathways to the CGIAR IDOs provides
realistic information about what kinds
of evidence individual projects can
provide, compared to more aggregate
approaches to indicators and evidence.
A glossary of terms used in this
learning note can be found in CCAFS
Theory of Change facilitation guide as
well as a set of questions that a CRP
can use to develop its own impact
pathway based ML&E system.12
CCSL Learning Brief No. 9 •• September 2014
Lessons in theory of change: monitoring, learning and evaluating Knowledge to Action
Notes
1. See presentation Tackling
Innovation in Climate Change
Research for more information:
http://ow.ly/B4YlD
2. The CCAFS Scenarios Program
focuses on the use of combined
climate/socio-economic scenarios
to guide policies and investments
at regional and national levels
in the CCAFS regions.n The
CCAFS regional scenarios are
developed from diverse stakeholder
knowledge and quantified through
agricultural economic and land-use
models.
3. The large groups of outputs,
products and deliverables (e.g. new
knowledge from research together
with new capacity in a specific
group of stakeholders to use that
knowledge and enact CSA policy)
that are necessary for achieving an
outcome.
4. The large groups of activities that
are necessary to create a major
output group.
5. A quantitative or qualitative
variable that represents
an approximation of the
characteristics, phenomenon or
change of interest (e.g. efficiency,
quality or outcome). Indicators
can be used to monitor research
or to help assess for instance
organizational or research
performance.
6. A metric that provides evidence
that the different components
of the impact pathway are
being implemented, leading
to production of MOGs critical
to achievement of the planned
outcome.
7. A metric that provides evidence
that sustainable change in nextuser behavior has occurred that
is consistent with the planned
outcome.
8. A metric that provides evidence
that a sustainable development
impact has occurred that is
consistent with the planned IDO.
9. Increased and stable access to
food commodities by rural and
urban poor (“Food security”);
Increased control by women
and other marginalized groups
of assets, inputs, decisionmaking and benefits (“Gender
and social differentiation”);
Increased capacity in low-income
communities to adapt to climate
variability, shocks and longer term
changes (“Adaptive capacity”);
Additional policies and institutions
supporting sustainable, resilient
and equitable agricultural and
natural resources management
developed and adopted by
agricultural, conservation and
development organizations,
national governments and
international bodies (“Policies
and institutions”).
Mario Herrero, Amanda Palazzo,
Ariella E.S. Helfgott, Angela
Wilkinsong, Petr Havlik, Daniel
Mason-D’Croz, Christine Jost.
2014. Challenges to scenarioguided adaptive action on
food security under climate
change. Global Environmental
Change. DOI: 10.1016/j.
gloenvcha.2014.03.001
12. Access the Theory of Change
facilitation guide here:
https://cgspace.cgiar.org/
handle/10568/41674
10. Carey C. 2014. The CCAFS
Regional Scenarios Programme:
External Evaluation Report on
Progress Towards Programme
Outcomes. CGIAR Research
Program on Climate Change,
Agriculture and Food Security
(CCAFS). Copenhagen,
Denmark. http://hdl.handle.
net/10568/34994.
11. Joost M. Vervoort, Philip K.
Thornton, Patti Kristjanson,
Wiebke Förch, Polly J. Ericksen,
Kasper Kok, John S.I. Ingram,
Acknowledgements
We aim to practice what we preach. This briefing series is the product of an on-going social learning process — the Climate Change and
Social Learning initiative (CCSL) — between the CCAFS team and its partners, in which knowledge has been co-constructed through many
different channels, including workshops, the CCSL ‘Sandbox’, and social media. Many thanks to everyone who has participated in this process
so far and to those who continue to do so.
CCSL Partners:
The CARIAA program is jointly funded by the International Development Research
Centre, Canada and the UK Department for International Development
Page 6