Day 1:
Introduction
Course Facilitators
4
Expectations of the Workshop
5
Agenda for Day 1
6
Workshop Process
• Start & end on time
• Give everyone a chance to speak & ask
questions
• « Issues and Concerns»
• Questions & comments at any time
• Questions et commentaires en français
7
Introductory Exercise
- Each person, share an achievement you`re
very proud of & the role you played
- Look for patterns in the stories
- Be prepared to share the patterns in plenary
Conceptual
Overview
Evaluation Challenges
1. Establishing cause & effect in an open
system
2. Timing
3. Encouraging iterative learning
4. Clarifying values
10
Challenge 1: Establishing Cause/Effect
in an Open System
• Multiple actors & factors contribute
• Causality is mutual, therefore usually
attribution is not possible
• Unintended results often ignored
11
Shifting Influence Over Time
Influence
High
Low
Time
Beneficiaries/Indigenous Actors
Program
12
Challenge 2: Timing
• Tracking incremental, cumulative
change
• Not assessing results as though final
13
Are we
having an
impact yet ?
Challenge 3: Encouraging Iterative
Learning
•
•
•
•
Emphasis on development results
Easy to measure vs. important to know
Keeping results within their context
Making time for reflection
15
Accountability & Learning:
A Balancing Act
16
Accountability & Learning:
A Balancing Act
17
Challenge 4: Clarifying Values
• Assessment is not value neutral
• Making explicit the implicit
• Building the capacity to engage
18
Challenges and Outcome Mapping
Challenges
OM Tools
1. Cause & effect in
open system
1. Vision & Outcome
Challenges
2. Timing
2. Outcome challenge &
Progress Markers
3. Org. Practices &
Monitoring Journals
4. Mission, Boundary
Partners, Progress
Markers, Org. Practices
19
3. Iterative Learning
4. Clarifying Values
What is Outcome Mapping?
• an integrated PM&E tool
• an approach that focuses on changes in the
behaviour, relationships,or actions of partners
(as outcomes)
• a methodology that characterizes and assesses
the program’s contributions to the achievement
of outcomes
• an approach for designing in relation to the
broader development context but assessing
20
within your sphere of influence
At What Level?
• Course is focusing at the program level
• Definition of Program: A group of related
projects and activities with a specific set of
resources (human, financial, capital) directed
to the achievement of a set of goals within a
specified period of time
• Can be used by projects, organizations, and
communities too
22
Key Concepts of Outcome Mapping
• Sphere of influence
• Outcomes as behavioural change
23
Sphere of Influence
Planning &
Design
Sphere of
Influence
Assessing
24
The Focus of Outcome Mapping
Inputs
Activities
Outputs
Outcomes
Impacts
Behavioural Changes 25
What is an « Outcome » in OM?
• Change(s) in the behaviour,
relationships, activities, and/or actions
of an individual, group, or organization
that the program was helpful in bringing
about.
• These changes are aimed at
encouraging human and ecological wellbeing.
26
Changes Sought
Changes in
State
Ecosystem
Human
System
Changes in
Behaviour
27
Why Behavioural Change?
• To stress that development is done by,
and for, people
• To illustrate that although a program can
influence the achievement of outcomes,
it cannot control them because ultimate
responsibility rests with the people
affected
28
How Can Outcome Mapping Be Used?
For a program to tell its performance story in
outcome terms by:
• articulating its goals and designing its activities
• designing a monitoring system for assessing
internal performance and outcomes of partners
• setting a use-oriented evaluation plan
29
Why Use Outcome Mapping?
Focussing on changes in partners’ behaviour,
relationships, or actions allows a program to:
• measure results within its sphere of influence
• obtain feedback about its efforts in order to
improve its performance
• take credit for its contributions to the
achievement of outcomes
30
• show progress towards outcomes
Stage 1:
Intentional
Design
« I have a dream! »
Martin Luther King
Step 1 : Vision Statement
• Reflects the broad human, social, &
environmental betterment in which the
program is engaged and to which it is
contributing
35
Example Vision Statement
Local authorities, communities, and international organizations in
developing countries in Africa recognize the value of HIV/AIDS
intervention as an integral part of social & economic development.
Municipal, regional, and national governments actively support
HIV/AIDS prevention activities by formulating and implementing
effective public health policies. Using research findings, they
have developed a comprehensive public health strategy to slow
down the infection rate. Formerly marginalized groups (e.g.
women and youth) are organized into advocacy groups that can
effectively formulate their needs to policy makers. All groups have
access to reliable and relevant technical information about
HIV/AIDS prevention and are able to make informed choices. In
essence, there are healthier, happier, and wealthier communities.
Facilitation Questions
“Imagine that in 3-5 years the program has
been extremely successful. What changes
will you have helped bring about? What are
your partners doing differently? What have
they achieved? In essence, what would
total success look like?”
37
VISION STATEMENT
Mission
The Mission is that “bite” of
the Vision Statement on
which the program is going
to focus
Step 2: Mission Statement
Describes how the program intends to:
• Operationalize its role in support of the
vision by determining the areas in which
it will work
• Support the achievement of outcomes
by its boundary partners
40
Facilitation Questions
“How can the program best
support or contribute to the
achievement of the vision?
What areas do you need to
work in? What do you need to
do? ”
41
Example Mission Statement
In support of this vision and on behalf of its donors, the program
will work in the areas of research, dissemination, capacity
building, & coordination. It will contribute to the production,
synthesis, & dissemination of research data, position papers, &
other information that will sensitize local & international actors to
HIV/AIDS prevention. The program will seek to expand the range
of disciplines involved in HIV/AIDS research. It will enhance
HIV/AIDS research capacity in order to produce credible
information for local, national, & international policy-making &
program development. It will promote an interest in HIV/AIDS
research among new researchers by providing research
fellowships, mentorship, & training opportunities. The program will
contribute to the development of linkages between Northern &
Southern researchers & encourage partnerships between
research organizations, advocates, & decision makers. It will
increase its visibility & credibility among the donor community &
will convince them of the utility of supporting HIV/AIDS prevention.
Developing a Vision or Mission
•
•
•
•
•
•
•
Warm-up – general conversation
Individual write up
Discuss & Post
Facilitator and/or participant draft
Present & discuss
Purge the jargon
Verify with partners (if required) &
Finalize
43
Developing a Vision with the
Community in Nagaland
44
Suggestions and Tips!
If time is a limiting factor you can carry
out an email survey in advance to
create a draft vision & mission before
the workshop starts. Use the draft
version as a starting point for discussion
in order to save time.
45
Exercise 1:
Correct Errors
in Vision & Mission
Statements
Step 3: Boundary Partners
Those individuals, groups, & organizations
with whom the program interacts directly to
effect change & with whom the program can
anticipate some opportunities for influence.
48
Program`s Sphere of Influence
Program
= Program`s Boundary Partners
49
Boundary Partners (have boundary
partners)
Program
Program’s Boundary
Partners
Boundary Partners’
50
Boundary Partners
Typology of IDRC Boundary Partners
IDRC
Program
Applied Research Community
e.g. researchers, institutions,
NGOs, networks
System Gatekeepers / Dev`t
Agents e.g. business, policy
makers, NGOs, gov`t
Beneficiaries
e.g. communities, farmers,
51
women, youth
Facilitation Questions
« In which individuals, groups, or
organizations is your program trying to
encourage change so that they can
contribute to the vision? With whom will
you work directly? »
52
Suggestions and Tips!
• A program can choose its boundary partners
because it wants to influence them or
because they will influence others
• Example: a rural development NGO chooses
to work directly with farmer organizations
who in turn, lobby the Minister of Agriculture
to effect change.
53
Exercise 2:
Identify the Program`s
Boundary Partners
Working on Your Own Case
Guiding Questions
In your context:
• How would the ideas of Outcome Mapping
resonate?
• Is a focus on behavioural change appropriate?
• How would you foster the imagination of your
group to come up with a vision?
• Are you clear about your contributions to the
vision? (i.e., your mission)
• Who are your boundary partners?
56
• How could you include them in the process?
Conclusion & Evaluation of
Day 1
What’s Working? What’s Not?
Day 2:
Review of Day 1 & Feedback
from Evaluation
Agenda for Day 2
60
Step 4: Outcome Challenge
Description of:
• The changed behaviours (relationships,
activities, and/or actions) of a boundary
partner
• How they could contribute ideally to the
vision
62
Example Outcome Challenge Statement
The program intends to see researchers in developing countries
that have gained the expertise required to carry out HIV/AIDS
research in a multidisciplinary fashion using quantitative,
qualitative, and/or participatory methodologies. They are
conducting research on the legislative, economic, social, and
health aspects of HIV/AIDS with special emphasis on vulnerable
groups including youth and women. They are providing credible
evidence that allows research findings to be translated into
effective HIV/AIDS policies and programs. They are skillfully
disseminating research results in the appropriate format for the
audience (e.g., to the scientific community, the general public,
NGOs, government officials). They are successfully obtaining
funding from multiple sources to conduct HIV/AIDS research.
More experienced researchers are mentoring those new to the
field and HIV/AIDS researchers globally are engaged in
productive communications using various means especially
electronic networks.
« Those Who Dream Make a
Difference! »
Ms. Kalpana Pant,
Chaitanya
Phrasing of an Outcome Challenge
“The program intends to see [boundary
partner] who [description of behaviours
in the active present tense]”
65
Facilitation Questions
“Ideally, in order to contribute to the
vision, how would the boundary partner
be behaving?”
“What would your boundary partner be
doing to contribute maximally to the
vision?”
66
Suggestions and Tips!
• Similarity of behaviours sought among
boundary partners (policy makers and
community members) in international model
forests was unexpected by program staff
67
Outcome Challenge Checklist
Outcome Challenge Statements:
Describe behavioural changes
About a single boundary partner
Set out the ideal behavioural changes
Describe the boundary partner`s
contributions to the vision
68
Exercise 3:
Identify Errors in Outcome
Challenge Statements
Program Framework
Vision:
Mission:
Boundary
Partner:
Outcome Challenge:
Boundary
Partner:
Outcome Challenge:
Boundary
Partner:
Boundary
Partner:
Outcome Challenge:
Outcome Challenge:
70
Overview Exercise:
Put Together a
Program Framework
Step 5: Progress Markers
• A graduated set of statements
describing a progression of changed
behaviours in the boundary partner
• Changes in actions, activities, &
relationships leading up to the ideal
outcome challenge statement
73
Levels of Progress Markers
The program sets out what it would:
•
•
•
Expect to see its boundary partners doing?
Like to see its boundary partners doing?
Love to see its boundary partners doing?
74
How Many Progress Markers?
Limit to a manageable amount of data to
be collected:
4 “expect to see”
8 “like to see”
3 “love to see”
75
Why Graduated Progress Markers?
• Articulate the complexity of the change
process
• Permit on-going assessment of partners’
progress (including unintended results)
• Encourages the program to think about how it
can intentionally contribute to the most
profound transformation possible
• Make mid-course corrections & improvement
easier
76
Example Progress Markers
The program Expects to See local communities:
1. Participating in regular model forest (MF)
partnership meetings
2. Establishing a structure for cooperation in the
partnership
3. Acquiring new skills for involvement in the MF
4. Contributing resources necessary to get the
MF operational (minimum)
77
Example Progress Markers
The program would Like to See local communities:
5. Articulating a vision for the MF that is locally relevant
6. Promoting their involvement with the MF nationally
7. Expanding the partnership
8. Calling upon external experts when necessary
9. Requesting new opportunities for training
10. Disseminating concrete examples of benefits arising
from MF activities
11. Seeking out new partners for the MF
12. Obtaining funding from different national sources
78
Example Progress Markers
The program would Love to See
local communities:
13. Helping other MFs establish themselves
14. Sharing lessons-learned internationally
15. Influencing national policy debates &
formulation on resource use and management
79
Facilitation Questions
« How can the program know the
boundary partner is moving toward the
outcome? What would they be doing? »
« What milestones would be reached as
the boundary partner moves towards
their intended role in contributing to the
vision? »
80
Suggestions and Tips!
• If a program has multiple boundary partners
but the primary result to be achieved is
changing their relationships with one
another then it is possible to create a single
outcome challenge & set of progress
markers for the whole group
81
Progress Marker Checklist
Each Progress Marker:
 Describes a changed behaviour by the boundary
partner
 Can be monitored & observed
As a set, Progress Markers:
 Are graduated from easier to more difficult to achieve
changes in behaviour
 Describe the change process of a single boundary
partner
82
Exercise 4:
Identify Correct Progress
Markers & Categorize as
Expect, Like, or
Love to See
Step 6: Strategy Map
• Outlines the program`s approach in working
with the boundary partner
• Indicates the relative influence the program
is likely to have on boundary partner
• Helps pinpoint strategic gaps in the approach
or if the program is overextended
• Suggests the type of evaluation method
appropriate to track and assess the
performance of the program
85
6 Types of Strategies
Strategy
Aimed at
the
Boundary
Parnter
Causal
Persuasive
Supportive
I-1
I-2
I-3
• Direct Output
• Arouse New
Skills/ Thinking
•Supporter who
guides change
over time
E-2
E-3
•Modify the
information system
•Create /
Strengthen a
Peer Network
E-1
Aimed at the
•Alter physical or
Boundary
regulatory
Partner`s
environment
Environment
86
6 Types of Strategies
87
Facilitation Questions
I-1
I-2
I-3
What will be
done to produce
an “immediate”
output?
What will be
done to build
capacity?
How will
sustained
support,
guidance, or
mentoring be
provided?
E-1
E-2
What will be
done to change
the physical or
policy
environment?
How will you use
the media or
publications?
E-3
What networks/
relationships will
be established
or utilized?
88
Suggestions and Tips!
• Facilitator asks the group questions
about their strategies
• While they are discussing them (s)he
quietly plots them on a strategy map
• After being shown the strategy map, the
group is asked to analyze it for gaps,
overextension, and the influence that
can be expected
89
Exercise 5:
Plot Activities on
a Strategy Map
« The oft repeated and echoing question in
Outcome Mapping « what or who needs to
change? » raised us from being providers of
development, achieving outputs, to actually
believing we were agents of change. »
Raj Verma, NEPED
Step 7: Organizational Practices
The things that you do as an organization to:
• foster creativity & innovation
• seek the best ways to assist your partners
• maintain your niche
93
Why Organizational Practices?
• Important to how the program is going
to function to effectively fulfill its mission
• Supporting change in its boundary
partners requires that the program be
able to change and adapt as well
94
8 Organizational Practices
1. Prospecting for new ideas, opportunities,
& resources
2. Seeking feedback from key informants
3. Obtaining the support of your next highest
power
4. Assessing & (re)designing products,
services, systems, and procedures
95
8 Organizational Practices
5. Checking up on those already served to add
value
6. Sharing your best wisdom with the world
7. Experimenting to remain innovative
8. Engaging in organizational reflection
96
Facilitation Question
“What do you have to be doing in order to
remain effective?”
97
Suggestions and Tips!
• This can be a difficult set of concepts to
get across; frame it in terms of changes
in behaviour internal to the organization
• Have the group think of the program as
an organization in its own right
98
Exercise 6:
Identify the
Organizational Practices
Exercise 6a:
Draw the 7 steps of
Outcome Mapping
Overview of Stage 1
101
Organizational Practices
Working on Your Own Case
Guiding Questions
In your own context:
• How would the idea of planning beyond what
can be achieved be accepted? (i.e. outcome
challenge and progress markers)
• What kinds of strategies do you use – causal,
persuasive, supportive?
• How will you get your group to talk about your
own organizational practices?
103
Evaluation of Day 2
Feedback from
« critical friends »
Day 3:
Review of Day 2 & Feedback
from Evaluation
Agenda for Day 3
107
Stage 2:
Outcome &
Performance
Monitoring
Outcome Mapping Monitoring
• Systematic collection of data
• A regular learning & improvement cycle
• Encourages the program to challenge
itself
• Collects & organizes, does not analyze
data
109
« Being busy creates a mindset that is not
conducive to innovation and creativity.
Without interaction, there is no innovation.
Time to discuss, reflect, and generate new
ideas is the ransom that outcome mapping
demands for innovation. »
Thierry Barreto-Fernandez
West Africa Rural Foundation, Senegal
Step 8: Monitoring Priorities
• Tempting to gather information on a whole range of
issues but this is not practical
• Base priorities on intended use of intended users:
–
–
–
–
–
Use(s) for findings and from process
Improve performance through learning
Help meet accountability/reporting requirements
Supply information for planned evaluation
Inform publicity documents, communication activities, or
case-study materials
112
Monitoring Plan
Monitoring Who Purpose When Who How
Priority
will
is it
will often
use
needed collect
Strategies
Proposed
Tool
Strategy
Journal
Org’l
Practices
Performance
Journal
Boundary
Partner
Outcomes
Outcome
Journal
113
Boundary
Partner
Program
Boundary
Partner
Outcome Challenge
& Progress Markers
Program
Boundary
Partner
Program
Strategies &
Activities
Boundary
Partner
Program
Organizational
Practices
Boundary
Partner
Outcome
Journal
Program
Strategy
Journal
Performance
Journal
Organizing Monitoring Meetings
• Regular face-to-face meetings to record
observed changes
• Filling out electronic data sheets on an
on-going basis, then aggregating
changes for entire program
• Interviews and/or focus groups with
boundary partners
119
Step 9: Outcome Journal
• Uses progress markers to chart
boundary partner`s progress
• Sets starting point or benchmark
• Captures details on changes in the
boundary partner, contributing factors &
actors, and sources of evidence
121
Outcome Journal
Work Dating From/To:
Contributors to Monitoring Update:
Outcome Challenge:
Low =
Medium =
High =
Expect to see:
Who?
LMH
1
2
3
122
Set Values for Low, Medium & High
• Can capture either quantity or quality of
change:
– Quantity (number of boundary partners)
– Quality (depth of change)
123
Baseline
• multiple individuals, groups or
organizations in each boundary partner set
• identify each boundary partner’s starting
point in terms of progress markers
• boundary partners may be at different
stages of development
– individual change process can be captured
over time
124
Step 10: Strategy Journal
• Captures data on the strategies being
employed to encourage change in the
boundary partner
• Helps the program gauge its contributions to
the achievement of outcomes and modify its
actions accordingly
• Feeds into work planning & (modifications)
or new activities
126
Strategy Journal
Working Dating From/To:
Contributors to Monitoring Update:
Strategy to be Monitored:
Effectiveness?
(How did it help
the boundary
partner?)
Outputs
127
Step 11: Performance Journal
• Records data on how the program is
operating as an organization to fulfill its
mission
• Includes information on the program’s
practices
• Feeds into work planning
128
Performance Journal
Working Dating From/To:
Contributors to Monitoring Update:
Practice 1: Prospecting for New Ideas, Opportunities, & Resources
Example or Indicators:
Sources of Evidence:
Lessons:
Practice 2: Seeking Feedback from Key Informants
Example or Indicators:
Sources of Evidence:
Lessons:
129
Exercise 7
Identify your monitoring
priorities (e.g. work on your
real case)
Stage 3:
Evaluation
Planning
« Evaluative thinking is a
learned behaviour that you
have to practice and improve
at on an on-going basis! »
Michael Quinn Patton
Step 12: Evaluation Plan
• A short description of the main planning
elements of an evaluation study
• For Whom? What? Where? How? By
Whom? When? How Much?
134
Elements of an Evaluation Plan
Evaluation Issue:
Date Cost
Info.
Evaluation Who Will
Who Will Use Questions
Sources Methods Conduct & (Start &
to be
Evaluation,
Manage the Finish)
How, & When? Answered
Evaluation?
135
Setting Evaluation Priorities
Criteria with which to select include:
• Learning needs
• Reporting requirements
• Partners’ information needs
136
Guiding Principles for Evaluation
• Balancing learning & accountability
• Action oriented
• Each study is unique
• Process not only product
• Build capacity at all levels through
participation
137
« Not everything that counts
can be counted. And not
everything that can be
counted, counts. »
Albert Einstein
An Evaluation of IMFNS
• Assessment of 1st phase
• Wanted to measure human, community, &
partnership elements of model forests
• Client acceptance of methodology essential
• OM provided a framework for the study
139
Evaluation Questions
1. Who has IMFNS reached?
2. How have the behaviours of
IMFNS`boundary partners changed?
3. What has been the nature of IMFNS`
interactions with its partners?
140
Process of the IMFNS Evaluation
• Recreated planning framework in 1 day with
IMFNS staff
• Developed progress markers for communities
& goverment officials across the 5 sites
• Conducted a document review & key
informant interviews
• Comparison of boundary partners across MF
sites for analysis
141
Comparing Boundary Partners
Progress
Marker
Chile Chiloe
Russia Gassinsky
Mexico Chihuahua
Mexico Calakmul
Mexico Monarch
Expect to see local communities:
1
!
!
!
!
!
2
!
!
!
!
!
3
!
!
!
!
!
4
!
!
!
!
!
Like to see local communities:
5
!
!
!
!
!
6
!
!
!
!
!
7
!
!
!
!
!
8
!
9
!
!
!
!
10
Potential
!
11
12
Love to see local communities:
13
14
15
!
!
!
142
Lessons from the IMFNS Study
• Difficult to link what the Secretariat did with
outcomes achieved without using cause-effect
• Boundary partners should have been involved
in the development of the framework & analysis
of data
• Should have included more analysis of
organizational practices
• Did not study the relevance of boundary
partners within their environment
143
A Self-Assessment of NEPED
• Preparation for external evaluation
• Process
– Fred knew people and context prior to workshop
– Retrospectively built the framework (took 1.5 days
including HML values)
– « Ah-ha » moment happened at end for some
participants
– Organizing data collection (preliminary
identification, collection and verification, analysis
and writing report took another 1.5 days)
144
Lessons from the NEPED Study
• Group dynamics (worked together well,willing
to trust Fred, cooperative)
• Data (they had the data but it wasn`t
organized)
• Preparation for external review (intended use)
• Relevance of work with state government
(beyond trees, number of women involved,
number of villages worked with)
• Could have had greater external participation
(Gov`t of Nagaland, etc.)
145
Conclusion
Conclusion
• A system to think holistically & strategically
about how to contribute maximally to
development
• A method to articulate contributions to the
development process through boundary
partners
• Outcomes are changes in the behaviour,
actions, and/or relationships of boundary
partners
• An approach that encourages on-going
learning, challenges, & improvement through
148
M&E activities
Back to Work…
How are you going to put your
training in Outcome Mapping
to use?
Were Your Expectations Met?
150
Evaluation of
the Workshop
In terms of the content and
process, what should we
keep, add, change?
Thank You!
http://www.idrc.ca/evaluation