Preparation Guideline for
Outcome Evaluation Report
March 2015
Prepared by Cap-Net
1. Introduction of Outcome Evaluation
Cap- Net is a result- based management organization which
needs constant feedback, learning and improving. Existing and
future plans have to be regularly improved and developed
based on the lessons learned from monitoring and evaluation.
To ensure Cap- Net’s goal and objectives can be achieved,
evaluation plays a vital role in measuring Cap-Net’s
contributions to the change of water resources management.
2. Purpose of This Guideline
Relevance, Efficiency,
Effectiveness, Impact,
Sustainability, and
External Utility are
Cap-Net’s Monitoring,
Evaluation and
Learning Principles.
(Cap-Net MELP,
2009)
This guideline provides guidance for Cap-Net’s affiliated
network to prepare their annual outcome evaluation report of
capacity development activities. For Cap-Net’s monitoring and
evaluation framework in detail, please refer to Cap-Net’s
Monitoring, Evaluation, and Learning Plan (MELP).
This guideline highlights the definitions of terminologies of
outcome evaluation, overview of the outcome evaluation
process, tools for outcome evaluation, guideline for result
categorization, and overview of the contents of outcome
evaluation report. A reporting template has been prepared and
it serves as a reference for network’s outcome evaluation
report. Networks are allowed to modify the content of the
report (e.g.: change the order of the section, add more details),
but networks are encouraged at least submit the information
that stated in the template.
1
3. Overview of Cap-Net’s Monitoring, Evaluation and Reporting Framework
Cap-Net Annual reports, Phase completion reports
Change in practices in the ground through knowledge
transmission
Output
number of people trained in each thematic targets, number of trainings,
countries achieved, individuals trained in each country
overview of thematic areas against countries, type of professions and
institutions reached etc.
Outcome
Percentage of response - distribution in country and institution types,
Individuals work in water and related sectors are knowledgebale on
IWRM, application of knowledge in sustainable water management,
knowledge sharing and actions by the participants and their institutions
actions towards changes in policies and practices (new projects,
drafting policiy briefs, applying the conservation rules and regulations,
acedemic curricula development, trainings conducted etc.)
Impact
changes in practices, IWRM implementation, policies for sustainable
water use, Resillience plans in place for climate change, new projects in
place for address the water pollution issue, innovative practices for
addressing corruptions, enforcement of regulations etc.
Technical reports with
participants list, course
evaluation reports
Annual reports of
networks
Meeting minutes and
newsletters
Course evaluation
reports
Outcome evaluation
reports
Network self
assessments in Annual
reports of the networks
Case studies
Peer reviews
Internal/ External Review reports/ Project Evaluation
2
4. Definition of Terminology
Capacity Development Activity
CDA
Water Resource Management
WRM
Terminology Definition
Immediate
Immediate outcome is the most immediate things we are trying to change.
Hence, for Cap-Net’s CDAs, the most immediate change should be
Outcome
participants’ knowledge or skills on WRM. For example, immediate
outcome of IWRM Training Course should be improved knowledge of the
participants on IWRM subject.
Mid-term
outcome
Mid-term outcome is the change happens on the person who participated in
Cap-Net’s CDA, for instance, mid-term outcome is when a person use
knowledge gained through Cap-Net’s CDAs in his or her day-to-day work
and cause to changes. Evaluating mid-term outcome is to prove that the
knowledge taught from Cap-Net’s CDA is useful and applicable, and also to
prove the competency of a person in WRM has already been improved
after he or she attended the Cap-Net’s CDA.
Long-term
outcome
Long-term outcome is the change happens within the institution that the
participant is working for OR changes happen because of shared
knowledge or interventions by the participant using the knowledge gained
through the training. Hence, expected long-term outcome of Cap-Net’s
CDAs aims to improve the efficiency and effectiveness of an institution in
solving water issues through the participants of the CDAs.
Example: Mr. X runs a company that provides water loss reduction service to a city, he
improve the operational efficiency of his team in solving pipe leakage issue by
introducing new water loss policy. And such policy is developed base on the
knowledge that he gained from a water loss management training which was
organized by Cap-Net.
Impact
Mid-term outcome refers to the change of a person, long-term outcome
refers to change of a water related institution, and thus impact should refer
to the change of water sector. Impact of Cap-Net’s CDAs refer to the
improvement of WRM practices, improvement of water and sanitation
service quality, and improved application of WRM instruments in
addressing climate change impacts. By using the example in long-term
outcome, impact of the CDAs should be the improved water loss
management in the city. Please be note that the impact is usually long-term
and it could be indirectly caused by Cap-Net’s CDA as big changes always
involve multiple factors and actors. Thus, the networks are required to
focus on evaluating the contribution not the attribution.
3
Report Preparation Guideline
5. Evaluating Outcomes and Impacts of
Knowledge and Capacity Development Activities by Networks
Overview of the Process
Evaluation
Tools
Timeline
Immediate Outcome
Tool 2 (course evaluation
form) + additional tool
Immediately after training
Mid-term Outcome
Tool 3 -Form A (outcome
evaluation question)
Six month after training
Long-term Outcome and
Impact
Tool 3 -Form A + follow up
interview
Could be six month up to
one year after training
Outcomes and impacts of
case studies/ training
materials/ meetings
Tool 3 - Form B, C, D
Could be 6-12 months or
more
Outcome Evaluation
Report Preparation
Preparation guideline and
excel sheet of result
categorization
Submit report in between
July and September
Note:
1. Result of immediate outcome evaluation is not covered in outcome evaluation
report, but it is included in the course/ activity report, which should be submitted
after every CDA. Generally, Tool 2 serves as a basic tool to evaluate the
immediate outcome of CDA. As the content of every course is different, networks
are recommended to develop an additional tool to evaluate the immediate
outcome of the CDA.
2. Only one outcome evaluation report should be submitted, and it should report the
mid-term outcomes, long-term outcomes, and impacts of network’s CDAs in the
reporting period, which are usually the CDAs that networks organized in the
previous year. For example, if you are preparing outcome evaluation report in
2015, you should report the mid-term outcomes, long-term outcomes, and
impacts of the CDAs that your network organized in 2014. Tool 3 will be use in
evaluating the outcomes of CDA, and knowledge development and networking
activities.
4
Report Preparation Guideline
6. Overview of the Content of Outcome Evaluation Report
Title and cover page
-
Logo and name of network, title of the report, name of the person
who prepared the report, and report submission date
Report’s content
page
-
The content of the report should be presented in this section,
together with the page number
Introduction
-
Introduction of network
-
Aim and objectives of outcome evaluation report
-
Explain how did you evaluate outcome in details
-
Describe the tools, instruments, and indicators that you used for
evaluating outcomes and impacts of your capacity building activity
-
Attach the questionnaire that you used for outcome evaluation as
per the Tool 3 (you can choose to attach it in appendix section)
-
State which course have been evaluated
-
Present and discuss respondents’ profile (e.g.: the response rate,
respondents’ gender, respondents’ institution etc.)
-
Present and discuss the results of outcome and impact evaluation
-
Please provide necessary figures in this section
-
This sub section (as a part of results and findings) reports about
any follow-up of the case studies produced in the previous year/s,
and important meetings conducted in the previous year
-
Tool 3 Form B, C, D of the Cap-Net MELP provides the tables for
recording follow up, but network are advised to come up with
innovative methods and reporting in this section.
-
Present the lesson learned from the outcomes of capacity building
activity, as well as the lesson learned from monitoring and
evaluation activities.
-
Provide recommendations with specific findings from previous
sections
Conclusion
-
Concluding remark
Appendix 1
-
Appendix 2
-
Attach the questionnaire that you used for outcome and impact
evaluation
Provide the tables that you used for categorizing the answers
Appendix 3
-
Quotes from the respondents (optional)
Reference
-
Please include the references that you used for preparing report
Tools and
Methodologies of
Outcome Evaluation
Results and
Findings
Outcomes of Case
studies/ Training
Materials/ meetings
Lesson Learned and
Recommendation
5
Report Preparation Guideline
7. Revised Outcome Evaluation Questions (Tool 3 – Form A)
When should this be applied?
All participants should be followed up between 6 – 12 months after course completion to
assess the outcomes and impacts of the training.
How are the results to be used?
Compile necessary information on the activities of the network against indicators and
targets. Primary is used to prepare the evaluation report. Used to review and revise
work plans and priorities. Used for promoting network and increased visibility. Improve
management, relevance and efficiency of future activities.
The answer of question 1 and question 2 have to be reported by using the tables in a
excel sheet template which can be downloaded from Cap-Net’s website. The answer of
question 3 need to be categorized by the network and/or reported in short stories format.
--------------------------------------------------------------------------------------------------------------------Dear……………
You participated in the course “…………………..” held from “……………..” at
“……………”. Could you please take a few minutes to answer these 3 questions? This
will help us to improve our work.
Question 1
a) Have you used the knowledge from the course to improve your working
performance?
Yes [ ]
No [ ]
b) If YES, please explain how you have used the knowledge. If NO, please mention
the limitations.
c)
Aim of first question
To investigate whether the respondent has already apply the knowledge gained through
Cap-Net’s CDAs in his or her day-to-day work and bring about changes.
6
Report Preparation Guideline
Question 2
a) Have you shared or spread the knowledge gained from the course within or
beyond of your organization?
Yes [ ]
No [ ]
b) If YES, please explain how you have shared/spread and to whom you have
shared/spread the knowledge. If NO, please mention the limitations.
Aim of second question
To investigate the way that the participant shared the knowledge as well as to investigate how
far Cap-Net’s information has reached.
Question 3
Have you identified positive or any specific change in water resource management that
contributed by the knowledge that you gained from the course? What change was that?
Aim of third question
To obtain more information about how water resources management has already been changed
due to the knowledge taught by Cap-Net’s training programmes. This question is crucial to
provide more information to identify the potential long-term outcome or impact of Cap-Net’s
training. The answer of this question will be used to develop the short stories. Specifically, this
question provides more information to network to decide which participant they should follow up
to obtain more information for developing the short story.
--------------------------------------------------------------------------------------------------------------------Note to networks: Networks are allowed to modify the questions or add more questions.
But networks are encouraged at least keeping the keywords in the questions.
7
Report Preparation Guideline
8. Result categorization Guide
Below tables serves as a reference to networks for categorizing results. Besides, by
referring below tables, network should think about the expected answers for the
question 1 and question 2 of the Tool 3 - Form A. More tables are available in the excel
sheet (result categorization guide); networks are required to fill in the tables in the excel
sheet and paste the tables in the report, then Cap-Net can refer to them in case need of
more information. Networks are allowed to add more categories or section in the table
according to the answers they received.
Table 1: How the participants have used the knowledge (Answers for Question 1)
Category of
answer
Raising
awareness
on
water
management
Sub category
Number
of
respondent
- Improve knowledge of water management
- Improve
sensitivity/
perception
towards
water
management
- Improve working motivation/ attitude
- Other (please specify)
Enhancing
- New technical skills have been developed
skills
- Improve skills in solving water issues
- Other (please specify)
Educational
- Teaching or lecturing
or academic - Conducting capacity building activity
purpose
- Conducting water campaign
- Conducting research
- Developing or improving educational or course material
- Other (please specify)
Formulating
- Propose or draft water policy / regulation
water policy/ - Improve or initiate water management strategy
regulation/
- Engage stakeholders in water management
strategy
- Other (please specify)
Supporting
- Design or plan water related project
water related - Manage water related project
project
- Implement water related project
- Other (please specify)
I haven’t used the knowledge from the course
Total
8
Percenta
ge
100 %
Report Preparation Guideline
Table 2: How the participants shared/ spread the knowledge (Answers for Question 2)
Category
Respondents’ Answer
Formal
conversation
-
Informal
conversation
Capacity
building
activity
Academic or
educational
activity
Document
sharing
Collaboratio
n
or
partnership
of
project/
programme
Stakeholder
involvement
in
water
management
Media
-
-
Number
of
Respondents
Percentage
Formal meeting
Debate
Group discussion
Other (please specify)
Daily/ casual conversation
Informal group/ personal discussion
Other (please specify)
Training of Trainer programme
Course training or capacity building activity
Engaged as facilitator of capacity building activity
Other (please specify)
Academic conference
Lecturing or teaching
Research activity
Water campaign
Other (please specify)
Distribution or share course material/ book/ other
document from the course
Sharing of course report
Other (please specify)
Knowledge spreading
was occurred through
partnership or collaboration with other institute or
organization of a water related project/ programme
Other (please specify)
- Knowledge spreading
was occurred through
community water management programme or
stakeholder involvement in water management
- Other (please specify)
- Mass media (TV, radio, newspaper, etc)
- Social media (Internet, Facebook, twitter, email, blog,
etc.)
- Other (please specify)
I haven’t shared the knowledge from the course
Total
9
100 %
Report Preparation Guideline
9. Guideline for Developing Short Stories of the Impact of Capacity
Development Activities
The purpose of impact and long-term outcome evaluation is to provide evidence to
prove that CDA is a crucial element in enhancing sustainable water management. Short
story is used as a tool to present the long-term outcome and impact of Cap-Net’s CDAs.
Generally, the answers from tool 3 provide basic information to identify which short
stories should be reported, especially answer of question three. Network is suggested to
obtain more information through an interview by choosing respondent according to the
answers from tool 3, particularly answer of question number three.
Networks should report the short stories by including below elements:
 Who has taken action and what course did he/she attend?
 What has he/ she done to move the country towards WRM and service provision?
 How has the mode of WRM and service delivery changed/ improved?
Example
Short story from Burundi:
The Climate Change Adaptation: Strategies and Tools training workshop was
conducted by the Nile Basin Capacity Building Network with the objective of
strengthening the capacities of the Nile basin water professionals in dealing with climate
change issues. Dr. Nyandwi Venant is an assistant lecturer at the University of Burundi
and a member of Friends of the Earth (Les Amis de la Terre Burundi – ATEBU),
Bujumbura, Burundi. He says that ‘After the course I’ve gained new ideas and I use
them in some work like consultancies and meetings as much as possible.’ He has
discussed some climate issues with members of Nile Discourse Forum (NDF) especially
on the need to include climate change issues in any project. The NDF is involved in
building some national policies as a team. Delegates from the NDF were active where
Burundians were elaborating a water policy and poverty reduction strategy framework.
Dr. Venant has trained 18 journalists in order to build their capacity on climate change.
This training was held on 12th September, 2011 in Bujumbura Press House. His
colleague Mr Apollinaire Niyirora conducted a session on environmental reporting
techniques for journalists. After the training course, the facilitators noticed some impact
of the training course. They were invited by some journalists to answer questions
related to the environment after the workshop and nowadays. Media and papers in
Burundi talk more about environmental issues now.
(Source: Gunawardana et al., 2013)
10
Report Preparation Guideline
10. References
1.
2.
3.
4.
5.
6.
7.
8.
Cap-Net Monitoring, Evaluation and Learning Plan (MELP) (2009)
Cap-Net MELP Guidance Sheet (2014)
Cap-Net Phase III Joint Donor Review Report (2014)
Cap-Net Annual Report 2013
Cap-Net Phase III Strategic Plan
Cap-Net Phase IV Strategic Plan
Cap-Net Network Attachment MELP Reports (2011- 2013)
Gunawardana, I., Leendertse, K., Handoko, W. (2013). Monitoring outcomes and
impacts of capacity development in the water sector: a Cap-Net UNDP experience.
Water Policy 15.
9. UNDP Handbook on Monitoringand Evaluating for Results (2002)
10. UNDP Handbook on Planning, Monitoring and Evaluating for Development Results
(2009)
11