1 - Monitoring and Evaluation
Jela Tvrdonova
2012
Outline of the presentation
 Strategy/program in development context
 Strategy/program logic
 Monitoring and evaluation in development context
 Evaluation processes
 Monitoring and evaliation framework
The strategy/program in the development
context
 The development of countries, regions are often governed
via strategies and programs, which are time limited
 In strategies/programs it is not possible to solve all
problems/needs of particular area, country or region in the
same time.
 Problems/needs are targeted by strategy/program
interventions
 It is necessary to decide which areas of life are the priority to
be targeted via interventions of the strategy/program life
time
The strategy/program in the development
context
Intervention contains:
 priority areas,
 overall and specific objectives,
 measures and actions to be taken
Intervention reflects:
 SWOT
 Identified needs and problems
 Vision established by development stakeholders
Strategy/program framework and
intervention logic
Stakeholders
Territory
Strategic objective
Vision
Situation description
Int
er
ve
ntion
SWOT
Problem analysis and
priorities
Specific objectives
Measures, activities,
Financial plan
Interventions in the development
context
 Interventions are always functioning in certain contextual
environment, which „goes its own way“ as well.
 Example of the EU rural development programs
Figure. 1 Context of the program/strategy in rural development
Rural areas as
defined by
national
definition/all
stakeholders in
rural areas
The area of
intervention rooted in
the EU Strategic
guidelines EC reg.
1698/2005, eligible
beneficiaries, EU
objectives
The intervention area
rooted in National
strategic plan for RD
,selected eligible
beneficiaries – national
objectives
Other
interventions and
objectives
 Example - Context of the development program in the EU
Context of the strategy/program and its
effects
 In such a complex situation is sometimes complicated to
evaluate effects of the intervention.
 The overall situation is located in changing environment,
which can influence the intervention of the
program/strategy in many different ways.
Development program/strategy
Direct effects
on eligible
beneficiaries
In-direct
effects on
nonbeneficiarie
s
Other
programs
effects
In-direct effects
on eligible
beneficiaries
Program/strategy´s effects
Other Nonprogram
effects
Program/strategy logic
 Strategies/programs are governance tools
 They shall bring sustainable solutions to territory´s needs and challenges.
 Program/strategy effects on the territory depend to the high extent on the
quality of its intervention logic design.
 It shall prove the efficient use of public and private funds in favour of whished
change
 In the ideal case the development of the area shall go with the line of its goals
and objectives although still influenced by external factors
 Only well established strategoes have realeffects on the territory in whished
directions supported by sufficient public and private means
Change
Cont
ext
of
deve
lop
men
t– all
facto
rs
influ
enci
ng
deve
lop
men
t
Needs,
visions
Change observable in
reality during and after
intervention
Whished change in
priority areas before
intervention
Overall /strategic
goal
Impact
Specific objectives
within development
priorities
Result
Output
Operational
objectives of
measures
Measures
Inputs
Overall contextual baseline socio-economic and environmental situation before intervention
Logic of the program/strategy
Monitoring and evaluation in
development context
 It is necessary to distinguish among various processes – audit,
monitoring and evaluation
Audit and Monitoring
 Audit , control is focused on the correct administrative
and financial management of measures
 Monitoring is continuous and systematic observation of
financial and other inputs, implemented projects and
activities and intermediate outputs within the
implementation of particular measures of the local
development strategy. It also contains the program
results, although these are part of the evaluation as well
Evaluation
 Evaluation is focused on the assessment of impacts of the
program/strategy within certain development context, netted out
from deadweigth, leverage, substitution, diplacement and
multiplying effects
 Evaluation means also looking at the effectiveness, efficiency and
the relevance of particular measures and offers useful reflexion for
re-formulating or overall change of policies.
Evaluation
 In this process the quality of the evaluation substantially depends
on the methods choosen
 It also depends on the quality of available data and information
collected in the monitoring processes or also contextually, which
means that both processes must be interlinked and designed in
interaction straight from the beginning.
Evaluation processes
On going evaluation is helping us to collect data which
help us to carry the evaluation exercise in time periods –
ex-ante, mid term and ex post
Periodical evaluations
 Ex ante evaluation – independent study, which helps us
to understand if the projected intervention is really
going to enable to achieve envisioned change expressed
in objectives.
 Modeling approaches could represents solid
methodological gound for this purpose
Evaluation processes
Mid term evaluation
 Is the independent exercise which is carried out in the
middle of the strategy/program implementation.
 The changes on the level of specific
objectives/development priorities shall be observed and
results shall be measured
 Analytical methods are envisioned for this stage
 At minimum evaluation framework for impacts shall be
included
Evaluation processes
Ex post evaluation
 The last independent study is carried out at the end of
the strategy implementation.
 The impact of the strategy implementation shall be
visible and strategic objectives shall be achieved
 Analytical methods are envisioned at tis stage
Contextual monitoring and evaluation
 This M/E is based on the prediction that the
intervention of the strategy exist in the environment
which might itself to strengthen or weaken our
intervention instruments
 It helps to distinguish to which extent the change was
caused by our intervention and to which extent the
change was caused by external factors
 Contextual indicators help to carry this process
Self-evaluation
 is the process which is carried out in the case when
stakeholders create partnerships, which
prepares,implements and evaluates the strategy or
program
 it shall help to steer/coach the strategy in such a way
that objectives are followed
 for this purpose any findings of not well prepared
interventions within the self-evaluation process shall be
discussed and necessary changes of the strategy shall be
implemented as the lessons learned
 this does not exclude the independent evaluation
Monitoring and evaluation framework – part
of the strategy/program design
 Already in the stage of the strategy/program design it is necessary




to prepare monitoring and evaluation framwork
It contains:
To set up proper monitoring mechanism – indicators – input and
output indicators (reflecting the activities and projects level)
To set up proper evaluation system – result (reflecting specific
objectives) and impact indicators (reflecting strategic goals)
To design the system together with the setting up objectives, goals,
priorities, measures and activities
Monitoring and evaluation
framework contains
 Evaluation questions
 Indicators
 Baselines
 Targets
All this is interlinked with the intervention
logics of the overall program, priority and
particular measures
Monitoring and evaluation framework
 shall be part of the strategy from the beginning
 It shall be organicaly incorporated into the strategy and shall
be designed based on the situation analysis, needs and
objectives, measures and activities
The strategy framework - completed
Partnershi
p
Territory
Vision
Strategic
objecive
Impact
Impact
Indicators
Specific
objectives
Result
Result
indicator
s
Measures
Output
Output
Indicator
s
I
n
t
e
r
v
e
n
t
i
o
n
Resource audit
Analysis
Priorities for
development
Input
indicators
E
v
a
l
u
a
t
i
o
n
Moni
torin
g
Evaluation questions
 Common – cross strategy evaluation questions and specific
(sector/priority/measures) – are those which shall be
answered in the evaluation process.
 Shall be well formulated so we will receive the answer if
objectives were reached
Indicators
Are instruments for the measurement of interventions
Indicators are on the level of:
Monitoring – input and output (physical financial) and results
Evaluation – results (priority level) and impacts (the cross
strategy level)
Indicators in monitoring and evaluation framework
Inputs
Resources
•money
•people
•supply
•equipment
•facilities
Activities
What plan
does
•Services
•Training
•Processing
Outputs
Products or
services.
Quantitative
changes
•Numbers of
products or
services
Results
Inmediate
results on the
priority level,
qualitative
changes
connected
directly with
intervetion
•New quality,
changes
•New
knowledge
•skills
•attitude
•nové správanie
sa
Impacts
Long term
qualitative
changes and
effects
connected
with
intervention in
certain
environment
•Achievement
of the strategy
objectives
•Improved
situation
Baselines
 Are base data we were able to collect in resource audit
or in any other additional data collection and are
available in existing data collection systems, which help
us to understand the initial situation before the
intervention starts
 Accessible data shall be in balance with M/E indicators
or vise versa – the preferably we shall select indicators
which can be filled with data
Target levels
Are wished situations in which the development priorities shall
be after intervention is completed and express the
quantitative level of the change
E.g. unemployment decrease in 25% is the target level
Objectives
Evaluation
questions
Indicators
Baseline
situation
(situation
analysis,
SWOT)
Target levels
Strategic
/overall
objectives
Cross strategy
/horizontal
evaluation
questions
Impact
indicators
Baseline values
of impact
indicators
Target
levels for
impact
indicators
Specific
objectives
Horizontal
evaluation
questions relating
to specific
objec/priorities
Result indicators
Baseline values
of result
indicators
Target
levels of
result
indicators
Operational
objectives,
measures
activities
Measure specific
evaluation
questions
Output
indicators
Baseline values
of output
indicators
Target
levels of
output
indicators
Monitoring and evaluation framework
Example of the monitoring framework
Inputs
Outputs
Input
indicator
The value Output indicators
of
indicators
Target
Baselin
e
Actual
status
Volume of the
financial funds
for the measure
20 000 00
0,- Sk
Number of newly built and
reconstructed places for
accomodation
Increase
with 26 %
15
19
Number of beds in the places
for stay
Increase
with 26 %
1600
1700
Number of houses offering
rooms for accomodation
Increase
with 26 %
120
150
Example of the evaluation framework
Evaluation question
Impact indicators
Target level
Baseline situation
Question № 1:
To which extent has
the
competitiveness
of the
microregion
increased?
Educational degree of the
population
Increase with 7
%
Primary (26,21%),
Special/professio
nal (19,21%),
Secondary (22,82 %),
University (2,93
%)
Annual turnover in SK
Increase with 10
%
No data available
Investments in SK
Increase with 15
%
No data available
Actual
Thank you for your attention!
The Helpdesk of the European Evaluation Network for
Rural Development
[email protected]
[email protected]
260, Chaussée Saint – Pierre - Etterbeek - 1040 Brussels – Belgium
Tel: +32 2 736 49 60
Email: [email protected]
http//enrd.ec.europa.eu/evaluation/