EVALUATION OF INSTITUTIONAL STRENGTHENING PROJECTS
PRESENTATION AT ECA NETWORK MEETING
Tirana, Albania, 26 March 2008
By
Stefan Musto
Evaluation Consultant of the
Multilateral Fund for the Implementation of the Montreal Protocol
Based on Document
UNEP/OzL.Pro/ExCom/54/13
1.
OVERVIEW OF IS PROJECTS
580 IS projects including extensions approved for US$63,921,291,
of which
205 have been approved for Africa (total funding US$17,986,103)
179 for Asia and the Pacific (US$22,527,742)
158 for Latin America and the Caribbean (US$19,373,105)
38 for Europe (US$4,034,341)
Ansgar Eussner
Senior Monitoring and Evaluation Officer
2.
IS APPROVAL BY AGENCY
France
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
Overall
Germany
IBRD
4
2
1
1
1
1
2
1
2
1
1
2
3
2
2
2
3
3
4
2
3
7
UNDP
5
9
5
2
9
4
11
5
13
6
12
6
12
7
10
12
24
UNEP
UNIDO
8
13
9
11
16
16
22
19
16
37
35
45
35
53
42
103
2
USA
1
2
1
1
3
3
3
1
2
3
4
2
3
11
1
Overall
10
21
19
13
22
21
30
33
37
27
53
47
63
50
69
61
143
Ansgar Eussner
Senior Monitoring and Evaluation Officer
3.
THE DISTRIBUTION OF DELAYS ACROSS THE
351 COMPLETED IS PROJECTS:
Early completion
21
On time
72
1-6 months delays
74
7-12 months delays
68
13-24 months delays
60
25 months delays or more
56
Ansgar Eussner
Senior Monitoring and Evaluation Officer
4.
IMPLEMENTATION DELAYS OF IS PROJECTS BY REGION
Region
Number of Number of Approved Total Percentage
Countries Countries Durations Months of Months
Approved Delayed (Months) Delayed Delayed
Africa
52
52
5,397
2,313
43%
Asia and the Pacific
44
40
4,480
1,506
34%
Europe
12
9
1,042
417
40%
Latin America and the Caribbean
33
33
4,134
1,587
38%
Total
141
134
15.053
5,823
39%
Ansgar Eussner
Senior Monitoring and Evaluation Officer
5.
OBJECTIVES SHOULD BE EXPRESSED IN ‘SMART’ TERMS:
Specific (clear, unambiguous, concrete)
Measurable (you can measure what has changed)
Achievable (you know it can be done – and how to do it)
Realistic (it is useful to do and you have resources to do it)
Time specific (it has a deadline)
Ansgar Eussner
Senior Monitoring and Evaluation Officer
6.
OBJECTIVE, ACTIVITY AND RESULT IN
LOGIC CHAIN APPROACH
Assessment
Clear
Partial
Unclear
Objective
Activity
To raise the
awareness of the
public about ozone
depletion and the
country’s
obligations under
the Montreal
Protocol.
To continue efforts
to raise public
awareness.
Produce leaflets and
posters, write articles
for newspapers and
organise a workshop
for the media.
To raise public
awareness.
Celebrate
International Ozone
Day.
To hold meetings to
celebrate
International Ozone
Day.
Result
Expected
Produce 4 leaflets,
2 posters and
regular articles for
the newspaper.
Achieved
4 leaflets, 3 posters
and fortnightly
articles produced.
Media workshop led
to more and better
reporting on Montreal
Protocol activities.
News media will
report International
Ozone Day so
public will be more
informed.
Increased public
awareness of
ozone.
Ozone Day was
reported widely,
including on the TV.
Public more aware of
ozone issues.
Ansgar Eussner
Senior Monitoring and Evaluation Officer
7.
THE EVALUATION WILL TRY TO ESTABLISH
WHETHER AND TO WHAT EXTENT THE NOU
– THROUGH IS SUPPORT – IS INVOLVED IN:
Shaping the political and administrative structures
Developing the legislation so it is logically constructed to ensure
compliance and is easily implemented
Ensuring that the legislation and other policy instruments are
enforced
Facilitating and fostering stakeholder cooperation
Ansgar Eussner
Senior Monitoring and Evaluation Officer
8.
THE EVALUATION ISSUES IDENTIFIED INCLUDE:
The results and impact of IS projects
The political and administrative context
The planning of IS projects
Implementation issues
Future work
Funding issues
Ansgar Eussner
Senior Monitoring and Evaluation Officer
8. a)
RESULTS AND IMPACT OF IS PROJECTS SO
FAR:
To investigate what has been funded and what has been achieved to
date that can be wholly or partially attributed to IS funding.
To find and present evidence of what IS projects have contributed to
Montreal Protocol implementation and achievement of compliance
in a wide range of Article 5 countries.
To identify best practice, common difficulties (and their solutions)
and lessons learned.
To define, if possible, quantifiable metrics that can be used to
demonstrate the value of IS projects.
Ansgar Eussner
Senior Monitoring and Evaluation Officer
8. b)
POLITICAL AND ADMINISTRATIVE CONTEXT:
To assess how well decision 30/7 has been implemented to date,
in particular how NOUs have managed to integrate their work
plans into the internal planning processes of national authorities.
To analyse how far the mainstreaming of ozone issues into
national planning and budgets has been realized.
To investigate NOUs influence and work with decision makers
in government and industry
To review the extent and success of NOU partnership working
with national, regional and international organisations.
Ansgar Eussner
Senior Monitoring and Evaluation Officer
8. c)
PLANNING AND REPORTING OF IS PROJECTS:
To assess how NOUs and implementing agencies plan IS projects to
maximise their contribution to achieving and sustaining compliance.
To look at how NOUs understand the links between objectives,
activities and expected results and whether they would benefit from
a ‘logic chain approach’.
To assess the quality and usefulness of terminal reporting to the NOU,
implementing agency and the Executive Committee.
Ansgar Eussner
Senior Monitoring and Evaluation Officer
8. d)
IMPLEMENTATION ISSUES:
The extent, causes and significance of implementation delays
and implications on compliance (on average IS projects
experience delays of over one third of their expected duration)
Recruitment, induction, training and retention of National
Ozone Officers and NOU staff
The extent, duplication, resource implications and usefulness
of reporting requirements on NOUs
How to make the most of links between NOUs and PMUs and
between IS projects and other MLF supported projects
Ansgar Eussner
Senior Monitoring and Evaluation Officer
8. e)
FUTURE WORK:
To investigate how the demands on NOUs will change in future,
including the introduction of HCFC controls, and the implications of
these changes for the organisation, activities, accountability and funding
of IS projects and National Ozone Units.
To assess the implications of the new focus on phase-out of HCFCs on
the volume and composition of tasks for the NOU and to determine the
potential for narrowing the activities supported
through IS.
To assess efficiencies to be gained from enhanced cross-NOU/regional
activities (e.g. for public awareness).
Ansgar Eussner
Senior Monitoring and Evaluation Officer
8. f)
FUNDING ISSUES:
To review the current uses of IS funding and the value for
money achieved.
To assess availability of funding against priority needs,
including scope for additional (counterpart) contributions.
To assess how decision 35/57 and the subsequent increase of
IS funding has and is being used by countries for awareness
activities and the implementation of the strategic approach
approved with this decision.
Ansgar Eussner
Senior Monitoring and Evaluation Officer
8. f) FUNDING ISSUES (cont’d)
To investigate the incentives provided by (i) linking funding to
ODS consumption and (ii) keeping levels of funding constant and
(iii) the impact of potential alternatives like gradually reduced
funding on NOU activities and performance when HCFCs remain
as only one significant group of chemicals to be phased out.
To identify the likely consequences if IS funding was not, or not
fully, available in its current form, together with factors that could
encourage NOUs to achieve long-term self-sustainability, taking
into consideration the different needs of VLVC countries and
LVC countries.
Ansgar Eussner
Senior Monitoring and Evaluation Officer
8. f) FUNDING ISSUES (cont’d)
To identify technical conditions that may need to be satisfied
for approval/disbursement of IS funding in the HCFC context
compared to those in the CFC context to improve efficiency
and value of IS, e.g. establishment of licensing schemes or
other basic regulatory measures.
Ansgar Eussner
Senior Monitoring and Evaluation Officer
9.
CONCLUSIONS:
To assess the value of IS projects in terms of results and impact
and its value after 2010.
To make recommendations on criteria and levels of IS funding
appropriate to future IS project activities.
To formulate requirements for effective and efficient
implementation.
Ansgar Eussner
Senior Monitoring and Evaluation Officer
10.
THE NEXT STEPS WILL BE TO:
Finalize the evaluation instruments (checklist of questions and
questionnaire) in light of the information and feedback
obtained from the initial series of interviews and questionnaires
received
Distribute revised questionnaire to all remaining NOUs and
establish a list of further interview partners
Ansgar Eussner
Senior Monitoring and Evaluation Officer
10.
THE NEXT STEPS WILL BE TO (cont’d):
Collect information at network meetings and during the
preparation of country case studies; other means of
communication will also be used, as required, such as fax
and e-mail
Prepare case studies for all regions
Prepare the synthesis evaluation report with findings and
recommendations for consideration by the 56th Meeting
of the Executive Committee
Ansgar Eussner
Senior Monitoring and Evaluation Officer