SYNTHESIS OF THE ANNUAL PROGRESS
REPORTS FOR 2007
CONCERNING ONGOING EVALUATION
Irina RAMNICEANU
Helpdesk Evaluation Manager
Evaluation Expert Committee
2nd meeting
OUTLINE OF THE PRESENTATION
• What is the purpose of the synthesis?
• How was the work organized?
• General findings
• Specific findings (per reported topic)
• Role of the Helpdesk
23 June 2009
Evaluation Expert Committee
2nd meeting
2
WHAT IS THE PURPOSE OF THE SYNTHESIS?
The Evaluation Helpdesk prepares it annually, to ...
 ... help improve ongoing evaluation systems in the MS
 ... enhance the quality of evaluation reporting
 ... identify and promote good practice in conducting and
reporting about evaluation activities
23 June 2009
Evaluation Expert Committee
2nd meeting
3
HOW WAS THE WORK ORGANISED?
National/Regional MC
Report
ongoing
evaluation
activities
EC (Desk Officers)
EC and MS
APR
summary
1.
2.
3.
4.
5.
6.
7.
….
….
….
Ongoing ev
….
….
….
Section
Section
APR
Section
APR
Section
Ongoing
Ongoing
1.
….
Ongoing
Ongoing
synthesis
evaluation
2. Section
….
evaluation
evaluation
evaluation
3.Ongoing
….
APR
APR
4.APR
Ongoing ev
APR
evaluation
5.
6.
7.
23 June 2009
….
….
….
Evaluation Expert Committee
2nd meeting
4
A USEFUL REFERENCE
The Guidance Note B – Evaluation Guidelines
23 June 2009
Evaluation Expert Committee
2nd meeting
5
GENERAL FINDINGS
Few of the reports for
2007 follow closely
the indicative
structure
90
80
70
60
50
40
30
20
10
0
Yes
No
They vary in quality,
length and
information content
23 June 2009
Evaluation Expert Committee
2nd meeting
6
GENERAL FINDINGS
43
...but 2007 was a special
year:
22
 first time such reporting required
 reports to focus on the ongoing
evaluation systems
18
6
Q2-07
 most of the RDPs approved late in
the year
23 June 2009
5
Q3-07
Q4-07
Q1-08
Q2-08
The number of RDPs approved , by quarter
Evaluation Expert Committee
2nd meeting
7
SETTING UP THE EVALUATION SYSTEMS
 25% of the programmes deal with the evaluation
questions and indicators
 50% of the programmes refer to administrative
preparations
 Type of ongoing evaluation system
 Actors involved (including a steering group)
 Contracting evaluators: TORs and beyond
23 June 2009
Evaluation Expert Committee
2nd meeting
8
SETTING UP THE EVALUATION SYSTEMS
 Most programmes (plan to) fully outsource their
evaluation activities, as a “single package”
 Evaluation activities bundled together and outsourced as
one set
 Leads to improved continuity and consistency of
evaluation throughout the programming period
 Alternatives refer to: multiple tenders and minimal
outsourcing (only mandatory steps)
 But difficult to draw a clear and comprehensive picture
EU-wide
23 June 2009
Evaluation Expert Committee
2nd meeting
9
SETTING UP THE EVALUATION SYSTEMS
Recommendations for the MS:
Cover all the key components of the evaluation system
Be clear and specific about how the ongoing evaluation is
implemented (type, duration, scope of contracts etc.)
Clarify what has been achieved and what is still planned
 In 2008, include information on your ongoing evaluation system if
not already done so in 2007
For 2008 and 2009, give particular attention to the administrative
preparations for the MTE
23 June 2009
Evaluation Expert Committee
2nd meeting
10
ONGOING EVALUATION ACTIVITIES
 Limited focus in the 2007, but increasing in 2008+
 Confusion about reporting on “borderline activities” ...
 Ex-post 2000-06 covered in 10 reports
 Ex-ante 2007-13 referred to in 20 reports
 ... and wide-ranging scopes and levels of detail
 Already some reference to methodological issues
23 June 2009
Evaluation Expert Committee
2nd meeting
11
ONGOING EVALUATION ACTIVITIES
Recommendations for the MS:
Ex-post 2000-06 only if clear relevance and follow-up in 2007-13
Only a concise reference to the activities of the ex-ante evaluation
Clearly state any activities supporting evaluation undertaken or
planned (e.g. thematic studies)
 You may refer to methodologies developed or planned for the
evaluation of your RDP
Include the evaluation capacity building activities
Note: data collection activities are detailed under a distinct chapter
23 June 2009
Evaluation Expert Committee
2nd meeting
12
SYSTEMS FOR DATA COLLECTION AND
MANAGEMENT
 Data collection described in 30% of the reports
 Focus on responsible parties and coordination
 Typically MA pools data together, but decentralised
models also available
 Any differentiation by data type?
 Limited information about data collection methodologies
 Development in progress, responsible bodies mentioned
 Data management systems in 15% of the reports
 Update on progress
 Institutions responsible
23 June 2009
Evaluation Expert Committee
2nd meeting
13
ONGOING EVALUATION ACTIVITIES
Recommendations for the MS:
Highlight progress and/or difficulties encountered with regard to
data collection (including methodologies)
Distinguish, as appropriate, between types of data collected
Describe institutional arrangements for data collection and
management only when set up; continue reporting changes (if any)
Explain arrangements made to develop/update IT systems
23 June 2009
Evaluation Expert Committee
2nd meeting
14
OTHER TOPICS
 Networking activities
 Very little information in 2007
 Most developed (or reported on) in multi-programme
countries
 Difficulties encountered and need for future work
 Only a few insights available in 2007
 Systems and methodologies for data collection, e.g.
enhance usability of IT systems, check functionality
 Methodological aspects, e.g. HNV indicator, determine
net intervention effects etc.
23 June 2009
Evaluation Expert Committee
2nd meeting
15
OTHER TOPICS
Recommendations for the MS:
Report progress against previous year(s) only, without repeating
past information
Clearly demarcate activities carried out from activities planned
[You may consider adding a timetable of planned activities]
Be clear and concise in your wording
Do not make extensive references to legal texts or guidelines
23 June 2009
Evaluation Expert Committee
2nd meeting
16
ROLE OF THE HELPDESK
 Support the EC desk officers in charge of the APRs
(including through training)
 Provide detailed recommendations in each annual
synthesis report
23 June 2009
Evaluation Expert Committee
2nd meeting
17