PREPARING for MTE of RDPs:
A SURVEY OF THE MEMBER STATES
Irina RAMNICEANU
Helpdesk Evaluation Manager
Evaluation Expert Committee
2nd meeting
OUTLINE OF THE PRESENTATION
• What are the aims of the survey?
• How was the work organized?
• The MTE in the ongoing evaluation process
• Coordination and capacity building
• Towards organising the MTE ...
• Difficulties encountered and Helpdesk support
23 June 2009
Evaluation Expert Committee
2nd meeting
2
WHAT ARE THE AIMS OF THE SURVEY?
 underpins the MTE Guidelines developed by the
Helpdesk
 sheds light on the design of the ongoing evaluation
systems – and on how the MTE fits in
 highlights challenges with the MTE
 provides an EU-wide reference on evaluation activities
23 June 2009
Evaluation Expert Committee
2nd meeting
3
HOW WAS THE WORK ORGANISED?
Helpdesk
GEs and
MAs
Helpdesk
• developed the
questionnaires
• filled out the
questionnaires
• drafted the survey
report
23 June 2009
62 returns
so far!
Evaluation Expert Committee
2nd meeting
4
THE MTE IN THE ONGOING EVALUATION PROCESS
1. “Minimal outsourcing” …
ex ante
ongoing
mid-term
ongoing TBD
ex post
… in 14 programmes
23 June 2009
Evaluation Expert Committee
2nd meeting
5
THE MTE IN THE ONGOING EVALUATION PROCESS
2. “In-house project” …
Activity 1
ex ante
mid-term
ongoing
Activity 2
23 June 2009
Evaluation Expert Committee
2nd meeting
ongoing
ex post
… in 2 programmes
6
THE MTE IN THE ONGOING EVALUATION PROCESS
3. “Sequential outsourcing (simple) ” …
ex ante
ongoing
mid-term
ongoing TBD
ex post
… in 4 programmes
23 June 2009
Evaluation Expert Committee
2nd meeting
7
THE MTE IN THE ONGOING EVALUATION PROCESS
4. “Sequential outsourcing (multiple) ” …
ongoing 1
ex ante
ongoing 2
mid-term
ongoing TBD
ex post
ongoing 3
… in 4 programmes
23 June 2009
Evaluation Expert Committee
2nd meeting
8
THE MTE IN THE ONGOING EVALUATION PROCESS
5. “Full-outsourcing (1) ” …
ex ante
ongoing
mid-term
ongoing TBD
ex post
… in 7 programmes
23 June 2009
Evaluation Expert Committee
2nd meeting
9
THE MTE IN THE ONGOING EVALUATION PROCESS
6. “Full-outsourcing (2) ” …
ex ante
ongoing
mid-term
ongoing
ex post
… in 26 programmes
23 June 2009
Evaluation Expert Committee
2nd meeting
10
COORDINATION AND CAPACITY BUILDING
training
75% of the MAs are
enhancing their
evaluation
capacities
39%
additional staff
27%
not needed
26%
redeployment of existing
resources
They typically use a
mix of instruments
23 June 2009
23%
other
investment in equipment and
facilities
Evaluation Expert Committee
2nd meeting
18%
8%
11
COORDINATION AND CAPACITY BUILDING
supervision
60% of the MAs have
an evaluation
steering group
45%
decision-making
39%
advice
Its role and composition
vary significantly
across MS
37%
coordination
other
35%
13%
.
23 June 2009
Evaluation Expert Committee
2nd meeting
12
TOWARDS ORGANISING THE MTE ...
no answer, 26
planning
23
20
MTE part of
ongoing, 39
tender
launched
Timely
appointment
of
independent
evaluators
is critical!
selection
concluded
contract started
1
1
1
23 June 2009
Evaluation Expert Committee
2nd meeting
13
TOWARDS ORGANISING THE MTE ...
36
 Progress on reviewing
evaluation questions,
indicators and
intervention logic is
limited.
37
36
26
25
22
Yes
No
Responsibility of the MAs?
Evaluation questions Evaluation indicators
Intervention logic
.
 Arrangements for evaluators’ access to data are well
advanced.
23 June 2009
Evaluation Expert Committee
2nd meeting
14
DIFFICULTIES ENCOUNTERED ...
Data collection and methodologies
80% of the programmes
expect difficulties.
A wide range of problems:
.



data gaps
retrieve data from/on
(non-)beneficiaries
net impacts
23 June 2009
need to optimize coordination
with other domestic institutions
47%
lack of clear definitions of the
indicators
47%
challenges in setting up IT
systems
42%
challenges in data collection
methodologies
other
Evaluation Expert Committee
2nd meeting
40%
29%
15
... AND HELPDESK SUPPORT
Data collection and methodologies:
Thematic working groups (HNV, assessment of
impacts)
Guidance on specific topics (e.g. GVA)
Other: e.g. good practice, questions from the MS
15 June
23
December
2009 2008
Evaluation Expert Committee
2nd meeting
16
DIFFICULTIES ENCOUNTERED ...
Administrative and other matters
56% of the programmes
need(ed) support on
the TORs.
Other matters:
. clarifications on indicator


fiches
explanatory notes on the
evaluation questions
glossary of terms
23 June 2009
tasks of the evaluators
31%
structure of the ToRs
19%
planning of contracted
activities
16%
required qualifications of the
evaluators
16%
other
Evaluation Expert Committee
2nd meeting
10%
17
... AND HELPDESK SUPPORT
Administrative and other matters:
MTE Guidelines for the MS
Dedicated good practice examples
Dedicated FAQ section on the website
Glossary of key evaluation terms
15 June
23
December
2009 2008
Evaluation Expert Committee
2nd meeting
18