Theory-Based Evaluation:
Untapped potential for EU
Regional Policy
Marielle Riché
Evaluation Unit, DG Regional policy,
European Commission
Helsinki, 4th October 20124 October 2012
Regional
Policy
Content of paper
• Evaluations in EU regional policy: state of play
• TBE are relevant for regional policy: throughout
policy cycle and for policy making
• TBE still has much untapped potential: too much
focus on funds absorption and lack of clear
theories of change
• 2014-2020: new opening for TBE (stronger focus
on results, new regulatory requirements)
• Looking forward
Regional
Policy
Evaluations in EU regional policy
• Programme and evaluation management: MS or
regions (EC: ex post evaluations)
• Programmes run for 9 years (2007-2013/15)
• Cover research, ICT, urban generation, SMEs,
environment, social infrastructures etc.
• May focus on specific territories, group of people,
sectors
• May use "integrated approaches", may support
"innovation systems" – complex programmes
Regional
Policy
Evaluations in EU regional policy
• MS/Regions: some 266 evaluations since 2007
• Most on implementation issues and policy
relevance, few on outcomes (27%)
• Most based on traditional qualitative techniques
and monitoring data
• EC role: improve evaluation practice
• Long-range aim: more rigorous evidence for
policy making, building shared knowledge
("Regio collaboration")
Regional
Policy
TBE are relevant for regional policy
• Programmes (should) rely on a theory of change
• TBE are relevant throughout the programming
cycle (ex ante, in itinere, ex post)
• Analyse the assumptions underlying the
programme
• Assess the plausibility of programme
achievements as planned
• Assess contribution, analyse mechanisms leading
to results, context, unintended effects
Regional
Policy
TBE are relevant for regional policy
• Fields of intervention where counterfactual
impact evaluations not possible
• Complicated/complex programmes
• Context analysis allows for policy adaptation and
replication
• Communication on programme outcomes
(narratives)
• Participative evaluations (partnership principle)
Regional
Policy
An untapped potential
• Programmes more focused on funds absorption
than on results
• Evaluations carried out at the beginning of the
period, focused on implementation
• No explicit theories in programmes
• Over-confidence in indicators to measure actual
outcomes
• "Myopia" of programmes (focus on first results)
• Reluctance to commit to outcomes which are not
"under control"
Regional
Policy
2014-2020: a new opening for more TBE
• Political commitment for a "result-based" policy
• New logical framework, taking into account
context and other factors, where impact is the
contribution to observed results
• Stronger requirements for ex ante evaluations
• Compulsory impact evaluations for each priority
axis of programmes (covers different interlinked
interventions)
• Our preparatory work with Member States:
guidance and pilots
Regional
Policy
How to implement these new
requirements for 2014-2020?
Our advice to Member States:
• Use the evaluation ex ante to strengthen the
programme theories of change
• Be open to new approaches and allow for
discussion on methods in Terms of Reference
• Be realistic: take into account budget and time
constraints, think of the primary use of
evaluation (improve implementation or assess
effectiveness? target audience?)
Regional
Policy
How to implement these new
requirements for 2014-2020?
Our advice to evaluators:
• Help commissioners to refine evaluation
questions
• Propose and defend TBE designs mixing
methods according to evaluation questions,
timing, budget, planned use
• Document data collection and analysis
processes in evaluation reports
Regional
Policy
Conclusion
• We should not miss the opportunity given by
ex ante evaluations
• Common goals: create a body of evidence on
regional development
• Need for critical mass of good quality TBE: mix
of methods framed by a rigorous design
• Need to develop strategies to minimise
potential bias in data collection and analysis
• Need to develop criteria and standards for
theory-based designs
Regional
Policy
Reference documents
On Inforegio Website:
http://ec.europa.eu/regional_policy/information/evaluations/index_en.cfm
•
•
•
•
Guidance documents
Impact evaluation center
Evaluation library (programmes)
EC evaluations
12
Regional
Regional
Policy
Policy