Ref. Ares(2015)2298948 - 02/06/2015
ESF 2007-2013 Ex-post
Evaluation Synthesis
Contract No. VC/2015/0098
EUROPEAN COMMISSION
Directorate-General for Employment, Social Affairs and Inclusion
Unit A3 — Analysis, Evaluation, External Relations
Contact: Andris Skrastins
E-mail: [email protected]
European Commission
B-1049 Brussels
EUROPEAN COMMISSION
ESF 2007-2013 Ex-post
Evaluation Synthesis
Contract No. VC/2015/0098
Directorate-General for Employment, Social Affairs and Inclusion
[Ex Post Evaluation Synthesis]
2014
EUR [number]EN
Contract No. VC/2015/0098
TABLE OF CONTENTS
Introduction ..................................................................................... 8
1.
OVERALL APPROACH to the synthesis ........................................ 9
1.1 The purpose of the ESF 2007-2013 ex post evaluation synthesis ........9
1.2 The scope of the ESF 2007-2013 ex post evaluation synthesis and
use of sources .............................................................................9
1.3 Evaluation criteria ...................................................................... 10
1.4 Our approach to the synthesis ..................................................... 12
1.4.1 Providing missing information ....................................................... 13
1.4.2 Drawing the data together ............................................................ 13
1.4.3 Allocating Priority Axes to one single Priority (and sub priorities) ........ 15
1.4.4 Aggregating output and result indicators ......................................... 15
2.
METHODOLOGY AND TOOLS ..................................................... 24
2.1 Main tasks and deliverables ......................................................... 24
2.2 Evaluation task 1: Follow up of ESF thematic ex-post evaluations ..... 28
2.3 Evaluation task 2: National level analysis ...................................... 36
2.3.1 National inventories ..................................................................... 36
2.3.2 Country synthesis reports ............................................................. 38
2.4 Evaluation task 3: EU level analysis .............................................. 46
2.4.1 Thematic synthesis reports ........................................................... 46
2.4.2 EU synthesis report ..................................................................... 59
3.
REPORTING ............................................................................. 70
3.1 The Inventory of Evaluations ....................................................... 70
3.2 The Country Reports .................................................................. 75
3.3 The Thematic Synthesis Reports .................................................. 77
3.4 The Final Synthesis Report .......................................................... 78
4.
WORK ORGANISATION ............................................................ 80
4.1 Roles and responsibilities of the team ........................................... 80
4.1.1 Functions, responsibilities and work flows ....................................... 81
4.2 Allocation of time and resources to the project and to the tasks........ 84
4.3 Timetable ................................................................................. 88
Annex 1: The allocation of Priority Axes to ESF Priorities ........................ 91
page 4
May 2015
Contract No. VC/2015/0098
TABLES, FIGURES AND BOXES
Table 1. Evaluation criteria, sources and additional data gathering .......... 11
Table 2. Overview of tasks, data sources and outputs ........................... 25
Table 3. Database structure for the thematic ex post evaluations and EU
synthesis report ................................................................... 34
Table 4. Overview of steps for the accomplishment of Task 1 ................. 36
Table 5. Overview of steps for the accomplishment of Task 2A ............... 37
Table 6. Table of contents for country synthesis reports ........................ 40
Table 7. Overview of steps for the accomplishment of Task 2B ............... 45
Table 8. EU Synthesis Report structure ............................................... 56
Table 9. Overview of steps for the accomplishment of Task 3A ............... 59
Table 10. Structure of the EU synthesis report ..................................... 60
Table 11. Context indicators for each ESF priority (as included in the
EU27 templates) .................................................................. 63
Table 12. Overview of steps for the accomplishment of Task 3B .............. 69
Table 13. Content of the inventory ..................................................... 71
Table 14. Working days allocated to the partners in the Consortium ........ 81
Table 15. Allocation of working days to the tasks .................................. 84
Table 16. Working days allocated to each member in the team (including
tasks) ................................................................................. 86
Table 17. Working days per expert ..................................................... 87
Table 18. Timetable.......................................................................... 89
Table 19. The allocation of pas in the thematic ex post evaluations and
proposed allocation to promoting partnerships and increasing
administrative capacity ......................................................... 91
Figure 1. Tasks and deliverables of the ex post evaluation synthesis........ 24
Figure 2. From evidence to findings to lessons learned .......................... 68
Figure 3. Overview of the team structure and work division.................... 80
Figure 4. Generalized work flow ......................................................... 84
Box 1. An example: Programme specific output and result indicators in
A2E .................................................................................... 20
May 2015
page 5
Contract No. VC/2015/0098
Box 2. Discussion points /guidelines on the construction and handing
over of the the database ....................................................... 30
Box 3. Updating financial and physical performance data with AIR 2014... 43
Box 4. Approach to efficiency in the three ex post evaluations ................ 66
Box 5. Template for the Phase 2 Country Reports ................................. 75
Box 6. Draft Thematic Synthesis Report Structure ................................ 77
Box 7. Final Report Structure ............................................................. 78
page 6
May 2015
Contract No. VC/2015/0098
LIST OF ABBREVIATIONS
May 2015
A2E
Access to employment and sustainable integration into
employment
AIR
Annual Implementation Report
ALMP
Active labour market policy
CAV
Community Added Value
CE
Country Experts
CIE
Counterfactual Impact Evaluation
CoE
Code of expenditure
CSR
Country synthesis report
EC
European Commission
EEN
Expert Evaluation Network
EES
European Employment Strategy
ESF
European Social Fund
EU
European Union
LFS
Labour force survey
LMP
Labour market policy
MA
Managing Authority
MS
Member State
NGO
Non-governmental organisation
OP
Operational Programme
QA
Quality assurance
PES
Public Employment Service
PL
Project leader
PR
Public relations
SFC
Structural Funds Common Database
SI
Social Inclusion
TA
Technical Assistance
TBE
Theory Based Evaluation
page 7
Contract No. VC/2015/0098
INTRODUCTION
This revised Inception Report is for the “ESF 2007-2013 ex post evaluation
synthesis” (Contract No. VC/2015/0098).
The contractor is a consortium led by Metis GmbH with Fondazione G.
Brodolini (FGB) and Panteia as partners. All partners will contribute to the
study and each has one key expert in the core team.
Herta Tödtling-Schönhofer (Metis) will manage the evaluation with
operative support from Isabel Naylon (also Metis). Building on previous
cooperation in the framework of the three thematic ex post evaluations1,
Isabel will manage Task 1. She will also manage Task 2 and ensure the
delivery of the updated inventories and the country synthesis reports. Metis
will supervise the drafting of the two thematic synthesis reports and the
overall EU synthesis reports (Task 3) which will be in the hands of the
Consortium partners Fondazione G. Brodolini (FGB) and Panteia
respectively. All tasks are led by very experienced experts who have been
analysing the current ESF programmes exhaustively in the recent past.
Benno Savioli will support the core team in the updating of the inventories
for which he produced the database in the ESF Expert Evaluation Network.
The overview team will support the core team by preparing and structuring
the available material and by supporting the core team members in drafting
the reports.
The present Inception Report includes a chapter on our overall approach to
the synthesis and a description of our proposed methodology and tools for
the management of the main tasks and outputs.
The main task of the evaluation synthesis will be bringing together all the
information and filling in the gaps as certain Priorities will have been
thoroughly covered in some countries but not in others and the degree of
depth of the information will differ. One of the main challenges in the
evaluation synthesis as in the three thematic ex post evaluations will be the
demarcation between the five2 ESF Priorities (according to Article 3 of the
ESF Regulation) which are the common thread throughout all the ex post
evaluation exercise. This will largely have been solved by the three
thematic ex post evaluations covering four Priorities (Adaptability and
Human Capital, Social Inclusion and Access to Employment). A further issue
will be linking the Annex XXIII and programme specific indicators to ESF
Priorities and aggregating these on country level and subsequently on
European level. We discuss these issues and our approach to them in
Chapter 1 and the overall methodology in Chapter 2. The structure of the
reports is given in chapter 3 and the overall management of the contract is
described in Chapter 4.
1
2
page 8
Isabel is managing the Ex Post Evaluation of Access and Sustainable Integration into Employment.
Adaptability and Human Capital are considered one Priority as they have been treated as such in the ex
post evaluation.
May 2015
Contract No. VC/2015/0098
1. OVERALL APPROACH TO THE SYNTHESIS
1.1 The purpose of the ESF 2007-2013 ex post evaluation
synthesis
In the course of 2014, three ESF 2007-2013 ex post evaluations were
launched by DG EMPL. These covered the ESF Priorities Adaptability and
Human Capital (grouped under the ex post evaluation on Human Capital),
the Integration of Disadvantaged Groups and Access and Sustainable
Integration into Employment. The purpose of Ex Post Evaluation Synthesis
is to synthesise the results of these thematic ex post evaluations and to
provide supplementary evaluation syntheses of the ESF Priorities Promoting
Partnership and Strengthening Institutional Capacity which were not
covered by separate contracts. The ex post evaluation synthesis also
includes 28 country reports summarising the outputs and results of the ESF
investments across the ESF Priorities and presenting findings and
conclusions as well as lessons learned and good practice.
The synthesis should draw lessons for economic and social cohesion policy
at Member State and EU level with reference to the European Employment
Strategy as incorporated into the Integrated Guidelines for Growth and
Jobs, within the framework of the Lisbon Strategy and the Europe 2020
Strategy, and the accompanying (country-specific) recommendations and
national reform programmes.
As well as serving the purpose of the ex post evaluation as defined in the
General Regulation (Art. 49.3), the ex post synthesis evaluation will also
have the function of supporting the communication of the Commission with
the Council and the European Parliament of the main results of the ESF as
well as to non-technical audiences.
1.2 The scope of the ESF 2007-2013 ex post evaluation
synthesis and use of sources
The synthesis will cover the interventions of the European Social Fund
during the 2007-2013 programming period in all 28 Member States. As
Croatia only joined the EU in 2013, at the end of the programming period
observed by the ex post synthesis, the analysis will only be able to cover
this short period of implementation. It will draw on the Operational
Programme document, Annual Implementation Reports 2013 and 2014 and
any available evaluations. The synthesis will cover all 117 + HR Operational
Programmes in the 28 Member States in both Convergence and
Competitiveness Regions.
The ex post evaluation synthesis will rely mainly on existing evaluation
material and information. However, to fill in the gap for the remaining two
ESF Priorities not covered in the three thematic ex post evaluations
(Promoting Partnerships and Strengthening Institutional Capacity), the
contractor will undertake additional research on the basis of existing
material and a limited number of interviews.
May 2015
page 9
Contract No. VC/2015/0098
The main sources of information will be:
 ESF 2007-2013 ex post evaluation: Access and sustainable integration
into Employment (2014)
 ESF 2007-2013 ex post evaluation: Investing in Human Capital (2014)
 ESF 2007-2013 ex post evaluation: Supporting the integration of
disadvantaged groups into the labour market and society (2014)
 Ex post evaluation of cohesion policy programmes 2007–2013
financed by the European Regional Development Fund (ERDF) and
Cohesion Fund (CF) — work package 12: delivery system
 DG Regio evaluation on effects of structural funds on GDP and macroindicators
 Evaluation of the reaction of the ESF to the economic and financial
crisis, Final Report by Metis GmbH in cooperation with wiiw
 SFC2007
 Annual Implementation Reports 2014 and previous AIRs when needed
 The deliverables of the ESF Expert Evaluation Network (2011-2013)3
 The deliverables of the Preparatory Study (2013)
 Operational Programmes and amendments/Implementing documents
 Relevant ESF Evaluations
 Relevant EU-wide evaluations
1.3 Evaluation criteria
The synthesis of the ex post evaluation should bring all information
together gathered during the 2007-2013 programming period on each of
the five ESF Priorities as defined the ESF Regulation (Art.3). The synthesis
will present the lessons learned in terms of policy choices, target groups
(with a special focus on specific target groups such as young people and
older workers), programming, implementation, monitoring and evaluation
of future programmes.
Particular attention will be paid to:
 the extent to which resources were used
 effectiveness (quantified results, factors contributing to success and
failure, identification of good practice)
 efficiency measured in terms of cost-effectiveness
 socio-economic impact
In addition, reflecting the focus of the thematic ex post evaluations,
findings will also be presented in terms of
 Community Added Value
 Gender sensitivity and
3
page 10
Including country reports, synthesis reports and inventories
May 2015
Contract No. VC/2015/0098
 Sustainability
The table below provides an overview of what evaluation issues the ex post
evaluation synthesis should report on, on Member State and EU level, and
the use of existing sources already produced as well as needs for additional
data gathering.
Table 1. Evaluation criteria, sources and additional data gathering
Evaluati
on
criteria
May 2015
Country
synthes
is
reports
(per
priority
)
Themati
c EU
synthes
is
reports
(per
priority
)
Contextu
ali-sing
ESF
Information about the
content and objectives
of the OPs and the types
of interventions funded
(by ESF priority).
Provide the context of
the
national
socioeconomic situation and
the national policies and
other
EU
level
strategies.
The
extent to
which
sources
were
used
Overview ESF spending
per
ESF
priority
(SFC2007 and AIR2014)
Effective
ness
Quantified results (the
outputs
of
the
interventions and the
results achieved).
Success factors
Good practice
Lessons learned
Efficienc
y
Cost effectiveness
Socio
economi
c impact
Analysis
of
socioeconomic
indicators
(ESF big part of funding
in
social
fields),
qualitative
evaluation
(other).
Other
criteria:
Community Added Value
Gender sensitivity
Existing Sources
27
filled-in
overview
country
templates
of
three ESF 2007-2013 Ex
post thematic evaluation
studies
Country
in-depth
assessment reports from
the relevant ESF 20072013 Ex post thematic
evaluation studies
Available country level
deliverables of the 'Ex
post'
evaluation
of
cohesion
policy
programmes 2007–2013
financed by the ERDF and
CF — work package 12:
delivery system
Annual
Implementation
Reports 2014
Operational
Programme
documents
Evaluation reports (till 30
June 2015
All
Expert
Evaluation
Network country reports
Relevant
country-level
deliverables
of
the
Preparatory study for the
ex post evaluation of ESF
2007-2013
EU wide data (Annex
XXIII and CoE)
Commission reports (1)
Employment and Social
Additional
data
gathering
Update
evaluations
produced by
MA
after
December
2013 till 30
June 2015
Information
from Croatia
(all
evaluation
reports, AIR
2013-2014,
OP
document)
Promoting
partnerships
and
Strengthenin
g Institutional
Capacity (OP
documents;
AIR
20072014;
Evaluations
available up
until 2015).
page 11
Contract No. VC/2015/0098
Evaluati
on
criteria
Themati
c EU
synthes
is
reports
(per
priority
)
Sustainability
at
the
level of individuals and
systems/structures/prod
ucts and in terms of the
sustainability
of
the
interventions
themselves
in
the
absence of ESF funding
Country
synthes
is
reports
(per
priority
)
Conclusi
ons
For the EU level policy
on economic and social
cohesion
Lessons
learnt
policy choices,
target groups,
programming,
and
implementation
monitoring
and
evaluations (with regard
to the new regulatory
requirements).
Existing Sources
Additional
data
gathering
Developments in Europe;
(2) Fourth report on
Economic and Social
Cohesion; (3)Fifth Report
on Economic, Social and
Territorial Cohesion; (4)
Sixth Report on
Economic, Social and
Territorial Cohesion
DG Regio evaluation on
effects of structural funds
on GDP and macroindicators
Evaluation of the reaction
of the ESF to the
economic and financial
crisis
1.4 Our approach to the synthesis
The Ex Post Evaluation Synthesis is based mainly on previous work in the
framework of the three thematic ex post evaluations and the approach to
the synthesis will be aligned with this work, e.g. with regard to definitions
of expenditure and structure of the presentation etc. It will also draw on the
ESF Expert Evaluation Network and Preparatory Study where appropriate,
e.g. in the updating of the Inventories of evaluations and the completion of
the thematic reports on Partnerships and Institutional Capacity. The Metis
report on the Evaluation of the reaction of the ESF to the economic and
financial crisis for DG EMPL will also be drawn on. The synthesis will also
integrate findings of the DG REGIO evaluation of the delivery systems as
results will be useful for the Institutional Capacity report. Other DG REGIO
reports that may be relevant are: a report on the effects of the crisis on the
(ERDF) programmes (July) and an econometric report in the framework of
Article 16 on the effects of structural funds on GDP and macro-indicators.
The analysis will select the best information available and formulate
synthetic messages on policy choices, target groups (with a special focus on
young people), programming, implementation, monitoring and evaluation of
future programmes.
page 12
May 2015
Contract No. VC/2015/0098
The challenge of this synthesis report, but also the three thematic ex post
evaluations is to bring together what has been gathered in other evaluation
exercises and (1) providing missing information, (2) drawing the data
together, (3) assuring a clear distinction between data related to each
Priority, and (4) aggregating output and result indicators. These challenges
should already have been dealt with in the individual thematic evaluations
but will nevertheless require further thought in the ex post synthesis.
1.4.1 Providing missing information
As mentioned above, the synthesis is based on existing information.
Nevertheless there are still some blank spots to be filled:
I.
Cumulated data from the 2014 AIRs (including financial figures and
programme specific output and result data)
II.
Evaluations by Managing Authorities (MA) implemented between
December 2013 and 30 June 2015
III.
Information on the Priority Themes Promoting Partnerships and
Strengthening Institutional Capacity. The synthesis should include
activities for drawing up country fiches finally feeding thematic EU
synthesis reports on these themes
IV.
Data on Croatia from the Operational Programme document, Annual
Implementation Reports 2013 and 2014 and available evaluations.
1.4.2 Drawing the data together
As indicated above there are a wide number of data sources on which the
thematic EU synthesis reports and country synthesis reports can draw.
However, one of the main challenges working with these data is that their
definitions and scope are overlapping and not always clearly demarcated
(the most striking example is the level at which financial data is stored
compared to data on physical progress). The challenge of the ex post
evaluation is how to link these different sources:
May 2015
I.
The purpose of the thematic ex post synthesis reports is to report on
the ESF Priorities as included in Article 3 of ESF Regulation
1081/2006. These do not always correspond to the Priority Axes of
the OPs.
II.
The Categories of Expenditure as defined in the Funds Regulation
1083/2006 and Implementation Regulation 1828/2006. The MS
submit to the Commission an indicative breakdown by these
categories. Categories 62-74 relate to the ESF. Each category can be
clearly linked to one of the ESF Priorities within the ESF Regulation
(Article 3) but not always very obviously to a Priority Axis or the
programme specific indicators. Nevertheless the Preparatory Study
for this ex post evaluation teaches us that around 70 percent of the
sub priorities / actions are linked to one specific category of
expenditure indicating that in most cases a clear link can be made.
page 13
Contract No. VC/2015/0098
Moreover, Member State Managing Authorities themselves have
allocated budgets to specific categories, making the reconstruction of
budgets by the evaluation team redundant.
III.
Annex XXIII data on participants are registered for every OP and
reported on Priority Axis level. Nevertheless, Priority Axes are not
always clearly linked to the Priorities in Article 3.
IV.
The 117 OPs and their respective AIRs, include Priority Axes
(first digit level), sub priorities / actions (second digit level),
interventions
(third
digit
level)
that
are
differently
operationalised across programmes. All programmes defined
programme specific indicators in relation to the Priority Axes and
sometimes actions, and also store information on the ESF budget.
Priority Axes do not always show a clear relation with Priorities or
Categories of Expenditure, and sometimes more Priorities are
addressed at the same time or, vice versa, one Priority is addressed
by more than one Priority Axis4. Working with Priority Axis data
allows us to combine financial spending figures with physical
progress indicators on OP level5. Besides, there are two crucial
issues:
V.

The titles (face value) of Priority Axes sometimes suggest a
different allocation to ESF Priorities than a careful reading of the
actual actions and interventions behind them suggests (some
headings do not accurately represent the content or are
politically driven).

Even if a Priority Axis is predominantly focused on one Priority, if
this Priority Axis is financially significant (over a billion Euro), the
other Priority that it also affects may also be highly significant in
financial terms.
Evaluations are implemented on different levels, sometimes dealing
with a Priority Axis, sometimes a specific objective, sometimes a
specific action, sometimes a concrete intervention or bundling of
projects. Evaluations can also be transversal addressing several
actions / interventions addressed by more than one Priority Axis.
All sources will be used in the synthesis while being aware of the problems.
Decisions will already have been made on how to deal with most of them in
the thematic ex post evaluations but the synthesis will have to report on
these decisions and their caveats.
4
There is a cluster of Priority Axes that have a direct link with a Priority (100% coverage); There is a cluster
of Priority Axes that is dominantly addressing one of the ESF Priorities, but at the same time includes
some actions / interventions related to another Priority (80-100 % coverage); and there is a last cluster of
Priority Axes that has an equal contribution to one or more ESF Priorities (20-80% coverage).
5 See outcomes of the Preparatory Study, based on a sample of 43 OPs, spending around 80
percent of ESF budget across Europe.
page 14
May 2015
Contract No. VC/2015/0098
1.4.3 Allocating Priority Axes to one single Priority (and sub
priorities)
The thematic ex post evaluations have allocated Priority Axes to ESF
Priorities as a basis for the evaluation synthesis. In order to avoid overlaps
in the EU synthesis report between ESF Priorities (including the same
Priority Axis) and possible double counting of budget and output/result
indicators, a clear methodology feeding the decision how to allocate Priority
Axes to ESF Priorities has been developed:
I.
Step 1: Identify the Priority Axes that show an overlap (address one
or more of the thematic ex post evaluations).
II.
Step 2: Further explore those Priority Axes that show overlap on
digit 2 level (and if necessary digit 3 level) to assess the dominant
Priority in the Priority Axis (based on budget). Guiding criteria are
how actions and interventions within this Priority Axis relate to the
demarcation of clusters of interventions as identified in each
thematic evaluation, the labelling of MA themselves, the labelling of
relevant categories of expenditures, and expert opinion assessing
the goals, target groups, and instruments of each action and
intervention.
III.
Step 3: Allocate the Priority Axis to one of the ESF Priorities. In case
it appears that Priority Axes substantially contribute to more than
one Priority a solution should be found. In order to do justice to the
actual nature of Priority Axes, the preferred option is to allocate
distinct actions (at digit 2 level) to each Priority in case financial
information and output and result data is available on this level.
The three thematic evaluations will have addressed this in such a manner
that the ex post evaluation synthesis can start with clearly demarcated
overview reports on each theme. Ideally, the evaluation synthesis team will
be able to start synthesising immediately without reallocating Priority Axes
(and actions) to the respective ESF Priorities.
1.4.4
Aggregating output and result indicators
Programme specific indicators are defined on OP level and in most cases
these are related to PA or sub priorities. OPs have defined a wide variety of
indicators measuring the progress of ESF also including different types of
measurements (such as absolute numbers, percentages, totals or additional
numbers / increasing percentage, etc). For some indicators targets are set
for, others not.
The challenge is to link these indicators to ESF Priorities (subject to the
thematic ex post evaluations) and to aggregate these on country level and
subsequently on European level for each ESF priority (but also across ESF
priorities feeding the EU synthesis). For example, in total 2,387 indicators
were found for A2E, of which 897 are labelled as output indicators, and
1,490 as result indicators. For the Human Capital theme, 955 result
indicators were found of which 820 indicators with 2007-2013 values.
May 2015
page 15
Contract No. VC/2015/0098
For the ex post synthesis evaluation, two types of analysis are foreseen on
these programme specific indicators:
1) Aggregating achievement values within and between OPs (such as
summing up total number of participants, entities supported, or
products developed)
2) Calculating (average) achievement rates within and between OPs
(total achievement in 2013 divided by the target set).
Both types of analysis pose some challenges that should be overcome in
each of the thematic evaluations. In order to ensure a common approach
across thematic evaluations, some guidelines and rules need to be drawn
up for aggregating data.
3) Aggregating achievements
In the context of the thematic evaluations as well as for the synthesis
report it would be valuable to sum up some key performance figures of the
ESF across programmes, ESF priorities, and countries. Monitoring
information that allows aggregations across OPs and countries are: financial
spending figures, Annex XXIII participant data, and (some) programme
specific output and result indicators.
I.
Financial data: Allocated and certified expenditure budgets per PA
and CoE can be summed up across PA, programmes, countries and
EU level, since they have the same unit of measurement, namely
Euro6.
II.
Annex XXIII participants data: For aggregation on outputs of
participants across PAs, OPs and countries the Annex XXIII
participant data provides a more solid basis for aggregation than the
programme specific output indicators related to individuals, since
Annex XXIII data is recorded in a systematic manner across
countries (total number of participants and for sub groups like
gender, employment situation, age, migrants/minorities, disabled,
other, and educational level7).
III.
Programme specific output and result indicators: Aggregating
achievement values of programme specific indicators can only be
done if indicators have a common basis (definition, scope and unit of
measurement). A first analysis of the programme specific indicators
shows that there is a wide variety of programme specific indicators
(having different units of measurement and different scope,
6Regarding
financial data we suggest to distinguish: (1) Allocations (OP, PA, CoE) of ESF-funding in terms
of what has been approved by the COM as EU-contribution up-front; (2) Commitments made by member
states when granting funds to beneficiaries (to be distinguished ESF money and national public); (3)
Expenditure declared (again: ESF and national public to be distinguished) – should be expenditure that is the
base for re-imbursement from the COM (while assessing the certified expenditure one should be aware of
the frontloading of ESF introduced with anti-Crisis-measures). Committed budget is not recorded in the data
collection template for task 1 (only allocated budget and certified expenditure).
7 Programmes are recording the number of entries and departures of individuals participating in ESF. In the
thematic evaluation information is collected on the number of entries (to get a more realistic picture on
programmes performance since the programme is not finalised yet, and to assure we are not missing a high
number of participants in the data set). One individual could however participate in more than one ESF
project and is therefore counted more than once. As a result Annex XXIII is reporting about the number of
participations and not the number of persons benefitting from ESF.
page 16
May 2015
Contract No. VC/2015/0098
addressing different objectives, types of interventions and target
groups). Nevertheless, there are still a number of (common)
indicators that frequently appear across and within each ESF
priority. Some of these also allow cross ESF priority aggregation
(which is interesting for the synthesis). In each of the thematic
evaluations we therefore need to carefully assess each programme
specific indicator whether these belong to common groups of
indicators, allowing aggregations of achievement values in the end.
In order to group indicators we propose to follow a deductive and
inductive approach.
a. Pre-define groups of indicators that have similar definitions,
scope and unit of measurement and allocate individual
indicators to these groups.
i. Starting point is to define whether an indicator relates
to the performance of an individual participant, entity
(such as an organisation, enterprise, network,
partnership supported / established), product (such as
a programme, project, tools, curriculum, approach
developed), or system (such as a reform
implemented).
ii. Secondly, where possible, indicators could be
classified in terms of type of intervention (does it
concern an intervention related to training, job
placement, counselling?). The definition of indicators,
however, does not often give an indication of the
types of intervention addressed. In many cases,
different types of activities referring to the same
objective are integrated in the same output indicator.
In these cases the evaluator is not able to construct
groups.
iii. Subsequently, one should assess whether an indicator
can be considered as output or result indicator. Output
indicators relate to operations supported and
measures an output which is directly
produced/supplied through the implementation of an
ESF operation, measured in physical or monetary units
(such as supported people, supported entities,
provided goods or services delivered). Result
indicators capture the expected effects on participants
or entities brought about by an operation. Result
indicators go beyond output indicators in so far as
they capture a change in the situation, in most cases
related to supported entities or participants, e.g. in
their employment situation. Programmes label the
indicators themselves as output or result indicators.
The reality is that sometimes the same indicator is
labelled as output indicator in one OP while in another
OP this indicator is labelled as result indicator
(depending on the action or the objectives of the
May 2015
page 17
Contract No. VC/2015/0098
action / programme). An example is that for Human
Capital “the number of participants gaining a
qualification or certificate” is often considered as result
indicator, while for Access to Employment this
indicator can be considered as an output indicator,
finally contributing to employment. While grouping
indicators we attempt to aggregate similar indicators
in terms of definition and level of measurement
despite whether these are labelled as output or
results. Moreover, sometimes result indicators are
actually macro-impact indicators (such as national or
regional employment rates of specific groups) and not
related to single participants. These indicators should
be considered as context indicators and will not be
further assessed in the analysis.
iv. Finally, one should assess the unit of measurement of
indicators. This mainly relates to (1) number or (2)
percentages. Where possible percentages will be
recalculated to absolute numbers (based on
information available on the total population if
available).
b. While allocating programme specific indicators to groups of
indicators, where necessary, new groups will be formed or
combined. In the first case, this means that in case we see
more indicators that have a similar definition and unit of
measurement within the pre-defined group, we decide to
define a sub group. In other cases, we bring indicators
together - being different - but having similar unit of
measurement and scope. For example, in case sustainable
employment is measured using different time periods (like 3,
6, or 12 months after the intervention), these can be grouped
together as one indicator called sustainable employment. Also
different types of entities supported (like SME, PES, NGOs)
can be regrouped in total number of organisations supported.
A suggestion is to split up this group in final beneficiaries and
intermediary organisations like PES and NGO’s.
c. Per group of indicators, where possible, values will be
summed up. In case the indicator is measuring numbers, we
are able to sum up numbers in case they concern the same
object (like individuals, entities, products). When indicators
are measuring the achievement in percentages we face some
specific challenges. Percentages are mostly used for
measuring the results of an intervention (like the percentage
of participants that successfully completed a training course,
gained a qualification or found a – sustainable – job). In
these cases it would be recommendable to calculate the
absolute number. Therefore, information is needed on the
page 18
May 2015
Contract No. VC/2015/0098
total number of participants8 that will be multiplied with the
percentage (like 30% of 1000 participants = 300). A precondition is that there is clear information on the total
population the indicator is reporting on (preferably a clear
relation between output – total number of participants – and
result). Only in this way are we able to calculate absolute
numbers. If this is not possible, another option is to calculate
average percentages between indicators (like the percentage
of participants that have a sustainable job). However, this is
not the preferred option since it does not provide robust
outcomes.
8
May 2015
The term ‘participants’ actually refers to ‘participations’ meaning number of entries into measures rather
than number of people who participated in measures. The fact that one person may well have participated
in more than one measure is taken into account.
page 19
Contract No. VC/2015/0098
Box 1. An example: Programme specific output and result indicators
in A2E
A close look at the output indicators reveals that most output indicators relate to
individuals (total 617), followed by products (176) and entities (107). Concerning the
number of output indicators across countries one see that Belgium, Germany, Italy,
Spain, and the UK already represent 69% of the total number of indicators. As these
countries also gave the largest number of OPs and the indicators are sometimes very
similar, particularly in Italy and Spain, this means a reduced overall number of indicators.
The indicators in the German, Belgium and the UK programmes are more distinct. A first
inventory of the volume and types of indicators per PA, OP and Member State shows a
quite diverse picture. Many of the indicators are unique in their own context and difficult
to aggregate. They can nevertheless be broadly grouped around some common
categories.
Concerning individuals, these would broadly correspond to the Annex XXIII data whereby
Annex XXIII data is probably the more robust being collected systematically by PA and OP
and covering all participants in ESF interventions.
Concerning entities, the categories discerned were: (1) number of enterprises created;
(2) number of enterprises supported; (3) number of networks supported; number of
centres developed; (4) number of PES supported; (5) number of organisations supported.
As indicated most indicators relate to the number of supported organisations, allowing
some form of aggregations on this very basic level. The problem, however, is that the
indicators do not have a complete coverage across PAs, OPs, and countries, making it
difficult to do comparative analysis.
Concerning products, the categories distinguished were: (1) number of projects; (2)
number of campaigns; (3) number of programmes; (4) number of trajectories; (5)
number of tools / databases; number of studies / evaluations; (5) number of agreements
/ conventions. These groups allow doing some aggregations at very basic level, however
it should be noticed that most indicators (except of the indicator on number of projects)
do not frequently appear across OPs.
In addition to the ‘content’ or definition issue, there is also an issue concerning indicator
values. Most output indicators are measured in absolute numbers; however 6.7% of all
output indicators (60 in total for A2E) use percentages as unit of measurement.
Sometimes the indicator measures the relative share of a particular target group of the
total population, and sometimes the increase in percentage compared to a certain
baseline or target.
In the second interim phase of the A2E ex post, the programme specific output indicators
will be further analysed, grouped, and assessed on their unit of measurement.
A large number of programme specific result indicators can also be found (in total 1,490
indicators) for the selected PA relevant for A2E. Of these 1,490 indicators, the majority
relate to individuals (1,337), a smaller number relate to entities (total 86) and products
(64) and very few to systems (3). Assessing the number of indicators across the different
MS, one sees that also here Italy, Spain, and the UK represent most of the indicators
page 20
May 2015
Contract No. VC/2015/0098
(68% of total indicators), mainly linked to the large number of OPs. Italy represents in
total 42% of the indicators. Again, in Italy, Spain and the UK, the indicators show little
variation between OPs. The large amount of indicators is in these cases also explained by
the fact that the same indicators are used to measure the results for all target groups
(men, women, young people, older people, migrants etc.). Another conclusion to be
drawn is that most OPs report on the number of job entries and do not provide evidence
on job sustainability. These would be the two most interesting indicators to observe in
this field in the synthesis and further research will be done in the next interim phase of
the A2E ex post.
(2) Calculating achievement rates across PA, programmes, and countries
For those indicators that have a target, achievement ratios can be
calculated. This can be done in two ways:
a. If targets are set for absolute numbers, these can be
compared with the achievement values (target /
achievement)
b. If targets are set for percentages, the achievement rate can
be calculated in two ways. The first option is to calculate the
relative share of the achievement compared to target (when
the target is 20% and the achievement is 10% we calculate
the achievement rate 10/20 = 50%). The second option is to
calculate the difference between target and achievement in
absolute terms (when the target is 20% and the achievement
is 10%, we can calculate the distance to target, by extracting
the target from achievement: 10-20=-10%). Since the first
option allows a better analysis between indicators, this is the
preferred type of analysis.
Subsequently, we can calculate the average achievement across indicators
for each PA, OP, and groups of programme specific indicators on EU level.
This provides input for making statements such as: 40% of indicators
across programmes achieved their target, or 60% of output indicators
related to individuals achieved their target. However, it is important to
mention that in some cases it is questionable whether targets set are
realistic. Indicators that show a high over- or underperformance therefore
need special treatment
It is being discussed with the teams responsible for the three thematic
evaluations, whether indicators with very high or low achievement values
have been excluded from the analyses. It is advisable to leave out those
indicators showing an over performance of 300% or those that have an
underperformance of 10%, while calculating average achievement
percentages across indicators. The parameters for excluding indicators is
being discussed and decided between the responsible evaluation teams.
May 2015
page 21
Contract No. VC/2015/0098
A critical note related to the target values set by programmes is that many
programmes have adjusted their targets over the programming period in
order to better align them to the actual progress made in the programme,
while other programmes kept the targets set from the start of the
programme. As a result, one could question whether measuring the
achievement rate is meaningful. With proper adjustments, the moving
target can be reached in the end and a potential failure to meet targets be
disguised.
Problem of exclusion of single indicators that do not have a value or
cannot be grouped
Some indicators cannot be included in a further analysis on country and EU
level due to the following reasons:
 Indicators that do not report an achievement value for the period
2007-20139.
 Indicators that are unique for certain programmes and not appear
frequently – or not at all – in other OPs and cannot be grouped
(such as the indicators measuring the yearly reduction of 5% of the
cost of an intervention, as included in the ESF programme for the
German Speaking Community in Belgium).
Each thematic evaluation should report for which indicators one of the
above described situations apply.
Concluding remarks
Based on the experience with the A2E ex post and the anticipated
experience of the other ex post evaluations, it is expected that a grouping
of common or similar programme specific indicators will be possible with
some caveats and compromises. Nevertheless, given the variety described
above, full aggregation and coverage will not be possible since ‘unique’
indicators not belong to a group should be excluded from the analysis.
Reviewing the indicators, one can conclude that the financial data and
Annex XXIII data comprise the best information base for providing an EU
wide overview of the output of ESF. The programme specific indicators only
represent a part of ESF spending. The output and result indicators for
individuals show the most commonalities, followed by the indicators related
to entities. Those formulated for products are very diverse and difficult to
aggregate. As a result, we also need to indicate that the programme
specific indicators are reporting on the minimum contribution of ESF. We do
not propose to extrapolate achievement values of programme specific result
indicators to the whole of ESF. For example sustainable employment
indicators for A2E are only measured in 16 countries (and most cases are
related to a specific PA or sub priority). As a result we propose to only
report on these indicators for the 16 countries (and will not extrapolate
these figures to the other 11 countries). The evaluators should always be
9
page 22
There are indicators that only have a value in 2014 (or even later) because the value should be produced
by a separate evaluation / study at the end of the programme.
May 2015
Contract No. VC/2015/0098
clear about the coverage of the indicators they are reporting on, avoiding
false interpretation by the reader of the final evaluation reports.
May 2015
page 23
Contract No. VC/2015/0098
2. METHODOLOGY AND TOOLS
2.1 Main tasks and deliverables
The ex post synthesis evaluation comprises three main tasks and a number
of sub-tasks. The deliverables and their interlinkage are shown in the figure
below.
Figure 1. Tasks and deliverables of the ex post evaluation synthesis
Task 2. National level analysis
Task 1:
Follow-up of
ex-post
evalu-ations
27+1 – updated
Inventories
27+1 – Country
Reports
Task 3. EU level analysis
2 Thematic reports
(partnership/cap)
EU Synthesis Report
As discussed above, the ex post will pull together the material that has
been produced so far and complement the existing information with
additional research. The main methods that will be used will be desk
research, comparative research, completing the database with statistical
information coming from Eurostat, the SFC2007 and national sources, some
complementary interviews and a focus group with high level experts and
Commission officials to validate the findings.
The main tools will be existing material coming from the three thematic ESF
ex post evaluations and the Ex Post Evaluation of Cohesion Policy
Programme – Work Package 12, the ESF Expert Evaluation Network, the
Preparatory Study, EU-wide evaluations (e.g. on gender or on specific
target groups, in particular young people and older people), newer
evaluations carried out by the Managing Authorities, OPs and implementing
documents, especially in Croatia, AIRs 2014, SFC2007, Eurostat, national
databases etc.
0Table 2 below summarises the main tasks, data sources and outputs
required for the fulfilment of the contract. The following chapter describes
how we intend to fulfil the individual tasks.
page 24
May 2015
Contract No. VC/2015/0098
Table 2. Overview of tasks, data sources and outputs
Tasks
Data sources
Output
Task 1: Follow-up of the ESF 2007-2013 thematic ex post evaluation studies
1: Follow-up of
the ESF 20072013 thematic ex
post
evaluation
studies
Three thematic ex post evaluations
Ex post evaluation ERDF: Work package 12
Minutes steering groups / reports
Attending steering group meetings of the four Ex Post Evaluations
(including Work Package 12 of the DG REGIO Ex Post Evaluation)
providing feedback and advice
Minutes
Task 2: National level analysis
2
A:
National
inventories
of
evaluation
Previous inventory by the EEN as a basis
ESF evaluation reports produced by MA after
December 2013 till 30 June 2015
All evaluation reports produced in Croatia
28 National Inventories
Overview Inventory of all Member States
2 B1: Country
synthesis reports
27 filled-in overview country templates of three
ESF 2007-2013 Ex post thematic evaluation
studies
Country in-depth assessment reports from the
relevant ESF 2007-2013 Ex post thematic
evaluation studies
Available country level deliverables of the 'Ex
post' evaluation of cohesion policy programmes
2007–2013 financed by the ERDF and CF — work
package 12: delivery systems
Annual Implementation Reports 2013 and 201410
SFC2007, in particular Annex XXIII and
programme-specific indicators
Operational Programme documents
28 Country synthesis reports structured by ESF priority including:
information on OP objectives and types of interventions
contextualisation of the above vis-à-vis national situation and
strategies and EU-level strategies
overview of expenditure, outputs and results achieved,
findings on all evaluation criteria with a particular focus on
effectiveness, based on evaluation evidence
conclusions for the policy on economic and social cohesion at
Member State level
outline lessons learned in relation to policy choices, target groups,
programming, implementation, monitoring and evaluation
at least one good practice per Member State according to robust
evidence-based criteria, from the list of good practices already
selected in the ex-post evaluations, or new good practices
10
May 2015
For the themes covered by the three thematic ex post evaluations, only the AIRs 2014 will need to be consulted to update information. For the themes PP and SIC,
the earlier AIRs, 2013 in particular, will need to be used.
page 25
Contract No. VC/2015/0098
Tasks
Data sources
Evaluation reports (till 30 June 2015)
All Expert Evaluation Network country reports
Relevant country-level deliverables of the
Preparatory study for the ex post evaluation of
ESF 2007-2013
Evaluations available up until 2015
Output
researched for the other priorities
The country synthesis reports will also include additional data on
Promoting Partnerships and Strengthening Institutional Capacity.
The country synthesis report for Croatia will also be included
based on the OP, AIR 2014 and any available implementation
information, contextual data (e.g. from Eurostat and national
statistics office) and evaluations.
country synthesis reports will be a minimum of 15 and an absolute
maximum of 30 pages long, including tables and a one-page
executive summary.
3A: Thematic EU
synthesis reports
on
promoting
partnerships and
strengthening
institutional
capacity
Country synthesis reports
EU wide data (Annex XXIII and CoE)
Context data (Eurostat)
Employment and Social Developments in Europe
Fourth report on Economic and Social Cohesion
Fifth Report on Economic, Social and Territorial
Cohesion
Sixth Report on Economic, Social and Territorial
Cohesion
One thematic synthesis report on Promoting Partnerships
One thematic synthesis report on Strengthening Institutional
Capacity
Each of maximum 50 pages
3B: EU Synthesis
report
Country synthesis reports
Overview assessment for 27 Member States from
the relevant ESF 2007-2013 Ex post thematic
evaluation studies
Thematic EU synthesis reports on promoting
partnerships and strengthening institutional
capacity
In-depth horizontal assessment of interventions
in Member States selected for the relevant ESF
One synthesis report structured along ESF Priorities which:
provides information on objectives established in the Operational
Programmes and types of interventions funded, together with a
contextualisation of this information vis-à-vis national situations
and strategies and EU-level strategies
provides an overview in terms of expenditure, and as regards the
outputs achieved, based on the most important data collected
through Annex XXIII and programme-specific indicators
presents findings on all evaluation criteria, with a particular focus
Task 3: EU level analysis
page 26
May 2015
Contract No. VC/2015/0098
Tasks
May 2015
Data sources
ex post thematic evaluation studies
Available deliverables of the 'Ex post' evaluation
of cohesion policy programmes 2007–2013
financed by the ERDF and CF — work package
12: delivery system
Expert Evaluation Network synthesis reports
Preparatory study for the ex post evaluation of
ESF 2007-2013
Other relevant evaluations of the ESF 2007-2013
carried out by the Commission
Output
on effectiveness, based on evaluation evidence
presents conclusions for the policy on economic and social
cohesion at EU level
outlines lessons learned in relation to policy choices, target
groups, programming, implementation, monitoring and evaluation
The EU synthesis report will be 50 pages long, including tables and
a three page executive summary.
page 27
Contract No. VC/2015/0098
2.2 Evaluation task 1: Follow up of ESF thematic ex-post
evaluations
Given the request to base the Ex Post Evaluation of Access and Sustainable
Integration into Employment (being carried out by Metis) on the other two
thematic ex post evaluations (being carried out by ICF), the Consortium has
a good overview of what is being done in all three thematic ESF evaluations
and is in good contact with the other two evaluation teams. The existing
good cooperation base will be used to accomplish Task 1 of the ex post
synthesis as efficiently and effectively as possible. A maximum effort is
being made to align the three ex post evaluations and this effort will be
maintained throughout the existing contracts in order to develop a common
methodological approach and to deliver comparable final reports for the ex
post synthesis evaluation.
In the thematic ex post evaluations, a common approach was used in the
selection of OPs in Member States with a large number of OPs for the indepth assessments, selection of interventions for in-depth study,
presentation of findings (report templates, tables, maps, definitions). For
the overview (Task 1), a common approach was also developed in order to
have comparable data from the three ex posts with a clear demarcation
between ESF Priorities. This will be extremely useful when it comes to
writing the synthesis report as important decisions on how to allocate OP
Priority Axes (and if necessary sub-priorities or actions) to ESF Priorities
have already been taken by the evaluation teams and agreed by the
Commission. A huge effort was made to avoid overlap in Task 1 but in a
limited number of cases it proved impossible to allocate the whole PA to one
Priority (e.g. in FI, LT, LU, SE). In these few cases, a special mention will made
in the Synthesis that the PAs have been allocated to one PA to avoid double
counting but that a significant amount of funding according to CoE has gone
to another Priority (in most cases from A2E to SI).
It is also expected that the presentation of the conclusions and lessons
learnt in relation to policy choices, target groups (in particular young
people), programming, implementation, monitoring and evaluation will be
aligned in all three thematic ex posts.
The contractors will familiarise themselves with the ex post evaluation of
Cohesion Policy Programmes 2007–2013 financed by the European Regional
Development Fund (ERDF) and Cohesion Fund (CF) — work package 12:
delivery system" (DG REGIO) published in June 2014. This ex-post
evaluation is concerned with the effectiveness and efficiency of the delivery
system of cohesion policy for the 2007-2013 period and will identify the
main success factors that support implementation of all funds. This
evaluation will provide useful information for the thematic report on
institutional capacity.
The approach to Task 1 of the ex post synthesis evaluation is based on the
existing good cooperation between the evaluation teams. Building on the
existing cooperation the project manager will:
page 28
May 2015
Contract No. VC/2015/0098
 (Continue to) familiarise herself with all the reports published under
the four ex post evaluations (including the DG REGIO ex post
evaluation, work package 12)
 (Remain in) contact with the evaluation teams concerning
methodology and deliverables with the aim of making these as
complementary and comparable as possible
 Ensure that overlaps and double counting have been dealt with
satisfactorily and agreed by the Commission, including for the
Priorities Promoting Partnerships and Strengthening Administrative
Capacity.
 Ensure that definitions, terms, funding categories used etc are
harmonised
 Participate in steering committee meetings as requested, e.g. on 21st
April 2015
 Cooperate
in
the
methodologies/deliverables
development
of
comparable
 Provide advice when appropriate and requested
 Encourage other evaluation teams to build a database in which
information is merged on Priority Axis of every programme, including
information on the ESF Priority it addresses, and basic information on
budget spent and output and result indicators in order to be able to
combine the databases for the ex post synthesis evaluation. At the
very least, an excel sheet should be produced collating all the
financial, output and result data for the EU27 – not only single country
fiches.
Preparation, setting up a database
In order to produce EU wide information we are dependent on the
consistency in the way data is collected in the separate thematic ex post
evaluations (and the thematic EU synthesis reports on Promoting
Partnerships, and Strengthening Institutional Capacity). In order to be able
to start from a structured information base, it is recommended that every
thematic evaluation produce a database (in Excel) with basic information
per relevant Priority Axis that is labelled to a single ESF Priority, including
information on financial allocations and expenditures, participant data, and
programme specific indicators.
In order to streamline the approach between the thematic evaluations two
notes were produced in the inception phase and shared with the three
evaluation teams and the European Commission. One note focused on
issues surrounding the aggregation of programme specific indicators (which
has been discussed in the previous section). Another note focused on the
construction and handing over of the database with information on financial
data and programme specific indicators. Several discussions have taken
place with the evaluation teams responsible for (1) the human capital
evaluation and (2) Access to Employment evaluation during the inception
phase while the discussion with the team responsible for the thematic
evaluation on Social Inclusion still needs to be had. The box below includes
May 2015
page 29
Contract No. VC/2015/0098
some important points that should be taken into account while producing
and handling over the database for each thematic evaluation.
Box 2. Discussion points /guidelines on the construction and
handing over of the the database
Discussion note on the construction and handling over of the
databases
Since the synthesis of the ex post further elaborates (and update) some
of the products delivered in each of the thematic evaluations, some
guidelines should be formulated how each thematic evaluation handles
over the data(bases) produced and analysed. This is important since the
evaluation team responsible for the “synthesis report” needs to update
the financial, Annex XIII data and programme specific indicators with
2014 values (AIR2014). Moreover, based on the AIR2014, the analysis
should be updated and reconstructed11. This has some consequences
for:
1. Set up of the database with information on financial and physical
data. The databases of every thematic evaluation should have a
similar structure (including similar variables) so easy to merge
into one database.
2. Transparency on the analysis done aggregating programme
specific indicators by means of log books / syntaxes (including the
link of individual programme specific indicators to groups of
indicators), Annex XXIII data and financial data. This information
is needed to reproduce the analysis done, including the most
recent AIR2014 data.
Evaluation teams should share their final databases and inform
the “synthesis team” how they dealt with the construction of (1)
the database (and relevant variables recorded) and (2) analysis
done aggregating data. Each aspect is further discussed below.
(1) Database structure:
In total every thematic evaluation produce four databases that can be
linked with a common identifier (CCI and PA). Most of the information
captured in the database is extracted from SFC2007 (information on PA
level). Especially in case information is reported on sub priority level (not
available in SFC2007) it should be completely transparent where financial
and physical information is stored in the AIR and whether additional
calculations were carried out by the thematic evaluators to construct a
value on financial and physical performance (in case information on sub
priority level is not available). The evaluation teams should provide log
books when they allocate budget and number of participants to a sub
11
page 30
We assume that the programme specific indicators for each OP have not changed dramatically since the
AIR2013.
May 2015
Contract No. VC/2015/0098
priority level based on some assumptions. As a general rule it was
decided that the EU27 templates only zoom in on the sub priority level
when financial information, Annex XXIII data, and programme specific
indicators are available in the AIR on this level. If this was not the case,
the whole PA should be allocated to one ESF priority.
At least the following information should be available in each
database that should be carefully checked within each evaluation
team (during / after the meeting)12.
A.














Database on programme specific indicators
Each record is an indicator
Country name
CCI
OP name
OP objective (convergence / competitiveness/multiple objective)
PA name and number
ESF priority (A2E, HC, ADAPT, SI)
Indicator name – original
Indicator name – translated in English
Unit of measurement (number / percentage)
Initial baseline
Final target
2007, 2008, 2009 …. 2013 Achievement
Total 2007-2013 achievement (already available in 2013
achievement number or reproduced by evaluation team)
 Achievement rate (target / final achievement 2013)
 Allocation to grouping of indicators (in case of aggregating
programme specific indicators)
B.











12
May 2015
Database on Annex XXIII data
Each record is a PA
Country name
CCI
OP name
OP objective (convergence / competitiveness/multiple objective) √
PA name and number
ESF priority (A2E, HC, ADAPT, SI)
Total number of participants
Male
Female
Employed
The assumption is that the databases show a lot of commonalities since these are based on the same
EU27 template.
page 31
Contract No. VC/2015/0098



























of which self-employed
Unemployed
of which LTU
Inactive
of which in education/training
Young people (15-24 years)
Older people (55-64 years)
Migrants
Minorities
Disabled
Others
Primary or lower secondary education (ISDEC 1 and 2)
Upper secondary education (ISCED 3)
Post-secondary non tertiary education (ISCED 4)
Tertiary education (ISCED 5 and 6)
Database with financial data per PA
Each record is a PA
Country name
CCI
OP name
OP objective (convergence / competitiveness/multiple objective)
PA name and number
ESF priority (A2E, HC, ADAPT, SI)
Total funding of the OP (Union and national)
Total amount of certified eligible expenditure paid by beneficiaries
Corresponding public contribution
Implementation rate (%)
C.






Database with financial data on CoE per OP
Each record is an OP
Country name
CCI
OP name
OP objective (convergence / competitiveness/multiple objective)
Total funding of the OP (Union and national) per CoE (allocated
budget)
(2) Transparency analysis performed aggregating financial and
physical data
In order to repeat the analysis done in each of the thematic evaluations,
each thematic evaluation should make transparent how they aggregated
page 32
May 2015
Contract No. VC/2015/0098
financial, Annex XXIII and programme specific indicators, by producing:




A log book on how this analysis is done and assumptions behind
o Which indicators are included / excluded in the analysis
o Special treatment of percentage values
o How to deal with indicators that have no target value
o How to deal with indicators that have no value for 2013, or no
value at all
o How to deal with calculating achievement rates (target /
achievement value)
o Additional calculations done on total achievement 2007-2013
(already included in 2013 achievement value or not)
o On what level aggregations are done (OP, country, EU level)
Syntaxes for the analysis performed in Social Statistical Software
Formulas in Excel for calculating totals
How individual indicators are allocated to groups of indicators
Each evaluation team should share this information with the
Commission and the “synthesis team”.
Answers on the above mentioned issues allows the evaluator - responsible
for the EU synthesis report - to merge databases, and to further work with
one database for further analysis feeding the EU synthesis report as well as
the country synthesis reports (see Table 3 below). This database will be
constructed during Task 1 but will be completed with updated information
from the AIR 2014 during the country synthesis work (see Task 2).
May 2015
page 33
Contract No. VC/2015/0098
Table 3. Database structure for the thematic ex post evaluations
and EU synthesis report
MS # # # # # # # # # # # # # # # # Total number of participants # # Male # # Female # # Employed # # of which self‐employed # # Unemployed # # of which LTU # # Inactive # # of which in education/training # # Young people (15‐24 years) # # Older people (55‐64 years) # # Migrants # # Minorities # # Disabled # # Others # # CCI Programme name Priority Axis Link PA to ESF Priority AIR version Background information Years covered Allocated budget Certified expenditure Budget Annex XXIII data page 34
May 2015
Contract No. VC/2015/0098
Primary or lower secondary education (ISCED 1 and 2) # # Upper secondary education (ISCED 3) # # Post‐secondary non tertiary education (ISCED 4) # # Tertiary education (ISCED 5 and 6) # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # Indicator title and definition Indicator title and definition – English Type I (output / result) Type II (individual, entity, product, system) Type III (type of group) Unit of measurement Priority Initial Baseline Final Target Achievement 2007 Achievement 2008 Achievement 2009 Achievement 2010 Achievement 2011 Achievement 2012 Achievement 2013 Achievement 2014 Achievement 2007‐2014 Achievement rate % This database will be used for updating some key figures from the EU27
analysis based on the AIR2014, mainly related to certified expenditures,
participant data (Annex XXIII), and 2014 values of programme specific
indicators. Based on this updated data some calculations are reproduced
within task 2 (country level) and task 3 (on EU level). The difference is that
May 2015
page 35
Contract No. VC/2015/0098
task 2 intends to analyse and aggregate data on national level across OP
and ESF priorities (and find common indicators on priority themes and
national level), while task 3 tries to aggregate data on EU level (and find
common indicators on priority themes and EU level).
Table 4. Overview of steps for the accomplishment of Task 1
Task
Who
Task 1: General overview
Step 1: Familiarisation with all the reports published under Project manager
the four ex post evaluations (including the DG REGIO ex post
evaluation, work package 12)
Step 2: Contact with and advice to the evaluation teams Project manager +
concerning methodology and deliverables
Bert-Jan Buiskool
Step 3: Participate in steering committee meetings as Project manager
requested
2.3 Evaluation task 2: National level analysis
2.3.1 National inventories
In the framework of the ESF EEN, 27 national inventories of evaluations
were created as Excel files. The aim of the inventories was to record in very
succinct form the contents of the ESF evaluations that had been produced
in the period, in particular with a view to filtering out the findings of the
evaluations according to ESF Priority. The inventories consist of a number of
sheets which present the evaluation plans, the evaluations, the evaluation
documents, the evaluation approach, evaluation method, the main findings,
and the main findings according to intervention types and target groups
(see below). All the inputs can be filtered according to the subject of
interest, e.g. counterfactual evaluations or evaluations concerning women
or young people, older workers etc.
Objectives
The main objective of this task (2A) is to update the existing inventories
with new evaluations bearing in mind that the last update was in October
2013. A new inventory will be created for Croatia. The overall aim is to
have a complete overview of the evaluation activity and the main findings
of the evaluations in relation to interventions and target groups in the
programming period. The tool is useful as a basis for the Evaluation
Synthesis in that the evaluation findings will provide additional and specific
evaluation evidence under the priorities which may not have been included
in the thematic ex post evaluations. The findings are categorised according
to Priority (Policy Field) and therefore easily allocated to the relevant
chapter of the country reports.
The tool is also useful for policy makers and researchers and any interested
stakeholders who would like to know more about what the ESF achieved
and how specific instruments worked.
page 36
May 2015
Contract No. VC/2015/0098
Data and information sources
For the update of the 27 existing inventories and the new inventory for
Croatia, the main sources of information will be the internet (the web pages
of the main stakeholders in the ESF: Managing Authorities, Public
Employment Service, national and regional authorities, EU and international
(e.g. ILO, OECD) sources). In addition, it may be necessary to contact the
ESF Managing Authorities to ask specifically about recently published
evaluations or even those in planning (as the Inventory also allows for nonfinalised evaluations). The geographic units of DG EMPL also be asked to
provide any evaluations they have been provided with or are aware of. The
cut-off date for the evaluations to be included will be June 2015.
Tools
The basic structure of the inventories is shown in the Chapter 3, Table 13.
The structure of the Inventory is a series of sheets covering: the evaluation
plans, the evaluations, the evaluation documents, the evaluation approach,
evaluation method, the main findings, and the main findings according to
intervention types and target groups. Each sheet has a number of columns
with information pertaining to the evaluation.
Activities and team members
The updating of the inventory will be coordinated by the project manager
with the support of Benno Savioli (research team). He originally created the
inventory in the framework of the ESF Expert Evaluation Network and will
be responsible for ensuring its smooth functioning and for aggregations of
findings for the country and final synthesis reports.
The country experts will be responsible for the actual filling in of the
inventory, i.e. for the identification and screening of the evaluations and for
the categorisation of the technical information and findings of the
evaluations. A guidance note will be provided to the experts for doing this.
The quality of the findings and the correct and complete categorisation will
be checked by the overview team.
Table 5. Overview of steps for the accomplishment of Task 2A
Task
Who
Task 2: National level analysis (Lead: Benno Savioli)
A: National inventories of evaluations
May 2015
Step 1: Preparation of Guidance for country experts
Benno Savioli
Step 2: Identification and reading of the evaluations by
country experts
Country experts
Step 3: Categorisation of the technical information and
findings of the evaluations
Country experts
Step 4: Quality check of the findings and correct
categorisation
Benno Savioli (with
support by Metis)
page 37
Contract No. VC/2015/0098
2.3.2 Country synthesis reports
Objectives
The country synthesis reports requested under Task 2B have the aim of
summarising the most recent and most robust evaluation evidence available
and providing a succinct overview of the outputs, results and impacts of the
ESF in each of the 28 Member States in relation to target groups and types
of intervention and the evaluation findings identified in the available
material. The most reliable information will be used – not a summary of all
evaluations available. Older sources such as the EEN will only be used when
information gaps need to be filled. The country reports include sections on
the evaluation criteria in which the achievements per ESF Priority are
analysed according to:
 Extent to which resources were used
 Effectiveness
 Efficiency/cost-effectiveness
 Socio-economic impact
 CAV
 Gender sensitivity
 Sustainability
Particular attention will be paid to the issue of effectiveness and information
will be synthesised from the in-depth studies (in the framework of the
thematic ex post evaluations) which have the aim of providing insights into
what worked and how.
Also important is the issue of gender sensitivity. This theme is explicitly
covered in the three thematic ex post evaluations and will also feature in
the country synthesis reports as one of the evaluation criteria covered in all
the packages of the ex post synthesis. Our understanding of gender
sensitivity is that it covers both the aspects of gender mainstreaming (equal
access and participation in interventions) and specific measures for women
and also gender budgeting, i.e. ensuring that equal amounts of funding
benefit men and women. Furthermore, the relative effectiveness of ESF
supported interventions for men and women will be looked at. This means
that gender balance will be looked at in terms of participation, budgets and
effectiveness across all of the ESF supported interventions in the country
synthesis reports.
An issue that will be of importance in the country and thematic studies on
Institutional Capacity in particular will be that of sustainability.
Sustainability will be investigated at the level of individuals and
systems/structures/products and in terms of the sustainability of the
interventions themselves in the absence of ESF funding.
The conclusions of the country synthesis reports will cover all the evaluation
criteria and the lessons learned will focus on the policy choices, target
groups, programming, implementation, monitoring and evaluation to draw
page 38
May 2015
Contract No. VC/2015/0098
lessons for the future. The focus will be particularly on specific target
groups such as young people, older workers, women and types of
interventions that have proven effective for these specific target groups.
An example of good practice will be included in each of the country
synthesis reports. It will make sense to use those examples that have
already been given in the thematic ex post evaluation in-depth country
studies as these will have been well-researched and can be relied upon to
be based on sound evidence. Care will be taken to take a diverse range of
Good Practice across ESF Priorities.
In order to select good practices we will isolate the most robust examples of
practices identified in the previous thematic ex-post evaluations which will
also mean researching additional good practice in the fields of Promoting
Partnership and Strengthening Institutional Capacity.
Data and information sources
The main challenge in this task will be to synthesise the information coming
from at least six country reports of one kind or another per country that will
have been produced in the framework of the various evaluation contracts
tendered out by the Commission. There is at least one per country
emerging from one of the three thematic ex post evaluations plus one per
country at EU27 level. Then there are at least one per country from the DG
REGIO study and at least three per country that are of relevance from the
ESF EEN (Access to Employment, Social Inclusion and the Final Country
Report as well as the country reports on women and young people). In
addition to the country reports, there are other sources of information that
it will be essential to draw on: the AIRs for 2014 and the newer evaluations
as captured in the inventories (Task 2A) as well as OPs and implementing
documents in cases where information is missing on activities (e.g. for
partnership and institutional capacity and for Croatia) (see also Table 2).
The geographic desks will also be consulted in the case of missing
information or to check if information is – to the best of their knowledge –
correct.
The information gathered will be summarised into a maximum of 30 pages
which will involve an extremely economical use of tables and a careful
visualisation in order to ensure that all the information is succinctly and
comprehensively reported.
Table of contents
The table of contents will broadly reflect the structure of the both the indepth country reports and the EU 27 overview templates which will be used
in the three thematic ex post evaluations The information will already be
structured as it is needed for the country reports in most cases (with the
exception of Croatia and for the two themes that have so far not been
treated). This will be particularly useful in those countries with many OPs as
country level aggregation will have taken place for the ESF Priorities already
covered in those countries where it is reasonably easy (Italy and Spain) and
solutions proposed in those countries where it is not (Germany). Given the
May 2015
page 39
Contract No. VC/2015/0098
shortness of the country synthesis reports, the information will have to be
condensed in those countries with many OPs although these will no doubt
be the longer country reports in the end. It is important that the detail is
not lost in the aggregations and this may lead to longer explanations in e.g.
Germany or countries in which the demarcation is particularly difficult on
account of the coverage of the Priority Axes such as Finland.
Table 6. Table of contents for country synthesis reports
Chapter/secti
on
page 40
Sub-chapter
Source
information
of
Pages
Executive
summary
1
Introduction
1
1.
Context,
general
policy
background
and ESF
investment
in all ESF
Priorities
1.1 Economic and social
challenges in the
Member State
assessing Europe
2020 indicators
(such as GDP
development,
employment and
unemployment
figures, education,
lifelong learning,
poverty index etc.)
and strategic
response
1.2 Investment
priorities in the MS
(in relation to
challenges and EU
level strategies)
Summary of main types
of interventions funded
across ESF Priorities
and main target groups
EU
27
country
templates from the 3
thematic ex posts
General
overviews
from the 3 thematic
ex posts
2
2
Financial
volumes,
outputs,
results and
findings by
ESF Priority
2.1 Increasing
adaptability and
Human Capital
ESF HC ex post (EU27
template and in-depth
reports and horizontal
analyses for specific
countries)
3–4
- Financial
absorption
(SFC2007
and
AIR 2014)
- Outputs
(SFC2007
and
AIR 2014)
- Results
(Programme
specific indicators
AIR 2014
ESF EEN final country
report
Relevant evaluations
May 2015
Contract No. VC/2015/0098
Chapter/secti
on
Sub-chapter
-
-
3
May 2015
Conclusions
Source
information
of
Pages
and other results
gathered in the
framework of the
thematic ex post
evaluation)
Extent to which
resources
were
used
Effectiveness
Efficiency/costeffectiveness
Socio-economic
impact
CAV
Gender
sensitivity
Sustainability
2.2 Enhancing
access
and
sustainable
integration
into
employment
- See 2.1
ESF A2E ex post
(EU27 template and
in-depth reports and
horizontal
analyses
for specific countries)
AIR 2014, SFC2007
A2E database
ESF EEN final country
report
Relevant evaluations
3–4
2.3 Reinforcing
inclusion
- See 2.1
ESF
Disadvantaged
Groups ex post (EU27
template and in-depth
reports and horizontal
analyses for specific
countries)
AIR 2014, SFC2007
ESF EEN final country
report
Relevant evaluations
3–4
2.4 Promoting
partnerships
- See 2.1
OPs, AIRs, SFC2007
ESF EEN final country
report
Relevant evaluations
2
2.5 Strengthening
institutional
capacity
- See 2.1
OPs, AIRs, SFC2007
ESF EEN final country
report
Relevant evaluations
2
In relation to resources
EU
3
social
27
country
page 41
Contract No. VC/2015/0098
Chapter/secti
on
Sub-chapter
used, effectiveness,
efficiency, CAV,
sustainability and
gender sensitivity and
socio economic impact
Source
of
information
templates from the 3
ex posts
In-depth reports from
the 3 ex posts
Pages
4
Lessons
learned
In terms of: policy
choices, target groups,
programming,
implementation,
monitoring and
evaluation
EU
27
country
templates from the 3
ex posts
In-depth reports and
horizontal
analyses
from the 3 ex posts
2
5
Good
practice
One evidence
example
In-depth and reports
and
horizontal
analyses from the 3
ex posts
2
based
Tools
The main tools to be used in the country synthesis reports will be the
reports listed in the ToR and above (under data sources) and the database
produced for Task 1 (at least under the ESF Access and Sustainable
Integration into Employment Ex Post Evaluation13).
Activities and team members
The country reports will be drafted on the basis of a thorough reading of the
existing material and drafting according to the above template. The basic
information for the country reports will be pulled together and updated
(with AIR 2014) by the overview team. This is exacting work and needs to
be done very carefully, especially in the case of constructed indicators. The
process is described in Box 3 below.
13 It
is recommended by the Access to Employment evaluation team that similar databases be
set up in the other two ex post evaluations.
page 42
May 2015
Contract No. VC/2015/0098
Box 3. Updating financial and physical performance data with AIR
2014
The 3 thematic evaluations include information on the financial and physical
progress for each OP and relevant PA from the AIR2013. This includes
information on the:
Allocated budget and certified expenditure per PA on 31-12-2013
Annex XXIII data per PA on 31-12-2013 (number of entries)
Programme Specific Indicators per PA (target and 2013 realisation)
This information will be updated in task 2 with the information included in
the AIR of 2014 (dated on 31-12-2014). This information is generally
available in the SFC2007. Only in those cases where PA are split between
different ESF Priorities is information gathered on sub-priority level. Since
this information is not available in SFC2007, the AIR should be used to find
more detailed financial and physical information on the level.
For every country, an Excel sheet will be developed that follows the
structure of the database as presented in task 3B
(including three
worksheets, one on financial data, one for Annex XXIII data, and one on
programme specific indicators per OP/PA). These worksheets are already
filled in with information collected during the thematic evaluations as much
as possible with an extra column in which the values for 31 December 2014
will be reported. These country sheets will be integrated in one database
with all indicators per ESF priority, providing input for the EU synthesis
report.
For the country reports, only the basic information will be reported on the
financial and physical progress of the programme. This includes updated
information of Section 4.4 of the EU27 template providing answers to the
question on which have been the key positive achievements and underachievements in the country in terms of the outputs, results and impacts
for ESF investment. The tables with programme specific indicators will be
included in the Annex.
For updating and aggregating the programme specific indicators with 2014
data some necessary steps need to be undertaken in each of the thematic
evaluations. In case individual indicators are aggregated on country level
(by for example summing up similar indicators between OPs) it should be
clear and transparent how this was done. Each thematic evaluation should
provide details on which indicators are grouped / grouped together and
explain the origin of every indicator, e.g. providing source and page number
if using the AIR 2013. If this is not done, the evaluator responsible for the
“synthesis” is not going to be able to recalculate the output and result of
the programme on the basis of the 2014 data. Therefore, each programme
specific indicator should be labelled and attributed to a group and the
formulas calculating aggregated values should be transparent and
reproducible.
May 2015
page 43
Contract No. VC/2015/0098
This is something that will have to be very carefully followed up in Task 1 of
the ex post synthesis to ensure that the data are transparent in all three
thematic ex posts.
The recent information included in the 27 country reports of the thematic ex
post evaluations will be a major source of information for the country
synthesis reports. The level of depth of information differs per country since
for every thematic ex post evaluation an in-depth assessment is made in
only a limited number of countries (8-12 countries) on that particular
theme. However, each country is covered by at least one in-depth study
and by all five thematic synthesis reports. Information will be taken from
the three thematic ex post evaluations and complemented by additional
information from the AIRS 2014, the final ESF Expert Evaluation Network
reports and recent domestic evaluations, in particular for the ESF Priorities
not covered by the three thematic ex post evaluations. With the aim of
ensuring a similar approach and quality across the 28 country reports, a
maximum amount of compilation will be done in-house (by Metis) with the
aim of producing reasonably complete basic country reports.
The national experts will be consulted for those sections of the reports
which cannot be done by the overview team. They will be asked to
contribute new findings (identified during the updating of the inventories
which will be one of the first tasks to be undertaken during the ex post
synthesis), provide analyses of the information compiled, draw conclusions
and identify lessons learned. They will also be expected to verify the good
practice or suggest other examples for the three ex-post evaluations.
The country research will be preceded by a pilot study (Pilot country
synthesis report) on the basis of AIR 2013 as this will be available at the
time of commencement of the contract. If all ESF Priorities are to be
covered in the pilot study as suggested in the ToR, the Member States
which come into question for the pilot study will be those which cover all
Priorities set out in Article 3 of the ESF Regulation, meaning that only
countries with Convergence OPs come into question due to the Priority
Strengthening Institutional Capacity which is only in Convergence regions.
Additional criteria for the pilot study could be:
 Limited number of ESF OPs
 Simple or relatively simple allocation of Priority Axis to ESF Priorities
 Ample evaluation material (i.e. evaluations completed)
If we comply with the above criteria (including that of a country with a
limited number of OPs for manageability), we would actually only have the
choice between Austria, Bulgaria and Poland. Other Convergence OPs in
countries with only a few or one ESF OP do not have all the ESF Priorities
covered by their Priority Axis. Partnership often seems not to enjoy its own
Priority Axis although it may be included as a transversal theme of course.
While the three countries mentioned: Austria, Bulgaria and Poland seem
page 44
May 2015
Contract No. VC/2015/0098
good choices for pilot studies – also on account of available evaluation
material. Of the three, it is proposed to take Bulgaria as Bulgaria has all
ESF Priorities and presents a number of challenges as a pilot as there is
overlap between the PAs. It will also be a test of the in-house overview
team cooperating with an external expert.
The remaining country synthesis reports are to be delivered in batches. The
criteria for deciding on the batches could be either to choose the countries
included in each of the thematic ex post evaluations (as these are almost
mutually exclusive with the exception of France and Italy) or those with
accepted AIRs for 2014. The logic of the first would be that the in-depth
country reports would cover at least one ESF Priority in depth in the
countries which could be an interesting point of departure for the remaining
Priorities in that country. It would make sense here to start with the
countries from the ex post that is the furthest advanced or finished which is
presumably the Integration of Disadvantaged Groups, then Human Capital,
and finally Access to Employment. A further criterion could be the
availability of the AIR 2014 as it is absolutely necessary to have the final
AIRs for completing the country reports.
The batches of country synthesis reports would be managed by one core
team expert. The first batch would be managed by Isabel Naylon, the
second by Costanza Pagnini, the third by Bert-Jan Buiskool. The core team
experts would ensure the high quality of the reports delivered and a quick
turnaround in the delivery of revised reports. Given the logic that the
reports are produced as far as possible in-house, it is hoped that quality
issues will not be an issue. Still, a comparative review of all reports in terms
of coherence check will be done by Isabel Naylon, and for editorial issues,
by Haris Martinos.
Table 7. Overview of steps for the accomplishment of Task 2B
Task
Who
Task 2: National level analysis
B: Country synthesis reports (Lead Isabel Naylon)
Step 1: Pilot country study
Core team and country
expert
Step 2: Screening and pulling together of the Overview team
existing material by the overview team
Step 3: Filling in of the blank parts of the country Country experts
report templates by the country experts
May 2015
Step 4: Finalisation of reports by overview team
Overview team
Step 5: Drafting
Core team
Step 6: Delivery of country reports in batches
Project
manager
with
responsible
sub-task
manager (for each of the
3 batches)
page 45
Contract No. VC/2015/0098
2.4 Evaluation task 3: EU level analysis
2.4.1 Thematic synthesis reports
Objectives of the thematic EU synthesis reports and overall
approach
The two thematic synthesis reports will provide an overview of the
implementation of the ESF Priorities Promoting Partnerships and
Strengthening Institutional Capacity at EU level in terms of participants,
outputs and results, of typologies of interventions and – on the basis of
available evaluations findings - effects and socio-economic impacts.
As these two ESF Priorities were not covered under the three thematic ESF
2007-2013 ex post evaluations, the additional two EU thematic synthesis
reports on Promoting Partnerships and Strengthening Institutional Capacity
will provide the key information needed in order to compile the EU
synthesis report covering all the ESF Priorities.
Promoting Partnerships and Strengthening Institutional Capacity Priorities
cover a marginal proportion of ESF resources and they differ from the other
Priorities addressed by the ESF as they are not directly focused on
education/training or employment-related objectives for individuals, but
rather focus on strengthening structures and entities directly or indirectly
involved in implementing such objectives. For this reason available data on
outputs and results might not be representative of the actual relevance and
effectiveness of related interventions and additional qualitative information
will have to be collected.
The tender specifications mention that these reports shall follow in their
structure and analytical approach the overview assessment of 27 MS of the
thematic ex post evaluations. Consequently our approach to the analysis of
thematic priorities is centred on the analysis of a basic set of information at
the MS level in line with what has been collected during Task 1 of the expost thematic evaluations for the preparation of the Country Templates.
This information encompasses quantitative and qualitative data.
Quantitative data refers to the collection of financial and physical indicators
(outputs and results) for the relevant Priority Axes (or, where clearly
identifiable, sub-PAs).
Qualitative information will be collected in order to give a more complete
overview of how these Priorities have been dealt with in the different MS, to
describe the objectives established in the OPs as regards each of the two
ESF Priorities and to provide information on the types of interventions
supported and the target groups. This information will be analysed also
having regard to the national situation and strategies and EU-level
strategies under the two ESF Priorities even though, necessarily, they will
not provide the same level of detail and in-depth analysis as the other three
thematic ex post evaluations.
page 46
May 2015
Contract No. VC/2015/0098
The analysis of this information will provide a first picture of the extent to
which the resources were used and of the effectiveness of implemented
interventions (in terms of results) and efficiency measured in terms of
financial resources spent in order to achieve them.
Information collected shall be consistent with the methodology elaborated
in Task 1 of the thematic ex post evaluations attributing specific Priority
Axes to ESF Priorities and ensuring that no overlap between ESF Priorities
happens.
In addition to information collected through the analysis of dedicated PAs in
the OPs, the EU synthesis thematic reports will also collect and analyse key
information on typologies of interventions that can be clearly traced back to
the relevant category of expenditure (cat. 80 for Promoting Partnerships
and 81 for Strengthening Institutional Capacity) outside of dedicated
Priority Axes. This information will be treated separately in order to ensure
that overlapping with other ex post thematic evaluations is clearly
identified.
The combination of these two criteria (analysis of data concerning both
dedicated PAs as well as CoE) thus provides a clear demarcation of the
scope of the evaluations, although they will not necessarily and
exhaustively cover all the interventions which can be traced back to these
Priorities. Partnership building in particular is sometimes supported as a
transversal theme across all Priority Axes and is more difficult to detect.
Likewise, capacity building objectives – as we will explain in the following
paragraphs – are pursued across Priority Axes outside of those explicitly
devoted to this Priority.
In addition to the above some specific and additional analysis will be carried
out. It will focus mainly on issues of programming strategies, effectiveness
and sustainability which represent an important element of analysis for both
Partnership and Institutional Capacity themes. The scope and methodology
for this additional research will be identified on the basis of information
collected through the Country synthesis reports and will necessarily focus
on a limited number of MS and OPs.
In the following we provide a more detailed description of the approach we
will adopt for the delivery of the two synthesis reports starting from the
specificities of each ESF Priority to be investigated, tacking stock of
accumulated knowledge.
EU Synthesis Report on Promoting Partnerships
Art. 3.1(e) of ESF Regulation 1081/2006, describes the policy field as
follows: “Promoting partnerships, pacts and initiatives through networking
of relevant stakeholders, such as the social partners and non-governmental
organisations, at the transnational, national, regional and local levels in
order to mobilize for reforms in the field of employment ad labour market
inclusiveness”. This description helps us to define the demarcation and the
main focus of the evaluation task, i.e. interventions formally included,
under each OP in all the MS, under the Promoting Partnership specific
May 2015
page 47
Contract No. VC/2015/0098
Priority when the Priority is explicitly and clearly identified within an OP
Priority Axis (as it is, e.g., in the case of Belgium, Bulgaria, Germany, and
France). When the adoption of such a criterion is not feasible, interventions
under Reg. 1828/2006 Priority Theme 80 (dedicated to Partnerships) will be
considered under each OP as falling under the ESF Priority.
Based upon a quick assessment of the main reference documents (MetisEureval, 2011 and Metis-EEN 2014 and related country reports, and the
Preparatory Study) we can already trace a first outline of the most relevant
characteristics of Promoting Partnership ESF Priority which can be used as
the basis for formulating key evaluation questions or drivers:
 The policy field, not particularly relevant for its financial weight, is
very differentiated in terms of interventions and it has been locally
implemented in highly contextualised ways;
 Partnership and networking have been (correctly) generally assumed
as a means for implementing activities/interventions (therefore
focusing on systems and structures) and not as an end in themselves;
 There are some key variables that may be used for understanding
(and probably grouping) the policy field implementation and results.
These are: a) the subjects involved in partnership/networking; b) the
territorial (or policy) level covered by partnership and/or networking
(national, regional, local); c) the types of intervention/s implemented
through partnership; d) the issue/s (such as items, themes) covered
by partnership and/or networking;
 Three main types of interventions can be identified around which the
grouping could be further detailed, these are: i) Bringing together a
range of actors from across public, private and non-governmental
sectors so as to promote, establish and sustain partnerships;
ii) Strengthening local support structures and policies, where the
focus is on the local dimension as a way of enforcing intervention
effectiveness; and iii) Promoting Transnational Partnerships, for
“sharing knowledge and collaborate on areas of common interests”;
 According to preliminary research carried out we can anticipate that in
some MS the PP priority is not part of the ESF programming or its
implementation cannot be clearly identifiable (such as Cyprus,
Estonia, Latvia, Luxembourg, Netherlands, Slovakia, Sweden); while
in some others the priority has been implemented transversally (we
could say it has been mainstreamed), also in these cases it is difficult
to “isolate” and evaluate its implementation, outputs and results
(France, Greece, Hungary, Ireland, Poland, United Kingdom, Finland,
Portugal);
 The Priority-related evaluation reports are (at least until 2013) very
few (for 2011, Metis et. al., p.69, “the evaluations so far hardly deal
with the promotion of partnership and networking, only 2% of the
findings address this policy field”) and, together with “other policy
fields”, very poor in presenting sound evidence about effectiveness
(2014, Metis et al., p. 91).
page 48
May 2015
Contract No. VC/2015/0098
From the same data sources we also learn that participants (read entries) in
these interventions were more than 136,000 and they are mainly
concentrated in five MS (CZ, DE, HU, IT, RO). We also have information on
their occupational status and gender. No information is available about
entities involved, which are otherwise relevant for such a policy field which
is focused on structures and systems rather than on services delivery. The
most important results (considering 2012 result indicators and 2013
evaluation reports) are strongly linked to the thematic focus of the
established partnerships (such as knowledge sharing, new target groups
involvement, good practices exchange, delivery of effective interventions)
and consequently not easily comparable.
Very few robust evaluations concerning effectiveness are available (up until
2013) while a number of evaluation reports (very few in general for this
policy field) identify facilitating factors for partnerships development and
operations (such as regular networking, financial incentives) as well as
ways in which partnerships contribute to a more effective evaluation of the
ESF in general. As a whole, a very limited number of good practices has
been identified for this policy field and they are concentrated in 4 MS (AT,
HU, IT, SI).
The above indicated key-variables (intervention logic, composition of
supported partnerships, types of interventions, issues covered by
partnerships), together with physical and financial information and duration
collected through AIRs and SFC are the most relevant elements that can be
collected within the scope of the present study and on which the analysis
will be focused, at MS level, in order to provide an overview of the PP
priority programming strategies and implementation.
Within this framework, the main research questions that will be tackled at
the MS level in the report are the following:
- How has the PP priority been integrated – at MS level – in ESF
programming (analysis of intervention models and logics and
programming set-ups)
- How has the PP priority been implemented in terms of typologies of
interventions (i.e. what type of networking activities: partnerships,
pacts and initiatives and at what level of intervention: transnational,
national, regional or local) and beneficiaries (such as social partners,
NGOs), resources spent
- How much has been achieved in terms of outputs and results and,
according to available evaluations, impacts (in terms of mobilisation
for reforms in the field of employment and labour market
inclusiveness)
- How do achievements compare with programming expectations (as
represented by the specific OP indicators aggregated by MS)
May 2015
page 49
Contract No. VC/2015/0098
- What can we say about the future sustainability, gender
sensitivity and CAV of the interventions promoted under the PP
priority?
- What is the level of consistency between the PP priority and the
relevant Country Specific Recommendations14;
The extent of the completeness with which these research questions will be
answered will depend to a great extent on PP programming set-ups adopted
by each MS. We expect that these questions will receive a more complete
answer for those MS in which the PP priority is totally or partially
implemented under dedicated priority axes/measures. According to
preliminary research carried out so far this is the case for Austria, Belgium,
Bulgaria, Czech Republic, Denmark, Italy, Lithuania, Malta, Romania,
Slovenia, Germany, and Spain). According to the ESF Expert Evaluation
Network (2014) in these MS training, employment, local development and
geographical mobility are the issues mostly covered by partnerships and
networking activities; VET providers, social partners, NGOs and local
institutions are the most frequently involved actors; in general, both system
actions and actions targeting people are equally supported within the PP.
Answering the above indicated questions will allow to draw an overall
picture of both quantitative and qualitative) evidence focusing on all the
programming and implementation steps of Promoting Partnerships Policy
Field; in addition, such evidence will allow to define MS clusters according
to PP programming and implementation strategies, on the one hand, and by
actual funding, output and results, on the other.
Furthermore, the PP evaluation questions match the thematic synthesis
report structure described in the “Table of contents” section below which
will be produced using evaluation questions’ answers as inputs.
From a methodological point of view, Promoting Partnerships evaluation
questions will be answered in the following ways:
- The consistency between the PF programming strategies and their
implementation pathways (interventions and actions; implementing
subjects; target-groups, etc.) will be equally assessed through a desk
analysis of existing information reinforced by the evidence deriving
from the in-depth research on a limited number of MS (case-studies);
- The correspondence between expected and actual outputs and results
indicators (mainly at MS level) is going to be evaluated through a
desk comparison between the two groups of indicators quantifications;
explanations of positive and/or negative differences will be based
upon country experts’ contribution as well as on official
implementation document analysis (mainly 2013 and 2014 AIRs);
- The coherence between the expected and actual PF expenditure will
be assessed in the same way as output/result indicators expected
versus actual values;
14
page 50
Covering the 2004-2013 period
May 2015
Contract No. VC/2015/0098
- The evaluation of the relevance of PF implementation impacts is going
to be mainly based on available PF specific impact evaluations (very
few at the moment) eventually integrated, for the involved MS, by
evidence coming from the case-studies carried on by country experts;
- Finally, the CAV, gender sensitivity and future sustainability
assessment of PF implementation pathways and (if possible) results
will be based on country experts analysis using dedicated grids and/or
check-lists.
Such methodological design includes, as far as possible, the establishment
of groups of indicators derived, at MS level, both from Annex XXIII Reg.
1828/2006 and Programme indicators. However, due to the peculiarity of
PP with respect to the other ESF priorities as well to the differences in
approaching the issue in each MS, it will be reasonably more feasible to
identify aggregated (that is at EU level) output indicators than result
indicators (which are likely to be quite different among MS and in some
cases among OPs within the same MS).
EU Synthesis Report on Strengthening Institutional Capacity
In line with the 2006 Community Strategic Guidelines for Cohesion, which
identify good governance and capacity building as key issues to be
addressed, Article 3.2(b) – Regulation EC 1081/2006 mentions the
strengthening of institutional capacity and the efficiency of public
administrations and public services at national, regional and local level and,
where relevant, of the social partners and nongovernmental organisations,
as one of the main ESF Priorities for the 2007-2013 period. This Priority has
been introduced for Convergence regions and Cohesion MS as a key
element for promoting structural adjustments, growth and jobs, as well as
economic development. It is seen as a tool which will contribute to reforms,
better regulation and good governance, especially in the economic,
employment, education, social, environmental and judicial fields.
Although the share of overall funding volumes is relatively small,
institutional capacity building plays an important role in improving the
delivery of ESF services, their efficiency and their effectiveness. In general,
the actions under the capacity building priority theme should support the
reform of public administrations and public services, including their
modernisation, both in attaining sustainable socio-economic growth and
reducing disparities and in ensuring good governance. The support provided
under the priority should enable public administrations and public services
to become a factor of competitiveness, development and growth for the
Member States and regions. Clearly, these actions are strongly dependent
on the specific national and regional context of each MS, making it
somehow more complicated to classify and compare them (with respect for
instance to other ESF Priorities).
Each eligible MS has outlined the strategy for reinforcing the institutional
capacity in the National Strategic Reference Framework, indicating whether
May 2015
page 51
Contract No. VC/2015/0098
strengthening institutional capacity is implemented through a separate
Operational Programme (with one or more Priority Axis), or through one or
more Priority Axis in other programmes, especially regional programmes or
a combination of both. In this context, four Member States (Bulgaria,
Romania, Hungary and Greece) set up a dedicated administrative capacity
building Operational Programme, while nine others (the Czech Republic, the
three Baltic States, Poland, Slovenia, Slovakia, Malta and the UK — in
Wales) included it as a Priority Axis in one of their programmes, mainly in
regional programmes. Others, like Italy, combined the two approaches with
a dedicated national programme and Priority Axes in regional ones.
As a starting point the EU synthesis report on Strengthening Institutional
Capacity will analyse and compare quantitative data (financial allocations,
outputs and results) concerning the interventions linked to the Priority in
line with the demarcation of axes across priorities. Following this, it will
examine ESF Operational Programmes or Priority Axes specifically dedicated
to Capacity Building across all Member States (with reference to the
Convergence objective).
In addition, in order to expand the qualitative assessment of the Priority
(for which overlapping is not an issue) and in consideration of its horizontal
nature, the report will also explore Priority Axes which are not specifically
targeted at Strengthening Institutional Capacity but which nonetheless
foresee a relevant component of capacity building activities. This can be
done by looking at the categories of expenditures. ESF Priority
Strengthening Institutional Capacity is in fact thematically linked with
category of expenditure n. 81 “Mechanisms for improving good policy and
programme design, monitoring and evaluation at national regional and local
level, capacity building in the delivery of policies and programmes”;
however it can be related to other important categories, such as category n.
80 (Promoting partnerships, pacts and initiatives through the networking of
relevant stakeholders for the mobilisation of reforms), which is the object of
the other thematic synthesis report; and categories n. 65 (Modernisation of
labour market institutions) and n. 72 (Reforming education and training
systems), which are covered under the three ex-post thematic evaluations.
Finally, an additional point to be considered by the report is the relationship
between the Strengthening Institutional Capacity Priority with actions and
objectives of the technical assistance interventions. The ESF finances
actions aiming at supporting the structures for the management and
implementation of the Fund under Technical Assistance and should
therefore be clearly distinguished from the activities financed under the
Institutional Capacity Priority. However, the Institutional Capacity Priority
aims at supporting the reform of the administrations irrespectively of their
role in Structural Funds management and implementation and some overlap
between these two typologies of interventions occur. For this reason it will
be necessary to verify the use of resources destined to capacity building
and if the actions analysed are really interventions of capacity building or
their specific objectives cover mainly the needs of the management of
Structural Funds.
page 52
May 2015
Contract No. VC/2015/0098
Strengthening
intervention:
-
-
institutional
capacity
relates
to
two
main
types
of
improvement of mechanisms of good policies and sound
programming and implementation (such as through studies and
expert advice), including support for dialogue between the public
administration and its citizens and private bodies
capacity for the delivery of policies and programmes (through
training and specific support to key services).
Clearly, while public administration capacity building targets primarily
institutions (systems and structures), the capacity building of individuals
(e.g. staff of institutions) can be equally important to improve the ability of
these institutions to perform in a more effective and efficient way.
That is why ESF promotes an integrated approach to such priority:
assistance to individuals (for personalised support such as training of staff)
and assistance to entities (single institutions or a system of such).
Furthermore any assessment of strengthening institutional capacity should
also look whether relevant programmes and measures follow a horizontal or
sectoral approach, i.e. whether they pursue:
-
overall modernization and administrative reform (including in
the judiciary) – horizontal approach or
improvement
of
mechanisms
and
strengthening
of
institutions (always according to the integrated approach targeting
both systems and individuals) working in specific areas of cohesion
policy (such as employment, education, health and social policies) sectoral approach
Another task of this analysis concerns the extent to which the SIC priority
responds to, or is coherent with, the development needs of MS as
formulated in the CSR, with a particular focus on the sub-categories like
Improving the effectiveness and efficiency of the public administration, the
Judiciary reform, the Improvement of the business environment and the
Anti-corruption Public procurement.
The above indicated key-variables provide us with a general analytical
framework for assessing the implementation of the SIC priority. These
elements, together with physical and financial information and duration
extracted from AIRs and SFC are the most relevant information that can be
collected within the scope of the present study and on which the analysis
will be focused, at MS level, in order to provide an overview of the SIC
priority programming strategies and implementation.
Within this framework, the main research questions that will be tackled at
the MS level in the report are the following:
- How has the SIC priority been integrated – at MS level – in ESF
programming (analysis of intervention models and logics and
programming set-ups)
May 2015
page 53
Contract No. VC/2015/0098
- How has the SIC priority been implemented in terms of typologies of
interventions and beneficiaries, resources spent?
Concerning interventions the assessment will explore whether for
example supported interventions are aimed at:
(i)
modernization and/or administrative reform (including
interventions targeted at judiciary reform). Interventions
will be grouped according to a “process dimension” (e.g.
modernizing
and
optimizing
internal
processes,
development of quality management systems, adoption of
IT systems, improving the interaction between institutions
and with stakeholders, e-government / e-justice...) or to a
structure and organizational dimension (developing
appropriate administrative structures, incl. through
reallocation of functions, decentralization, improving
management structures).
(ii)
Skills enhancement of relevant staff? In this case
interventions will be grouped according to the specific
actions envisaged: e.g. traineeship programme, learning
network, training.
Sectoral interventions will be grouped according the relevant policy
field
One more element of analysis of the interventions will relate the
level of intervention (national, regional or local)
Concerning beneficiaries this question will explore in particular
which bodies and services have benefitted from ESF support for
SIC (public administrations and public services but also social
partners and non-governmental organizations) at the
- How much has been achieved in terms of outputs and results and,
according to available evaluations, impacts (with a view to reforms,
better regulation and good governance especially in the economic,
employment, education, social, environmental and judicial fields).
- How do achievements compare with programming expectations (as
represented by the specific OPs indicators aggregated by MS)
- What can we say about the future sustainability, gender
sensitivity and CAV of the interventions promoted under the SIC
priority?
- What is the level of consistency between the SIC priority and the
relevant Country Specific Recommendations15.
Such methodological design includes, as far as possible, the establishment
of groups of indicators derived, at MS level, both from Annex XXIII Reg.
1828/2006 and Programme indicators. Given the peculiarity of the SIC
15
page 54
Covering the 2004-2013 period
May 2015
Contract No. VC/2015/0098
priority, it will be reasonably more feasible to identify aggregated (that is at
EU level) output indicators than results indicators (which will be probably
quite different among MS and in some cases among OPs within the same
MS).
Data and information sources
As already mentioned above, the data and information sources feeding the
thematic EU synthesis reports refer in the first instance to the material
which is already available and which has been described in detail in the
Tender Specifications and in the previous chapters of the present Inception
Report. These refer in particular to:
 Operational Programmes and Annual Implementation Reports (with
particular reference to the 2013 and 2014 AIRs which were not
reviewed by the EEN Country Reports); AIRs are a key source of
information in order to update the quantitative assessment and double
check information from the SFC2007, to provide more detailed
information on typology of interventions implemented under the
dedicated Priority Axes, and to identify Priority-relevant objectives and
actions outside of dedicated Priority Axes. The analysis of the AIR will
consider the following parts of the document related to the
Programmes or Priority Axes for Promoting Partnerships and
Strengthening Institutional Capacity:
1. Section 2 “overview of the implementation of the Operational
Programme” in particular for the general information about the
Programme e for the financial allocation referred to the relevant
categories of expenditures;
2. Section 3: “implementation by priority” including the description
of the actions activated and their physical implementation.
 SFC2007 database; for compiling the quantitative assessment
 The updated inventory of evaluations – up to 30 June 2015 – that will
be analysed in detail in order to identify evaluations (on-going, final
but above all ex post; thematic and ESF priority-wide; results and
impacts-focused, also those produced in the framework of ex post
evaluation of Cohesion Policy Programmes;) that have been carried
out at the EU or MS level and that might provide important elements
concerning
effects,
impacts,
lessons
learned
and
policy
recommendations relevant to the two Priorities (the ESF assessment
on administrative and institutional capacity and the forthcoming ERDF,
ESF and CF ex post study on delivery system will be of particular
importance for the Institutional Capacity report)
 DG REGIO ex post evaluation of delivery systems, WP12 which is
expected to provide interesting insights and complementary
information on the delivery systems of Cohesion Policy
 Preparatory Study; this report will be particularly useful in providing
information for identifying the main clusters of interventions, but also
the programme architecture of 43 OPs (Priority axes, actions, and
May 2015
page 55
Contract No. VC/2015/0098
interventions) including “Promoting Partnerships” and “Strengthening
Institutional Capacity”
 EEN synthesis and country reports – last round; these provide
information on outputs, results and short descriptions of the ESF
Priorities (up to 2013) and will be useful for completing the
quantitative assessment as well as for providing key information for
filling-in the dedicated sections of the Country synthesis reports
 Context data, such as Employment and Social Developments in
Europe; Fourth report on Economic and Social Cohesion; Fifth and
Sixth Report on Economic, Social and Territorial Cohesion, Quality of
public administration report16, 2015 EU Justice Scoreboard17.
It should be noted that most of the relevant information derived from the
above data sources will be included and analysed in the Country synthesis
reports (sections on Promoting Partnerships and Strengthening Institutional
Capacity); consequently these reports will represent a key information
source for the drafting of the EU synthesis reports.
In addition, as further explained in “Tools and activities” section/Step 2
below, the PF implementation in some MS (max 5, to be agreed with the
Commission) will be more deeply analyzed through specific case studies
aimed at more deeply exploring the scope and effectiveness of a sample of
PF implementing interventions.
Table of contents
Each thematic EU synthesis report will be 50 pages long and its table of
contents broadly reflect the structure of the other thematic ex post
evaluations. A tentative structure is outlined below:
Table 8. EU Synthesis Report structure
Introduction
Executive summary
Background and report objectives and
scopes
Overall progress in ESF Priority
implementation
Analysis of clusters of interventions
Evaluation results and evidences
Conclusions
Report background and objectives (3
pp)
Report objectives
ESF and Promoting
Partnerships/Capacity building
Key-data concerning ESF Priority
delivery: a general overview (10 pp.)
Analysis of outputs (EU and MS level)
Analysis of results (EU and MS level)
ESF Priority operations and
Types of interventions and interventions
16
http://ec.europa.eu/enterprise/policies/industrial-competitiveness/monitoring-member-states/improvingpublic-administration/index_en.htm
17 COM(2015)116
page 56
May 2015
Contract No. VC/2015/0098
interventions (10 pp)
clustering: numbers and facts (EU and
MS level)
Types of interventions: numbers and
facts (EU and MS level)
Evaluation data, approaches, methods
and results (15 pp.)
Evaluation sources, approaches and
methods
Evaluation evidences (EU and MS level)
Main factors influencing interventions
results and impacts
Conclusions: implications for future
policies (3 pp)
Tools and activities
Step 1. Desk analysis of existing information
During this first step all relevant information and documents for the drafting
of the EU synthesis reports on Promoting Partnerships and Strengthening
Institutional Capacity will be collected and analysed, in particular
concerning implementation of the relevant ESF priorities in OPs/MS,
identification of dedicated PAs consistent with the other three ex-post
thematic evaluations, implementation processes, results and evidence from
specific evaluations (see the data sources indicated above). This activity will
entail a review of available material and will result in a preliminary draft of
the overview assessment of the 28 MS; it will also serve to identify the
information that is needed from Country experts and to pre-compile the
relevant sections of the Country synthesis reports (see Task 2). During this
first step the core team will also build a preliminary clustering of typologies
of interventions implemented in response to each of the two priorities.
In addition the following tools will be implemented:
 MS/OPs table with Priority Axes dedicated to Promoting Partnerships
and Strengthening Institutional Capacity. A preliminary table,
compiled according to information in EEN country and synthesis
reports, is provided in Annex I. The table will be revised to make sure
that no overlapping exists among the five ESF priorities. On the basis
of this table it will be possible to identify for each dedicated Priority
Axis the relevant financial allocation, outputs and results indicators,
targets and beneficiaries. This forms the basis for the qualiquantitative assessment.
 MS/OPs table with financial allocations per category of expenditure
which have a direct relevance with each of the two Priorities (n. 80 for
PP and n. 81, 65 and 72 for SIC). This information will be matched
with the MS/OPs table with dedicated Priority Axes so as to map MS
according to their programming decisions on how to deal with the two
priorities (i.e. whether they have dedicated specific OPs/PAs or have
adopted a “transversal” approach whereby relevant interventions are
distributed across OPs/PAs or a combination of both); MS positioning
May 2015
page 57
Contract No. VC/2015/0098
might be used as an additional element for the selection of a number
of MS/OPs for a more in depth analysis.
Step 2. Identification of information
information at country level
gaps
and
collection
of
On the basis of the analysis undertaken during Step 1, the sections of the
country synthesis reports dedicated to Promoting Partnerships and
Strengthening Institutional Capacity will be precompiled and circulated to
the country experts for double-checking and updating on the basis of the
analysis of the AIRs and relevant evaluations carried out in MS. Country
experts will also be asked to fill in information gaps concerning dedicated
OPs/PAs objectives.
During this step, a limited amount of additional (mostly field) research will
be envisaged aiming at studying in more detail the implementation and
results of the two ESF priorities in a restricted number of MS. This
information will help shed light on the scope and effectiveness of specific
typologies of interventions or ESF set-ups for the implementation the
priorities.
Field research will be carried out by Country experts and will mainly involve
the collection of additional information/data through telephone interviews
(with MAs and/or relevant stakeholders).
The selection of case studies will be agreed with the European Commission
once Country synthesis reports have been completed and will be based as
far as possible on the criteria used for the ex post thematic evaluations
(e.g. relative importance of dedicated Priority Axes/OPs; socio-economic
contexts). In addition representativity of different ESF set-ups for
implementing the priorities will be taken in consideration (i.e. MS that have
dedicated specific OPs/PAs to the priorities vs MS that have dealt with the
priorities “horizontally” i.e. outside of dedicated Priority Axes).
Step 3. EU-level overview of ESF Priority and drafting of the
Thematic EU synthesis report
This step is dedicated to synthesising the information collected from
Country experts (Country synthesis reports and additional information from
case studies) and other EU-level relevant documents. The final result shall
be an overview of the implementation, outputs, results and effects of ESF
Priorities Promoting Partnerships and Strengthening Institutional Capacity.
The overview will adopt a comparative approach analysing available
information across MS, objectives, financial allocations, actions, outputs and
results. MS/OPs will be classified according to the importance that they
attribute to each of the ESF priorities Promoting Partnerships and
Strengthening Institutional Capacity, and whether they do so through
dedicated OPs/Priority Axes or through a clear horizontal approach. This will
allow making comparisons in terms of results, impacts, and MS socioeconomic contexts.
page 58
May 2015
Contract No. VC/2015/0098
As previously mentioned, the overview will include both quantitative as well
as qualitative analyses. Quantitative analysis will provide comparable and
standardised information, consistent with that of the three ex-post ESF
thematic evaluations covering financial information, outputs and results.
Qualitative information will provide information on objectives established in
the Operational Programmes and types of interventions funded, together
with a contextualisation of this information vis-à-vis national situations and
strategies and EU-level strategies. Particular attention will be given to
synthesise any existing evaluative evidence on impacts and effects of
implemented strategies with a view to identifying strengths and weaknesses
as well as facilitating factors or obstacles.
Team members
One core team member (Costanza Pagnini) will be in charge of coordinating
the activities and drafting the Thematic EU synthesis reports. She will be
supported by a member from the research team (Diego Teloni), who will
discuss and review the reports.
Country experts will draft a separate small chapter during their
complementary information provision and provide country-specific insight
concerning the need for further in depth assessment.
Table 9. Overview of steps for the accomplishment of Task 3A
Task
Who
Task 3: EU level analysis
A: Thematic EU synthesis reports (Lead: Costanza Pagnini)
Step 1. Desk analysis of existing information
Core team
Step 2. Identification of information gaps and collection of
information at country level
Core
team
and
Country experts
Step 3. Comparative analysis and drafting of the reports
Core team
2.4.2 EU synthesis report
Objectives
The second part of the EU level analysis concerns the main product of this
project: the production of the EU synthesis report. This report will
synthesize the analysis, findings, conclusions and lessons learned from the
ex post evaluation of the ESF 2007-2013. This report will be structured
around the ESF priorities and will both be used as a basis for the
communication to the Council and the EU Parliament and be the main
communication to other relevant audiences. From this perspective the
language will have to be non-technical and the report very concise
(maximum 50 pages).
May 2015
page 59
Contract No. VC/2015/0098
The template for the EU synthesis report
The ToR indicates which information the EU synthesis report should include.
Based on these requirements the following structure of the synthesis report
is proposed.
Table 10. Structure of the EU synthesis report
Executive Summary
Introduction
Adaptability of workers
Access
and
Sustainable
Integration
into
Employment
Social Inclusion
Human Capital
Promoting Partnerships
Strengthening Institutional Capacity
Conclusions and lesson learned on the EU level
policy on economic and social cohesion
Policy context
Aims
Methods
Contextualising ESF
The extent to which resources were
used
Effectiveness
Efficiency
Socio economic impact
Community Added Value
Gender sensitivity
Sustainability
Conclusion and lessons learned
See above
See above
See above
See above
See above
policy choices
target groups
programming
implementation
monitoring
evaluation of future programmes
Data and information sources
The EU synthesis report will be based on several sources as already
identified in the ToR.
The main secondary sources for the EU synthesis report are:
 The country synthesis reports (all 28 MS)
 Overview, comparative analysis, conclusions, lessons learnt from the
four thematic ex post evaluation reports and the two thematic EU
synthesis reports,
Other possible sources are:
 Additional background material: such as Employment and Social
Developments in Europe, the reports on Economic, Social (and
page 60
May 2015
Contract No. VC/2015/0098
Territorial) Cohesion, relevant reports from Eurofound and Cedefop,
national studies with an EU perspective
 The new Regulations for the Cohesions Funds and the ESF 2014-2020
in particular and the Guidance documents related to these regulations
(in particular in connection to the objective to provide lessons learned
which could be useful for the new programming period).
The level of depth of information differs per country since for every
thematic ex post evaluation an in-depth assessment is made in only a
limited number of countries (8-12 countries) on that particular theme.
However, each country is covered by at least one in-depth study and by all
five thematic synthesis reports.
Despite, or rather because of, the fact that the EU synthesis report is
mainly based on existing data sources, we suggest organising a peer group
seminar in Brussels near the end of the process to reflect on the main
conclusions and lessons learnt. For this seminar we would like to invite
senior officials from the EC, the main evaluators for each of the thematic ex
post evaluations, and a limited number of senior experts. The aim would be
to validate the main findings and conclusions and lessons learnt in a peer
setting.
In the following sub sections we explore our approach for every sub section
in the EU synthesis report (see Table 10 on the structure of the report
above).
Activities and tools
Step 1: Aggregating information in the database
Based on the work done in Tasks 1 (setting up of the database) and Task 2
(filling in of the database at country level), Task 3 tries to aggregate data
on EU level (and find common indicators between priority themes and EU
level).
Step 2: Contextualising ESF (Section 1 in the synthesis report)
First of all, the EU synthesis report should provide information on objectives
established in the Operational Programmes and types of interventions
funded, together with a contextualisation of this information vis-à-vis
national situations and strategies and EU-level strategies.
This analysis will be implemented by plotting all relevant contextual
information (Europe 2020 indicators relating to economic development,
employment, education, lifelong learning and social inclusion), general
trends in Country Specific Recommendation over the programming period,
and focus of ESF programmes in countries. Here we would like to build
further on the scoreboards that were developed in each of the thematic ex
post evaluations where the key indicator trends are presented for each
Member State and the EU-27 average. Each thematic evaluation also
May 2015
page 61
Contract No. VC/2015/0098
provided an overview of relevant Priority Axes per ESF Priority. Having this
information, the following outputs can be created:
I.
page 62
Sub-step 1: Typology of different country contexts per ESF Priority
(3-4 clusters) assessing Europe 2020 indicators (such as GDP
development, employment and unemployment figures, education,
lifelong learning, poverty index etc.) and other. Here we take the
development over the 2007-2013 period, also to identify whether
contexts are positive.
May 2015
Contract No. VC/2015/0098
Table 11. Context indicators for each ESF priority (as included in the
EU27 templates)
A2E
HC
GDP
Real
expenditure per
capita
(in
PPS_EU27)
Bienual
(%)
growth
Employment
rates by age and
sex
Employment
rates by highest
level of education
attained
Unemployment
rates from 15 to
64 years
Temporary
employees
as
percentage of the
total number of
employees
Self-employed
(total
number:
thousands)
Young people not
in
employment
and not in any
education
and
training
Job vacancy rate
(2008 onwards)
May 2015
Higher
education:
HE
attainment
(Eurostat)
(disaggregated by gender
and age where available)
Continuing adult education:
Employment of low skilled
(Eurostat), Participation in
LLL
(Eurostat)
(disaggregated by gender
and age where available)
Funding:
Expenditure
on
E&T (Eurostat), Expenditure
on R&D (Eurostat).
Part-time
employment
Funding:
spending
participants
Initial education: Extent of
ESL
(Eurostat),
Unemployment
of
young
people
(Eurostat),
PISA
scores (2006, 2009, 2013,
OECD) (disaggregated by
gender and
age where
available)
ADAPT
SI
Percentage
of
enterprises
providing CVET
Employment rate
Percentage
of
companies
systematically
checking
of
training needs
%
of
establishment
giving
employees time
for training
Unemployment
rate
Long
term
unemployment rate
Inactivity
rates
among
the
low
skilled (ISCED 1-2)
Inactivity
rates
among individuals
of
foreign
nationality
Share of young
people (15-24) not
in education not in
training
Share
(%)
of
population aged 18
and above at risk
of poverty or social
exclusion
Share
(%)
of
individuals
(aged
18 and above) born
in a foreign country
at risk of poverty
or social exclusion
Percentage
of
expenditure
in
different
social
protection areas in
relation with the
GDP
LMP
and
page 63
Contract No. VC/2015/0098
II.
III.
IV.
Sub-step 2: Trends of development challenges based on the
analysis of the Country Specific Recommendations during the
programming period. Types of CSR will be clustered and countries
will be mapped along clusters of CSR, showing different challenges
identified across the EU.
Sub-step 3: Typology of programmes assessing type of focus of
programmes per ESF Priority (type of interventions supported and
relative financial contribution ESF). It will be assessed whether
countries concentred their budget on a limited number ESF priorities
(concentration principles) or not. You could identify programmes
that (1) focus on a wide variety of ESF priorities, (2) medium
number of interventions, or (3) low number of interventions, having
a more focused approach. Moreover, we will identify the focus of
each programme (by identifying the ESF priority / COE that receives
the largest budget).
Sub-step 4: countries will also be compared on how the way the
ESF A2E investment contributed to the national policies. Overall
assessment are made whether ESF is mainly used as additional
funding to mainstream policies, or whether ESF was used to test
new and innovative activities, or whether ESF was used to reach new
target groups, or finally whether ESF was used to improve the
delivery systems and methods.
Based on the four steps above, countries can be clustered and illustrated
with some examples coming out of the country synthesis reports (task 2)
and the thematic evaluation reports on each of the 6 ESF Priorities. It would
be interesting to see whether ESF is addressing the exact issues in a
country that shows the greatest need (linking the different sub steps).
Step 3: Assessment of evaluation criteria
The extent to which resources are used: The first step in the
implementation of ESF is the mobilisation of the financial inputs provided by
the EU and the Member States. The utilisation of the financial resources is a
condition sine qua non for achieving any kind of impact.
Per ESF Priority we will assess:
 Budget allocated to every Priority Axis and finally spent (certified
expenditure), calculating absorption rates (here we makes use of the
latest AIR2014). These budgets will be aggregated on ESF Priority
level.
 Budget allocated to the relevant Categories of Expenditure per ESF
Priority.
 Information
analysis).
page 64
explaining
factors
for
low
absorption
(qualitative
May 2015
Contract No. VC/2015/0098
By comparing budgets spent on Priority Axis and the budgets allocated to
categories of expenditure, we are able to identify deviations, possibly
explained by the allocation of Priority Axis to one single ESF Priority (cross
check).
Effectiveness: For each ESF Priority an overview will be provided in terms
of outputs and results achieved based on the most important data collected
through Annex XXIII and programme specific indicators.
 Annex XXIII indicators are registered on Priority Axis level, and
centrally captured by the European Commission. Within task 2, this
database will be updated with participant data coming from AIR 2014.
 Programme specific indicators are stored on Priority Axis level, and
OPs follow different approaches leading to a wide variety of indicators,
also including different types of measurement like absolute numbers,
percentages, total or additional, etc. Therefore, the ex post evaluation
synthesis can only aggregate values of indicators that have a common
ground (unit of measurement) and can therefore only provide a
picture of the minimum contribution of ESF (see chapter 1.4.4 on
programme specific indicators). Moreover (average) achievement
ratios are calculated for the programme specific indicators
(achievement value / target value) per PA, OP, ESF priority, country
(captured by task 1) and EU level (captured by task 3)18.
For each ESF Priority an overview will be provided of the main physical
outcomes of ESF financed interventions.
The results of the analysis in the thematic ex post evaluations and the
thematic synthesis reports for the other two ESF priorities PP and SIC will
be screened and used for evidence of effectiveness, success factors, good
practice, and lessons learned.
The aggregated findings from the evaluation inventories will also be drawn
on for the assessment of the effectiveness of ESF interventions as well as
relevant EU level evaluations and research projects.
We will also identify some common indicators across ESF priorities and
countries to provide a picture on the performance of ESF integrally (see
Section 1.4.4).
Efficiency is related to the question of whether the results have been
achieved in a cost-efficient way. Since it is hardly possible to assess the
efficiency of spending by linking financial spending per Priority Axis and
output and result indicators (also given the broadness of interventions
included in each Priority Axis), we will base the assessment of efficiency
mainly on the assessment made in the thematic ex post evaluations and the
18The
problem, however, is that a relatively large amount of indicators do not have a target value, so the
average achievement rate is not representative for all indicators. Another challenge is that many OPs have
adjusted their targets over the programming period, in order to better align them to the actual progress
made in the programme. As a result, one could question whether measuring the achievement rate is
meaningful. With proper adjustments, the moving target can be reached at the end this way disguising a
failure.
May 2015
page 65
Contract No. VC/2015/0098
thematic synthesis reports for the other two ESF priorities (in-depth country
reports) and summarise the outcomes. This is assessed in terms of the
extent to which the desired effects were achieved at a reasonable cost, how
economically the resources used were converted into outputs and results,
and the relationship between the outputs and results of ESF interventions
and the resources used to deliver them19. The main components of the
approach are listed in the box below. The Evaluation Synthesis will
summarise the results of this investigation in the three ex post evaluations.
Box 4. Approach to efficiency in the three ex post evaluations
The analysis of the efficiency of interventions in the three ex post
evaluations compares the scale and scope of interventions (cost per
participant) and the relative difference the intervention makes. This
requires:

A clear understanding of how the costs of ESF interventions compare to
their outputs and results achieved and at what costs the desired results
have been delivered, including mapping the relationships between the
cost vs outputs and results achieved.

An understanding and analysis of the measured efficiency of ESF
interventions compares relative to other similar interventions funded
from different national/EU funding streams.

An analysis of the relative efficiency of the different ESF interventions
and the reasons behind these differences in efficiency.
In order to answer these questions, the approach to the efficiency analysis
of interventions in the three ex post evaluations incorporates the following
key elements:

Comparison of the levels of output/result per EUR invested per person /
entity across different projects supported under the interventions.

Ranking of the
interventions.

Comparison of the results of different ESF interventions and other
similar interventions (funded from different streams) to support similar
themes (i.e. obtaining information on cost per unit of matched
comparators).

Analysis of the views of the ESF stakeholders and other independent
experts about cost-effectiveness gathered in the interviews and the A2E
e-survey.
19
page 66
cost
per
participant
/
entity
of
different
ESF
See definition in the Commission’s guidance on evaluation, European Commission (2001) and Evalsed
Sourcebook: Methods and techniques: P.150-151.
May 2015
Contract No. VC/2015/0098
Socio economic impact: We will synthesise evidence gained on socio
economic impact for every ESF priority as included in the thematic
synthesis reports for the three thematic ex post evaluations and the
thematic synthesis reports on Promoting Partnerships and Strengthening
Institutional Capacity (in-depth country reports).The country synthesis
reports will be checked for updates (AIR 2014).
Community Added Value: We will synthesize evidence gained on
Community Added Value for every ESF priority as included in the thematic
synthesis reports for the three thematic ex post evaluations and the
thematic synthesis reports on Promoting Partnerships and Strengthening
Institutional Capacity (in-depth country reports). The country synthesis
reports will be checked for updates (AIR 2014).
Gender Sensitiveness: We will synthesize evidence gained on gender
sensitiveness for every ESF priority as included in the thematic synthesis
reports for the three thematic ex post evaluations and the thematic
synthesis reports on Promoting Partnerships and Strengthening Institutional
Capacity (in-depth country reports). The country synthesis reports will be
checked for updates (AIR 2014).
Conclusions and lessons learned: Conclusions and lessons learned
should be concise, logical, evidence based, analytical and relevant.
However, experience shows that in many evaluations, a logical fault line
appears between findings, conclusions and lessons learned. We will attempt
to produce a common thread between these different results.
Step 4: Conclusions and lessons learned
As a first sub-step the core team will cluster the conclusions for the EU level
policy on economic and social cohesion and lessons from the different
thematic evaluations and country synthesis reports along the following
lines:
 Policy choices: an assessment will be provided whether the focus of
programmes are in line with the emerging development needs over
the programming period.
 Target groups: assessing whether and how the relevant target
groups are addressed by the programmes giving the challenges
countries face, in particular concerning young people.
 Programming,
and
implementation:
assessing
whether
programming and implementation can be improved, also taking into
account the new Regulations for the 2014-2020 programming period
(ESF Regulation and Common Provisions Regulation) that have
introduced many changes and novelties, and reinforced obligations for
the ESF.
 Monitoring and evaluations
requirements): improving
addressing
the issue to
interpretations and consistent
May 2015
(with regard to the new regulatory
monitoring systems, amongst else
further standardise the definitions,
collection of monitoring data (outputs
page 67
Contract No. VC/2015/0098
and results), including linking better administrative systems Lessons
will be drawn on the planning, implementation, and use of evaluations
in policy making. Special focus is on the topic how to improve the
evidence on the (net) effectiveness of interventions.
In a second sub-step the core team will categorise these lessons based on
categories like time-frame (short/mid/long term), addressee (MA, IB,
beneficiaries etc.), type of action (e.g. termination of an activity,
intensification etc.)
In a third sub-step the evaluation team would like to discuss the outcomes
during a peer group seminar in Brussels (Step 5) for which we would like to
invite the following discussants (maximum 15):
 Senior officials from the EC (DG EMPL, DG EAC, DG REGIO),
 Responsible evaluators for each thematic ex post evaluation,
 Limited number of experts that were involved in the ex post
evaluation exercise
During this seminar we want to prioritize the conclusions and lessons
learned for the future (also taking into account the political, institutional
and practical feasibility to implement the lessons learned). The figure below
provides a schematic overview how we are planning to translate the
evidence collected into concrete findings, provide a judgment, and finally
develop conclusions and lessons for the future.
Figure 2. From evidence to findings to lessons learned
Evidences from
data collection …
… are cross‐analyzed
to obtain findings…
… which are then
used for judgement…
… and to formulate
Lessons learned
Evidence #1
Finding #a
Evidence #2
Evidence #3
Finding #i
Lessons learned
Finding #b
Evidence #4
Finding #ii
Finding #c
Evidence #5
Evidence #6
page 68
Finding #d
May 2015
Contract No. VC/2015/0098
Table 12. Overview of steps for the accomplishment of Task 3B
Task
Who
Task 3: EU level analysis
B: EU synthesis report (Lead: Bert-Jan Buiskool)
May 2015
Step 1: Preparation, setting up a database
Core
team
with
support of overview
team
Step 2: Contextualising ESF
Core team
Step 3: Assessment of evaluation criteria
Core team
Step 4: Conclusions and lessons learned
Core team
Step 5: Validation of findings through Peer Review Seminar
Core team + peer
reviewers
Step 6: Drafting
Core team
page 69
Contract No. VC/2015/0098
3. REPORTING
This chapter presents the tables of contents of the various deliverables.
3.1 The Inventory of Evaluations
The structure for the inventory was already developed in the framework of
the ESF Expert Evaluation Network. The structure is robust and has proved
useful in filtering for various elements of information, e.g. types of
interventions or target groups evaluated, types of evaluations carried out,
findings on various themes etc.
The structure is given in Table 13 below.
page 70
May 2015
Contract No. VC/2015/0098
Table 13. Content of the inventory
Evaluation Plan (table I_EvanPlan) – no update expected
Basic info on evaluation plans
0
admin. Country /polit. (MS)
Unit
No.
Version number
Version date
Level for which evaluation plan is available
Please specify level
No. of eval. to Link to be moni‐
carried toring
out
Specification of arrangements
Specifications I ‐ Content & Use
Description of the scope of the evaluations
Specification of scope(s)
Main eval. Questions
Specification of main evaluation questions
Potential use of the Specification of the potential use
evaluation
Eval Plan Specifications II ‐ Time, Money & Overall comment
Time frame for evaluation
May 2015
Specification
Information on Finanical resources foreseen
Specification
Overall Comments to Evaluation Plan
Your notes and excerpts
page 71
Type of evaluation
Contract No. VC/2015/0098
Implementation of the evaluation (table II_EvanImpl)
II.1 ‐ Basic info on a specific evaluation
17
admin. Country /polit. (MS)
Unit
No.
Short title (english)
Full title of evaluation contracted /announced (original language)
Title of Evaluation contracted /announced (english translation)
Authority/ Institution/ Organisation in charge
II.2 ‐ Specification : Type, evaluator, periods & status
Type of evaluation
Evaluator
End of the Start of the Period Period covered: evaluation evaluation covered: (planned or (planned or END BEGIN [dd/mm/yyyy]
actual) actual) [dd/mm/yyyy]
[dd/mm/yyyy] [dd/mm/yyyy]
Status of evaluation
II.3 ‐ Specification: Coverage (Programmatic & policy level, territorial level) and intended use
Level of evaluation
page 72
SFC Code of OP
please specify evaluation level
Territorial level
If other, please specify
Potential If 'mixed' or if otherwise 'unclear' ‐ (intended) use of please specify
the evaluation
May 2015
Contract No. VC/2015/0098
Evaluation documents (table III_EvanImpl_Doc)
(
characters!)
III. 1 ‐ Evaluation documents ‐ Title, type & topics (and availabilty)
18
admin. Country /polit. (MS)
Unit
No.
Ref. Eval
Type of evaluation document available
Short title (english)
Title of the document
Title in english translation
Details /main topics
Comments
Evaluation approaches (table III_EvalImpl_Appr)
(not more than
510 characters!)
III. 2 ‐ Evaluation approaches ‐ Specifications
20
admin. Country /polit. (MS)
Unit
Ref. Eval
Evaluation approach
If Impact Evaluation ‐ If 'Impact other', please specify
which design?
If other, please specify
Evaluation approach ‐ general comments
Data collection method (table III_EvalImpl_Meth)
III. 3 ‐ Data collection methods
53
admin. Country /polit. (MS)
Unit
Ref. Eval
Data collection methods
If other, please specify
General comments
Findings of the evaluation (table III_EvalImpl_Find)
III. 4 ‐ Findings reported ‐ Basic info on content and reference
200
May 2015
admin. ID Country Ref. /polit. (MS)
Finding Eval
Unit
Describe or specify finding
Provide sources and references (short) ‐ Preferably the most relevant reference on TOP please!
page 73
Contract No. VC/2015/0098
III. 4 ‐ Findings reported ‐ Evaluation criteria and policy field(s) relating to
Finding refers (1st) Finding refers (2nd) Finding refers (3rd) to Key Eval Finding refers (1st) to Policy Field
to Key Eval Finding refers (2nd) to Policy Field
to Key Eval Finding refers (3rd) to Policy Field
Criterion
Criterion
Criterion
General comments
Findings by Intervention types (table IV_EvalFind_IntTpe)
IV. 1 ‐ Findings & Interventiontypes ‐ Assignments and Sources
188
Country Ref. Ref. (MS)
Eval Findg
Types of interventions
Additional sources used (if not already listed in this very inventory) and bibliographic information
Comments
Findings by target group (table IV_EvalFind_TrgtGrp)
IV. 2 ‐ Findings & Target groups ‐ Assignments and Sources
250
page 74
Country Ref. Ref. Types of Target Addressees
(MS)
Eval Findg
Sub‐type of Addressee
Additional sources used (if not already listed in this very inventory) and bibliographic information
May 2015
Comments
Contract No. VC/2015/0098
3.2 The Country Reports
Box 5. Template for the Phase 2 Country Reports
Table of contents: Country Report for XXX all Priority Axes
Executive summary
Introduction
1. Context, general policy background and ESF investment in all ESF
Priorities
1.1 Economic and social challenges in the Member State assessing
Europe 2020 indicators (such as GDP development, employment
and unemployment figures, education, lifelong learning, poverty
index etc.) and strategic response
1.2 Investment priorities in the MS (in relation to challenges and EU
level strategies)
1.3 Summary of main types of interventions funded across ESF
Priorities and main target groups
2. Financial volumes, outputs, results and findings by ESF Priority
2.1 Increasing adaptability










Financial absorption (SFC2007 and AIR 2014)
Outputs (SFC2007 and AIR 2014)
Results (Programme specific indicators and other results
gathered in the framework of the thematic ex post evaluation)
Extent to which resources were used
Effectiveness
Efficiency/cost-effectiveness
Socio-economic impact
CAV
Gender sensitivity
Sustainability
2.2 Enhancing access and sustainable integration into employment

See 2.1
2.3 Reinforcing social inclusion

See 2.1
2.4 Enhancing human capital

See 2.1
2.5 Promoting partnerships

See 2.1
2.6 Strengthening institutional capacity

See 2.1
2.7
May 2015
page 75
Contract No. VC/2015/0098
3. Conclusions
3.1 resources used
3.2 effectiveness
3.3 efficiency
3.4 CAV
3.5 sustainability
3.6 gender sensitivity
3.7 socio economic impact
4. Lessons learned
4.1 What are the key lessons in terms of policy choices - were the
policy choices made appropriate? What actions were missing?
4.2 What are the key lessons in terms of target groups - were the
appropriate target groups chosen?
4.3 What are the
programming?
key
lessons
in
terms
of
the
appropriate
4.4 What are the key lessons in terms of the effective
implementation - were the right types of activities funded? Were
the target groups chosen prioritised in the implementation? Has
the response to the challenges posed by the crisis been
adequate?
Were
the
delivery
mechanisms
functioning
appropriately? How could the effectiveness of activities funded by
ESF been increased?
4.5 What are the key lessons drawn in terms of the robustness of the
monitoring (such as setting of appropriate output and result
indicators, any challenges in the regular monitoring and
reporting on the indicators chosen)?
4.6 What are the key lessons drawn in terms of the robustness of the
evaluation systems (such as availability of data measuring
progress on output, result indicators, the choice of evaluations
conducted and not conducted, the use of impact evaluation
methods such as counterfactuals etc)
5. Good practice
5.1 Description of one evidence based good practice in Member State
Annexes:
Information sources
page 76
May 2015
Contract No. VC/2015/0098
3.3 The Thematic Synthesis Reports
Box 6. Draft Thematic Synthesis Report Structure
Table of contents
Executive Summary
1. Synthesis
1.1
Scope and aims of evaluation, links to other evaluations
2. Promoting partnerships/Capacity building: context analysis
3. ESF Programming: Promoting partnerships/Capacity building
3.1
Focus on PP/CB policy priorities at MS level
3.2
Target groups addressed
3.3
Objectives, types of interventions and clustering under PP/CB
priority
3.4
Contribution of PP/CB priorities to Europe 2020 and CSRs
4. Financial and physical performance
4.1
Financial performance of PP/CB-related OPs/PAs
4.2
Analysis of outputs (EU and MS level)
4.3
Analysis of results (EU and MS level)
5. Evidence from evaluation data
5.1
Sources, approaches and methods
5.2
Evidences
5.3
Main factors influencing interventions, results and impacts
6. In-depth assessment of ESF investment in PP/CB (case studies)
7.
May 2015
Conclusions: implications for future policies
page 77
Contract No. VC/2015/0098
3.4
The Final Synthesis Report
Box 7. Final Report Structure
Table of contents
Executive Summary
Introduction
 Policy context
 Aims
 Methods
1. Adaptability of workers
 Contextualising ESF
 The extent to which resources were used
 Effectiveness
 Efficiency
 Socio economic impact
 Community Added Value
 Gender sensitivity
 Sustainability
 Conclusion and lessons learned
2. Access and Sustainable Integration into Employment
 The extent to which resources were used
 Effectiveness
 Efficiency
 Socio economic impact
 Community Added Value
 Gender sensitivity
 Sustainability
 Conclusion and lessons learned
3. Social Inclusion
 The extent to which resources were used
 Effectiveness
 Efficiency
 Socio economic impact
 Community Added Value
 Gender sensitivity
 Sustainability
 Conclusion and lessons learned
4. Human Capital
 The extent to which resources were used
 Effectiveness
 Efficiency
page 78
May 2015
Contract No. VC/2015/0098
 Socio economic impact
 Community Added Value
 Gender sensitivity
 Sustainability
 Conclusion and lessons learned
5. Promoting Partnerships
 The extent to which resources were used
 Effectiveness
 Efficiency
 Socio economic impact
 Community Added Value
 Gender sensitivity
 Sustainability
 Conclusion and lessons learned
6. Strengthening Institutional Capacity
 The extent to which resources were used
 Effectiveness
 Efficiency
 Socio economic impact
 Community Added Value
 Gender sensitivity
 Sustainability
 Conclusion and lessons learned
7. Conclusions and lessons learned on the EU level policy on economic
and social cohesion
 policy choices
 target groups, in particular young people
 programming
 implementation
 monitoring
 evaluation of future programmes
May 2015
page 79
Contract No. VC/2015/0098
4. WORK ORGANISATION
This chapter presents the organization and management of the work with
regard to the following five aspects:
 Roles and responsibilities of the team
 Allocation of time and resources to the project and to the tasks,
including the allocation to each expert proposed
 The timetable and the overview of the workflow
 The quality assurance measures
4.1 Roles and responsibilities of the team
This sub-chapter presents the roles and respective responsibilities of the
proposed team, considering also the different roles and responsibilities of
the partners in the Consortium.
Figure 3 provides an overview of the structure of the work and the team,
including sub-task coordinators.
Figure 3. Overview of the team structure and work division
Metis GmbH leads the Consortium and both Consortium partners,
Fondazione Brodolini and Panteia, participate in the core team and take the
role of sub-task coordinators.
We have set up three teams: the core team, the overview team as support
to the core team and the research group, i.e. the national experts.
The distribution of work between the core team, the overview team and the
research group among the partners is shown in Table 14. We have allocated
page 80
May 2015
Contract No. VC/2015/0098
a total of 432 days to this synthesis. About 35% of the days have been
allocated to the core team, 38% to the overview team and 27% to the
country experts.
Table 14. Working days allocated to the partners in the Consortium
Research
Group
(Geographic
experts)
Total
Partner
Core Team
Overview
team
Metis GmbH
(lead)
65
115
92
272
34
6
80
Panteia
40
FGB
46
14
20
80
Total
151
163
118
432
4.1.1 Functions, responsibilities and work flows
There is a clear allocation of responsibilities between teams. To ensure
consistency, all core team members follow the synthesis work through all
stages.
The core team incorporates the experiences from the previous studies and
ensures that the synthesis is built on solid ground. There are four core team
members: Herta Tödtling-Schönhofer as project director, Isabel Naylon as
project manager and expert to cross-check the methodology and synthesis
work at a horizontal level, to ensure compatibility with the other three expost evaluations and to review the deliverables. Furthermore, we have
included Bert-Jan Buiskool (Panteia) and Costanza Pagnini (FGB) to
coordinate sub-tasks. All core team members are Category I or II experts.
The overview team includes senior experts with specific responsibilities for
individual tasks (e.g. Benno Savioli for the inventory, Diego Teloni for the
thematic reports, Douwe Grijpstra for the synthesis report), the quality
assurance (Haris Martinos). Furthermore there are three junior experts in
the team (Alexandra Frangenheim and Jakob Weiss (to be approved) from
Metis and Gert-Jan Lindeboom from Panteia), who will support the core
team members by compiling the available information.
The research group (country experts) are responsible for providing
additional information for task 2 and supplementary information for the two
thematic reports on partnership and capacities. Their work will be designed
by the core team and closely guided by the project manager, who will be
supported by Alexandra Frangenheim.
Herta Tödtling-Schönhofer will be the contract and project director for
this study. She is responsible for the overall performance of the contract,
for the appropriate quality and quantity of services delivered and will
May 2015
page 81
Contract No. VC/2015/0098
ensure the overall coherence and consistency of the reports. She will
organise all quality assurance measures for securing the delivery of all
reports with highest content-related, formal and linguistic quality. She will
participate in the kick-off meeting, the steering group meetings and the
meetings with DG Employment and any others as deemed necessary.
She will be supported by Isabel Naylon who will have the role of project
manager. She is the contact person for DG Employment for all operational
issues within the project. She is responsible for leading the Core Team,
guiding the Research Team and for the day-to-day co-ordination of the
work and the establishment of an information system within the project.
Isabel Naylon will be in charge of all three tasks outlined in the ToR. She
will coordinate and guide the work, always with a view on achieving
consistent, verifiable and clear results of this synthesis. However, different
subtasks will be allocated to core team members.
Task 1 lies with Isabel Naylon with the support of Bert-Jan Buiskool for the
database.
Task 2
For task 2, Isabel Naylon will elaborate the concept and the pilot pilot with
support of Jakob Weiss and Alexandra Frangenheim for the operational
tasks. Further sub-tasks within task 2 will be dealt with in the following
way:
The update of the inventory is the responsibility of Benno Savioli. Mr
Savioli has been designing and developing the inventories throughout the
EEN and is thus fully familiar with the content and technical features of the
inventory. Mr Savioli will prepare the files to be sent to the country experts,
check them for consistency and adapt the entire inventory.
Country experts will be asked to update the inventories and contribute
additional information through a review of the most recent evaluations.
Handling the information flows lies with Alexandra Frangenheim, a junior
Metis staff member, who has recently been dealing with similar
assignments.
The country reports have to be produced in three batches – with 10, 9
and 9 country reports to be delivered consecutively. Junior experts (Jakob
Weiss and Alexandra Frangenheim from Metis) from the overview team will
sort and compile the information. The responsible core team expert will
check their work for inconsistencies and gaps.
For the three batches of country reports we have assigned three different
core team members with key responsibilities. They will receive the compiled
material and edit the reports respecting the length, content and quality
required. Ms Naylon, Ms Pagnini and Mr Buiskool will take the responsibility
for editing one batch each.
page 82
May 2015
Contract No. VC/2015/0098
Task 3
Task 3 is on the EU level analysis and will be coordinated by Isabel
Naylon.
The two thematic reports will be drafted by Costanza Pagnini (FGB). She
is involved in the A2E ex post evaluation and thus highly familiar with ESF
evaluations. For both thematic reports she will also use input from the
overview team on the information already available and complementary
data and information from the country experts. Mr Teloni will review the
reports.
The EU synthesis report lies within the responsibility of Bert-Jan Buiskool
(Panteia). He is responsible for task 1 in the A2E evaluation and was also
involved in the Preparatory Study. He will use the compilation of the
information from the country reports (to be produced by Gert-Jan
Lindeboom from the overview team) for drawing up the synthesis report.
Douwe Grijpstra will discuss and review the report during drafting.
However, at the beginning and at the draft final stage, the entire core team
will be involved in the elaboration of the report.
Two rounds or checks of all the reports are foreseen (see also 0 and
chapter 4.4):
 Herta Tödtling-Schönhofer will check the coherence and consistency of
the reports. She is the project director of the A2E ex-post evaluation
and has had the same function for all three rounds of EEN. She is thus
familiar with all Member States and all issues related to ESF
evaluations.
 Haris Martinos, a very experienced consultant in the fields of
employment and labour market policies and ES will review and edit all
reports.
The back office in Vienna will provide secretarial and administrative
support. It will also ensure an appropriate layout of all deliverables.
May 2015
page 83
Contract No. VC/2015/0098
Figure 4. Generalized work flow
Task 2. National level analysis
Savioli
27+1 – updated
Inventories
Overview
team
Compilation
of available
inf ormation;
identif ication
of gaps
Core team
Research
group
(CE)
Pilot
country
and
template
f or additional inf or
mation
Overview
team
Country Synthesis
Reports
Batch 1 (10 CR)
Updated
complementary
inf ormation
Country Synthesis
Reports
Batch 2 (9 CR)
Compilation
of inf ormation
f rom task 2
and task 3A
Country Synthesis
Reports
Batch 3 (9 CR)
2 Thematic reports
(partnership, capacity
building)
EU Synthesis report
Task 3. EU level analysis
4.2 Allocation of time and resources to the project and to the
tasks
In this sub-chapter we present the details on the allocation of time and
resources. In order to ensure work and outputs of high quality, the
allocation was based on the following principles:
 All core team members are involved in all stages of the synthesis
 Sub-tasks have clear allocations of responsibilities
 Sufficient resources are foreseen for compilation of information
 Sufficient resources for quality and consistency checks
Table 14 presents the allocation of days to the tasks with a detailed breakdown of activities within each of the tasks.
Table 15. Allocation of working days to the tasks
Tasks
Task
1
Task
2
page 84
Follow-up of the ESF 2007-2013 thematic expost evaluation studies
Allocated
days
12
Compare reports
9
Suggest improvements for comparability
3
National level analysis
266
May 2015
Contract No. VC/2015/0098
Tasks
Task
2A
National inventories
Task
2B
Country synthesis reports
Task
3
EU level analysis
Task
3A
Synthesis report on partnership
47
219
111
23
Synthesis report on inst. Capacity
Task
3B
EU Synthesis report
Task
0
Organisation and management
Total
Allocated
days
35
53
41
Meeting with EC (ISSG)
16
Review and finalisation of reports, ex sums
27
432
Table 15 presents the working days allocated to each member in the team,
broken down by teams and tasks.
May 2015
page 85
Contract No. VC/2015/0098
Table 16. Working days allocated to each member in the team (including tasks)
Schönh
ofer
Metis
Tasks
Task 1
Task 2
Task
2A
Task
2B
Task 3
Task
3A
Task
3B
Task 0
Follow-up of the ESF
2007-2013 thematic
ex-post evaluation
studies
Compare reports
Suggest improvements
for comparability
National level analysis
National inventories
Country Synthesis
reports
EU level analysis
Synthesis report on
partnership
Synthesis report on inst.
Capacity
EU Synthesis report
Organisation and
management
Meeting with EC (ISSG)
Review and finalisation
of reports, ex sums
Total
page 86
Core team
Buiskoo
Naylon
Pagnini
l
Metis
Panteia
FGB
Overview team
Metis
Martinos
Metis
Grijpstra
Panteia
Savioli
Teloni
Weiss
FGB
Metis
Frange Linden-heim
bom
Metis Panteia
Country
experts
Total
I
I
II
II
Total
I
I
I
I
III
IV
IV
total
I/II
0
5
2
0
7
0
0
0
0
5
0
0
5
0
12
0
4
0
0
4
0
0
0
0
5
0
0
5
0
9
0
1
2
0
3
0
0
0
0
0
0
0
0
0
3
10
20
20
16
66
8
8
0
0
35
50
0
101
99
266
0
2
0
0
2
8
0
0
0
0
0
0
8
37
47
10
18
20
16
64
0
8
0
0
35
50
0
93
62
219
3
5
17
29
54
0
7
5
6
0
0
20
38
19
111
1
1
1
10
13
0
1
0
3
0
0
0
4
6
23
1
1
1
15
18
0
1
0
3
0
0
0
4
13
35
1
3
15
4
23
0
5
5
0
0
0
20
30
0
53
11
11
1
1
24
2
0
9
8
0
0
0
19
0
43
3
3
1
1
8
2
0
3
3
0
0
0
12
0
16
8
8
0
0
16
0
0
6
5
0
0
0
7
0
27
24
41
40
46
151
10
15
14
14
40
50
20
163
118
432
Contract No. VC/2015/0098
On a more detailed basis, the allocation of working days per expert is
shown in Table 16
Table 17. Working days per expert
Name
Company
Category
days
Tödtling-Schönhofer
Metis
I
24
Naylon
Metis
I
41
Buiskool
Panteia
II
40
Pagnini
FGB
II
46
Core team
Total
151
Overview team
Weiss
Metis
IV
40
Frangenheim
Metis
IV
50
Grijpstra
Panteia
I
14
Lindeboom
Panteia
IV
20
Savioli
Metis
I
10
Martinos
Metis
I
15
Teloni
FGB
I
14
Total
163
Research team (Country experts)
May 2015
Naylon
AT
I
3
Buiskool
BE
II
3
Devetakova
BG
I
4
Sofianopoulos
CY
I
2
Kaufmann
CZ
I
5
Savioli
DE
I
7
Schmidt
DK
I
3
Hagel
EE
I
4
Sanchez Esteban
ES
I
7
Arnkil
FI
I
2
Kolosy
FR
I
3
Karantinos
GR
I
4
Karzen
HR
I
5
Hars
HU
II
5
Humphreys
IE
I
3
Vergani
IT
I
6
Dumcius
LT
I
4
page 87
Contract No. VC/2015/0098
Name
Company
Category
days
Savioli
LU
I
2
Ramina
LV
II
3
Tonna
MT
I
2,5
Grijpstra
NL
I
3
Siekiera
PL
II
8
Perista
PT
I
4
Ionescu
RO
I
4
Tranquist
SE
II
3
Bozinac Mohorčič
SI
I
3
Huba
SK
I
4
Martinos
UK
I
4,5
Days allocated
111
Reserve
7
Total
118
Grand total
432
The country experts will be asked to provide information for the update of
the inventory, the country report (mainly extracting information from the
AIR 2014, in some cases also going back to the OP, and screening new
evaluations) and the synthesis reports on partnerships and capacities. We
have differentiated the allocation according to some estimates on the
number of OP, the complexity of the information to be generated and the
occurrence of partnership and administrative capacity. We have kept a
reserve of 7 days for unforeseen needs.
4.3 Timetable
Table 17 presents the timetable with milestones and details on each of the
tasks. For task 2 and task 3 detailed steps have been developed, which
have been explained in the previous chapters.
page 88
May 2015
Contract No. VC/2015/0098
Table 18. Timetable
Actions/deliverables
Date
Signature of the contract
11.02.15
Kick-off meeting
11.02.15
Draft inception report
25.03.15
Inception ISSG meeting
17.04.15
Final inception report
11.05.15
Pilot Country synthesis report
12.06.15
ISSG Meeting on Pilot Country
synthesis report
26.06.15
Final Pilot
report
synthesis
24.07.15
First batch of 10 draft Country
synthesis reports and national
inventories of evaluations
23.09.15
Second batch of 9 draft Country
synthesis reports and national
inventories of evaluations
23.10.15
Third batch of 9 draft Country
synthesis reports and national
inventories of evaluations
25.11.15
First batch of 10 final Country
synthesis reports and national
inventories of evaluations
11.12.15
Second batch of 9 final Country
synthesis reports and national
inventories of evaluations
23.12.15
Third batch of 9 final Country
synthesis reports and national
inventories of evaluations
11.01.16
Draft Thematic EU synthesis
reports
on
promoting
partnerships and strengthening
institutional capacity
11.01.16
Country
ISSG
meeting
reports
page 90
on
thematic
25.01.16
Final Thematic EU synthesis
reports
on
promoting
partnerships and strengthening
institutional capacity
25.02.16
Draft EU Synthesis report
25.03.16
Final ISSG Meeting
11.04.16
Final EU Synthesis report
11.05.16
May 2015
Contract No. VC/2015/0098
ANNEX 1: THE ALLOCATION OF PRIORITY AXES TO ESF PRIORITIES
Table 19. The allocation of pas in the thematic ex post evaluations and proposed allocation to promoting
partnerships and increasing administrative capacity
Country Country code CCI
OP name (EN)
Austria AT
2007AT051PO001 Operational Programme Burgenland 2007‐2013: Convergence objective /Phasing Out/ ESF Austria AT
2007AT052PO001 Belgium BE 2007BE051PO001 Operational Programme Employment Austria 2007‐
2013 ESF Operational Programme: Convergence Hainaut May 2015
Objective Priority Axis relevant to Adaptability policy field (sub‐
priorities/actions in brackets) Priority Axis relevant to Human capital policy field (sub‐
priorities/actions in brackets) Priority Axis relevant to Access to employment (sub‐
priorities/actions in brackets) Priority Axis dedicated to Social Inclusion Convergence
1
3 2.2 PA 3
NA
2.1 to A2E 2.2 to SI. Data is available at level 2. Regional Competitiveness 1
4 2 (since no financial, Annex XXIII, programme specific indicators are available on sub priority level we took the data for the whole PA 2 and allocate it to A2E) 2
PA 3
PA 5
NA
Convergence
1
2 No PA assigned
NA
NA
page 91
Promoting Partnerships Strengthening Institutional Capacity Comments Belgium BE 2007BE052PO001 Belgium BE 2007BE052PO002 Belgium BE 2007BE052PO003 Belgium BE 2007BE052PO004 Belgium BE 2007BE052PO005 page 92
ESF 2007‐2013, German‐
speaking Community of Belgium ESF Operational Programme: Regional Competitiveness and Employment ‐ Troika Wallonia (excluding Hainaut) Brussels ESF Federal Operational Programme: Regional Competitiveness and Employment ESF Operational Programme: Regional Competitiveness and Employment ‐ Brussels‐Capital Region: Employment and Social Cohesion Regional Competitiveness 1
4 2
PA 3
NA
NA
Regional Competitiveness 1
2 No PA assigned
PA 3
NA
NA
Regional Competitiveness no PA assigned*
no PA assigned* No PA assigned
PA 1 & 2
NA
NA
Regional Competitiveness no PA assigned*
no PA assigned* 1.2 and 1.3
3
NA
1.1 to A2E 1.2 and 1.3 to SI Data availability is to be confirmed "ESF Operational Programme ""Regional Regional Competitiveness 3
no PA assigned* PA 1 and PA2 (since no financial, Annex XXIII, programme specific indicators are available on sub priority level we took the data for the whole PA 1 and allocate it to A2E) PA 1
PA 2
NA
NA
May 2015
Contract No. VC/2015/0098
Bulgaria BG
Bulgaria BG
Cyprus CY Czech Republic CZ May 2015
2007BG051PO001 Competitiveness and Employment"" of Flanders" Operational Programme Human Resources Development 2007CY052PO001 OP Adminitsrtaive Capacity Employment, Human Capital and Social Cohesion 2007CZ052PO001 Operational Programme Prague Adaptability Convergence
2
2.1 3, 4 Convergence
NA
NA Regional Competitiveness Regional Competitiveness 1 1
3 PA 1, 2, 6, and 7(since no financial, Annex XXIII, programme specific indicators are available on sub priority level we took the data for the whole PA 2 and 5 and allocate it to A2E NA
No PA selected PA 2
page 93
5
PA 7
PA 6
NA
NA
PA 2 (2.4)
NA
2.1 to Adapt 2.2 and 2.3 to A2E 5 to SI Data is available at level 2 PA 1, PA 2, PA 3, PA 4 NA
?
Data is not available at level 2 (in‐
depth country in SI evaluation), PA 2 wholly to SI theme. 1.3 is SIC but already covered under HC PA 2 wholly to A2E, no PA for SI (very few SI activities in the OP) Czech Republic CZ Czech Republic CZ 2007CZ05UPO002 Germany DE
2007DE051PO001 Operational Programme ESF Brandenburg 2007‐2013 Germany DE
2007DE051PO002 Operational Programme ESF Mecklenburg‐
Vorpommern 2007‐2013 Germany DE
2007DE051PO003 Operational Programme ESF Lower Saxony ‐ Lueneburg region 2007‐
2013 Germany DE
2007DE051PO004 Operational Programme ESF Saxony 2007‐
2013 page 94
2007CZ05UPO001 Operational Programme Human Resources and Employment Operational Programme Education for Competitiveness Multi objectives
1 (1.1, 1.2)
Multi objectives
no PA assigned* 1, 2,3, 4 PA 2
No PA assigned
PA 3 (realised exclusively within the Objective Convergence) No PA assigned Convergence 79% of AD alloc to PA A, 17% to PA B, 4% to PA E
58% of PA A funded by AD, PA E = 21%, PA E =5% 96% of HC alloc to PA B 86% of PA B funded by HC PA C 99% of SI alloc to PA C
PA C = 46% funded by SI Convergence 100% of AD alloc to PA A, 74% of PA A funded by AD 97% of HC alloc to PA B 100% of PA B funded by HC No PA assigned 100% of SI alloc to PA C
PA C = 63% funded by SI Convergence 94% of AD alloc to PA A, 6% to PA E
94% of PA A funded by AD, PA E =40%, 93% of HC alloc to PA B 100% of PA B funded by HC No PA assigned 93% of SI alloc to PA C
PA C = 45% funded by SI Convergence 100% of AD alloc to PA A, 86% of PA A funded by AD 90% of HC alloc to PA B 100% of PA B funded by HC PA C 100% of SI alloc to PA C
PA C = 44% funded by SI May 2015
PA 5 (5.1)
PA 4 (4.1)
NA
2.4 belongs to SIC but is already covered under HC See table 2 NA
NA
NA
NA
NA
NA
Contract No. VC/2015/0098
Germany DE
2007DE051PO005 Operational
Programme ESF Saxony‐Anhalt 2007‐2013 Germany DE
2007DE051PO006 Operational Programme ESF Thuringia 2007‐
2013 Germany DE
2007DE052PO001 Operational Programme ESF Baden‐
Württemberg 2007‐2013 Germany DE
2007DE052PO002 Operational Programme ESF Bavaria 2007‐
2013 Germany DE
2007DE052PO003 Operational Programme ESF Berlin 2007‐
2013 Germany DE
2007DE052PO004 Operational Programme ESF Bremen 2007‐
May 2015
Convergence 92% of AD alloc to PA A, 8% to PA E
52% of PA A funded by AD, PA E =100%, 100% of HC alloc to PA B 41% of PA B funded by HC PA B 100% of SI alloc to PA C
PA C = 46% funded by SI Convergence 89% of AD alloc to PA A, 6% to PA E, 6% to PA B
67% of PA A funded by AD, PA E =32%, PA B =2% 93% of HC alloc to PA B 89% of PA B funded by HC, PA E =31% PA C 81% of SI alloc to PA C, 19% to PA B
PA C = 35% funded by SI, PA B =9% Regional Competitiveness 100% of AD alloc to PA A, 72% of PA A funded by AD 100% of HC alloc to PA B 84% of PA B funded by HC No PA assigned 100% of SI alloc to PA C
PA C = 64% funded by SI Regional Competitiveness 88% of AD alloc to PA A, 12% to PA C
64% of PA A funded by AD, PA C =4% 100% of HC alloc to PA B 97% of PA B funded by HC PA C 100% of SI alloc to PA C
PA C = 45% funded by SI Regional Competitiveness 100% of AD alloc to PA A, 61% of PA A funded by AD 100% of HC alloc to PA B 100% of PA B funded by HC PA C % 100% of SI alloc to PA C
PA C = 29% funded by SI Regional Competitiveness 100% of AD alloc to PA A, 64% of PA A 100% of HC alloc to PA B 68% of PA B No PA assigned 86% of SI alloc to PA C, 14% to B
page 95
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
2013 Germany DE
2007DE052PO005 Operational Programme ESF Hamburg 2007‐
2013 Germany DE
2007DE052PO006 Operational Programme ESF Hesse 2007‐
2013 Germany DE
2007DE052PO007 Germany DE
2007DE052PO008 Operational Programme ESF Lower Saxony ‐ without region Lueneburg 2007‐2013 Operational Programme ESF Nordrhein‐
Westfalen 2007‐
2013 Germany DE
2007DE052PO009 Operational Programme ESF Rhineland‐
Palatinate 2007‐
2013 Germany DE
2007DE052PO010 Operational Programme ESF Saarland 2007‐
2013 page 96
funded by AD funded by HC PA C = 84% funded by SI, PA B = 11% Regional Competitiveness 100% of AD alloc to PA A, 84% of PA A funded by AD 100% of HC alloc to PA B 100% of PA B funded by HC PA C 100% of SI alloc to PA C
PA C = 33% funded by SI Regional Competitiveness 100% of AD alloc to PA A, 100% of PA A funded by AD 100% of HC alloc to PA B 100% of PA B funded by HC No PA assigned 100% of SI alloc to PA C
PA C = 89% funded by SI Regional Competitiveness 100% of AD alloc to PA A, 85% of PA A funded by AD 100% of HC alloc to PA B 100% of PA B funded by HC No PA assigned 100% of SI alloc to PA C
PA C = 65% funded by SI Regional Competitiveness 100% of AD alloc to PA A, 100% of PA A funded by AD 100% of HC alloc to PA B 100% of PA B funded by HC No PA assigned 100% of SI alloc to PA C
PA C = 100% funded by SI Regional Competitiveness 100% of AD alloc to PA A, 86% of PA A funded by AD 100% of HC alloc to PA B 100% of PA B funded by HC No PA assigned 100% of SI alloc to PA C
PA C = 62% funded by SI Regional Competitiveness 58% of AD alloc to PA A, 42% to PA C
100% of PA A funded by AD, PA C =32%, 90% of HC alloc to PA B, 10% to PA E 100% of PA B funded by HC, 100% of PA E No PA assigned
100% of SI alloc to PA C
PA C = 55% funded by SI May 2015
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
Contract No. VC/2015/0098
Germany DE
2007DE052PO011 Operational Programme ESF Schleswig‐
Holstein 2007‐
2013 Regional Competitiveness 100% of AD alloc to PA A, 67% of PA A funded by AD 100% of HC alloc to PA B 85% of PA B funded by HC PA C
Germany DE
2007DE05UPO001 Multi objectives
DK
2007DK052PO001 100% of AD alloc to PA A, 41% of PA A funded by AD 1
100% of HC alloc to PA B 100% of PA B funded by HC 1 PA A
Denmark Operational Programme ESF BUND 2007‐
2013 More and better jobs Estonia EE 2007EE051PO001 Operational Programme for Human Resource Development Convergence
4
1, 2 May 2015
Regional Competitiveness PA 2
PA 3
page 97
53% of SI alloc to PA B, 47% to PA C
PA C = 40% funded by SI, PA B = 13% 100% of SI alloc to PA C
PA C = 62% funded by SI No PA allocated Not PA or action level, but code 71: Social inclusion NA
NA
NA
NA
NA
No SI allocation possible at the PA level and sub‐PA level, although some measures are relevant especially under PA2. Data is available at level 2 Sub‐priority 1.2 is PP but is already covered under HC 4 to Adapt 3 to A2E. SI is a horizontal priority, no allocation possible at PA and sub‐PA level, although some NA
PA 5
Spain ES 2007ES051PO002 Operational Programme ESF Castile and La Mancha Convergence
1
3 PA 2
Code 71
PA 4
PA 5
Spain ES 2007ES051PO003 Convergence
1
3 PA 2
Not specified
PA 4
PA 5
Spain ES 2007ES051PO004 Convergence
1
3 PA 2
Not specified
PA 4
PA 5
Spain ES 2007ES051PO005 Convergence
1
3 PA 2
Not specified
PA 4
PA 5
Spain ES 2007ES051PO006 Convergence
1
3 PA 2
Not specified
PA 4
NA
Spain ES 2007ES051PO007 Convergence
1
no PA assigned* PA 2
Not specified
PA 4
NA
Spain ES 2007ES051PO008 Convergence
1
3 PA 2
Not specified
NA
NA
Spain ES 2007ES051PO009 Convergence
1
3 PA 2
Not specified
NA
NA
Spain ES 2007ES052PO001 Regional Competitiveness 1
3 PA 2
Not specified
PA 4
NA
Spain ES 2007ES052PO002 Operational Programme ESF Extremadura Operational
Programme ESF Galicia Operational Programme ESF Andalusia Operational Programme ESF Asturias Operational Programme ESF Ceuta Operational Programme ESF Melilla Operational Programme ESF Murcia Operational Programme ESF Canary Islands Operational Programme Castile and León measures are relevant under PA1 and PA3. SI allocation to PA or action level difficult. Suggested approach: Code 71, applies to all regional OPs in Spain. Regional Competitiveness 1
3 PA 2
Not specified
PA 4
NA
page 98
May 2015
Contract No. VC/2015/0098
Spain ES 2007ES052PO003 Spain ES 2007ES052PO004 Spain ES 2007ES052PO005 Spain ES 2007ES052PO006 Spain ES 2007ES052PO007 Spain ES 2007ES052PO008 Spain ES 2007ES052PO009 Spain ES 2007ES052PO010 Spain ES 2007ES052PO011 Spain ES 2007ES05UPO001 Spain ES 2007ES05UPO002 May 2015
Operational Programme Valencian community Operational Programme Aragon Operational Programme Baleares Operational Programme ESF Cantabria Operational Programme ESF Catalonia Operational Programme ESF Madrid Operational Programme ESF Navarra Operational Programme ESF Basque Country Operational Programme ESF La Rioja Operational Programme ESF Adaptability and Employment Operational Programme ESF Fight Against Discrimination Regional Competitiveness 1
3 PA 2)
Not specified
Regional Competitiveness 1
3 PA 2
Not specified
PA 4
NA
Regional Competitiveness 1
3 PA 2
Not specified
PA 4
NA
Regional Competitiveness 1
no PA assigned* 3 PA 2
Not specified
PA 4
NA
Regional Competitiveness 1
3 PA 2
Not specified
PA 4
NA
Regional Competitiveness 1
3 PA 2
Not specified
PA 4
NA
Regional Competitiveness 1
3 PA 2
Not specified
NA
NA
Regional Competitiveness 1
no PA assigned* PA 2
Not specified
NA
Na
Regional Competitiveness 1
3 PA 2
Not specified
PA 4
NA
Multi objectives
1
3 PA 2
Not specified
PA 4
NA
Multi objectives
no PA assigned*
no PA assigned* PA 2
PA2
PA 4
NA
The OP included PA2, PA4 (transnational‐
interreg coop) and PA5 (technical page 99
Finland FI 2007FI052PO001 Mainland Finnish ESF OP 2007‐2013 EU Regional Competitiveness and Employment Regional Competitiveness 1
3 PA 2
PA 2 (35% of PA 2) Horizontal or PA 4 (transnational partnerships… NA
Finland FI 2007FI052PO002 Operational Programme ESF Åland 2007‐
2013 Regional Competitiveness no PA assigned*
no PA assigned* PA 1
no PA assigned* NA
NA
France FR 2007FR051PO001 Convergence
1
no PA assigned* PA 2
PA3
NA
NA
France FR 2007FR051PO002 Convergence
no PA assigned*
2 PA 1
PA3
NA
NA
France FR 2007FR051PO003 Convergence
2
no PA assigned* PA 1
PA3
PA 4
NA
France FR 2007FR051PO004 Convergence
no PA assigned*
1 No PA assigned
PA2
NA
NA
France FR 2007FR052PO001 Operational Programme ESF Martinique Operational Programme ESF Guadeloupe Operational Programme ESF Guyane Operational Programme ESF Reunion National Operational Programme ESF assistance). No data available below PA level PA 2 is called A2E and SI. The division proposed is 65% to A2E and 35% to SI – on the basis of the SI country report OP not covered in the EEN country report and SI thematic evaluation. Suggest to include OP in A2E Regional Competitiveness 1
4 PA 2
PA3
NA
NA
PA 4 is also PP but is addressed under HC page 100
May 2015
Contract No. VC/2015/0098
Greece GR
2007GR051RV001 Greece GR
2007GR05UPO001 Greece GR
2007GR05UPO002 Greece GR
2007GR05UPO003 Hungary HU
2007HU05UPO001 Hungary HU
2007HU05UPO002 Ireland IE 2007IE052PO001 Human Capital Investment Operational Programme Italy IT 2007IT051PO001 Italy IT 2007IT051PO002 ESF Regional Operational Programme Campania ESF Regional Operational Programme May 2015
National Contingency Reserve Operational Programme Human Resources Development Operational Programme Education and lifelong learning Operational Programme Administrative Capacity Operational Programme for Social Renewal State Reform OP
Convergence
no PA assigned*
no PA assigned* PA 1 and 2
no PA assigned* NA
NA
Multi objectives
2
no PA assigned* PA 1 and 3
no PA assigned* 4, 5 NA
NA
PA 2 to Adapt 1 and 3 to A2E Multi objectives
no PA assigned*
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 No PA assigned
no PA assigned* NA
NA
NA NA
NA
Multi objectives
2
3, 4, 6 PA 1
PA 5
NA
NA NA
NA
NA
All PA
Regional Competitiveness 2
no PA axis assigned 1.3, 1.4, 2.3, 2.4 and 2.5 PA 1 and 2
NA
NA
Convergence
1
4 PA B
Not specified. Social Inclusion (SI) is a horizontal priority in the OP and thus operations across the OP Suggest to use Code 71 for approximation PA 3 PA 5
PA 7
Convergence
1
4 PA B
PA 3 PA 5
PA 7
page 101
NA
All PA
1.3, 1.4, 2.3, 2.4 and 2.5 to HC Data is available at level 2 2007 ‐
Calabria Italy IT 2007IT051PO003 Italy IT 2007IT051PO004 Italy IT 2007IT051PO005 Italy IT 2007IT051PO006 page 102
2013 ESF Regional Operational Programme 2007‐2013 Sicily ESF Regional Operational Programme 2007‐2013 Basilicata ESF Regional Operational Programme Apulia National Operational Programme: Governance and System Actions Convergence
1
4 PA B
PA 3 PA 5
PA 7
Convergence
1
4 PA B
PA 3 PA 5
PA 7
Convergence
1
4 PA B
PA 3 PA 5
PA 7
Convergence
1
4 PA B
no PA assigned* PA F
PA E
(National OPs do not contain explicit reference to SI even where the Ministry of Education OP implicitly deals with SI through the improvement of quality and performance of educational systems in Convergence areas) May 2015
Contract No. VC/2015/0098
Italy IT 2007IT051PO007 National Operational Programme: Competencies for Development Italy IT 2007IT052PO001 Italy IT 2007IT052PO002 Italy IT 2007IT052PO003 Italy IT 2007IT052PO004 Italy IT 2007IT052PO005 ESF Regional Operational Programme Abruzzi ESF Regional Operational Programme Emilia Romagna ESF Regional Operational Programme Friuli Venezia Giulia ESF Regional Operational Programme Lazio ESF Regional Operational Programme Liguria May 2015
Convergence
1
4 PA B
no PA assigned* NA
PA 2
Regional Competitiveness 1
4 PA B
PA 3 PA 5
NA
Not specified (national OPs do not contain explicit reference to SI even where the Ministry of Education OP implicitly deals with SI through the improvement of quality and performance of educational systems in Convergence areas) Regional Competitiveness 1
4 PA B
PA 3 PA 5
NA
Regional Competitiveness 1
4 PA B
PA 3 PA 5
NA
Regional Competitiveness 1
4 PA B
PA 3 PA 5
NA
Regional Competitiveness 1
4 PA B
PA 3 PA 5
NA
page 103
Italy IT 2007IT052PO006 Italy IT 2007IT052PO007 Italy IT 2007IT052PO008 Italy IT 2007IT052PO009 Italy IT 2007IT052PO010 Italy IT 2007IT052PO011 Italy IT 2007IT052PO012 Italy IT 2007IT052PO013 Italy IT 2007IT052PO014 page 104
ESF Regional Operational Programme Lombardy ESF Regional Operational Programme The Marches ESF Regional Operational Programme Molise ESF Regional Operational Programme Autonomous Province of Bolzano ESF Regional Operational Programme Autonomous Province of Trento ESF Regional Operational Programme Piedmont ESF Regional Operational Programme Tuscany ESF Regional Operational Programme Umbria ESF Regional Operational Programme: Regional Competitiveness 1
4 PA B
PA 3 PA 5
NA
Regional Competitiveness 1
4 PA B
PA 3 PA 5
NA
Regional Competitiveness 1
4 PA B
PA 3 PA 5
NA
Regional Competitiveness 1
4 PA B
PA 3 PA 5
NA
Regional Competitiveness 1
4 PA B
PA 3 PA 5
NA
Regional Competitiveness 1
4 PA B
PA3
PA 5
NA
Regional Competitiveness 1
4 PA B
PA 3 PA 5
NA
Regional Competitiveness 1
4 PA B
PA 3 PA 5
NA
Regional Competitiveness 1
4 PA B
PA 3 PA 5
NA
May 2015
Contract No. VC/2015/0098
Italy IT 2007IT052PO015 Italy IT 2007IT052PO016 Italy IT 2007IT052PO017 Lithuania LT 2007LT051PO001 Luxembourg LU
2007LU052PO001 May 2015
Employment 2007‐2013 Valle d'Aosta ESF Regional Operational Programme Veneto ESF Regional Operational Programme Sardinia National Operational Programme: System Actions Operational Programme for the Development of Human Resources 2007‐
2013 Operational Programme ESF Luxembourg Regional Competitiveness 1
4 PA B
PA 3 PA 5
NA
Regional Competitiveness 1
4 PA B
PA 3 PA 5
NA
Regional Competitiveness 1
4 PA B
PA 3 PA D
NA
Convergence
1
1.1 2, 3 1.3
Regional Competitiveness 2
3 PA 1(since no financial, Annex XXIII, programme specific indicators are available on sub priority level we took the data for the whole PA 1 and allocate it to A2E) PA 1 (since no financial, Annex XXIII, programme specific indicators are available on sub priority level we took the data for the whole PA 1 and allocate it to A2E ( page 105
1.4
NA
PA 4
NA
1.1 to Adapt 1.2 to A2E 1.3 to SI Data is available at level 2 A measure under a subpriority is dedicated to PP NA
1.4 to SI Data is available at level 2 Latvia LV 2007LV051PO001 Operational Programme Human Resources and Employment Convergence
1.3.2. Measure
"Health at
work" 1.1, 1.2 PA 3
PA 1.4
Malta MT
2007MT051PO001 Operational Programme II ‐ Empowering people for more jobs and a better quality of life Convergence
2
1 PA 3 PA 3
Code 71 Netherlands NL
2007NL052PO001 Operational Programme ESF 2007‐2013 Regional Competitiveness 3.D, 3.E PA 1
2.B + 2.C
page 106
May 2015
NA
NA
PA 1.5
PA 4
NA
NA
Data is available at level 2 PA 2 to Adapt, PA 3 to A2E Data is not available at level 2 One operational objective is addressed to PP Data is available at level 2. PA 2c in NL should fall under social inclusion. Part of the Social Inclusion PA (PA2) containing Actions B (detainees) and C (youth in special education). In ex post evaluation concerning social inclusion of disadvantaged Contract No. VC/2015/0098
Poland PL 2007PL051PO001 Human Capital Operational Programme Convergence
2, 8
3, 4, 9 PA 1 and 6
Portugal PT 2007PT051PO001 Convergence
NO PA assigned
1 2
NO PA assigned no PA assigned* Portugal PT 2007PT052PO001 Operational Programme Employment Azores Valorisation of Human Potential and Social Cohesion 1 PA 2
no PA assigned* May 2015
Regional Competitiveness page 107
7
NA
PA 5
NA
NA
NA
groups this has been labelled social inclusion. In HC ex post evaluation as HC, but the desk does not agree and thinks it should be considered social inclusion. PA 2 removed from A2E (more relevant to Adapt) PA 6 wholly to A2E PA 7 wholly to SI PA 6 (6.3), and 7 (7.3) would also go to PP but PA 6 and 7 are already covered under A2E and SI PA NA
Madeira
Portugal PT 2007PT05UPO001 Romania RO
2007RO051PO001 Romania RO
Sweden SE 2007SE052PO001 Slovenia SI 2007SI051PO001 Slovakia SK 2007SK05UPO001 page 108
Operational Programme Human Potential 2007‐
2013 Sectoral
Operational Programme Human Resources Development 2007 ‐ 2013 OP ACD –
Operational Programme Administrative Capacity Development National Structural Fund for Regional Competitiveness and Employment (ESF) Operational Programme, Development of human resources for the period 2007‐2013 Operational Programme Education Multi objectives
3, 8 (aprt from 8.6), 9 (apart from 9.6) 1, 2, 4 Convergence
2 (2.3), 3 (3.1, 3.2) 1 (1.1‐1.5), 2 (2.1, 2.2) Convergence
2
PA 5, 7 PA 4 PA 5 PA 6, 8.6 (Algarve) & 9.6 (Lisboa) NA
PA 6 PA 3.3, PA 6.4
NA
All PAs
Regional Competitiveness 1.3 Prevention of long‐term sickness absenteeism 1.1 Skills training for development in line with working life demands PA 2 (no sub‐
priorities) 1.2 Counteract discrimination and promote equality Convergence
1
3 PA 2 4 PA 5.3
Multi objectives
no PA assigned*
1 to 4 no PA assigned* NA
no PA assigned*
May 2015
Data available at level 2 PA 5 ( 5.1, 5.2) NA
Financial data and data on outputs is available at level 2, results data on level 2 is not available PA 2 wholly to A2E, 4 wholly to SI Contract No. VC/2015/0098
Slovakia SK 2007SK05UPO002 Operational Programme Employment and Social Inclusion Multi objectives
No PA assigned*
no PA assigned* PA 1 and 3 NA
PA 4
Mainstreamed throughout PA 1, 2 and 3 – concentrated but not exclusively in Priority 1 Mainstreamed throughout PA 1 and 2 NA
NA
) United Kingdom UK
2007UK051PO001 Highlands and Islands of Scotland ESF phasing out Convergence programme Convergence
2
3 PA 1
United Kingdom UK
2007UK051PO002 West Wales and the Valleys ESF Convergence programme Convergence
3
no PA assigned* PA 1 and 2
United Kingdom UK
2007UK052PO001 East Wales ESF Regional Competitiveness and Employment programme Regional Competitiveness 2
no PA assigned* PA 1
United Kingdom UK
2007UK052PO002 Regional Competitiveness 2
3 PA 1 and 5
United Kingdom UK
2007UK052PO003 Lowlands and Uplands of Scotland ESF Regional Competitiveness and Employment programme Northern Ireland ESF Regional Regional Competitiveness 2
no PA assigned* PA 1
May 2015
PA2
page 109
PA1
NA PA 4 Data is available at level 2 PA 3 is to be split between A2E and SI in the research For SI, use categories of funding (69, 70, 71) as proxy NA
NA
Mainstreamed throughout PA 1, 2 and 3 – concentrated but not exclusively in P1 NA
NA
Mainstreamed throughout PA 1 and 2 ‐ NA
NA
For SI, use categories of funding (69, 70,71) as proxy For SI, use categories of funding (69, 70,71) as proxy For SI: no allocation by categories of United Kingdom UK
page 110
2007UK05UPO001 Competitiveness and Employment programme England and Gibraltar ESF Convergence, Competitiveness and Employment Programme Multi objectives
2
no PA assigned* PA 1 and 4
May 2015
concentrated but not exclusively in Priority Axis 1 Mainstreamed ( PA 1, 2, 4 and 5) – concentrated in PA 1 and PA 4 expenditure available include PA1 NA
NA
For SI, use categories of funding (69, 70,71) as proxy