Human Resource Development Operational Programme Complement,
Annex I - Evaluation Strategy
EVALUATION STRATEGY
Human Resource Development
Operational Programme Complement
VERSION 1.2
JUNE 2005
17. 6. 2005
Number of Edition: 1
Number of Revision: 2
Page 1 of 12
Human Resource Development Operational Programme Complement,
Annex I - Evaluation Strategy
NTRODUCTION
Regular evaluation in various stages of preparation and implementation of the
programme is one of the European Commission's essential requirements for
quality assurance and efficiency of use of the aid administered from Structural
Funds. Therefore, the Managing Authority will provide all types of mandatory
evaluation associated with the programme period of 2004-2006. At the same
time, the Managing Authority will also use other procedures beyond the level of
essential obligations to ensure the systematic nature and quality of evaluation as
well as creating an expert capability for the purpose.
The HRD OP for the 2004-2006 period substantiates the first opportunity to use
ESF finance for the Czech Republic. Evaluation will therefore be a very
significant tool with regards the confirmation of the appropriateness of defined
priorities, the performance of existing or newly-established implementation
structures, the absorption capacity of project applicants, the methods and
procedures of project selection and control. Evaluation will be a crucial starting
point for programme preparation for the following planning period.
Besides the established (i) quantity monitoring of individual projects as well as
the programme, there will be (ii) tracking studies to find complementary
information and other quality-related data on the results and impact of
programme measures, (iii) cross-measure studies to find the extent of fulfilment
of the horizontal themes and (iv) surveys to confirm the quality and capacity of
the implementation and management processes.
1. GENERAL PROCEDURES OF EVALUATION
In accordance with European Commission requirements associated with
efficiency measurements of the use of Structural Funds finance (Resolution No.
1260/99, Articles 40-43), programme evaluation will be performed at all stages
that are binding upon the Czech Republic; this means preliminary (ex-ante)
evaluation, final evaluation and ex-post evaluation.
Considering the fact that the Operational Programme is planned for the short
period of 2004 – 2006 only, the Czech Republic has no obligation to provide midterm evaluation.
Preliminary evaluation (ex-ante)
The ex-ante evaluation process will contribute towards the quality improvement
of programme documents, according to the European Commission requirements
and the relevant instruction. Therefore, the ex-ante evaluation is more of a
comment process, or a discussion with those dealing with the programme and
formulating recommendations throughout the entire course of programme
document compilation.
17. 6. 2005
Number of Edition: 1
Number of Revision: 2
Page 2 of 12
Human Resource Development Operational Programme Complement,
Annex I - Evaluation Strategy
The Initial Report of ex-ante evaluation has been compiled that contains
evaluation methodology, procedures and time schedules. The Report has been
discussed by the Evaluation Work Group (EWG). The methodology has been
handed over to programme team leaders so that the specified requirements can
be tracked, even during the preparation of the Human Resource Development
Operational Programme and its Complement.
The general ex-ante evaluation report is a part of the programme document. It
summarises all improvements achieved based upon the Evaluation team‘s
recommendations and assesses the document from the following points of view:
the appropriateness of the focus of the Operational Programme strategy and its
relationship to the relevant national and European strategies, the internal
consistency and logic of the programme's inner connections, the quality of
definition and the level of quantification of the determined objectives, priorities
and measures, the representation of horizontal themes, the expected social and
economic impact and adequacy of funding allocation, the quality of the
implementation and monitoring mechanism, the relationship of the Human
Resource Development Operational Programme to other operational
programmes from the point of view of preventing duplication and ensuring
synergy.
Evaluation of achieved progress up-to-date in the implementation of OP
HRD as a part of final evaluation
According to the methodological guidelines of the Commission (The 2000-2006
Programming Period: Methodological Working Papers WORKING PAPER no. 9
The Update of the Mid Term Evaluation of Structural Fund Interventions, DG
Regio) there is no requirement for the new member states to realize mid-term
evaluation of OP HRD interventions. However, it is recommended that new
member states should focus on building internal evaluation capacity within the
managing authorities in the short programming period 2004-2005. Commission
also suggests to undertake a limited number of evaluations over the period which
would add value to the implementation of the current programmes and provide
useful learning for 2007-2013. In the context of this activity, the Commission
recommends that the new Member States should, if possible, undertake an
update of progress in the implementation of their programmes in the same time
frame as the mid term evaluation update for the older Member States.
In regard to these recommendations a limited number of evaluations will be
undertaken. One of the evaluations will be the achieved progress in the
implementation up-to-date. Other evaluations will be thematic evaluation studies.
Their content will be specified by the Evaluation Work Group in co-operation with
the Managing Authority.
The update of achieved progress, which is at present being commissioned, will
focus on detailed analysis of outputs and achieved first results to date with
relation to defined goals and financial performance on the level of priorities with
17. 6. 2005
Number of Edition: 1
Number of Revision: 2
Page 3 of 12
Human Resource Development Operational Programme Complement,
Annex I - Evaluation Strategy
specific focus on core indicators. The estimation of anticipated and feasible
results by the end of programming period should also be undertaken. The
analysis will link expenditures outlaid to date (inputs) with the outputs and
outcomes, in other words cost analysis of interventions will be undertaken. It
should allow to draw conclusions about the effectiveness and efficiency of the
programme interventions and it will also provide useful information on quality of
defined goals in the programme documents and how they could be adjusted and
improved in the future. One of the parts of this evaluation exercise is also
assessment of implementation system and monitoring system and identification
of problematic areas in the implementation of the programme and formulation of
suggestions for the next programming period. The aim is to provide to the
Managing Authority a feedback which should allow to identify hitches and ensure
their correction. Financial and real performance will also be evaluated.
This evaluation for OP HRD and SPD 3 could be concieved as a component of
final evaluation which is mentioned in the programme document and it should
also provide inputs for the partial mid-term evaluation at the CSF level which has
been started.
One of the suggested thematic evaluation studies shall, within the limits of the
short programming period, evaluate results and impacts of the programme
among individual beneficiaries. The aim of this tracking study should be to
acquire qualitative data and detailed information about the effects of delivered
support among target groups, to acquire knowledge about different means of
support for target groups and gather differential perception of obtained support
among individual beneficiaries, their opinions and views on the obtained help.
The methods for this study could be directed inquiries, analysis of secondary
data from job centers and analysis of statistical data. These analyses should be
supplemented by qualitative research, i.e. questionaires and observations, case
studies, focus groups etc. Themes of these analyses could be for example
experience with discrimination on the labour market and/or reconciliation of work
and family life. Tracking study of individual beneficiaries should provide data and
basis for subsequent comparative or longitudinal studies.
Ex-post evaluation
This evaluation is the responsibility of the European Commission, which cooperates with the Managing Authority. Evaluation is performed by an
independent evaluator chosen through a public tender. Funding will be provided
from European Commission funds.
The subject of ex-post evaluation is the use of the funds, the efficiency and
effectiveness of programme objective fulfilment and the programme impact from
the point of view of benefit to economic and social cohesion. Ex-post evaluation
must be executed within three years of the end of the programme period, that is
by the end of 2009.
17. 6. 2005
Number of Edition: 1
Number of Revision: 2
Page 4 of 12
Human Resource Development Operational Programme Complement,
Annex I - Evaluation Strategy
Preliminary evaluation (ex-ante) for the following programme period
With respect to the fact that, according to the expected planning steps, the
programme document for the following programme period will have to be ready
by the end of 2006, it is necessary to take into consideration that the ex-ante
evaluation of the following programme document will have to take place by,
approximately, mid-2006. Funding will be provided from the funds allocated as
part of evaluation within the programme period of 2004 – 2006.
2. ORGANISATIONAL FRAMEWORK OF EVALUATION
The Managing Authority, in Concordance with the European Commission,
is responsible for evaluation (except for ex-post evaluation). Conditions,
guidelines, general evaluation strategy, including appropriate monitoring and
evaluation indicators for the programme, are also approved by the Managing
Authority. The Monitoring Committee discusses the evaluation report results
and issues decisions based upon the evaluation reports. Besides that, the
Monitoring Committee also discusses and approves final and annual reports on
programme implementation.
The organisational framework of evaluation respects the fact that evaluation
activities are demanding in terms of expert background and skills in the area of
analysis and knowledge of national as well as European policies.
Simultaneously, independence of processing and implementation of the
programme on the one hand and its evaluation on the other hand must be
guaranteed.
Therefore the proposed organisational framework is as follows:
• The Managing Authority will be responsible for the entire evaluation process
and its organisation. The MA in co-operation with Evaluation Work Group will
be responsible for the formulation of tasks (TOR), for the selection of
independent external evaluators and for monitoring of task fulfilment within
the framework of evaluation. The MA will discuss the relevant issues with the
Monitoring Committee and with the European Commission and will cooperate with the Evaluation Work Group.
•
Independent external evaluators will be selected for each project through a
tender by the Managing Authority and will work out thematic evaluations,
evaluation of achieved progress to date, tracking studies, inquiries and
analyses. Results from the studies will be available for the Managing
Authority and for Evaluation Work Group, with which the evaluators will cooperate on methodological and other issues.
17. 6. 2005
Number of Edition: 1
Number of Revision: 2
Page 5 of 12
Human Resource Development Operational Programme Complement,
Annex I - Evaluation Strategy
• The Evaluation Work Group
The Evaluation Work Group for the HRD and Employment Programme
Evaluation was founded according to the European Commission
recommendation1 in June 2002 by the Human Resource Development
Programme Monitoring Committee as its technical body. The activity framework
of the work group was defined by the statutes and focused in particular on the
development and harmonisation of evaluation methodologies and procedures in
all programme evaluation stages (that is ex-ante evaluation, mid-term evaluation
and final evaluation), on evaluation result assessment, the publication of
information and other activities leading towards the development of an expert
capability and the exchange of experience and working examples.
The EWG primarily assessed the draft of the evaluation procedures and
methodologies, the structure and definition of the monitoring indicators and the
ex-ante evaluation of the HRD OP and SPD report for Objective 3. On the basis
of this EWG activities the renewed Evaluation Work Group with new members
has been established according to Monitoring Committees’ rules of procedures,
which will operate conjunctively under all of the three Monitoring Committees for
OP HRD, SPD 3 and CIP EQUAL. The activities of the Group are defined by its
statutes and rules of procedure. European Commission has also a representative
in the group.
Moreover, in order to co-ordinate activities associated with the evaluation and
search for a joint solution across the operational programmes and the
Community Support Framework, the Evaluation and Technical Assistance Work
Group of the CSF has also been established.
The group consists of
representatives of individual operational programme Managing Authorities and
the CSF.
3. MONITORING
Evaluation demands relevant information being provided in a timely manner.
Quantity-related data will form the foundation of the information used in
evaluation and will be collected within the framework of monitoring.
Data collection:
Data collection is the responsibility of the Managing Authority that is responsible
for the implementation of the uniform information system MSSF (Monitoring
System of the Structural Funds) to collect reliable statistical and financial data
and information on the implementation of aid from the HRD OP from all entities
that monitor performance against the programme. Monitoring data are collected
from the application forms and final reports of the projects as well as being the
subject of regular reporting from the Final Beneficiaries. All data on projects and
1
Guidelines for systems for monitoring and evaluating ESF assistance in the period 2002 - 2006
17. 6. 2005
Number of Edition: 1
Number of Revision: 2
Page 6 of 12
Human Resource Development Operational Programme Complement,
Annex I - Evaluation Strategy
programmes (including the defined indicators) are stored in the MSSF Monit and
with that are transmitting to the database MSSF Central maintained by the
Ministry for Regional Development (MRD), which is responsible for the
monitoring of structural funds. Methodological management and database
maintenance is performed by the MRD. The database will allow three types of
access: sector access, access outside the sectors, regional access. Independent
evaluators and other evaluators will have guaranteed access to the
required/requested data from the database. Data for the evaluation requirements
will be available in pre-defined data compilations that will form a part of the
MSSF as well as in ad-hoc compilations that will allow focused processing of
data from the central database.
• The procedures for defining the monitoring indicators for the HRD OP were
based upon the partnership approach and consultations with the
participating Ministries. Consultations also took place within the framework of
the Evaluation Work Group, which discussed the issues of selecting the
relevant indicators with respect to their relevance to the programme contents
and results and individual programme priorities. The methodology for
expressing the brutto and netto effects of the programme and priorities was
also discussed.
Monitoring indicators:
•
Input, output, result and impact indicators are defined for all levels, that is for
the level of the programme, individual priorities and individual measures.
•
The common core indicators are defined as a given minimum number of
output character indicators for which a uniform methodology and definition
of measuring, including up to the programme level, has been established.
The indicators have been defined to allow a more detailed characterisation of
individuals supported by the programme, organisations that won a subsidy
and the number of projects within the framework of individual measures.
The grouping of individuals allows for distinguishing between services clients
from services providers and individuals promoting services who can also win a
subsidy within the framework of the given measure. All individuals (clients as
well as providers) are grouped according to sex (total, men, women).
Individuals - clients are further divided so that the programme focus might be
evaluated according to the principle of equal opportunity in the labour market.
For that reason the individuals are grouped only into basic groups that show a
certain level of disadvantage in the labour market: young people aged 25 or
under, elderly people over 50, individuals with a physical disability, long-term
unemployed individuals, those with low qualifications. According to their
standing in the labour market, the individuals - services clients are grouped
into the following categories: employees, self-employed, long-term
unemployed, other unemployed, other individuals.
17. 6. 2005
Number of Edition: 1
Number of Revision: 2
Page 7 of 12
Human Resource Development Operational Programme Complement,
Annex I - Evaluation Strategy
The supported organisations are grouped according to the type (large
enterprises, medium and small enterprises, public administration bodies, other
public organisations, NGOs, other) and according to the character of their
activities (educational and advisory organisations, organisation providing
social services, organisations providing employment services, organisations
promoting those services, other).
It is obvious from the afore-mentioned descriptions that the indicators
differentiate between the direct support of individuals and organisations and
indirect support administered within the framework of measures that focus on
system improvements. The indicators that monitor the supported service
providers (individuals as well as organisations) within the framework of system
measures are considered significant at this stage since they allow a view of
the development of capacity of the relevant service sector and therefore also
of the future absorption capability, quality improvement and efficiency of the
future programmes associated with the ESF.
•
A part of the core indicators is also a group of indicators that show the
fulfilment level of horizontal themes. To track the development of the
information society, an output indicator of the number of participants in
courses with an information technology component has been introduced, as a
result indicator the number of newly created courses with an information
technology component and as an impact indicator the proportion of
employment in the ICT sector in the overall employment statistics. Tracking
environmental aspects is provided at the output level by the indicator of the
number of participants in courses with an environmental component and at
the result level by the number of newly created courses with an environmental
component. To track equal opportunity the numbers of women are
consistently stated for all indicators related to individuals, including data for
specific population groups. Horizontal themes will also be monitored through
the following context indicators: the percentage of the population that
regularly use the Internet, the proportion of recyclables in home waste and
the proportion of women entrepreneurs.
•
Data collection periodicity will respect both evaluation requirements and the
frequency of relevant data acquisitions within the programmes of short
duration. Input indicator collection (financial data) and output indicator
collection will be performed quarterly because these are the categories of
most frequent data exchange. Result indicators will be collected annually.
Impact indicators usually depend on data from tracking studies. Therefore,
these indicators will be collected during 2005 due to the short programme
period and the need to provide information for the final evaluation (see below
under tracking studies).
•
The core indicators database will allow the aggregation of indicators from the
measure level on to the individual priority level and further on to the
17. 6. 2005
Number of Edition: 1
Number of Revision: 2
Page 8 of 12
Human Resource Development Operational Programme Complement,
Annex I - Evaluation Strategy
programme level. Individuals in charge of evaluation will be allowed access to
the database. The Managing Authority will guarantee that the database
provides data at all times and allows for flexible compilations according to any
chosen analytical aspects for evaluation purposes.
4. OTHER DATA SOURCES FOR EVALUATION
Besides the data acquired within the framework of monitoring, evaluation will also
use other sources of information. These sources will primarily provide qualityrelated information that impact upon individual specific groups of programme
participants, information proving the permanent nature of effects and information
on contextual conditions that affect programme results.
•
Reports and conclusions of inspections of those who implement End User
projects
•
Tracking studies – surveys amongst the participants from target groups:
The studies will confirm the programme impact on individuals according to
their participation in various types of measures, according to their belonging
to specific defined groups, etc. Forms of impact, social consequences of the
programme, relevant aspects of horizontal themes (especially specific
benefits to equal opportunities, contributions to the information society and
access to information technology). Surveys will be carried out via direct or
telephone interviews.
•
Survey amongst supported institutions (Labour Offices, small and medium
enterprises, educational institutions, NGOs). Surveys will be carried out as
electronically distributed questionnaires in the case of public institutions
where a very good return rate is ensured. In the case of company surveys the
vehicle used will be direct interviews and telephone enquiries.
•
Statistically tracked data that characterises the labour market development,
employment policy, and education (Czech Statistical Office, Labour Force
survey, Ministry of Labour and Social Affairs - work OK system, Institute of
Information in Education) will be used for contextual analysis and the
definition of average or starting level as the control level.
•
Studies to confirm implementation quality (partnerships, success at
inspections, financial flows, project selection, exchange of information) will be
carried out as:
- Questionnaires mailed or e-mailed to project managers/project applicants
- Surveys amongst experts in the decision-making area and in key
institutions (a combination of mailed questionnaire method and telephone
enquiry method will be used)
17. 6. 2005
Number of Edition: 1
Number of Revision: 2
Page 9 of 12
Human Resource Development Operational Programme Complement,
Annex I - Evaluation Strategy
•
Theme studies - will confirm the programme benefits with respect to
fulfilment of horizontal themes. They will be based upon monitoring data, data
from tracking studies (see above), statistical data and, should it be necessary,
from data acquired in special surveys required by the given study.
•
Expert surveys – questionnaire surveys (combination of mailed
questionnaires and telephone enquiries) of experts focusing on a special
topic, confirmation of specific impact of the programme.
•
Focus groups - composed of various groups of experts and key partners
focusing on an assessment of program efficiency with respect to the defined
priorities and specified problems. Attempts will also be made to compare the
program impact in regions with varying social and economic environments.
With respect to the necessity to process the final evaluation within a very short
time period (by the end of 2005 – for details see item 1), an appropriate scope for
the tracking studies, theme studies and surveys executed will have to be
determined.
5. GENERAL TIME SCHEDULE
Ex-ante evaluation of the programme document…………….....…..January 2003
Preparation of study, survey, etc. methodology ……………..…..3/2004 – 6/2005
Monitoring data analysis ……………………………….....….....end of 2004, 2005
Tracking studies of individuals - programme participants ……………end of 2005
Survey of institutions…………………………………………………......end of 2005
Focus group organisation …………………………………………….………… 2005
Implementation study .….......……………………………………………...........2005
Theme studies…... …………………………………………………….……….…2005
Final evaluation .....…………............………………………………..........…12/2005
Ex-ante evaluation for the following programme period..………………...…9/2006
6. PUBLICATION OF THE EVALUATION RESULTS AND INFORMATION
Evaluation reports should be structured in such a way as to allow the use of
evaluation results for the following purposes:
•
Informing the Managing Authority, Monitoring Committee and European
Commission on the programme evaluation results;
•
Recommendations for the following programme period: recommendations on
changes in programme contents (what to change, what to enhance and what
to reduce), recommendations on programme management, recommendations
on monitoring;
•
Recommendations towards the National Employment Policy;
17. 6. 2005
Number of Edition: 1
Number of Revision: 2
Page 10 of 12
Human Resource Development Operational Programme Complement,
Annex I - Evaluation Strategy
•
Analysis towards and establishment of a working example and experience
repository.
Providing the greatest evaluation result publishing rate possible is an important
element of the evaluation strategy in several levels. The objective is to promote
both the enhancement of programme efficiency, as such, and the use of its
results, as well as to increase knowledge levels of the public, especially of the
potential project applicants. That will increase the absorption capability for the
involvement of the European Social Fund which is, under the current conditions
of the less developed institutional background of the Czech Republic, one of the
important aspects. Publication of the evaluation results will be administered by
the Managing Authority within the framework of the general programme
information campaign strategy.
Evaluation results will be publicised by means of:
•
Web-pages that will feature a link to the Ministry of Labour and Social Affairs‘
web-site and to the Evaluation Work Group web-site. Those interested will be
able to obtain evaluation reports, figures suitable for benchmarking,
evaluation analyses and tracking study conclusions, monitoring report
conclusions;
•
Material containing generalisation of experience and elaboration of working
examples;
•
Seminars and workshops;
•
Discussions within the framework of expert theme groups;
•
Participation in international discussion.
7. EXPENSES ASSOCIATED WITH THE EVALUATION STRATEGY
In the European Commission material (Working Paper No 8, page 10) it is
specified that financial funding allocated to evaluation may not be less than
0.1 per cent of the operational programme‘s annual budget.
However, in the Czech Republic's specific conditions it is necessary to consider
the fact that programme evaluation is only beginning and that it will be necessary
to apply adequate funding to the evaluation tool development, tracking study
initiation, initiation of studies focused on companies and institutions participating
in the programme, initiation of surveys monitoring the system impact, initiation of
theme studies, surveys and questionnaire surveys. Thus the expenses will be
associated not only with the execution of all the necessary studies and surveys
but appropriate funding will have to be provided for the preparation of
methodology (for example, preparation for the initiation of tracking studies took
18 months in Great Britain), specification of the size and structure of respondent
samples (to guarantee objectivity of the survey and thus also the statistical
relevance), unification of procedures, etc. From the point of view of evaluation
17. 6. 2005
Number of Edition: 1
Number of Revision: 2
Page 11 of 12
Human Resource Development Operational Programme Complement,
Annex I - Evaluation Strategy
objectives, it will be necessary to test the monitoring system, the appropriateness
of numbers and the structures of the indicators, etc. and propose any potentially
necessary changes.
For the afore-mentioned reasons, the start of the entire evaluation system will be
extremely demanding. The proportion of funding for evaluation, therefore, should
be at least 0.25 per cent of the total programme budget so that it can cover all
the activities necessary for a good-quality programme evaluation.
Besides these costs, costs of communication with the MSSF database (Benefit,
Monit, Central) also have to be calculated within the framework of the technical
assistance expenses. These are activities associated with obtaining data from
the database in compilations customised for evaluation requirements and for
providing feedback between the evaluation results and possible modifications in
the structure of indicators. These activities should be guaranteed by the
Managing Authority, which should ensure that the appropriate funding is reserved
within the framework of technical assistance for these purposes.
17. 6. 2005
Number of Edition: 1
Number of Revision: 2
Page 12 of 12