Consultation Strategy for the PAR Evaluation
A. Approach
1.
The present note outlines a comprehensive consulting strategy for the “Thematic evaluation
on IPA and ENI support to Public Administration Reform.” It aims at reaching out to the highest
possible number of relevant stakeholders, trying to ensure a balanced geographical coverage. The
relevant guidelines for consultations are covered under chapter 7 of the better regulation guidelines.1
B. Stakeholder Consultations to collect needed information
2.
This evaluation will involve two types of consultations. The first will involve exchanges
internally (i.e.; EU staff) and externally (i.e.; mainly government officials both policy makers and
beneficiaries at desk stage, to be expanded to other development partners, the civil society and others
at a later stage during field visits in the context of country case studies). The objective of these
exchanges is to obtain information on the various aspects of issues listed under evaluation questions.
This process is especially important in the case of evaluation because even though most EU
strategies and relevant regional programmes have been independently evaluated (in the past 4 years);
few if any operations have been completed. This means that the consultants will have to rely on
selective stakeholder feedback to gain an understanding of implementation experience and
ownership, results, impact and sustainability, and ultimately what worked and what did not and why.
3.
The team of evaluators are aware that such an approach entails risks:
(i) Issue of sample choice and self-selection. Important stakeholders may not be available or not
identified, and those who respond (typical response rate in such case is usually below 50%)
may not be the most representative.
(ii) There is a lot of ground to cover as each country includes up to a dozen relevant operations.
4.
These risks will be mitigated as follows:
1.
A mapping of stakeholders (discussed below) will be carried out during the inception phase
and should result in the identification of the most important counterparts, especially in government.
2.
Most members of the evaluation team are familiar with the countries being covered and their
officials. They will make extensive use of this prior knowledge to identify the right specific national
counterparts.
3.
Given focus is on PAR the fact that dozens of operations may have been implemented during
the review period can be managed because counterpart institutions would generally be stable over
time. However, as one goes back in time, finding stakeholders knowledgeable of actions that took
place over 5 years ago may prove a challenge – especially where turnover of public officials and civil
servants is high. In practice, many of those consulted in the context of the evaluation are likely to be
those with more recent knowledge of PAR and EU contributions. This bias may be reduced through
field work.
4.
Very small operations will not be given the same weight as larger ones (especially as our
preliminary analysis suggests they are poorly documented (no Results-Oriented Monitoring reports).
C. Stakeholder Mapping
5.
The stakeholder mapping will be a two-step process coinciding with the Desk Review and the
Field Work. They include European Stakeholders, national stakeholders, and others (namely
development partners). A stakeholder mapping has already been started as part of the preparation of
the IR and will be largely completed at the start of the desk phase. Most of them will be consulted at
the desk phase. The field phase may also complete the list as needed and add a handful of others,
while increasing the number of people consulted, including through focus groups. The desk report will
1
http://ec.europa.eu/smart-regulation/guidelines/ug_chap7_en.htm
include an updated stakeholder mapping. The TOR provides a fairly comprehensive list of the type of
counterparts by function to be consulted. Others will be added to the list through review of operations
(for instance any key counterpart who recent oversaw an EU project or continues to do so) and the
consultative process will become less granular over time, especially during the field phase. Table A7.1
below provides a stakeholder mapping given the present state of knowledge. The intention would be
to obtain about 10 responses (see below) per country, assuming a 50% or so response rate.
Table A7.1: Stakeholder mapping
European
DG NEAR Directorate A, B, C
and D
DGs HR, BUDG
EU Delegations in beneficiary
countries, European Union Office
in Kosovo
TAIEX and Twinning
management, and at least one
leading Member State active in
the country
Field Phase
European development partners
(SIDA, GIZ, etc.)
Desk Phase Onwards
National
National IPA (NIPAC) and ENI
aid Coordinators, relevant
structures, responsible for
programming, implementation,
monitoring of IPA I and IPA II
and ENI assistance
General Secretariats of
Government/ Deputy Prime
Minister´s or Prime Minister´s
Offices, Ministries of Finance,
European Integration, and others
in charge of public
administration reform
Procurement institutions, Civil
Service Management Agencies,
other relevant Public
Administration Bodies, Supreme
Audit Institutions.
Parliamentary Secretariats, etc.;
International Organisations
Organisation for Economic
Cooperation and Development
(OECD;
World Bank, International
Monetary Fund (IMF) –
Washington-based staff
Representatives of PAR Special SIGMA
groups2
Regional School of Public
Administration (ReSPA)
Local International donors,
playing a role in the area of PAR
(UNDP, World Bank, USAID etc.
Academic, civil society
SIGMA
NB: This list may be expanded for the field phase and presented in the Desk Study
D. Consultative Mechanism
6.
The consultations are linked to the work-plan of the evaluation, the key features and methods
of which are presented in tables A7.2 and A7.3 below.
2
The composition of these groups varies from country to country but typically includes donor (including from Member States)
and government representatives. They meet periodically as part of the policy dialogue.
Table A7.2 – Desk Phase Consultations (January-March 2017)
Activity
Complete the
documentation
collection
Brief Description
of
relevant
Strategic documents largely available. However,
operational level documents are generally not readily
obtainable and have been requested from the
Delegations and geographic desks.
Update the level of information by means
of interviews, e-mail questionnaires or
any other relevant tools with relevant
actors (EC officials, Government officials,
other stakeholders)
Throughout December 2016, the evaluation team will
compile a list of about 20 interviewees for each country
and will contact each either with a request for phone
interviews or filling in a questionnaire. Every attempt will
be made to meet with Brussels-based staff face to face.
Analyse
the
documentation
in
compliance with the evaluation questions
The idea is to analyse all operations for which
information is available. The consultants recognise that
there is a general absence of evaluation reports, which
will be filled through exchanges with stakeholders.
Understand better SIGMA support and
collect country level and specific activity
level information
This activity, initially envisaged as part of the Field
Phase, will be undertaken under the Desk phase in
January 2017.
Table A7.3 – Field Phase Consultations (April-June 2017)
Activity
Brief Description
In-person interviews in Brussels with the relevant
staff from the pertinent line DGs and in the
beneficiary countries with relevant stakeholders.
The evaluators will meet with the Delegations,
national and international stakeholders relevant for
each country. ReSPA will be visited during the field
phase.
Country case studies will be phased so that
the approach is piloted and written-up by the
team leader before it is undertaken in other
countries,
Other case studies will be led by each team
member, with the team leader participating in
some field visits.
The field phase will also include analysis of
relevant information. Phone interviews, online
questionnaires (only if other means are not
feasible) and other tools may complement personal
interviews and analysis of information, with relevant
local and international stakeholders to collect data
on PAR supported by IPA/EN(P)I assistance.
This is related to the previous point and a
continuation of work initiated under the desk
phase, albeit focused on fewer countries.
In order to organise the web based Open public
consultation the evaluators shall elaborate a
document summarising the preliminary findings,
conclusions and recommendations including some
open questions to provide structure for the possible
inputs and to guide those wanting to contribute,
(max 20 pages).
This document (and summary in French) is
intended for the Internet-based public
consultation (via a link provided on the
Secretariat General website to another
website managed by the DG responsible for
the Open Public Consultation).
The Draft document to be used for public
consultation may also be used as the basis for a
possible workshop in Brussels of the main
stakeholders organised by the Commission to
Expected in June 2017
Activity
Brief Description
discuss preliminary findings, conclusions and
recommendations stemming from the field and
desk phases.
The Open public consultation will then be initiated
and organised by the Commission at the end of the
field phase to verify preliminary findings and to
seek views from as wide an audience as possible.
It will be open on line on the SG/DG NEAR website
for 14 weeks for all interested parties to provide
their input. Inputs from the public will be welcome in
all EU languages.
July-October 2017
Dissemination event of the Final Report in Brussels
Early 2018, tbc
7.
The means of communication with stakeholders will include phone and face-to-face
interviews, written questionnaires or emails when other means are not feasible (to be kept as concise
as possible to encourage good response). This list is in order of preference, as it is the evaluators’
experience that people are less responsive to purely written requests for information. It is thus the
intention to have a team member to conduct a cluster of face-to-face interview with Brussels-based EU
staff. As a first step of the desk study the consultants will prepare a standard list of questions in each
case to allow for comparability, even if operation and country-specific questions are also added. The
differences in the nature of commitment on PAR between Neighbourhood and Enlargement countries
will also be considered. As far as possible, the consultants will provide stakeholder counterparts with a
list of questions a day prior to phone or to face-to-face interviews – this approach can encourage
better preparation and reduces some of the stress associated with being interviewed by evaluator.
The field phase will consists mainly of face-to-face discussions with individual and focus groups. This
work will only take place in capital cities as subnational governments are not being covered.
8.
E. Feedback to Interviewees
The final report will be placed on the DG NEAR website.
F. Public Consultation and Civil Society (open public consultation)
9.
The second type of consultation concerns a formal external public consultation process open
to all following the field phase and preparation of a report. This process3, which is managed by the EU,
involves the formal collection of comments from any interested stakeholders. The documents are
posted on a public website for 14 weeks which can collect feedback. These comments are
subsequently summarised in a document and taken into consideration when drafting the final report.
3
https://myintracomm.ec.europa.eu/sg/better_regulation/Documents/Checklist_public_consultations.pdf