Blue Paper
Measurement Procedures
Q6 and Q7
The specific approaches taken by the Participating Partners in
calculating the performance indicators »Customer satisfaction«
Fourth Draft
BP 28
February 2007
Synthesis Forschung
Gonzagagasse 15/3
1010 Vienna
Telefon +43 1 310 63 25
Fax +43 1 310 63 32
E-Mail [email protected]
http://www.synthesis.co.at
ÖSB Consulting GmbH
Meldemannstraße 12-14
1200 Vienna
Telephone +43-1-33 168-0
Fax +43-1-33 168-101
E-Mail [email protected]
http://www.oesb.at
Preface
The Participating Partners have agreed upon a set of
indicators which reflect upon the performance of
Public Employment Services.
Though there is an agreed »core procedure« for arriving
at the numerical values for specific variables, it is to be
expected that some Participating Partners might have
to make »adjustments« in their data retrieval process.
This paper provides background information about such
»adjustments« in case a Participating Partner had to
make use of them.
This draft paper is subject to revisions. These revisions
might turn out to be substantial.
For the team of consultants:
Prof. Michael Wagner-Pinter
Vienna, 2 February 2007
Content
1
Are the indicators already part of the management information system?
4
2
Are there indicators already part of the management information system that comes
close to the indicator of the project with regard to data provision?
5
3
Are the customer satisfaction data available?
6
4
By whom is the customer satisfaction survey carried out?
7
5
Additional questions
8
6
How are the customer satisfaction data defined?
9
Appendix
List of papers of the Working Group
Imprint
16
22
Measurement procedures: The specific approaches taken
1
Are the indicators already part of the PES management information system?
Q6:
Customer satisfaction
Job seekers
yes
AT
BE/ORBEM
BE/VDAB1
DE
FR
IE
FI
LV
NL
SE
SI
SK
UK
1
x
x
x
x1
x
x
x
x
x
x2
x
x
x1
x
x
x
x
x
. Customer surveys are
casrried out from 2006
on.
2
no
Q7:
Customer satisfaction
Employers
yes
. First pilot survey
among job-seekers was
carried- out by PES in
2006
4
no
x
x
x
x
Measurement procedures: The specific approaches taken
2
Are there indicators already part of the management information system that comes
close to the indicator of the project with regard to data provision?
Q6:
Customer satisfaction
Job seekers
yes
AT
BE/ORBEM
BE/VDAB1
DE
FR
IE
FI
LV
NL
SE
SI
SK
UK
2,3
x
no
x
Q7:
Customer satisfaction
Employers
yes
x
x2
x3
x
x
x
. Every year, ANPE
realizes a survey on
customers’ satisfaction.
This survey is very
detailed and concerns all
the services available to
job seekers (and
employers). There is no
question on the global
level of satisfaction.
That’s why we propose
to calculate an indicator
synthesizing several
questions.
5
x
no
x
Measurement procedures: The specific approaches taken
3
Are the customer satisfaction data available?
AT
BE/ORBEM
BE/VDAB
DE
FR
IE
FI
LV
NL
SE
SI
SK
UK
1
.
Customer surveys
are casrried out from
2006 on.
6
Job seeker
Employers
yes
no
yes
yes 1
yes
yes
yes
yes
yes
yes
no
yes
yes 1
yes
yes
yes
yes
yes
no
yes
yes
yes
Measurement procedures: The specific approaches taken
4
By whom is the customer satisfaction survey carried out?
AT
BE/ORBEM
BE/VDAB1
DE
FR
IE
FI
LV
NL
SE
SI
SK
UK
1
Q6:
Customer satisfaction
Job seekers
By the PES
By an external
himself
institute
x
-
x
x
-
carried out in 2005 by
an external intitute on
FÁS‘ behalf.
. Since 2004.
. Since 2005, the
x
. In addition an annual
3
x1
x
x2
Employer Survey was
2
Q7:
Customer satisfaction
Employers
By the PES
By an external
himself
institute
x
-
method has changed.
7
x
x
x3
x
x
Measurement procedures: The specific approaches taken
5 Additional questions
Were there any changes made to the methodology used for calculating the »customer
satisfaction Indicator« in the course of the observation period? (Please tick)

no
yes
no
If yes: Which one?
Were there changes within the organisation which might have influenced the »customer
satisfaction Indicator« in the course of the observation period? (Please tick)

yes
If yes: Which one?
8
Measurement procedures: The specific approaches taken
6 How are the customer satisfaction data defined?
9
Measurement procedures: The specific approaches taken
AT
BE/ORBEM
BE/VDAB
DE
FR
CUSTOMER SATISFACTION
Job seekers
Quarterly survey by telephone by an external institute. Every 3 months 30 persons
(who had any kind of contact to AMS) per local office (99 local offices) are asked:
altogether 11.880 persons (50% women) per year in Austria, representative by age
and qualification. Beside a lot of specific questions on all kind of services there is one
final question: Alltogether are you satisfied with the services you got from Your AMS
office? Answers are 1 ( very satisfied) to 6 (Not satisfied at all). This means all
answers 1-3 are counted as a positive answer ( the internal indicator counts only 1
and 2 answers)
A survey including questions about a selection of items such as accessibility, waiting
times, information, staff members, quality of provided services… is sent to a
representative sample of jobseekers throughout the year. They can give a score from
1-10 (1-4: totally insufficient, 5-6 insufficient, 7-8 good, 9-10 excellent). These
results are available 2 times a year. The jobseekers are divided by which service
they’ve used: universal services, vocational training, training on the floor… The
jobseekers satisfaction results are also separated according to the channel they’ve used
being: online – service center – contact service. A question about the overall
satisfaction of the specific used service is being asked, not about the whole PES.
Since 2006 there are quarterly customer surveys, where we are asking a quarter of the
yearly number of customers to be asked (25 employers and 30 job-seeker in each of
the 178 local-PES; yearly 100 employers and 120 job-seekers in each of the local
PES).
We propose to synthesize three questions from our annual survey on jobseekers’
satisfaction:



Globally, are you satisfied with the services and facilities available in your local
agency (information, physical and internet job-boards, ...)?
Globally, are you satisfied with the proposals and advices given by the ANPE
concerning your job-search?
Globally, are you satisfied with the answers found to your needs?
The calculation is a simple average of the answers to these questions. It has to be
noted that only a third of the total sample (around 1500 persons) answers to the
third question.
The questionnaire was changed in 2004. So it’s not possible to build a reliable
indicator for years 2000 to 2003.
In 2004, 70,8% of jobseekers were satisfied («very satisfied» + «quite satisfied»). In
2005, they were 67,1%.
10
Measurement procedures: The specific approaches taken
IE
CUSTOMER SATISFACTION
Job seekers
FÁS carries out an Employment Services Customer Satisfaction Survey in which we ask
the question: Thinking of all the services on offer by FÁS Employment Offices, how
satisfied or dissatisfied are you overall with the services that FÁS provided for you?
FI
LV
(which is equivalent to the suggested question, how are you satisfied with the overall
services the Public Employment Service has delivered?) In terms of its measurement,
we have a five point scale (from very satisfied to very dissatisfied) which allows for a
neutral category (neither satisfied nor dissatisfied). Excluding this category, as outlined
in the instructions, we obtain a satisfaction rating of .84 out of 1 in the previous
survey.

A customer feedback survey is carried out every other year; 2002, 2004, 2006
and from 2006 onwards every year

Telephone survey of 23 000 interviews based on stratified random sample from
the IT system of PES

Both job seekers´ and employers´ satisfaction is measured by means of their
statements concerning the different components of services

The grading system consists of 5 options: very poor, poor, satisfactory, good, very
good. The option in the middle of the scale is dropped out, when counting the
shares of satisfied job seekers and employers.
In total 1003 respondents were interviewed in the quantitative research of job seekers.
The size and structure of sample (number of interviews in each local office of the
SEA) were divided considering the SEA statistic data on number of the registered job
seekers by regions of the Republic of Latvia and cities.
The grading system consists of 5 options: The service was useful, was rather useful,
was rather not useful, was not useful and it’s hard to say. very poor, poor,
satisfactory, good, very good. The option in the middle of the scale is dropped out,
when counting the shares of satisfied job seekers and employers.
At the end of this year the results of the second satisfaction study of clients of the
SEA will be available. The “standard” question “How you are satisfied with the overall
services the Public Employment Service has delivered?” along with the four possible
answers described on page 4 were included in the questionnaire elaborated for this
second study. So it will be possible to compare the results and indicators regarding the
satisfaction of the clients of the SEA.
11
Measurement procedures: The specific approaches taken
NL
SE
SI
SK
UK
CUSTOMER SATISFACTION
Job seekers
The questionnaire has changed in 2004:The measurement is based on ( external)
surveys. The rate of satisfaction is expressed by a figure ( on a ten points scale) in
stead of a percentage. Since this year the survey takes place quarterly by on line
interviewing. The results are used for improving the services.
First pilot survey among job-seekers was carried-out by PES in 2006. For 2007 it
is planned to improve the sample design and to add a question about overall
satisfaction with PES services.
There is an annual telephone survey of 3,300 Jobseeker customers conducted on
behalf of the PES by an independent researcher.
The Jobseekers are asked about their overall satisfaction with the service and those
who respond with very or fairly satisfied make up the percentage score.
12
Measurement procedures: The specific approaches taken
AT
BE/ORBEM
BE/VDAB
DE
FR
CUSTOMER SATISFACTION
Employers
Quarterly survey by telephone by an external institute. Every 3 months 20 employers
per local office (99 offices) who have posted a vacancy are asked (alltogether
9.900 employers a year in Austria). In the questionnaire there are a lot of specific
questions on all kinds of services. Finally there is one final question: Alltogether are
You satisfied with the services You got from your AMS office? Answers are 1 – 6 (1=
very satisfied, 6= not satisfied at all). This means all 1 – 3 answers are counted
(internal indicator is only 1 and 2 answers)
A survey including questions about a selection of items such as accessibility, waiting
times, information, staff members, quality of provided services…is sent to a
representative sample of employers throughout the year. They can give a score from 110 (1-4: totally insufficient, 5-6 insufficient, 7-8 good, 9-10 excellent). These results
are available 2 times a year. The employers satisfaction results are separated according
to the channel they’ve used being: online – service center – contact service. A
question about the overall satisfaction of the specific used service is being asked, not
about the whole PES.
Since 2006 there are quarterly customer surveys, where we are asking a quarter of the
yearly number of customers to be asked (25 employers and 30 job-seeker in each of
the 178 local-PES; yearly 100 employers and 120 job-seekers in each of the local
PES).
On the same principle than indicator Q6, we propose to combine three questions from
our annual satisfaction survey on employers:



Globally, what’s your level of satisfaction concerning the conditions under which
your job vacancy has been registred?
Globally, what’s your level of satisfaction concerning the follow-up of your
recruitment done by the counsellor?
Globally, what’s your level of satisfaction concerning the candidates refered by
ANPE on this vacancy?
The calculation is a simple average of the answers to these questions.
The questionnaire was changed in 2004. So it’s not possible to build a reliable
indicator for years 2000 to 2003.
In 2004, 76,5% of employers were satisfied («very satisfied» + «quite satisfied»). In
2005, they were 77%.
13
Measurement procedures: The specific approaches taken
IE
CUSTOMER SATISFACTION
Employers
FÁS carries out a monthly employer survey asking employers to rate the recruitment
service. There are two questions asked, depending on the means of vacancy
notification (If you registered by Internet/Telephone/Etc, how would you rate this
service?). Employers are asked to rate the service from 1 - 5, with 5 being the
highest.
‘In 2005, FÁS PES commissioned independent Consultants to carry out a survey of
employer usage and awareness of FÁS services.
The aims of the research were to:
1.
measure which FÁS services, if any, employers were using at the time
2.
determine how satisfied those using FÁS services were with the individual
services used
3.
identify any issues with services that were of a concern
4.
estimate FÁS vacancy market share
This was within the overall context of the wider usage of FÁS services by employers.
The survey asked employers to rate the FÁS services they used and were asked to
rate their overall satisfaction with FÁS services on a rating of very satisfied, fairly
satisfied, neither, fairly dissatisfied, very dissatisfied, don’t know. Employers were also
asked if they ever used FÁS services for recruitment purposes and to rate the services
under various measurement indicators. Information was collected on the various reasons
for satisfaction and dissatisfaction with recruitment services. Information was also
collected on the awareness of recruitment services and the frequency of usage. FÁS
received useful feedback as a result as to how it could improve its services and what
did not require change and worked well. FÁS intends to carry out a similar survey in
2007’.
14
Measurement procedures: The specific approaches taken
FI
LV
CUSTOMER SATISFACTION
Employers

A customer feedback survey is carried out every other year; 2002, 2004, 2006
and from 2006 onwards every year

Telephone survey of 23 000 interviews based on stratified random sample from
the IT system of PES

Both job seekers´ and employers´ satisfaction is measured by means of their
statements concerning the different components of services

The grading system consists of 5 options: very poor, poor, satisfactory, good, very
good. The option in the middle of the scale is dropped out, when counting the
shares of satisfied job seekers and employers.
In total 800 respondents were interviewed in the qualitative research of employers.
Random quota method was used in the research. The Central Statistical Bureau (CSB)
data on number of economically active enterprises of the 3rd quarter of 2005 was used
for sampling. Also CSB Enterprise Register data base was used to select a sample. In
order to analyze groups of middle and big companies separately, in the sample they
were represented proportionally in greater numbers than it would be necessary, therefore
the data was placed under a weighing procedure for balancing significance of answers
in conformity with CSB statistical data on size (number of employees) of economically
active enterprises, region, planning region of the SEA and the main type of economic
activity of the enterprise.
The grading system consists of 5 options: The service was useful, was rather useful,
was rather not useful, was not useful and it’s hard to say. very poor, poor,
satisfactory, good, very good. The option in the middle of the scale is dropped out,
when counting the shares of satisfied job seekers and employers.
At the end of this year the results of the second satisfaction study of clients of the
SEA will be available. The “standard” question “How you are satisfied with the overall
services the Public Employment Service has delivered?” along with the four possible
answers described on page 4 were included in the questionnaire elaborated for this
second study. So it will be possible to compare the results and indicators regarding the
satisfaction of the clients of the SEA.
NL
Since 2005 employers are surveyed (externly) with regard to their opinion about their
satisfaction concerning the aspect of filling vacancies by CWI’s.
15
Measurement procedures: The specific approaches taken
SE
SI
CUSTOMER SATISFACTION
Employers
Employers’ satisfaction survey that comes close to the benchmarking project definition
was first carried-out in 2005. As the question about the overall satisfaction with PES
services was not included, the indicator represents an average of the answers to the
different components and dimensions of services: job-mediation, information-support,
delivery of services in due time, staff behavior in contacts, working time schedule for
clients. The employers rate the services by using 5 grade-scale (1-very dissatisfied,
5-very satisfied). To comply with the indicator definition the (neutral) answers under
the grade 3 were split between “satisfied” and “unsatisfied”. In the 2006 survey a
question on the overall satisfaction with services will be added.
The survey on satisfaction is added to a questionnaire on annual employment plans and
needs of employers. It is carried-out by Employment Service and covers all employers
with 10 or more employees. The overall response rate in 2005 was 78 %.
Until 2005 the questions on the satisfaction were specific and related only to the
assessment of referred job-seekers, and thus were not in accordance with the definition
of benchmark indicator.
SK
UK
There is a monthly survey of 4,500 employers that use the PES to fill their vacancies.
The employers are contacted by an independent market research company within 3
months of a vacancy being closed.
Employers are asked about their overall satisfaction with the service and those who
respond with very or fairly satisfied make up the percentage score.
16
Measurement procedures: The specific approaches taken
List of papers of the
Working Group
All papers of the Working Group will be distributed by
the group of consultants to each of the Participating
Partner:
White Papers
Blue Papers
Red Papers
Yellow Papers
Green Papers
There are several series of papers. Some of them can
be used for general circulation, some of them are for
internal use only; none of the papers are strictly
confidential.
– White Papers deal with general topics and cover the
topics of the final report of the Working Group
(unrestricted circulation)
– Blue Papers list performance indicators and discuss
their strong and weak points (unrestricted circulation)
– Red Papers report on harmonized data forwarded by
the Members of the Working Group (circulation only
with permission of the Members who provided data)
– Yellow Papers contain only information which serves
to organize the work in progress (of no use to
outsiders of the Working Group)
– Green Papers cover issues related to the choice of
good practices and their presentations (unrestricted
circulation)
17
Measurement procedures: The specific approaches taken
List of Papers (latest draft)
Type of paper
White Papers
Reference
Date
WP 01
PES-Performance, Indicators, Good Practice
19/01/06
WP 02
Reporting the Benchmarking Results to the head
of PES. The Performance of AMS Austria
(English Version). First Draft
Reporting the Benchmarking Results to the head
of PES. The Performance of AMS Austria
(English Version). First Draft
Reporting the to the European Commission. First
Draft
Supporting Flowcharts
15/01/07
WP 03
WP 04
Blue Papers
Title
BP 01
BP 04
15/01/07
17/01/07
23/01/06
Measuring effectiveness and operational efficiency
of the Finnish PES
A list of eight performance indicators
(Second draft)
Operational definitions of performance indicators
Q1–Q3 (Fourth draft)
03/03/06
BP 13
Operational definitions of performance indicators
Q4 and Q5 (Second draft)
27/07/06
BP 14
Operational definitions of performance indicators
Q6 and Q7 (First Draft)
27/07/06
BP 15
Operational definition of performance indicator
Q8 (First Draft)
Information Technology as an Input to PES
Performance: A standardized data format
(First Draft)
Information Technology as an Input to PES
Performance: Data on Austria
Capturing Context: A radar chart approach
(Third Draft)
Measurement Procedures: The specific approaches
taken by the Participating Partners Q1-Q3
(Third Draft)
Measurement Procedures: The specific approaches
taken by the Participating Partners Q4 and Q5
(Third Draft)
Measurement Procedures: The specific approaches
taken by the Participating Partners Q6 and Q7
(Third Draft)
27/07/06
BP 06
BP 12
BP 20
BP 21
BP 22
BP 23
BP 24
BP 25
18
03/04/06
27/07/06
13/11/06
13/11/06
15/11/06
19/12/06
19/12/06
19/12/06
Measurement procedures: The specific approaches taken
19
Measurement procedures: The specific approaches taken
List of Papers (latest draft)
Type of paper
Yellow Papers
Reference
YP 01
YP 02
YP 08
YP 10
Red Papers
RP 04
RP 05
RP 06
RP 07
RP 08
RP 09
RP 10
RP 11
RP 12
RP 13
RP 14
RP 15
Title
Date
Public Employment Services in Europe:
a large spectrum of missions
Performance Indicators:
A first survey on data availability and ranking
PES Background and Content Information and
Indicators in the year 2005 (Third draft)
A tight schedule
(Fifth Draft)
Transition from unemployment to employment:
Indicator 1 (Second Draft)
Transition from unemployment to employment
before unemployment lasts long: Indicator 2
(Second Draft)
Transition from measures to employment:
Indicator 3 (Second draft)
A cluster approach towards different PESenvironments
(First Draft)
Benchmarking:
Performance Indicator Charts – 8 Indicators
(First Draft)
Benchmarking:
Participating Partner Charts Countries
(First Draft)
Opening access to a large share of vacancies
through the PES information system: Indicator 4
(First Draft)
Making sure that vacancies are filled: Indicator 5
(First Draft)
Customer satisfactions: Indicators 6 and 7
(First Draft)
Information technology indicator
(First Draft)
Benchmarking of PES Performance Indicators
2004: 13 PES, 7 Indicators, 4 Context Variables
(First Draft)
Benchmarking of PES Performance Indicators
2005: 13 PES, 7 Indicators, 4 Context Variables
(First Draft)
20
11/01/06
12/01/06
30/07/06
19/01/07
08/09/06
08/09/06
08/09/06
08/09/06
12/09/06
12/09/06
12/09/06
12/09/06
12/09/06
12/09/06
19/12/06
19/12/06
Measurement procedures: The specific approaches taken
List of Papers (latest drafts)
Type of paper
Green Papers
Reference
Title
Date
GP 01
Looking for good practices: A draft questionnaire
01/06/06
GP 03
(42) Examples of Good Practices: A summary
12/09/06
GP 04
Do Benchmarking Results Tell about Good
Practice? (First Draft)
Assessing Good Practices: A draft questionnaire
(Second Draft)
PROMOTION OF APPRENTICESHIP TRAINING:
A good practice pertinent to the core process
»Servicing and supporting job seekers«.
Austria Good Practice I. (First Draft)
IMPROVING SERVICES FOR EMPLOYERS: A
good practice pertinent to the core process
»Servicing and supporting employers«. Austria
Good Practice II
(First Draft)
15/11/06
GP 05
GP 06
GP 07
21
20/11/06
09/01/07
09/01/07
Measurement procedures: The specific approaches taken
List of Papers (previous drafts)
Type of paper
Blue Papers
Reference
BP 02
BP 03
BP 05
BP 07
Date
Transition from Unemployment to Employment
(First draft)
Transition from Unemployment to Employment
(Second draft)
A preliminary list of eight performance indicators
(First draft)
Operational definitions of performance indicators
Q1–Q3 (Third draft)
10/02/06
28/02/06
06/03/06
03/04/06
BP 08
Operational definitions of performance indicators
Q4 and Q5
03/04/06
BP 09
Operational definitions of performance indicators
Q6–Q8
Measurement Procedures: The specific approaches
taken by the Participating Partners
(Second Draft)
Capturing Context: A radar chart approach
03/04/06
Capturing Context: A radar chart approach
(Second Draft)
Measurement Procedures: The specific approaches
taken by the Participating Partners Q1-Q3
(Second Draft)
Measurement Procedures: The specific approaches
taken by the Participating Partners Q4 and Q5
(First Draft)
Measurement Procedures: The specific approaches
taken by the Participating Partners Q6 and Q7
(First Draft)
A tight schedule
(Second draft)
PES Background and Content Information and
Indicators in the year 2005 (Second draft)
A tight schedule
(Third draft)
A tight schedule
(Fourth draft)
08/08/06
BP 10
BP 11
BP 16
BP 17
BP 18
BP 19
Yellow Papers
Title
YP 05
YP 06
YP 07
YP 08
22
29/05/06
29/05/06
12/09/06
12/09/06
12/09/06
06/03/06
12/05/06
07/06/06
12/09/06
Measurement procedures: The specific approaches taken
List of Papers (previous drafts)
Type of paper
Red Papers
Reference
RP 01
RP 02
RP 03
Green Papers
GP 02
Title
Date
Transition from unemployment to employment:
Indicator 1
Transition from unemployment to employment
before unemployment lasts long: Indicator 2
Transition from measures to employment:
Indicator 3
Assessing Good Practices: A draft questionnaire
(First Draft)
23
30/05/06
30/05/06
30/05/06
08/08/06
Measurement procedures: The specific approaches taken
Imprint
Owner/Publisher:
Synthesis Forschung Gesellschaft m.b.H.
Gonzagagasse 15/3
1010 Vienna
Editor:
a.o. Univ.-Prof. Dr. Michael Wagner-Pinter
Synthesis Forschung
Vienna 2007
24