The European Conference on
Quality in Official Statistics
Cognitive Test related to Labour Force Survey
ad-Hoc Module on Migrants and their
Immediate Descendants
Mario Albisinni
Elisa Marzilli
Federica Pintaldi
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
Introduction: Labour Force Survey

The Labour Force Survey (LFS) began on quarterly basis in
1959
 It is the biggest and most complex sample survey
- 75.000 households / 170.000 individuals
- average sampling rate: 1/300
- statistics for small geographical areas (NUTS II)
 The LFS is a survey of households that provides information
about people's employment status and conditions
 Official source for employment and unemployment
 Many Eu Regulations
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
Introduction: Labour Force Survey
 In order to match EUROSTAT standards. on 2002 ISTAT
developed a new Labour Force Survey
 It replaced the previous old survey in 2004
 The main changes concern both content and technical
points of view:
- the survey has become continuous
- from a PAPI technique to a mixed technique CAPI-CATI
- a new questionnaire
- new information
Since 2005 we have started to diffuse
information about non nationals
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
Introduction: Labour Force Survey
Ad hoc modules: specific items every year
2000
Transition from school to working life
2001
Length and patterns of working time
2002
Employment of disabled people
2003
Lifelong learning
2004
Work organization and working time
2005
Reconciliation between work and family life
2006
Transition from work into retirement
2007
Accidents at work and work-related health problems
2008
Labour market situation of migrants
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
LFS: Ad-hoc Module 2008
The aim is:



complete identification of the population of migrants
and their immediate descendants
comparable data at European level
comparison between migrants (and
descendants)
and
nationals
about
characteristics of labour market
their
main
analysis of the factors affecting integration in the
labour market
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
LFS: Ad-hoc Module 2008
In order to improve the quality of
survey data for migrants by decreasing
language barriers we conducted a
cognitive test
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
Cognitive Test




the cognitive test is a method to reduce non-sample
errors and improve the quality of the data
the cognitive interview identifies the aspects of the question
that lead to errors in the answers
the cognitive interview decomposes and codifies the
thoughts of the interviewees when they answer a
questionnaire
the purpose is to stimulate the respondents to suggest the
difficulties encountered in answering questions
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
Cognitive Test
The cognitive process can be drawn up into 4 steps:
1.
Comprehension: the respondent tries to understand the
question and its terms
2.
Retrieval of relevant information from the memory:
the respondent retrieves and selects the information in
order to be able to answer the question
3.
Decision-making (or judgment) process: the
respondent reflects on what is the correct answer and
about personal experience to share
4.
Answer process: the respondent decides on the answer
to be given and communicates it to the interviewer
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
AM 2008: Cognitive Test
Methodology




An interviewer accompanied by one observer
conducted a face-to-face qualitative interview
Both the interviewer and the observer had a trace
they had to follow and which allowed them not to go
off the point
Each interview was registered in order to be able to
listen to it again and transcribe it
A standardized scheme was compiled to evaluate the
interviewee’s level of question comprehension
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
Cognitive Test
Sample
 We
interviewed a selected group of 30
migrants
by
sex.
citizenship
and
professional status
 The sample has been divided in two:
- first 15 interviews: test questionnaire
some questions were re-formulated because of problems
emerged
- second 15 interviews: test new questions
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
Cognitive Test
The interview
The questionnaire contains 3 kind of questions:



Questions to be tested: read exactly the way they are
written on the questionnaire to evaluate whether the
terms are correct and comprehensible
Probe questions: asked after the questions to be
tested allow to examine more in-depth the meaning of
the answers given
Questions for the observer: allow to survey the nonverbal reactions of the communication process and
suggest what kind of information should be written on
for each question
Moreover the probe questions can be integrated with other questions the
interviewer may freely elaborate regarding aspects that emerge during the
interview
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
Cognitive Test
The probe questions concern:
 problems
relative to the question comprehension/
interpretation
 re-wording of the question by the interviewer
 certainty of the answer given
 strategy used for retrieving past information
 opinion on a specific aspect discovered during
the
interview
 reason
for choosing a specific answer and the
difficulties encountered in formulating it
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
Cognitive Test
The cognitive test questionnaire: example
QUESTIONNAIRE QUESTIONS
D1. In which year did you come to live in
Italy for the first time?
-
PROBE QUESTIONS
-
Do you understand “to live in Italy”?
-
Is there a difference between “live” and
“reside” in Italy?
-
What does “residence” make you think of?
-
What were you thinking of when answering
the question?
1|_| To work
-
Has that reason changed over time?
2|_| To study
-
What does “main reason” make you think of?
-
The reason you mentioned. to what year does
it refer to?

Read the modalities
-
Are the answers provided clear? Is it easy to
choose an answer?
Year___________
D9. Which is the main reason for coming to
Italy?
3|_| For family reason (reunion. transfer)
4|_| For marriage. affective relationship
5|_| To ask for political asylum
996|_| Other (specify_____________)
997|_| Does not know
Rome. 9-11 July 2008
-
Is there an answer you would like to add?
Could this question be annoying? In your own
words…
The European
Conference on Quality in
Official Statistics
Cognitive Test
The analysis
The first step was to transcribe the dialogues of the
interviews and notes of the observer.
For each question. we indicated:
 answer given
 respondent’s difficulties
 observer’s comments
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
Cognitive Test
The analysis
We compiled a standardized scheme to
evaluate the interviewee’s level of
comprehension:
 interviewee’s question
 answer modality
 pertinence of the answer
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
Cognitive Test
The interviewee’s question comprehension:
A1 – Did the interviewee ask to repeat the question?
1 – yes
2 – no
A2 – Did the interviewee understand the question immediately?
1 – yes
2 – no
A3 – How difficult was for the interviewee answering the question?
1 – a lot
2 – some
3 – not much
4 – none at all
A4 – Did the interviewee understand the time reference?
1 – yes
2 – no
3 – not present (the question did not have a time reference)
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
Cognitive Test
The interviewee’s question comprehension:
A5 – Did some words of the question result not very comprehensible?
1 - yes
2 - no
A5a – (if no) State the not very comprehensible words
A6 – Other question comprehension problems?
1 - yes. question too long
2 - yes. complex sentence
3 - yes. other problems (________________________________________________________)
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
Cognitive Test
The answer modality:
A7 – Were the answer modalities clear?
1 - yes. very clear
2 – yes. rather clear
3 – no
4 – not present (the question did not provide for answer modalities)
A8 – Were the answer modalities exhaustive?
1 – yes
2 – no
3 – not present (the question did not provide for answer modalities)
A9 – How difficult was for the interviewee choosing an answer modality?
1 – a lot
2 – some
3 – not much
4 – none at all
5 – not present (the question did not provide for answer modalities)
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
Cognitive Test
The pertinence of the question with the interviewee’s condition :
A10 – Through probe questions did the interviewee’s initial answer change?
1 – yes. it changed
2 – no. the answer was always the same
A11 – How pertinent was the interviewee’s answer to his/her condition?
1 – a lot
2 – enough
3 – not a lot
4 – not at all
5 – I do not know (difficult to understand whether the interviewee’s answer is correct)
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
Cognitive Test


for each question. the combination of the
information from the content analysis has allowed
reconstructing a comprehension index
it was created adding the “positive comprehension”
answers and dividing these per total number of
answers
Comprehension Index
=
answers with “positive comprehension”
total answers
The index moves from 0 to 100 (max comprehension)
Rome. 9-11 July 2008
*100
The European
Conference on Quality in
Official Statistics
Cognitive Test
In relation to the comprehension index of wording
questions we considered positive “comprehension
answer”:





Did the interviewee understand the question immediately?
[Modality: yes]
How many difficulty was for interviewee answering the
question? [Modality: not much/not at all]
Did the interviewee understand the time reference?
[Modality: yes]
Did some words of the question
comprehensible? [Modality: no]
result
not
very
Other question comprehension problems? [Modality: no]
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
Cognitive Test
In relation to the comprehension index of modalities of
answer we considered positive “comprehension answer”:

Were the answer modalities clear? [Modality: yes]

Were the answer modalities exhaustive? [Modality: yes]

How difficulty was for the interviewee have to choose an
answer modality? [Modality: not much/not at all]
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
Cognitive Test: results
Questions
Comprehension
Index of wording
question
Comprehension
Index of modalities of
answer
1st group
2nd group
Year of arrival in Italy
94.7
82.7
Open
Age at arrival in Italy
72.2
94.7
Dichotomised
Main reason for coming to Italy
76.8
91.7
78.6
91.1
Coming for work
75.0
94.4
80.0
96.3
Help received to find a job
57.6
76.1
81.0
93.3
Services used during the first two years
50.7
81.0
76.9
95.6
Type of permit
81.5
93.3
100.0
95.6
Permit that allows to work
83.7
100.0
Improve Italian language for adequate job
46.7
53.3
88.4
86.7
Recognition of educational degree
70.0
86.5
86.7
97.4
Knowledge of Italian language
75.0
98.2
83.3
92.0
Rome. 9-11 July 2008
1st group
2nd group
Dichotomised
The European
Conference on Quality in
Official Statistics
Cognitive Test: results
The new wordings improved the comprehension
levels: the average comprehension level of
wording question arises from 71% of the 1st
group to 87% of 2nd group
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
Cognitive Test: results
The main problems that could arise when using cognitive test
and how they can be corrected
Problem
The structure of the question
is complex
Type of correction
New wording of the question
Decomposition of the question
The terms used are not clear
Wording
The answer modalities are not
pertinent or exhaustive
Elimination/addition of some
modalities
The question is superfluous or
not easy to understand
Elimination of the question
Two close questions do not
appear logically
consequential
Change in order of the
questions
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
Cognitive Test: some examples
Which is the main reason for coming to Italy?
1|_|To work
2|_|To study
3|_|For family reasons (reunion. transfer)
4|_|For marriage. affective relationship
5|_|To ask for political asylum
996|_|Other (specify____________)
997|_|Does not know
PROBLEMS:
The answer modalities were neither clear nor exhaustive. The
modality “to work”. produced some uncertainties for those who
came to Italy to search for a job.
R: to create myself a future. I cannot say for work because I did not have a
work contract; I came to Italy to find a job
The reason changes in the time
R: To work. Well. at first not to stay in Italy. My objective was to go to
Spain and continue to study. Then. I had to stay in Italy because I did not
have much money anymore; I found a job and stayed…
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
Cognitive Test: some examples
Which is the main reason for coming to Italy?
1|_|To work
2|_|To study
3|_|For family reasons (reunion. transfer)
4|_|For marriage. affective relationship
5|_|To ask for political asylum
996|_|Other (specify____________)
997|_|Does not know
CORRECTIONS:
 the first modality was divided in “look for a job” and “to work”
 the verb was put to the past tense to focus the attention on
the reason for migrating rather than on the reasons that
subsequently came to create
 a new modality was added “short stay in Italy to go to another
country” to survey when Italy is a transitory destination rather
than the reason for migrating
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
Cognitive Test: some example of results
In the 2 years subsequent to your arrival in Italy. did you use one of the following
services supplied by public or private institutions/associations:
1|_| Assistance for searching a job
2|_| Professional training (including the one on the workplace)
……
PROBLEMS:
the structure of the question resulted to be complex and the time
reference was not always understood.
I: is it clear in the question that we refer only to first two years? R: No. I
understood the whole period
CORRECTIONS:
 the wording of the question was simplified
During the first 2 years you lived in Italy. did you use one of the
following services:
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
Cognitive Test: some example of results
Do you need to know better Italian to find a job that is more adequate to your
skills and/or previous working experiences?
1|_|Yes
2|_|No
PROBLEMS:
the phrase structure resulted complex: interviewees asked several
times to repeat the question. three terms were not very
comprehensible: “Italian”. “skills” and “adequate”
most of the interviewees answered only the first part of the question
(knowledge of the Italian language) without considering the second part of
the question (work more adequate to one’s own skills and/or previous
working experiences)
R: what does “Italian” mean? The Italian language or Italian as person?”
R: “adequate” though is difficult …I would say “better”…
R: to persons who do not understand well Italian. the question is too long.
At first. one is not sure of what it is all about
R: Instead of “adequate”. I would say “more right for the things you can do”
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
Cognitive Test: some example of results
CORRECTIONS:
 the question was divided into two parts
 in the first part. respondents are asked if they have an adequate job
 if “no” we asked if the difficulty is because of poor knowledge of the
Italian language
Do you have a job close to your studies and competences?
1|_|Yes
2|_|No
To have a job close to your studies and competences. do you need to
improve the knowledge of the Italian language?
1|_|Yes
2|_|No
Rome. 9-11 July 2008
Ad-hoc Module 2008: Cognitive Test
Conclusion
 after a second cycle of interviews the questionnaire becomes
definitive
 the cognitive test has been particularly useful in order to set-up
of the ahm 2008
 we reduce the non-sample error that bias the accuracy of data
Cognitive test highlights hidden errors
Rome. 9-11 July 2008
The European
Conference on Quality in
Official Statistics
THANK YOU A LOT!
Rome. 9-11 July 2008