Do Interviewers Influence
Respondent Propensity to ‘Satisfice’?
Gosia Turner, Prof Patrick Sturgis, Prof Chris Skinner
[email protected]
AAPOR 67th Annual Conference - May 17-20, 2012
1
NCRM is funded by the Economic and Social Research Council
Cognitive aspects of survey
•
•
•
•
•
•
•
•
•
•
•
Selecting first response alternative that
seems ‘reasonable’
Agreeing with assertions
Endorsing status quo
‘Straightlining’ (same answer on the rating
scale)
Saying ‘Don’t know’
Mental ‘coin-flipping’
Heaping
Comprehension
Retrieval
Judgement
Response
NCRM is funded by the Economic and Social Research Council
SATISFICING
2
1
Satisficing
• Difficulty of the task
– Questionnaire design
– Difficulty of the question
• Respondent’s ability to do the task
– Cognitive skills (proxy: education)
• Respondent’s motivation to do the task
– Interest in and familiarity with the subject
– External circumstances
– Interviewer effect (motivating, assistance given, social
desirability)
NCRM is funded by the Economic and Social Research Council
3
Heaping
• When respondent is rounding to ‘some commonly
used measure’ or ‘neat’ value:
– Behavioral frequency questions
• How many cigarettes a day? (8  10/1 small pack a day)
– Duration/distance questions
• How long to drive to work? (36 minutes  40 min/half an hour)
– Income/spending questions
• What is your annual income? (31,450  30,000/32,000)
• Histograms show apparent spikes or ‘heaps’ at certain
values
NCRM is funded by the Economic and Social Research Council
4
2
Data used:
• National Travel Survey (2002-2008)
– Household level
• Information on interviewer (paradata)
– Administrative information
– Admin block
• Middle Super Output Areas (MSOA) characteristics
(derived from Census 2001)
NCRM is funded by the Economic and Social Research Council
5
How long…?
• ‘How long would it take (me) to walk to the nearest
railway station’?
• ‘How long would it take (me) to get to the nearest shop
selling groceries on foot or by public transport, using
whichever is the quickest?’
• In total 3 ‘transport’ items and 6 ‘amenities’ items.
• Strong evidence of heaping:
NCRM is funded by the Economic and Social Research Council
6
3
Nearest grocery store
10000
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
0
5
10
15
20
25
30
40
45
50
55
60
7
NCRM is funded by the Economic and Social Research Council
Nearest hospital
2500
2000
1500
1000
500
0
0
5
10
15
20
25
30
35
40
45
NCRM is funded by the Economic and Social Research Council
50
55
60
65
70
75
80
85
90
8
4
Do interviewers ‘satisfice’?
• Evidence from the process data (paradata):
– How long did it take to place and explain the diary?
– How long did it take to pick up and check the diary(ies)?
• Visible heaps at the multiples of 5 minutes
9
NCRM is funded by the Economic and Social Research Council
How long did it take to pick up and
check the diary(ies)?
8000
7000
6000
5000
4000
3000
2000
1000
0
0
5
10
15
20
25
30
35
40
45
NCRM is funded by the Economic and Social Research Council
50
55
60
65
70
75
80
85
90
10
5
Research question
• Do heaping interviewers encourage heaping
behavior among their respondents?
NCRM is funded by the Economic and Social Research Council
11
Methods
• Constructed ‘heaping’ indicators for
respondents and interviewers
• Multilevel multinomial ordered x-classified
model (estimated using MlwiN and MCMC)
– Interviewer and area effect separated
– Control for area and respondent characteristics
– Control for remoteness
NCRM is funded by the Economic and Social Research Council
12
6
Heaping indicator
• If the reply is a multiple of 5 then score ‘1’ / ‘0’ otherwise
• For the respondent: average taken over 3 transport items and
average taken over 6 amenities items
• Two dependent variables are ordered discrete variables with
four categories:
–
–
–
–
High heapers
Medium heapers
Low heapers
No heapers
• For the interviewer: average taken over 2 admin block items and
then over all interviews conducted – continuous independent
variable
13
NCRM is funded by the Economic and Social Research Council
Distribution of the dependent variables:
55.7%
54.9%
30.9%
25.9%
16.8%
11.9%
1.5%
High heapers
Medium heapers
Transport
NCRM is funded by the Economic and Social Research Council
Low Heapers
2.5%
No heapers
Amenities
14
7
1. Interviewer random effect
• Interviewer’s effect on the satisficing variables:
– Transport
• interviewer related variance – 5.3%
• area related variance – 7.2%
– Amenities
• interviewer related variance – 7.3%
• area related variance – 4.2%
NCRM is funded by the Economic and Social Research Council
15
2. Interviewer’s characteristics
• Include:
– NTS experience (number of NTS administrations)
– Age in 2010
– Gender
• None of them affect the transport satisficing variable
• Male interviewers encourage less heaping than female
interviewers on the amenities satisficing variable
NCRM is funded by the Economic and Social Research Council
16
8
3. Effect of interviewer’s heaping
Transport:
Logit (probability of being a high heaper) = -1.656 + 0.664 x Int.Heapingl + hjkl
Amenities:
Logit (probability of being a high heaper) = -0.246 + 0.666 x Int.Heapingl + hjkl
Interviewer heaping indicator ranges from 0.16 to 1; average 0.831
17
NCRM is funded by the Economic and Social Research Council
Predicted probabilities of being a high heaper
60.2%
57.5%
52.1%
46.5%
17.5%
0.16
24.6%
20.8%
0.5
Transport
0.831
26.7%
1
Amenities
X axis – the interviewer heaping indicator
NCRM is funded by the Economic and Social Research Council
18
9
Conclusions
• Interviewers influence respondent’s satisficing
behaviour measured by heaping
• Interviewer’s characteristics in general do not
influence the respondent’s satisficing
• Interviewer’s own satisficing behaviour affects
(increases) respondent’s satisficing
• Implications for survey practice
NCRM is funded by the Economic and Social Research Council
19
10