The Relations among Different
Cognitive Shortcuts in Surveys
Presented to 68th Annual AAPOR
Conference
Roger Tourangeau, Westat
Becca Medway, AIR
Stanley Presser, JPSM
Boston, MA, May 18, 2013
Notion of Cognitive Shortcuts
• Respondents take various shortcuts to minimize burden
• Krosnick (1991, 1990) on “satisficing”:
Distinguishes such
manifestations as response order effects, nondifferentiation, DK/NO opinion responses, acquiescence
• Other have distinguished other low-effort strategies (use of
estimates for behavioral frequency questions, reporting in
round values, skimming the question)
2
Consistently Bad Respondents?
• This has been a concern since Cannell’s work; he and his
colleagues saw the job of the interviewer as preventing shortcuts
• Conrad, Tourangeau, Couper, and Kennedy (2009)—“Hard-core
speeders,” people who answer so quickly that they are unlikely to
have read the questions
• Continued to “speed” even after they were prompted to slow
down
• Latent class analysis: Kaminska, McCutcheon, and Billiet (2010)
suggest that the respondents to the European Social Survey may
fall into two latent classes, with one group more prone to
satisficing and other shortcutting behaviors than the other group
3
What is the Relation among Use of Various
Shortcuts?
• At least three possibilities here
— First, some respondents use lots of shortcuts, others only a few; this
would produce positive relations among different shortcuts
(“speeders” in Conrad et al. also prone to straight-line in grids)
— Second, different forms of straightlining could trade off; the same R
can’t show primacy, recency, and DK with a single item; this could
produce negative relationship across different methods of shortcutting
— Third, maybe there isn’t a consistent relation
According to Krosnick, satisficing most likely when difficulty of the
item exceeds R’s ability and motivation
Ability are partly R-level variables, but have item-level components
as well (familiarity makes an item less difficult, interest in topic
heightens motivation)
Difficulty primarily item-level characteristic
4
Empirical Findings
• Krosnick (1999, p. 555) cites several studies demonstrating
a positive correlation between yea-saying , on the one
hand, and non-differentiation and giving no opinion
answers, on the other
• Malhotra (2008) found a positive relation among loweducation respondents between completing a Web survey
quickly and selecting one of the initial options in a set of
unipolar scales
• Kaminska et al. find somewhat mixed results in their latent
class analysis
5
Method
•
We examined the level of use of different forms of shortcutting in three studies
— choosing the first and last response options
— yea-saying
— giving don’t know and no opinion responses
— Straight-lining among answers to similar questions
— reporting numerical answers as round values
•
Looked at 2000 ANES pre-election survey, conducted by the Michigan’s SRC,
which randomly assigned sample members to FTF or telephone. Q’aires were
very similar.
•
Also, examined the first wave of the 2008-2009 ANES Panel; done via the
Internet by Knowledge Networks; panel initially recruited by telephone.
Questions not identical to those asked in the 2000 ANES Study, but did focus on
similar topics.
6
Sample Sizes
ANES
FTF
ANES
Telephone
ANES
Web
1,006
801
1,623
985
767
1,609
Complete interviews
Answered at least 75
percent of items
7
Our Analysis
Indicator
Eligible items
Straight-lining
At least 3 items in a series with the same response
options; each item has at least 4 response options
.
Items with at least 4 response options
Items with at least 4 response options
Attitude items where respondent could indicate
that they agreed or disagreed
Behavior items where respondent could indicate
that they had or had not done something
Items where respondents provided a numerical
response without the aid of response options
Items where respondents were explicitly offered a
“don’t know” option
All items
All items
All items
Primacy
Recency
Acquiescence:
Attitude
Acquiescence:
Behavior
Round Value
Selecting Offered
“Don’t Know”
Item Nonresponse
Speed
Shortcutting Index
8
Do People do the Same Thing in Both Halves?
Straight-lining
Primacy (no showcard)
Recency (no
showcard)
Primacy (showcard)
Recency (showcard)
Acquiescence
(attitude)
Acquiescence
(behavior)
Round value
Selected offered
“don’t know”
Item nonresponse
Shortcutting index
.
FTF
Telephone
Web
*
0.21
0.07
*
0.38
0.11
*
*
*
0.30
0.20
0.10
--0.10
--0.04
0.32
0.18
*
*
0.40
*
0.37
*
*
0.68
0.43
0.65
0.45
0.13
0.05
9
What about relations among shortcuts?
0.0%
100%
13.2%
17.1%
14.8%
10.0%
80%
29.6%
60%
47.2%
37.1%
90.0%
40%
55.6%
20%
39.6%
>=0.30
0.10-<0.30
<0.10
45.7%
0%
ANES FTF
(n=53)
ANES
Telephone
(n=35)
ANES Web
(n=27)
Opt-in Web
(n=20)
10
What about the direction?
• About half in the expected direction
.
Mean Correlations between Behaviors, by Survey
Mean
ANES FTF
-0.03
ANES
Telephone
ANES Web
-0.03
0.01
11
Conclusions
• We found relatively weak relations among our indicators of
shortcutting
• The results were similar across modes of data collection
• There is some consistency in use of shortcuts within an
interview
• Also, a factor analysis suggests that some respondents may
have a tendency to select (or avoid) early options, which
may also be reflected in high (or low) levels of “yea-saying”
12