Adaptation of Standard Cognitive
Interview Methodology for use with
Spanish-Speaking Respondents
Patricia Goerman and Ryan King,
U.S. Census Bureau
Presented at the 69th annual conference of the American
Association for Public Opinion Research (AAPOR)
Anaheim, CA: May 15-18, 2014
Outline of talk






Cognitive interviewing (CI)
Past research
Research questions
Study methods
Findings
Next steps
Cognitive interviewing (CI)
Definition:
 One-on-one interviews to evaluate whether
respondents interpret, comprehend and
respond to survey questions as intended.
Think aloud procedure
Probes (concurrent or retrospective)
Cognitive interview probes
Meaning oriented
 “What does the term ‘foster child’ mean to you in this
question?”
Process oriented
 “How did you arrive at/choose that answer?”
Paraphrasing
 “Can you tell me in your own words what that question
is asking?”
Recall
 “How do you remember that you (saw a dentist 3
times) in the last year?”
Past research
 Monolingual cognitive interview (CI) methods:
 Presser et. al. 2004; Willis 2005; Beatty and Willis
2007; Blair and Conrad 2011
 Adaptation of CI Method for use across
languages/cultures:
 Dean et al. 2007; Willis and Zahnd 2007; Fitzgerald and
Miller 2009; Pan et al. 2010; Harkness et al 2010
Research questions
 Why have previous studies found that CI
techniques cause discomfort among nonEnglish respondents?
 What specific interview techniques and
probes work best with Spanish-speaking CI
respondents?
Methods
 48 cognitive interviews
 Spanish speakers
 Segment of American Community Survey (ACS):
CATI/CAPI
 Types of question tested:
 Person level: age, sex, DOB, marital, Hispanic
origin/race, place of birth, year of entry, education,
ancestry, language spoken, English proficiency
 Household level: Type unit, year built, when moved,
number of acres, number rooms and bedrooms,
plumbing, telephone, vehicles
Demographic characteristics of
Spanish speakers (n=48)
National origin
44 Mexican origin
4 Some other origin
Education level
35 Less than HS
13 HS diploma or more
Gender
42 Female
6 Male
Age
Range: 24-74
Mean: 46
Median: 44
Missing: 4
Type of interviews:
Standard v. experimental
 Variation of introduction and probe wording
 Standard interviews
 Direct translation of typical U.S. English protocol
(procedure and wording)
 Experimental interviews
 Variation of introductory statements/conversation
 Different sample probes to begin
 Flexibility in probe wording
Pre-interview interactions:
Both standard and experimental
 Discussions of the interview process prior to
beginning
 Reason for doing the interview
 Confidentiality
 Reasons for taping the interview
Pre-interview interactions:
Experimental v. standard
Experimental
N=28
Standard
N=20
Information provided about Int.
36%
10%
Family Background
36%
5%
How learned Spanish
36%
5%
Time spent abroad
11%
5%
Discuss Job at Census
36%
0%
Small talk between interviewer and respondent about:
The respondent
71%
40%
Interview location
61%
60%
Census Bureau / surveys
39%
10%
Interview city
18%
0%
Coding scheme/analysis
Yes/no codes:
1. Was probe administered?
2. Was probe administered as worded in protocol?
3. Did respondent understand when read as worded?
4. Was probe reworded?
5. Was probe understood when reworded?
6. Did probe cause discomfort?
7. Did respondent provide “useful” answer to probe?
Descriptive codes:
1. Type probe rewording done? (description)
2. What type of discomfort did it cause? (description)
Inter-coder reliability
 Application of Kappa statistic
 Kappa Scores
 Total of 7 cases coded by both interviewer/coders
 Overall: 0.68 (n=892)
 Yes/No: 0.71 (n=784) (good)
 Type probe rewording: 0.50 (n=69) (fair-good)
 Discomfort: 0.38 (n=39) (poor)
Frequency of rewording by
probe type and interview type
100%
80%
90%
86%
90%
85%
77%
76%
70%
61%
60%
50%
41%
40%
30%
Experimental
30%
26%
21%
20%
10%
0%
Meaning
Oriented
E=132,
S=94
Paraphrasing
E=92,
S=94
Process
Oriented
E=44,
S=61
Think
Aloud
E=48,
S=47
Recall
E=17,
S=19
Standard
Usefulness of probes by
interview type
100%
90%
80%
100%
89%
91%
77% 74%
70%
75%
70%
72%
60%
50%
42% 43%
40%
Experimental
30%
Standard
20%
10%
0%
Meaning
Oriented
E=132,
S=94
Paraphrasing
E=92,
S=94
Process
Oriented
E=44,
S=61
Think
Aloud
E=48,
S=47
Recall
E=17,
S=19
Overall Discomfort
100%
90%
80%
70%
60%
53%
50%
Experimental
40%
31%
30%
20%
15%17%
10%
0%
Total
E=333
S=315
Standard
21%
16%
12%
6%
18%
13%
21%
8%
0% 2%
Pause Embarass Annoying Emotional Laughing
E=27
E=8
E=3
E=0
E=9
S=11
S=16
S=6
S=1
S=7
Other
E=4
S=11
Overall usefulness of probes by
education level and treatment
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Less Than High School Experimental
Less Than High School Standard
More Than High School Experimental
Meaning
Oriented
LTHS-E=99,
LTHS-S=68,
MTHS-E=33,
MTHS-S=26
Paraphrasing
LTHS-E=69,
LTHS-S=64,
MTHS-E=23,
MTHS-S=30
Process
Oriented
LTHS-E=33,
LTHS-S=40,
MTHS-E=11,
MTHS-S=21
Think
Aloud
LTHS-E=33,
LTHS-S=32,
MTHS-E=15,
MTHS-S=15
Recall
LTHS-E=14,
LTHS-S=14,
MTHS-E=3,
MTHS-S=5
More Than High School Standard
Frequency of “successful”
major probe rewording
100%
100%
90%
80%
100%100%
91%
82% 80%
79%
70%
59%
60%
50%
50%
35%
40%
Experimental
30%
Standard
20%
10%
0%
Meaning
Oriented
E=95,
S=20
Paraphrasing
E=14,
S=2
Process
Oriented
E=22,
S=17
Think
Aloud
E=36,
S=31
Recall
E=12,
S=3
Probe rewording: Think aloud
 Original/scripted probes:
 ¿Podría pensar en voz alta cuando está
decidiendo cómo contestar la pregunta?
 Por favor dígame qué está pensando.
 Rewording:
 Y si quisiera hablar mientras piensa, me
gustaría.
 Me encantaría oír lo que está pensando
mientras mira…
Probe rewording: Paraphrase
 Original/scripted probe:
 ¿Podría decirme con sus propias palabras qué
información pide esa pregunta?
 Rewording:
 ¿Y qué información le esta pidiendo esta
pregunta? ¿Qué le parece?
 Si ud tuviera que preguntar eso a alguien,
cómo lo diría?
 ¿Qué le parece que quieren preguntar (o
saber)?
Next steps
 Additional analysis:
 Look at emergent probes in experimental cases
 Look at debriefing, people’s experience with
reading, literacy, education, etc.
 New research:
 Different national origin Spanish speakers
 Speakers of additional languages
Adaptation of Standard Cognitive
Interview Methodology for use with
Spanish-Speaking Respondents
Patricia Goerman and Ryan King,
U.S. Census Bureau
For more information:
E-mail: [email protected]
Disclaimer:
This presentation is intended to inform people about research and to
encourage discussion. The views expressed are those of the authors and not
necessarily those of the U.S. Census Bureau.