6/26/2014
Outline of talk
Adaptation of Standard Cognitive
Interview Methodology for use with
Spanish-Speaking Respondents
Patricia Goerman and Ryan King, U.S. Census Bureau
Presented at the 69th annual conference of the American Association for Public Opinion Research (AAPOR)
Anaheim, CA: May 15‐18, 2014
Cognitive interviewing (CI)
Definition:  One‐on‐one interviews to evaluate whether respondents interpret, comprehend and respond to survey questions as intended.
Think aloud procedure
Probes (concurrent or retrospective)






Cognitive interviewing (CI)
Past research
Research questions Study methods
Findings
Next steps
Cognitive interview probes
Meaning oriented
 “What does the term ‘foster child’ mean to you in this question?”
Process oriented  “How did you arrive at/choose that answer?” Paraphrasing
 “Can you tell me in your own words what that question is asking?” Recall  “How do you remember that you (saw a dentist 3 times) in the last year?”
1
6/26/2014
Past research
 Monolingual cognitive interview (CI) methods:
 Presser et. al. 2004; Willis 2005; Beatty and Willis 2007; Blair and Conrad 2011
 Adaptation of CI Method for use across languages/cultures:  Dean et al. 2007; Willis and Zahnd 2007; Fitzgerald and Miller 2009; Pan et al. 2010; Harkness et al 2010
Methods
 48 cognitive interviews  Spanish speakers  Segment of American Community Survey (ACS): CATI/CAPI
 Types of question tested:  Person level: age, sex, DOB, marital, Hispanic origin/race, place of birth, year of entry, education, ancestry, language spoken, English proficiency
 Household level: Type unit, year built, when moved, number of acres, number rooms and bedrooms, plumbing, telephone, vehicles Research questions
 Why have previous studies found that CI techniques cause discomfort among non‐
English respondents?  What specific interview techniques and probes work best with Spanish‐speaking CI respondents? Demographic characteristics of
Spanish speakers (n=48)
National origin
Gender
44 Mexican origin
42 Female 4 Some other origin 6 Male
Education level
Age
35 Less than HS
Range: 24‐74
13 HS diploma or more Mean: 46
Median: 44
Missing: 4
2
6/26/2014
Type of interviews:
Standard v. experimental
 Variation of introduction and probe wording
 Standard interviews
 Direct translation of typical U.S. English protocol (procedure and wording)
 Experimental interviews
 Variation of introductory statements/conversation
 Different sample probes to begin
 Flexibility in probe wording
Pre-interview interactions:
Experimental v. standard
Experimental Standard
N=28
N=20
Information provided about Int.
36%
Family Background
36%
10%
5%
How learned Spanish
36%
5%
Time spent abroad
11%
5%
Discuss Job at Census
36%
0%
Small talk between interviewer and respondent about:
The respondent
71%
40%
Interview location
61%
60%
Census Bureau / surveys
39%
10%
Interview city
18%
0%
Pre-interview interactions:
Both standard and experimental
 Discussions of the interview process prior to beginning
 Reason for doing the interview  Confidentiality  Reasons for taping the interview
Coding scheme/analysis
Yes/no codes: 1. Was probe administered? 2. Was probe administered as worded in protocol?
3. Did respondent understand when read as worded? 4. Was probe reworded?
5. Was probe understood when reworded? 6. Did probe cause discomfort? 7. Did respondent provide “useful” answer to probe? Descriptive codes: 1. Type probe rewording done? (description)
2. What type of discomfort did it cause? (description)
3
6/26/2014
Frequency of rewording by
probe type and interview type
Inter-coder reliability
 Application of Kappa statistic
 Kappa Scores 100%
80%
 Total of 7 cases coded by both interviewer/coders
 Overall: 0.68 (n=892)
 Yes/No: 0.71 (n=784) (good)
 Type probe rewording: 0.50 (n=69) (fair‐good)
 Discomfort: 0.38 (n=39) (poor)
76%
70%
61%
60%
50%
41%
40%
30%
80%
77%
74%
70%
75%
70%
26%
21%
Standard
20%
10%
0%
100%
89%
91%
Experimental
30%
Usefulness of probes by
interview type
90%
85%
77%
Meaning
Oriented
E=132,
S=94
100%
90%
86%
90%
Paraphrasing
E=92,
S=94
Process
Oriented
E=44,
S=61
Think
Aloud
E=48,
S=47
Recall
E=17,
S=19
Overall Discomfort
100%
90%
80%
72%
70%
60%
60%
50%
42% 43%
53%
50%
40%
Experimental
30%
Standard
20%
31%
30%
20%
10%
Experimental
40%
15%17%
10%
0%
Meaning
Oriented
E=132,
S=94
Paraphrasing
E=92,
S=94
Process
Oriented
E=44,
S=61
Think
Aloud
E=48,
S=47
Recall
E=17,
S=19
0%
Total
E=333
S=315
Standard
21%
16%
12%
6%
18%
13%
21%
8%
0% 2%
Pause Embarass Annoying Emotional Laughing
E=9
E=0
E=3
E=8
E=27
S=7
S=1
S=6
S=16
S=11
Other
E=4
S=11
4
6/26/2014
Overall usefulness of probes by
education level and treatment
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Frequency of “successful”
major probe rewording
90%
80%
100%100%
100%
100%
91%
82% 80%
79%
70%
Meaning
Oriented
LTHS‐E=99,
LTHS‐S=68,
MTHS‐E=33,
MTHS‐S=26
Paraphrasing
LTHS‐E=69,
LTHS‐S=64,
MTHS‐E=23,
MTHS‐S=30
Process
Oriented
LTHS‐E=33,
LTHS‐S=40,
MTHS‐E=11,
MTHS‐S=21
Think
Aloud
LTHS‐E=33,
LTHS‐S=32,
MTHS‐E=15,
MTHS‐S=15
Recall
LTHS‐E=14,
LTHS‐S=14,
MTHS‐E=3,
MTHS‐S=5
Less Than High School ‐
Experimental
60%
Less Than High School ‐
Standard
40%
59%
50%
50%
More Than High School ‐
Experimental
30%
More Than High School ‐
Standard
10%
35%
Experimental
Standard
20%
0%
Meaning
Oriented
E=95,
S=20
Paraphrasing
E=14,
S=2
Process
Oriented
E=22,
S=17
Think
Aloud
E=36,
S=31
Recall
E=12,
S=3
Probe rewording: Think aloud
Probe rewording: Paraphrase
 Original/scripted probes:  ¿Podría pensar en voz alta cuando está decidiendo cómo contestar la pregunta?
 Por favor dígame qué está pensando.
 Rewording:  Y si quisiera hablar mientras piensa, me gustaría.  Me encantaría oír lo que está pensando mientras mira…
 Original/scripted probe:
 ¿Podría decirme con sus propias palabras qué información pide esa pregunta?
 Rewording:  ¿Y qué información le esta pidiendo esta pregunta? ¿Qué le parece?
 Si ud tuviera que preguntar eso a alguien, cómo lo diría?
 ¿Qué le parece que quieren preguntar (o saber)?
5
6/26/2014
Next steps
 Additional analysis:  Look at emergent probes in experimental cases
 Look at debriefing, people’s experience with reading, literacy, education, etc.
 New research:
 Different national origin Spanish speakers
 Speakers of additional languages
Adaptation of Standard Cognitive
Interview Methodology for use with
Spanish-Speaking Respondents
Patricia Goerman and Ryan King,
U.S. Census Bureau
For more information:
E-mail: [email protected]
Disclaimer: This presentation is intended to inform people about research and to encourage discussion. The views expressed are those of the authors and not necessarily those of the U.S. Census Bureau.
6