Can Embedded Help Text Links
in Web Survey Items Improve
Data Quality?
Presented by: Natasha Janson
RTI International
May 17, 2013
Natasha Janson
Christopher Bennett
Lesa Caves
Melissa Cominole
Bryan Shepherd
Jennifer Wine
Introduction
•
•
•
•
Examples of help text in web surveys
Impact of help text on data quality
Summary of our analyses and preliminary results
Conclusions and implications for future investigation
Formats for displaying “help” in
web surveys vary:
Always-visible help:
Formats for displaying “help” in
web surveys vary:
Behind-the-scenes help:
“Help” in the context of surveys
• Definition: “To make more pleasant or
bearable”
• Synonyms: abet, aid, assist, back, backstop,
prop (up), support, bear a hand, to stand one
in good stead
• Antonyms: hinder
-Merriam-Webster
Intent of “help” in surveys for:
Survey respondents
Data analysts
• Elicit confidence in item
response for:
• Elicit confidence in item
response for:
– Difficult concepts or terms in
questions
(standardizes definitions)
– Difficult concepts or terms in
questions
(standardizes definitions)
– Less-common scenarios
(sets guidelines)
– Less-common scenarios
(sets guidelines)
When help can “hinder”:
• If it exists, but respondents:
• can’t find
• ignore it
• If what help is provided:
• is irrelevant
• contradicts question wording
• is lengthy (burdensome)
Past studies:
• Good help is good to find! Can improve data quality by
improving accuracy of item response (e.g., by helping to
standardize definitions and set guidelines)
• Inconclusive/varied on best way to get respondents to
use help when they need it
Our study examined:
• Two different forms of behind-the-scenes help for web
mode of surveys:
• Embedded links vs.
• Help buttons only
• Impact of embedded links on:
• Help text access
• Timing on forms
• Response (missingness, not accuracy)
Example:
Help button
Embedded link
and help button
Analysis #1:
2008/12 Baccalaureate and Beyond Longitudinal Study (B&B:08/12)
• Longitudinal, nationally representative study of bachelor’s degree
recipients’ education and employment experiences
• Random assignment--half of field test sample received treatment of
embedded links on 45 forms:
• Schooling (loans, post-bachelor’s degree education)
• Career (future career expectations, teaching)
• Expenses (car loans, dependents)
• Demographics (citizenship status, marital status, military status,
household composition, language spoken)
• Results: Respondents who saw links accessed help more than twice
as often as others (p < .05)
Analysis #2
2011-12 National Postsecondary Student Aid Study (NPSAS:12)
• Large, nationally representative sample of students at all levels of
postsecondary education with emphasis on how students and their families
pay for postsecondary education
• Natural experiment--comparison of three matching items between field test
and full-scale studies:
• Intensity of enrollment at main school
• Intensity of enrollment at other schools
• Received federal loans
Results: Help Text Access
Field Test:
Help button
Full-Scale:
Embedded
link
Statistical
Significance
Enrollment Intensity:
Main
0.2%
1.9%
p < .001
Enrollment Intensity:
Other
0.0%
0.5%
p < .001
Federal loans
0.2%
1.7%
p < .001
help text access 3 of 3 items
Results: Timing
Field Test:
Help button
Full-Scale:
Embedded
link
Statistical
Significance
Enrollment Intensity:
Main
14.2 s
14.1 s
Not sig.
Enrollment Intensity:
Other
9.4 s
10.8 s
Not sig.
11.3 s
13.0 s
p < .05
Federal loans
Only slight
in timing on 1 of 3 items
Results: Missingness
Field Test:
Help button
Full-Scale:
Embedded
link
Statistical
Significance
Enrollment Intensity:
Main
0.43%
0.30%
Not sig.
Enrollment Intensity:
Other
0.04%
0.05%
Not sig.
Federal loans
1.48%
0.54%
p < .01
response 1 of 3 items
Results: Response Distributions
Statistical
Significance
Enrollment
Intensity: Main
p < .01
Enrollment
Intensity: Other
Not sig.
Federal loans
p < .01
Different response distributions on 2 of 3 items
Summary Results:
help text access 3 of 3 items
Only slight I in timing on 1 of 3 items
response 1 of 3 items
Different response 2 of 3 items
Preliminary Conclusions
Embedded help links:
•
•
•
•
Seemed to increase awareness of help text
Minimal impact on timing burden
Resulted in less missingness (item-level nonresponse)
Resulted in differences in response distributions (not yet
evaluated against external benchmarks)
Contact Information
Natasha Janson
[email protected]
919-316-3394