NONSAMPLING ERROR RESEARCH
IN PRACTICE
J. Michael Brick and Graham Kalton
Westat
OUTLINE
• Review sources of nonsampling error
• Discuss examples of nonsampling error research:
– NHES
– YATS
– NALS
– RCGS
– NIPRCS
• Discuss how we choose which methodological
studies to be conducted
2
TOTAL SURVEY ERROR
• Sampling error
• Nonsampling error
– Missing data
• Coverage error
• Nonresponse error
– Measurement error
• Response error
• Processing (coding, data entry) error
3
COVERAGE ERROR
• Undercoverage to avoid missing persons within
households
• Undercoverage due to missing households
– Studies of estimation methods to reduce bias
– Studies of efficient designs with lower coverage
rates
– Studies of the level of coverage bias for specific
topics.
4
NATIONAL HOUSEHOLD EDUCATION
SURVEY (NHES)
• Repeating RDD survey on education topics.
• Two topics of interest in 1989 were high school
dropouts and preschool enrollment of 3 to 5 year
olds.
• Concerns about undercoverage lead to an
evaluation using data from CPS supplement that
covered these topics and could be classified by
telephone status.
5
NHES COVERAGE BIAS ESTIMATES
Coverage rates-14-21 yr olds = 92%; 3-5 yr olds = 88%
Estimate
Characteristic
Bias of estimate
Tel.(%) Nontel.(%) Simple
Post.
High school dropout
7.2
33.6
-23%
-15%
Enrolled in preschool
58.9
41.6
3%
0%
6
NONRESPONSE ERROR
• Nonresponse bias studies to evaluate the level of
nonresponse bias in estimates based on:
– Frame data,
– Nonresponse follow-ups,
– Simulations.
• Studies evaluating estimation methods (e.g., use
of different auxiliary variables) to reduce bias.
• Studies evaluating methods of increasing
response rates.
7
NHES:2003 INCENTIVE EXPERIMENT
Incentives
(initial/refusal
conv)
Relative
Resp. rate
Cost ratio with refusal
subsampling rate
1.0
0.7
0.5
1. $0/$0
1.00
1.0
1.0
1.0
2. $0/$2
1.07
1.1
1.1
1.1
3. $0/$5
1.12
1.2
1.1
1.2
4. $2/$0
1.09
1.2
1.2
1.3
5. $5/$0
1.13
1.6
1.6
1.7
6. $2/$2
1.13
1.3
1.3
1.3
8
NATIONAL ADULT LITERACY SURVEY
(NALS) 1992
• Adults interviewed and given literacy tests.
Concern that nonresponse was related to literacy.
• An incentive experiment offered $0, $20, and $35.
• Response rates for $20 & $35 were about 9 pct. pts.
higher than $0; for minorities 20 pts. higher.
• Scores substantially higher for $0 vs. $20 & $35.
• Data collection cost lowest for $20.
9
RESPONSE ERROR
• Studies evaluating the level of errors due to:
– Recall
– Questionnaire design
– Sensitive items
– Interviewers
10
NATIONAL IMMUNIZATION
PROVIDER RECORD CHECK STUDY
• Parents reported children’s immunizations in a
supplement to the NHIS.
• Concerns about the accuracy of the parent reports
(especially if reported by recall rather than from
shot cards) lead to checks with medical providers.
• Provider and parent reports reconciled to create
“best” values which are treated as “true values”.
11
GROSS AND NET DIFFERENCE RATES
Survey
response
Yes
No
Total
Record data
Yes
No
A
B
C
D
A+C
B+D
Total
A+B
C+D
N
• Gross difference rate gdr = (B + C)/N
• Net difference rate (bias) ndr = (B – C)/N
12
NDR AND GDR FOR DTP,
BY USE OF SHOT CARD, 1994-1996
1994
1995
1996
ndr
Recall Shot card
-3.4
-11.6
-8.7
-14.1
-4.6
-13.0
gdr
Recall Shot card
30.9
14.9
34.0
17.0
28.0
17.0
• Parents substantially underreported DTP.
• Greater underreporting when shot cards used.
• Greater accuracy when shot cards used.
13
NHES 1995 REINTERVIEW STUDY
• The 1995 Adult Education Survey had a response
variance reinterview (n = 1,109 out of 19,722)
• 21% reported work-related (WR) activities;
– gdr = 12.5% ndr = −5.7%
• 22% reported personal development (PD);
– gdr = 14.3% ndr = −1.2%
14
NHES INTENSIVE BIAS STUDY
• Used an intensive, cognitive-type reinterview to
determine “true values”
• Small sample (n = 206) chosen to explore reporting
AE participation in WR and PD
Type Original estimate Bias estimate
WR
21%
16%
PD
20%
14%
Adj. estimate
37%
34%
15
YOUTH ATTITUDE TRACKING STUDY
• Annual cross-sectional RDD survey of 16-24 year
olds conducted for the DoD to track attitudes
towards military service.
• Design shifted to include a panel component.
• Annual enlistment propensities declined because
panel members had lower propensities to enlist.
16
YATS ADVISORY GROUP
• Panel attrition and conditioning were the main
sources considered.
• Few variables consistently related to panel attrition
and enlistment propensity.
• Revised weighting adjustments did not narrow the
gap between RDD and panel estimates.
• DoD reverted to a fully cross-sectional design.
17
THE 1991 RECENT COLLEGE
GRADUATE SURVEY
• The RCGS included:
– A nonresponse study,
– A reinterview study,
– An interviewer variance study,
– A record check study, and
– Other evaluation studies
• Made strong assumptions of additive errors to model
mean square error of estimates.
• Major contribution is understanding general
magnitude of errors by source.
18
FACTORS INFLUENCING RESEARCH
CHOICES
• Study the major error sources for the specific
survey design
• Include substantively important variables
• Conduct studies with the potential for assessing
current estimates and/or designing future surveys
• Take advantage of opportunities for research
– Small studies can be valuable
– Inexpensive studies on low priority issues or
using less rigorous methods can be worthwhile
19
References
•
•
•
•
•
•
•
Brick, J.M., Burke, J., and West, W. (1992). Telephone undercoverage bias of 14- to 21-yearolds and 3- to 5-year-olds (Technical report No. 2, NCES 92-101). Washington, DC: U.S.
Department of Education.
Brick, J.M., Cahalan, M., Gray, L., and Severynse, J. (1994). A study of selected nonsampling
errors in the 1991 Survey of Recent College Graduates. U.S. Department of Education, Office
of Educational Research and Improvement, NCES 95-640.
Brick, J.M., Hagedorn, M.C., Montaquila, J., Roth, S.B., and Chapman, C. (2004). Using an
experiment to design an RDD survey. Proceedings of the Survey Methods Section of the
American Statistical Association [CD-ROM], 4923-4928.
Brick, J.M., Kalton, G., Nixon, M., Givens, J., and Ezzati-Rice, T. (2000). Statistical issues in
a record check study of childhood immunization. Proceedings of the 1999 Federal Committee
on Statistical Methodology Research Conference (Statistical policy working paper 30, 625634).
Brick, J.M., and Morganstein, D. (1996). Estimating response bias in an adult education
survey. Proceedings of the Survey Research Methods Section of the American Statistical
Association, 728-733.
Brick, J.M., Wernimont, J., and Montes, M. (1996). The 1995 National Household Education
Survey: Reinterview results for the adult education component (NCES 96-14). Washington,
DC: Office of Educational Research and Improvement, U.S. Department of Education.
Mohadjer, L., Berlin, M., Rieger, S., Waksberg, J., Rock, D., Yamamoto, K., Kirsch, I.,
Kolstad, A. (1997). The role of incentives in literacy survey research, Chapter 10 pp 209-244
in Adult Basic Skills: Innovations in Measurement and Policy Analysis, eds. Tuijnman, Kirsch,
and Wagner, Hampton Press, 1997.
20