Optimal Adaptive Survey
Design
Lars Lyberg, Frauke Kreuter,
and James Wagner
ITSEW 2010
Stowe, VT, USA, June 16
The Survey Process
Research Objectives
Population
Mode of Administration
Questions
Sampling
Questionnaire
Design
Data Collection
Data Processing
Analysis/Interpretation
revise
revise
Concepts
What Should Be Designed?



Requirements+specifications+operations
Ideal goal+ Defined goal+Actual results
Good survey design means control of
accuracy through the specs (QA) and
control of operations (QC)
Some Early Thinking


Hansen-Hurwitz-Pritzker 1967
 Take all error sources into account
 Minimize all biases and select a minimum-variance
scheme so that Var becomes an approximation of
(a decent) MSE
 The zero defects movement that later became Six
Sigma
Dalenius 1969
 Total survey design
Some More Thinking



Textbook on total survey design
 Hansen-Hurwitz-Cochran-Dalenius
Survey models and specific error sources
Cochran’s comment from 1968
Alternative Criteria
of Effectiveness





Minimizing MSE for a given budget
while meeting other requirements
Maximizing fitness for use for a given
budget
Maximizing comparability for a given
budget
All these reversed
Something else?
The Elements of Design






Assessing the survey situation (requirements)
Choosing methods, procedures, “intensities”, and
controls (specifications)
Allocating resources
Assessing alternative designs
Carry out one of them or a modification of it
Have a Plan B
So, What’s the Problem?






No established survey planning theory
Multi-purpose, many users
The information paradox
Uninformed clients/users/designers
Much design work is partial, not total
Limited knowledge of effects of
measures on MSE and cost
More Problems




Decision theory and economics theory
not used to their potential
New surveys conducted without
sufficient consideration of what is
already known
No one knows the proper allocation of
resources put in before, during and
after
The literature is small
Various Skills Needed
Which Calls for a Design Team





Survey methodology
Subject-matter
Statistics (decision theory, risk analysis,
loss functions, optimization, process
control)
Economics (cost functions, utility)
IT
The Adaptive Element

The entire survey process should be
responsive to anticipated uncertainties that
exist before the process begins and to real
time information obtained throughout the
execution of the process
or

Use process data (paradata) to check, and if
necessary, adjust the process
We Should Assemble
What We Know





Assessment methods
Design principles
Trade-offs and their effects
The potential offered by other
disciplines
We shouldn’t accept partial designs
Apply Design Principles




If
If
If
If
pop is skewed then….
pop is nested then….
questions are sensitive then….
a high NR rate is expected then…
Apply SOPs, CBMs
or Best Practices

Part of the design is to use known,
dependable methods
Examples of Trade-offs







Accuracy vs timeliness
Response burden vs wealth of detail
Conduct survey vs other information
collection
Large n vs smaller n
Mixed vs single mode
NR bias vs measurement error
NR vs interpretation by family members
Process view





Upstream thinking (prevention)
Understanding variation
Measure cost of poor quality and waste
Intervention or improvement actions
should be based on good data and
statistical analysis
Continuous monitoring
Tentative Course Syllabus




The elements of design
Real world examples (e.g., CPS
Technical Paper 63, PIAAC, the Monthly
Retail Trade Survey, the Annual Survey
of Hale Mountain Fish & Game Club,
VT)
The literature on optimal decisions
Theory for adaptive treatment design
and risk management
Course syllabus continued






Data for monitoring and decision making
Analysis of such data
Design lessons learned
Examples of bad designs and not so great
trade-offs
Student project with TSE perspective
Student presentations