Total Survey Error in
Disability Assessments
Measuring Physical and Cognitive Capacity in the
National Health and Aging Trends Study (NHATS)
Brad Edwards and Tamara Bruce, Westat
Presented at the International Total Survey Error Workshop
Quebec, Canada
June 2011
Overview




NHATS and capacity measures
Self report and performance
NHATS assessments
Strategies for error reduction
- design, training, standardization, parsing
out nonresponse
 Pretest results
 National experience
 Future research
2
NHATS
 Westat working with a team led by Johns
Hopkins Bloomberg School of Public Health
 Funded through cooperative agreement from NIA
 CAPI panel study with 9,000 Medicare
beneficiaries, annual interviews starting 2011
 New and improved measures of disability for
producing data on trends and trajectories; selfreports, mental and physical assessments,
eventually biomarkers, links with admin data
3
Conceptual Framework
 Blend of Nagi’s model and WHO
perspective
 Distinguish explicitly between
– Capacity to do something – basic
building blocks
– What people actually do in actual
environment – activities
 Accommodations may fill gap
4
Capacity Is Critical
Measures of capacity over time are key
elements in understanding individual patterns
of progression to activity limitations. They
allow us to track trends in function that are
independent of environmental changes or
accommodations, for understanding the
disablement process, and as targets for
interventions to prevent or slow disability.
5
Reducing Error in Capacity Measures
 Questionnaire design
 Importance of performance measures
 Challenges for quality management
– New to surveys
– Complex
– Expect high level of interviewer variance
6
Physical Capacity: Upper Extremity
Self report
Performance
 Able to
 Grip strength
– Put book on shelf/reach
overhead
– Open jar/grasp small object
7
Physical Capacity: Lower Extremity
Self Report
Performance
 Able to
 Walking speed
 Balance stands
– Walk 6/3 blocks?
– Kneel/bend over?
– Lift and carry 20/10
pounds?
– Walk up 20/10 stairs?
–
–
–
–
–
Side by side
Semi-tandem
Full-tandem
One leg eyes open
One leg eyes closed
 Chair stands
8
Cognitive Capacity: Memory
Self report
Performance
 At present time?
 Memory problems
interfere with activities?
 Memory compared to one
year ago?
 10 word recall
9
– Immediate
– Delayed
Other Cognitive Performance Measures
 Orientation
– Day of week, date, naming President and
Vice President
 Overall cognitive screening/executive
function
– Clock drawing
 Attention & interference/executive
function
– Stroop test (computerized)
10
Standardizing Administration
 Performance measures of capacity can be hard to
standardize in large-scale surveys with lay interviewers
 NHATS uses multiple methods to improve consistency
& accuracy:
– Activities Booklet design & use within CAPI framework
– Standardized presentation of cognitive tasks using flash
displays
– Survey design & interviewer feedback using CARI coding
– Formal certification process using live respondents
– Follow-up web-based recertification
11
Training Approaches
 Broader scope of demands on NHATS interviewers for
performance measures
– tests require using a variety of equipment
– navigation of unfamiliar environments
– potentially uncomfortable instructions to respondents
 Integration of video components
– recruitment
– in-person interviewer training modules such as walking
course
– certification & re-certification processes
Walking Course Timing 1
Walking Course Timing 2
Walking Course Timing 3
15
Training Video Demo
16
Item Nonresponse
 Performance measures used in depth set of
exclusion criteria to minimize burden
 High completion rate for both cognitive &
performance tasks for those eligible
 Comprehensive list of reasons why test was not
conducted allow detailed analyses
 Majority of unit nonresponse due to inability to
complete easier task & safety concerns
 Performance related to age, health rating, care
setting, memory
17
Respondent Reactions
 Avoiding “test” and “performance”
 Training interviewers how to respond
to requests (e.g., “How did I do?”)
 Physical activities: “7th inning stretch”
 Positive feedback at the end
 Interest in assessing respondent
satisfaction and rapport in relation to
nonresponse in later rounds
18
Comparison of Performance and Self
Report
 High functioning performance group
(attempting one-leg balance stands,
meeting accuracy thresholds for
Stroop) reported better self-reported
health and memory and (for the
Stroop) had higher performance on
two memory tests
19
Comparison with Other Surveys
 NLTCS screening questions in
NHATS form a bridge from previous
trend data to detailed NHATS self
reports and assessments
 Possibility of future bridge to 6
disability questions developed by
NCHS and used on the American
Community Survey
20
Conclusions
 Reducing error in capacity measures
based on performance assessment
– sharpens understanding of individual
trajectories and accommodations
– is expected to improve predictive ability
for health outcomes
 NHATS protocol
– standardizes administration
– focuses on reasons for nonresponse
21
Conclusions (2)
As a new survey, NHATS has been able to give
consideration to the implications of doing these
types of assessments from the outset, starting with
interviewer recruitment and training. Experience
from pretesting has led to refinements of these
procedures (e.g., use of videos in recruitment;
certification procedures in training) and to the
NHATS Activities Booklet. Training materials and
data collection instruments will be available later
this year at www.nhats.org.
22
Future Research
 Interviewer variance study
 Examination of
– interviewer and respondent conditioning effects
– “drift” trajectories
– item nonresponse as predictor of unit nonresponse
 Error comparison/tradeoffs between self-reports and performance
 Analysis of cognitive performance and data quality
 More development
– distance learning
– SPC charts, integration with paradata for management
– re-certification via WebEX, 2-way video
23
For more info, contact:
Brad Edwards
[email protected]
or Tamara Bruce
[email protected]
24