Oh, Just One More Thing …
Leveraging “Leave-Behinds” in Data Collection
Presenter:
Michael W. Link
Chief Methodologist /SVP- Nielsen Measurement Institute
The Nielsen Company
1
NSF Survey Workshop
Nov 8, 2012
Title of Presentation
Copyright © 2012 The Nielsen Company. Confidential and proprietary.
Key Lessons of “Leave-Behinds”
•  Fairly prevalent in practice, but rare as a research topic;
•  Much is known of their components, but not much of the methodology
itself;
•  Typically the province of large, more complex data collection efforts;
•  Used for many different purposes;
•  Come in many sizes and forms – and may or may not be a traditional
“survey”;
•  Growth of new technologies (mobile & online) and decline in funding
for many studies will likely make these more attractive in the future
2
Copyright © 2012 The Nielsen Company. Confidential and proprietary.
What is a “Leave-Behind” (and What is Not)?
•  Term “Leave-behind or “leave-behind survey” not defined in
Encyclopedia of Survey Research (Lavrakas et al) or other
prominent survey texts;
•  As a search term it is nearly useless:
–  Only research publication: John P Robison (1969) “Television and Leisure
Time: Yesterday, Today, and (Maybe) tomorrow. Public Opinion Quarterly
33:210-222.
•  Ask survey professionals and most are either confused by the
term or interpret it quite differently
3
Copyright © 2012 The Nielsen Company. Confidential and proprietary.
All Else Fails, Ask Your Social Network Friends!
• Nielsen
• RTI International
• GfK
• Westat
• NORC
• Abt Associates
• Abt/SRBI
• NCHS/CDC
• Bureau Labor Statistics
• Census
• Kaiser Family Foundation
• Google
• ICFI (formerly Macro)
• Washington State U
• U of Michigan
• U of South Carolina
• U of Maryland
• U of Indiana
• SSRS
• Pew Center
Input on defining the concept and
identifying examples and research in
this area
4
Copyright © 2012 The Nielsen Company. Confidential and proprietary.
Characteristics of “Leave-Behinds”
•  Involves self-administration;
•  Data collection mode is often different than the initial mode (multimode total design);
•  Typically provides addition information for a study, but can in rare
instance actually collect the primary data;
•  Task completed after close-out of an initial survey or interview (data
collection in stages);
•  Nearly always used with large, more complex studies – rarely with
smaller, straightforward studies
•  Task may be a survey or could involve other form of data collection –
diary, electronic monitor, physical specimen collection;
5
Copyright © 2012 The Nielsen Company. Confidential and proprietary.
Defining the Concept
•  Leave-Behind (def.): “A form of data collection which is selfadministered by the respondent and completed sometime after
completion of an initial survey”
•  For example, would include:
–  Survey left for respondent to complete after an initial face-to-face
interview;
–  Transfer of respondent from telephone interview to Interactive Voice
Response IVR) system for additional questions;
–  Recruitment and screening interview via one mode, then left or sent an
activity diary
•  For example, would not include:
–  Traditional mail or self-initiated online survey;
–  Audio Computer-Assisted Self Interview (ACAI) during a larger in-home
interview;
6
Copyright © 2012 The Nielsen Company. Confidential and proprietary.
Why Use Leave-Behinds?
•  Collect more information -- expand initial data collection effort;
•  Reduce respondent burden – allow them to complete a portion of data
collection on their schedule;
•  Privacy/sensitive topics – less social-desirability, higher quality data;
•  Data quality -- More effective than recall, especially for capture of
multiple, frequent incidences (i.e., use of activity dairy)
•  Unique information -- Only way to capture required information – ex.
Outside-of-lab physiological measures
7
Copyright © 2012 The Nielsen Company. Confidential and proprietary.
What the Empirical Research Tells Us
•  We know a lot about the components or concepts associated with
leave-behind approaches (specific modes, design of tools, etc,), but
know very little about leave-behinds as a methodology;
•  Conducted extensive search:
–  Public Opinion Quarterly
–  International Journal of Public Opinion Research
–  Journal of Official Statistics
–  Field Methods
–  Survey Practice
–  American Journal of Epidemiology
–  Epidemiology
–  Google Scholar: Top 250 entries for “leave behind surveys”
•  Turned up 4 articles which mention the approach, but none focus on
assessing the methodology
8
Copyright © 2012 The Nielsen Company. Confidential and proprietary.
Why the Dearth of Research?
• Studies may have been conducted on this topic, but not
described the method as “leave behind”
–  Very likely, particularly in the public health literature
–  Requires pre-knowledge of the study or scowering of the
methodology sections of articles to identify
• Discussed as a component of a broader study, but not the
specific focus;
• Viewed as “adjunct data” and not the main focus of a
study, hence not explicitly examined or findings kept
internal and not reported publicly
–  Learnings are there, but not widely-shared
9
Copyright © 2012 The Nielsen Company. Confidential and proprietary.
Example #1: Health & Retirement Survey (HRS)
• Longitudinal study of health, retirement and aging
• Sponsored by National Institute on Aging
• Conducted by University of
• Study population: 50+ nationwide in US (~13,000
respondents)
• Panel survey conducted every two years
–  Alternate face-to-face and telephone modes every other cycle
10
Copyright © 2012 The Nielsen Company. Confidential and proprietary.
HRS Leave Behind Component
•  Respondents are left a booklet which examines cognitive status and
psychosocial topics as well as subjective work and retirement
questions
•  Researchers use this to:
–  Enhance/expand the data they collect in conjunction with the face-to-face
component (which is already 140 minutes)
–  Provide privacy for sensitive questions and time for reflection for
subjective questions
•  Cooperation rates: 89.9% (2006), 85.1% (2008)
–  Design changes cause of decline; 2010 not yet available
–  Lower cooperation among Hispanics, Blacks, those most recently added
to the panel
•  More information: http://hrsonline.isr.umich.edu/
11
Copyright © 2012 The Nielsen Company. Confidential and proprietary.
Example #2: Consumer Expenditure Survey (CE)
•  Panel study collecting data on buying habits of American consumers,
expenditures, income and consumer characteristics
–  Contributed to revision of Consumer Price Index (CPI)
•  Sponsored by Bureau of Labor Statistics [Contact: Adam Safire]
•  Conducted by Census
•  Study population: Household and person-level information for noninstitutionalized US population
•  Panel study with two separate sample components of (a) Quarterly
Interview and (b) Purchase Diary
–  ~12,000 households per sample component
12
Copyright © 2012 The Nielsen Company. Confidential and proprietary.
CE Leave Behind Component
•  Expenditure Diary:
–  Step 1: Face-to-face administration of Household Characteristics
Questionnaire – demographics, HH composition, work experience, and
earnings
–  Step 2: Leave-behind 1 week expenditure diary – for entire HH, covering
food expenditures (in and out of home), clothing, and all other goods and
services purchases
–  Step 3: Pick up Week 1 Diary and leave-behind Week 2 Diary
–  Step 4: Pick-up Week 2 Diary and administer final questionnaire on work
experience and income
•  Researchers use leave-behind to:
–  Capture detailed activity data which is too extensive to be captured or
assessed via standard recall questions
13
Copyright © 2012 The Nielsen Company. Confidential and proprietary.
CE Leave Behind Component (con’t)
•  Cooperation rates: 76.8% (2009)
–  Lower cooperation among Hispanics, Blacks, and younger adult
households
•  Undertaking extensive redesign (Project Gemini) of both Diary and
Quarterly Interview components:
–  Move from paper to electronic version (tablet) of Diary to improve data quality and
completeness
–  Examining other techniques as alternatives: SKU scanning, collection of receipts,
download of credit card purchase information, 3rd party vendor data
•  More information: www.bls.gov/cex
14
Copyright © 2012 The Nielsen Company. Confidential and proprietary.
Example #3: National Health and Nutrition
Examination Survey
•  Study designed to assess health and nutritional status of adults and
children in the US
•  Sponsored and Conducted by National Center for Health Statistics /
Centers for Disease Control & Prevention
•  Study design combines multiple interviews, physical examines, and
leave-behind components
• 5,000 persons each year, nationally representative from
15 selected counties
–  In-person recruitment with initial questionnaire administration
–  Examines and blood work in special mobile vans w additional
interview questions
15
Copyright © 2012 The Nielsen Company. Confidential and proprietary.
NHANES Leave-Behind Component
•  Two leave-behind components:
–  Home urine collection (6+ yrs old) to check kidney functioning
•  Urine kit provided before respondent leaves exam center
•  Tech explains how to collect and return specimen
•  This is a second sample (first collected at exam center)
–  Physical Activity Monitor (6+yrs old) to measure physical activity levels
and compliance with physical activity recommendations
–  Tech provides wrist-strap-on device at exam center
–  Respondent wears and returns device after 7 days
•  Researchers use leave-behind to:
–  Collect unique data than cannot be provided otherwise;
–  Reduce burden (eliminate need to return to exam center)
–  Provide more accurate data than can be obtained via recall
•  Cooperation rates:
–  Home urine collection: 94.1% (2011), 94.6% (2012)
–  Physical activity monitor: 88% mailed back (2012)
•  More information: www.cdc.gov/nhanes
Selected Other Examples of Leave-Behinds
•  Media & Consumer Study (GfK):
–  Questionnaire booklet on consumer behaviors left with respondents after
initial interview
•  Kids and Media Use Study (Kaiser Family Foundation)
–  Children provided 2 day media use diaries after completion of in-school
interview
•  Pathways of Diagnoses and Services (NCHS/CDC)
–  Leave-behind questionnaire mailed to households with children after
completion of a 40 minute telephone interview
•  Project for Excellence in Journalism (Pew Research)
–  Questionnaire left with business leaders after initial interview – questions
pertain to business and may be filled out by others in the company
•  Metered TV Ratings Service (Nielsen)
–  After extensive screening and household characteristics questionnaire,
respondents have “People Meters” installed in each room with a TV to
record their presence and TV viewing in the room
How New Technologies May Change the Landscape
•  Online, Mobile platforms (phones and tablets), Blue Tooth enabled
devices and other new technologies offer new and innovative leavebehind methodologies
•  Online: Provides an easy, relatively inexpensive and consistent
interface across multiple devices and platforms (traditional PCs,
laptops, tablets, mobile phones) for capture and transmission of data
–  Examples:
 Web-based survey to augment initial survey, particularly for sensitive
items
 Electronic versions of traditional eave-behind tools (such as activity
diaries)
 Can facilitate quick, easy communication with respondents during selfadministered leave-behind phase of data collection (rather than more
expensive and intrusive face-to-face or telephone contacts)
18
Copyright © 2012 The Nielsen Company. Confidential and proprietary.
Leave-Behinds and New Technologies (con’t)
•  Mobile: A platform offering a diverse set of potential data collection
tools – surveys, image collection, audio capture, GPS, app-based
entries, Twitter, Blue Tooth enabled devices, etc.
–  Examples:
 Leave-behind surveys accessible on web via mobile;
 Detailed transportation / mobility studies to augment survey data;
 Audio journals kept by cancer patients;
 “Twitter Diaries” capturing and submitting information in real-time about
out-of-home activities;
 Capture still pictures of stores and brands purchased –or – video of
family living conditions
 Devices capturing physiological readings uploaded and sent via Blue
Tooth and mobile.
19
Copyright © 2012 The Nielsen Company. Confidential and proprietary.
Leave-Behinds and New Technologies (con’t)
•  Blue Tooth: Hundreds of devices are now available which measure or
capture an array of activities and transmit the information via a BTenabled mobile device.
•  Examples:
• Measuring exposures to environmental hazards
• Capturing uptake of medicines
• Record blood glucose, blood oxygen and pulse data passively
20
Copyright © 2012 The Nielsen Company. Confidential and proprietary.
Areas Ripe for Investigation
•  What lessons can be drawn by focusing on leave-behinds as a
distinct methodological approach? – or – are leave behind
approaches and techniques so unique it makes them difficult to
compare and generalize?
•  Unit Non-Response Errors: Cooperation & compliance :
–  How does this differ from the “primary” collection effort in terms of
demographics, modes, approaches, incentives, motivational appeals,
etc?
–  Which techniques/approaches work best in this context?
–  Do leave-behinds really reduce respondent burden?
–  Influence of having developed a relationship during the primary data
collection – how does that affect leave-behind compliance & quality?
21
Copyright © 2012 The Nielsen Company. Confidential and proprietary.
Additional Areas of Potential Investigation
•  Measurement error:
–  Are mode effects different in the context of a leave-behind than when
different modes are used as the primary data collection vehicle?
–  Survey data -vs – data obtained via other techniques or monitoring?
–  If initial data collection is lengthy, does leave-behind
–  Does item non-response differ in a leave-behind situation
–  Is there greater satisficing in the leave-behind context than in other
survey contexts
•  Data quality issues:
–  How do leave-behind approaches differ from other self-administration?
–  Do timing and context change responses one might obtain otherwise?
–  Compatibility Effects - The difficulty of comparing surveys done at
different times by different groups using different methods.
22
Copyright © 2012 The Nielsen Company. Confidential and proprietary.
Key Lessons of “Leave-Behinds” Revisited
•  Fairly prevalent in practice, but rare as a research topic;
•  Much is known of their components, but not much of the methodology
itself;
•  Typically the province of larger, complex data collection efforts;
•  Used for many different purposes – additional information, privacy,
data quality, unique data capture;
•  Come in many sizes and forms – and may or may not be a traditional
“survey”;
•  Growth of new technologies (mobile & online) and decline in funding
for many studies will likely make these more attractive in the future
23
Copyright © 2012 The Nielsen Company. Confidential and proprietary.
Thank you!
Michael Link
[email protected]
24
Copyright © 2012 The Nielsen Company. Confidential and proprietary.