Evaluating the
Effectiveness of Two
Strategies to Improve
Telephone Survey
Response Rates of
Employers
AAPOR
May 17, 2013
Presented by:
Jeremy Pickreign
Heidi Whitmore
Background
• Low response rates in establishment surveys are common
• Common approaches to improve response rates
• Advance notice to participants
• Follow-up reminders
• Personalized communication
• Household surveys commonly use incentives to improve response
rates.
• Notable lack of studies on the impact of incentives in establishment
surveys
• Recent literature shows mixed results
–
–
–
–
–
Cycyota and Harrison (2002)
Biemer, Ellis, Pitts, and Robbins (2007)
Ott and Beckler (2007)
Moore and Ollinger (2007)
Cook, LeBaron, Flicker, and Flanigan (2009)
Evaluating the Effectiveness of Two Strategies to Improve Telephone Survey Response Rates of Employers
2
Research Goal
• Conduct a randomized experiment of firms on the impact
of financial incentives and personalized advance letters
on survey response rates
• Research Questions:
• Do respondents receiving a financial incentive have a higher
response rate than respondents not receiving a financial
incentive?
• Do respondents receiving a prepaid financial incentive have a
higher response rate than respondents receiving a promise of a
financial reward?
• Do respondents sent a personalized advance letter have a higher
response rate than those sent a generic advance letter?
Evaluating the Effectiveness of Two Strategies to Improve Telephone Survey Response Rates of Employers
3
The Survey
• 2011 California Employer Health Benefits Survey
•
•
•
•
Sponsor: California HealthCare Foundation
Conducted annually since 1999
Since 2004, response rates between 35% and 40%
Panel component for firms with 10+ workers
• Computer Assisted Telephone Interviewing by National
Research, LLC
• Field period was from June through October
• Final response rate
• All Firms: 36%
• Non-Panel Firms with 3-49 Workers: 27%
Evaluating the Effectiveness of Two Strategies to Improve Telephone Survey Response Rates of Employers
4
Current Activities to Maintain
Response Rates
• Attempt to re-interview most firms responding to previous rounds of
the survey
• Pre-call smallest firms (3-49 workers) to obtain accurate contact
information prior to fielding the survey
• Send advance letters prior to call attempts
• Make numerous call attempts per firm during the field period
• Offer to send respondents completing the survey a copy of the results
to benchmark against their health benefits
• Financial incentives have never been a part of the survey process
Evaluating the Effectiveness of Two Strategies to Improve Telephone Survey Response Rates of Employers
5
Methods
• Sampled non-panel employers with 3-49 workers
(n=1024) randomly assigned to one of three groups
1. receiving a financial incentive with the initial mailing (n=250)
2. receiving a promise of a financial reward upon completion of
the survey (n=250)
3. control group with no financial reward (n=524)
• The financial incentive was $20
• Pre-called non-panel employers with 3-49 workers.
• 513 firms successfully contacted
• 511 firms not successfully contacted
Evaluating the Effectiveness of Two Strategies to Improve Telephone Survey Response Rates of Employers
6
Analysis Group Sample Sizes*
Sent
Personalized
Advance Letter
Sent Generic
Advance Letter
Total
Prepaid Monetary
Incentive
109
114
223
Promised Monetary
Incentive
129
99
228
No Incentive / Control
Group
275
298
573
Total
513
511
1,024
* 49 Firms with Returned Advance Letters Assigned to Control Group
Evaluating the Effectiveness of Two Strategies to Improve Telephone Survey Response Rates of Employers
7
Response Rate (95% CI) by Incentive Group and Advance
Letter Group for Non-Panel Firms with 3-49 Workers
45.0%
40.0%
38.1%
35.7%
35.0%
33.2%
30.0%
31.1%
30.0%
31.0%
30.0%
27.3%
28.1%
23.5%
23.0%
26.3%
25.0%
22.0%
21.8%
24.4%
20.0%
18.3%
15.0%
14.1%
12.2%
10.0%
5.0%
0.0%
All Firms
Prepaid Monetary
Promised
Incentive
Monetary Reward
No Incentive / Sent Personalized Sent Generic
Control Group
Advance Letter Advance Letter
Evaluating the Effectiveness of Two Strategies to Improve Telephone Survey Response Rates of Employers
8
Response Rate (95% CI) by Incentive Group by Advance
Letter Group for Non-Panel Firms with 3-49 Workers
50.0%
44.0%
40.0%
30.0%
39.5%
31.1%
27.3%
29.1%
30.8%
38.2%
31.8%
27.9%
26.9%
25.4%
23.5%
20.0%
40.1%
21.4%
19.9%
18.6%
14.9%
10.0%
11.9%
9.9%
6.4%
0.0%
-2.2%
Sent Personalized Advanced Letter
Sent Generic Advanced Letter
-10.0%
All Firms
Prepaid
Monetary
Incentive
Promised
Monetary
Reward
No Incentive /
Control Group
Prepaid
Monetary
Incentive
Evaluating the Effectiveness of Two Strategies to Improve Telephone Survey Response Rates of Employers
Promised
Monetary
Reward
No Incentive /
Control Group
9
Impact of Incentives on
Completion Counts
Sample
Size
Response
Rate
Completion
Count
Expected
Completion
Count with
No
Incentives
All Firms
1024
27.3%
142
148
-4.1%
Prepaid
Monetary
Incentive
223
22.0%
23
32
-28.1%
Promised
Monetary
Reward
228
30.0%
36
33
9.1%
No Incentive
/ Control
Group
573
28.1%
83
83
0.0%
Evaluating the Effectiveness of Two Strategies to Improve Telephone Survey Response Rates of Employers
Percentage
Improvement
Due to
Incentives
10
Impact of Pre-Calling on
Completion Counts
Sample
Size
Response
Rate
Completion
Count
Expected
Completion
Count with
Generic
Advance
Letter
All Firms
1024
27.3%
142
56
153.6%
Sent
Personalized
Advance
Letter
513
31.0%
114
28
307.1%
Sent Generic
Advance
Letter
511
18.3%
28
28
0.0%
Evaluating the Effectiveness of Two Strategies to Improve Telephone Survey Response Rates of Employers
Percentage
Improvement
Due to
Personalized
Advance
Letter
11
Base Logistic Regression
• Dependent Variable: Survey Completion
• Key Independent Variables:
• Advance Letter Indicator
• Incentive Indicator
• Control Variables:
• Firm Size
• Principle Industry
• Model included an interaction between the key
independent variables
• Model fits the data fairly well
• Prediction accuracy of 62%
Evaluating the Effectiveness of Two Strategies to Improve Telephone Survey Response Rates of Employers
12
Modified Logistic Regression
• Missing from the base model are key meta-variables
• Number of Dial Attempts
• Callback Conversion Eligibility
• Meta-variables for 2011 unavailable
• Used basic imputation techniques to assign values from
the 2010 survey to the 2011 database
• 20 strata based on firm size, advance letter indicator, and survey
response category
• 2010 observation values randomly assigned with replacement to
a 2011 observation within each stratum
• Modified model fits data fairly well
• Prediction accuracy of 74%.
Evaluating the Effectiveness of Two Strategies to Improve Telephone Survey Response Rates of Employers
13
Logistic Regression Results
Odds Ratio
Odds Ratio
(with Meta Data)
0.1257 *
0.4249 *
2.4685 *
3.0514 *
REF
REF
0.5851 *
0.6780
Promised Monetary Incentive
1.3210
1.2387
No Incentive / Control Group
REF
REF
Independent Variable
Intercept
Advance Letter
Personalized Advance Letter
Generic Advance Letter
Incentive
Prepaid Monetary Incentive
Meta Variables
Dial Attempts
0.9843 *
Callback Conversion Eligibility
0.2049 *
Evaluating the Effectiveness of Two Strategies to Improve Telephone Survey Response Rates of Employers
14
Discussion
• Response rates did not significantly improve over
the control group with the offer of an incentive,
whether prepaid or promised
• Consistent with other research (Biemer et al., 2007;
Cycyota & Harrison, 2002).
Evaluating the Effectiveness of Two Strategies to Improve Telephone Survey Response Rates of Employers
15
Discussion (continued)
• The promise of a monetary reward versus a prepaid
monetary incentive produced mixed results.
• Contrary to other research (Church, 1993; Cook et al., 2009; Ott
& Beckler, 2007) which suggests that a prepaid incentive is more
likely to produce a completed interview.
• Promising a monetary reward among respondents receiving a
generic advance mailing appears to be effective (p = 0.0344) at
improving response rates
• Logistic regression results find that promising a monetary reward
has no effect while providing a prepaid incentive reduces the
likelihood of obtaining a completed interview.
Evaluating the Effectiveness of Two Strategies to Improve Telephone Survey Response Rates of Employers
16
Discussion (continued)
• Sending a personalized advance letter produces a
significant improvement in response rate.
• Consistent with previous research (Martin, Duncan, Powers, &
Sawyer, 1989; Roth & BeVier, 1998; Yammarino, 1991)
• Logistic regression shows that firms receiving a personalized
advance letter are 2.5 to 3 times more likely to complete an
interview.
Evaluating the Effectiveness of Two Strategies to Improve Telephone Survey Response Rates of Employers
17
Limitations
• The $20 cash offer might have been insufficient to encourage
participation.
• Cash offer might be inappropriate for some firms.
• The lack of a choice in incentive or reward might have dampened
participation.
• Non-significant findings might be due to insufficient sample size.
• Effect of the Advance Letter might be confounded because all
successfully pre-called respondents received the personalized
advance letter.
Evaluating the Effectiveness of Two Strategies to Improve Telephone Survey Response Rates of Employers
18
Conclusions
• A personalized advance letter is effective in improving
response rates
• Monetary incentives appear to have no effect in improving
response rates
• The effectiveness of a personalized advance letter far
exceeds the effectiveness of either a prepaid monetary
incentive or the promise of a monetary reward
• Much work remains to provide clear guidance to
researchers conducting establishment surveys
Evaluating the Effectiveness of Two Strategies to Improve Telephone Survey Response Rates of Employers
19
References
• Biemer, P., Ellis, C., Pitts, A., & Robbins, K. 2007. Do Monetary Incentives Increase Business
Survey Response Results? Results from a Large Scale Experiment. Paper presented at the Third
International Conference on Establishment Surveys, June 18-21, Montreal, Quebec, Canada.
• Church, A. H. 1993. Estimating the Effect of Incentives on Mail Survey Response Rates: A MetaAnalysis. Public Opinion Quarterly, 57: 62-79.
• Cook, S., LeBaron, P., Flicker, L., & Flanigan, T. S. 2009. Applying Incentives to Establishment
Surveys: A Review of the Literature. Paper presented at the American Association for Public
Opinion Research 64th Annual Conference, Hollywood, Florida.
• Cycyota, C. S., & Harrison, D. A. 2002. Enhancing Survey Response Rates at the Executive Level:
Are Employee- or Consumer-Level Techniques Effective? Journal of Management, 28: 151-176.
• Martin, W. S., Duncan, W. J., Powers, T. L., & Sawyer, J. C. 1989. Costs and Benefits of Selected
Response Inducement Techniques in Mail Survey Research. Journal of Business Research, 19: 6779.
• Ott, K. & Beckler, D. 2007. Incentives in Surveys with Farmers. Paper presented at the Third
International Conference on Establishment Surveys, June 18-21, Montreal, Quebec, Canada.
• Roth, P. & BeVier, C. 1998. Response Rates in HRM/OB Survey Research: Norms and Correlates,
1990-1994. Journal of Management, 24: 97-117.
• Yammarino, F. J., Skinner, S. J. & Childers, T. L. 1991. Understanding Mail Survey Response
Behavior. Public Opinion Quarterly, 55: 613-629.
Evaluating the Effectiveness of Two Strategies to Improve Telephone Survey Response Rates of Employers
20
Jeremy Pickreign
[email protected]
Thank You!