Persuasion and Persistence:
A Large Scale Field Experiment
David W. Nickerson
Associate Professor
Department of Political Science
Temple University
How Politicians Learn
Princeton University
May 17, 2017
Outline
Example of field experiment that OFA conducted
– Can volunteers persuade voters?
– How long does the effect last?
The Field Experiment
Work would not have been possible without the
assistance of :
– Peter Backof
– OFA Analytics
– OFA Data
– OFA Field Staff
– OFA Volunteers
Practitioner State of the Art
• Volunteers generally not used for persuasion
– Volunteer recruitment
– Mobilization
– Events / Rallies
– ID calls
Practitioner State of the Art
• Volunteers generally not used for persuasion
• Viewed as too hard
– May go off message
– Hostile reactions
– Preaching to converted
Practitioner State of the Art
• Volunteers generally not used for persuasion
• Viewed as too hard
• Volunteer persuasion mainly used for low
profile elections
– Interactions often viewed more educational than
persuasive
Practitioner State of the Art
• Volunteers generally not used for persuasion
• Viewed as too hard
• Volunteer persuasion mainly used for low
profile elections
Question: Can volunteers effectively persuade
people to vote for a candidate?
Academic Skepticism
• Few good targets
– Most People have made up their minds
– “Undecideds” actually break in pre-ordained ways
Academic Skepticism
• Few good targets
• Message may not be heard
– Many competing messages
– Voters may resist/reject messages from overtly
partisan sources
Academic Skepticism
• Few good targets
• Message may not be heard
• Campaigns in general have minimal effects
– Why should one component of a campaign be
different?
Academic Skepticism
•
•
•
•
Few good targets
Message may not be heard
Campaigns in general have minimal effects
Prior persuasive experiments
– Low salience offices: Mixed results
– Higher salience offices: Null or negative
– Treatment effect very short lived
Academic Skepticism
• Few good targets
• Message may not be heard
• Campaigns in general have minimal effects
• Prior persuasive experiments
But …
1) Volunteer phone calls are successful at
boosting turnout and donations
2) Seems plausible and need to do something
Goal of the Client
Question #1: Are volunteer calls successful at
persuading voters?
Question #2: Can we predict the type of person
who is susceptible to persuasive outreach?
Question #3: Are the same types of people
persuadable across platforms?
Questions Answered Today
• Can a volunteer be heard over the din?
• Can a volunteer call change the impression of
an incumbent?
• Can a volunteer call change a target’s vote?
• Are there populations particularly susceptible
to volunteer calls?
• How long does this persuasion effect last?
Experimental Design
Treatment
Group
Calls
Made
Survey
50%
Random
Registered
Voters
Robodialer
Control
Group
Survey
50%
15
Scope of the Experiment
• 19 states
– AZ, CO, FL, GA, IA, IN, ME, MI, MN, MO, NC, NH,
NM, NV, OH, OR, PA, VA, WA
• Calls placed between January 23rd and
February 24th, 2012
• In-state volunteer callers
• 345,481 calls attempted
• 54,649 calls completed (16% contact rate)
Scale of OFA Volunteer Call Experiment
8K
1K
5K
16K
7K
20K
17K
19K
34K
3K 43K
25K
26K
14K
9K
20K
11K
7K
62K
17
Volunteers
• Anyone who wanted to volunteer for the
campaign
• Highly heterogeneous, but broad tendencies
– Older
– Educated
– Female
– Democrats
• Slightly more experienced than later
volunteers
Treatment: Introduction and ID
• Greetings
• Trying to learn the perspective of people
• Have you thought about who you’ll support in
the election for President this year? Are you
100% certain of that?
– If definite Obama, then Volunteer ask.
– If definite Romney, then polite goodbye with
request to keep an open mind.
– Else  persuasion
Treatment: Persuasion Message
Okay, we’re calling people like you to talk about the important
economic issues facing the country. (Engage using the talking
points below while personalizing your story.)
o The promise of this country has always been that if you work hard and
play by the rules, you can provide a decent life for your family.
o But for decades, that promise of middle class security has been
slipping away for too many Americans. Even as they worked harder,
wages stagnated, health care costs soared and fewer employers
offered retirement benefits. Good manufacturing jobs were shipped
overseas and college tuition costs skyrocketed.
o And after a difficult decade, in late 2008, middle class families were
devastated when their 401ks were wiped out on Wall Street, the value
of their houses plummeted and millions of jobs disappeared
overnight.
o This is what the President has been working to turnaround.
…
Measuring Outcome
• 2 days after volunteer call attempted
• Independent vendor (8 questions)
– In November 2012 there will be an election for
US President and other offices. Will you
definitely vote, probably vote, is there a 50/50
chance you will vote, will you probably not vote
or will you definitely not vote in this presidential
election in November 2012?
Measuring Outcome
• 2 days after volunteer call attempted
• Independent vendor (8 questions)
– Vote intention
– If the November 2012 presidential election were
held tomorrow, would you vote for [ROTATE:
Democrat Barack Obama / the Republican
Candidate] or would you vote for [ROTATE: the
Republican Candidate / Democrat Barack
Obama]? Are you absolutely certain?
Measuring Outcome
• 2 days after volunteer call attempted
• Independent vendor (8 questions)
– Vote intention
– If the November 2012 presidential election were
held tomorrow, would you vote for [ROTATE:
Democrat Barack Obama / Republican Mitt
Romney] or would you vote for [ROTATE:
Republican Mitt Romney / Democrat Barack
Obama]? Are you absolutely certain?
Measuring Outcome
• 2 days after volunteer call attempted
• Independent vendor (8 questions)
– Vote intention
– Vote choice
• Do you APPROVE or DISAPPROVE of the way Barack Obama is
handling his job as President? [PROBE:] And is that STRONGLY
(approve/disapprove) or SOMEWHAT (approve/disapprove)?]
• Do you AGREE or DISAGREE with the following statement:
Barack Obama has been effective as President? [PROBE:] And is
that STRONGLY (agree/disagree) or SOMEWHAT
(agree/disagree)?
Measuring Outcome
• 2 days after volunteer call attempted
• Independent vendor (8 questions)
– Vote intention
– Vote choice
– Approval
– Finally, in the past few days, have you received
any contact from political campaigns?
Measuring Outcome
• 2 days after volunteer call attempted
• Independent vendor (8 questions)
– Vote intention
– Vote choice
– Approval
– Recall
• Response rate roughly 9%
• N = 18,554
Balance Among Respondents
Condition
Female
Age
Black
White
D
Ind.
R
Control
58%
60
6%
91%
30%
37%
33%
Treatment
57%
60
6%
91%
30%
37%
33%
Condition
Support
Turnout
2010
2008
2006
2004
Control
46
73
84%
92%
66%
69%
Treatment
46
74
85%
92%
67%
70%
Treatment Contact Rates
Among the 350K attempts: 16% contact rate
Among survey respondents: 38% contact rate
External validity to non-respondents is an open
question.
Did People Recall the Contact?
Condition
Recall
Treatment
45.7%
Control
30.8%
ITT
14.9pp
(0.7)
Contact Rate
ATE
38%
39.3pp
(1.7)
Did stated “Intent to Vote” Increase?
Condition
Definitely Will Vote
Treatment
88.3%
Control
87.4%
ITT
0.9pp
(0.45)
Contact Rate
38%
ATT
2.5pp
(1.3)
0.040
0.035
ATT on “Definitely will Vote”
by Support Score
0.038
0.030
0.025
0.025
0.020
0.015
0.010
0.010
0.005
-0.004
0.000
-0.005
Overall
-0.010
Low
Middle
High
(0 - 20)
(20 - 80)
(80 - 100)
0.35
0.32
0.30
0.25
ATT on “Definitely will
Vote”
by Turnout Score
0.20
0.17
0.15
0.15
0.10
0.04
0.05
0.00
Overall
Bottom 25%
Middle 50%
Top 25%
(0 - 60)
(60 - 88)
(88 - 100)
Does Barack Obama’s Approval
Improve?
Treatment
Control
Strongly Disapprove (1)
40%
42%
Somewhat Disapprove (2)
10%
10%
Neither Approve nor
Disapprove (3)
6%
6%
Somewhat Approve (4)
18%
18%
Strongly Approve (5)
25%
24%
Mean
2.8
2.7
ITT of Mean
ATT of Mean
0.06
(0.03)
0.16
(0.07)
Does view of Barack Obama’s
Effectiveness Improve?
Treatment
Control
Strongly Disagree (1)
41%
42%
Somewhat Disagree (2)
11%
11%
Neither Agree nor
Disagree (3)
4%
4%
Somewhat Agree (4)
18%
18%
Strongly Agree (5)
26%
25%
Mean
2.8
2.7
ITT of Mean
ATT of Mean
0.06
(0.03)
0.15
(0.07)
0.35
0.30
Effective
Approval
0.29 0.30
0.25
0.20
0.15
0.15
0.16
0.15
0.09
0.10
0.12
0.11
0.05
0.00
Overall
Low Support
Middle Support
High Support
(0 - 20)
(20 - 80)
(80 - 100)
Does it change vote choice?
Higher hurdle to cross
– especially hard to move opinion on a sitting
President.
Could simply move people out of undecided
category.
Look at Vote Obama and Vote Romney
separately, but ultimately care about horse-race.
Did Vote Choice Change?
Condition
Vote Obama
Treatment
39.3%
Control
38.0%
ITT
+1.3pp
(0.7)
Contact
Rate
ATT
38%
+3.5pp
(1.9)
Did Vote Choice Change?
Condition
Vote Obama Vote Romney
Treatment
39.3%
42.5%
Control
38.0%
44.5%
ITT
+1.3pp
(0.7)
-2.0pp
(0.7)
38%
38%
+3.5pp
(1.9)
-5.3pp
(1.9)
Contact
Rate
ATT
Did Vote Choice Change?
Condition
Vote Obama Vote Romney
Two-way
Horse Race
Treatment
39.3%
42.5%
48.0%
Control
38.0%
44.5%
46.0%
ITT
+1.3pp
(0.7)
-2.0pp
(0.7)
+2.0pp
(0.8)
38%
38%
38%
+3.5pp
(1.9)
-5.3pp
(1.9)
+5.3pp
(2.1)
Contact
Rate
ATT
0.100
0.080
0.060
0.040
Obama
Romney
0.082
0.035
0.020
0.038
0.008
0.000
-0.020
Overall
-0.040
Low Support
(0 - 20)
Middle
Support
High Support
(20 - 80)
(80 - 100)
-0.060
-0.053
-0.080
-0.100
-0.062
-0.079
-0.022
0.100
0.080
0.060
0.040
Obama
Romney
0.082
0.035
ATE on Vote Choice
by 2010 Turnout Score
0.040
0.020
0.008
0.000
-0.020
Overall
Bottom 25%
Middle 50%
Top 25%
(0 - 60)
(60 - 88)
(88 - 100)
-0.056
-0.059
-0.040
-0.060
-0.053
-0.080
-0.051
0.080
0.071
0.060
0.040
0.035
0.032
0.032
0.020
0.000
-0.020
Overall
18 - 35
36 - 60
60+
-0.019
-0.040
-0.060
-0.053
-0.080
-0.100
-0.120
Obama
Romney
-0.067
-0.095
ATE on Vote Choice
by Age
How long does the effect last?
• Six weeks after experiment ended
• Sent IVR poll to the 18,557 respondents
• Key question: Dichotomous vote choice
question.
• 2,575 valid survey response
• Response rate 13%
– Extremely high for IVR survey
Is it unbalanced?
Condition
Female
D
Ind.
Age
Black
White
Hispanic
Control
65%
37%
29%
66
6%
92%
2%
Treatment
66%
36%
28%
66
6%
92%
2%
Condition
Support
Turnout
2010
2008
2006
2004
Control
48
76
90%
92%
72%
74%
Treatment
48
76
90%
93%
74%
76%
No, but the IVR sample definitely differs from original.
Evidence of Persistence
Full Sample
Treatment
48.1%
Control
46.0%
Difference
0.020
Contact
ATE
Persistence
38%
0.053
Evidence of Persistence
Full Sample
IVR
Respondents
Treatment
48.1%
49.5%
Control
46.0%
46.7%
Difference
0.020
0.028
38%
46%
0.053
0.061
Contact
ATT
Persistence
Evidence of Persistence
Full Sample
IVR
Respondents
6 Weeks
Treatment
48.1%
49.5%
45.2%
Control
46.0%
46.7%
43.0%
Difference
0.020
0.028
0.022
38%
46%
46%
0.053
0.061
0.048
Contact
ATT
Persistence
Evidence of Persistence
Full Sample
IVR
Respondents
6 Weeks
Treatment
48.1%
49.5%
45.2%
Control
46.0%
46.7%
43.0%
Difference
0.020
0.028
0.022
38%
46%
46%
0.053
0.061
0.048
Contact
ATT
Persistence
79%
Evidence of Persistence
Full Sample
IVR
Respondents
6 Weeks
12 Weeks
Treatment
48.1%
49.5%
45.2%
45.1%
Control
46.0%
46.7%
43.0%
46.1%
Difference
0.020
0.028
0.022
-0.01
38%
46%
46%
49%
0.053
0.061
0.048
-0.020
79%
N.A.
Contact
ATT
Persistence
What did campaign do?
• Increased size of field operation
• Replicated 4 times over remainder of
campaign
• Modeled persuadability
• Used model to target persuadables
– Field
– TV buying
– Mail*
Thank you!