Treatment of don’t knows
and turnout weighting
Stephen Fisher, University of Oxford
Slides for presentation at the BPC/NCRM event on Opinion Polling in the EU Referendum,
at the Royal Statistical Society, 7th December 2016
Referendums Prior to the EU ref
• The change option more often declines in
popularity during the campaign
• Final polls tend to overestimate change
– The more so the more Don’t Knows (DK) there
are, perhaps because they split disproportionately
for the status-quo
• So, potential to improve polling accuracy by
estimating how the undecideds will split
– Typically with “squeeze” questions
Referendum
AUT13 Conscription
AUS99 Republic
MLT15 Spring hunting
AUS99 Constit. preamble
FRA05 EU Constitution
NOR94 EU accession
DNK00 Euro
ITA16 Constitution
DNK15 EU opt out
GRC15 Bailout proposals
GBR16 Brexit
IRL08 EU Lisbon
SVN05 State broadcasting
SVN10 Croatian Border
SWE94 EU accession
MLT11 Divorce
FRA92 EU Maastricht
DNK92 EU Maastricht
HUN04 Dual citizenship
PRT98 Abortion
IRL02 Abortion
IRL95 Divorce
Poll average for change
(%)
45.2
45.5
45.7
46.1
46.7
46.9
47.2
47.3
48
48
48.4
49
49.8
51.2
51.9
52.1
52.3
52.4
52.8
53
53
53.2
Share for change
(%)
40.3
45.1
49.6
39.3
45.3
47.8
46.8
40.9
46.9
38.7
51.9
46.6
50.7
51.5
52.7
53.2
51
49.3
51.6
49.1
49.6
50.3
Difference
-4.9
-0.3
3.9
-6.8
-1.3
0.9
-0.4
-6.4
-1.1
-9.3
3.5
-2.4
0.9
0.4
0.9
1
-1.2
-3.1
-1.2
-3.9
-3.5
-2.9
Passed
Change >
50%
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Online = red, Telephone = blue
Undecideds in the Final polls
• 6% to 15%, lower than at GE15
– As in GE DK & Refuse higher in phone higher than online
• % of DKs reporting 10/10 Likelihood of Voting some
20 points lower than for Remain and Leave intenders
in the same survey
• Squeeze questions split in favour of Remain, on
average 32% to 25%
• Squeeze or reallocation on other questions typically
negligible effect on headline figures
– At most 1 point to Remain
Did the Undecideds really split in
favour of Remain?
• Mixed evidence from YouGov recontact
– Yes on the day; No with later recontacts
• Yes according to:
– TNS:
• DKs split 31% Remain, 18% Leave, 51% Abstain.
– British Election Study internet panel
• DKs split 46% Remain, 39% Leave, 15% Abstain.
• So, DK squeeze made headline figures slightly
worse for the right reasons
Likelihood of Voting
• Leave intenders reported being more likely to
vote
– Only by 2-5 points, and 2 outliers
• But, high general election turnout
demographics more Remain
• What to weight by?
– Likelihood of voting questions (LTV) only?
– GE15 demographics only?
– A mix? Or Nothing?
Impact of turnout weighting
• Likelihood of voting questions (LTV) only
– Increase Leave share by up to 1 point
• GE15 demographics only
– Increase Remain share by 1.5 points
• Mixes
– Effects in different directions depending on
dominance of LTV question
• Assumptions affected implied profile of voters
– More C2DE, more Leave (Luke Tayor)
How good were the LTV questions?
• Typically suggested turnout c. 90%
– Recontact reported turnout higher still
– Still suggests samples over engaged even allowing
for over reporting
• LTV highly correlated with who votes
– TNS, Opinium and BES recontact surveys
– Likelihood of voting and GE15 turnout good prior
predictors, but thereafter no sociodemographics
• No turnout validation study though
Were Leave intenders really more
likely to vote?
• No difference in Opinium and YouGov
recontact surveys
• Yes according to:
– TNS recontact
– British Election Study internet panel study
• No signs of differential turnout misreporting
– NatCen mixed-mode random probability panel
Whether voted in EU referendum by
how intended to vote
Vote intention pre-referendum
Remain
Leave
Total
(%)
(%)
(%)
81
89
84
Voted
Whether voted in
Did not
EU referendum
19
11
16
vote
100
100
100
Total
Vote intention pre-referendum
in population (%)
Unweighted bases
53
47
100
652
573
1,274
Base: all adults aged 18+ NatCen Panel May and September 2016 survey. Source: Swales 2016
Whether voted in EU referendum by whether voted in 2015 GE
Whether voted in 2015 General Election
Voted Did not vote
Whether voted in
EU referendum
Total
Voted
Did not vote
Whether voted in 2015 General
Election in population (%)
Unweighted bases
Total
(%)
94
6
100
(%)
54
46
100
74
26
100
1,138
252
1,391
Base: all adults aged 18+ NatCen Panel September 2016 survey. Source: Swales 2016
• Also, those who voted in the referendum but not GE15 split 60:40 in
favour of Leave.
• So turnout weighting that emphasised 2015 turnout demographics
made things worse.
(%)
83
17
100
Lessons for future polls
• Not the main thing but assumptions about Don’t
Knows and Turnout matter
– With LTV (dominated) turnout weighting polls, on average
would have had Leave a point higher
– But should have been more
• Referendums different from general elections
– But squeeze and likelihood to vote questions help for both
• Reconsider sampling and how well baseline
weighting captures less interested electors