The Polls in 2017
Will Jennings
University of Southampton
[email protected]
@drjennings
Outline
1. Historical trends in campaign polls in Britain
2. Are polling errors getting bigger (everywhere)?
3. What errors, biases and groupthink should we
beware in 2017?
The timeline of elections…
• How do polls lines up with the election result
over the campaign?
• Calculate mean absolute error ( |Vote - Poll| )
of the daily poll-of-polls, for each day of the
election timeline, for all major parties*, across
all elections, from 1945 to 2015.
• Focus on the last 50 days…
* Excludes UKIP, as we only have poll data for 2010-15 election cycle.
The timeline of elections…
6
5
4
3
2
1
0
50
40
30
20
Days Until Election
10
0
Labour Party, 1979-2015
12
10
8
6
4
2
0
-2
-4
-6
50
40
30
20
Days Until Election
10
0
Conservative Party, 1979-2015
10
8
6
4
2
0
-2
-4
-6
-8
50
40
30
20
Days Until Election
10
0
Are polling errors getting bigger?
Analysis of data for 13 countries 1960s to 2010s
• Australia, Canada, France, Germany, Ireland,
Netherlands, New Zealand, Norway, Portugal,
Spain, UK, US, Denmark.
• Mean absolute error of the daily poll-of-polls,
for all parties/candidates for every day of final
week of election campaign.
Are polling errors getting bigger?
All elections
15
10
5
0
1965
1975
1985
1995
2005
2015
Are polling errors getting bigger?
Parliamentary elections
15
10
5
0
1965
1975
1985
1995
2005
2015
Are polling errors getting bigger?
Analysis of 8 national elections in 2015-16:
• Greece, Spain, Denmark, Ireland, Iceland, US,
Canada, Australia.
• Average mean absolute error of all final polls
for the main parties (i.e. including smaller
parties reduces MAE due to sampling error
being a function of the vote share – and even
this comparison is not perfect).
Polling errors worldwide, 2015-16
Mean absolute error on the final polls
8
6
4
2.3 = the average MAE of
polls for ‘large parties’
(>20% vote share) in 139
elections in 23 countries
(from Jennings & Wlezien
2016).
2
0
GB
2015
Denmark* Greece
2015
2015
Canada
2015
Ireland
2016
EU Ref
2016
Spain*
2016
Australia Iceland*
2016
2016
* In multi-party systems where polls for >2 parties overlap, average MAE is for 3 parties.
U.S.
2016
2.7 = the average MAE of
polls in this 2015-16 set
of elections
Anchoring bias? Shaping narratives
Is what really matters getting the story right?
• Polls regularly showed Leave ahead (should really
have been seen as a coin-flip?)
• National polls in 2016 US presidential election
were less wrong than 2012.
• In knife-edge elections, polling error can be small
but get the result wrong. How communicate that?
Expectation bias?
• Ahead of the EU referendum, much discussion
of ‘status quo bias’ of past votes.
• After the 2015 election polling miss, random
probability (‘gold standard’) surveys noted as
getting closer to the final result. Over course of
referendum campaign, random probability
surveys (by NatCen and others) pointed to a
Remain win. Did this influence expectations?
Summary
1. History advises caution.
2. But polling errors aren’t getting worse, so still
are valuable.
3. Extent/variety of methodological adjustment
requires caution
4. Need to beware conventional wisdom, overcorrection, hyping of results and new political
alignments…