Sampling and mode of interview
Patrick Sturgis, University of Southampton
Opinion Polling in the EU Referendum: Challenges and Lessons
Royal Statistical Society, 8 December 2016
Lessons from the EU ref?
• Online ✔
• Phone ✗
• Random ✗
Quota sampling methodology
• For quota sampling to ‘work’, following condition must be
met:
– Within levels of the quota/weighting variables, sample
should have the same vote intention as same group in
population
• A strong assumption
Final polls vs. Post-election surveys 2015
4
2015 Polling Inquiry Report:
“Our conclusion is that the primary cause of the polling miss
in 2015 was unrepresentative samples. The methods the
pollsters used to collect samples of voters systematically
over-represented Labour supporters and under-represented
Conservative supporters. The statistical adjustment
procedures applied to the raw data did not mitigate this
basic problem to any notable degree.”
Difference in Con lead phone-online
2010-2015
Online vs. phone – EU Ref polls
15
Net error for Remain (%)
10
5
0
-5
-10
-15
14-April
28-April
12-May
Online
26-May
Telephone
9-June
23-June
Mode skews
Missing at Random assumption
Z
X
P
Y
Interest in Politics
English v British identity
Young people today don’t
respect British values
Death penalty most appropriate
sentence for some crimes
Random v Quota
• Inquiry called for funding of a random probability
sample for 2020 election
• Suggested online panel funded by ESRC, as polls
will not move to random due to cost
• Not simply because random = better but because
more transparent, use for calibration of errors
• Our call was answered sooner than anticipated…
Random Prob performance
• 2 random surveys undertaken (in public domain):
– ICM 55/45 (n=875, RR=60%, f-t-f,1-19 June)
– NatCen 53/47 (n=1594, RR=19%, online+phone, 16
May-13 June)
• One random survey not in public domain
Random surveys were wrong?
• Accounting for clustering and weighting gives
95% confidence intervals for ICM survey:
– 48-62% (Remain)
– 38-52% (Leave)
• Using reported vote from post-election wave,
NatCen panel estimates Leave share of 51%
• Third (private) random survey?
Inference from n=1
r(response rate, nonresponse bias)
Groves & Peytcheva (2006)
Conclusions
• Unrepresentative samples likely still a problem
• No silver bullet for this
• Cannot conclude online produces better samples
than phone
• Or that random sampling ‘failed’
• But cost:accuracy ratio strongly favours online for
future election polling