Are interviews costing £0.08 a waste of money? Reviewing Google Surveys for Wisdom of the Crowd projects Geoffrey Roughton & Iain MacKay X-MR Limited ASC conference, September 2016 G W Roughton & I M MacKay Interviews costing £0.08… 0 Introduction G W Roughton & I M MacKay Interviews costing £0.08… 1 Table 1: Surveys & Interviews Surveys 30 Row Labels 125 UK G W Roughton & I M MacKay Surveys Interviews UK 125 65514 US 30 15076 Grand Total 155 80590 US Interviews costing £0.08… 2 Table 2: Surveys & Interviews - II Subject Surveys Surveys UK 100 EU Referendum 46 Labour leadership 7 London Mayor UK General election 2015 8 39 US 13 Republican candidate 8 G W Roughton & I M MacKay Wrong 17% 21 Democratic candidate Grand Total How many were right and how many were wrong Right Right 83% Wrong 121 Interviews costing £0.08… 3 Figure 1:Specimen question presentation G W Roughton & I M MacKay Interviews costing £0.08… 4 Table 3: Length of interview (seconds) Subject UK Length in seconds UK General election 2015 19 Labour leadership 18 London Mayor 26 EU Referendum 18 US 22 Republican candidate 24 G W Roughton & I M MacKay 2 2 23 Democratic candidate Grand Total Number of trackers 19 2 Under 20 seconds 20-23 seconds 24+ seconds 20 Interviews costing £0.08… 5 Table 4: Impressions & completes Response rate 64605 Response rate by subject EU REFERENDUM 26% DEMOCRATIC CANDIDATE 18% REPUBLICAN CANDIDATE 18% 371596 UK GENERAL ELECTION 2015 17% LONDON MAYOR Impresions G W Roughton & I M MacKay Completes 10% 0% Interviews costing £0.08… 5% 10% 15% 20% 25% 30% 6 Table 5: Source of interviews Source of impressions (invitations to complete) NEWS 41.3% OTHER 27.1% MOBILE APP 14.7% REFERENCE 7.7% ARTS & ENTERTAINMENT 7.5% MOBILE 1.3% SHOPPING 0.2% 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 35.0% 40.0% 45.0% Source G W Roughton & I M MacKay Interviews costing £0.08… 7 Table 6: Example Google weighting Weight 65+ 55-64 45-54 FEMALE ENGLAND NORTHERN IRELAND 35-44 MALE WALES 25-34 18-24 SCOTLAND 0.00 Scotland 18-24 G W Roughton & I M MacKay 1.66 1.49 1.18 1.12 1.07 1.00 0.94 0.90 0.84 0.81 0.70 0.64 0.20 25-34 Wales 0.40 Male 0.60 35-44 0.80 1.00 Northern Ireland 1.20 England Interviews costing £0.08… Female 1.40 45-54 1.60 55-64 1.80 65+ 8 Table 7: UK Population& Google Population& Standard Region 25.0% 20.0% 15.0% 10.0% 5.0% 0.0% East Midlands East of England London North East North West UK G W Roughton & I M MacKay Northern Ireland Google Scotland South East South West Wales West Midlands Yorkshire and the Humber Referendum Interviews costing £0.08… 9 Wisdom of crowds methodology G W Roughton & I M MacKay Interviews costing £0.08… 10 Why wisdom of crowds? • Scottish referendum 2014 – much less close than polls predicted • Maybe people report other’s behaviour more reliably than their own • Surowiecki’s 2004 book – “The Wisdom of Crowds” G W Roughton & I M MacKay Interviews costing £0.08… 11 Criteria for a wise crowd Criterion Explanation How addressed by sample surveys Diversity of opinion Each person should have private information even if it's just an eccentric interpretation of the known facts. We draw a random sample; people are not selected based on any relationship to the issue in question. Independence People's opinions aren't determined by the opinions of those around them. This is more questionable, especially if we are asking people what they believe is the consensus opinion of others. Decentralization People are able to specialize and draw on We (attempt to) draw a nationally local knowledge. representative sample Aggregation Some mechanism exists for turning private judgments into a collective decision. G W Roughton & I M MacKay Interviews costing £0.08… We arrive at the collective decision by simple aggregation of responses. 12 Studies of UK political events G W Roughton & I M MacKay Interviews costing £0.08… 13 General Election outcome – initial wording G W Roughton & I M MacKay Interviews costing £0.08… 14 General Election outcome – revised wording “Who do you think most likely to be UK Prime Minister after the May 2015 election (whatever your personal preference)?” David Cameron Ed Miliband Nick Clegg Nigel Farage Others (specify) … G W Roughton & I M MacKay Interviews costing £0.08… 15 General election time series G W Roughton & I M MacKay Interviews costing £0.08… 16 Conventional polls showing party prospects Labour Conservative Liberal Democrat UKIP Green 15-day average trend line of poll results from 6 May 2010 to 7 May 2015 (Wikipedia) G W Roughton & I M MacKay Interviews costing £0.08… 17 UK general election 2015 - results Who will be next PM? Party X-MR final % Share of vote Share of seats David Cameron Conservative 48% 37% 51% Ed Miliband Labour 35% 30% 36% Nick Clegg Liberal Democrat 5% 8% 1% Nigel Farage UKIP 10% 13% 0% Other Other Parliamentary 3% 9% 12% G W Roughton & I M MacKay Interviews costing £0.08… 18 Labour Leadership Election 2015 – X-MR polls “Who do you think is most likely to be elected Leader of the Labour Party in the September 2015 election (whatever your personal preference)?” 60% 50% 40% 30% Jeremy Corbyn Andy Burnham 20% Yvette Cooper Liz Kendall 10% 0% July 24, 2015 July 31, 2015 August 7, 2015 August 14, August 21, August 28, September 2015 2015 2015 4, 2015 G W Roughton & I M MacKay Interviews costing £0.08… 19 Labour Leadership Election 2015 (results) X-MR final poll YouGov poll Result 04-Sep 10 Aug 12-Sep Jeremy Corbyn 52% 53% 59% Andy Burnham 20% 21% 19% Yvette Cooper 16% 18% 17% Liz Kendall 12% 8% 5% Candidate G W Roughton & I M MacKay Interviews costing £0.08… 20 London Mayoralty Election 2016 Respondents in England 50.0% 45.0% 40.0% 35.0% 30.0% 25.0% 20.0% 15.0% 10.0% 5.0% 0.0% Zac Goldsmith Respondents in London 60.0% 50.0% 40.0% Sadiq Khan G W Roughton & I M MacKay 30.0% 20.0% 10.0% 0.0% Interviews costing £0.08… 21 London Mayoralty Election 2016 (results) First round result Final X-MR poll (Londoners) Final X-MR poll (all England) Caroline Pidgeon (Lib Dem) 5% 9% 6% 3% 4% Peter Whittle (UKIP) 4% 11% 11% 4% 5% Sadiq Khan (Labour) 44% 38% 39% 35% 48% Siân Berry (Green) 6% 15% 10% 4% 5% Zac Goldsmith (Conservative) 35% 23% 32% 26% 35% Other 8% 4% 4% 2% 3% G W Roughton & I M MacKay Interviews costing £0.08… Final conventional Final poll, adjusted Poll 26/4 -1/5 22 Brexit referendum 2016 – initial wording “In a "UK in/out of the EU" referendum, what do you think most voters in the UK will choose (whatever your own preference)?” 70.0% 60.0% 50.0% 40.0% 30.0% 20.0% 10.0% 0.0% 5/8/2015 6/8/2015 7/8/2015 8/8/2015 9/8/2015 UK remains in the EU G W Roughton & I M MacKay 10/8/2015 UK leaves the EU 11/8/2015 12/8/2015 1/8/2016 2/8/2016 3/8/2016 There won't be a referendum Interviews costing £0.08… 23 Brexit referendum 2016 – final wording “In the referendum on Thursday 23rd June, which answer do you think most voters in the UK will choose (whatever your own preference)?” 70.0% 60.0% 50.0% 40.0% 30.0% 20.0% 10.0% 0.0% Remain a member of the European Union G W Roughton & I M MacKay Leave the European Union Interviews costing £0.08… 24 Brexit - result Outcome X-MR results on 20-June USurv on 21-June Final poll-of-polls Leave 52% 39% 41% 48% Remain 48% 61% 59% 52% G W Roughton & I M MacKay Interviews costing £0.08… 25 Other and ongoing studies • Other o Exploratory studies leading up to the general election o US Presidential primaries • Ongoing o Labour leadership 2016 o US Presidential Election G W Roughton & I M MacKay Interviews costing £0.08… 26 Conclusions • We have demonstrated that £0.08 interviews are not a waste of money • Google are not alone in offering bargain basement prices • Large scale general population survey with relatively low data collection costs are with us now • Setting these up, analysing & interpreting them are an opportunity for market researchers. • We have a solution. All we need now is to find the problems it helps solve. G W Roughton & I M MacKay Interviews costing £0.08… 27
© Copyright 2024 Paperzz
