With the global leader in sampling and data services The True Meaning of Represen6vity Pete Cape Global Knowledge Director © 2014 Survey Sampling Interna6onal | 1 | Usually more helpful than this… © 2014 Survey Sampling Interna6onal | 2 | Usually more helpful than this… “in the state of being representa4ve” © 2014 Survey Sampling Interna6onal | 3 | BeEer •  In a sample each person “represents” all the other people who are just like him, but were not selected for the survey •  If everyone is represented, in their correct propor6ons, then we will have the truth •  How do we know we chose a sample that has all the different types of people we need to get to the truth? •  And in the correct propor6ons? © 2014 Survey Sampling Interna6onal | 4 | Because we chose them at random Sampling 101 – why sampling gets you to the truth Consider 10 coin tosses © 2014 Survey Sampling Interna6onal | 5 | Because we chose them at random How many ways can 10 heads occur? Just 1 How many ways can 9 heads occur? 10 ways So which is more likely? Which is most likely? 1, 2, 3, 4, 5, 6, 7, 8, 9 or 10? © 2014 Survey Sampling Interna6onal | 6 | Distribu6on of possible outcomes n=10 Centres on the true probability; 5/10 © 2014 Survey Sampling Interna6onal Sampling facts •  If you take a sample you are going to get an answer •  If you sample at random you are most likely to get close to the truth •  The truth for your popula6on…… © 2014 Survey Sampling Interna6onal | 8 | The Population
The Sample Frame
The Samples
© 2014 Survey Sampling Interna6onal Sampling facts •  If you take a sample you are going to get an answer •  If you sample at random from a list that is close to complete for the popula6on… •  you are most likely to get close to the truth •  QED BUT © 2014 Survey Sampling Interna6onal | 10 | Small problems with random samples •  Unlike in quality control •  Samples can refuse —  And they do so at differen6al rates •  We do not have access to all the possible samples —  And this may not be an “ignorable sample design” •  And we may not be competent to assess the item —  We ask it the wrong ques6on •  Which s6ll leaves us with an answer….. AND © 2014 Survey Sampling Interna6onal | 11 | We don’t do random samples… QUOTA SAMPLES
© 2014 Survey Sampling Interna6onal | 12 | Not this sort of quota sampling! •  Interview one 65+ Male plus any four Men 25-­‐34 © 2014 Survey Sampling Interna6onal | 13 | Quota sampling in the telephone world Because we were in a hurry And the last was the same as the first (within reason!) We just set quotas on random samples And went for it… © 2014 Survey Sampling Interna6onal | 14 | Stra6fying a random sample 10 26 33 22 14 14 1 © 2014 Survey Sampling Interna6onal Look at the sample any way you like… 10 © 2014 Survey Sampling Interna6onal 27 8 18 23 10 23 Quota sampling/Quasi random sampling •  Is a means of achieving a Stra6fied Random Sample •  Without having to worry about differen6al non-­‐response •  Or weigh6ng the data •  In a random world you could use any variable you like © 2014 Survey Sampling Interna6onal | 17 | But… •  The frames are s6ll universal •  And the sampling methods random •  What happens in a convenience sampling world? © 2014 Survey Sampling Interna6onal | 18 | Stra6fying a convenience sample 10 26 33 22 14 14 1 © 2014 Survey Sampling Interna6onal Now look at the sample this way… 10 7 © 2014 Survey Sampling Interna6onal 27 35 8 4 18 14 23 25 10 6 23 26 Quota sampling in a convenience world •  Only guarantees the variables you quota on will look like the popula6on •  Underneath you are missing whatever the frame is missing © 2014 Survey Sampling Interna6onal | 21 | Other unpleasant side effects •  Adding in more and more of a biased sub-­‐set of people might be making the point es6mate worse, not beEer •  And we do this all the 6me in online — 
Over 65’s © 2014 Survey Sampling Interna6onal | 22 | Other unpleasant side effects •  How representa6ve of all over 65s are these people? © 2014 Survey Sampling Interna6onal | 23 | Let’s not get too frightened •  Most of the 6me it doesn’t maEer •  Although there is no way to know •  So what harm can it do? •  Let’s experiment! © 2014 Survey Sampling Interna6onal | 24 | Hold on! What do you mean it doesn’t maEer? •  Think…. •  If men and women feel the same does it maEer how many men and how many women there are in the sample? •  If the young and the old feel the same, what does it maEer the balance between them? •  You need to balance the things that do maEer…. •  So let’s experiment! © 2014 Survey Sampling Interna6onal | 25 | Which sample is most representa6ve? Sample A
Census
Sample B
Sample C
© 2014 Survey Sampling Interna6onal Who chose… •  Sample A? •  Sample B? •  Sample C? •  Who said...... •  It depends on what you want to measure? © 2014 Survey Sampling Interna6onal Es6ma6ng eye colour distribu6on © 2014 Survey Sampling Interna6onal What quotas would you set on your sample? • 
• 
• 
• 
• 
Age? Gender? Region? Income? Ethnicity? •  None? © 2014 Survey Sampling Interna6onal Different quota control Age Sample
Gender ARegion
“Nation”
Ethnicity
Sample/ Heritage
B
Sample
The Crowd
C
© 2014 Survey Sampling Interna6onal © 2014 Survey Sampling Interna6onal © 2014 Survey Sampling Interna6onal How important is eye colour in your choice of partner? © 2014 Survey Sampling Interna6onal © 2014 Survey Sampling Interna6onal Age Gender Region Average 5’7” height Median height 5’7” Ethnicity The Crowd 5’6” 5’6” 5’7” 5’7” the average American is 5’6”
© 2014 Survey Sampling Interna6onal © 2014 Survey Sampling Interna6onal Extremely likely to buy 18oz box of StartRight cereal
go
no go
© 2014 Survey Sampling Interna6onal New defini6on of represen6vity •  In a sample each person “represents” all the other people who are just like him, in terms of the things that ma.er to the outcome of the survey, but were not selected for it •  If everyone is represented, in their correct propor6ons, then we will have the truth Is this really new? © 2014 Survey Sampling Interna6onal | 38 | What’s the textbook say Pete? •  On choosing a stra6fica6on variable: Worcester & Downham (ESOMAR 1986): “The object is to divide the total popula<on into strata that differ markedly in respect of the characteris<cs measured in the survey” Zikmund (an old book on my shelf…): “The criteria for a stra<fica<on variable is that it be a characteris<c of the popula<on elements known to be related to the dependent variable or other variables of interest” © 2014 Survey Sampling Interna6onal What’s the textbook say Pete? •  On choosing a stra6fica6on variable: Chisnall (1991): “Pilot surveys can help iden<fy characteris<cs relevant to a survey” © 2014 Survey Sampling Interna6onal Should we give up? •  No, just be aware •  Quotas on Age, Gender and Region are not going to guarantee represen6vity Only that the sample looks like the popula6on —  One Age, Gender and Region! — 
•  For trackers: Look for the key drivers and ensure consistency of this over 6me •  For ad hoc: Think about what might drive key variables Add that ques6on to your survey —  You might be able to weight on it — 
© 2014 Survey Sampling Interna6onal “Nat Rep” •  For a sample to represent a na6on •  All elements must be able to be chosen •  Unless it doesn’t maEer Because there is no difference in opinion —  Or the numbers concerned are too small —  Or their opinions do not maEer! — 
•  In the West the marke6ng “na6on” is essen6ally everyone •  In emerging markets it may not be so… •  Design your sample to fit your target market © 2014 Survey Sampling Interna6onal For more informa6on Email us at [email protected]: If you would like to request addi6onal informa6on, please put webinar in the subject line and request one or all of the following in the body of your email: •  Today’s presenta6on •  POV on Nat Rep Please provide your name, 6tle, company © 2014 Survey Sampling Interna6onal Q & A © 2014 Survey Sampling Interna6onal To con6nue the conversa6on Visit our Facebook page: facebook.com/surveysampling or Tweet #SSIWebinar © 2014 Survey Sampling Interna6onal surveysampling.com [email protected] Pete [email protected] © 2014 Survey Sampling Interna6onal | 46 |