Administering Surveys
GV917
Theory and Methods



There are both theoretical and methodological
considerations to examine in the administration of
surveys.
Theoretical considerations relate to what it is that we
are trying to explain and the models we use to do
this. The models have to be operationalised
properly by questions used in the surveys
The methodological considerations relate to the
mode of administering the survey, what sampling is
used, the coding frames used in answers and any
scaling done.
Theory – Explaining Turnout
Some Theoretical Models of Turnout

Cognitive Engagement

General Incentives

Valence
Cognitive Engagement Theory
Electoral Participation is motivated by:





The willingness of individual’s to acquire political
information
Their ability to process such information
Their interest in the political process and their
comprehension of issues in the election
If they are ‘critical citizens’ and can judge the
effectiveness of policy delivery
Dalton (2000); Lupia and McCubbins (1998);
Norris(2000)
General Incentives Theory
Electoral Participation is motivated by:








Perceptions of the collective benefits of participation
Feelings that the individual can influence the political
process
Perceptions of the Costs of participation
Perceptions of the private benefits – selective
incentives
Perceptions of Group benefits
Altruistic motives
Social norms
(Seyd and Whiteley, 1992, 2002); Whiteley and Seyd
(2002)
Valence Theory
Electoral participation is motivated by:





The government delivering on the policies which matter
to voters
In particular the existence of a party that can deliver on
the economy
Leaders doing a good job at governing
Democracy working well
(Stokes (1963); Clarke, Sanders, Stewart and Whiteley,
2004)
An Example of Studying Turnout with
Cross-National Data and the Valence Model




The European Social Survey for 2002
A total of 37,497 respondents when the data are
weighted by population and design weights
A total of 22 countries – 21 in Europe plus Israel
A variety of measures of participation in addition to
voting turnout at the individual level
European Social Survey Voting Question







B13 Vote
Some people don’t vote nowadays for one
reason or another.
Did you vote in the last [country] national
election in [month/year]?
Yes
No
Not eligible to vote
(Don’t know)
1 (ASK B14)
2
3 (GO TO B15)
4
Self-Reported Turnout in the Previous National Election
European Social Survey Data 2002 (N=37,497 weighted)
100
90
Percentage Reported Turnout
80
70
60
50
40
30
20
10
0
AT BE Ch CZ DE DK ES FI FR GB GR HU IE IL
IT LU NL NO PL PT SE SL
The Relationship between Actual Turnout and Aggregated Self-Reported
Turnout in the European Social Survey, 2002
(Sources: European Social Survey, 2002 and IDEA)
Operationalising Valence Theory
- Signature variables




Approval of Government’s performance
Evaluations of politicians and party
leaders
Evaluations of the State of the Economy
Satisfaction with democracy
Approval of Government’s Performance

B31 StfGov

Now thinking about the [country] government,
how satisfied are you with the way it is doing
its job?

Use this card:

00 01 02 03 04 05 06 07 08 09 10 88

Extremely Dissatisfied
Extremely Satisfied
DK
Satisfaction with National Government in
the 2002 ESS
stfgov How satisfied with the national government
Valid
Mis sing
Total
0 Extrem ely diss atis fied
1 1
2 2
3 3
4 4
5 5
6 6
7 7
8 8
9 9
10 Extrem ely s atis fied
Total
77 Refusal
88 Don't know
99 No ans wer
Total
Frequency
3573
2271
4079
5445
4923
6735
4114
2920
1591
374
304
36329
30
1116
23
1169
37497
Percent
9.5
6.1
10.9
14.5
13.1
18.0
11.0
7.8
4.2
1.0
.8
96.9
.1
3.0
.1
3.1
100.0
Valid Percent
9.8
6.3
11.2
15.0
13.6
18.5
11.3
8.0
4.4
1.0
.8
100.0
Cum ulative
Percent
9.8
16.1
27.3
42.3
55.9
74.4
85.7
93.8
98.1
99.2
100.0
Trust in Politicians



















Question B 4 -10
Using this card, please tell me on a score of 0-10 how much you personally trust each of the institutions I read out.
0 means you do not trust an
institution at all, and 10 means you have complete trust. Firstly...
Instruction(s): Pre: CARD 8: Post: READ OUT...
Values and categories
00 No trust at all
01 1
02 2
03 3
04 4
05 5
06 6
07 7
08 8
09 9
10 Complete trust
77 Refusal
88 Don't know
99 No answer
Trust in Politicians
trstplt Trust in politicians
Valid
Missing
Total
0 No trust at all
1 1
2 2
3 3
4 4
5 5
6 6
7 7
8 8
9 9
10 Complete trust
Total
77 Refusal
88 Don't know
99 No ans wer
Total
Frequency
4429
2763
4671
5801
5310
8441
4496
3364
1557
385
258
41477
16
771
94
881
42358
Percent
10.5
6.5
11.0
13.7
12.5
19.9
10.6
7.9
3.7
.9
.6
97.9
.0
1.8
.2
2.1
100.0
Valid Percent
10.7
6.7
11.3
14.0
12.8
20.4
10.8
8.1
3.8
.9
.6
100.0
Cumulative
Percent
10.7
17.3
28.6
42.6
55.4
75.7
86.6
94.7
98.4
99.4
100.0
Satisfaction with the Economy



















Question B 25
On the whole how satisfied are you with the present state of the economy in [country]?
Instruction(s): Pre: STILL CARD 10: Post: Still use this card.
Variable name and label: STFECO How satisfied with present state of economy in country
Values and categories
00 Extremely dissatisfied
01 1
02 2
03 3
04 4
05 5
06 6
07 7
08 8
09 9
10 Extremely satisfied
77 Refusal
88 Don't know
99 No answer
Satisfaction with the Economy
stfeco How satisfied with present state of economy in country
Valid
Mis sing
Total
0 Extremel y dis satis fied
1 1
2 2
3 3
4 4
5 5
6 6
7 7
8 8
9 9
10 Extremely satisfied
Total
77 Refusal
88 Don't know
99 No answer
Total
Frequency
3053
1969
3827
5437
5213
6967
5101
4865
3175
966
579
41152
16
1120
70
1206
42358
Percent
7.2
4.6
9.0
12.8
12.3
16.4
12.0
11.5
7.5
2.3
1.4
97.2
.0
2.6
.2
2.8
100.0
Valid Percent
7.4
4.8
9.3
13.2
12.7
16.9
12.4
11.8
7.7
2.3
1.4
100.0
Cumul ative
Percent
7.4
12.2
21.5
34.7
47.4
64.3
76.7
88.5
96.2
98.6
100.0
Satisfaction with Democracy

B32 StfDem
And on the whole, how satisfied are you with
the way democracy works in [country]?

00 01 02 03 04 05 06 07 08 09 10 88

Extremely Dissatisfied
Extremely Satisfied
DK
Satisfaction with Democracy in the 2002
ESS
stfdem How satisfied with the way democracy works in country
Valid
Mis sing
Total
0 Extrem ely diss atis fied
1 1
2 2
3 3
4 4
5 5
6 6
7 7
8 8
9 9
10 Extrem ely s atis fied
Total
77 Refusal
88 Don't know
99 No ans wer
Total
Frequency
1758
1079
2306
3809
4166
7010
4739
5106
4015
1268
745
36002
96
1379
21
1495
37497
Percent
4.7
2.9
6.1
10.2
11.1
18.7
12.6
13.6
10.7
3.4
2.0
96.0
.3
3.7
.1
4.0
100.0
Valid Percent
4.9
3.0
6.4
10.6
11.6
19.5
13.2
14.2
11.2
3.5
2.1
100.0
Cum ulative
Percent
4.9
7.9
14.3
24.9
36.4
55.9
69.1
83.3
94.4
97.9
100.0
Conclusions about Theory


Theories need to be translated into models,
so that they specify which variables have an
influence on which other variables
Adequate indicators of the different models
need to be present in data sets if they are to
be tested properly
Methodological Issues




The mode? – face-to-face interviewing, mail
questionnaires, telephone polls or internet
surveys.
Samples? – Random samples, quota
samples, or other types of samples eg
snowball samples.
Design? – Descriptive, analytical or
experimental.
Scaling? – Open ended or closed ended
responses, types of scales.
Face to Face Probability Surveys The Gold Standard?






Probability samples are a kind of Gold Standard since they
allow us to infer the characteristics of the populations from
which the samples are taken with the help of statistical theory
But probability Samples Yield Lists (i.e. the sampling frame)
Not Achieved Samples
Non-Response is a growing problem in all probability surveys
(55 percent response rate for 2015 BES)
respondents are unlikely to be a random subset of the list – ‘topups’ do not help
They are very expensive (The American National Election Study
in 2004 cost $5500 per voter interviewed)
They are slow moving – two to three months to get the data
Are Telephone Surveys an alternative?








Probability based surveys based on Random Digit Dialling
(RDD) can be conducted by telephone
They are less expensive than face-to-face surveys
But increasingly they have horrific response rates (often below
10 per cent)
The growth of mobile phones, telephone preference system and
telemarketing has made sampling problematic
Complex questions and/or routing protocols are very difficult to
administer by telephone
You can’t keep someone on the phone for longer than 20
minutes (if that).
Experimental designs are difficult to execute by phone
If one wants a large N they take time – several weeks
Is the Internet an Alternative?






Fast responses to internet surveys – typically 95 per cent of
responses are in within a couple of days. This is ideal for
campaign studies focusing on short-term dynamics
They are very cost effective – in the BES significant savings are
obtained in terms of the costs per interview. They are currently
cheaper than telephone surveys
Large N’s are possible – the marginal cost of adding a respondent
is small
Experiments are possible with large treatment groups
No interviewer effects – which can be a problem when interviewers
are socially homogenous
BUT – they are not random samples. In the BES they are a type of
quota sample – which means that inferring from the sample to the
underlying population is problematic
BES 2005 CORE FACE-TO-FACE PANEL SURVEY:
Wave 1 Pre-election
Probability Sample,
Face-to-Face N=3589
128 Primary Sampling Units
Wave 2 Post-election
Probability Sample,
Face-to-Face N=4161
Including top-up,
mail-back; 128 Primary
Sampling Units
BES 2005 INTERNET CAMPAIGN PANEL SURVEY:
Wave 1
Pre-campaign
Baseline Survey
N=7793
Wave 2
Campaign survey
275 interviews per
day for 30 days
N=6068
Wave 3
Post-election
Interview
N=5910
Wave 4
One Year Out
Interview
N=6186
Waves 5, 6, 7
Annual interviews
through to 2010
Conclusions



The choice of mode depends on the budget
available, the time frame over which the survey is
conducted, and the importance of strict statistical
inference
There is no uniquely ‘best’ mode. Thus in campaign
studies which require a quick turnaround of data
collection face-to-face probability surveys would be
useless
In a national survey of political values with a focus
on looking at regional variations a probability survey
would be best.