European Social Survey
THE EUROPEAN
SOCIAL SURVEY
INFRASTRUCTURE:
PAST, PRESENT AND
FUTURE
3rd International
ESS Conference
Lausanne
15 July 2016
Dr. Rory Fitzgerald
Director, ESS ERIC
City University London
ESS is a European Research Infrastructure Consortium (ESS ERIC)
European Social Survey infrastructure
Today’s session:
Looking back: the past
Where we currently are: the present
Looking forward: the future
2
THE PAST
3
European Social Survey: the past
Why we needed ESS
“There is a wealth of data on individuals and households that are regularly collected
by statistical offices and other administrative agencies. These data, however, are
frequently not comparable across nations and are often not accessible to researchers
for reasons of administrative self-interest or data protection. Absolutely crucially,
however, they do not deal with the whole range of individual orientations (attitudes,
beliefs and behaviours) that are central in understanding modern societies, and that
can be assessed by survey research.
“Very many individual surveys have been and will be conducted in particular European
countries. Inevitably, however, these are tailored to individual projects and interests.
The social sciences, if they are to make progress, require regular cross-national surveys
that are conceptually well anchored, conducted according to rigorous methodological
standards and are available at little cost to the entire research and policy
community.”
ESF Blueprint, March 1998
4
European Social Survey: the past
Key milestones for ESS:
ESS commission funding (2001)
22 countries took part in Round 1 (2002)
36 European countries involved (2002-16)
Received Descartes Prize (2005)
Infrastructure funding (from 2006)
Groves Review (2008)
ESFRI Roadmap
ERIC (2013)
ESFRI landmark (March 2016)
“The panel unanimously finds that the
importance of ESS, its demonstrated success
in initial launch, and its clear signals of
impact justify fully continuous funding at
levels necessary to achieve its vision and
maintain its quality.”
Groves Review, 2008
5
THE PRESENT
6
European Social Survey: the present
Use of ESS infrastructure
Registered ESS Users (January 2005 - January 2016)
100000
90000
80000
70000
60000
50000
40000
30000
20000
10000
0
janv..05
7
janv..06
janv..07
janv..08
janv..09
janv..10
janv..11
janv..12
janv..13
janv..14
janv..15
janv..16
European Social Survey: the present
Data users and data sessions per month
5000
4500
4000
3500
3000
2500
Data users
2000
Data sessions
1500
1000
500
0
Dec 2015
Dec 2014
Dec 2013
Dec 2012
Dec 2011
Dec 2010
Dec 2009
Dec 2008
Dec 2007
Dec 2006
Dec 2005
Dec 2004
Dec 2003
8
European Social Survey: the present
Data users and data sessions per year
40000
35000
30000
25000
Data users
20000
Data sessions
15000
10000
5000
0
2003
9
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
European Social Survey: the present
Data downloaders per year
14000
12000
10000
8000
Downloaders
6000
4000
2000
0
2003
10
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
Trends in ESS academic publishing
(Google Scholar, 2003-2014, N=2705)
450
400
350
300
416
2705 publications overall:
▪ 1281 journal articles
▪ 472 books & chapters
▪ 470 working papers
▪ 330 conference papers
375
379
2012
2013
313
292
263
250
193
210
200
133
150
89
100
31
50
11
0
2003
2004
Overall
2005
2006
Journal articles
2007
2008
2009
Books & chapters
2010
2011
2014
Conference & working papers
Journals with most ESS publications
(2003-2014, N=1281)
Sociology
35.1
Political science
19.2
Economy
11.5
▪ Social Indicators Research
Methods, Statistics
60
▪ European Sociological Review
8.1
55
▪ European Societies
32
▪ Journal of European Social Policy
Health, Medicine
6.3
▪ International Journal of Public Opinion Research
▪ Survey Research Methods
Psyshology
19
17
▪ International Journal of Sociology
5
18
14
▪ International Journal of Comparative Sociology
Media, Public opinion
▪ Comparative Political Studies
2.9
14
▪ West European Politics
Other
11.9
0
5
10
15
14
▪ Journal of Cross-Cultural Psychology
20
25
30
14
35
40
13
45
50
Topics addressed in ESS publications
(2003-2014, N=2705)
Politics, democracy
Immigration
Public policies, welfare
Work
SWB-QOL
Family
Social capital
Economy
Inequalities
Culture, values
Age groups
Citizenship
Gender
Health
Education
Religion
Crime
Media-ICT
Nation
Europe
Environment
Transition
Methods
600
338
289
282
234
233
227
222
209
185
159
139
139
136
120
116
114
111
95
Political participation
199
Political trust
102
Political parties
83
Political protest
73
Political attitudes, culture
71
Democratic system
67
0
100
47
33
33
405
0
100
200
300
400
500
600
700
200
300
Distribution of 1st author's affiliation across countries
(2003-2014, N=2705)
363
United Kingdom
Germany
USA, Canada
Netherlands
Belgium
Sweden
Spain
Italy
Norway
Switzerland
France
Finland
Denmark
Poland
Portugal
Ireland
Estonia
Israel
Greece
Austria
Hungary
Romania
319
272
245
164
150
137
106
72
71
64
63
61
60
56
55
52
38
37
36
32
31
0
100
Europe-West
Europe- North
Europe-South
Europe-East
North America
10.5
8.9
49.2
18
13.4
200
300
400
% share of publications where a country
was included (1)
(sorted according to the number of rounds fielded, 2003-2014, N=1520)
Germany
United Kingdom
Sweden
Denmark
Spain
Belgium
Netherlands
Finland
France
Portugal
Norway
Ireland
Poland
Slovenia
Hungary
Switzerland
73.7
73.5
72.4
72.3
71.4
70.9
70.5
68.9
66.6
65.6
63.9
60.3
60
56.1
55.8
55.7
Austria
Czech R
Estonia
Slovakia
7 rounds
52.9
6 rounds
48
39.7
38.1
0
20
40
60
80
100
Trend of multiple round usage
% share of publications where more that one ESS round was used, (2003-2014,
N=1580)
100
90
80
70
60
49.8
50
42.4
40
30
25.4
26.1
2008
2009
2010
33.7
2011
2012
15
20
10
23
34.2
6.3
0
0
<2005
2006
2007
more than 1 round
3 rounds
4 rounds
2013
5 rounds
2014
European Social Survey: the present
Non-academic impact of the ESS
Setting new standards in cross-national research
Open data
Methods adopted
Policymakers and Government
Press coverage
18
European Social Survey: the present
Non-academic impact of the ESS
19
HAS PRESENT MET
CHALLENGES OF PAST
20
MEASUREMENT EQUIVALENCE
ACHIEVED?
In each round two complex concepts in the ESS are tested for
measurement equivalence, i.e. whether the same construct is being
measured across countries or linguistic groups.
Testing in the ESS is done in three steps:
Configural invariance: testing evaluates whether a theoretical
model holds cross-nationally, i.e. whether the factor structure is
the same.
Metric invariance: testing restricts the factor loadings to be equal
across groups. Its interpretation is that respondents across
groups attribute the same meaning to the latent concept under
study.
→Only if metric invariance is established the relationship with
other variables can be studied.
Scalar invariance: testing restricts (in addition) the intercepts to
be equal across groups.
→ If established groups’ latent means can be compared.
Scalar invariance - Allowing for comparison
of relationships and means across groups in
each Round
Concept tested
Number
of
indicator
s
Across countries / groups (combination of
language and country)
Interest in Political Issues
presented in the Media
3
29/30 countries (R1-R4)
Depression
8
6/27 countries (R3 and R6)
Trust in police procedural
fairness
3
18/27 countries (R5)
Political satisfaction
2
10/25 (R1- R5)
Quality of state services
2
7/25 (R1 – R5)
Trust in political institutions
2
21/33 groups (R2-R5)
Trust in political authorities
2
33/33 groups (R2-R5)
Internal political efficacy
3
14/18 groups (R7 first data release)
External political efficacy
3
15/ 18 groups (R7 first data release)
Scalar invariance - Allowing for comparison
of relationships and means across rounds
within each group
Concept tested
Number
of
indicator
s
Within country longitudinal results
Interest in Political Issues
presented in the Media
3
29/30 countries (R1-R4)
Depression
8
22/22 countries (R3 and R6)
Political satisfaction
2
For most countries that participated in 4 or 5 rounds,
within-country scalar equivalence established over 45 rounds.
Quality of state services
2
For most countries that participated in 4 or 5 rounds,
within-country scalar equivalence established over 45 rounds.
Equivalence achieved?
• Scalar invariance test is a strict test requiring loadings and intercepts to
be equal across groups, even though it could be established for most of
the groups tested.
• Sometimes substantive researchers are more interested in relationships
with other variables than in the comparison of latent means. For that metric
invariance is a sufficient requirement. ESS concepts exhibit high metric
invariance.
• When invariance is established the measurement of a concept is not
dependent on group membership, revealing differences across countries
as substantive.
• Invariance testing provides valuable information for analysts, for instance:
Common understanding of political affairs across Europe:
-
Concepts asked in the ESS about political attitudes exhibit high
measurement equivalence.
-
‘Interest in political issues presented in the media’ was scalar invariant
across 4 rounds tested and 29 out of 30 countries
-
‘Trust in political authorities’ was scalar invariant in 4 rounds tested and
33 out of 33 groups
Interpretation of the ‘depression scale’ is not the same across European
countries although it is consistent over time within countries.
THE FUTURE
25
European Social Survey: the future
ANALYSIS CHALLENGES
Weighting
Not always applied (not clear in journals)
Most common user query
Complex sampling designs not accounted for
Use of a harmonized methodology for post stratification weights needs to
be evaluated
Interviewer effects not accounted for
Differences in question quality rarely accounted for
Equivalence testing not always performed
The curse of the ESS total column in tables
More on-line training and guidance required in future
Combining data across rounds to increase sample size of sub groups –
issues?
26
European Social Survey: the future
Reasons for rejecting mixed mode interviewing
Synthesis report (Villar and Fitzgerald, 2016)
ESS has always used face-to-face interviews
Six studies into measurement, representativeness and cost efficiency
Same sampling frame as the standard ESS
Response rates remained lower in all mixed-mode designs compared
to face-to-face design
Telephone interviews had poor response rates
Mixed-mode online performed better – results still disappointing
Nonresponse bias may be increased by moving to mixed-mode
Mode effects would be introduced but no easy way to correct for
these
Estonia – hope for future?
27
www.seriss.eu
@SERISS_EU
A collaboration between:
•
European Social Survey (ESS ERIC)
•
Survey of Health, Ageing and
Retirement in Europe (SHARE ERIC)
•
Consortium of European Social
Science Data Archives (CESSDA AS)
•
Generations and Gender Programme
(GGP)
•
European Values Study
•
WageIndicator Survey
This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement No 654221.
www.seriss.eu
@SERISS_EU
Designed to:
•
Strengthen social science research across Europe and beyond by
overcoming fragmentation and fostering interoperability,
harmonisation and innovation
•
Equip social science infrastructures to play a major role in addressing
key societal challenges and ensure that national and European policy
making is built on a firm socio-economic evidence base
•
Promote the value of the social sciences to the wider research
community, and policymakers
This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement No 654221.
www.seriss.eu
@SERISS_EU
Achieved by:
•
Addressing key challenges for cross-national data collection e.g.
accurately representing the population, achieving equivalence through
translation
•
Breaking down barriers between social science infrastructures
via training and networking events and the development of shared
online tools to facilitate harmonised data collection and
documentation.
•
Embracing the future of the social sciences by examining the legal
and ethical challenges associated with new forms of data, developing
a cross-national probability-based web survey and exploring
automated coding for socio-economic variables
This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement No 654221.
www.seriss.eu
@SERISS_EU
This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement No 654221.
European Social Survey: the future
Methodology: CRONOS web panel
Assessing viability of obtaining accurate data online
CRONOS will use ESS to recruit respondents from three countries
Recruits will take part in 12-month web panel
Respondents will be asked to complete a 20-minute web survey every
other month at any point during a two-month period
Surveys will be programmed and administered using Questback-EFS
Questions will be taken from European Value Study, ESS and European
Quality of Life Survey
Respondents will be given a small token of appreciation
Some data will be freely accessible
If positive outcome, could be adopted by ESS in future
32
Commitment to the ESS
16 countries ESS ERIC
36 Countries have taken
part in at least one round
of the survey:
27 EU states (not Malta)
+ 9 others:
Norway, Switzerland,
Israel, Turkey, Russia,
Ukraine, Iceland, Kosovo,
Albania
Members (6 Guests)
Austria
Portugal
Belgium
Slovenia
Czech republic
Sweden
Estonia
Switzerland*
France
UK
Germany
(Hungary)
Ireland
Lithuania
(Finland)
Netherlands
(Iceland)
Norway
(Israel)
Poland
(Russia)**
European Social Survey: the future
ESS SUSTAIN
Membership Development activities
Membership Development Advisor appointed
Roaming Ambassadors appointed
Events / seminars in target countries
Funding and Governance Working Group (including BREXIT)
Impact case study
Enhanced communications
Increasing media presence
Launched social media presence
Updating News section on ESS website
Working with national teams to promote ESS in media across Europe
Funded by Commission (H2020)
34
European Social Survey: conclusions
Clear success of the ESS has been demonstrated and the current model is
sustainable
Have to keep making the case for rigour
Need to further increase visibility of the project
Use of data and comparability findings suggest ESS is a leader in the field
Mixed mode would undermine quality but FtoF fieldwork costs rising: web
additions in the future?
ESS core questionnaire will be reviewed
Rotating modules continue to allow new topics to be included
SERISS project offers opportunities for joint methodological development
Especially CRONOS panel
SUSTAIN project helping to secure the future but challenging landscape
BREXIT is a new challenge
35
CONTACT
[email protected]
www.europeansocialsurvey.org
@ESS_Survey
european-social-survey