The Philosophy of French experiments
on Internet and mixed modes data collection
Tiaray Razafindranovona
September 26th, 2013
Introduction
2
•
Better quality, more complexity …
•
… in a general context of budgetary reduction !
•
Mixed modes to face these challenges ?
•
Experimental surveys will help to answer
Principles of French experiments
•
An experiment for (almost) each new household survey
–
–
•
No interference with main survey
–
–
•
Workload
Sampling frames
Each survey must test something new
–
–
3
General results on mixed modes surveys not sufficient
Experiments in parallel of each new CAPI survey
Accumulation of experience
Test of new specific points not already handled
Standard protocol
•
Sample frames
–
–
•
Questionnaire
–
–
•
Fitted subset of the CAPI questionnaire
Length around 10/15 minutes
Notification letter, reminders
–
–
4
Name and adress for massmailing
Available in taxpayers files
First, a postal mail with a notification letter
2 reminders (one with paper questionnaire)
Examples of experiments
•
Housing 2010 (Amiel and Denoyelle, 2012)
–
–
•
Safety survey (SASU) 2013
–
–
•
Focus on rental amount
External documents without the interviewer ?
Wealth 2014
–
–
5
First experimental survey on victimization
2 protocols to control the problem of self-selection
Housing 2014
–
–
•
Comparison with results of Housing 2006 (face-to-face)
Internet respondents less satisfied (calibration techniques)
Focus on value of real estate property
Deviations from market value without the interviewer ?
Working conditions
Quality of life at work (QLW)
•
Working conditions survey (face-to-face survey)
Survey adresses many topics surrounding working conditions :
– working hours and organization of working time
– psychosocial constraints
– relations with the public …
•
Experimental parallel survey : Quality of life at work
Tackles same topics : shorter version of the questionnaire.
Sample drawn in a base built on taxpayers files :
only people that have declared earnings (in 2010) from work can be selected.
6
QLW – Aims and protocols
•
Aims
–
Comparison of
•
•
•
–
–
•
7
Response rates between internet and internet+paper
Quality and content of responses between traditional and experimental surveys
Responses between 2 protocols involving different notification letters
Test questions of the future survey on psychosocial risks (2015)
Open questions to raise topics for this survey
Protocols
–
2 different models of notification letters (psychosocial risks)
–
For half of the sample, internet and paper (standard protocol),
for the other half, only internet
QLW – Response rates
At the end of the data collection
Subsample 1 - model A letter, internet and paper available as modes of response
Subsample 2 - model B letter, internet and paper available as modes of response
Subsample 3 - model A letter and only internet available as mode of response
Subsample 4 - model B letter and only internet available as mode of response
(model B letter insists on psychosocial risks)
8
QLW – Sample vs internet / paper
Comparison on sample frame variables
9
QLW – Internet vs paper respondants
Comparison on survey variables
10
QLW – Examples of mode effects
•
Differentiation on responses scale (satisficing)
–
–
–
•
Emotional well-being
–
–
–
11
Control of selection with kernel matching on propensity score
Index of response differentiation (McCarty and Shrum, 2000)
Result : differentiation more pronounced on internet than on paper
Control of selection with kernel matching on propensity score
WHO-5 well-being index (de Wit et al., 2007)
Result : emotional well-being higher on paper
Conclusion
12
•
Mixed modes : adaptations to technological changes
•
Not to be forced without prior evaluation of quality
•
Future results of experiments will help to re-design
•
A standard scheme ? Trade-off with society requests
References
Amiel, M. H,. and Denoyelle, T. (2012), Enquêtes en ligne : comparaison de modes de questionnement sur le
thème du logement, XIèmes Journées de Méthodologie Statistique de l’Insee
de Peretti, G., and Razafindranovona, T. (2013), Les enquêtes multimode : multi-solution ou multi-problème ?,
45e Journées de Statistique de la SFDS.
de Wit, M., Pouwer, F., Gemke, R., Delemarre-van de Waal, H. and Snoek, F. (2007), Validation of the WHO5 Well-Being Index in Adolescents With Type 1 Diabetes, Diabetes care, 30.
Heckman, J., Ichimura, H., and Todd, P. (1998), Matching as an Econometric Evaluation Estimator, Review of
Economic Studies,65.
Krosnick, J. (1991), Response strategies for coping with the cognitive demands of attitude measures in
surveys, Applied Cognitive Psychology. Vol 5.
Lugtig, P. J., Lensvelt-Mulders, G. J., Frerichs, R., and Greven, F. (2011), Estimating nonresponse bias and
mode effects in a mixed mode survey, International Journal of Market Research, 53(5).
Lyberg L. (2012), La qualité des enquêtes, Techniques d’enquête, 38(2).
McCarty, J. A., and Shrum L. J. (2000), The measurement of personal values in survey research: A test of
alternative rating procedures, Public Opinion Quarterly, 64.
13