GfK Media
Non differentiation in a web-panel - Lex van Meurs
Respondent characteristics as explanations for uninformative
survey response: sources of non differentiation in a web-panel
Lex van Meurs, Thomas Klausch and Klaus Schönbach
--------------------------------------------------------------------------------
4 October 2010
GfK Media
Non differentiation in a web-panel - van Meurs, Klausch & Schönbach
15 March 2011
Recommendations to prevent dubious responses
2
• Prevent fraudulent behaviour in online questionnaires
by taking recruitment and panel management serious
• Increase data integrity by improving questionnaires:
• Limit the length
• Include rewarding questions
• Pilot questionnaires with extensive feedback
• Discuss questionnaire issues with client
• Check for quality of responses and questionnaires
• Standardise questionnaire feedback
• Check on response quality
GfK Media
Non differentiation in a web-panel - van Meurs, Klausch & Schönbach
15 March 2011
3
Kwaliteitscontrole bij onderzoek
GfK Media
Non differentiation in a web-panel - van Meurs, Klausch & Schönbach
15 March 2011
Flags: the GfK Survey and Panellist Quality Filter
4
1. Response time
2. Non-differentiation (straight lining)
3. Open-ended questions
Do not know
4. Screening
questions
5. Do not know
6. Traps
Traps
GfK Media
Non differentiation in a web-panel - van Meurs, Klausch & Schönbach
15 March 2011
5
Introduction
question
Non differentiation in the GfK Appreciation Panel
GfK Media
Non differentiation in a web-panel - van Meurs, Klausch & Schönbach
15 March 2011
Programme selection
screen
6
GfK Media
Non differentiation in a web-panel - van Meurs, Klausch & Schönbach
15 March 2011
7
Standard questions
asked for all
programmes seen
GfK Media
Non differentiation in a web-panel - van Meurs, Klausch & Schönbach
15 March 2011
8
In-depth questions
GfK Media
Non differentiation in a web-panel - van Meurs, Klausch & Schönbach
15 March 2011
Non differentiation
9
• Dependent variable: proportion of questionnaires with non differentiation
• Non differentiation defined as zero variance in one of three item batteries with
minimum length
• Standard questions: at least 5 items
• In-depth questions: at least 10 items
• Shorter questionnaires were not analyzed (14%)
• Generally non differentiators give high appreciation scores
• Research question: what respondent characteristics can explain non
differentiation?
GfK Media
Non differentiation in a web-panel - van Meurs, Klausch & Schönbach
15 March 2011
Sample and modeling
10
• GfK Appreciation Panel in the Netherlands
• Non differentiation identified in all online questionnaires issued between January 1
and June 30 2009
• 502,750 questionnaires of 7,793 panel members that were part of the panel
when the survey was conducted in August 2009
• Stratified random probability sample (n=1,200)
• n=700 low non differentiation group
• n=500 high non differentiation group
• Modeling: negative binomial regression
GfK Media
Non differentiation in a web-panel - van Meurs, Klausch & Schönbach
15 March 2011
Questionnaire: a hierarchy of impact on response behavior
11
Background Characteristics
Contextual Norms
Behavioral Norms
Response Motivation
Survey Experience
GfK Media
Non differentiation in a web-panel - van Meurs, Klausch & Schönbach
15 March 2011
1. Survey experience
12
Survey length
The questionnaires are often too long.
Cognitive effort
I often feel that I am not up to the demands of the surveys.
Boredom
Often the survey is so boring that it makes me want to stop filling in the
questionnaire.
GfK Media
Non differentiation in a web-panel - van Meurs, Klausch & Schönbach
15 March 2011
2. Response motivation
13
Incentive Focus
• If the financial compensation for my study participation was
taken away, I would most likely stop participating in the
panel.
• The financial compensation is the most important reason
why I started participating in the panel.
• If I heard that the panel is continued without a financial
compensation, I would be really angry.
Involvement
• Enjoyment: I participate in the panel, because I enjoy doing it.
• Topic interest: I participate in the panel, because I enjoy rating television
programs.
Perceived Obligations
• Though I receive a regular incentive, I do not feel that I owe anything to Intomart
GfK.
GfK Media
Non differentiation in a web-panel - van Meurs, Klausch & Schönbach
15 March 2011
2. Response motivation (2)
14
Relevance of survey
• Relevance to TV: I participate in the panel, because I think
the appreciation questionnaires are an important way to
improve the quality of television.
• Relevance to TV makers: I participate in the panel, because
this is a way to give my opinion to TV makers.
• Scarcity: I participate in the panel, because I am one of the
few to give an opinion in the name of a large TV audience.
• Self-efficacy: My answers in the panel contribute to the
production of programs that people like me like.
• Self-efficacy (reversed): TV makers are not interested in the
answers that people like me give in the appreciation panel.
GfK Media
Non differentiation in a web-panel - van Meurs, Klausch & Schönbach
15 March 2011
3. Behavioral norms
15
Social norms
• As I believe that many people do not fill out the surveys
very properly, I am not doing it either.
Individual norms
• I think it is not a problem to fill out the surveys a bit sloppy
from time to time.
GfK Media
Non differentiation in a web-panel - van Meurs, Klausch & Schönbach
15 March 2011
4. Contextual norms
16
Honesty
• Is it justified not to tell the truth?
Helping
• I am a person who enjoys helping other people a lot.
• People often ask me for help with their problems.
• Helping other people makes me happy.
Social integration
• Compared to other people of your age, how often would you say you take part in
social activities (other than work or duty)?
GfK Media
Non differentiation in a web-panel - van Meurs, Klausch & Schönbach
15 March 2011
5. Background characteristics
17
Socio-demographics controlled for:
• Gender
• Age (16-29, 30-44, 45-59, 60+)
• Education (high, medium, low, none)
• Income (4 groups)
GfK Media
Non differentiation in a web-panel - van Meurs, Klausch & Schönbach
15 March 2011
Results and discussion
19
• Norms are very relevant (by 29% per sd for lower social
norms, by 9% per sd for lower individual norms)
• Honest people respond more honestly: respondents
rating honesty important have 32% lower SL rate
than others
• Perception of behavior of others is equally relevant
• But we cannot influence these effects – or can we?
• High age, low education and low income all increase SL
GfK Media
Non differentiation in a web-panel - van Meurs, Klausch & Schönbach
15 March 2011
Results and discussion (2)
20
• Survey experience: perceived effort increases non
differentiation (by 13% per sd)
• What is perceived as causing effort is individually different
• Generally useful to avoid long or complicated
questionnaires
• Respondents who perceive the relevance of the survey as high use non
differentiation more frequently (by 34% per sd)
• Instrumentalisation: “I like everything I see and now I can tell them”?
• Respondents showing this behavior are generally lower educated
• No effects due to incentive, if at all: positive impact
• Use incentives without worries
• But effects of income may point to higher attractiveness of
incentive for this subgroup
GfK Media
Non differentiation in a web-panel - van Meurs, Klausch & Schönbach
15 March 2011
Acknowledgements
21
• Publications
• Meurs, Lex van; Ossenbruggen, Robert van & Nekkers, Liesbeth. (2007). Do rotten
apples spoil the whole barrel? Exploring quality issues in panel data. In: ESOMAR Panel
Research 2007 Conference (CD-ROM). Amsterdam: ESOMAR.
• Meurs, Lex van; Ossenbruggen, Robert van & Nekkers, Liesbeth. (2009). Rotte appels?
Controle op kwaliteit van antwoordgedrag in het Intomart GfK online panel. In: Bronner,
A.E et all (eds) Ontwikkelingen in het marktonderzoek 2009: Jaarboek
MarktOnderzoekAssociatie, dl. 34, 2009, p 61-81. Haarlem: SpaarenHout.
• Winner of the MOA Wetenschapsprijs 2009
• Upcoming publication in collaboration with Thomas Klausch (University Utrecht) and Prof
Dr. Klaus Schönbach (University of Vienna)
• Supported by
• Dutch Public Broadcasters (NPO)
• Intomart GfK
• The Amsterdam School of Communication Research