What happens if you offer a mobile
option to your web panel?
Results from a pilot study comparing a desktop
and mobile device survey
Vera Toepoel
(Utrecht University, [email protected])
Peter Lugtig
(Utrecht University/University of Essex, [email protected])
1
Vast increase in mobile phone use

According to Statistics
Netherlands, 50% of all 12 to
72 year olds with Internet
access do this via a mobile
device in 2011

Smartphone penetration rate
explosion
– In 2011 1 out of 5 people
opened their email on a
smartphone (source: Emerce)
http://www.emerce.nl/research/mobiel-emails-lezen-2011-verdubbeld
2
Smartphone as research tool?

Online probability based panel of Market Response
– RDD, Internet population
– Panel members ask if they can complete the survey on their mobile
phone
– Young panel members (age 18-34) have the lowest response
probabilities in the panel and are continuously underrepresented in
the panel (reasons for non response: don’t feel like it, no time)
– Current software not capable of using mobile friendly design

Confirmit software offers ‘responsive design’ (survey
automatically optimized for the device chosen)
3
Research Design
Two parallel surveys

June 2012 two surveys with similar sampling frame
– (NL, 18+, Internet users, mobile registration):
– Standard QSL (current online method)
– Confirmit with response design (respondent could chose
prefered device)
» Soft check for mobile completion
Invitation: QSL
Confirmit
4
Research Questions
What happens if you offer a mobile option
to your web panel?

Amount of people using mobile
– Substantial?

Socio-demographic characteristics
– Coverage bias: age, edu, gender, urbanization, hh size+composition




Response (break offs and item non-response)
GPS
Evaluation
Response quality (measurement error)
– Questionnaire already adapted to mobile (no experiment)
– Open-ended questions, check-all-that-apply, primacy
5
Research Questions
Amount of people using mobile

57% used mobile phone throughout the survey (N=443)
–
–
–
–
252 used mobile (57%)
33 people switched from desktop to mobile during the survey (7%)
4 people switched from mobile to desktop(1%)
154 people used desktop (35%)
– Note: we define mobile as smart phone only. Additional analyses
with mobile=all with touchscreen (incl. ipads etc.) produces similar
results.
6
Research Questions
Socio-demographics

Age, hh size and hh composition (kids) significant predictors
of mobile use
– Older people
– Larger households
– No kids

No effect of: urbanization, gender, social class, education,
household position, family income
7
Hard-to-reach-group
Young: mobile increases response probability
8
Research Questions
Response

Break offs limited
–
–
–
–

Short questionnaire
4 out of 285 break off on mobile
3 out of 153 on desktop
No evidence that mobile results in higher break offs
Item non-response
– 7% over 3 questions
– No significant difference mobile versus desktop
– No significant difference in non-substantive answers (don’t know)
9
Research Questions
time to complete
10
Research Questions
Day questionnaire completed
11
Research Questions
GPS
Mobile
Desktop
Total
GPS ok
38%
8%
25%
N
253
190
443
%
Home
Work
On the road
outside
abroad
Mobile
67
14
11
6
2
N=110
12
Research Questions
Evaluation



%
Mobile
Desktop
Total
confirmit
survey
qsl
Positive
(8-10)
42
44
43
38
Neutral
(6-7)
50
47
49
55
Negative
(1-5)
8
9
8
7
n(mobile)=281, n(desktop)=151, N(QSL)=498
Chi-square test across mobile and desktop not significant
Chi-square test across all three groups not significant
13
Evaluation
Higher evaluation young people
18-24
7.7
25-34
7.1
35-44
7.5
45-54
7.1
55-64
6.9
65+
6.4
7.2
7.2
7.2
7.2
14
Research Questions
Response Quality: open-ended questions
%
Mobile
Desktop
No answer
35%
33%
Total
(Confirmit)
33%
0-50 characters 52%
49%
51%
51-100
charcters
100 +
characters
10%
16%
13%
3%
3%
3%
Notes: N(mobile)=285, N(desktop)=154
No significant difference
15
Research Questions
Response Quality: check-all-that-apply
%
0
1
2
3
4
5
6
7
8
Mean number of
shops visited
Mobile
24.2
9.1
15.4
14.7
14.4
8.8
7.0
4.6
1.8
2.7
Desktop
21.4
9.7
18.2
12.3
15.6
8.4
6.5
4.5
3.2
2.8
Total (Confirmit)
23.2
9.3
16.4
13.9
14.8
8.7
6.8
4.6
2.3
2.8
Number of shops visited in past year
Notes: N(mobile)=285, N(desktop)=154
No significant differences
16
Research Questions
Response Quality: primacy in radio buttons
%
One a week or
more often
About once a
month
About every 3
months
About once
every 6 month
Once a year o
less often
Don’t know
Mobile
3.9
Desktop
0.6
Total Confirmit
2.7
23.9
27.7
25.2
37.2
32.1
35.4
21.4
26.4
23.2
10.2
9.4
9.9
3.5
3.8
3.6
Notes: N(mobile)=285, N(desktop)=154
Frequency of buying clothes in a shop nearby
No sigificant difference
17
Feedback mobile option
Perfect format, excellent speed! Super!
The scrollbar was not working properly.
Did not like it. it is easier on
desktop(evaluation 3: woman, 58)
I am glad you figured out that desktop pcs are getting used less often and
that questionnaires should be made mobile friendly. I don’t need to get my
laptop (evaluation 8: woman, 29)
Looovvedddd
it(evaluation 9: woman,
18)
I like this very much. I do
not need to get my laptop
(evaluation 8: man, 20)
Very good!!! I will
definetely get higher
participation rates if this
will be implemented
(evaluation 9: man, 32)
Good. But it should not
take me too long,
otherwise I’ll break
off(evaluation 8: man, 51)
Good! I often complete
surveys on my mobile
phone but this is the first
one that reads well,
perfect format, excellent
speed! Super! (evaluation
9: woman 24)
Handy! Better then the desktop
since I tend to forget the survey
invitations (evaluation 8:
woman, 38)
Don’t like it. I have
a pc which I use for
all my survey
activities. Next time
I won’t be doing the
survey(evaluation 1:
man, 71)
18
Conclusion







Considerable amount (57%) uses mobile
No effect on non-response
No effect on response quality
Similar evaluation
GPS gives additional insights
No reason to believe that mixed-device is a
problem
Able to attract hard-to-reach group such as
young people
19
Discussion

What about difficult questionnaires?
– Long
– grids



What about privacy?
How to offer additional information to
researchers?
Coverage bias?
Thank you!
20