The Effects of Adding a MobileCompatible Design
to the American Life Panel
Vera Toepoel, Utrecht University
Alerk Amin, RAND Corporation
Peter Lugtig, Utrecht University
RAND American Life Panel
• Nationally representative sample
• Internet surveys
• Provide computers / internet access to
those who don’t have it
• Link responses to over 400 past
surveys
• English / Spanish
Slide 2
ALP Summer 2014
• ~5000 respondents
• “Desktop Only” web interface for
surveys
Slide 3
ALP Fall 2014
• Panel Refreshment
• Mid-Term Election Study
• Mobile-Compatible Interface
Slide 4
Panel Refreshment
• Recruit ~900 new respondents
• Recruitment with RDD
– Asked to take a web survey
– In web survey, asked about
interest/consent to join ALP
– Added to panel and immediately eligible
to be selected for surveys
Slide 5
Mid-Term Election Study
• A series of surveys leading up to the
November 2014 mid-term elections
• 6 weekly surveys
– 1st survey opened September 28 2014
– 6th survey opened November 5 2014
• Administered to all ALP panel members
– Existing members
– New RDD sample
Slide 6
Mobile-Compatible Interface
• New responsive design
• Compatible with desktops, phones,
tablets
Slide 7
Original Interface - Desktop
Slide 8
Original Interface - Phone
Slide 9
New Interface - Desktop
Slide 10
New Interface - Phone
Slide 11
Introduction of Mobile Interface
Existing panel
members sent
notification about new
interface with some
screenshots
New RDD sample never
saw the “desktop-only”
interface
Slide 12
Research Questions
1. Has usage of mobiles and tablets
changed since introduction of new
design?
2. Are there differences in mobile adoption
rates between existing panel members
and new RDD recruits?
3. Which socio-demographics predict
device use?
4. Is there a correlation between device
use and voting behavior?
Slide 13
Analysis
• Track device usage during Mid-Term
Election Study
– User-Agent Strings to identify device
Slide 14
Surveys
Survey
Field Start
Field End
# Questions
Note
385
April 23
May 27
11
Last desktoponly survey for
entire panel
406
September 28
October 04
9
1st mobilecompatible
survey in ALP
407
October 05
October 11
6
408
October 12
October 18
6
409
October 19
October 25
6
410
October 26
November 01
6
Last survey
before election
411
November 05
November 13
6
Post-election
Slide 15
Introduction of New Design –
Existing Panel Members
80
70
60
50
385
406
40
30
20
10
0
PC
Tablet
Smartphone
Other
Slide 18
Device Usage
%
PC
Tablet
Phone
Other
N
PM
73.4
10.2
15.2
1.2
2909
71.8
11.1
16.0
1.1
1799
PM
68.8
10.0
19.4
1.7
1152
New ORC
77.1
13.0
9.9
0.0
647
Wave 385
Wave 406 Total
Chi
Square
Wave 407 Total
42.18***
73.2
10.2
15.7
.9
1755
PM
70.7
9.6
18.3
1.3
1112
New ORC
77.4
11.3
11.3
0.0
647
Chi
Square
Wave 408 Total
25.33***
72.1
10.5
16.4
1.0
1755
Present
69.5
9.5
19.3
1.6
1101
Slide 19
Device Usage for 406
80
70
60
50
Existing
New RDD
40
30
20
10
0
PC
Tablet
Phone
Other
Slide 20
Existing Panel Members Device Usage
Over Time
100%
90%
80%
70%
60%
Other
Phone
Tablet
PC
50%
40%
30%
20%
10%
0%
385
406
407
408
409
410
411
Slide 21
Switching Devices Across Waves
Surveys
%
Existing
385-406
406-407
407-408
408-409
409-410
410-411
19.8
10.6
12.2
11.6
11.4
10.9
% New
RDD
Sample
9.9
9.4
10.5
9.7
12.2
Total
Chi
Square
N
19.8
10.3
11.1
11.2
10.7
11.4
.19
2.82
.45
1.18
.65
1073
1513
1467
1557
1666
1685
Slide 22
Longitudinal Device Use –
Existing vs New Panel Members
% New RDD Sample Total
Always PC
% Existing Panel
Members
60.8
67.4
63.3
Switch PC-tablet
6.7
7.3
6.9
Switch PC phone
15.3
12.4
14.2
Always tablet
5.1
6.6
5.7
Always phone
12.0
6.3
9.8
N
1187
742
1929
Pearson Chi-Square 23.13, p=.000
Slide 23
Demographic Comparison –
Existing vs New Panelmembers
%
% Existing
Panel
Members
% New RDD
Sample
Total
Chi Square
N
BornUS
White
Black
Hispanic
Male
Separated
Never Married
86.3
69.2
14.4
28.1
38.1
21.8
21.9
92.6
85.2
8.4
4.8
48.3
22.6
18.1
87.5
72.3
13.2
23.5
40.1
22.0
21.2
26.17***
92.24***
22.42***
215.99***
30.97***
.26
6.13**
4543
Edu low
Edu high
Income low
Income high
hhsize>2
work
27.7
31.3
18.5
19.4
22.6
58.6
11.1
55.2
7.6
37.8
12.9
55.8
24.4
36.0
16.4
23.0
20.7
58.0
106.95***
177.28***
62.06***
137.34***
41.34***
2.32
Slide 25
Nominal Regression Device Use Over Six Waves With
Demographics - Always PC Is Reference
B
Switch PC-tablet
Switch PC-phone
Always tablet
Always phone
Intercept
1.78
.26
.23
2.24**
Age/10
-.01
-.54***
-.19**
-.71***
Born US
.38
-.01
.17
.41
White
.22
.29
-.02
-.01
Black
.83*
.69*
.79*
1.04***
Hispanic
.71**
.42***
.99***
.88***
Male
-.40**
-.18**
-.45**
-.17
Separated
-.50*
-.07
-.56**
-.01
Never Married
-.67*
-.38*
-.09
-.54*
Edu low
-.36
-.09
.15
.14
Edu high
.28
-.26*
-.24
-.56**
Income low
-.17
-.05
.25
.21
Income high
.44**
.22
.04
-.44
hhsize>2
.29
.07
.14
.34
work
.17
.36*
-.45*
.06
-.06
.61**
.003
New RDD Sample .004
Slide 26
Voting Behavior in 2014 House Elections
Always
PC
Republica 70.6
n
Democrat 73.4
Other
73.4
Switch
Switch
Always
PC-tablet PC-phone tablet
8.4
10.4
4.6
Always
phone
6.0
7.0
1.6
4.0
6.3
11.2
15.6
4.3
3.1
No significant differences
Chi-square= 10.37, p=.22
Slide 30
Conclusions
• New design led some existing panel
members to switch from PC to phones
• Tablet usage did not change
• Overall usage switched during
introduction, then remained steady
– About 10% “churn” each wave
• Device not correlated to voting
behavior
Slide 31
Socio-Demographic Conclusions
More Likely to Use Mobile
• Young
Less Likely to Use Mobile
• Male
• Black or Hispanic
• Single
• Working (switch PC-phone)
• Higher Education Level
• Working (tablet)
Slide 32
Thank You
[email protected]
[email protected]
[email protected]
Slide 33
Doing Surveys Online
20%
DISCOUNT
Vera Toepoel - Utrecht
University
•
October 2014
•
£25.99 £20.75
This book helps students and researchers in identifying
•
ISBN: 978-1-44624-967-3
possible strategies to make the best use of online surveys,
providing pro’s and con’s, and do’s and don’ts for each
strategy. It also explores the latest opportunities and
developments that have arisen in the field of online surveys,
including using social networks, and provides expert guidance
and examples of best practice throughout.
Enter DISCOUNT CODE UK15CF31 at sagepub.co.uk or visit the SAGE stand to order
Offer valid till 31/12/2015 and cannot be used in conjunction with another offer.
Los Angeles | London | New Delhi | Singapore | Washington DC
Slide 34