What They Can’t See Can Hurt You:
Improving Grids for Online Surveys
on Mobile Devices
Randall K. Thomas, Frances M. Barlas,
Patricia Graham & Thomas Subias
GfK Custom Research
Acknowledgements
Special thanks go to the
following people from GfK
Research for their inspiration,
design, dedicated hard work, and
innovative programming skills
required to develop the grid
alternatives examined in this
study:
Jason Knight,
Tetyana Shvets,
Eugene Kagan, and
Liju James
2
Introduction
Screen real estate is a key determinant of
successful measurement in online surveys.
 The screen size we use in developing our
surveys on desktops or laptops is often not as
small as what participants use when they take
our online surveys:
 25% to 35% of surveys are being started
with a mobile device.
 20% to 30% of surveys are being completed
on a mobile device (higher levels of breakoffs occur with mobile devices).
3
Introduction
Single items on a single screen requiring a single
response are common in online surveys but are
not very efficient because they…
• Take time to read and understand the item
stem and the unique responses
• Take time for unique manual effort to select
the appropriate response
• Take time to select next to advance to next
question
4
Introduction
The grid format evolved to be more efficient
than single items across screens. Typically, a
grid presents responses in columns with the
elements to evaluate in the rows.
• The grid obtains efficiency because:
 Manual responding is standardized
 Response meanings are standardized
5
Introduction
Researchers and statisticians have some
materially misguided beliefs:
• MYTH 1: Increasing the number of response
categories from 3 to as much as 11 response
categories increases true score variance
relative to total variance.
• MYTH 2: Aggregate estimates are improved
by increasing the number of items
measuring the same concept.
6
Introduction
As researchers developed grids with large
numbers of items and a high number of responses,
grids have become less efficient and counterproductive - causing higher levels of participant
fatigue and reactance:
 Break-offs
 Speeding
 Non-differentiation – straightlining
 Lower response rates in subsequent surveys
Time to STOP the Grid Insanity!!!
7
Introduction
With the significant rise in mobile devices used
for online surveys, especially smartphones,
participants often run into two problems with
grids that have many response columns:
 If the grid reformats to be readable, not all
columns can be seen simultaneously, or…
 If the grid reformats so that all items and
response columns are seen simultaneously
(by shrinking the display), the items and
responses become unreadable and
responses are harder to make.
8
Introduction
To improve the grid for mobile presentation, we
explored two possibilities in this study:
• Simplify grid presentation - Simplify the
nature of the presentation of grids and/or
responses in order to reduce the total real
estate required.
• Make grid measurement more mobile
friendly - Reduce the number of response
categories to make the survey more mobile
friendly.
9
Method
10
Method
Sample was drawn from GfK's probability-based
KnowledgePanel®, the largest web-based panel
designed to be representative of the U.S.
population.
 Respondents for this panel were first selected
randomly using phone or mail survey
methods, and later interviewed online.
 People selected for KnowledgePanel who
didn't otherwise have access to the Internet
were provided access at no cost to them.
11
Method
To be eligible for this study, participants had to
have both a desktop or laptop computer and one
or both of a smartphone or a tablet.
 Once selected, participants were randomly
assigned to take the survey on a specific
device type when selected, which was
confirmed within the survey.
 10,672 were selected to participate
 4,555 completed on assigned device
 About 1,500 completed on each device type
(desktop/laptop, tablet, smartphone)
12
Research Question 1:
Are there simpler and better grid
presentation formats than the
Traditional Grid?
13
Method – Alternative Grid Formats
We developed 3 different grid questions and randomly
assigned respondents 1 of 4 formats, 3 that were new
and we believed would simplify the presentation and
response***:
1. Traditional Grid - row and column matrix
14
Grid Format – Traditional Grid
15
Method – Alternative Grid Formats
We developed 3 different grid questions and randomly
assigned respondents 1 of 4 formats, 3 that were new
and we believed would simplify the presentation and
response:
1. Traditional Grid - row and column matrix
2. Banked Items - items with the responses
presented horizontally under each element.
16
Grid Format – Banked Items
17
Method – Alternative Grid Formats
We developed 3 different grid questions and randomly
assigned respondents 1 of 4 formats, 3 that were new
and we believed would simplify the presentation and
response:
1. Traditional Grid - row and column matrix
2. Banked Items - items with the responses
presented horizontally under each element.
3. Responses on Right - Elements presented along
the left in rows and response options presented
vertically on the right.
18
Grid Format – Responses on Right
19
Method – Alternative Grid Formats
We developed 3 different grid questions and randomly
assigned respondents 1 of 4 formats, 3 that were new
and we believed would simplify the presentation and
response:
1. Traditional Grid - row and column matrix
2. Banked Items - items with the responses
presented horizontally under each element.
3. Responses on Right - Elements presented along
the left in rows and response options presented
vertically on the right.
4. Focal element – one element presented at a time
with the responses presented underneath.
20
Grid Format – Focal Element
21
Grid Presentation Results
22
Grid 1 Results – Time to Complete
Grid 1 presented 8 beverages and asked for liking of each using a 3
category scale. The traditional grid format took the least amount of
time to complete, the focal element presentation took the most time.
23
Grid 1 Results – Item Means
Mean liking of beverages were not substantially affected
by grid format.
24
Grid 2 Results – Time to Complete
Grid 2 asked people to rate the importance of 7 different aspects of
beverages in choosing one for between meals. Smartphones took
longer to complete, the Traditional Grid took the least amount of time.
25
Grid 2 Results – Item Means
Mean ratings of importance were not affected by Response Format
or Device.
26
Grid 3 Results – Time to Complete
Grid 3 presented 8 brands to rate liking using a 3 category scale.
Smartphones took longer to complete overall, the traditional grid
took the least amount of time to complete.
27
Grid 3 Results – Item Means
Mean ratings of brand liking were not affected due to Response
Format or Device.
28
Research Question 2:
Can we make grid response formats
more mobile friendly?
29
Mobile Friendly Response Formats
In addition to testing alternative designs for
grids, we were interested in making grids more
mobile friendly by shortening the response
scales. To do this, we used 4 traditional grids
and randomly assigned either:
 Mobile-friendly grids – having a smaller
number of responses
 Mobile-unfriendly grids – having a larger
number of responses
All responses in the grids were fully anchored
(each response had a semantic label).
30
Grid Manipulation – Mobile Unfriendly
Grid Manipulation – Mobile Friendly
Mobile Friendly Results
33
Grid A Results – Time to Complete
Grid A had 5 items on tech attitudes, with an agreement
scale, either 7 bipolar categories (unfriendly) or 4
unipolar categories (friendly).
34
Grid A Results – Item Means
Means for the Friendly and Unfriendly scales were
different in ways normally found for bipolar and unipolar
scales, but were in the same order.
35
Grid A Results – Validity
Somewhat higher validity for friendly scales - correlations
of tech items with total number of tech products owned
were higher with friendly scale than unfriendly scale.
36
Grid B Results – Time to Complete
Grid B had 5 items evaluating their survey experience,
using an agreement scale. The friendly version had 4
unipolar responses, the unfriendly had 5 bipolar
responses.
37
Grid B Results – Item Means
Agreement ratings using 4 unipolar versus 5 bipolar
response format, results in same order.
38
Grid C Results – Time to Complete
Grid C asked participants to rate the importance of various TV
features, with either a 5 category unipolar scale (unfriendly) or a 3
category unipolar scale (friendly).
39
Grid C Results – Item Means
Importance ratings using 3 versus 5 category unipolar
response format, results not different, and in same order
40
Grid D Results – Time to Complete
Grid D asked participants to rate the importance of features of
toothpaste in the purchase decision with either a 5 category unipolar
scale (unfriendly) or a 3 category unipolar scale (friendly).
41
Grid D Results – Item Means
Importance ratings using 3 versus 5 category unipolar
response format, results in same order.
42
Conclusions
Do Alternative Grid Presentation Formats Work?
In 3 different grids, varying by topic and measurement
concept (importance, liking) we randomly assigned 4 grid
formats.
 As has been found in other areas of research with
measures like sliders, fancy and interactive
measures often fall short of the efficiency of simpler
measurement techniques. Most of the new grid
formats took longer to complete, though none
showed significantly different means for items from
the Traditional Grid.
 The Traditional Grid appears to function well across
all devices – though we used only 3 category scales
for all grids in this comparison.
43
Conclusions
Can Grid Response Formats be Made Mobile Friendly?
 For 4 grids, we compared longer, more mobileunfriendly scales (with more responses) with shorter,
more mobile-friendly scales (with fewer responses).
Mobile-friendly scales…
• Did NOT substantially alter findings, having
equivalent differentiation capability
• Took less time to complete
• In some cases, may be more valid.
Yes, response formats can be made mobile friendly - 3 to 4
category unipolar fully labeled scales work well and may
be ideal for measurement in the mobile-oriented future.
44
Conclusions
Generally, the traditional grid can be improved by
decreasing response categories. Some caveats –
1. This was a purposefully mobile study and within a
sample that had access to multiple devices, including
at least 1 mobile device.
2. This study had a higher quality sample (probability
sample, high cooperation rates). Accidental mobile
opt-in participants may respond differently.
3. While these recommendations work nicely for new
surveys under development, changing response
formats that have a history (e.g., multiple waves for
which there is a historical trend) is difficult, often
requiring a bridge study to map the new to the old.
45
Thank You!
Randall K. Thomas
[email protected]