Screen Size vs. Interface vs. Double-Entry:
Combinations that Yield
Low Error Rates
When Entering Dates on Touchscreen Devices
Dale Rhoda1
Andrea Guillot2, Temitope Adeyoju2, Mary Prier1,
Nikhil Patil2, Mary Kay Trimner1, Thomas Albani1
1 Biostat
Global Consulting, Worthington, Ohio, USA
2 eHealth Africa, Kano, Nigeria
Three Commonsense Assertions
• We want survey
responses to be
recorded accurately –
with very low data
entry error rates
• In some surveys, dates
are an important
category of response
• Error rates for dates
are likely higher than
rates for some other
response types
2
Photo credit: Andrea Guillot
Our Experiment
3
Vaccination Records
• We did an experiment (the first of
several) looking at screen size,
interface and confirmation protocol
• We made up vaccination records for
30 imaginary children and wrote
them on vaccination record cards
• Up to 19 dates per card (mean = 10)
• All dates ordered D-M-Y & handwritten
• We hired 24 data collectors to spend
a day entering all 30 cards 2X each
• 12 participants on each of two days
• At eHA offices in Kano, Nigeria
• We obtained ethical approval from
the Kano State Ministry of Health
4
Devices
• Tablet-- Asus Xplore iX101T1
• Screen dimensions
• Length - 23.5 cm (9.25 inches)
• Width - 14 cm (5.5 inches)
• Phone-- HUAWEI Ascend Y320 U30
• Screen dimensions
• Length - 9 cm (3.54 inches)
• Width - 5.6 cm (2.20 inches)
http://ruggedpcreview.com/images2/xplore_rangerx_angle_400.jpg
http://compareindia.news18.com/media/gallery/images/2013/oct/y320_2_311331322688.jpg
5
Interfaces
Open Data Kit (ODK) is a popular,
free, mobile survey tool for
Android devices that is commonly
used for data collection in lowresource settings
Our Date Interfaces included:
• Calendar widget (ODK default)
• Wheel widget (‘nocalendar’ option)
• A homegrown radio button
interface
https://opendatakit.org/
6
Wheel Interface
• Default to today’s date
• Spin 3 wheels to select
month, day & year
• Left swipe to complete entry
• Constraint:
order is M-D-Y instead of D-M-Y
• Note: Month uses abbreviation
instead of ##
7
Calendar Interface
Tablet
Phone
•
•
•
•
Default to today’s date
Spin wheels to select month & year
Tap calendar grid to select day
Left swipe to complete entry
• Constraint:
order is M-Y-D instead of D-M-Y
• Note: Month uses abbreviation
instead of ##
• Narrow phone screen required
left/right scrolling which is easily
mistaken for left/right swiping
8
Radio-button Interface (not an ODK date widget)
• No default date
• Three consecutive screens
• Select one and swipe left to
proceed
• Ordered D-M-Y (like card dates)
• Month lists both ## and
abbreviation
• Phone screen required user to
scroll upward to select day > 15
• No check for nonsensical dates
(e.g. 31 February)
9
No Confirmation
Double-entry
Is there a
date for X
dose?
Is there a
date for X
dose?
No
No
Yes
Enter date
Yes
Enter date again
Enter date
Up to 4X
No
Dates
agree or
5X
disagree?
Yes
Next dose
Next dose
10
Hypotheses (a priori)
• Unconfirmed error rates are likely to be > 1%
• We hoped that confirmation would bring error rates down < 1% for all
combinations of interface & device
• Tablet will likely yield lower error rates than phones due to little or no
need to scroll screen
• No strong prediction about difference between interfaces
11
Balanced Design for Strong Inference
• Design is meant to eliminate possibility of confounding factor-related
differences in error rates (which we hope to see) with a learning
effect or with fatigue
• All participants
•
•
•
•
Enter data from 5 cards at each of 12 stations (60 cards total per person)
Enter all 30 cards using confirmation; enter all 30 cards unconfirmed
Enter 30 using phone; 30 using tablet
Enter 20 cards using each of the three interfaces
• Each block of 5 cards holds 55 dates in total
• Order of blocks of cards was randomized and balanced
12
Latin
Squares
This slide shows the
order of factor
combinations for 24
participants
13
Protocol for Experiment Day
• Participants were oriented & gave consent
• Filled out questionnaire re: demographics & technology use
• Received individual packet of 60 vaccination cards in prescribed order
• Visited 12 data entry stations in prescribed order
(order customized for each participant according to design)
• Could take as long as they needed at each station
• Checked with experiment proctor before moving to next station
14
Results
Photo credit: Andrea Guillot
15
Demographics & Technology Use
• All 24 identified as right handed or ambidextrous
• Use right hand to sign their name & right hand only or both to type on phone
• Over 90% of participants between 18-34 years old
• 75% female
• Device ownership
• 87.5% (n=21) own smartphone; 19 used it 7 days in prior week
• 20.8% (n= 5) own tablets; all used it 5+ days in prior week
16
Analyze Entries that Should Have Dates
• In this analysis we consider only entries that should have had dates:
10,830 entries
• Respondents did sometimes lose their place on the card and enter
dates where they shouldn’t have…
and that is an interesting type of data entry error…
but not our primary concern here
17
We Use Inchworm Plots to
Represent 95% Confidence Intervals (CIs)
Inchworm
No Bar
Bar with 95% CI
50
55
60
65
70
75
80
85
90
18
Mean error rate
across all 12
combinations:
13.2%
Error Rate
by Person
For the remainder of
this presentation I
exclude the five
participants with the
highest error rates –
we could either fire
them or train them to
do better…
19
Mean after
excluding bad
faith and high
error rates:
5.1%
Error Rate
by Person
20
Error Rates
for 12
Combos
= No Confirmation
= Double-entry
21
Sobering Conclusions & Implications
1. Error rates vary considerably by participant
Train & practice & test before hiring workers and sending to the field
2. People cheat
Hire based on test results & make it clear that you will check their work
3. The default ODK interface yields the worst error rates here
More work to recommend optimal solution, but radio buttons are promising
4. Even the best rates here are too high
Use onboard camera to photograph original document for later verification
Consider making accuracy a factor in compensation
22
Next Steps
Additional Analyses
More Experiments
• Understand double-entry errors
on wheel interface with tablets
• How many errors could be
caught with logic checks?
• How many errors give
misleading results in vaccination
quality indicators?
• Better training & practice with
feedback
• Cognitive & usability testing with
double-entry
• Try wheel with D-M-Y order and
month in numbers
• Lots more ideas…
23
Acknowledgements & Feedback
We are grateful for partial funding from the
Bill & Melinda Gates Foundation
and for in-kind services from
eHealth Africa, Kano, Nigeria
Send suggestions or request for
forthcoming manuscript to
[email protected]
24
Backup Slides
25
Accurate Data Entry is Important
• There is a rich history of research on data entry error rates, first by
keyboard and more recently using mobile devices
• Strategies to reduce errors include:
•
•
•
•
Double entry with forced matching
Visual checking
Read-aloud checking
Onboard or post-hoc logic checks for plausibility
26
Themes in Literature Search
• Mobile technologies are popular for data collection
• Skip patterns are well-executed
• Data entry is complete as soon as field work is complete
• Onboard GPS & camera are useful
• Entering dates is more error prone than other field types, but not well
explored with mobile devices
• Open fields versus widgets
• Higher rates of error when a default value present
• Double data entry confirmation seems to help
27
Incomplete Responses & Bad Faith Data Entry
Q1
Q2
Q3
Q4
Missing
Bad Faith Entry (repeatedly entered default date or repeatedly entered numbers off-by-one)
28
Incomplete Responses, Bad Faith Entry
& Very High Error Rates
Q1
Q2
Missing
Q3
Q4
High Error Rate
Bad Faith Entry (repeatedly entered default date or repeatedly entered numbers off-by-one)
29
Number of Cards
Data are
Reasonably
Balanced
Even After
Omissions
TW
Tablet Wheel
TR
Tablet Radio
TC
Tablet Calendar
PW Phone Wheel
PR
Phone Radio
PC
Phone Calendar
30
No
Evidence of
Aggregate
Learning
Effects
31
Results if we exclude only the bad
faith entries…(so include cards
from 23 of 24 respondents)
32
Error Rates
for 12
Combos
= No Confirmation
= Double-entry
33
Error Rates
for 12
Combos
= No Confirmation
= Double-entry
34
Learning
Effects
Error rates
dropped slightly
from Q2-Q4
(This slide includes
23 of 24
respondents)
35
Future Experiments
• Incorporate usability testing!!!
• Reverse radio button interface to use M-D-Y order and see if it gets worse
• Modify wheel and calendar widgets to show date in D-M-Y order
• Revise double-entry instructions: goal is to enter the correct date twice
(not just the same date twice)
• Detect bad faith entries in real-time & do not pay for them
• Experiment with read-aloud protocol
• Same task with double-entry using a computer keyboard
• Conduct orientation & training by video (for consistency)
36