Memorability of Pre-designed
and User-defined Gesture Sets
Miguel A. Nacenta,Yemliha Kamber,
Yizhou Qiang, and Per Ola Kristensson
1
User-defined gestures are easier to
remember than pre-designed gestures
2
Gestural Interfaces
Image credits: wikimedia commons, leap motion, Samsung.com, IbTimes.com
3
Previous research
GDT (Long, Landay, Rowe)
Charade (Baudel & Beaudoin-Lafon)
(Wobbrock & Ringel Morris & Wilson)
SHARK (Zhai & Kristensson)
4
Possible advantages
•
•
•
•
•
Any location
Multiple people
Less clutter (menus)
Many possible gestures
Many possible actions
5
Remembering Gestures
• Critical part of learning an interface
6
What type of gestures are most
memorable?
• Pre-designed
• User-defined
• Stock
7
Three Studies
Exp 1 (6 people)
Exp 2 (9 people)
Exp 3 (18 people)
8
Experiment 3
• Within-subjects, 18 participants
• 3 conditions (Pre-designed, User-defined, Stock/Random)
– Learn 22 Gestures (associated to 22 actions)
– Reinforcement (11 Gestures)
~24 h
– Next-day Test
– Repeat
9
Experiment 3
• 3 Example Applications
– Image Editing
– Web Browsing
– Word Processing
• Application Selection Criteria
– Multiple commands/actions
– Familiar to participants
– Representative of other apps
10
Applications and Actions
• 22 Actions selected for each application
• Action selection criteria
– Not similar to generic actions
• Cut, Paste, etc.
– Not similar to actions in
the other conditions
Example:
Image Editing
- Add new layer
- Crop
- Fill
- Select ellipse
- Increase brush
- Clone
- Smudge
- Copy image
- Etc.
11
Setup
• Wizard of Oz - approach
12
Phase 1: Learning
13
Pre-designed Gestures
Add new layer
Crop
Erase
Fill
Flip horizontally
Flip vertically
Incr. brush size
Decr. brush size
Pan left
Pan up
Rotate
Select airbrush
Select ellipse
Select pen
Select rectangle
Smudge
Duplicate image
Zoom in
Clone
Scale image
Incr. brightness
Ripple effect
•
•
•
Different from other
gestures in set
Different from gestures in
the other sets (applications)
Different from common
touch manipulations
14
Phase 2: Reinforcement
• Only 11 of the 22 gestures
• Quiz on these gestures
• Feedback (correct or incorrect)
• See video again + repeat
15
Phase 3: Next-day Test
• Tested on all 22 gestures
• In a different order
16
Condition: User-defined Gestures
• Same conditions
• But gesture creation instead of learning
• Recorded by a webcam
– For experimenter verification
17
Condition: Stock (random) Gestures
Actions
Image Editing
- Add new layer
- Crop
- Fill
- Select ellipse
- Increase brush
- Clone
- Smudge
- Copy image
- …
Designed gestures
Web Browsing
- Home
- Settings
- Open private browser win.
- New tab
- Print page
- Refresh
- Add page to bookmarks
- Access history
- …
18
Measures
• Correctness: (for reinforcement and next-day measures)
– Correctly Recalled
– Close recall – partial gesture errors
– Error – association errors
• Time
• Subjective measures
– Effort Likert scales
– Rankings of preference
19
RESULTS
20
Next-day Test (recalled)
12
10
User-defined
Non-reinforced
8
6
4
Reinforced
2
0
12
10
Pre-designed
8
6
4
2
0
12
10
Random
8
6
4
2
0
0
1
2
3
4
5
6
7
8
9 10 11
21
Results: Recall Rates
Recall Rate
(all)
Recall rate
(reinforced)
Recall rate
(not-reinforced)
User-defined
79%
85%
72%
Pre-designed
55%
68%
41%
Stock
25%
38%
11%
22
Next-day Test (errors)
12
10
Association Errors
Partial Gesture Errors
8
User-defined
6
4
2
0
12
10
Pre-designed
8
6
4
2
0
12
10
Random
8
6
4
2
0
0
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22
23
Next-day Test (errors)
12
10
Association Errors
Partial Gesture Errors
8
User-defined
6
4
2
0
12
10
Pre-designed
8
6
4
2
0
12
10
Random
8
6
4
2
0
0
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22
24
Results: Subjective
User-defined
• required less concentration
• easier to articulate
• more fun
• Preferred overall (17 of 18)
25
Are User-defined Gestures Better?
• Better memorability (44% more gestures!)
• Effort to create not an issue
• Personalizable
– Adapt to your culture/experience
– Adapt to your body
– Adapt to your task
However…
26
Are User-defined Gestures Better?
• Recognizer limitations
– Still not trivial to design by example
– Different for the user ≠ different for the machine
• Collaborative awareness and transferability
– Gestures are also indirect communication
• Shared gestures are valuable
– Will a standard set of gestures evolve?
27
Recommendations
• Allow end-users to design their own gestures
(if recognizer allows)
• To design memorable gestures, the association
between gesture and action is most important
• Avoid stock gestures if you can
28
User-defined gestures are easier to
remember than pre-designed gestures
29
THANKS!
http://sachi.cs.st-andrews.ac.uk
http://nacenta.com
http://pokristensson.com/
@SACHI_research
@miguelnacenta
@pokristensson
30
Experimental Interface
Experimenter
control area
Participant
Area
31
Conclusions of the Study
• User-defined best memorability
– (44% more gestures remembered!)
•
•
•
•
User-defined also preferred
Stock really bad
Differences due to association errors
Partial gesture errors more or less constant
32
Results
User-defined
Reinforcement Test (recalled)
Training/Learning Time (log10(Time)
12
10
1.42, 26.9s
8
6
4
2
0
Pre-designed
12
1.47, 29.8s
10
8
6
4
2
0
1.49, 31.2s
Random
12
10
8
6
4
2
0
0
1
2
3
4
5
6
7
8
9 10 11
1.3
1.4
1.5
1.6
33
Random Pre-designed User-defined
Results
12
10
8
6
4
2
0
Reinforcement Test
(recalled)
Next-day Test (recalled)
12
10
8
6
4
2
0
NonReinforced
reinforced
12
10
8
6
4
2
0
12
10
8
6
4
2
0
12
10
8
6
4
2
0
12
10
8
6
4
2
0
12
10
8
6
4
2
0
12
10
8
6
4
2
0
12
10
8
6
4
2
0
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10 11
Next-day Test (errors)
Training/Learning Time
(log10(Time)
1.42,
26.9s
Association Errors
Partial Gesture Errors
1.47,
29.8s
1.49,
31.2s
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
1.3
1.4
1.5
1.6
34
Experimental Interface
Experimenter
Reference Video
Experimenter
control area
Correctness
Buttons
Participant
video area
Action presentation area
-Before Action
-Action Description
-Post-action
Participant
Area
Action area
35
Condition: Stock (random) Gestures
Condition
Pre-designed
User-Defined
Stock
Application
Image Editing
Web Browsing
Word Processing
Actions set
Image Editing
Web Browsing
Word Processing
Gesture set
Image Editing
The participant’s
Web Browsing
36
Subjective Measures
• Likert-scale questions (after each condition)
–
–
–
–
How much did you have to concentrate?
How hard were the gestures to remember?
How hard were the gestures to articulate?
How boring were the gestures to create?
• Ranking
–
–
–
–
Difficulty of learning
Difficulty of remembering
Time spent learning/remembering
How fun were the gestures to remember
• Overall Ranking (preferred condition)
37
Results
Random
Pre-designed User-defined
Next-day Test (recalled)
12
10
Next-day Test (errors)
12
0
10
8
6
4
2
0
12
12
10
8
6
4
10
8
6
4
2
0
2
0
12
10
8
12
10
8
6
4
2
0
6
4
2
0
8
6
4
2
Non-reinforced
Reinforced
0 1 2 3 4 5 6 7 8 9 10 11
Association Errors
Partial Gesture Errors
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
38