A case study in UI design and
evaluation for computer
security
Rob Reeder
January 30, 2008
CMU Usable Privacy and Security
Laboratory
http://cups.cs.cmu.edu/
Memogate: A user interface
scandal !!
• CMU Usable Privacy and Security Laboratory •
2
Overview

Task domain: Windows XP file permissions

Design of two user interfaces: native XP
interface, Salmon

Evaluation: Which interface was better?

Analysis: Why was one better?
• CMU Usable Privacy and Security Laboratory •
3
Part 1: File permissions in
Windows XP

File permissions task: Allow authorized
users access to resources, deny
unauthorized users access to resources

Resources: Files and folders

Users: People with accounts on the
system

Access: 13 types, such as Read Data,
Write Data, Execute, Delete
• CMU Usable Privacy and Security Laboratory •
4
Challenges for file permissions UI
design

Maybe thousands of users – impossible to
set permissions individually for each

Thirteen access types – hard for a person
to remember them all
• CMU Usable Privacy and Security Laboratory •
5
Grouping to handle users

Administrators

Power Users

Everyone

Admin-defined
• CMU Usable Privacy and Security Laboratory •
6
A problematic user grouping
Xu
Ari
Bill
Miguel
Cindy
Group A
• CMU Usable Privacy and Security Laboratory •
Yasir
Zack
Group B
7
Precedence rules

No setting = Deny by default

Allow > No setting

Deny > Allow

(> means “takes precedence over”)
• CMU Usable Privacy and Security Laboratory •
8
Grouping to handle access types
Execute
9
Moral

Setting file permissions is quite
complicated

But a good interface design can help!
• CMU Usable Privacy and Security Laboratory •
10
The XP file permissions interface
• CMU Usable Privacy and Security Laboratory •
11
The Salmon interface
ProjectF
12
Expandable Grid
13
Example task: Wesley

Initial state
• Wesley allowed READ & WRITE from a group

Final state
• Wesley allowed READ, denied WRITE

What needs to be done
• Deny Wesley WRITE
• CMU Usable Privacy and Security Laboratory •
14
What’s so hard?

Conceptually: Nothing!

Pragmatically:
• User doesn’t know initial group membership
• Not clear what changes need to be made
• Checking work is hard
• CMU Usable Privacy and Security Laboratory •
15
Learning Wesley’s initial
permissions
1
2
Click “Effective
Permissions”
Click “Advanced”
3
4
View Wesley’s
Effective Permissions
Select Wesley
16
Learning Wesley’s group membership
Bring up Computer
5
Management interface
6
Click on
“Users”
Read
Wesley’s
group
membership
9
7
Doubleclick
Wesley
Click
“Member
Of”
8
17
Changing Wesley’s permissions
10
11
Deny
Write
Click
“Add…”
Click
“Apply”
12
18
Checking work
13
14
Click “Effective
Permissions”
Click “Advanced”
15
16
View Wesley’s
Effective Permissions
Select Wesley
19
XP file permissions interface:
Poor
20
Part 2: Common security UI design
problems

Poor feedback

Ambiguous labels

Violation of conventions

Hidden options

Omission errors
• CMU Usable Privacy and Security Laboratory •
21
Problem #1: Poor feedback
1
2
Click “Effective
Permissions”
Click “Advanced”
3
4
View Wesley’s
Effective Permissions
Select Wesley
22
Salmon: immediate feedback
ProjectF
23
Grid: consolidated feedback
24
Problem #2: Labels (1/3)
Full Control
Modify
Read & Execute
Read
Write
Special Permissions
25
Problem #2: Labels (2/3)
Full Control
Traverse Folder/Execute File
List Folder/Read Data
Read Attributes
Read Extended Attributes
Create Files/Write Data
Create Folders/Append Data
Write Attributes
Write Extended Attributes
Delete
Read Permissions
Change Permissions
Take Ownership
26
Salmon: clearer labels
ProjectF
27
Grid: fewer, clearer labels
• CMU Usable Privacy and Security Laboratory •
28
Problem #3: Violating interface conventions
29
Problem #3: Violating interface conventions
30
Salmon: better checkboxes
ProjectF
31
Grid: direct manipulation
32
Problem #4: Hidden options
• CMU Usable Privacy and Security Laboratory •
33
Problem #4: Hidden options
1
2
Click “Advanced”
Double-click entry
3
Click “Delete”
checkbox
• CMU Usable Privacy and Security Laboratory •
34
Salmon: All options visible
ProjectF
35
Grid: Even more visibility
36
Problem #5: Omission errors
37
Salmon: Feedback helps prevent
omission errors
ProjectF
38
Grid: No omission errors
39
FLOCK: Summary of design
problems

Feedback poor

Labels ambiguous

Omission error potential

Convention violation

Keeping options visible
• CMU Usable Privacy and Security Laboratory •
40
Part 3: Evaluation of XP and
Salmon

Conducted laboratory-based user studies

Formative and summative studies for
Salmon

I’ll focus on summative evaluation
• CMU Usable Privacy and Security Laboratory •
41
Advice for user studies

Know what you’re measuring!

Maintain internal validity

Maintain external validity
• CMU Usable Privacy and Security Laboratory •
42
Common usable security metrics

Accuracy – with what probability do users
correctly complete tasks?

Speed – how quickly can users complete
tasks?

Security – how difficult is it for an
attacker to break into the system?

Etc. – satisfaction, learnability,
memorability
• CMU Usable Privacy and Security Laboratory •
43
Measure the right things!

Speed is often useless without accuracy
(e.g., setting file permissions)

Accuracy may be useless without security
(e.g., easy-to-remember passwords)
• CMU Usable Privacy and Security Laboratory •
44
Measurement instruments

Speed – Easy; use a stopwatch, time users

Accuracy – Harder; need unambiguous
definitions of “success” and “failure”

Security – Very hard; may require serious
math, or lots of hackers
• CMU Usable Privacy and Security Laboratory •
45
Internal validity

Internal validity: Making sure your results
are due to the effect you are testing

Manipulate one variable (in our case, the
interface, XP or Salmon)

Control or randomize other variables
• Use same experimenter
• Experimenter reads directions from a script
• Tasks presented in same text to all users
• Assign tasks in different order for each user
• Assign users randomly to one condition or other
46
External validity

External validity: Making sure your experiment
can be generalized to the real world

Choose real tasks
• Sources of real tasks:
 Web forums
 Surveys
 Your own experience

Choose real participants
• We were testing novice or occasional filepermissions users with technical backgrounds
(so CMU students & staff fit the bill)
• CMU Usable Privacy and Security Laboratory •
47
User study compared Salmon to XP

Seven permissions-setting tasks, I’ll
discuss two:
• Wesley
• Jack

Metrics for comparison:
• Accuracy (measured as deviations in users’
final permission bits from correct permission
bits)
• Speed (time to task completion)
• Not security – left that to Microsoft
• CMU Usable Privacy and Security Laboratory •
48
Study design

Between-participants comparison of
interfaces

12 participants per interface, 24 total

Participants were technical staff and
students at Carnegie Mellon University

Participants were novice or occasional file
permissions users
• CMU Usable Privacy and Security Laboratory •
49
Wesley and Jack tasks
Wesley task

Jack task
Initial state
• Wesley allowed

Final state
• Wesley allowed

What needs to be
done
• Deny Wesley

READ & WRITE

WRITE, & ADMINISTRATE
READ, denied
WRITE

Initial state
• Jack allowed READ,
Final state
• Jack allowed READ,
denied WRITE &
ADMINISTRATE
WRITE
• CMU Usable Privacy and Security Laboratory •
What needs to be
done
• Deny Jack WRITE &
ADMINISTRATE
50
Salmon outperformed XP in
accuracy
Percent successful completions by
task
300%
100
100
50
25
0
Wesley task
25
XP
58
Salmon
75
improvement
Salmon
83
XP
Percent of Users Who
Correctly Completed
Tasks
43% improvement
Jack task
Task Name
• CMU Usable Privacy and Security Laboratory •
51
Salmon outperformed XP in
accuracy
p = 0.09
83
100
p < 0.0001
50
25
0
Wesley task
25
XP
58
Salmon
75
Salmon
100
XP
Percent of Users Who
Correctly Completed
Tasks
Percent successful completions by
task
Jack task
Task Name
• CMU Usable Privacy and Security Laboratory •
52
Salmon did not sacrifice speed
Speed (Time-to-Task-Completion)
Results
250
173
150
Salmon
100
50
XP
183
Salmon
208
200
XP
Time (seconds)
208
Successful XP users
Successful Salmon users
0
Wesley task
Jack task
Task Name
• CMU Usable Privacy and Security Laboratory •
53
Salmon did not sacrifice speed
250
Speed (Time-to-Task-Completion)
Results
p = 0.35
p = 0.20
173
150
Salmon
100
50
XP
183
Salmon
208
200
XP
Time (seconds)
208
Successful XP users
Successful Salmon users
0
Wesley task
Jack task
Task Name
• CMU Usable Privacy and Security Laboratory •
54
Part 4: Analysis

What led Salmon users to better
performance?
• CMU Usable Privacy and Security Laboratory •
55
How users spent their time Wesley
45
40
35
All XPFP users
30
25
20
15
All Salmon users
Successful XPFP users
Successful Salmon users
wo
Ch
rk
ec
k
gr
ou
Le
ps
ar
n
in
te
rfa
Co
ce
ns
ul
tH
Se
el
tp
p
er
m
iss
M
io
an
ns
ag
e
wi
Re
nd
m
ow
ov
s
e
fro
m
AC
L
to
Ad
d
Ch
ec
k
AC
L
an
Pl
ta
sk
10
5
0
Re
ad
Time (seconds)
Average behavior time per participant for Wesley task
Behavior
• CMU Usable Privacy and Security Laboratory •
56
Where Salmon did better - Wesley
45
40
35
All XPFP users
30
25
20
15
All Salmon users
Successful XPFP users
Successful Salmon users
wo
Ch
rk
ec
k
gr
ou
Le
ps
ar
n
in
te
rfa
Co
ce
ns
ul
tH
Se
el
tp
p
er
m
iss
M
io
an
ns
ag
e
wi
Re
nd
m
ow
ov
s
e
fro
m
AC
L
to
Ad
d
Ch
ec
k
AC
L
an
Pl
ta
sk
10
5
0
Re
ad
Time (seconds)
Average behavior time per participant for Wesley task
Behavior
• CMU Usable Privacy and Security Laboratory •
57
Where XP did better - Wesley
45
40
35
All XPFP users
30
25
20
15
All Salmon users
Successful XPFP users
Successful Salmon users
wo
Ch
rk
ec
k
gr
ou
Le
ps
ar
n
in
te
rfa
Co
ce
ns
ul
tH
Se
el
tp
p
er
m
iss
M
io
an
ns
ag
e
wi
Re
nd
m
ow
ov
s
e
fro
m
AC
L
to
Ad
d
Ch
ec
k
AC
L
an
Pl
ta
sk
10
5
0
Re
ad
Time (seconds)
Average behavior time per participant for Wesley task
Behavior
• CMU Usable Privacy and Security Laboratory •
58
How users spent their time - Jack
Average behavior time per participant for Jack task
60
All XPFP users
40
All Salmon users
30
Successful XPFP users
20
Successful Salmon users
10
wo
Ch
rk
ec
k
gr
ou
Le
ps
ar
n
in
te
rfa
Co
ce
ns
ul
tH
Se
el
tp
p
er
m
iss
M
io
an
ns
ag
e
wi
Re
nd
m
ow
ov
s
e
fro
m
AC
L
to
Ad
d
Ch
ec
k
AC
L
an
Pl
ta
sk
0
Re
ad
Time (seconds)
50
Behavior
• CMU Usable Privacy and Security Laboratory •
59
Where Salmon did better - Jack
Average behavior time per participant for Jack task
60
All XPFP users
40
All Salmon users
30
Successful XPFP users
20
Successful Salmon users
10
wo
Ch
rk
ec
k
gr
ou
Le
ps
ar
n
in
te
rfa
Co
ce
ns
ul
tH
Se
el
tp
p
er
m
iss
M
io
an
ns
ag
e
wi
Re
nd
m
ow
ov
s
e
fro
m
AC
L
to
Ad
d
Ch
ec
k
AC
L
an
Pl
ta
sk
0
Re
ad
Time (seconds)
50
Behavior
• CMU Usable Privacy and Security Laboratory •
60
Where XP did better - Jack
Average behavior time per participant for Jack task
60
All XPFP users
40
All Salmon users
30
Successful XPFP users
20
Successful Salmon users
10
wo
Ch
rk
ec
k
gr
ou
Le
ps
ar
n
in
te
rfa
Co
ce
ns
ul
tH
Se
el
tp
p
er
m
iss
M
io
an
ns
ag
e
wi
Re
nd
m
ow
ov
s
e
fro
m
AC
L
to
Ad
d
Ch
ec
k
AC
L
an
Pl
ta
sk
0
Re
ad
Time (seconds)
50
Behavior
• CMU Usable Privacy and Security Laboratory •
61
Common UI problems summary

Feedback poor

Labels ambiguous

Omission error potential

Convention violation

Keeping options visible
• CMU Usable Privacy and Security Laboratory •
62
User interface evaluation
summary

Know what you’re measuring

Internal validity: Control your experiment

External validity: Make your experiment
realistic
• CMU Usable Privacy and Security Laboratory •
63
Rob Reeder
[email protected]
CMU Usable Privacy and Security
Laboratory
http://cups.cs.cmu.edu/
• CMU Usable Privacy and Security Laboratory •
64
x-x-x-x-x-x-x END x-x-x-x-x-x-x
• CMU Usable Privacy and Security Laboratory •
65
Results
Small-size
Grid
Windows
Accuracy
Task type
Speed
Small view simple
View simple
View complex
Change simple
Change complex
Compare groups
Conflict simple
Conflict complex
Memogate simulation
Precedence rule test
89%
56%
Grid
Windows
Small view complex
0.20
0.40
0.60
0.80
Windows
Small change simple
0.20
0.40
0.60
0.80
Windows
Small change complex
0.00
0.20
0.40
0.60
1
0.80
0
0
Windows
0.00
0.20
0.40
0.60
0.80
0.00
0.20
0.40
0.60
0.80
50
0.20
0.40
0.60
0.80
Windows
50
0.00
0.20
0.40
0.60
0.80
0.00
0.20
0.40
0.60
0.80
100%
94%
50
89%
94%
Windows
0.40
0.60
0.80
100
1.00
100
50
Grid
Windows
0.20
0.40
0.00
Grid
Windows
100
20s
66s
0
50
100
150
0.20
0.40
• CMU Usable Privacy and Security Laboratory •
100
0.60
0.80
Grid
Windows
0.20
0.40
0.60
0.80
Windows
0.20
0.40
0.60
50
0.80
0.00
0.20
0.40
0.60
0.80
Windows
0.20
0.40
0.60
0.80
150
42s
118s
0.00
0.20
0.40
0.60
0.00
0.80
100
6%
94%
78%
0.60
0.80
1.00
Grid
Windows
50
100
50
150
100
100s
143s
150
111s
126s
Grid
Windows
50
100
1
0
150
73s
104s
Grid
Windows
50
100
150
52s
Insufficient data
Grid
1
Windows
Large Memogate
0
50
100
150
105s
116s
Grid
1
Windows
Large Precedence
0
1.00
78%
78%
Windows
0.40
150
Large conflict simple
1.00
Grid
0.20
Windows
Windows
0
1.00
Windows
1
Grid
Large conflict complex
Grid
1
150
Grid
0
Grid
Windows
Large Memogate
100
1
1.00
Grid
1
50s
42s
1
1.00
67%
83%
72%
61%
100
39s
67s
Large compare groups
Windows
1
Windows
Large change complex
0
Grid
Grid
Large precedence
150
0
1
1.00
67%
17%
1
Grid
Large change simple
1.00
100
100
50
Large compare groups
0.00
150
Windows
50
0.80
0
1
Large conflict complex
Grid
0
100
39%
Windows
Small precedence
1
1.00
Grid
1
0.00
150
55s
103s
Windows
50
0.60
Large conflict simple
Grid
0
0.80
1
0.00
150
39s
103s
29s
Insufficient data
1
0.60
Large change complex
150
100
0.40
Small Memogate
1.00
Grid
0.20
0
1
1.00
Windows
1
0.00
1.00
89%
0%
Grid
Small precedence
Windows
1
42s
61s
1
Large view complex
Windows
Small conflict complex
Windows
1
0.00
150
Grid
100
Windows
Grid
Small conflict simple
0
Grid
Grid
Small Memogate
0.20
1
Large change simple
70s
Insufficient data
Small conflict complex
0.00
Windows
100
1
1.00
67%
61%
1
Grid
Windows
0
1.00
89%
83%
Windows
1
0.00
150
Small compare groups
Grid
Small conflict simple
100
35s
55s
30s
52s
Grid
Large view complex
Grid
1
Small compare groups
1
50
Large view simple
61%
56%
1
Small change complex
0
Grid
50
1
1.00
61%
0%
Windows
Small change simple
1.00
89%
94%
Grid
1
Grid
Small view complex
1
Speed
Large view simple
29s
64s
1
1.00
94%
17%
Grid
1
0.00
Accuracy
Small view simple
1
0.00
Large-size
50
100
150
71s
115s
Grid
1
Windows
0
50
100
150
66
Measure the right thing!
Keystroke dynamics analysis poses a real threat to any
computer user. Hackers can easily determine a user’s
password by recording the sounds of the users'
keystrokes. We address this issue by introducing a new
typing method we call "Babel Type", in which users hit
random keys when asked to type in their passwords. We
have built a prototype and tested it on 100 monkeys
with typewriters. We discovered that our method
reduces the keystroke attack by 100%. This approach
could potentially eliminate all risks associated with
keystroke dynamics and increase user confidence. It
remains an open question, however, how to let these
random passwords authenticate the users.
• CMU Usable Privacy and Security Laboratory •
68