www.site.uottawa.ca/~elsaddik
SEG 3210
User Interface Design & Implementation
Prof. Dr.-Ing. Abdulmotaleb El Saddik
University of Ottawa (SITE 5-037)
(613) 562-5800 x 6277
elsaddik @ site.uottawa.ca
abed @ mcrlab.uottawa.ca
http://www.site.uottawa.ca/~elsaddik/
1
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
Unit B: UI Evaluation
2
Unit B-Introduction
(c) elsaddik
1.
Objectives of User Interface Evaluation
2.
Where Evaluation Fits in the Development Process
3.
A Preliminary Case Study: Hotel Reservations
4.
Overview of Interface Evaluation Methods
5.
Details: Heuristic Evaluation
6.
Malfunction Analysis
7.
Details: Videotaped Evaluation
8.
Details: Experiments
9.
Details: Usability Engineering
10.
Details: Cognitive Walkthroughs
11.
Key Points to Review
www.site.uottawa.ca/~elsaddik
1. Objectives of User Interface Evaluation
Key objective of both UI design and evaluation:
•Minimize malfunctions
Key reason for focusing on evaluation:
• Without it, the designer would be working “blindfold”
• Designers wouldn’t really know whether they are solving
customer’s problems in the most productive way
3
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
1. Objectives of User Interface Evaluation
Questions answered by various evaluation techniques:
1. What is the user’s real task?
•
Prevent later malfunctions
•
•
Present and work with a UI
•
•
by doing evaluation as part of requirements analysis
to help formulate the requirements
Inappropriate tasks/requirements are a major source of malfunctions
2. What problems do or might users experience with the UI?
•
Directly find malfunctions
3. Which of several alternative UI’s is better?
•
Pick the version that leads to fewer malfunctions
4. Has the UI met usability targets?
•
Ensure that malfunction counts are sufficiently low
5. Does the UI conform to standards?
•
4
Unit B-Introduction
(c) elsaddik
Leverage of collective wisdom to reduce malfunctions
www.site.uottawa.ca/~elsaddik
1. Objectives of User Interface Evaluation
But, in order for evaluation to give feedback to
designers...
...we must understand why a malfunction occurs
Malfunction analysis:
• Determine why a malfunction occurs
• Determine how to eliminate malfunctions
We will discuss this while working through this unit.
5
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
2. Where Evaluation Fits in the Development Process
Throughout the lifecycle!
• During rough sketching or prototyping
• During iterative design
The more evaluation the better
• Especially when users are involved
Formative evaluation:
• When designing and maintaining software that we are
developing
Summative evaluation:
• When judging a finished product developed by someone else
6
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
3. A Preliminary Case Study: Hotel Reservations
UI Evaluation performed for Forte Travelodge
Performed in a special usability lab
Aims:
• Identify and eliminate malfunctions
• Hence make system easier to use
• Avoid business difficulties caused by these malfunctions
• Develop improved training material and documentation
• Avert potential malfunctions by teaching users how to avoid them
Setup of IBM usability lab:
• Resembles TV studio
•
•
•
•
Microphones and video equipment
One way mirror
Technicians, observers sit on one side
Users sit on other side in realistic environment
• User environment resembles reception desk
7
Unit B-Introduction
(c) elsaddik
• Non-threatening
www.site.uottawa.ca/~elsaddik
3. A Preliminary Case Study: Hotel Reservations
Aspects of system to be evaluated:
• How quickly can a booking be made?
• (while operator is on telephone)
• Is each screen productive to use?
• Are help and error messages effective?
• Can non-computer-literate operators use the system?
• Is complexity minimized?
• Is training and documentation effective?
8
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
3. A Preliminary Case Study: Hotel Reservations
Procedure:
• 15 common task scenarios developed:
• Among others: basic registration, cancellation, request for specific
room, extension of existing stay etc.
• Four days of testing with multiple users performing various sets
of tasks
• Users were told evaluation is of system, not them
• All actions were recorded
• Debriefing sessions held
• Videos then analyzed for malfunctions
• 62 identified
• Priorities:
•
•
•
•
9
Unit B-Introduction
(c) elsaddik
Navigation speed needs improvement
Screen titles and formats need tuning
Hard to refer to documentation
Physical difficulties with telephone headsets and furniture
www.site.uottawa.ca/~elsaddik
3. A Preliminary Case Study: Hotel Reservations
Results:
• Higher productivity of booking staff
• tasks completed more quickly
• guest requirements better met
• Training costs kept low
• Morale kept high
• More customers booked by phone
• 14500
10
Unit B-Introduction
(c) elsaddik
27000 per week
www.site.uottawa.ca/~elsaddik
4. Overview of Interface Evaluation Methods
• Three types of methods
• Passive evaluation
• Active evaluation
• Predictive evaluation / usability inspections
• All types of methods useful for optimal results
• Used in parallel
• All attempt to prevent malfunctions
• Before trying methods, do pilot studies first
11
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
4. Passive evaluation
•
•
•
Usage of software is monitored
Performed while prototyping, in alpha test and later
Does not actively seek malfunctions
•
•
only finds them when they happen to occur infrequent (but
possibly severe) malfunctions may not be found
Generally requires realistic use of a system
•
Users become frustrated with malfunctions
Gathering Information:
a) Problem report monitoring:
•
•
12
Unit B-Introduction
(c) elsaddik
Users should have an easy way to register their frustration /
suggestions
Best if integrated with software
www.site.uottawa.ca/~elsaddik
4. Passive evaluation: Gathering Information
b) Automatic software logs
•
Can gather much data about usage
•
•
•
•
•
•
Privacy is a concern
System must be designed for testability (DFT)
Logs can be taken of:
•
•
13
Unit B-Introduction
(c) elsaddik
command frequency
error frequency and pre-error patterns
undone operations (a sign of malfunctions)
just keystrokes, mouse clicks
full details of interaction
• The latter make accurate playback easier
www.site.uottawa.ca/~elsaddik
4. Passive evaluation: Gathering Information
c) Questionnaires / surveys
•
•
•
Unit B-Introduction
(c) elsaddik
Proper statistical means are needed to analyze results
Gathers subjective data about importance of malfunction
•
•
•
•
•
•
14
Useful to obtain statistical data from large numbers of users
automated logs omit importance
less frequent malfunctions may be more important
users can prioritize needed improvements
Limit on number of questions
Very hard to phrase questions well
Questions can be closed- or open-ended
www.site.uottawa.ca/~elsaddik
4. Active evaluation
•
•
Actively study specific activities performed by users
Performed when prototyping and later
Gathering Information:
d) Experiments & usability engineering
•
Prove hypotheses about measurable attributes of one or more
UI’s
•
•
•
•
•
e.g. speed/learning/accuracy/frustration
In usability engineering test against preset targets
Can be expensive
Knowledge of statistics needed
Hard to control for all variables
e) Observation sessions
•
•
•
15
Unit B-Introduction
(c) elsaddik
Also called ‘interpretive evaluation’
Simple observation or cooperative evaluation
Described in detail later
www.site.uottawa.ca/~elsaddik
4. Predictive evaluation
•
•
Studies of system by experts rather than users
Performed when UI is specified and later
•
•
•
useful even before prototype developed
Can eliminate many malfunctions before users ever see
software
Also called ‘usability inspections’
Gathering Information:
f) Heuristic evaluation
•
•
•
Based on a UI design principle document
Analyze whether each guideline is adhered to in the context of
the task and users
Can also look at adherence to standards
g) Cognitive walkthroughs
•
16
Unit B-Introduction
(c) elsaddik
Step-by-step analysis of:
•
•
•
steps in task being performed
goals users form to perform these tasks
how system leads user through tasks
www.site.uottawa.ca/~elsaddik
Summary of evaluation techniques
Technique
When to use
a) Problem reporting
Always
b) Automatic logs
In any moderately complex system and
whenever there are large numbers and
commands
c) Questionnaires
Whenever there are large number of
users
d) Experiments &
Usability Engineering
In special cases where it is hard to
choose between alternatives, or when
fine tuning
e) Observation sessions
Almost always, especially when user has
to interact with a client while using the
system
f) Heuristic evaluation
Always
g) Cognitive Walkthrough When usability must be optimized
17
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
Comparison of key questions to evaluation techniques
A
B
Prob Log
What is task?
What are current
malfunctions?
Does UI conform to
standards?
++: Very good techniques
+ : OK technique
0 : possible technique
18
Unit B-Introduction
(c) elsaddik
D
E
F
G
Exp Obs Heu Wlk
++
+
+
Which UI is better?
Has UI met targets?
C
Q?
+
+
++
+
++
++
++
++
++
++
+
+
+
++
+
0
+
++
++
www.site.uottawa.ca/~elsaddik
5. Details: Heuristic Evaluation
• A type of predictive evaluation
• Use HCI experts as reviewers instead of users
• Benefits of predictive evaluation:
• The experts know what problems to look for
• Can be done before system is built
• Experts give prescriptive feedback
• Important points about predictive evaluation:
• Reviewers should be independent of designers
• Reviewers should have experience in both the application domain
and HCI
• Include several experts to avoid bias
• Experts must know classes of users
• Beware: Novices can do some very bizarre things that experts
may not anticipate
19
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
5. Details: Heuristic Evaluation
• Planning for heuristic evaluation
• Based on UI guidelines (heuristics about what is best)
• Multiple passes needed
• one pass to look for each kind of problem
• passes to follow different routes through screens and dialogues (i.e. different
tasks)
•
•
•
•
1-2 hour sessions is good
1 expert evaluator finds only 33% of problems
5 evaluators needed to find 75% of problems
15 more to find about 99%
•
•
•
•
•
•
•
•
•
Use simple and natural dialogue
Speak the user’s language
Minimize memory load
Be consistent
Provide feedback
Provide clearly marked exits
Provide shortcuts
Provide good error messages
Prevent errors
• Example heuristics for heuristic evaluation (Many more to be covered
later in course)
20
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
6. Malfunction Analysis
A disciplined approach to analyzing malfunctions
•
Provides feedback into the redesign process
1. Play protocol, searching for malfunctions
2. Answer four distinct questions:
•
Q1. How is the malfunction manifested?
•
•
Q2. At what stage in the interaction is it occurring?
•
•
Goal forming, action decision, action execution, interpretation of
results
Q3. At what level of the user interface is it occurring?
•
•
What do you notice and who noticed it?
Physical element level to task level
Q4. Why is it occurring?
•
What is its root cause
3. List and prioritize possible cures
21
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
Q1. How is the malfunction manifested?
a) Malfunctions detected by the system (easiest to
detect)
• omission of an argument
• incorrect date format
Cure:
• Better prompts, consistency, visible examples, more forgiving
of alternatives
b) Malfunctions detected by the user during
operation
•
•
•
•
taking wrong path in menu hierarchy
not finding required help
not being able to perform a certain action
not being able to tell which state system is in
Cure:
22
Unit B-Introduction
(c) elsaddik
•
Improve functionality, feedback, clarity, simplicity
www.site.uottawa.ca/~elsaddik
Q1. How is the malfunction manifested?
c) Malfunctions undetected (until later)
• output produced is wrong due to wrong inputs
• unnecessary work performed
Cure:
• Improve feedback indicating consequences of input; simplify
d) Inefficiencies
•
•
•
•
•
excessive response time
excessive think time
unnecessarily long command sequences
unnecessary repetitions
complex operations that require use of reference
Cure:
• Simplify, speed system up
23
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
Q2. What Stage in the Interaction the Malfunction Occur?
a) When the user decides on next goal (Forms an
intent to do inappropriate thing)
• decides to empty a field because user thinks it is unimportant
(when it is important)
• decides to charge default exchange rate (when should obtain
current exchange rate)
Cure:
• Lead user through task better; better feedback; better training
b) When the user specifies the action (Action does
not match the goal)
• deletes the record instead of emptying a field
• charge reciprocal of exchange rate
Cure:
24
Unit B-Introduction
(c) elsaddik
• Improve clarity, feedback, prompts, conceptual model
www.site.uottawa.ca/~elsaddik
Q2. What Stage in the Interaction the Malfunction Occur?
c) When the system executes the action
• Defects in functionality
Cure:
• Fix functionality in normal way
d) When the user interprets the resulting system
state
•
•
•
•
thinks bank account has been debited when it has not
thinks system has ‘hung’ when it has not
thinks some data must be entered when it is the default
cannot understand resulting error message
Cure:
• Better feedback, better conceptual model
25
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
Q3. At Which Level Does the Malfunction Occur?
a) Task level (Task and goals not supported)
• What the user wants to do cannot be done by the system
• Functionality is not provided
Cure:
• Add functionality
b) Conceptual level (User has wrong mental model;
does not understand intended conceptual model)
• thinks that money is being deducted from bank account when it
is being charged to a credit card
• thinks that dragging a file to the desktop means they are no
longer on the disk
• thinks that dragging a disk to the trash can icon deletes disk
contents
Cure:
• make conceptual model clearer; improve metaphors
26
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
Q3. At Which Level Does the Malfunction Occur?
c) Interaction style level (system wide problem)
•
•
•
•
does not know how to pull down a menu
scrolls a page instead of a line
goes to next screen instead of scrolling
retypes command after an error instead of editing it
Cure:
• make operation of the interface more intuitive and consistent
d) Interaction element level (specific detail
inappropriate)
• selects wrong button because label is misinterpreted
• specifies invalid command syntax
• specifies wrong code for option
Cure:
27
Unit B-Introduction
(c) elsaddik
• More attention to details of the interface, simplification
www.site.uottawa.ca/~elsaddik
Q3. At Which Level Does the Malfunction Occur?
e) Physical element level (Physical execution
incorrect)
•
•
•
•
presses wrong key accidentally
clicks on wrong pixel in image
out-types machine (actions lost)
types ahead when system is computing; keystrokes later
applied to wrong action
Cure:
• Defenses to protect user from consequences; better hardware
design; fix bugs in code
28
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
Q4. Why Does the Malfunction Occur?
Lack of (on the part of the user):
• Motivation:
• Poor job satisfaction
• Attention:
• User is pre-occupied with other things.
• Input information processing:
• No feedback provided to tell user what is going on
• or cues provided by the system are not recognized
• or cues are misinterpreted
Cures: Clearer, more consistent feedback
• Discrimination:
• user is unable to tell certain things apart
• e.g. red/green colour discrimination
• e.g. two icons that are similar
Cures: Improved expression of information
29
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
Q4. Why Does the Malfunction Occur?
• Physical coordination:
• e.g. wrong item selected because of difficulty positioning cursor
with mouse.
Cures: Alternate interaction mechanisms, better feedback
• Recall:
• User did not remember command , syntax etc.
Cures: Better mnemonics, online help, quick lookup
mechanisms, command completion
• Knowledge / lack of learning:
• User does not have business or software knowledge to make right
choice.
30
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
Q4. Why Does the Malfunction Occur?
Learning difficulties that cause malfunctions:
• Learning is difficult
• users get frustrated
• learning takes time; can be hard to apply
• Learners make ad-hoc interpretations
• they may not recognize their problem
• they may falsely think they have a problem
• Learners generalize from what they know
• they assume computers work like manual methods
• they assume consistency
• Learners have trouble following directions
• they often ignore them even if they see them
• they do not easily understand them
31
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
Q4. Why Does the Malfunction Occur?
• Problems and features interact
• they do not see that one problem can cause another
• Prerequisites and side-effects confuse learners
• Help facilities do not always help
• they do not know what to ask for
• too much detail is often provided
• Other causes of malfunctions:
•
•
•
•
•
32
Unit B-Introduction
(c) elsaddik
Excessive resource demands
External events (e.g. noise)
Misleading or inadequate training
Unrealistic task definitions
Intrinsic human variability
www.site.uottawa.ca/~elsaddik
7. Details: Videotaped Evaluation
A software engineer studies users who are actively
using the user interface
• To observe what problems they have
• Rather than to measure numbers
• The sessions are videotaped
• Can be done in user’s environment
Activities of the user:
• Performs pre-defined tasks
• With or without detailed instructions on how to perform them
• Preferably talks to herself as if alone in a room
• Yields ‘think-aloud protocol’
• This process is called ‘co-operative’ evaluation when the software
engineering and user talk to each other
33
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
7. Details: Videotaped Evaluation
The importance of video:
• Without it, ‘you see what you want to see’
• You interpret what you see based on your mental model
• In the ‘heat of the moment’ you miss many things
• Minor details (e.g. body language) captured
• You can repeatedly analyze, looking for different problems
Tips for using video:
• Several cameras are useful
• Software is available to help analyse video by dividing into
segments and labeling the segments
• Evaluation can be time consuming so plan it carefully
34
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
Steps for videotaped evaluation
1. Select 6 to 8 representative users per user class
• E.g. client, salesperson, manager, accounts receivable
2. Invite them to individual sessions
• Sessions should last 30-90 minutes
• Schedule 4-6 per day
3. If system involves user's clients in the interaction:
1. Have users bring important clients
2. or have staff pretend to be clients
4. Select facilitators/observers and notetakers
5. Prepare tasks:
• Select the most commonly used tasks plus a few less important
tasks
• Write task instructions for users
• Estimate the time it will take to complete each task plus extra time
for discussion
6. Prepare notebook or form for organizing notes
35
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
Steps for videotaped evaluation
7. Set up and test equipment
1. Hardware on which to run system
2. Audio or video recorder
3. Software logs
8. Do a dry run (pilot study)!
9. At the Start of an Observation Session
•
explain:
•
•
•
•
•
•
•
•
•
Sign informed consent form:
•
36
Unit B-Introduction
(c) elsaddik
nature of project
anticipated user contributions
why user's views are important
focus is on evaluating the user interface, not evaluating the user
all notes, logs, etc., are confidential
user can withdraw at any time
usage of devices
relax!
very important
www.site.uottawa.ca/~elsaddik
Steps for videotaped evaluation
37
Unit B-Introduction
(c) elsaddik
10. Start user verbalizing as they perform each task (thinking
aloud)
•
•
For co-operative evaluation, software engineer also verbalizes
Appropriate questions to be posed by the observing software
engineer:
Question
Malfunction if
What do you want to do?
They do not know; the system
cannot do what they want
What do you think would
happen if ...?
They do not know; they give wrong
answer.
What do you think the
system has done?
They do not know; they give wrong
answer.
What do you think is this
information telling you?
They do not know; they give wrong
answer.
Why did the system do that?
They do not know; they give wrong
answer.
What were you expecting to
happen?
They had no expectation; they were
expecting something else.
www.site.uottawa.ca/~elsaddik
Steps for videotaped evaluation
11. Hold a wrap-up interview (de-briefing)
1. What were the most significant problems?
2. What was most difficult to learn?
3. Etc.
12. Analyze the videotape to find malfunctions
Lab exercise:
•
38
Unit B-Introduction
(c) elsaddik
Videotaped evaluation of a software product
www.site.uottawa.ca/~elsaddik
8. Details: Experiments (Details in Experimentation)
1. Pick a set of subjects (users)
1. A good mix to avoid biases
2. A sufficient number to get statistical significance (avoid random
happenings effect results)
2. Pick variables to test
•
Independent: Manipulated to produce different conditions
•
•
•
•
Should not have too many
They should not affect each other too much
Make sure there are no hidden variables
Dependent: Measured value affected by independent
3. Develop a hypothesis
•
•
•
39
Unit B-Introduction
(c) elsaddik
A prediction of the outcome
Aim of experiment is to show this is correct
E.g. Some change in an independent variable causes some
change in a dependent variable
www.site.uottawa.ca/~elsaddik
8. Details: Experiments (Details in Experimentation)
4. Design experiments to test hypotheses
•
Create a null (inverse) hypothesis
•
•
•
e. a change in independent variable causes no change in dependent
variable
Disprove null hypothesis!
Experiment design is difficult.
5. Conduct experiments
6. Statistically analyze results to draw conclusions
•
•
e.g. using ‘t-tests’
conclusions will be correct within a margin of error 19 times out of
20
7. Decide what action to take based on conclusions
40
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
Two major types of UI experimentation
• Traditional
• Micro level
• (e.g. testing which colour is best for a certain icon)
• Only done when usage will be very high or you are a university
researcher!
• Usability engineering
• Tests significant part of system
• Relaxes ‘scientific constraints’
• Because we cannot control all variables
• Used to prove hypotheses that certain usability goals have
been met
• More later
You should understand experimentation
• Much UI research is experimental
• You have to interpret and apply others’ results
41
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
Example Experiment: Text Selection Schemes
Early GUI research at Xerox on the Star
Workstation
•
•
Traditional experiments
Results were used to develop Macintosh
Goal of study:
•
Evaluate how to select text using the mouse
Steps:
1. Subjects
•
Six groups of four
•
42
Unit B-Introduction
(c) elsaddik
In each group, only two are experienced in mouse usage
www.site.uottawa.ca/~elsaddik
Example Experiment: Text Selection Schemes
2. Variables
•
Independent:
•
Selection schemes:
•
•
•
•
•
6 strategically chosen patterns involving
--> Which mouse button (if any) could be double/triple/quad clicked
to select character/word/sentence
--> Which mouse button could be dragged through text
--> Which mouse button could adjust the start/end of a selection
Dependent
•
•
Selection time
Selection errors
3. Hypothesis
•
43
Unit B-Introduction
(c) elsaddik
Some scheme is better than all others
www.site.uottawa.ca/~elsaddik
Example Experiment: Text Selection Schemes
4. Detailed experiment design
•
•
•
•
Null hypothesis: No difference in schemes
Assign a selection scheme to each group
Train the group in their scheme
Measure task time and errors
•
•
Each subject repeated 6 times
A total of 24 tests per scheme
5. Conduct Experiment
6. Analysis
•
F-test used - scheme F found to be significantly better
•
•
Point and draw through with left mouse
Adjust with middle mouse
7. Action
44
Unit B-Introduction
(c) elsaddik
•
•
Try another combination similar to scheme F
Left mouse can be double-clicked
www.site.uottawa.ca/~elsaddik
Questions to ask when reviewing experiments
Not all published experiments are done well!
•
•
•
•
Were users adequately prepared?
Were tasks complex enough to allow adequate evaluation?
Did the task become boring to the users?
Although effects are found to be statistically significant, does
that matter?
• Maybe not if a particular task is rarely performed
• Are there any other possible interpretations?
• Maybe users have learned to do better at task B because they did
task A first!
• Are dependent variables consistent?
• e.g. users may prefer slower method
• Can results be generalized?
• Maybe selection results also apply to graphics, maybe not
45
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
9. Details: Usability Engineering
“A process whereby the usability of a product is specified
quantitatively, and in advance. Then as the product is built it
can be demonstrated that it does or does not reach the
required levels of usability”
Partly engineering:
• Design-evaluate-redesign
Partly science:
• Experimentation methodology
• Although not with full controls
46
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
Usability Engineering Steps
1. Pick ‘benchmark’ tasks
•
2.
3.
4.
5.
Simple tasks that can be repeated and performance measured
Pick usability metrics
Set planned levels of usability
Design initial interface using usability guidelines
Analyze impact of design(s) using experiments
1. i.e. a new batch of users is measured running the benchmark
tasks
6. If goals are achieved, stop
7. Incorporate user derived feedback in design
8. Go back to step 5
Major problem with usability engineering:
•
47
Unit B-Introduction
(c) elsaddik
Benchmark tasks are rarely performed in a truly natural
environment
www.site.uottawa.ca/~elsaddik
Some suitable usability metrics
•
•
•
•
•
•
•
•
•
•
•
Time to complete task
Percentage of task completed per unit time
Ratio of successes to failures
Percentage of time spent dealing with errors
Percentage of competing products that have better speed
measures than our product
Number of repetitions of failed commands
Percentage of available commands used
Number of times user had to undo an action
Number of unnoticed errors
Number of times user did not use the expected method to
accomplish the task
Think time required for task
• i.e. ignoring system response time
• a good UI should lead user through system with minimal ‘cognitive
load’
48
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
10. Details: Cognitive Walkthroughs
A form of predictive evaluation
Detailed reviews based on psychological theory,
focusing on:
• Goals a new user must form to execute a task
• How well the system leads the user to form those goals
• i.e. how well the system supports the user
• The method is highly structured
• Forms are provided to guide the evaluator
• More time consuming that ordinary heuristic evaluation
• Less time consuming than experiments
49
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
Cognitive Walkthrough Steps
1. Choose a task to evaluate
2. Describe the task exactly
a) First describe the task in one sentence
• Use simple language
• The wording should be from a first-time user’s point of view
e.g. Record a newly-received item in inventory.
b) Describe the initial state of the system
e.g. Main menu is displayed
c) List the atomic actions needed to correctly perform the task, e.g.
1. Click on ‘add to inventory’ in the menu.
2. If you don’t know the part number, hit ‘return’ to perform look up the part
number, then go to action 4.
3. Type the part number into the ‘part number’ field
4. Press tab
5. Type the number of items in the ‘Number’ field
6. Hit <return> or click on ‘Add’.
7. If the system prints out a bar-code sticker, affix it to the new item.
d) Describe classes of users who may perform the task
50
Unit B-Introduction
(c) elsaddik
e.g. Receiver - knows about inventory, but not yet about the system
www.site.uottawa.ca/~elsaddik
Cognitive Walkthrough Steps
e) Describe the ‘Goal Structure’ (or task structure) users would
likely have in their minds before starting the task
• High-level and system independent
• Indent subgoals/subtasks
• Note if there are actions for which the user has no goals, the
system must stimulate the user to think of these goals by the time
they must perform the task
• If different classes of user may have different goal structures, list
these too.
e.g.
Record a received item in inventory
Started the inventory program
Enter the item
51
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
Cognitive Walkthrough Steps
3.
I.
For each action specified in step 2c, do the following (I to
IV):
Write down the goal structure
... that the user would need to have in order to perform the
action correctly
e.g. For action 4
•
•
•
•
•
52
Unit B-Introduction
(c) elsaddik
Record a received item in inventory
Record the number of items
Press tab
Enter the number
Cause the system to process the transaction
www.site.uottawa.ca/~elsaddik
Cognitive Walkthrough Steps
II.
•
•
•
Verify that the user will have the correct goal structure
given their initial goals
given the system’s response to the previous action
Estimate the percentage of users who might have each of
the following possible problems:
•
Failure to add goals
•
•
Failure to drop goals
•
•
Unit B-Introduction
(c) elsaddik
e.g. The user may have a goal to, notify the person who ordered the
parts
This would not b needed if the system performs this automatically
Addition of spurious goals
•
53
e.g. For action 2
The system must make it clearly visible that pressing return with
nothing entered will invoke a lookup mechanism
e.g. There may be a field marked ‘Description’
However this only needs to be filled in if the type of item is not in the
database
www.site.uottawa.ca/~elsaddik
Cognitive Walkthrough Steps
54
Unit B-Introduction
(c) elsaddik
•
No-progress impasse
• e.g. After adding an item, the system might just clear the screen ready
for another entry.
• The user may think the transaction failed (i.e. goal not achieved)
•
Premature loss of goals
• e.g. The user enters an item and hits “return”
• A message ‘transaction accepted’ is printed (meaning the transaction
has been started)
• The user powers off the computer thinking the goal is reached
• The system never got around to printing the label
www.site.uottawa.ca/~elsaddik
Cognitive Walkthrough Steps
III. Verify that the actions match the goals
Possible problems:
•
Correct action doesn’t match goal
•
•
•
•
e.g. User wants to delete an item that was stolen.
Correct action is to select ‘add to inventory’ and specify a negative
number
System does not help user match the goal to the action
Incorrect actions match goals
•
•
e.g. User wants to add a new type of item to inventory (for which no
items have yet been received)
Upon seeing ‘add to inventory’, user selects this incorrect menu item
IV. Verify that the user can physically perform the action
Possible problems:
•
Physical difficulties
•
•
55
Unit B-Introduction
(c) elsaddik
e.g. recognizing an icon, holding down shift-ctrl-alt-a to perform a
command
Time-outs
•
i.e. running out of time – the system gives up
www.site.uottawa.ca/~elsaddik
11. Key Points to Review
• Objective of evaluation: Minimize malfunctions
• Key questions:
• What is real task? Problems? Which is better?
• Met targets? Is it standard?
•
•
•
•
•
Evaluate throughout lifecycle!
Formative vs. summative evaluation
Pilot studies important
Use all techniques in a balanced approach
Use cost-benefit analysis to see if an expensive technique will
pay off
• Passive methods
• Problem reporting
• Software logging
• Questionnaires/surveys
56
Unit B-Introduction
(c) elsaddik
www.site.uottawa.ca/~elsaddik
11. Key Points to Review
• Active methods
• Traditional experiments
•
•
•
•
•
Investigate a single UI element
Pick subjects
Independent and dependent variables
Hypotheses
Experimental designs:
• independent subject
• matched subject (control for differences among subjects)
• repeated measures (reuse subjects)
• Usability Engineering
•
•
•
•
Test realistic ‘benchmark’ task
Set targets for usability metrics
Evaluate-redesign-evaluate until targets met
Partly engineering, partly science
• Observation sessions (Videotaped Evaluation)
57
Unit B-Introduction
(c) elsaddik
• Study active use on realistic tasks
• Think-aloud protocol on video
• Co-Operative Evaluation involves dialogue
www.site.uottawa.ca/~elsaddik
11. Key Points to Review
• Predictive evaluation: involve experts
• Heuristic Evaluation: based on guidelines
• Cognitive Walkthroughs: goals and actions
• Describe task, actions, users, goal structure
• For each action, verify that users:
• ... add and drop goals as needed
• ... don’t add unneeded goals
• ... can tell when a goal is reached
• ... don’t drop needed goals
• ... can see what action to take
• ... are not mislead into taking wrong action
• ... have no physical difficulties with action
• Malfunction analysis
• How manifested?
• Detected by: system, user
• Undetected, inefficiencies
• What stage in interaction? When user...
• Decides on goal?
• Executes action?
Specifies action?
Interprets result?
• Level? (physical to task)
• Why occurring?
58
Unit B-Introduction
(c) elsaddik
• Users lacks: e.g. motivation, input, recall