Chapter 12/13:
Evaluation/Decide
Framework
Question 1
It’s part of a design process!
• Every organization that
builds things has some
design process.
• Good idea to have it be
a conscious one, that
you can follow
consistently…
• And change consciously
if it’s not working!
For ID - Why Evaluate?
• Why: to check that users can use the product and
that they like it.
• Designers need to check that they understand
users’ requirements.
• Avoid wasting money
Question 2
What to Evaluate
• What: a conceptual model, early prototypes of a
new system and later, more complete prototypes.
Where to Evaluate
• Where: in natural and laboratory settings.
When to Evaluate
• When: throughout design; finished products can be
evaluated to collect information to inform new
products.
The language of evaluation
•
•
•
•
•
Analytical evaluation
Controlled experiment
Formative evaluation
Heuristic evaluation
Predictive evaluation
• Summative evaluation
• Usability laboratory
• User testing
Start Question 3
Evaluation approaches
• Usability testing
• Field studies
• Analytical evaluation
• Combining approaches
Start Question 4!
Characteristics of approaches
Usability
testing
Field
studies
Analytical
Users
do task
natural
not involved
Location
controlled
natural
anywhere
When
prototype
early
prototype
Data
quantitative
qualitative
problems
Feed back
measures &
errors
descriptions
problems
…More for Question 3
Evaluation approaches and
methods
Method
Usability
testing
Field
studies
Observing
x
x
Asking
users
x
x
Asking
experts
Testing
Modeling
x
Analytical
x
x
x
End Questions 3, 4
Six evaluation case studies
• Evaluating early design ideas for a mobile device for rural
nurses in India.
• Evaluating cell phones for different markets.
• Evaluating affective issues: challenge and engagement in a
collaborative immersive game.
• Improving a design: the Hutch World patient support system.
• Multiple methods help ensure good usability: the Olympic
messaging system (OMS).
• Evaluating a new kind of interaction: an ambient system.
Team Extra Credit
Book Exercise in Chapter 12 (Interaction Design Book –
See p 450 in the 3rd Ed). Due with Milestone 4.
DECIDE: a framework to
guide evaluation
•
•
•
•
•
•
Determine the goals.
Explore the questions.
Choose the evaluation approach and methods.
Identify the practical issues.
Decide how to deal with the ethical issues.
Evaluate, analyze, interpret and present the data.
Determine the goals
• What are the high-level goals of the evaluation?
• Who wants it and why?
• Some examples of goals:
 Check to ensure that the final interface is consistent.
 Investigate how technology affects working practices.
 Improve the usability of an existing product
.
Explore the questions
• All evaluations need goals & questions to guide them.
• What questions might you ask about the design of a cell phone?
Choose the evaluation approach
& methods
• The evaluation approach influences the methods used,
and in turn, how data is collected, analyzed and
presented.
• E.g. field studies typically:
Involve observation and interviews.
Do not involve controlled tests in a laboratory.
Produce qualitative data.
Identify practical issues
For example, how to:
•
•
•
•
•
Select users
Stay on budget
Stay on schedule
Find evaluators
Select equipment
Question 5
Decide about ethical issues
• Develop an informed consent form (see example on
pp 465-7 of the ID book!)
 Do one for class on Friday!
• Participants have a right to:
- Know the goals of the study;
- Know what will happen to the findings;
- Privacy of personal information;
- Leave when they wish;
- Be treated politely.
Question 6
Evaluate, interpret & present
data
• The approach and methods used influence how data is
evaluated, interpreted and presented.
• The following need to be considered:
- Reliability: can the study be replicated?
-
Validity: is it measuring what you expected?
Biases: is the process creating biases?
Scope: can the findings be generalized?
Ecological validity: is the environment influencing the findings?
i.e. Hawthorne effect.
Question 7