Strategies for evaluating information visualization tools:
Multi-dimensional In-depth Long-term Case studies (MILCs)
(BELIV 2006 at AVI Conference)
http://dl.acm.org/citation.cfm?id=1168158
Ben Shneiderman &
Catherine Plaisant
University of Maryland
Human-Computer Interaction Lab
Test of Time Award from BELIV Workshop 2016
http://beliv.cs.univie.ac.at/
Multi-Dimensional In-Depth Long-term Case Studies (MILCs)
• Domain experts (+ our research team)
• Exploring their own data
• To produce their research results
• Using new interactive tools
• Time-timited structured evaluation (4-8 weeks)
• Observations, interviews, surveys, etc.
• Replicated case studies = Hypothesis testing
(Shneiderman & Plaisant, BELIV 2006)
TimeSearcher: PhD 2003
2 case studies
• Genome biology: DNA expression
• Nucleotide sequences
Hierarchical Clustering Explorer: TVCG 2006
• 3 case studies
• Biologist, statistician, meteorologist
• Survey results from 57 users
• 94 citations
SocialAction: CHI 2008
• 4 case studies
• Business consultant, medical librarian
• Journalist, terrorism analyst
• Pushback from reviewers
• But we succeed
• 162 citations
Treeversity PHD: 2013
• 13 case studies
Treeversity PHD: 2013
Multi-Dimensional In-Depth Long-term Case Studies (MILCs)
Other PhD students who conducted MILCs
• David Wang
• Krist Wongsuphasawat
• Cody Dunne
• Megan Monroe
• Sana Malik
• Fan Du
The New ABCs of Research (Oxford, 2016)
Guide for
Junior researchers
Manifesto for
Senior researchers
Academic administrators
Business leaders
Funding agencies
www.cs.umd.edu/hcil/newabcs
Research Methods
Method
Definition
Pros
Controlled
Experiments
Defined task, hypotheses, alter IVs
(treatments), measure DVs, run stats,
prove hypoths
Scientific, reproducible Biases can undermine
results
Usability
Testing
Evaluate users efficiency & subjective
Fast, insight to user
reaction in use of tool for specific tasks, thought processes
list of changes
Small user sample,
short-term, limited
scope
Case Studies
(Design Study
Methodology)
Longer term, with domain experts,
design tool, write up
reflections/experiences
Takes a long time,
questionable
generalizability
insight to user thought
processes, guidance to
refine tool
Cons