Comp 15 - Usability and
Human Factors
Unit 6c - Electronic Health
Records and Usability
This material was developed by Columbia University, funded by the Department of Health and Human Services, Office of the National Coordinator
for Health Information Technology under Award Number 1U24OC000003.
Usability Challenges in
Healthcare
Complex needs,
time pressure
Vary from setting
to setting,
specialty
Clinician mobility,
primary focus on
patient (should
be)
Legal, ethical,
errors, highstakes
Confidentiality
makes in situ
observation,
testing difficult
Component 15/Unit 6b
50 specialties;
other professions
• (OT, PT, pharmacists,
respiratory therapists
etc.)
Multiple
standards,
evidence-based
Medicine
Cost of change
(time, vendor,
consensus, cost)
Health IT Workforce Curriculum
Version 2.0/Spring 2011
Team-based work
Hard to get
clinician feedback
Interruptions,
time pressure,
institution policies
2
More issues
• Clauses mean
institution liable
Learned
intermediary
• Clinician
responsible even
if no choice or
access to guts of
software
Hold harmless
Component 15/Unit 6b
• 25% medication
errors (2006)
involved computer
technology
• 82% of these
CPOE/data entry
Patient Safety
Health IT Workforce Curriculum
Version 2.0/Spring 2011
3
Basic Minimums
Usability = Efficiency of use + usefulness + ease of use?
For EHRs, we want to:
• Minimize likelihood of user error
• Provide cognitive support – guide interaction to foster good work,
no errors
Avoid:
• Errors of commission: e.g. wrong patient chart
• Errors of omission: e.g. fail to notice abnormal result
• Failure to complete task (due to interruption and no handling of
interruption)
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
4
Evidence-based Usability/Cognitive
Support
• Usability is not just screen or software
design
– Affected by workflow, time pressure, physical
space layout, lighting, policies for use, and
even user experience during implementation
• Design needs to be evidence based, and
evidence is available
» Karsh, 2010
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
5
Evidence-based Usability/Cognitive
Support (cont.)
Fact:
• Healthcare is
complex, training
will be required
Myths:
• No training needed with
good usability
From Karsh, 2010
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
6
Evidence-based Usability/Cognitive
Support (cont.)
Fact:
• Users not trained in
science of usability and
cognition
• What they want may be
wrong, mis-specified, or
inarticulate
Myth:
• User-centered design = give
users what they want
From Karsh,
2010
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
7
Evidence-based Usability/Cognitive
Support (cont.)
Myth:
• Health IT should
integrate into workflow
Fact:
• Healthcare workflow is
emergent
• what is needed is more
flexible availability of data
From Karsh, 2010
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
8
Evidence-based Usability/Cognitive
Support (cont.)
• Usability is not a fixed target
• Don’t think we have the answers
• Ongoing research is needed
» Karsh, 2010
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
9
Evidence-based Design Exists: AHRQ
Resources
•
Electronic Health Record Usability:
–
–
–
•
Interface Design Considerations Recommended actions to support development of an objective
EHR usability evidence base, formative policies
Evaluation and Use Case Framework literature and best practices regarding the usability of EHRs,
use cases for primary care IT evaluation
Vendor Practices and Perspectives: current usability processes, practices, and perspectives of
certified EHR vendors
Available at: http://healthit.ahrq.gov
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
10
Usability Testing
“In routine handling, the
Sorrento feels responsive
in corners, with nicely
weighted, quick steering…
The gated zigzag shifter is
awkward to use”
“It posted a commendable
speed through our
avoidance maneuver.
Avoidance maneuver,
max. spd: 51.5 mph. 0 to
60 mph: 7.6 sec”
Two Voices in Consumer Reports Car Reviews
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
11
Subjective v. Objective
From Friedman, 2010
Subjectivist/Qualitative (“art
criticism”)
• Not everything of importance can be
quantified
• Differences of opinion are okay
• The value is in the “thick description”
• Rigorous methods exist (one is formal
criticism)
Component 15/Unit 6b
Objectivist/Quantitative
• Believable knowledge derives from
measurement of attributes that are
inherent in objects
• All observers should agree on
measurement results
(intersubjectivity)
• On what result is “better” (polarity)
• Accuracy
• Response time
• Time to identify / Time spent
searching
• Eye gaze
Health IT Workforce Curriculum
Version 2.0/Spring 2011
12
Heuristic Evaluation
Expert(s) evaluate system according to heuristics
(Norman)
10 main axes:
•
•
•
•
•
•
•
•
•
•
1. Visibility of system status
2. Match between system and the real world
3. User control and freedom
4. Consistency and standards
5. Error prevention
6. Recognition rather than recall
7. Flexibility and efficiency of use
8. Aesthetic and minimalist design
9. Help users recognize, diagnose, and recover from errors
10. Help and documentation
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
13
Focus Groups
Use for formative evaluation
•
•
•
•
•
Find out information designers can use
What should the system do?
What are users worst problems?
What is their workflow?
How do they do the task now?
Use for summative evaluation (after system is built)
Method: get typical users to view and discuss
system and related ideas
• Group similar users together
• Don’t put supervisors with staff (people should feel free to speak)
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
14
Focus Groups (cont.)
Prepare open-ended questions, not yes/no
Provide fixed time, food
Compensate users
• Clinicians are busy and highly paid
• Compensate appropriately, e.g. $100/hour
Get permission in writing
• Institutional Review Board
Define privacy policy
• e.g. participants will be anonymous in any publications, talks
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
15
Focus Groups (cont.)
Record meeting (two digital or tape recorders);
obtain permission of subjects
Transcribe and code thematically
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
16
Usability Testing
Thinkaloud, in-lab
• Subject (typical user) uses system in quiet office,
software captures video of their screen actions
and voice (and face, if desired)
• User is told to think aloud while using software for
typical tasks
• Video is coded and analyzed for themes, action
patterns, problems, time for tasks…
• Morae software is a common tool
– http://www.morae.com
– Requires webcam with sound, can do remote testing
• Video can be edited for administrators, clients,
decision-makers, programmers
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
17
Field Usability Testing
Examine user’s interactions in their normal workplace
setting
Field observation
• Important method to establish conditions of work, workflow
Answer questions such as
• What are time constraints, interruptions, noise, information sharing with
colleagues, information flow, information sources, needs, team members?
Observation, logfile analysis, user interviews, and perhaps
remote
• Morae testing will be the main methods
Problems not detected in a laboratory test likely to come to
light after deployment
Monitor usage closely soon after deployment
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
18
Cognitive Walkthrough
• Method of inspecting software for
problems
• Have goal, subgoal, actions taken, system
response, potential problems
• Classify problems:
1. Cosmetic
• Need not be
fixed unless
time available
Component 15/Unit 6b
2. Minor usability
• Low priority for
fix
3. Major usability
• High priority
Health IT Workforce Curriculum
Version 2.0/Spring 2011
4. Usability
catastrophe
• Imperative to fix
19
Scenes from a Walkthrough
Context: Patient Presents with a medical problem to a Family
Physician’s Office
Task: Diagnostic Reasoning
Goal Structure and Action Sequence
Goal: Represent chief complaint
Chief Complaint: Patient complains that she has been feeling
pretty tired for the past 6 months
Subgoal 1: Characterize Patient’s Observation of Tiredness
Action 1: Open Top-Level Category General Condition
System Response: Displays Findings Organized by Category
Potential Problem: System has sluggish response
Subgoal 2: Enter Finding Fatigue
Subgoal 3: Locate Finding
Action 2: Open Category/Select Finding "Fatigue"
Subgoal 4: Describe Severity/Quality
Subgoal 5: Translate quality "Pretty Tired" into Quantity
indicating Severity
Action 3: Enter severity: Three +++
Action 4: Selects Quality: Lack of energy
Subgoal 6: Indicate Chronology
Action 5: Selects chronology duration of 6 months
System Response: Displays chronology in "years“
Potential Problem: “years” is not appropriate unit for 6 months
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
20
The special case of CPOE
• Poor CPOE design can facilitate medical
error
– e.g. overdose, wrong patient given drug
• Quantitative and qualitative studies show
flaws leading to errors
• Many adverse drug events due to poor
interface
• Heavy cognitive demands
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
21
Koppel:
EHR Interface flaws
•
•
•
•
•
•
•
Can’t clearly identify the patient
Can’t view all meds on single screen
Log-in/log-out failures
Extra steps required to ‘activate’ orders
Automatic cancellation of pre-surgical orders
Downtime delays
Orders near midnight interpreted as
‘tomorrow’
• Cumbersome interface – charting difficult
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
22
CPOE-related Usability Problems
Avoid:
•
•
•
•
•
•
•
•
•
Deep navigational structures requiring multiple clicks
Too close screen elements -> errors in selections
Same color for data entry fields and others
Long drop-down menus requiring scrolling
Documentation templates different from clinician
cognitive model of ordering
Use of string sensitive data fields for abbreviations
Excessive alerts, alerts which do not appear at
appropriate time (when clinician would look for that
information)
Excess screen density, or too many screens
Obscure hierarchies of orders or order sets
Khajouei, Jaspers 2008
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
23
CPOE Design Recommendations
View complete medication record on one screen, avoid scrolling/other
screens/fragmentation
Avoid subtle differences in layout, forms, labels for large functional differences
Explicit time indications
Map interface to ordering workflow
Maximum 3 layers of screens
Use consistent terms, organize elements into logical groups, separated by
space, alignment
Distinguish active and passive elements
Consistent, sparing color
Khajouei, Jaspers 2008
Technology Effects
Effects of
technology
• How performance
changes when one
uses a system
• Longer-term effects
of technology on
cognition, even when
the technology is no
longer being used
Effects with
technology
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
25
Information Gathering Strategies
Hypothesisdriven
strategy:
• Requests for information guided by
clinician’s hypothesis independent
of the screen displays
Screen-driven:
• Guided by the ordered sequence of
information on the computer screen
With
experience:
Component 15/Unit 6b
• Novice changed from hypothesisdriven strategies to screen-driven
Health IT Workforce Curriculum
Version 2.0/Spring 2011
Patel et al. 2000
26
Study Results
• Paper-based records
– Narrative form, with connected and linked text and
sentences
• EMRs
– More info on patient’s past medical history, lifestyle, and
primary diagnosis
– Information entered in point form, not linked in narrative
– Followed structure and sequence of system
– Time course not adequately captured
• Post EMR paper-based record
– Closely resembled EMR in structure and format
– No connecting narrative
– Limited info on time course
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
Patel et al. 2000
27
Web 2.0 and modern
approaches
‘Web 2.0’ is a change in internet approaches
• Give the user more control
Both philosophical approaches & technical
approaches
Social networking applications
• e.g. Facebook
Crowd sourcing:
• Obtain information or judgment from a large group of users
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
28
Web 2.0 and EHRs
• Facilitate user control, a better user
experience, new forms of interactive
information display, and social networking
• Address problems of clinician
collaboration, optimal design of EHRs,
flexibility to meet rapid change, and other
problems
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
29
Conclusion
• EHR usability is a complex area in which we do not yet
have standards, but best practices are being intensively
studied
• Usability is one of the most important factors affecting
adoption, satisfaction, and optimal use of EHRs
• Usability should be an important factor in selection and
deployment of a system
• In the next two years much research will be done in this
area; it is important to keep abreast of developments
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
30
References
1.
2.
3.
4.
5.
6.
7.
8.
9.
Patel V, Kushniruk A. Cognitive and usability engineering methods for the evaluation of
clinical information systems. Journal of Biomedical Informatics. 2004;37(1):56-76.
Patel VL, Kushniruk, A.W., Yang, S., & Yale, J.F. . Impact of a computerized patient record
system on medical data collection, organization and reasoning. J of the American Medical
Informatics Association. 2000;7(6):569-85.
Senathirajah Y, Bakken, S., editor. A User-Configurable EHR using Web 2.0 Approaches.
AMIA Spring 2008; 2008; Phoenix, AZ: AMIA.
Shabot M. Ten commandments for implementing clinical information systems. Proc (Bayl
Univ Med Cent). 2004;17(3):265-9.
Staggers N, Mills ME. Nurse-Computer Interaction: Staff Performance Outcomes. Nursing
Research. 1994;43(3):144-50.
Zhu X, Gold SA, Lai A, Hripcsak G, Cimino J, editors. Using Timeline Displays to Improve
Medication Reconciliation. 2009 International Conference on eHealth, Telemedicine, and
Social Medicine; 2009.
Belden J. EHR usability: an illustrated guide. AHRQ/NIST EHR Usability Conference; 2010;
Washington DC: National Institute of Standards and Technology.
Karsh B-TB. Health IT Design and Usability: Myths and Realities. AHRQ/NIST EHR
Usability Conference; 2010; Washington DC.
Friedman C. A Range of Methods for Assessing Usability. AHRQ/NIST EHR Usability
Conference; 2010; Washington DC.
9. Koppel et al. 2005. JAMA 293(10): 1197-1203
10. Khajouei, R., Jaspers, MWM. CPOE System Design Aspects and Their
11. Qualitative Effect on Usability. eHealth Beyond the Horizon – Get IT There. S.K. Andersen et al.
(Eds). IOS Press, 2008. 309-314
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
31
Useful Resources
•
One Insurance company’s case descriptions of EHRs causing errors:
–
•
Electronic Health Records: Recognizing and Managing the Risks "Claims Rx"
–
•
http://Mscui.net
Joint Commission: official ‘Do Not Use’ abbreviations list:
–
•
www.usability.gov/pdfs/guidelines.html
http://Ui-patterns.com
Open source standards for medical information display created by National Health Service
(Britain) through a rigorous process
–
•
http://healthit.ahrq.gov
Research‐Based Web Design & Usability Guidelines
–
–
•
http://www.useit.com/papers/heuristic/heuristic_list.html
AHRQ usability reports
–
•
http://www.himss.org/ASP/topics_FocusDynamic.asp?faid=358
Nielsen’s Ten Usability Heuristics
–
•
http://www.norcalmutual.com/publications/claimsrx/oct_09.pdf
Presentations on usability by HIMSS members
–
•
http://hcrenewal.blogspot.com/2010/07/norcal-mutual-insurance-company.html (Scot Silverstein's blog)
www.jointcommission.org/NR/rdonlyres/2329F8F5-6Ec5-4E21-B932-54B2B7D53F00/0/dnu_list.pdf
Site discussing EMR usability, author is Dr. Jeffrey Belden
–
http://www.toomanyclicks.com
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
32
Copyright urls
• Early car
– http://commons.wikimedia.org/wiki/File:Splendid_early_car_at_Petersfie
ld_Station_-_geograph.org.uk_-_1249493.jpg
• Rotary phone
– http://commons.wikimedia.org/wiki/File:WE302dialphone.jpg
• Intermediate cell phone
– http://en.wikipedia.org/wiki/File:DynaTAC8000X.jpg (attribute)
• 3 iphones - no attribution needed
– http://commons.wikimedia.org/wiki/File:Iphone2g3g3gson.jpg
Component 15/Unit 6b
Health IT Workforce Curriculum
Version 2.0/Spring 2011
33