Quality Improvement
Measuring Quality
This material (Comp12_Unit10) was developed by Johns Hopkins University, funded by the Department of Health and Human
Services, Office of the National Coordinator for Health Information Technology under Award Number IU24OC000013.
Measuring Quality
Learning Objectives
• Understand the basic concept of variation.
• Explain the attributes of an effective reporting
system.
• Examine the importance of having standardized
and structured health information so that you
can use those data to make valid reports.
• Discuss how HIT can facilitate data collection
and reporting for improving quality and patient
safety.
Health IT Workforce Curriculum
Version 3.0/Spring 2012
Quality Improvement
Measuring Quality
2
Why Measure?
• We have limited ability to accurately selfassess.
• Neither the act of measuring performance
nor the resulting data accomplish anything
by themselves.
• So why measure?
To understand your system
Health IT Workforce Curriculum
Version 3.0/Spring 2012
Quality Improvement
Measuring Quality
3
Understanding Variation
• We are surrounded by variation.
• We make decisions based on how we
interpret that variation.
• To make good decisions, we need to be
able to understand whether variations are
actual trends or the result of random
differences.
Health IT Workforce Curriculum
Version 3.0/Spring 2012
Quality Improvement
Measuring Quality
4
Different Interpretations
of Variation
Health IT Workforce Curriculum
Version 3.0/Spring 2012
Quality Improvement
Measuring Quality
5
Types of Variation
Common Cause Variation
• Variation happens on a
regular basis
• Variation is predictable
• Variation happens within
historical parameters
• No change in the system
or our knowledge of the
system (Deming, 1975)
Health IT Workforce Curriculum
Version 3.0/Spring 2012
Special Cause Variation
• New phenomena happens
within the system
• Variation is unpredictable
• Variation happens outside
historical parameters
• Evidence of some change
in the system or our
knowledge of the system
Quality Improvement
Measuring Quality
6
Tools to Understand Variation
The RUN chart
• Focuses on the time order that data are collected
• Can be applied when traditional methods to determine statistical
significance are not useful.
• Used for improvement activities to:
– Display data to make process performance visible
– Determine if changes tested resulted in improvement
– Determine if we are holding the gains made by our improvement
– Allow for a temporal (analytic) view of data versus a static
(enumerative) view (Perla, Pronovost & Murray 2011)
Health IT Workforce Curriculum
Version 3.0/Spring 2012
Quality Improvement
Measuring Quality
7
Graphical Representation of Data
p <0.001
Health IT Workforce Curriculum
Version 3.0/Spring 2012
Quality Improvement
Measuring Quality
8
Run Charts
Health IT Workforce Curriculum
Version 3.0/Spring 2012
Quality Improvement
Measuring Quality
9
Statistical Rules to Identify
Non-Random Signals
in Run Charts
• Three probability-based rules below are used to
objectively analyze a run chart for evidence of
nonrandom patterns in the data based on an α error of
p<0.05.
– Shift
– Trend
– Run
• One subjective rule to recognize the importance of the
visual display of the data in a run chart
– Astronomical point (Perla, Pronovost & Murray, 2011)
Health IT Workforce Curriculum
Version 3.0/Spring 2012
Quality Improvement
Measuring Quality
10
Statistical Rules to Identify
Non-Random Signals
in Run Charts
• Shift: Six or more consecutive points either all
above or all below the median.
• Trend: Five or more consecutive points all going
up or all going down.
• Run: a series of points in a row on one side of
the median.
• Astronomical point: point that is obviously, even
blatantly, different from the rest of the points.
(Perla, Pronovost & Murray, 2011)
Health IT Workforce Curriculum
Version 3.0/Spring 2012
Quality Improvement
Measuring Quality
11
Statistical Rules to Identify
Non-Random Signals
in Run Charts
Median
Health IT Workforce Curriculum
Version 3.0/Spring 2012
Quality Improvement
Measuring Quality
12
Statistical Rules to Identify
Non-Random Signals
in Run Charts
Health IT Workforce Curriculum
Version 3.0/Spring 2012
Quality Improvement
Measuring Quality
13
Statistical Rules to Identify
Non-Random Signals
in Run Charts
Health IT Workforce Curriculum
Version 3.0/Spring 2012
Quality Improvement
Measuring Quality
14
Statistical Rules to Identify
Non-Random Signals
in Run Charts
Health IT Workforce Curriculum
Version 3.0/Spring 2012
Quality Improvement
Measuring Quality
15
Measure
• PROCESS MEASURE: Are
we doing what we must to get
the improvement we seek?
• OUTCOME MEASURE Are
we getting what we expect?
• BALANCING MEASURE Are
we causing new problems in
other parts of the system?
Health IT Workforce Curriculum
Version 3.0/Spring 2012
Quality Improvement
Measuring Quality
16
Role of HIT in Data Collection
• EHR as data source
– Structured data vs. free text
– What is the source of the data?
• Reporting structure
– ‘Canned’ reports
– Individualized reports
Health IT Workforce Curriculum
Version 3.0/Spring 2012
Quality Improvement
Measuring Quality
17
How to Get More Valid Data
• Structure data entry or data collection forms clarify who what when where and how
• Pilot test entry to see if staff understand
• Train and evaluate competency
• Evaluate data quality (look at data)
– Missing data, outliers, repeat values
• Ask if the consumer of the data believes it is
valid and useful
Health IT Workforce Curriculum
Version 3.0/Spring 2012
Quality Improvement
Measuring Quality
18
Measuring Quality
Summary
• Measurement is an essential component of quality
improvement.
• Understanding variation provides a context to interpret
the data we collect by allowing us to determine if there
has been a change introduced to the system or it is
random variation we are detecting.
• Run charts are a simple statistical methodology to
interpret the data collected.
• HIT has an important role to play in any measurement
strategy.
Health IT Workforce Curriculum
Version 3.0/Spring 2012
Quality Improvement
Measuring Quality
19
Measuring Quality
References
References
•
Berenholtz SM, Needham DM, Lubomski LH, Goeschel CA, Pronovost PJ. Improving the quality of quality
improvement projects. Jt Comm J Qual Patient Saf 2010, in press
•
Deming, W E (1975) On probability as a basis for action, The American Statistician, 29(4), pp146–152
•
Needham DM, Sinopoli DJ, Dinglas VD, Berenhottz SM, Korupolu R, Watson SR, Lubomski, Goeschel C,
Pronovost PJ. Improving data quality control in quality improvement projects. Int J Qual Health Care 2009 Apr;
21(2):145-150. Epub 2009 Feb 13.
•
Nolan TW, Pronovost LP. Understanding variation. Quality Press. 1990 (May)
•
Perla,RJ, Provost LP, Murray SK. The run chart: a simple analytical tool for learning from variation in healthcare
processes. BMJ Qual Saf 2011;20:46e51. doi:10.1136/bmjqs.2009.037895
Images
Slide 5: Different interpretations of variation. Adapted From Nolan TW, Pronovost LP. Understanding variation. Quality
Press. 1990 (May)
Slide 8: Graphical Representation of Data. Adapted from Perla, Provost & Murray (2011) by Dr. Anna Maria IzquierdoPorrera
Slide 9: Run Charts. Adapted from Perla, Provost & Murray (2011) by Dr. Anna Maria Izquierdo-Porrera
Slide 12: Statistical Rules to identify non-random signals in run charts. Adapted from Perla, Provost & Murray (2011) by
Dr. Anna Maria Izquierdo-Porrera
Slide 13: Statistical Rules to identify non-random signals in run charts. Adapted from Perla, Provost & Murray (2011) by
Dr. Anna Maria Izquierdo-Porrera
Slide 14: Statistical Rules to identify non-random signals in run charts. Adapted from Perla, Provost & Murray (2011) by
Dr. Anna Maria Izquierdo-Porrera
Slide 15: Statistical Rules to identify non-random signals in run charts. Adapted from Perla, Provost & Murray (2011) by
Dr. Anna Maria Izquierdo-Porrera
Slide 16: P.O.B. Dr. Anna Maria Izquierdo-Porrera
Health IT Workforce Curriculum
Version
. 3.0/Spring 2012
Quality Improvement
Measuring Quality
20