Model for Improvement and
Measuring for Quality
Seminar 3
Objectives
• Review and apply relevant statistical concepts,
including sensitivity, specificity, predictive
value and probability.
• Discuss the differences between measuring
for research versus quality improvement work.
• Introduce the model for improvement, plando-study-act cycles, and run charts.
• Apply these concepts to the Pat Smith error.
Diagnostic Decision Making
• Goal: reduce uncertainty regarding the
diagnosis
• Pretest probability
• Epidemiologic data
• Individual risk data
• Clinical presentation
• Diagnostic testing
• Posttest probability
Steps Toward High Value, CostConscious Care1
1. Understand the benefits, harms, and relative costs of the
interventions that you are considering
2. Decrease or eliminate the use of interventions that
provide no benefits and/or may be harmful
3. Choose interventions and care settings that maximize
benefits, minimize harms, and reduce costs (using
comparative-effectiveness and cost-effectiveness data)
4. Customize a care plan with the patient that incorporates
their values and addresses their concerns
5. Identify system level opportunities to improve outcomes,
minimize harms, and reduce healthcare waste
The 2x2
Positive test
Negative test
Disease
A
C
No Disease
B
D
Sensitivity and specificity evaluate the TEST.
Sensitivity = A/A+C
Specificity = D/B+D
The 2x2
Positive test
Negative test
Disease
A
C
No Disease
B
D
Positive and negative predictive values evaluate the
DISEASE. These will change based on the POPULATION.
PPV = A/A+B
NPV = D/C+D
Likelihood Ratio (LR)
• Combine sensitivity/specificity with pretest
probability
• Helps evaluate how powerful a test will be in your
diagnosis
• LR = 1 test has no influence on your diagnosis
• LR < 1 decreases your pretest probability
• LR > 1 increases your pretest probability
• Goal is to use this tool on patients with
INTERMEDIATE pre-test probability
Likelihood Ratio Table2
Chest Pain Evaluation3
Chest Pain Evaluation3
Pat Smith
• Pat Smith returns to clinic with a sore throat.
• She has had subjective fevers, cough, fatigue,
and runny nose. Her spouse was given a
prescription for strep throat at a nearby
urgent care.
• She does not have a fever today, but does
have mild tonsillar erythema and shotty
nontender lymphadenopathy.
Rapid Strep
Positive test
Negative test
Disease
70
30
• Sensitivity = 70/70+30 = 70%
• Specificity = 98/2+98 = 98%
No Disease
2
98
Rapid Strep
Positive test
Negative test
Disease
70
30
• PPV = 70/70+2 = 97%
• NPV = 98/30+98 = 77%
No Disease
2
98
Rapid Strep
Positive test
Negative test
Disease
7
3
• Sens = 70/70+30 = 70%
• Spec = 98/2+98 = 98%
No Disease
20
980
• Sens = 7/7+3 = 70%
• Spec = 980/20+980 =
98%
Rapid Strep
Positive test
Negative test
Disease
7
3
• PPV = 70/70+2 = 97%
• NPV = 98/30+98 = 77%
No Disease
20
980
• PPV = 7/7+20 = 26%
• NPV = 980/3+980 =
99.7%
Determining Pre-Test Probability4
• Part of clinical decision making.
• Should be based on patient history and physical.
• In streptococcal pharyngitis, determine using
age, clinical setting, and season.
• In adults, pretest probability starts at 5-10%.
– Higher in fall and winter and lower in spring and
summer.
– Higher in ED or urgent care settings and lower in
office settings.
CENTOR criteria for Group A Strep
Pharyngitis5
CENTOR criteria (one point for each positive):
–
–
–
–
History of fever
Tonsillar exudates
Tender anterior cervical adenopathy
Absence of cough
The Modified Centor Criteria add the patient's age to
the criteria:
– Age <15 add 1 point
– Age >44 subtract 1 point
Probability of Strep Pharyngitis4
Patient complains of sore throat: 10%
Sore throat with fever, tonsillar exudate, and
swollen glands on exam: 20%
Sore throat with fever, tonsillar exudate, swollen
glands, and positive rapid strep test: ~75%.
(If rapid strep test negative, probability ~3%).
How Probability Changes4
Likelihood
Centor
Ratio
Score
0
0.16
Pretest Probability
5%
10%
15%
20%
25%
1%
2%
2%
3%
5%
1
0.3
2%
3%
5%
7%
9%
2
0.75
4%
8%
12%
16%
20%
3
2.1
10%
19%
27%
34%
41%
4
6.3
25%
41%
53%
61%
68%
What about Pat?
• What is her pretest probability?
• How does the Centor Criteria change her
pretest probability?
• What if you did do a throat culture since the
clinic was out of rapid strep tests?
• LR for positive test 12.1.
• LR for negative test is 0.16.
• Would the throat culture change your
treatment plan if it was positive?
Likelihood Ratio Table2
Pat Smith
• But she really wanted the antibiotic…
• Why do we order unnecessary tests or give
unnecessary antibiotics?
Why do we order unnecessary
diagnostic tests/meds?6
• Don’t know they are not helpful.
• Don’t trust your own clinical judgment. Tests
feel “objective” but they are not.
• Desire for "baseline" information.
• Easier than explaining to patients why you’re
not ordering it.
• “CYA” – scared of malpractice suits.
Research
•
•
•
•
•
Randomized Control Trial
Cohort Study
Null hypothesis
Blinding
Confounding variables
Rapid Response Teams7
• Only 10-15% of non-ICU patients survived cardiac
arrest.
• Rapid Response teams were created.
• Thought to improve teamwork, reduce staff anxiety,
decrease code blues, and possibly reduce mortality.
• MERIT trial (2005): RRT had no beneficial effect.
•
•
•
•
Cluster randomized prospective trial
Study was underpowered
Potentially cross-contaminated
Claimed to be a negative trial but inconclusive at best
• Quality data is not well measured when using classic
science research techniques.
Measurement for Research versus
Quality7
Measurement of Research
Purpose
Tests
Biases
Data
Duration
Measurement for Process
Improvement
Measurement for Research versus
Quality7
Purpose
Tests
Biases
Data
Duration
Measurement of Research
Measurement for Process
Improvement
Discover new knowledge
Bring knowledge into practice
Measurement for Research versus
Quality7
Measurement of Research
Measurement for Process
Improvement
Purpose
Discover new knowledge
Bring knowledge into practice
Tests
One large blind test
Many sequential observable
tests
Biases
Data
Duration
Measurement for Research versus
Quality7
Measurement of Research
Measurement for Process
Improvement
Purpose
Discover new knowledge
Bring knowledge into practice
Tests
One large blind test
Many sequential observable
tests
Biases
Control for as many biases
as possible
Stabilize the biases from test
to test
Data
Duration
Measurement for Research versus
Quality7
Measurement of Research
Measurement for Process
Improvement
Purpose
Discover new knowledge
Bring knowledge into practice
Tests
One large blind test
Many sequential observable
tests
Biases
Control for as many biases
as possible
Stabilize the biases from test
to test
Data
Gather as much as possible
Gather enough to learn and
adjust for new cycle
Duration
Measurement for Research versus
Quality7
Measurement of Research
Measurement for Process
Improvement
Purpose
Discover new knowledge
Bring knowledge into practice
Tests
One large blind test
Many sequential observable
tests
Biases
Control for as many biases
as possible
Stabilize the biases from test
to test
Data
Gather as much as possible
Gather enough to learn and
adjust for new cycle
Duration
Long periods of time
Short duration to accelerate
change
Model for Improvement8
• Three questions
• Aim
• Measures
• Change
• PDSA cycle
Model for Improvement: In Detail8
• Aim
• What are we trying to accomplish?
• Be specific: How good? By when? For whom?
• Identify who will and should help you
accomplish these changes.
Model for Improvement: In Detail8
• Measures
•
•
•
•
How will we know the change is an improvement?
Outcome measures
Process measures
Balancing measures
Model for Improvement: In Detail8
• Changes: What changes can we make that will
result in an improvement?
• Don’t pick one change and stick with it but
plan to re-evaluate with each small change.
PDSA Cycle8
PDSA Cycle8
• Tests should be small and
specific.
• Each test should influence
the next one.
• Expand conditions if a
test will work under
different circumstances.
• Use the rule of 5s to
expand testing.
• Results should evaluate if
a test is promising.
Example of a Simple Run Chart9,10
• Easy means of tracking
and displaying data.
• X-axis shows time.
• Y-access shows an
outcome or process
measure.
• Include annotations to
show when and where
different interventions
started.
Run Charts Rules10,11
Back to Pat Smith’s Allergic Reaction
You feel terrible about poor Mrs. Smith receiving
Augmentin despite her PCN allergy.
You decide to initiate an improvement project to
ensure this doesn’t happen again.
• Where are areas in her RCA that are potential
opportunities for change?
• Give an example of an Aim, Measure and
Change?
• How would you plan for her PDSA cycles?
Apply to your project
• Complete the PDSA worksheets for your idea
for improvement.
• Identify your aim, different measures and
potential change you can make to prepare for
your chosen project.
References
1.
ACP HVCCC Curriculum. Adapted from Owens, D, et al. High Value, Cost Conscious Health Care: Concepts for clinicians to
evaluate the benefits, harms, and costs of medical interventions. Ann Intern Med. 2011;154:174-180.
2. Alguire, P, et al. Utilizing Biostatistics in Diagnosis, Screening, and Prevention . American College of Physicians and Alliance
for Academic Internal Medicine High Value Care curriculum, Version 2 (2013-4), presentation 3 of 6. Accessed September
2013.
3. Alguire, P, et al. Utilizing Biostatistics in Diagnosis, Screening, and Prevention. American College of Physicians and Alliance for
Academic Internal Medicine High Value Care curriculum, Version 2 (2013-4), presentation 3 of 6, Small Group Worksheet,
Case 1. Accessed September 2013.
4. Ebell MH et al. The rational clinical examination: does this patient have strep throat? JAMA. 2000; 284:2912-2918.
5. Centor RM, Witherspoon JM, Dalton HP, Brody CE & Link K (1981). "The diagnosis of strep throat in adults in the emergency
room". Medical Decision Making 1 (3): 239–246.
6. Alguire, P, et al. Overcoming Barriers. American College of Physicians and Alliance for Academic Internal Medicine High Value
Care curriculum, Version 2 (2013-4), presentation 5 of 6. Accessed September 2013..
7. Berwick, DM. The science of improvement. JAMA. March 12, 2008; Vol 299 (10): 1182-84.
8. Lloyd, R. Murray, S. Provost, L. QI 102: The Model for Improvement: Your Engine for Change. [IHI Open School online course].
Cambridge, Massachusetts: Institute for Healthcare Improvement; 2009. http://app.ihi.org/lms/onlinelearning.aspx. June 1,
2009. Accessed March 2016.
9. Schriefer, J and Leonard, MS. Patient Safety and Quality Improvement: An Overview of QI. Pediatrics in Review. Aug 2012:
33(8); 353-360.
10. Lloyd, R. Murray, S. Provost, L. QI 103: Measuring for Improvement. [IHI Open School online course]. Cambridge,
Massachusetts: Institute for Healthcare Improvement; 2009. http://app.ihi.org/lms/onlinelearning.aspx. June 1, 2009.
Accessed July 2016.
11. Perla RJ et al. The run chart: a simple analytical tool for learning from variation in healthcare processes. BMJ Qual Saf 2011;
20: 46-51.