Clinical Decision Rules:
Clinical
Decision Rules:
How to Get Started
How to Get Started
CAEP 2012 Niagara Falls
Ian Stiell MD MSc FRCPC Ian Stiell MD MSc
Ian Stiell MD FRCPC
Professor and Chair, Dept of Emergency Medicine
Professor and Chair, Dept of Emergency Medicine
Distinguished Professor & University Health Research Chair
Research Chair
Senior Scientist, Ottawa Hospital Research Institute
Disclosures
None
What is a Clinical Decision Rule?
What is a Clinical Decision Rule?
Definition: Definition:
A tool that helps clinicians make diagnostic and therapeutic decisions at the bedside
Derived from original research
Incorporate 3 or more variables
co po ate 3 o o e a ab es from history, exam, or simple o
sto y, e a , o s p e
tests
Examples : which patients with ‐
Examples :
which patients with ‐
ankle injury need x‐rays?
possible DVT need imaging?
p
g g
heart failure need admission?
headache need CT/LP?
Prognostic/survival models are different
Clinical Decision Rules:
How to Get Started
Introduction to CDRs
Examples from Ottawa
Examples from Ottawa
Methodological Standards for Derivation
Prospective Validation p
Implementation Trial
Knowledge Transfer
What Conditions are Suitable for Clinical Decision Rules? The Need:
Common, high‐volume
, g
conditions, e.g. chest pain, cough and , g
p ,
g
fever, extremity injury, shortness of breath
Inefficient use of resources, e.g. diagnostic tests or hospitalization Variation in current practice
The Purpose:
The Purpose:
Standardize care based on evidence
Improve safety
Improve safety
Improve efficiency
JAMA 1997
JAMA 2000
Theory and Reality of Developing y
y f
p g
Clinical Decision Rules
Introduction to CDRs
Examples from Ottawa
Examples from Ottawa
Methodological Standards for Derivation
Feel free to ask questions
Clinical Decision Rules
Developed
Developed at U Ottawa
at U Ottawa
Ottawa Ankle and Knee Rules
Ottawa Ankle and Knee Rules
Canadian C‐Spine and CT Head Rules
Head Rules
Canadian Subarachnoid Rule – Jeff Perry
Wells Criteria for PE and DVT Wells
Criteria for PE and DVT
– Phil Wells
Ottawa Heart Failure and Ott
H t F il
d
COPD Rules
The Canadian C‐Spine Rule
Spine Rule
Imaging for Alert & Stable Trauma Patients
Stable Trauma Patients Potential C‐Spine Injury:
Potential C‐
The Clinical Problem
>> 8 million potential neck 8 million potential neck
injury cases per year in Canadian and US EDs
Canadian and US EDs
Most are alert and stable with <1% having c‐spine fracture
<1% having c‐spine fracture
C‐spine diagnostic imaging use i ffi i t d
i bl
inefficient and variable
High volume items add to health care costs
Prolonged immobilization and g
ED overcrowding
Development of the Canadian
Canadian C
Canadian C‐
C‐Spine Rule
C‐
Spine Rule
Variation and Inefficiency (N=6,855)
CMAJ 1997
CMAJ 1997
Derivation of the Rule (N=8,924) JAMA 2001
JAMA 2001
Prospective Validation (N=8,283) g J Med 2003 New Engl
Multicentre Implementation (N=11,648) Br Med J 2009
Awareness and Use (N=1,150) Acad Emerg Med 2008 Validation by Paramedics
Validation by Paramedics Ann Emerg Med 2009
Validation by ED Nurses (N=3,633)
Validation by ED Nurses (N
3,633) CMAJ 2010
Derivation of Canadian C‐Spine Rule Spine Rule ‐ JAMA 2001 Spine Rule ‐
JAMA 2001
Objective:
Derive a clinical decision rule highly sensitive for acute cervical spine injury
sensitive for acute cervical spine injury
Methods (N=8,924):
Prospective cohort ‐ 10 Canadian EDs
Alert and stable, adult trauma patients
MDs assessed 20 clinical findings
Clinically important c‐spine injury
y p
p
j y
Recursive partitioning analyses
2003
2003
Potential Radiography Rates
P < 0.001
80%
66.6%
55.9%
60%
40%
NEXUS
Canadian
2009
Ottawa Risk Scale for Ottawa
Risk Scale for
ED Patients with COPD
D
Department of Emergency Medicine
t
t fE
M di i
June 2011
COPD Patients in the ED: The Clinical Problem
Common ED presentation
With bed shortages, MDs under bed shortages MDs under
pressure to send home Adverse outcomes common, especially in those discharged
Little evidence to guide disposition decisions
disposition decisions
Need risk scales for rational and safe admission decisions
safe admission decisions
Methods
Design: Prospective cohort study
g
g ,
y
p
Setting: 6 EDs of large, tertiary care Canadian hospitals
Subjects: Adults with exacerbation of COPD ‐ both admitted and discharged
admitted and discharged
Excluded: Very ill patients
Standardized Assessment: variables from history, exam, lab, chest x‐ray, ECG
,
y,
Serious Adverse Event: death, intubation, critical care, ,
MI, return to ED with admission
Serious Adverse Events (N=69/945)
Total SAEs (N=945)
(N=945) Admitted Patients (N=354)
Patients (N=354) Discharged Patients (N=591)
Patients (N=591) Ottawa COPD Risk Scale ‐
Ottawa COPD Risk Scale ‐ Identify ED Patients at High Risk for SAE
ED Patients at High Risk for SAE
Theory and Reality of Developing y
y f
p g
Clinical Decision Rules
Introduction to CDRs
Examples from Ottawa
Examples from Ottawa
Methodological Standards Methodological Standards ‐‐ Derivation
Feel free to ask questions
Was the Rule Derived According to g
Methodological Standards? 1. Outcome Measure
2. Predictor Variables
Predictor Variables
3. Reliability of Predictors
4. Study Subjects
5. Sample Size
6. Mathematical Techniques
7 Sensibility for Clinicians
7.
Sensibility for Clinicians
8. Classification Accuracy Was the Rule Derived According to g
Methodological Standards? 1. Outcome Measure
Clinically important
Clearly defined
Assessed blindly
Clinically Important C‐‐Spine Injury
Clinically Important C
Serious Adverse Event for COPD:
Death
Intubation or NIV
Myocardial infarction
Admission to monitored unit
Admission to monitored unit Relapse back to ED requiring admission < 14 days
Was the Rule Derived According to g
Methodological Standards? 2. Predictor Variables
Standardized definition
Collected prospectively with data form
Assessed before outcome
20 Variables from History and Exam
30 Variables from History, Exam, Lab, ECG, Xray
30 Variables from History, Exam, Lab, ECG, Xray
Innovative 3‐min walk test
Univariate Correlation of COPD SAEs Variables from Exam and Labs (N=945)
Variables from Exam and Labs (N=945)
SaO2 on arrival (%)
CTAS level
Urea Glucose
pCO2 HGB
ECG ischemia
CXR congestion Too ill to do walk test (%)
Walk test highest HR (N=43, 749) SAE
No SAE
P‐Value
91.3
2.5
9.8
8.2
60.2
122.8
7.5
25.7 41.9
98.7
93.5
2.7
7.3
7.1
43.6
133.7
1.9
9.1
13.0 104.0 0.01
0.02
<0.01
0.03
0.01
<0.01
<0.01
<0.01
<0.0001
0.05
Was the Rule Derived According to g
Methodological Standards? 3. Reliability of Clinical Variables
Interobserver agreement beyond chance:
kappa (dichotomous / nominal data)
weighted kappa
weighted kappa (ordinal data)
(ordinal data)
intraclass correlation coefficient (interval)
3. Reliability: Interobserver Agreement
f C-Spine
for
C S i Findings
Fi di
Characteristic
Midline neck pain
Immediate neck pain
Weakness in extremities
Numbness / tingling
Upright position
Distracting injuries
F i l Injury
Facial
I j
Kappa
(N=150)
.69
.48
.54
.77
.78
78
.41
.75
75
Was the Rule Derived According to g
Methodological Standards? 4. Study Subjects
Defined inclusion criteria
Unbiased selection
Characteristics described
Setting
lert stable trauma patients with neck pain
Alert stable trauma patients with neck pain Adults with exacerbation of COPD ‐ both admitted Adults with exacerbation of COPD ‐
and discharged
Excluded: very ill patients:
O2 Sat < 85% on room air, HR > 130, SBP < 85, chest pain or acute ECG changes
COPD Patient Characteristics (N=945) Mean age
Male (%)
Male (%)
Ambulance arrival (%)
Duration of symptoms, hours
i
f
h
CTAS score, mean
Associated HF in ED (%)
Past Medical History (%)
Past Medical History (%)
Heart Failure
Admission for respiratory distress
Admission for respiratory distress
Intubation for respiratory distress
Current smoker (N=792)
Current smoker (N=792)
Home oxygen (%)
72.6
51 6
51.6
48.3
87.0
2.7
10.2
21.1
28.4
2.9
31 6
31.6
12.5
Was the Rule Derived According to g
Methodological Standards? 5. Sample Size
Requires adequate number of positive outcomes
Methods for determining SS:
rule of thumb – 10 outcomes / predictor
degree of precision in CI around measure of accuracy
8,924 neck injury patients with 151 important c
8,924 neck injury patients with 151 important c‐‐spine injury cases (1 7%)
injury cases (1.7%)
945 COPD patients with 74 SAE cases (7.8%)
Sample Size: C‐Spine Phase I
Estimated
‐ 100% sensitivity with 95% CI 97‐100%
j y
‐ 120 injury cases
‐ 1.5% incidence important injury
,
p
‐ 8,000 patients
Actual
‐ 100% sensitivity (98‐100)
j y
‐ 151 injury cases
‐ 1.7% incidence
p
‐ 8,924 patients
Was the Rule Derived According to g
Methodological Standards? 6. Mathematical Techniques
Univariate Kappa
Multivariate:
Chi‐square recursive partitioning
Logistic regression
Variables with strong univariate associations and kappa > 0 6  recursive partitioning
kappa > 0.6 recursive partitioning
Univariate analyses  logistic regression
Independent Predictors of SAE from Logistic Regression (N=945)
Logistic Regression (N=945)
Variable
β
Hx of PVD intervention
Prior CABG
Prior CABG
Prior intubation
ECG acute ischemia
ECG acute ischemia
CXR pulmonary congestion
Too ill to walk after treatment
Too ill to walk after treatment
HR on ED arrival > 110
HgB < 100
HgB < 100
Urea > 12
Serum CO2 > 35
Serum CO2 >
0.90
0 71
0.71
1.32
1 18
1.18
0.63
1 25
1.25
1.12
1 59
1.59
0.89
0 66
0.66
Hosmer‐Lemeshow Goodness‐of‐fit P = 0.70
Area under ROC curve = 0.80 (95%CI 0.74 ‐0.85)
OR
2.46
2 03
2.03
3.73
3 25
3.25
1.88
3 50
3.50
3.05
4 90
4.90
2.43
1 91
1.91
Was the Rule Derived According to g
Methodological Standards? 7. Sensibility for Clinicians
Clinical validity
Simple and easy to use
Yes/No vs probability
Yes/No Algorithm for C‐
C‐Spine
Yes/No Algorithm for C
Risk Scale for COPD
Was the Rule Derived According to g
Methodological Standards? 8. Accuracy – Classification Performance
8. Accuracy –
2 x 2 tables with sensitivity, specificity
ROC curve analysis
Predicted probability
Bootstrapping internal validation
Calibration
C‐Spine Spine –– 2 x 2 table
COPD –– probability
COPD Classification Performance and Potential Admissions for Ottawa COPD Risk Scale
COPD Risk COPD
Risk
Potential
Score
Sensitivity Specificity Admission 0
1
2
3
4
5
6
7
8
10
1.0 0.91
0.81
0.60
0.52
0.25
0.19
0.07
0.06
0.03
0.0
0.45
0.60
0.84
0.90
0.97
0.99
0.996
0.999
1.0
100%
57.6%
43.2%
20.0%
Calibration Between Observed vs
Calibration Between Observed vs Expected SAE Score in COPD Patients
SAE Score in COPD Patients
Homer‐Lemeshow goodness‐of‐fit p‐value = 0.67
Theory and Reality of Developing y
y f
p g
Clinical Decision Rules
Introduction to CDRs
Examples from Ottawa
Examples from Ottawa
Methodological Standards for Derivation
Prospective Validation Implementation Trials
Implementation Trials
Knowledge Transfer
Feel free to ask questions
Prospective Validation of O
Ottawa Decision Rules
ii
l
Ottawa Ankle Rule (N=1,485)
Ott
A kl R l (N 1 485)
JAMA 1993
O
K
R l (N 1 096)
Ottawa Knee Rule (N=1,096) JAMA 1996 Canadian C‐Spine Rule (N=8,283) New Engl J Med 2003
Canadian CT Head Rule (N=2,707)
JAMA 2005 Validation by Paramedics (N=1,949)
Ann Emerg Med 2009
Validation by ED Nurses (N=3,633) Can Med Assoc J 2010
Was the Rule Prospectively and p
y
Explicitly Validated? New patients and settings
Explicit application
Explicit application
Outcome measures
Accuracy:
The rule
Physicians
Reliability
e ab ty
Physician comfort
P t ti l i
Potential impact
t
Validation of Canadian C‐Spine Rule Spine Rule ‐ NEJM 2003 Spine Rule ‐
NEJM 2003
Objective: Prospectively compare the clinical Prospectively compare the clinical
performance of the CCR and the NEXUS criteria
Methods (N=8,283):
(
)
Prospective cohort ‐ 9 Canadian EDs
Alert and stable, adult trauma patients
349 MDs assessed patients for both rules –
p
data forms
2nd observer where feasible
169 important c‐spine
169 important c
spine injury cases by imaging or F/U
injury cases by imaging or F/U
Other Validation Measurements
Physician Accuracy
91 2%
91.2%
Reliability (kappa)
0 64
0.64
Clinical Sensibility
“Uncomfortable” with rule
8.0%
Potential Impact:
C S i Radiography
C-Spine
R di
h Rates
R t
80%
71 7%
71.7%
P < 0.0001
0 0001
RR = 22.0%
55.9%
60%
40%
Actual Rate
Potential Rate
Potential Impact:
Total Time in ED in Minutes
250
232.8
Diff = 109.8
P < 0.0001
200
150
123.0
100
Radiography
N=4,129
No Radiography
N=1,779
Actual vs Potential C‐Spine Clearance Rates (N=3,633)
50%
40.6%
25%
0%
0%
0%
Actual Rate
Potential Rate
Theory and Reality of Developing y
y f
p g
Clinical Decision Rules
Introduction to CDRs
Examples from Ottawa
Examples from Ottawa
Methodological Standards for Derivation
Prospective Validation Implementation Trials
Implementation Trials
Knowledge Transfer
Feel free to ask questions
Implementation Trials of O
Ottawa Decision Rules
ii
l
Ottawa Ankle Rules (N=2,342)
O
A kl R l (N 2 342)
JAMA 1994
Ottawa Ankle Rules (N=12,777)
Br Med J 1995
Ottawa Knee Rule (N=3,907) JAMA 1997
JAMA 1997 Canadian C‐Spine Rule (N=11,824 Br Med J 2009
Br Med J 2009
Canadian CT Head Rule (N=4,531) Can Med Assoc J 2010 Has the Rule been Implemented p
into Practice to Assess Impact? Controlled or cluster randomized trial design
Impact on clinical care:
Impact on clinical care:
Process measures: e.g. hospital admission, use of imaging
fi
i
Patient outcomes: e.g. mortality, missed injuries
Other:
Accuracy of the rule
Physician acceptability
Physician acceptability
Patient acceptability
2009
Implementation of Canadian C‐Spine Rule Spine Rule –
Spine Rule – Br Med J 2009 Br Med J 2009
Objective: To evaluate the effectiveness of an active strategy to implement the CCR into multiple EDs
i l
h CCR i
l i l ED
Methods (N=11,824):
(
,
)
Matched pair cluster randomized trial
12 university and community hospital EDs
12 university and community hospital EDs
Alert and stable, adult trauma patients
6 hospitals intervention 6 control
6 hospitals intervention, 6 control
Active strategies : Ed ti
Education ‐
rounds, handouts, posters, pocket cards, d h d t
t
k t d
PDA, screen‐savers
Policy Real‐time reminders
Diagnostic Imaging Rates (N=
(N 11,648)
P<0
0.001
001
70%
70
P < 0.01
P < 0.01
% Im
maging
61.7%
59.8%
53.7%
53.8%
After
P i d
Period
45%
20%
20
Before
Period
6 Intervention
6 Control
Hospitals
Hospitals
Study Sites
Secondary Outcomes (N=11,648)
Outcomes
Intervention Hospitals Control Hospitals
Before
After
Before After
N=3,266 N=3,624 N=2,413 N=2,345
Missed Fractures
0
0
0
0
Adverse Outcomes
0
0
0
0
206
215
187
210
Time in ED (Mins)
The “mother
mother of all negative trials”
trials !!
PRIMARY OUTCOME (N=4,531)
Diagnostic Imaging Rates
P = 0.64
P < 0.01
76.2%
80
80%
62.8%
% Im
maging
P < 0.01
74 1%
74.1%
67.5%
Before
Period
After
P i d
Period
40%
40
0%
0
6 Intervention
6 Control
Hospitals
Hospitals
Study Sites
Barriers to CCH Rule Use:
Pre-Study Survey - Brehaut 2003
Forget details of CCH Rule
32%
Trauma/NS will order anyway
30%
Patient/family expectation
10%
Research is flawed
6%
g to no CT
See no advantage
6%
Busy ED – can’t observe
6%
Takes too much time
2%
Rule not safe for patients
2%
Resent concept of guidelines
0%
Barriers to Use:
Use: Post
Post--Study Survey
Physician Beliefs and Attitudes:
• Bent rule to meet their needs
• Didn’t ‘believe’ the rule
• Didn’t like being
g directed
Electronic Ordering:
• Physical restriction of accessing computer
• Ability to circumvent the rule
Busy and Overcrowded EDs:
• Easier to CT and discharge
• Tests that speed flow are used
CT Head is Standard of Care:
• Access to CT easy at this site and becoming routine care
Theory and Reality of Developing y
y f
p g
Clinical Decision Rules
Introduction to CDRs
Examples from Ottawa
Examples from Ottawa
Methodological Standards for Derivation
Prospective Validation Implementation Trials
Implementation Trials
Knowledge Transfer
Feel free to ask questions
How does Clinician Uptake p
occur for Decision Rules? How do we close the evidence
How do we close the evidence‐‐practice gap?
Passive Diffusion
Passive Diffusion
Journal articles, scientific meetings
Dissemination
Targets an audience – mailouts, speakers
l
i
id li
Meta‐analyses, reviews, guidelines
Implementation
p
Active, local, persistent
Administrative, educational strategies
,
g
BMJ 2003
Evaluation of the Dissemination and Uptake of Ottawa Decision Rules
k fO
ii
l
Attitudes and Use Ankle/Knee in Canada (N=232)
A i d
d U A kl /K
i C
d (N 232)
Graham ‐ Acad Emerg Med 1998
Awareness and Use Ankle/Knee in 5 Countries (=1,769)
Graham ‐ Ann Emerg Med 2001
CCR and CCHR in Canada (N=262) Brehaut ‐ Acad Emerg Med 2006 Brehaut Med 2006
CCR and CCHR in 4 Countries (N=1,150) Eagles Acad Emerg Med 2008 Eagles ‐
Med 2008
Theory and Reality of Developing y
y f
p g
Clinical Decision Rules
Introduction to CDRs
Examples from Ottawa
Examples from Ottawa
Methodological Standards for Derivation
Prospective Validation Implementation Trials
Implementation Trials
Knowledge Transfer
Feel free to ask questions
Theory and Reality of Developing y
y f
p g
Clinical Decision Rules
Was the Rule Derived According to g
Methodological Standards? Clinical Decision Rules:
Clinical
Decision Rules:
How to Get Started
How to Get Started
CAEP 2012 Niagara Falls