The Quest for Quality:
Monitoring and Measuring Outcomes in
Continuing Nursing Education
Session 1: Outcomes for Learning
Activities
Pamela S. Dickerson, PhD, RN-BC, FAAN
Director of Continuing Education, Montana Nurses Association
President, PRN Continuing Education
Presented by Ohio Nurses Association
and Montana Nurses Association
1
Disclosures
• Purpose: Participants will effectively develop
learning and provider unit processes that lead to
measurable outcomes demonstrating quality
patient care and/or enhanced professional
development of registered nurses.
• Criteria for Successful Completion: Participate in this
entire webinar. Complete the evaluation and
registration information and return to ONA as
directed.
• Conflict of Interest: There is no conflict of interest for
any planner or presenter for this activity.
2
Objective
• Examine strategies for development of learning
activities that produce measurable outcomes.
3
Self-Assessment
I know how to
Strong =
3
Moderate = 2
Weak = 1
Conduct a meaningful needs assessment
Analyze gaps
Develop an outcome statement
Measure outcomes
Total Score =
4
Why Do Outcomes
Matter?
• Evidence related to patient safety
• Evidence related to quality of care
• Evidence related to life-long learning
• Accountability
o
o
o
o
To nurses
To patients
To employers
To the public
5
Benefits of Outcomes Evaluation for
Providers of Continuing Education
• Credibility
• Sustainability
• Provide evidence to support the value of
continuing nursing education
• (Read: How to keep your job!)
6
What is an Outcome?
• Webster: “A final consequence”, “result”
• NPD Scope & Standards, 2010: “something that
follows, is the result of, or is the consequence of a
project, program, or event”
• Outcome Measurement: “The process of observing,
describing, and quantifying predefined indicators of
outcomes of performance.”
7
What Do We Mean By
“Measurable”?
• Webster – Able to be measured; of distinguished
importance – SIGNIFICANT
• Webster – “Measure” = basis of comparison
• Quantifiable
EVIDENCE
• Observable
8
Considering Evaluation Strategies:
A Traditional Kirkpatrick Model
• Process – what are we doing?
• Product – what did we do?
• Outcomes – what difference did it make?
• Impact – how is our organization better as a result?
9
A Different Perspective
• Consider turning this upside down:
o How do we want to make our organization better?
(Where are the gaps?)
o What outcomes do we want to achieve in our
learning activity to help that happen?
o What behaviors do we want to see in our learners to
achieve that outcome?
o What learning objectives and strategies will help
learners achieve these behaviors?
10
Evaluation in Context
Target Audience
Needs
Assessment
Gap Analysis
Outcome
Purpose
(long term
evaluation)
Short-term
Evaluation
Objectives
Content/
Teaching
Methods
11
Educational Design
Components
• Target Audience: who needs this information?
• Gap Analysis: why do they need it?
• Learning Styles: how can they learn it best?
• Purpose: what will learners be able to do as a result
of participating in the learning activity?
• Outcome: what measurable evidence will show
that the gap has been filled?
12
Planning for Outcomes Starts
at the Beginning
• With the given target audience, what is the gap in
knowledge, skill, or practice?
• Gap Analysis Worksheet
What is
the
current
state?
What is
the
desired
state?
What
evidence
supports
the need?
What do
we need
to
measure?
How do
we
capture
the data?
13
Gap Analysis
• In 2012, the Joint Commission released a sentinel
event alert regarding opioid use in hospitalized
patients. We need to teach this to our nurses.
• Is this a strong gap analysis statement?
o Yes
o No
14
15
Gap Analysis
• What nurses need this information?
• Among this group, is there currently a problem in
the way opioid medication is administered?
o
o
o
o
o
Too much?
Too frequently?
Not enough attention to alternatives
Not thorough assessment of patient need
Other
• What data do we have that shows a gap?
• If there is a gap, is it in knowledge, skill, or
application in practice?
• What do we need to do to fill the gap?
16
A Stronger Statement
• In light of the 2012 Joint Commission Sentinel Event
Alert regarding appropriate use of opioid
medications among acute care patients and the
fact that, in our medical-surgical unit, 80% of
patients have received opioid medication in the
last 14 days without evidence of assessment for
other potentially effective types of analgesic
support, there is a gap in …
17
Next Questions
• Do the nurses know about the recommendations
from TJC?
• Are prescribers aware of the recommendations?
• How are pain assessments currently done? Are they
thorough enough? Do they consider nonpharmacological and non-opioid options?
• What are the documentation options in the EMR?
18
Starts
at the Beginning
• How do you decide on a purpose for a learning
activity?
• How does the purpose provide the learner with
information about what he/she is expected to do
as a result of participating in this activity (as
opposed to what the provider will offer)?
19
Purpose
• The purpose of this activity is to provide information
about safe opioid use among patients in the acute
care hospital.
• Is this a strong purpose statement?
o Yes
o No
20
21
Purpose
• The fact that information is being provided may be
of value to the provider but is of no benefit to the
learner.
• A better approach – the nurse will use opioid
analgesia judiciously in accordance with TJC
recommendations when caring for medical-surgical
patients.
22
Outcome
• What is the desired outcome of our educational
intervention?
• How will we be able to know when “success” has
been achieved? (the gap has been filled – or
lessened)
23
Outcome Statement
• Medical-surgical patients will receive pain
management in accordance with TJC
recommendations regarding opioid use.
• Is this a strong outcome statement?
o Yes
o No
24
25
Outcome Statement
• We know the population – medical-surgical patients
• We know the standard – TJC recommendation
• We don’t know what “success” looks like – how
many patients? What assessment data will be
collected? Documented? Is success based on the
assessment or the intervention?
26
A Stronger Statement
• From April 1-14, 2014, 90% of patients on the
medical-surgical unit will have documentation
reflective of nursing assessment related to type of
pain and options for non-pharmacologic and/or
non-opioid pain management.
• Previous data is 20% - gap to close is between 20
and 90
• Why this outcome? The nurse cannot control the
prescribing, but can advocate for the patient by
collecting and documenting relevant data
27
For Consideration
• Is self-report a valid “measure” of accomplishment?
28
• Yes
• No
29
For Consideration
• Self-report is a valid way to capture measurable
data regarding a person’s perceptions
• This is the basis of phenomenological research
• May want to consider other ways of validating that
data
30
Other Options
• Pre-test / Post-test / Follow-up test
• Surveys of managers
• Nurse educator observations of practice
• Chart reviews
• Monitoring of incidents, e.g. medication errors
• Other
31
External Sources of Data:
Examples
• National Database for Nursing Quality Indicators:
o Clinical: falls, pressure ulcers, pain assessments, healthcareacquired infections, etc.
o Non-clinical: workforce data, practice environment, etc.
• AHRQ: Extensive data base related to health care
quality initiatives
• Centers for Disease Control and Prevention
32
Internal Sources of Data:
Examples
Risk management
Quality / Performance Improvement
Incident reports
Recruitment / Retention Data
Finance – cost of care, cost of equipment &
supplies, etc.
• Readmission data
•
•
•
•
•
33
You’re Not Alone
• You don’t have to reinvent the wheel – or keep
spinning yours. Seek help in data collection and
analysis – both to support the need for the activity
(gap analysis) and to measure the results (your
outcome).
34
Does Your Educational Plan Meet
the Assessed Need?
•
•
•
•
•
•
•
Issue / gap clearly identified
Desired outcome clear and measurable
Supported by others
It’s physically, mentally, emotionally “do-able”
Supplies, equipment, resources available
Value identified and shared
Educational plan specific and targeted
35
Self-Assessment
I know how to
Strong = 3
Moderate = 2 Weak =
1
Conduct a meaningful needs
assessment
Analyze gaps
Develop an outcome statement
Measure outcomes
Total Score =
36
Your Own Measurable Outcome
• What was your score at the beginning of the session?
• What is your score now?
• If your score has increased, you have narrowed the gap
between what you knew and what you now know. The
best score is a 12. My pre-session score was a 5, my postsession score is a 9. Therefore, I have decreased the
knowledge gap by 4 points.
• Now, how will you use this information? (Retitle the tool “I
am able to” and assess yourself in 60-90 days.) – note: if
ONA/MNA wanted to collect outcomes data, we could
have you submit this additional information.
37
Questions, Ideas, and Inspirations
38
For Further Information
Pam Dickerson, PhD, RN-BC, FAAN
[email protected]
[email protected]
39
References / Resources
American Nurses Association and National Nursing Staff
Development Organization (2010). Nursing Professional
Development: Scope and Standards of Practice. Silver
Spring, MD: Nursebooks.org.
DeSilets, L. (2009). Connecting the dots of evaluation.
Journal of Continuing Education in Nursing 40(12), 532533.
Dickerson, P., and Chappell, K. (2012). Purpose, objectives,
and outcomes: tying the pieces together. Journal of
Continuing Education in Nursing 43(9), 389-390.
40
References / Resources
• Dickerson, P. (2011). Evaluating an activity: Beyond the “form”.
Journal of Continuing Education in Nursing 42(7), 292-293.
• Dickerson, P. (2008). Addressing barriers to enhance
outcomes. Journal of Continuing Education in Nursing 39(10),
437-438.
• Gibbs, J. and Reissour, S. (2010). Evaluating Outcomes:
Evidence of Success, Journal of Continuing Education in
Nursing, 41(11), 484-485.
• How to evaluate learning: The Kirkpatrick model for the 21st
century. http://www.dashe.com/blog/informal-learning/howto-evaluate-learning-the-kirkpatrick-model-for-the-21st-century
41