PFS Evaluation
Presented by Alaska PFS State Evaluators:
Bridget Hanson and Jodi Barnett
WELCOME! WHO’S JOINING US?
 Let’s introduce ourselves
Name
Community
Project role
Please use the chat box
Meeting Guidelines
We encourage participation!
Share your comments, questions, ideas.
Be positive, be problem solvers.
Keep chat box comments on-topic.
“One of the great mistakes
is to judge policies and programs
by their intentions
rather than their results”
Milton Friedman
Webinar Objectives
Alaska PFS staff will understand:
project evaluation expectations
importance of baseline data collection
the difference between the federally
required versus Alaska PFS evaluation
Your Turn (Raise Your Hand)
How many of you
have read the
evaluation section
of the guidance
document in the
past month?
Alaska PFS timeline
Key Elements of the Alaska PFS Project
Data driven
Key Elements of the Alaska PFS Project
Ultimate goal is to prevent/reduce
non-medical use of prescription opioids (12-25 year olds)
use of heroin (18-25 year olds)
Reminder: Federal vs Alaska PFS evaluation
Community Level Instrument (CLI-R) is a
federal evaluation requirement
Strategy evaluation efforts (using time series
graphs) are specific to the Alaska PFS project
Community-Specific Logic Model Elements
Social Availability
Community
Strategies
“What strategies
will we choose to
address the
community
factors”
Community
Factors
“Why does Rx
opioid
misuse/abuse
occur specifically in
our community?”
Easy access to Rx opioids
through social sources
Retail Availability
Easy access to Rx opioids
through providers &
dispensers
Perceived Risk of Harm
from Rx opioid misuse or
heroin use
Your Turn! (Chat Box)
Describe 1 or 2 community
factor(s) that rose to the top
in your data assessment?
Please use the chat box
Alaska PFS Evaluation Planning
Step 1: Finalize Community Logic Models
 snapshot of what the project is trying to accomplish
 communication tool
 help guide and focus planning efforts
Alaska PFS Logic Model Components
Community
Strategies
Community
Factors
“What strategies will “Why specifically
we choose to
does Rx opioid
address the
misuse/abuse
community factors” occur specifically in
our community?”
Intervening
Variables
Consumption
Consequences
“Why does Rx
opioid
misuse/abuse
occur?”
“Rx opioid
misuse/abuse
among 12-25
year olds”
“Consequences of
Rx opioid misuse /
abuse (i.e. Why do
we care?)”
Direction of influence once implementation begins
If-then statements
Example Community Logic Model
State-level evaluators at CBHRS have
identified indicators and will collect data for
intervening variables, consumption, and
consequences related to NMUPO and heroin
use at the state and funded community levels.
This data will be provided to communities in
order to monitor outcomes. Communities will
focus on local data related to community
factors for the PFS outcome evaluation.
Your Turn! (Raise Your Hand)
Has your coalition
discussed any potential
strategies to target
community factors?
Alaska PFS Evaluation Planning
Step 2: Develop Strategy Evaluation Plans
(for time series graphs)
Identify outcomes, indicators, and measures to answer:
 How much are you doing?
 How many are you reaching?
 Is change on community factors occurring?
STRATEGY: MULTI-LEVEL CAMPAIGN TO INCREASE KNOWLEDGE ABOUT HARMS OF SOCIAL ACCESS AND
WAYS TO PREVENT SOCIAL ACCESS TO RX OPIOIDS
Community Factor: Lack of community
knowledge among adults about harms
associated with easy access to Rx opioids
Community Factor: Lack of knowledge
among adults about how to prevent social
access to Rx opioids
KEY STRATEGY OUTCOMES
Multi-level campaign to:
Intervening Variable:
Social Availability of Rx opioids
1.
1) Increase knowledge about harms associated 2.
with easy access to Rx opioids
2) Increase knowledge about ways to reduce
social access to Rx opioids
3.
3) Increase safe storage, monitoring, and
disposal of Rx opioids
4.
(O) Outcome indicators; (P) Process indicators
INDICATORS
# and reach of awareness products
disseminated (posters, articles, etc) (P)
% of adults who have seen campaign
messages about safe storage and
disposal of Rx opioids (P)
% of adults reporting safe storage and
monitoring of Rx opioid prescriptions
(or intentions) (O)
% of adults who report disposing of
Rx opioids safely (or intentions) (O)
CSAP
Category:
Information
Dissemination
Strategy Target
Population:
Adults 18+
METHOD / MEASURE
1. Dissemination tracking
2. Quarterly random survey of 100
adults (e.g., DMV)
3. Quarterly random survey of 100
adults (e.g., DMV)
4. Quarterly random survey of 100
adults (e.g., DMV)
STRATEGIES: 1) INCREASE THE NUMBER OF SAFE DISPOSAL SITES AND
2) IMPLEMENT A CAMPAIGN TO INCREASE AWARENESS AND USE OF SAFE DISPOSAL SITES
CSAP Category
Community Factor: Lack of convenient
Intervening Variable:
and/or recognized sites for adult community Social Availability of Rx
members to dispose of Rx opioids safely
opioids
KEY STRATEGY OUTCOMES
1.
Increase safe disposal sites to:
2.
1) Increase access to safe disposal sites
3.
Multi-level campaign to:
4.
1) Increase awareness of safe disposal sites
2) Increase use of safe disposal sites
Create safe
disposal sites:
Environmental
INDICATORS
# of disposal sites created and # of days
open (P)
# and reach of awareness products
disseminated (posters, radio ads, etc.) (P)
% of adults aware of at least one local safe
disposal site (O)
# visitors of visitors and/or pounds of Rx
opioid pills discarded at disposal sites (O)
(O) Outcome indicators; (P) Process indicators
Campaign to
promote safe
disposal sites:
Information
Dissemination
Strategy Target
Population:
Adults 18+
METHOD / MEASURE
1. Activity tracking
2. Dissemination tracking
3. Quarterly random survey of 100
adults (e.g., DMV)
4. Activity tracking
Alaska PFS Evaluation Planning
Step 3: Collect Strategy Data
(for time series graphs)
 collect data on a regular basis over time
 graph results
 Interpret results
 revise strategy efforts as needed
CF: Lack of convenient and/or recognized sites for adult
community members to dispose of Rx opioids safely
Cumulative # of safe disposal sites
that are open for use
Cumulative # of TV ads seen per
adult about locations of safe dipsosal
sites
4
3
2
10
8
6
4
1
0
2
0
*Process indicators
CF: Lack of convenient and/or recognized sites for adult
community members to dispose of Rx opioids safely
# of people who drop off
prescriptions at safe disposal sites
per quarter
% of adults who can identify at
least one safe disposal site per
quarter
100
160
80
120
60
40
80
20
40
0
0
*Outcome indicators
B
Baseline Data
Rule of thumb for PFS: Two quarters
prior to implementation
Quantitative Data Collection Ideas
Policy Change
• progress towards policy establishment
• compliance with policy
• efforts to ensure policy compliance
• # of individuals reached
Telling Stories
Quantitative Data
“head”
vs
Stories
“emotions”
“The measure of our success
will not be determined
by the number of evaluations done
or even upon the quality of the findings …
success will depend on the ability to use
evaluation findings to strengthen our efforts
and sharpen our decision-making”
U.S. Agency for International Development. (2011).
Evaluation learning from experience:
USAID evaluation policy. Washington, DC: USAID.
When choosing strategies, consider:
 relevance to your chosen CF/IV
 time needed to plan and implement
 evaluation requirements (baseline)
 budget available
 potential for community level change
 sustainability
Next cohort call: Discuss strategic planning &
strategy selection - May 22nd at 10:00am
YOUR TURN! (MULTIPLE CHOICE)
How confident do you feel in your team’s
ability to fulfill the PFS evaluation expectations?
1)
2)
3)
4)
Not at all confident
Somewhat confident
Mostly confident
Very confident
Questions?
Please use the chat box