Got 10 Weeks? Try The Short-Term
Impact Evaluation of The Annual
Tobacco Treatment Conference
Natalie Kuhn, Lori Travis
EERS 40th Annual Conference-2017
April 3rd at 1:45p
Natalie Kuhn
• Senior at Hamilton college,
major in neuroscience
• Summer internship at
MaineHealth-2016
• Research experience: 4 years
• Evaluation experience: 1 year
• Coffee: Starbucks
• Cool fact: semester in
Denmark
2
Lori Travis
• Masters in Statistics
• Program Manager at
MaineHealth-CTI
• Research experience: 15 years
• Evaluation experience: 3 years
• Preferred coffee: Speckled Ax
• Cool fact: rappelled down a 13story building in Portland, ME
3
Quick Poll
• How many years conducting • When out and about where do
program evaluation?
you typically stop for “coffee”?
- <1
- Dunkin Donuts
- 1-4
- Starbucks
- 5-9
- Convenience Store
- 10-14
- Other
- 15+
- Don’t do coffee
4
Overview
• Background
- Tobacco Use in ME
- MaineHealth-CTI
- Maine’s Annual Tobacco Treatment Training Conference
• Evaluation over the 10-weeks: 6/6/16 to 8/12/16
- Methods
- Findings
- Recommendations
• Questions
5
Background: Maine Tobacco Use
• Maine tobacco prevalence- 19%
• Spending on health care costs directly
caused by smoking- $811 million
• Disparities based on social
determinants of health (2014)
- MaineCare adults reported using
cigarettes- 48%
- Non-MaineCare adults- 17%
• Want to quit- 60%
Sources: BRFSS, Campaign for Tobacco-Free Kids; The Toll of
Tobacco in Maine, Maine CDC
6
Background: MaineHealth Center for Tobacco
Independence (CTI)
• Sponsored by the Maine CDC since 2001
• Maine Tobacco Helpline: 1-800-207-1230
• Training and Education Programs
-
Basic and Intensive Skills Trainings
Webinars
Ad-Hoc Trainings
Annual Tobacco Treatment Training Conference
Clinical Outreach Sessions
• Tobacco Use Prevention since 2017
7
Background: The Conference
• Theme:
- Addressing Disparate Populations
• Goal:
- Increase statewide ability to treat tobacco
users with evidence-based methods and
reduce tobacco use
• Objectives:
- Bring together tobacco treatment
community in Maine
- Provide training as part of certification
process
8
Timeline: 10 weeks- June 6th to August 12th
Week 1:
met
team,
describe
Week 2:
plan,
LM
Week 3:
draft
survey
Week 4:
pilots,
IRB
Week 5:
sent
survey
Week 6:
1 ppt
FG
Week 7:
expert
FG
Week 8:
stats
Week 9:
share
results
Week 10:
final
report
9
Activity: Evaluation of a Donut (Sorry!)
• Identify objectives
• Make a logic model
• Set goals of evaluation
• Design a survey
• Hold focus groups
• Analyze data
• Report to stakeholders
• Future use recommendations
10
Methods: Logic Model
11
Methods: Survey
• Objectives
- Measure effectiveness on short and intermediate term outcomes
- Asses attitudes on logistics and networking
• Designed a short-tem impact survey
- 46 questions, 5 Sections:
» knowledge gained, behavioral changes, networking, logistics
- IRB reviewed, pilot tested and cognitive interviews
- Emailed via REDCap with incentives 6 weeks post-conference
• Analysis
- Descriptive statistics, chi squares, t-tests, and ANOVAs
- SPSS software
12
Methods: Focus Groups
• Objectives
- Gain qualitative context to survey
- Learn about planning, marketing, and education strategies
• Planned and moderated 3 sessions
- 1-hour long, 4 sections:
» knowledge gained, behavioral changes, networking, logistics
- Held one CTI focus group: participants only (10)
- Held two educational events expert focus groups:
» MaineHealth (8 invited, 6 attended) & CORE (4)
• Analysis
- Transcribed audio recordings
- Coding using MAXQDA software
13
Findings: Responses
• Response rate overall- 59%
• Felt the purpose of conference had been achieved- 100%
• Significant clinical behavior changes:
- Using 5As, CBT, MI, intervention integration
- MTHL referrals, motivating patients, NRT prescription
• Significant non-clinical behavioral changes:
- Educating coworkers/peers, address substance use consistently
- Conscious choice of nonjudgmental language
• Made changes in practice policy- 15%
• Breakouts preferred on all conference days- 83%
14
Findings: Responses
• Respondents reported
interacting with tobacco users
in their professions- 50%
• Math: to see the effect of the
conference trained attendees…
- Tobacco user interactions
weekly: 260-308
- Tobacco user interactions since
the conference: 2,080-2,464
- Tobacco user interactions
annually: 13,520-16,016
Number of
Interactions
Frequency
Percent
1 to 5
5
20
6 to 10
6
24
11 to 15
1
4
16+
13
52
Total
25
100
Amount of interactions with tobacco users
participants report having on a weekly basis
15
Findings: Knowledge Gained
4
3.5
3
2.5
2
1.5
1
0.5
0
Before
After
16
Findings: Clinical Behaviors
5
4.5
4
3.5
3
2.5
2
1.5
1
Before
After
0.5
0
17
Findings: Non-clinical Behaviors
4.5
4
3.5
3
2.5
2
1.5
1
0.5
Before
After
0
18
Findings: Networking
• Participants networked at
conference- 90%
• Connection follow-up- 40%
• Dedicated 1-2 hours- 74%
• Planning to attend next years
conference- 88%
• Qualitative responses
- Networking room with attendee
information, cocktail hour
- More breakouts
Value of networking responses
19
Findings: Participant Focus Group
 Knowledge Gained
» “I didn’t really know anything and now I
feel I have a greater empathy their struggle
and I am more aware of that.”
 Behavioral Changes
» “I was already doing it every day (CBT)
but I am doing it better because of the
conference.”
 Networking
» “The social activities were fun and a
better opportunity for networking”
 Logistics
» “For me it (learning) had a lot to do
with the speaker- it sticks in my
mind when I’m really engaged and
curious.”
20
Findings: Expert Focus Groups
 Planning
» “And you have to be bold and assertive
enough, even with national speakers to say
this is not the right direction or this is too
long and really work with them over time”
 Marketing
» “If you cast the net too broadly is you now
all of a sudden are reaching individuals who
are no interests in the particular event…If
you do that too frequently, you end up
diluting your brand”
 Training Techniques
» “The most effective kind of presentations
I’ve seen are kind of a combination of
winning hearts and minds so some data
and some heart stories and that can be
really powerful.”
 Networking
» “I definitely think networking is huge and
it’s a huge benefit to attend any meeting or
conference. The more built in
opportunities there are for that to happen,
from my perspective as an attendee it is
very valuable.”
21
Limitations
• Survey
- Immediate feedback survey (2 days post) was not designed for
comparison
- Relied on participant recollection of knowledge and behavior
- Online distribution method only
• Focus Groups
- Disorganization of focus group recruitment methods
- Limited time and resouces
» Lack of planning
» Lack of saturation in responses
22
Recommendations
1. Continue education on disparate populations in Maine
2. Spend ample time selecting and preparing speakers
3. Appeal to the hearts and minds of event participants
4. Include time networking (formal and informal)
5. Integrate technology into all areas of training
6. Continue evaluation
23
More than a survey?
24
What’s next?
• Take Away
- Maine’s Annual Tobacco Treatment Training Conference-2016 was
highly successful
• Next Steps
- Use of results at the next annual conference in May 2017
- Summer 2017: Hire a new summer intern to implement the 2nd
annual “Got 10-Weeks” evaluation project
25
Questions?
26
Contacts
Natalie Kuhn
Hamilton College
- [email protected]
Lori Travis
MaineHealth-CTI Evaluation and Reporting
- [email protected]
27