An Ounce of Prevention…. Practical
Approaches to Evaluating
Community Health Initiatives
Liz Radel Freeman – [email protected]
Chera Sevcik – [email protected]
The Improve Group
• Evaluation, research, strategic planning, and capacitybuilding
• Data-driven to answer questions that will help leaders
make confident decisions
• Coaching for programs to do internal evaluation and
research
• Local (Minnesota), National, and International clients
• Blog and other resources at:
www.theimprovegroup.com (Facebook and Twitter)
www.theimprovegroup.com
Faribault, Martin and Watonwan
Counties
www.theimprovegroup.com
Faribault, Martin and Watonwan
Counties
• Statewide Health Improvement Program
(SHIP) grantee
• Serving diverse, rural communities
• Strategic target areas:
•
•
•
•
School nutrition and physical activity
Community nutrition and physical activity
Comprehensive worksite wellness
Healthcare referrals
• Community-led implementation of strategies
through mini-grant process
www.theimprovegroup.com
What good is a bike rack
without a biker?
Tools for measuring health behavior changes that
follow systems, policy, and environment changes
Overview/Introduction
• Community-level changes around
nutrition and physical activity
• Challenges around collecting
behavior change data
Photo by Matthew Hankey,
Sun Newspaper, June 30, 2011
From Burnsville Patch
newspaper, April 9, 2011
www.theimprovegroup.com
Tools
• Bike rack tally
• Amenity survey – users of bike
racks and wayfinding signs
• Community gardener survey
• Food shelf survey
www.theimprovegroup.com
Bike Rack Tally
Purpose: observe use of newly installed
bike racks
www.theimprovegroup.com
Bike Rack Survey Findings
• Initial findings showed limited use of
bike racks
• Observers felt this was due to
weather and the recent installation of
bike racks
www.theimprovegroup.com
Amenity Survey
Purpose: to learn about the use of newly
installed bike racks and wayfinding signs
• Signs were posted at
the location of the new
bike racks and signs
• Convenient for user
• Incentive of $15 gift
card
• Quick response
www.theimprovegroup.com
Amenity Survey
www.theimprovegroup.com
Amenity Survey Logistics
•
Preparing signs and materials
•
•
•
Use landscape and portrait orientations
Laminate signs
Bring a variety of fasteners
•
Good map needed to place signs
•
Relatively low cost
•
Immediate response from cell phones
•
QR code option in future
www.theimprovegroup.com
Amenity Survey Sample Questions
1. When did you first start to use the
amenity?
2. How often would you say you use the
amenity?
3. What does having this resource available
in the community mean to you?
4. Does having this amenity available change
your habits or help you have a healthy
lifestyle?
www.theimprovegroup.com
Amenity Survey Findings
Respondents indicated bike racks:
• Are convenient and secure
• Encourage participants to get more exercise
• Send a message the community encourages
physical activity
• Make it easy to stop at local parks
www.theimprovegroup.com
Gardener Survey
Purpose: to learn about behavior changes of
gardeners in SHIP community gardens
• Surveys were distributed by garden
coordinators in 9 gardens at the end of
the growing season
• 54 adult gardeners and 51 youth
gardeners responded (17% response rate)
Photo by Mike Shaughnessy,
from Sun Newspaper, June
16, 2010
www.theimprovegroup.com
Gardener Survey
www.theimprovegroup.com
Garden Survey Logistics
• Administered by garden
coordinators at the end of growing
season
• Envelopes available to help protect
respondent confidentiality
• Spanish and English surveys
available
www.theimprovegroup.com
Garden Survey Findings
8 in 10 gardeners indicated:
• Working in the garden helps them be healthier
5 in 10 gardeners indicated:
• They eat more fruits and vegetables after
participating in the garden
www.theimprovegroup.com
Food Shelf Recipient Survey
Purpose: to better understand how food
shelf recipients used the fresh fruits and
vegetables they received.
• 112 people responded to the survey
www.theimprovegroup.com
Food Shelf Recipient Survey
www.theimprovegroup.com
Food Shelf Recipient Survey Logistics
• First step – commitment from food
shelves
• Tool available in paper format or inperson interview options
• Very brief survey
Photo by Rick Orndorf, from This
Week newspaper, May www.theimprovegroup.com
27, 2010
Food Shelf Recipient Survey Findings
• Nearly all (96%) participants would
select fresh fruits or vegetables if
available
• The process uncovered barriers to
using fresh fruits and vegetables
available at food shelves
www.theimprovegroup.com
Lessons Learned
• Measuring behavior change from
community-level interventions takes time
and creativity
• Minimizing burden helps you get the data
you need
• Tools must be brief and convenient to use
www.theimprovegroup.com
A chance to shine, a chance to
learn
Using case studies in Faribault, Martin and Watonwan
Counties to document, inform and inspire
Goals of case study approach
Document
Inform
Inspire
www.theimprovegroup.com
Selecting cases
Diversity in site characteristics and experiences
• Settings: schools, community organizations,
healthcare facilities and worksites
• Process: collaboration, empowering specific
populations
• Characteristics: size, populations
• Experiences: successes, challenges, barriers
www.theimprovegroup.com
Two approaches for the case study
All cases studies focused on the process of making changes
and the resulting behavior changes in individuals
Targeted settings:
• School Nutrition: 3 school districts
Work groups:
• Latino Council
• Senior Voices Council
• Active Living Coalition
www.theimprovegroup.com
Data Collection
Challenges to using the case study approach
• Time
• Budget
• Burden
Targeted settings:
• Half-day site visits
• Interviews during and outside of visit
Work Groups:
• Attended meetings and events
• Interviews
www.theimprovegroup.com
Targeted Settings: Schools
Gathering data
• Interviews: School nurses (grant managers),
Food Service Director, grant coordinator
• Document review: site work plans, previous
evaluation information collected from sites,
policy drafts
• Observations: physical space, activities
www.theimprovegroup.com
Targeted Settings: Schools
Reporting
• Story of the site – changes made, behavior changes
• Common lessons learned – challenges and successes
From Rosemount Town Pages newspaper, April 29, 2010
www.theimprovegroup.com
Targeted Settings: Schools
What we learned – part 1
• Wellness champions are needed to
move changes forward
• Changes are time-intensive
• Value added by working across
districts
www.theimprovegroup.com
Targeted Settings: Schools
What we learned – part 2
• Changes that are supported and
doable will go far
• Administrative support helps create
change
• Involve diverse perspectives to win
people over
www.theimprovegroup.com
Work Groups: Active Living Coalition
Gathering data
• Interviews: Community Education Director
(Coalition Chair), grant coordinator
• Document review: previous evaluation reports,
walkability study
• Observations: meetings, events
www.theimprovegroup.com
Meeting Observation Protocol
Record descriptions
•
•
•
•
•
•
Members
Location
Format
Objectives
Discussion topics
Team functioning
‒ Leadership
‒ Participation and representation
‒ Organization and structure
‒ Communication
‒ Results and Actions
www.theimprovegroup.com
Work Groups: Active Living Coalition
Reporting
• Story of meetings
• Accomplishments
• What they learned through this process
www.theimprovegroup.com
Work Groups: Active Living Coalition
What we learned
• Big changes take time, so have
a plan in place to seize opportunities as they arise
• Address resistance with education
• Involve the right people
• Move forward on what you can
Photo by Rick Orndorf, from This Week
www.theimprovegroup.com
newspaper, October 6, 2010
Work Groups: Senior Voices Council
Gathering data
• Interviews: Community Education Director
(Council Chair), grant coordinator, Council
members
• Document review: previous evaluation
reports, previous meeting minutes and
agendas, promotional materials
• Observations: meetings, Senior Expo event
www.theimprovegroup.com
Work Groups: Senior Voices
What we learned – part 1
• Building a sustainable council
takes time
• The right facilitator will help a new
council flourish
• Shared leadership results in
member buy in
www.theimprovegroup.com
Work Groups: Senior Voices
What we learned – part 2
• Start with an area people care
about and build on it
• Allow people to participate in the
way they feel most comfortable
www.theimprovegroup.com
Work Groups: Latino Wellness Council
Gathering data
• Interviews: Grant manager (Council Chair),
grant coordinator, Council member
• Document review: policy and proposal,
previous evaluation reports
• Observations: meeting
www.theimprovegroup.com
Work Groups: Latino Wellness Council
What we learned – part 1
• Focus efforts on what moves the
community
• Understand cultural perspectives on
prevention
• Facilitator must bridge between cultures
www.theimprovegroup.com
Work Groups: Latino Wellness Council
What we learned – part 2
• Capitalize on opportunities for change
• Staff support needed
• It takes time to build a council
• Make participation convenient
www.theimprovegroup.com
Full case study reports
Faribault, Martin, and Watonwan Counties SHIP
Reports can be found at http://tinyurl.com/3o6zn5v
• Latino Wellness Council Case Study
• Active Living Blue Earth Case Study
• School Nutrition Project Case Study
• Martin County Senior Voices Case Study
www.theimprovegroup.com
Discussion questions
1.
2.
3.
In what ways have you struggled to show the
impact of your health prevention programs?
How could you apply these strategies to your
work?
Do you have ideas for disseminating the case
studies to other health practitioners?
www.theimprovegroup.com
Contact Info
Liz Radel Freeman, MPH:
Email: [email protected]
Website: www.theimprovegroup.com
Phone: (651) 315-8919
Chera Sevcik, MS:
Email: [email protected]
Website: shipfmw.blogspot.com
Phone: (507) 236-5376
www.theimprovegroup.com