Results from the Cancer Alliance of Texas
2015 Evaluation
August 20, 2015
Roadmap
•
•
•
•
•
Objectives
Methods
Results
Recommendations
Lessons Learned
Background and Evaluation
Methods
Charles Shumate, MPH, CHES
Research Specialist
Office of Surveillance, Evaluation and
Research
2015 Strategic Planning Focus Groups
Objectives:
1. Provide follow-up measurements for domains of coalition
functioning.
2. Explore member beliefs on coalition successes,
experiences and sustainability.
3. Offer recommendations to coalition and evaluation.
4
Coalition Domains
Empowering
Leadership
Priority Work
Plans
Value Added
Collaboration
Diversified
Funding
Effective
Communication
Dedicated
Staff
Shared
Decision Making
5
2015 Interview Guide
Questions
1. What is your role in CAT? How long have you been in this role?
2. In what ways is the Cancer Alliance of Texas (CAT) fulfilling its mission statement?
3. What, if any, expectations did you have when you first joined CAT? How have your
expectations changed?
4. If you could change anything about CAT, what would you change?
5. How would you describe your experience as a CAT member/partner? Can you provide
examples?
6. Are you engaged in one of the three new priority workgroups? If so how would you describe
how these priority workgroups are functioning?
7. Do you have suggestions about how CAT could improve the likelihood of sustainability of its
membership? How can we grow and expand our organization?
8. Do you have suggestions about how CAT can diversify its funding base?
9. Do you have suggestions about how CAT could improve the likelihood of its sustainability in
its processes or organizational culture?
6
2015 Coalition Member Survey
Evaluation
• Planned
– Member survey
– Key informant interviews (KII)
• Recruitment
– Staff were emailed in February 2015
• Member Roster
– Calls scheduled ~ 30 minutes
• Analysis
– Thematic
– Summed scores, averages
7
2015 Coalition Member Survey
Evaluation
– Overall survey participation 17% (n=23)
• CAT Meeting (n=16)
• Online (n=7)
– KII participation 57% (n=21)
– Analysis
• Quantitative
– Percentages vs. summed scores
• Qualitative
– No change to analysis plan
8
Participant Characteristics
9
Results
2015 Coalition Member Survey
Results
Empowering Leadership
Reponses(%): Empowering Leadership
Strongly Disagree
Disagree
Neutral
Our coalition maintains clear roles, responsibilities, and
procedures
Our coalition efforts are controlled by just a few people
Coalition leaders are actively involved in coalition decisions and
efforts
Agree
4
Strongly Agree
17
13
9
43
30
4 4
No response
22
13
39
26
30
13
4
13
13
11
2015 Coalition Member Survey
Results
Value-Added Collaboration
Responses (%): Value-Added Collaboration
Strongly Disagree
Disagree
Neutral
Agree
Members believe the coalition works on value-added efforts - things
that wouldn’t happen if the coalition didn’t work on them
9
17
Coalition members distrust one another and / or the leadership
Our coalition has successfully maintained or increased its credibility
Our coalition’s efforts do not translate into meaningful influence in
the larger community
Strongly Agree
35
26
48
4
26
30
No Resonse
17
22
17
13
9
35
39
9
13
13
9
9
12
2015 Coalition Member Survey
Results
Effective Communication
Responses (%): Effective Communication
Strongly Disagree
Disagree
Neutral
Meetings are didactic and mostly focus on reports
Members feel their voice is heard and have channels to
communicate with coalition leadership
Several modes of communication (such as websites, emails,
newsletters and calls) are used to communicate to members
There is little and irregular communication from the coalition
leadership to its members
Agree
13
4
Strongly Agree
39
13
4 4
22
48
13
13
22
52
26
No Response
13
26
48
13
9
4
13
13
2015 Coalition Member Survey
Results
Shared Decision Making
Responses (%): Shared Decision Making
Strongly Disagree
Disagree
Neutral
Decisions are made based on personal preferences and the
loudest voices
It is understood how decisions are made
Decisions are made by only a few people
Agree
17
4
13
13
Strongly Agree
No Response
26
26
13
17
26
26
13
17
35
13
26
13
14
2015 Coalition Member Survey
Results
Dedicated Staff
Responses (%): Dedicated Staff
Strongly Disagree
Disagree
Neutral
The roles and responsibilities of staff are unclear/ not communicated
Dedicated staff is sufficient and effectively coordinated to make progress
Agree
Strongly Agree
17
13
48
43
No Response
4
9
30
4
17
13
15
2015 Coalition Member Survey
Results
Flexible Structure
Responses (%): Flexible Structure
Strongly Disagree
Disagree
Neutral
Agree
Our coalition structure (workgroups, team leads, etc.) has not changed
since our planning days
Our coalition develops specific roles and responsibilities for members
based on their interests and skills
New members are welcomed and effectively oriented to the group
22
9
26
4
9
43
35
17
26
43
17
22
9
48
26
9
4
48
13
Members are recruited haphazardly
No Response
26
17
Our coalition works largely in isolation of the community
Our coalition members are motivated and inspired
Strongly Agree
17
35
13
4
17
4
17
17
16
2015 Coalition Member Survey
Results
Clear Roles and Accountability
Responses (%):Clear Roles and Accountability
Strongly Disagree
Disagree
Accountability is valued and evident because the vast majority of
members follow through with assignments and meet deadlines
The roles of members and workgroups are unclear and not
communicated
Coalition meetings are perceived as unproductive
Neutral
Agree
9
Strongly Agree
17
22
26
No Response
39
13
39
13
48
22
9
17
4 4
17
17
2015 Coalition Member Survey
Results
Priority Work Plans
Responses (%): Priority Work Plans
Strongly Disagree
Disagree
Neutral
Members are involved in developing plans to implement
strategies and these plans are written with tasks, timelines, and
responsibilities
The coalition’s efforts are focused and broad enough that
members find a place for their interests and skills
The coalition takes actions that are not related to priorities of the
Texas Cancer Plan
13
4
Strongly Agree
4
48
13
13
Coalition members lack a clear understanding of the priorities of
the coalition
Coalition priorities from the Texas Cancer Plan are set and
periodically reviewed and revised
Agree
43
22
43
22
9
13
13
48
57
13
13
17
18
2015 Coalition Member Survey
Results
Diversified Funding
Responses (%): Diversified Funding
Strongly Disagree
Disagree
Neutral
Agree
We have resource plans and seek out new funding opportunities
9
The resource needs for our implementing our coalition efforts are
not known
9
We have a diversified funding base
13
Strongly Agree
17
No Response
39
17
39
13
35
48
17
17
4
22
19
2015 Coalition Member Survey
Results
Other Issues
Responses (%): Other Issues
Strongly Disagree
Disagree
Neutral
My “home” organization has benefited from the Cancer Alliance
of Texas
The coalition is addressing the disparity in cancer control services
in Texas
Agree
4
13
9
Strongly Agree
13
9
26
48
No Response
35
21
9
13
20
Take Aways From KIIs
Question
Themes
Illustrative Quotes
Fulfilling Mission
•
•
•
•
•
Products
Awareness
Connections
Networking
Texas Cancer Plan
“It [CAT] has helped me stay focused to make decision
about the direction of my own work so that [it] is better
aligned with the priorities set forth in the plan [TCP].”
Expectations
•
•
Need for awareness
Networking
opportunities
Work requirement
Inherent value
“To learn what others were doing and opportunities to
collaborate, to help foster efforts towards to cancer
control.”
Transparency
Location
CPRIT
Increase
engagement
Topics
“[How are] meetings scheduled. There are no doodle
polls. [It is like] ‘this is when they are’. That is frustrating.”
•
•
Changes to
CAT
•
•
•
•
•
“I do wish they [CAT] were a bit friendlier at remote
attendance. [You] can’t always make it in person and to
attend in remotely, not conference, but being able to
watch slides on a computer and be more [engaged].”
21
Continued
Question
Member
Experience
Themes
•
•
Rewarding
Tension
Direct Quotes
“Very fulfilling, glad to be engaged and good to work with
people. None of these people are paid people.”
“It kind of lacks engagement sometimes. [Occasionally]
there is a meeting; it doesn’t make me want to go. I
missed the last meeting. There is no hook to get me to
go. That is how it has impacted my participation.”
Benefits of Priority
Work Groups
(PWGs)
•
•
They are working
Needs
improvement
“It’s going well given the time crunch because everybody
is a volunteer. We don’t get paid.”
Benefits of
PWGs
•
•
Implement TCP
Opportunity to
Contribute
Produce
deliverables
“The idea is to move the model. To implement the plan.”
“Get some concrete time where there can be discussion
and planning on how to move the needs and actually
strategize the use of CATs resources to accomplish
[things].”
“Again, providing additional tools for people, to get best
practices to patients and technical assistance for their
home organization.”
•
22
Continued
Question
Membership
Themes
•
•
•
Direct Quotes
Provide
meaningful
engagement
Quality time
Not a problem,
membership is
strong
“Everybody has to make hard decision about spending time. If I
am getting something out of things and my contributions is
important, [I am] more likely to attend and more likely to stay.”
“Need to go back to dedicated time for members. Not just
business updates. I don’t know what that would look like [maybe
a] once a year kind of thing.”
“[This topic is a] broken record. We have sustained interest by
good people who are doing their part. I don’t think we are losing.”
Funding
•
•
•
501c3
DSHS
Grants
“[This is an ongoing] discussion topic. Unless we choose to be
something that we are not. Should we become 501c3? Should we
have a foundation? Once you get into that, that Jockeying
between doing the mission and seeking funding. We need to
decide who we are and what we can do. Form follows function.
We would have to have a big change in overall perceptive
[regarding 501c3 status].”
“Department of State Health Services needs to give more money.
DSHS should have a bigger buy in.”
Culture
•
•
Transparency
Maintain status
quo
“And yeah would not hurt to review bylaws. You know the roles
different seats [executive committee] hold. Often times they
[members] don’t want to run and are afraid to nominate someone
since they don’t know right person to nominate to run. Those
kinds of things.”
“How reflective how the organization is, how will CAT learn from
this evaluation of the organization. How and what extent will the
organization look at results and make changes related to the
results? That is very important in terms of sustainability.”
23
Lessons Learned
Results Support
• CAT is fulfilling its mission and provides members
with important networking opportunities and
information on cancer control efforts at the state and
national level.
• CAT needs to increase coalition transparency. Idea
was expressed by leadership and rank and file
membership alike. Transparency was woven
throughout interviewee responses.
25
Results Support
• CAT members identified many accomplishments in
both the survey and interviews, it became apparent
that CAT lacks a process to document its
successes, promote itself to the external
community, or internally to new members.
• CAT should increase opportunities for rank and file
members to have a voice in meeting activities,
diffusion of workgroup responsibilities, and sustain
post quarterly meeting engagement using the
existing communication channels.
26
Results Support
• Review CAT mission, goals and bylaws
• Provide members “member time” at CAT
meetings
• Document and share CAT
success/accomplishments
• Utilize breakout sessions to address member
driven topics
27
Evaluation Process
Limitation and mitigation to existing data validity threats
Limitation
Data accuracy
Unknown data elements
Selection bias
Confidentiality
Small sample size
Solutions to minimize
Assess reasonableness of the data
Focused on percentage rather than
changes in absolute values
Ensuring the sample was appropriate
Obtain de-identified data
Focus on data as illustrative of case
study vs. in-depth/inferential statistics
• Focus the member and partner survey to a few domains
that are of CAT and DSHS interest.
• Explore evaluation of CAT networking strength
28
Discussion Points
• What surprises you about results?
• What doesn’t surprise you?
• What other information about the evaluation would
be useful to you?
29
Contact Information
Charles Shumate, MPH, CHES
Research Specialist
[email protected]
Phone: 512-776-2522