BUILDING A STRONGER PQI PROGRAM WITH OUTCOME
MEASUREMENT & COA SELF‐STUDY AND SITE REVIEW
KENIA CRIMINALE, LCSW LIFELINES COUNSELING CENTER, MOBILE, ALABAMA
ROCHELLE HAIMES, ACSW
CONSULTANT, PITTSBURG, PA
Presentation Pre‐Test
Instructions: Please check the best answer choice for each item. My knowledge of the following: 1. Definition of an outcome
2. The importance of measuring outcomes
3. What/How/When to measure outcomes
4. Limitations/obstacles
5. Who should be involved
6. How to write surveys
7. How to collect the data
8. What to do with the results
9. How to get started measuring outcomes
10. What a Peer Reviewer will be looking for in a Self‐Study
11. What a Peer‐Reviewer will be looking for at
a Site Study Excellent
Good
Fair
Poor
Agenda
What are we going to talk about today? Item 1
Pre‐Test
Item 8
Writing Item 2
Training Objectives
Item 9
Collecting Data
Item 3
Define Key Terms
Item 10
Bringing it all Together
Item 4
Why Measure Outcomes
Item 11
Communicating Results Item 5
Who is Involved
Item 12
COA Self‐Study
Item 6
Obstacles
Item 13
COA Site Visit
Item 7
What‐How‐When to Measures
Item 14
Post‐Test
Training Objectives
•
Gain basic understanding of outcome measurements in order to implement a stronger PQI program.
•
Learn how to keep a running score of how your programs are doing.
•
Be more prepared for the accreditation process.
How Do We…
•
Show our donors that we are using their contributions effectively?
•
Show our board that we are holding ourselves accountable for results?
•
Demonstrate that we are fulfilling our mission?
•
Know we are making a difference?
Outcome Measurement Defined
The process of assessing the results of an agency’s programs for its participants, stakeholders and the community.
What are Outcomes? •
Outcome
•
Benefits or changes for individuals and/or populations during or after participating in program activities.
•
May relate to knowledge, attitudes, values, skills, behavior, condition or other attribute.
•
The result of a service.
More About Outcomes
OUTCOMES INCLUDE…
•
•
•
•
•
•
•
•
•
Symptom reduction
Change in clinical status
Change in functional status
Health, welfare and safety
Permanency in life situation
Behavioral change
Change in psychosocial skills
Change in knowledge
Service quality—accessibility, availability, efficiency, continuity, safety, timeliness, and respectfulness
AN OUTCOME IS NOT…
•
•
•
•
•
Number of clients served
Quantity of services, programs or activities provided
Number of volunteers
Amount of donations received
Client satisfaction
Why Measure Outcomes? •
•
•
•
•
•
•
•
•
•
COA
Builds public confidence
Encourages improvement of agency services
Funders Fundraising
Keep focus
Increase accountability
Improve efficiency
Improve performance
Increase achievement of program goals
•
•
•
•
•
Reinforce mission and vision
Improve client satisfaction and outcomes
Give competitive edge
Build client confidence
GREAT FEEDBACK!
“Quality is Everyone’s Responsibility”
‐‐W. Edwards Deming
Stakeholders Involved In Outcome Measurement
•
Executive Director
•
Board of Directors
•
Management
•
Staff
•
Clients
•
Community stakeholders
Ways Stakeholders Can Be Involved
•
•
•
•
•
•
•
•
•
Give input on outcomes
Participate in PQI committee
Review reports and provide feedback
Complete pre/post tests and surveys
Help identify positive practices
Recommend improvements when necessary
Work with the Board of Directors and Management on short‐ and long‐term planning
Administer measurement tools and collect data
Administer/tally surveys
Limitations/Obstacles to Outcome Measurement
•
Cost
•
Time‐consuming
•
Reliability/Validity
•
Some outcomes more difficult to measure
•
Getting buy‐in
•
Sensitivity of a client’s situation
•
Outcome levels over time
Getting Started Measuring Outcomes
•
Identify key PQI staff member
•
Get major players committed
•
Get staff committed
•
Educate
•
Utilize PQI committee
•
Develop an action plan and schedule for implementation
•
Allocate resources
•
Identify programs/areas where needed the most
Measuring!!
•
•
Keep with Mission and Vision
What does each program/staff want to accomplish?
•
•
•
•
•
•
•
Knowledge enhancement
Skill development
Attitudinal change
Behavioral change
Situational change
Research
What information do the funders want?
•
•
•
•
•
Resources available
Research
Type of measure Designate who is responsible
Collect in such a way that can easily and pleasantly show results
•
•
•
•
During or immediately after a service provided
Short‐term
Long‐term
Drop‐outs
Writing Surveys/Pre‐ and Post‐tests
Types of information to be included:
•Information on the client’s behavior and/or condition and the client’s perception of the degree of improvement
•Ratings of service timeliness, accessibility of staff and service facility, condition and safety of facilities
•Overall satisfaction with the services provided
•Reasons for any poor ratings given
•Suggestions for improvement
Identify Key Characteristics to Link to Outcome Information:
•Gender
•Age Group
•Race/ethnicity
•Income group
Other considerations:
•Remember population
•Make sure questions relate to what you are trying to measure
•User‐friendly
•Keep it as short and simple as possible
•Closed‐ ended vs. open‐ended questions
•Trial and error •Don’t reinvent the wheel
•Type of disability
•Educational level
•Housing type
Collecting Data
•
Who will collect?
•
How will information be collected?
•
When collected?
•
How will results be aggregated?
Pulling It ALL Together
•
Who gets the data and generates the reports?
•
Making the reports user‐friendly and pleasant
•
How often are reports generated?
Communicating Results •
•
•
•
Quarterly reports
Yearly reports‐compilation of quarterly reports
Annual report
Employees
•
•
•
•
New staff/intern orientation
Agency staff meetings
Department staff meetings
Newsletter
•
•
•
•
•
•
•
•
PQI committee
Reports to United Way
Reports to funders
Board members
Program brochures
Partnering agencies
Agency Lobby
Website
Results of Reports Are Used To…
•
Build upon strengths
•
Eliminate or reduce identified problems
•
Determine possible causes when issues or concerns are revealed
•
Ensure good practice standards
•
Implement and monitor the effectiveness of corrective action
•
Make recommendations and plan of improvement
•
Improve services
WHAT A PEER REVIEWER WILL BE LOOKING FOR IN A SELF‐STUDY
WHAT A PEER REVIEWER WILL BE LOOKING FOR IN A SITE VISIT
Presentation Post ‐Test
Instructions: Please check the best answer choice for each item. My knowledge of the following:
Excellent Good
1. Definition of an outcome
2. The importance of measuring outcomes
3. What/How/When to measure outcomes
4. Limitations/obstacles
5. Who should be involved
6. How to write forms 7. How to collect the data
8. What to do with the results
9. How to get started measuring outcomes
10. What a Peer Reviewer will be looking for in a Self‐Study
11. What a Peer‐Reviewer will be looking for at a Site Study 12. Overall satisfaction with presenter #1
13. Overall satisfaction with presenter #2
13. Overall satisfaction with presentation
14. Strengths of the presenters/presentation:
15. Weaknesses of the presenters/presentation Fair
Poor
Pre‐Test
Post‐Test
Outcome
Sample Item:
My knowledge of the importance of measuring outcomes. Pre‐Test
Post‐Test
“EVERYTHING CAN BE IMPROVED.” ‐ C.W. Barron
THANK YOU!