The RRFSS Evaluation Framework 2005
Background Information
and Progress to Date
Presenters
 Amira Ali
 Jane Hohenadel
 Kate O’Connor
 Deborah Radcliffe
June 22, 2005
1
Group Members


RRFSS Evaluation Group:
Amira Ali, Elaine Hector, Jane Hohenadel, Irene
Koren, Kathy Moran, Karen Moynagh, Lynne
Russell, Louisa Wong
PHRED RRFSS Evaluation Group:
Jane Hohenadel, Adriana Newbury, Kate
O’Connor, Louise Picard, Deborah Radcliffe
2
Outline of Presentation




Link to strategic planning session
Link to previous evaluation (2002)
Evaluation Framework – RRFSS Evaluation
2005
Tasks and Timelines
3
History




Project was identified in March 2004
Proposal developed in May 2004 and an Evaluation
committee was formed with representation from each
of the RRFSS regional working groups
The PHREDs were identified as partners in the project
– proposal submitted to them in October 2004
Terms of reference for the RRFSS group and the
PHRED group were finalized in December 2004
4
Link to Strategic Planning Session
5
Strategic Planning Session

Strategic Planning Session in December 2004
• To develop strategic statements (vision,
mission, values, goals, strategies) that will guide
the collective RRFSS work over the next 3
years
• To determine the specific objectives and
activities for the next year
• To create an action document summarizing the
directions set
6
Strategic Planning Session


Vision: RRFSS envisions that all decisions
within the public health system to promote and
protect health and wellbeing and prevent
adverse health events are informed by valid,
timely and relevant health intelligence
Mission: Ontario’s RRFSS is a flexible, timely
and responsive surveillance system designed to
meet local Public Health intelligence needs and
address information, geographic and timerelated data gaps
7
Strategic Planning Session

Four strategies were identified:
1. Advocacy
2. Communication
3. Governance
4. Quality Improvement and Development
8
Strategic Planning Session

Strategy: Quality Improvement and
Development
Priority Activity: “Conduct the RRFSS 2005
Evaluation to identify areas of the surveillance
system in need of improvement”
The full report “Strategic Plan for Action” can be
viewed at: www.rrfss.on.ca
9
Link to RRFSS Evaluation 2002
10
RRFSS Evaluation 2002



Covered first year of operation (2001)
Number of health units ranged from 6 to 12 during the
evaluation period
Purpose:
 Inform change to system procedures
 Contribute to continuous quality improvement
 Act as a baseline and template for future evaluations
The full evaluation report can be viewed at:
www.rrfss.on.ca
11
RRFSS Evaluation 2002

Evaluation Framework covered 4 areas:
 Process Effectiveness
 Collaboration Effectiveness
 Usefulness
 Cost Effectiveness
12
RRFSS Evaluation 2002

Performance criteria linked to following
outcomes expected from RRFSS:
 Increased local awareness of surveillance
information
 Increased involvement in generating
appropriate, accurate information
 Increased practice of evidence-based program
planning
 Demonstrated or potential cost benefit due to
RRFSS
13
RRFSS Evaluation 2002
Fourteen Key Issues Identified
(grouped as below for 2005 evaluation)



Local Resources (2)
Centralization Issues
• Provincial Implications (3)
• Implications for ISR (4)
Process Considerations (5)
14
Key Issues from 2002 Evaluation
Local Resources

Adequate staff/resources for data analysis and
making data useable

RRFSS coordinators overwhelmed by work at
the local level
15
Key Issues from 2002 Evaluation
Centralization – Provincial Implications



Need for stability and support for the central
work of the RRFSS partnership
Sustaining funding for existing and new
participants (smaller health units in particular);
includes funding for survey administration and
analysis
Provincial representation of data
16
Key Issues from 2002 Evaluation
Centralization – Implications for ISR




Increasing the sample size for a health unit
Access to ISR, decision-making, workload and
resources
Improved timeliness of data from ISR
ISR Ability to handle increase in HU
participation
17
Key Issues from 2002 Evaluation
Process Consideration Issues





Increasing use of data in program planning
and evaluation in the health units
Limited new question/module capacity under
current questionnaire
Process for revising old modules and
developing new modules
Upgrading RRFSS website
Sharing data among organizations
18
RRFSS Evaluation 2005
19
2005 Evaluation Framework
Objectives
1.
2.
3.
To document progress made in the key areas
identified in the 2002 Evaluation
To document some of the “best practices”
used by Health Units
To envisage what it would take to get RRFSS
on board for all Ontario Health Units and how
it could work as a provincial system
20
2005 Evaluation Framework
Refinement of the Key Issues


To address the evaluation objectives, we began
with what were identified as Key Issues in the
2002 Evaluation
Evaluation group members were asked to review
each of those 14 Key issues, and:
•
•
•
report on progress
whether it was still an issue now
if so, how important it was for the 2005 evaluation
21
2005 Evaluation Framework
Refinement of the Key Issues


It was felt that seven issues had been largely addressed,
and that the 2005 evaluation needed only to document
their evolution since 2002
The issues that we felt were addressed:
•
•
•
•
increasing the sample size for a health unit
improved timeliness of data from ISR
access to ISR, decision-making, workload and
resources, and
limited new question/module capacity
22
2005 Evaluation Framework
Refinement of the Key Issues

The issues that we felt were addressed (cont’d):
•
•
•
limited new questionnaire/module capacity
under current questionnaire
process for revising old modules and developing
new modules
upgrading RRFSS website
23
2005 Evaluation Framework
Refinement of the Key Issues

The remainder were felt to still be issues for the
RRFSS partnership and should remain in the
2005 evaluation

The outstanding issues were refined, defined and
expressed as evaluation questions
24
2005 Evaluation Framework
Evaluation Questions for Outstanding Issues
Issue
Local Resources
Question
Do HUs have adequate staff and resources for
regional and local administrative tasks of RRFSS?
Data Collection Tool
Questionnaire
Source of Data
RRFSS reps, MOH of participating HU
25
2005 Evaluation Framework
Evaluation Questions for Outstanding Issues
Issue
Local Resources
Question
Do HUs have adequate staff and resources locally
for data analysis and making data useable?
Data Collection Tool
Questionnaire
Source of Data
RRFSS reps, MOH of participating HU
26
2005 Evaluation Framework
Evaluation Questions for Outstanding Issues
Issue
Local Resources
Question
Are local RRFSS reps able to meet the local work
demand for RRFSS module development?
Data Collection Tool
Questionnaire
Source of Data
RRFSS reps, MOH of participating HU, program
and research staff, RRFSS coordinator, RRFSS
Evaluation Group, ISR
27
2005 Evaluation Framework
Evaluation Questions for Outstanding Issues
Issue
Local Resources
Question
Are local RRFSS reps able to meet the local work
demand for RRFSS module development?
Data Collection Tool
Questionnaire
Source of Data
RRFSS reps, MOH of participating HU, program
and research staff, RRFSS coordinator, RRFSS
Analysis Group, ISR
28
2005 Evaluation Framework
Evaluation Questions for Outstanding Issues
Issue
Local Resources
Question
Are local RRFSS reps able to meet the local work
demand for RRFSS module review?
Data Collection Tool
Questionnaire
Source of Data
RRFSS reps, MOH of participating HU, program
and research staff, RRFSS coordinator, RRFSS
Analysis Group, ISR
29
2005 Evaluation Framework
Evaluation Questions for Outstanding Issues
Issue
Centralization Issues: Provincial Implications
Question
What resources are needed currently to support
and to ensure the stability of the provincial and
administrative work of the RRFSS Partnership?
Data Collection Tool
Questionnaire
Source of Data
RRFSS reps, MOH of participating HU, ISR,
RRFSS Coordinator
30
2005 Evaluation Framework
Evaluation Questions for Outstanding Issues
Issue
Centralization Issues: Provincial Implications
Question
What funds are required to enable continued
participation of HUs currently participating in
RRFSS and allow non-participating HUs to
join?
Data Collection Tool
Key Informant Interviews
Source of Data
RRFSS reps, MOH of all HUs, ISR, MOHLTC
31
2005 Evaluation Framework
Evaluation Questions for Outstanding Issues
Issue
Centralization Issues: Provincial Implications
Question
Is provincial representation of data a desired
product of RRFSS?
Data Collection Tool
Key Informant Interviews
Source of Data
RRFSS reps, MOH of all HUs, MOHLTC,
PHRED, Chief MOH, external partners
32
2005 Evaluation Framework
Evaluation Questions for Outstanding Issues
Issue
Centralization Issues: Provincial Implications
Question
What would it take to extend RRFSS to all
Ontario HUs?
Data Collection Tool
Key Informant Interviews
Source of Data
RRFSS reps, epidemiologists in non-participating
HUs, MOHLTC, ISR, Chief MOH, MOH of all
HUs
33
2005 Evaluation Framework
Evaluation Questions for Outstanding Issues
Issue
Process Considerations
Question
To what extent are RRFSS data being used in
PP&E in the health units?
Data Collection Tool
Questionnaire, Case Examples
Source of Data
RRFSS reps, program and research staff
34
2005 Evaluation Framework
Evaluation Questions for Outstanding Issues
Issue
Process Considerations
Question
To what extent are provincial RRFSS data being
shared among agencies and organizations?
Data Collection Tool
Questionnaire, Key Informant Interview
Source of Data
RRFSS Coordinator, ISR, RRFSS reps
35
Evaluation Framework for RRFSS
Evaluation Questions for Outstanding Issues
Issue
Process Considerations
Question
To what extent are health unit specific RRFSS
data being shared among agencies and
organizations?
Data Collection Tool
Questionnaire
Source of Data
RRFSS reps
36
2005 Evaluation Framework
Evaluation Question for Objective 2
Issue
Process Considerations
Question
What Success Stories can be identified for
RRFSS?
Data Collection Tool
Workshop, Questionnaire
Source of Data
RRFSS reps, program and research staff
37
2005 Evaluation Framework
Evaluation Question
Issue
Process Considerations (Parked Issue)
Question
What is the quality of RRFSS data?
Data Collection Tool
Questionnaire
Source of Data
RRFSS reps, ISR, partners
38
Tasks and Timelines
39
RRFSS Evaluation Tasks




Finalize development of research tools.
• Questionnaire
• Focus groups
• Key informant interviews
• Document reviews
Data collection:
• RRFSS Workshop-- “Success stories”
• Primarily August and September
Data Analysis
Draft and finalize the report
40
RRFSS Evaluation Timelines
RRFSS Evaluation Project Tasks and Timelines, 2005
Month
Task
June
July
Develop Research
Tools
August
Sept
Data Collection
Oct
Nov
Dec
Data
Analysis
Draft
Report
Finalize
Report
41
42