Using Data to Drive Health System
Performance
Commissioned from Ovations by the National
Primary and Care Trust Development Programme
1
Health Care Evaluation &
Measurement for PCTs
Some Key Evaluation Terms
Acceptable
Accessible
Accomplishment
Accountable
Accuracy
Analysis
Appropriate
Available
Cohorts
Comparison
Content
Context
Control
Cost
Data: primary and
secondary
Goals
Judgment
Metrics
Norms
Objectives
Outcomes
Outputs
Precision
Process
Purpose
Quality
Quantity
Recording
Reliability
Reporting
Standards
Synthesis
Time
Timelines
Validity
Value
Weighting
Worth
2
Health Care Evaluation &
Measurement for PCTs
Things to Consider
Comparing and contrasting various types of evaluation
Identification of problems that may hinder an effective
evaluation
Reasons to conduct an evaluation
The planning and conducting of an evaluation
Defining metrics and methods to measure and report
evaluation findings
3
Health Care Evaluation
Key Ideas for PCTs
Evaluation is critical for all health care, and prevention/
health promotion initiatives.
Evaluation must be designed early in the process of health
care planning. Remember PRECEDE/PROCEED!
The process of designing an evaluation must be a
collaborative effort of all stakeholders.
Evaluation ultimately becomes “judgment.” Who has the
power to decide?
Evaluation does not need to be a formal academic study
Do you report successes, failures, strengths, weaknesses?
4
Health Care Evaluation
Purposes for PCTs
1. To determine achievement of objectives related to
improved health status
2. To improve health program implementation
3. To provide accountability to funders
4. To increase community support
5. To contribute to the scientific base for community.
public health interventions
6. To impact policy decisions
5
Health Care Evaluation
Definition
Evaluation:
Determining the value or worth of the health care
initiative against a standard of acceptability.
To examine or judge.
(The key is who establishes the standard and who judges!)
6
Health Care Evaluation
Elements
Context: What, when , where, and who
Process: How care is organized and
delivered
Content: Program elements to be provided
and why - available birdseed
Output:
How many times did the bird flap
its wings?
Outcome: Did the bird fly?
Impact:
How high? How far? Where?
7
Health Care Evaluation
Types
Process Evaluation: Examines the procedures
and tasks involved during the implementation of
a program.
8
Health Care Evaluation
Types
Process Evaluation
When to use: As soon as the health initiative
begins
What it shows: How well a program is
working as it goes
Why is it useful: Identifies early problems
9
Health Care Evaluation
Types
Outcome Evaluation: Used to obtain descriptive
data on a project and to document short-term
results. Focuses on an ultimate goal of a health
care program or treatment. Generally measured by
vital statistics in a population.
10
Health Care Evaluation
Types
Outcome Evaluation
When to use: For ongoing programs at
appropriate intervals or for one time programs
when program is complete
What it shows: Has program reached its
ultimate goal.
Why is it useful: Learn from successes and
for future funding.
11
Health Care Evaluation
Types
Impact Evaluation: Is the most comprehensive
type of evaluation because it focuses on the longrange results and the resultant improvements in
health status.
Impact evaluation is the most costly.
Information obtained from an impact evaluation
can include changes in e.g.,morbidity and mortality.
12
Health Care Evaluation
Types
Impact Evaluation
When to use: After the health program has
made “contact” with at least one person or a
population
What it shows: Changes in knowledge,
attitudes, and beliefs
Why is it useful: Allows management to
modify resources effectively
13
Health Care Evaluation
Lets pick a population health programme idea
for your specific PCT and consider aspects of
process, outcome and impact evaluation.
14
Metrics and Methods
to Measure and Report
Indicators of Hospital Utilisation
Assumption:
> Hospital admission is an adverse event
that should be avoided whenever possible
> Not all admissions are preventable
Two major indicators
> Admissions per 1000
> Days per 1000
15
Metrics and Methods
to Measure and Report
Indicators of Hospital Utilisation
Admissions per 1000
> In part measures a systems success at
preventing acute disease episodes or
caring for them in alternative systems
> Considered a prevention indicator
Days per 1000
> Reflects both frequency of admission and
length of stay
> Measure episode characteristics
16
Metrics and Methods
to Measure and Report
Indicators of Hospital Utilisation
Both measures are typically collected
separately for:
> Adult and pediatric medical
> Surgical
> ICU
> Obstetrics
> Boarder babies (newborn days after mother’s
discharge)
> Mental health
> Substance abuse
17
Metrics and Methods
to Measure and Report
Risk Adjustment
Prospective vs. retrospective
In general populations the ‘theoretical ceiling
to predication is between 18-22%
>This is higher in sicker, chronic populations
Demographic models
>Typically explain 2-5% of variance in
utilization
18
Metrics and Methods
to Measure and Report
Risk Adjustment
Adding prior use increase the predicative
ability of models to between 8 – 14%
> Some antidotial evidence that adding Rx
and Lab can increase predicative ability to
the 18-22% range
Adding health status changes the weights but
does not increase the predicative ability of
models
19
Metrics and Methods
to Measure and Report
Traditional Health Care Performance
Indicators
Finances
> Revenue and costs
Clinical Activities
Quality of Care
> Includes access
20
Metrics and Methods
to Measure and Report
Guidelines for Selecting Indicators
The ability of systems to produce the data required to
generate the indicator
The perceived importance of the indicator to the
development and assessment of the system
The current utility of the indicator in organizations
within the system
The comparability of the indicator across systems
The uniqueness of the applicability of the indicator
21
Metrics and Methods
to Measure and Report
Indicators of System Performance
Population Health
> SF-30, SF-12, etc
> Mental Health Index (such as the DIS)
> Disease specific incidence and prevalence
rates
> Disease specific outcomes of care (SF-36,
ADLs, IADLs)
22
Metrics and Methods
to Measure and Report
Indicators of System Performance
Quality of Care
> Compliance with guidelines by type of
provider
> Incidence of Sentinel Events:
- Unplanned re-visits
- Avoidable hospitalisations
- Hospital re-admission
- Mortality
23
Metrics and Methods
to Measure and Report
Indicators of System Performance
Quality of Care (cont.):
> Peer review
> Disease specific mortality
- Surgical
- Infant mortality
- Deaths due to cervical cancer
> Frequency of preventable acute episodes
for chronic conditions
24
Metrics and Methods
to Measure and Report
Indicators of System Performance
Quality of Care (cont.):
> RX
- Inappropriate poly pharmacy
- ADRs
Episode Prevention
25
26
27
28
Calculating Bed Days
Works for any given time period (e.g.
day, month to date, year to date)
[A/ (B/365)] /(C/1000)
> A = Gross bed days per unit of time
> B = Days per unit of time
> C = population size
29
Knowledge Management
Planning Cycle for Driving PCT Health Care
System Performance
Social/Health
Indicators
Evaluation
Data & Analysis
Implementation
Useful Information
Operations Planning
Knowledge Management
Strategy
30