Feeding back
Clinical Outcomes
to Frontline Teams
UKRCOM
22nd January 2015
Outcomes Analyses
• Have wanted to embed the routine
measurement, analysis and feeding back of
clinical outcomes to frontline teams for
several years
• Improve clinical effectiveness through
reflective practice, shared learning, identifying
gaps in service, training needs etc
• Have had to deal with re-organisations and
loss of data, changing priorities etc
CQUINs
• Have used CQUINs to promote the use of
outcome data – has ensured the Trust
devoted resource from the Information team
to develop analyses
• CQUINs initially required recording of HoNOS
scores at certain events eg acceptance to
service, admission, discharge, CPA review
• This year’s CQUIN required evidence that
analyses of outcomes were actively fed back
to teams
2014/15 CQUIN for CNWL & WLMHT
Numerator
Sample of 50 patients’ paired scores per CCG, across a range of care clusters
per month - the patients chosen will be from within the same group of clusters
i.e. non-psychosis ( 1-8); psychosis (10-17) organic (18-21); cluster groups may
vary by individual CCG in agreement with commissioners
Denominator All patients paired scores.
Rationale for
inclusion
-If ratings are incorporated into care plans then objectives can be quantified
- By using Outcome Measures, individual clinicians / teams can build up a
picture over time of their service-users patterns of response to interventions
and events that might not be easy to achieve without measurement.
-Sharing outcome analyses with frontline staff helps to improve accuracy of
scoring, as staff see analyses of their interventions on their patients, and clinical
effectiveness through reflection
-Managers can examine differences between outcomes between different
teams and interventions on similar service-user groups.
-Commissioners can move from a purely activity/structured approach to a more
rational purchasing model involving health gain.
CQUIN Milestones
Date/period
milestone
relates to
Rules for achievement of milestones (including evidence to Date
be supplied to commissioner)
milestone
to be
reported
Quarter 1

Develop an Audit tool to allow for analysis of paired
HoNOS /CROM scores
Quarter 2

Undertake the Audit - with different teams using CROM or
HoNOS
Quarter 3


Quarter 4
30th June
2014
30th
September
2014
Review the outcome of the Audit and feedback to clinicians 31st
December
Provide evidence that services are routinely sharing clinical 2014
outcomes analyses with frontline teams

Develop an action plan based on the Audit

Implement the Action Plan

Re-Audit using paired HoNOS / CROM scores
28th
February
2015
Milestone
weighting
(% of
CQUIN
scheme
available)
15%
25%
35%
25%
The Presentations:
“How much do we help our patients?”
• How do we know whether the interventions
we provide are effective? Eg:
–
–
–
–
An antipsychotic?
A “brief intervention” eg course of CBT?
An admission to an acute ward?
A 2 year admission to a rehabilitation/ forensic
unit?
• How interested are we in whether we make a
difference to our patients health and quality
of life?
Why would we want to know if we were being
clinically effective?
• Delivering clinically effective interventions is arguably the
most important thing we do for patients!
• GPs, patients, NHSE, Monitor, CQC all want to know whether
we provide a good (effective) service to patients
• Commissioners want to know that actual clinical outcomes for
patients using our services do improve
– Measurement and analysis of outcomes provides this evidence
• Really importantly, there is also clinical utility to measuring
outcomes:
– Systematic analyses of outcomes provide evidence of teams’ clinical
effectiveness
– Enrich clinicians & managers understanding of morbidity in their
locality
Is there any evidence we make a difference?
• For several years we have been recording HoNOS
scores at key times during patients’ pathway through
our services:
1. At first assessment
2. When there is a significant change in need eg
admission
3. At CPA
4. At discharge
• Comparing a patient’s scores from eg point 1 to
point 4 gives us a measure of our effectiveness
Outcomes analyses
 After years of collecting HoNOS scores, we now want to analyse these at a
team level, identifying how we are doing
 What are we doing well?
 In what areas are we providing the most help for our patients?
 Where are we doing less well and could benefit from further training,
different staff mix etc
• We have several analyses of outcomes scores in different formats and are
really interested in your views as to which (if any) you find most helpful to
understand whether you are delivering clinically effective care
METHODOLOGY
Paired HoNOS scores for selected Service-lines per CCG covering the period
April 2013 to September 2014 were analysed using the following method:
• Scores extracted from JADE at point 1 and point 2 for each selected
patient
• For new patients, Point 1 consisted of first assessment scores. For existing
patients, Point 1 consisted of scores at the start of a new cluster episode.
• Point 2 will be scores on discharge to GP or at start of new cluster episode
There are four potential pathway scores:
1. New to Discharge
2. New to Review
3. Review to Discharge
4. Review to Review
Sufficient paired HoNOS scores were found for pathways 1 and 4.
DATA ANALYSES
Analysis of HoNOS scoring will consist of comparing aggregated mean scores for
patients at point 1 and point 2 using:
• Mean total HoNOS scores at point 1 and point 2 and the difference for each sub
sample.
• HoNOS Four factor model showing differences in scores between point 1 and
point 2 – for each sub-sample.
• The HoNOS Categorical Change model. HoNOS scales were rated - 0 to 2 as LOW
and 3 to 4 as HIGH. Scores were then classified as follows, from point 1 to 2:
- Low score to Low score
- Low score to High score
- High score to Low score
- High score to High score
• Mean individual HoNOS scores at point and point 2 – for each sub-sample.
HoNOS SCALE
FOUR FACTOR MODEL
1.
Overactive, aggressive, disruptive or
agitated behaviour
Personal
Well Being
2.
Non-accidental self-injury
3.
Problem drinking or drug taking
4.
Cognitive problems
5.
Physical illness or disability problems
6.
Problems associated with hallucinations
and delusions
7.
Problems with depressed mood
8.
Other mental and behavioural problems
9.
Problems with relationships
10.
Problems with activity of daily living
11.
Problems with living conditions
12.
Problems with occupation and activities
4.
5.
10.
12.
Cognitive Problems
Physical illness or disability or
disability problems
Problems with activities of daily
living
Problems with occupation and
activities
Emotional
Well Being
2.
7.
8.
Non-accidental self injury
Problems with depressed mood
Other mental and behavioural
problems
Social
Well Being
3.
9.
11.
12.
Problem-drinking or drug taking
Problems with relationships
Problems with living conditions
Problems with occupation and
activities
Severe
Disturbance
1.
Overactive, aggressive, disruptive or
agitated behaviour
Problems associated with
hallucinations and delusions
6.
Note: The four factor score is derived using the sum of the items in each factor/dimension. Note item 12 (problems with occupations and activities) appears in both
personal and social wellbeing factors. This is because this item contributes equally to both factors.
SAMPLE
CCG
NHS West London
(K&C)
NHS Central London
(Westminster)
SERVICE LINE/TEAM
CLUSTERS
PATHWAY
SAMPLE SIZE
ABT
1-5
6-8
10 - 15
1
1
1
93
19
28
Recovery
12
6-8
10
11
12 - 13
16 - 17
11 - 17
3-5
7-8
10 - 15
18 - 19
20
4
4
4
4
4
4
4
1
1
1
1
4
50
23
19
34
93
6
52
45
80
243
238
6
NHS Brent
NHS Hillingdon
Rehab
Acute
NHS Harrow
OPHA
OPHA
MEAN TOTAL HoNOS SCORES
20
K&C ABT - Mean Total HoNOS Scores at First Assessment and Discharge
Mean Total HoNOS Score
15
10
5
0
-5
-10
1-5
6-8
10 - 15
Cluster Group
Mean Total HoNOS Score at First Assessment
Mean Total HoNOS score at Discharge
Mean Total HoNOS score difference
FOUR FACTOR CHANGE
ABT [K&C] - HoNOS Four Factor Model Change
Mean Four Factor Score
7
6
5
4
3
2
1
0
PWB
EWB
SWB
1-5
SD
PWB
EWB
SD
PWB
6-8
Cluster Group
Mean Four Factor score at First Assessment/CPA Review
PWB: Personal Wellbeing
SWB
EWB: Emotional Wellbeing
EWB
SWB
10 - 15
Mean Four Factor score at CPA Review/Discharge
SWB: Social Wellbeing
SD: Severe Disturbance
SD
ABT [K&C] CATEGORICAL CHANGE
HoNOS Categorical Change (CLUSTERS 1-5)
100
90
80
Frequency
60
Low to Low
50
Low to High
40
High to Low
30
High to High
20
10
0
1
2
3
4
5
6
7
8
9
Low to Low
Low to High
High to Low
High to High
1
10 11 12
2
3
4
5
6
7
HoNOS Scale
HoNOS Scale
HoNOS Categorical Change (CLUSTERS 10-15)
30
25
Frequency
Frequency
70
HoNOS Categorical Change (CLUSTERS 6-8)
20
18
16
14
12
10
8
6
4
2
0
20
Low to Low
15
Low to High
High to Low
10
High to High
5
0
1
2
3
4
5
6
7
HoNOS Scale
8
9
10
11
12
8
9
10
11
12
ABT [K&C] HoNOS PROFILE CHANGE
K&C ABT - HoNOS Profile Change [Clusters 1-5]
2.5
K&C ABT - HoNOS Profile Change [Clusters 6-8]
3
2.5
2
Mean Score
2
Mean Score
1.5
1
0.5
1.5
1
0.5
0
0
2
3
4
5
6
7
8
9
10
11
1
12
Mean HoNOS Score at Point 1
2
3
4
5
6
7
8
9
10
11
12
HoNOS Scale
HoNOS Scale
Mean HoNOS Score at Point 1
Mean HoNOS score at Point 2
K&C ABT - HoNOS Profile Change [Cluster 10-15]
2.5
2
Mean Score
1
1.5
1
0.5
0
1
2
3
4
5
6
7
8
9
10
11
12
HoNOS Scale
Mean HoNOS Score at Point 1
Mean HoNOS Score at Point 2
Mean HoNOS Score at Point 2
How do we compare with other ABTs?
• Different demographics, but in comparison
with the other ABTs, how are we doing?
SAMPLE
CCG
NHS West London
(K&C)
SERVICE LINE/TEAM
CLUSTERS
PATHWAY
SAMPLE SIZE
ABT
1-5
6-8
10 - 15
1
1
1
93
19
28
NHS Central London
(Westminster)
ABT
NHS Brent
ABT
NHS Hillingdon
ABT
1-5
6-8
10 - 15
1-5
6-8
10 - 15
1-5
6-8
10, 11, 13, 15 (no 12 or
14 in sample)
1-5
7, 8 (no 6 in sample)
10 – 14 (no 15 in
sample)
1
1
1
1
1
1
1
1
113
42
50
46
12
36
67
11
1
1
1
10
94
14
1
22
NHS Harrow
ABT
Paired HoNOS Categorical Change: CLUSTERS 1-5
K&C ABT
100
90
LL
60
50
LH
40
HL
30
Frequency
70
HH
20
80
Frequency
100
80
LL
60
LH
40
HL
HH
20
10
0
0
1 2 3 4 5 6 7 8 9 10 11 12
HoNOS Scale
Brent ABT
50
45
40
35
30
25
20
15
10
5
0
LL
LH
HL
HH
1 2 3 4 5 6 7 8 9 10 11 12
1 2 3 4 5 6 7 8 9 10 11 12
HoNOS Scale
HoNOS Scale
Hillingdon ABT
80
Harrow ABT
100
90
70
80
60
70
50
LL
40
LH
30
HL
20
HH
Frequency
Frequency
Frequency
Westminster ABT
120
60
30
HL
20
HH
10
0
0
2
3
4
5 6 7 8
HoNOS Scale
9 10 11 12
LH
40
10
1
LL
50
1
2
3
4
5
6
7
8
HoNOS Scale
9 10 11 12
Paired HoNOS Categorical Change: CLUSTERS 6-8
18
40
16
35
12
HH
10
LH
8
HL
6
Frequency
14
HH
4
10
25
20
LH
15
HL
10
HH
0
0
1 2 3 4 5 6 7 8 9 10 11 12
12
LL
5
Brent ABT
14
30
2
LL
8
LH
6
HL
4
HH
2
0
1
2
3
4
HoNOS Scale
5
6
7
8
1 2 3 4 5 6 7 8 9 10 11 12
9 10 11 12
HoNOS Scale
HoNOS Scale
Hillingdon ABT
12
Harrow ABT
16
14
10
12
Frequency
Frequency
Westminster ABT
45
Frequency
K&C ABT
8
LL
6
LH
4
HL
HH
2
Frequency
20
10
LL
8
LH
6
HL
4
HH
2
0
0
1
2
3
4
5 6 7 8
HoNOS Scale
9 10 11 12
1
2
3
4
5
6
7
8
HoNOS Scale
9 10 11 12
Paired HoNOS Categorical Change: CLUSTERS 10-15
K&C ABT
30
Westminster ABT
60
25
50
20
40
Brent ABT
40
35
30
LH
HL
20
HL
HH
10
LH
10
LL
5
HH
3
4
2
3
4
5 6 7 8 9 10 11 12
HoNOS Scale
5
6
7
8
15
HL
10
HH
1 2 3 4 5 6 7 8 9 10 11 12
9 10 11 12
HoNOS Scale
Harrow ABT
25
Hillingdon ABT
12
LH
HoNOS Scale
10
20
8
LL
6
LH
4
Frequency
2
LL
20
0
1
1
25
5
0
0
Frequency
15
Frequency
LL
Frequency
Frequency
30
LL
15
LH
10
HL
HL
2
HH
0
1
2
3
4
5
6
7
8
HoNOS Scale
9 10 11 12
HH
5
0
1
2
3
4
5 6 7 8
HoNOS Scale
9
10 11 12
Conclusion of Presentations:
• Providing interventions which make a genuine, positive
contribution to our patients’ lives is (or should be!) our top
clinical priority
• It is not always easy to determine how successfully we are
achieving our aims
• Systematic measurement and analysis of outcomes can help
us to understand where we as individuals and teams are
doing well and where we might need more development
• Please let us have your thoughts on the utility of outcome
measurement, so we can improve how scores are analysed &
fed back to teams in the future
Staff were asked to evaluate the sessions:
1. How useful was it to receive an analysis of team
outcomes using the 4 models?
2. Which model was most helpful?
3. Are there alternative ways of presenting outcomes
which might be more useful?
4. How often should outcomes analyses be presented
to teams?
5. Which other staff might benefit from being fed
back outcomes analyses?
Results of Evaluation (n=26)
How interested are you in finding out whether the patients you
treat get better?
• Very Interested 24/26
• Interested 2/26
• Not sure/ Not Interested 0/26
How useful was it to receive an analysis of team outcomes using
the 4 models?
• Very Useful 10/22
• Useful 11/22
• Not Sure 1/22
• Not Useful 0/22
Which models were useful in helping you
understand changes in patients’ symptoms?
Type of analysis
Not
Useful
Very
Useful
Total HoNOS score
1
1
2
11
8
4 factor model
1
3
5
8
9
Categorical Change model
0
1
2
9
11
Profile Change
0
1
2
9
12
Evaluation
How else could outcomes analyses be presented? (free text response)
• Benchmarking against other teams, to identify service or demographic differences,
or to highlight where teams are doing well/ less well.
• Compare results with patient / carer responses
• Use GP feedback
• Undertake further analysis for patients whose scores remain high despite treatment
• Separate out by diagnosis (as well as cluster) eg do personality disorder patients do
differently?
• Function on Jade to produce individual change graphs which can be shown to
patients
• Use case studies alongside outcomes analyses
Who else would benefit from attending presentations on outcomes analyses? (free
text)
• Whole of the team including admin, managers / Senior management team
• Service user groups...Commissioners
Conclusions
• Results showed all staff who completed the feedback forms
were interested in knowing whether the patients they treated
improved as a result of their interventions.
• The outcomes analyses that were shared with teams looked
at paired HoNOS scores using 4 different models. All but one
responder found the HoNOS analyses useful or very useful.
• Most staff were unfamiliar with the models before the
presentation. However responders found all four of the
models either useful or very useful in helping them
understand their outcomes (Total score change 83%, 4 factor
model 65%, categorical change model 87%, profile change
88%). 4 factor was the model with the highest proportion of
staff being unsure or finding it not useful (35%)
Conclusions
• Responders preferred feedback to be given at either 3 monthly intervals
(40%) or six monthly intervals (36%). Although numbers were relatively
small, rehab staff had a preference for longer periods between
presentations (6-12 monthly)
• Additional outcomes analyses which staff thought would be useful
including patient completed measures. Triangulation with PROMS would
help add validity to clinician rated measures
• Added contextual information such as diagnosis and demographics was
thought to be helpful
• Staff thought all members of the team including admin staff and senior
managers should be presented analyses of outcomes. Some also
supported outcomes analyses to be presented to commissioners
Next Steps
• Roll out outcomes analyses to all frontline
mental health teams during the course of
2015.
• Analyses should be actively presented to all
members of teams (eg for 30 minutes during
an MDT), by outcomes leads who understand
the models and can facilitate discussions on
what analyses mean.
Next Steps
• Each Divisional Medical Director to identify an
outcomes leads for their mental health teams.
The role of the leads will include:
– To liaise with the Information Team to ensure the
correct analyses are being prepared for their
allocated clinical teams
– To attend training on the models and on how to
facilitate a feedback session.
– To develop a programme to deliver presentations
to each of their allocated teams during the course
of 2015.
Questions?
Advice?