Patient experience metrics:
using the data to good effect
Veena Raleigh
The King’s Fund
HSRUK
Patient feedback: Potential or problem in a changing NHS?
Exeter University, 7 April
Introduction
• Patient experience data used widely – and wisely?
• Some issues to consider:
- clarity about aims
- familiarity with the data
- realistic expectations for change
- case-mix
- wider system impacts
• Examples from analysis of trends over 9 years in inpatient survey data for
156 NHS acute trusts (KF/Picker collaboration)
Clarity about aims
• Patient experience data used by multiple audiences for multiple purposes:
Aim
Purpose
Audience
Quality improvement
Internal use
Providers
Performance
assessment, P4P etc
External judgment
DH, NHSE, CQC,
commissioners
Transparency, patient
choice
Public use
Public, patients
• Common misconception that one tool can serve multiple aims eg FFT
• Clarity about aim of measurement vital
• Aim should drive choice of data, ensuring metrics are fit for purpose
Matching choice of data to aims
DATA COLLECTION MODES
(examples)
Surveys
FFT, other
real-time data
collections
Social
media
Quality improvement
INTERNAL USE



Performance management,
CQC ratings, P4P etc
EXTERNAL JUDGMENT

X
X
Information for patients,
public
PUBLIC USE



AIM OF MEASUREMENT
Features of data on PE
DATA COLLECTION MODE (examples)
Features
Surveys
Large, representative
sample

Standard data
collection methods

Case-mix adjustment

Statistical reliability

Comparative data
across organisations

FFT, other realtime data
collections
Social media
Timely data


Locality specific data


Free text data


Understanding the data
1. Trusts consistently show higher performance in some areas of patient experience than others.
2. Inter-trust differences are consistently wider in some areas than others.
Q 37: Were you given enough privacy when being examined or treated? Q 21: How would you rate the hospital food?
3. Much year-on-year variation is random, regression to the mean.
Q 59: Did staff tell you about danger signals to watch for after you went home? Q 67: Overall, did you feel you were treated with respect & dignity?
Taking the long view can be useful
Q 55: Did staff explain the purpose of the medicines in a way you could understand?
Not significant
Q17 How clean was the hospital room or ward that you were in?
Maidstone & Tunbridge Wells NHS Trust
100
90
sig p<0.01
Trust score
80
70
60
50
40
2005
2006
2007
2008
2009
2010
2011
2012
2013
Q 27: When you had important questions to ask a nurse, did you get answers that you could understand?
sig p<0.01
Having realistic expectations about
change
• National data show relatively little change over a decade
• At trust level, performance is mixed – some improvement, some decline
• Most trusts show statistically significant change on few questions, and the
magnitude of change is generally small
• Should be taken into account by eg commissioners when setting contracts,
assessing performance, in P4P
Change in national scores for selected
questions
100
0.99
0.24
90
1.17
-3.14
2.01
-0.95
0.50
0.40
6.59
1.16
1.51
0.43
-1.62
-1.54
0.51
80
National average score
1.71
70
60
50
-2.03
-1.51
1.01
4.09
2.10
40
30
20
10
0
Q06 Q07 Q09 Q31 Q52 Q59 Q32 Q55 Q56 Q24 Q26 Q27 Q29 Q15 Q16 Q17 Q21 Q37 Q39 Q67 Q75
(2011
only)
2005-2007
2011-2013
4. Evidence of more improvement where performance is lower and a ceiling effect.
Q 17: How clean was the hospital room or ward that you were in?
Case-mix is a confounder
• Several factors influence how patients respond, irrespective of quality:
- age, gender, social class, self-reported health status, deprivation, ethnicity,
LTCs, specialist vs general acute services
• Case-mix varies between trusts and changes over time
• Should be taken into account when assessing performance, comparing
organisational performance, in P4P
• Other factors: does lack of change reflect changed expectations over time?
Consider wider system effects
• Many trusts showed improvements in policy priority areas with targets:
- cleanliness
- waiting times to admission
• In contrast, many trusts showed deterioration resulting from wider system
pressures:
- waiting time to get to a bed after admission
- noise levels at night
- delayed discharge
5. Some aspects of patient experience showed widespread evidence of deterioration.
Q9 From arrival at hospital, length of wait to get to a bed on a ward
Q52 On the day you left hospital, was your discharge delayed/?
Final thoughts
• NHS patient survey programme one of the largest internationally
• Data under-used nationally and locally for QI
• Barriers to use cited by trusts but also examples of changes in practice
• Policy-makers, regulators, commissioners should be cognisant of data-related
issues and set realistic expectations for performance improvement
• Risks in inappropriate use of data eg misuse of resources
• More guidance needed on using the data appropriately and to good effect
• KF/Picker report makes recommendations for policymakers, commissioners and
providers
Patients’ experience of using hospital services:
an analysis of trends in inpatient surveys in
NHS acute trusts in England, 2005-13
V Raleigh
James Thompson
Joni Jabbal
Chris Graham
Steve Sizmur
Alice Coulter
December 2015
http://www.kingsfund.org.uk/publications/patients-experience-using-hospital-services