Targeting Regulatory Inspection with Big Data
Healthcare: Does the existing ‘Big Data’ approach
help to prioritise hospital inspections?
1. How are inspections prioritised?
The health and social care regulator in England, the
Care Quality Commission (CQC), is responsible for
overseeing 50,000 providers including150 NHS
Hospital Trusts. In order to prioritise it’s limited inspection resource it has devised its “Intelligent
Monitoring” statistical surveillance tool.
“Intelligent Monitoring” assesses each Hospital
Trust against approximately 150 indicators selected
in consultation with experts, focusing on whether
services are safe, caring, responsive to peoples’
needs, effective, and well-led.
Depending on how far from the average the Trust’s
performance deviates on each individual indicator, the
Trust is either flagged as ‘No evidence of risk’, ‘Risk’ or
‘Elevated Risk’. The proportion of indicators flagging as
a ‘Risk’ or ‘Elevated Risk’ is then used to determine an
overall risk score for the Trust (1 for the highest risk
down to 6 for the lowest risk).
The “Intelligent Monitoring” reports are published by
the CQC and used to prioritise inspection activity.
2. Was the system accurate? No.
We paired the outcome of 110 inspections of
NHS Hospital Trusts in the two years following the launch of “Intelligent Monitoring”.
Authors:
Alex Griffiths, Centre for Analysis of Risk and Regulation (CARR), LSE. [email protected]
David Demeritt, Department of Geography, KCL. [email protected]
Henry Rothstein, Department of Geography, KCL. [email protected]
The authors gratefully acknowledge funding by the Economic and Social Research Council (No.
ES/K006169/1) and the Quality Assurance Agency which made this research possible.
Higher Education: Can any collection of indicators
prioritise HE quality assurance inspections?
5. What data is available?
HESA Performance
Indicators
National Student
Survey
150 Universities with
2,000,000 students
Regulatory
History
Staff
Characteristics
Applications
260
Colleges providing
HE for 150,000
students
Student
Characteristics
119
Alternative Providers
for 60,000 students
Destination of
Leavers
Finance
Overseas
Activity
QAA ‘Concerns’
The 2011 HE White Paper proposed a data-driven, risk-based approach for prioritising inspections. But, if a specific selection of
indicators are unable to succesfully prioritise the inspections of hospitals, can any selection of indicators succesfully perform the
same function in HE?
We gathered all available data that could
feasibly form part of a data -driven,
risk-based approach to quality assurance
and applied machine learning techniques
on hundreds of historic QAA reviews.
Further to collecting nearly 1,000 indicators
covering a 7 year period, change-over
-time and benchmark indicators were calculated resulting in over 3,000 HE indicators.
6. What techniques were used?
The ‘elastic net’ approach, a dynamic
blending of lasso and ridge logistic
regression, was used with 5-fold cross
validation.
This combination allows for variable
selection in a ‘dimensionally cursed’
dataset while avoiding overfitting.
Although not strictly necessary when
using cross-validation, a portion of
An initial inspection of the data showed
there to be little if any relationship between data was held back for model validarisk scores generated by “Intelligent Monitor- tion where possible.
ing” and the subsequent outcome of comprehensive inspections.
7. What was the best possible model?
The model predicted the probability of a QAA resulting in an ‘unsatisfactory’ judgement as:
Further analysis showed the median risk score
preceding a inspection-based quality rating of
“Good” was actually higher - indicating a
greater risk - than preceding an inspectionbased quality rating of “Requires improvement”.
A regression analysis showed there was no significant relationship between the “Intelligent
Monitoring” risk scores and subsequent
inspection findings
3. Was the system at least helpful? No.
Performing Performing
Poorly
Well
High Risk
11
39
Low Risk
10
43
Whilst being able to identify where on a four
point quality scale a Hospital Trust will be
represents the ideal for CQC, “Intelligent
Monitoring” could still be useful if it could
differentiate between those Trusts performing well, and those performing poorly.
We therefore aggregated the risk scores and
inspection-based quality ratings to see if “Intelligent Monitoring” could pass this less demanding test. It could not. Indeed, the risk scores were wrong more often than they were
right.
4. Why did the system fail?
There are a number of possible reasons why the wealth of performance data could not predict
the outcome of comprehensive inspections of NHS Hospital Trusts:
1. The tool may be too simplistic. For example, the tool weights each of the c.150 indicators
equally and does not include hundreds of possible alternative indicators.
2. The majority of data is only available at Trust level. This may be too coarse in scale to
discern the localised pockets of poor quality detected by skilled inspectors.
3. The ratings assigned by inspectors are unreliable; however, the comprehensive trust-wide
inspections by large teams of specialist inspectors, clinicians and ‘experts by experience
have received widespread support and would be challenging to expand further.
4. The performance data and inspectors are simply measuring different things.
Further Information
Griffiths, A., Beaussier, A.-L., Demeritt, D. & Rothstein, H. (2016). Intelligent Monitoring? Assessing the Ability of the Care Quality Commission's Statistical
Surveillance Tool to Predict Quality and Prioritise NHS Hospital Inspections. BMJ Quality & Safety.
Beaussier, A.-L., Demeritt, D., Griffiths, A. & Rothstein, H. (2016). Accounting for Failure: Risk-Based Regulation and the Problems of Ensuring Healthcare
Quality in the NHS. Health, Risk & Society, 1-20.
Griffiths, A. (2016). Forecasting Failure: assessing risks to quality assurance in higher education using machine learning. PhD Thesis. King’s College London.
Havergal, C. (2015). Risk-Based Quality Assessment 'Cannot Work', Study Concludes. Times Higher Education. 26th Nov 2015.
Where:
- APL006_Ca1 - the one-year change in the proportion of successful applicants whose age is
known who are aged 25 & above.
- KFI020_Abs - the percentage ratio of contribution from research grants & contracts to
research grants & contracts income (a measure of the difference between how much
money an HEI has received for, and spent on, research in a given year).
- STA062_Ca1 - the one-year change in the proportion of full-time equivalent (FTE) staff who
are principally financed by the institution.
8. Did the model perform well? No.
Ordering the QAA reviews by their predicted likelihood of being ‘unsatisfactory’,
the model succesfully prioritises 4 of the
first 6 reviews. However, performance
rapidly diminishes:
- There is little difference between the
predicted probabilities for the majority
of universities
- 161 out of 174 reviews would have been required to identify all ‘unsatisfactory’ provision
- No predicted probability exceeds 30%. All probabilities cluster around the likelihood any
given university will be ‘unsatisfactory’ regardless of the available data.
9. What does this mean for ‘Big Data’ & Regulation?
Whilst the healthcare study shows a particular combination of indicators does not succesfully
prioritise hospital inspections, this study shows that no combination of performance indicators can effectively prioritise quality assurance reviews in higher education.
Failure to identify indicators that can prioritise HE reviews may, like healthcare, be due
to concerns over the data, the validity of inspection judgements, or the fact that the data and
the inspections are measuring two different things.
The use of ‘Big Data’ to prioritise regulatory inspections needs to treated with great care and
be subject to thorough evaluation. Failure to properly devise and assess any predictive
models will result in good-quality providers being unfairly burdened with additional inspections whilst low-quality provision goes undetected to the detriment of service users.
©