CUSUM plots
•
•
The HFEA has chosen to use cumulative sum (CUSUM) analysis to assess
how well centres are doing in terms of success rates and multiple
pregnancy rates.
An example of a “two sided” CUSUM plot is shown below
CUSUM plots - outcomes
•
In simple terms, when a cycle is reported, the difference between that centres
success rate at that moment and the sector average is calculated and the
difference is plotted. Sometimes the centre is performing above average,
sometimes below, so the cumulative sum value can go up and down.
CUSUM plots
•
The “two sidedness” of the plot is a device to allow positive or negative
trends to be identified quickly.. The two lines are representations of the
same data.
CUSUM plots - outcomes
•
•
We have found that we can plot all of the cycles from patients aged less
than 38 years together without age impacting on success rates: including as
many cycles as possible makes the statistical analysis more robust.
All cycles for patients aged more than 38 years are plotted together.
CUSUM plots
On a plot of outcomes the centreline at zero (x axis)
represents the sector average
CUSUM plots - outcomes
The y-axis of a CUSUM plot usually represents the
difference between an outcome value and a derived
standard or target – in this case the centres outcome and
the sector average
CUSUM plots - outcomes
The x-axis of a CUSUM plot usually represents time – in
this case time measured in cycle number. Each shaded
bar represents a month with the chart showing data for a
rolling year
CUSUM plots
Where a centre’s performance is in line with that of the
rest of the sector, monitoring of their outcomes will
produce a plot like this which oscillates between the
upper (green) and lower (red) threshold lines.
CUSUM plots - outcomes
Where a centre’s performance is consistently
better than the average this produces a
progressively upward trend line above the x
axis. For outcomes this indicates good
performance.
CUSUM plots
Where an upper green threshold line is crossed
in an outcomes CUSUM plot this indicates that if
the centre's performance continues on this
trajectory then they are likely to perform better
than the sector at statistically significant level
CUSUM plots - outcomes
Where a centre’s performance is consistently
worse than the average this produces a
progressively downward trend line (shown in
black) below the x axis. For outcomes this
may indicate poor performance.
CUSUM plots - outcomes
Where the lower red threshold line is crossed this indicates
that if the centre's performance continues on this trajectory
then they are likely to perform worse than the sector at
statistically significant level
CUSUM plots - outcomes
When a lower “red” threshold is crossed centres will be sent
an automated email to alert them to this. It is expected that
centres will review their practice and implement changes to
improve performance.
CUSUM plots - outcomes
Because there is a time lag of up to 3 months between
treatment and outcome reporting downward trends may
continue for several months even when action is taken
by a centre as soon as a downward trend is identified.
CUSUM plots - outcomes
Because the plot line remains below the threshold the centre would
continue to receive alert emails every month. To avoid this, a
second line – shown in blue – resets the CUSUM plot. This
“resetting CUSUM” allows the impact of any changes to be
monitored more effectively
CUSUM plots - outcomes
Where resets continue to occur this is indicative that corrective actions
implemented by the centre may not be effective. Every time the blue line
breaches the threshold the centre will be sent an automated email. If the
plot continues to reset then this might prompt a HFEA management review.
CUSUM plots - outcomes
If no early outcome form is submitted then our system
assumes that the outcome is negative. This means
that poor submission of treatment forms can cause a
negative trend line.
CUSUM plots - outcomes
A table like is included below each CUSUM plot: the table
shows where there are missing outcomes by month. Centres
can extract a report through EDI that shows exactly which
patient’s forms are missing. If the forms are submitted the
apparent poor performance may be resolved.
CUSUM plots – multiple births
•
We also use CUSUM plots to monitor clinical multiple pregnancy rates
(cMPR)
CUSUM plots – multiple births
On a cMPR CUSUM plot the centreline at zero (x axis)
represents the current cMPR target extrapolated from the
multiple live birth rate. These plots are restarted
whenever a new target is set.
CUSUM plots – multiple births
The x-axis of a cMPR CUSUM plot represents time – in
this case, measured in cycle number. Each shaded bar
represents a month with the chart showing data from the
month that the current target came into force
CUSUM plots – multiple births
Where a centre’s performance is in line with the target,
monitoring of their cMPR will produce a plot like this
which oscillates between the lower (green) and upper
(red) threshold lines.
CUSUM plots – multiple births
Where an lower green threshold line is crossed
in a cMPR CUSUM plot this indicates that if the
centre's performance continues on this trajectory
then they are likely to have a multiple live birth
rate below the current target
CUSUM plots – multiple births
Where an upper red threshold line is crossed in a cMPR CUSUM
plot this indicates that if the centre's performance continues on
this trajectory then they are unlikely to meet the current multiple
live birth rate target
CUSUM plots – multiple births
When an upper “red” threshold is crossed centres will be
sent an automated email to alert them to this. It is expected
that centres will review their practice and implement changes
to improve performance.
CUSUM plots – multiple births
Because the plot line (shown in black) remains above the threshold
the centre would continue to receive alert emails every month. To
avoid this, a second line – shown in blue – resets the CUSUM plot.
This “resetting CUSUM” allows the impact of any changes to be
monitored more effectively
CUSUM plots – multiple births
Where resets continue to occur this is indicative that corrective actions
implemented by the centre may not be effective. Every time the blue line
breaches the threshold the centre will be sent an automated email.
If the plot continues to reset then this might prompt a management
HFEA review of performance.
CUSUM plots
• The CUSUM analysis used by the HFEA is part of an
area of statistics called Statistical Process Control (SPC)
which has been used in manufacturing for many years
and is now gaining popularity in health statistics.
• The explanations given here are simplified for a nonstatistical audience: detail about the methodology used
can be found by referring to the original publication.
Leandro G., Rolando N., Gallus G., Rolles K. and Burroughs A. K. (2005) Monitoring
surgical and medical outcomes: the Bernoulli cumulative SUM chart. A novel
application to assess clinical interventions. Postgrad Med J; 81:647–652