BuzzFeed
This Algorithm Accidentally Predicted Which
Hospital Patients Were Most Likely To Die
The algorithm didn’t work the way it was supposed to. But it ended up having a more powerful
potential: telling doctors who their sickest patients were.
posted on Sept. 19, 2016, at 2:30 p.m.
Stephanie M. Lee
Thomas Northcut / Getty Images
Sepsis is one of the biggest hospital hazards you’ve maybe never heard of. When the body
overreacts to an infection, it can trigger widespread inflammation that can in turn cause tissue
damage and organ failure. It causes one-third to one-half of all deaths in US hospitals.
But because sepsis’s symptoms, like fever and difficulty breathing, sometimes look a lot like
other illnesses, it can be hard to detect, especially in the early stages. So a team at Banner Health,
a hospital system in Phoenix, Arizona, turned to computer science for a solution. Maybe they
could develop an algorithm that constantly monitored electronic health records and warned
hospital staff in real time when patients were at high risk for sepsis.
It didn’t work. At least, not in the way Banner had hoped for.
Five years after Banner put the alert in place, it turns out to not have done a very good job of
diagnosing sepsis. But the team behind it, led by Dr. Hargobind Khurana, discovered it had an
unexpected upside: It was good at identifying patients who were generally much sicker than
average, even if they didn’t have sepsis. Although the alert mostly failed at its main goal, it
ended up having a different, perhaps even more powerful potential: steering clinicians to their
most vulnerable patients.
Compared with patients who didn’t set off the alert,
those who triggered it had four times the chance of
dying the next day.
Algorithms have infiltrated almost every part of our lives, quietly yet deftly shaping both the
mundane — calendar alerts, Facebook ads, Google predictions — and the vital. One of the most
critical roles algorithms play is in electronic medical record software, which hospitals and
doctor’s offices use to track and manage patients’ health and illnesses. Algorithm-based alerts
are supposed to point out important information hidden in mountains of data — things like when
someone’s medication needs to be refilled, or when a patient has an unusually high heart rate.
At their best, these alerts save busy doctors and nurses precious decision-making energy and
draw attention to dangers that would otherwise go unnoticed. Too often, however, they dilute
their usefulness and urgency by beeping, buzzing, and flashing tens of thousands of times a day,
often without a good reason.
Banner Health’s experiment demonstrates some of the core challenges of merging health care
with 21st-century digital automation. It’s a continuing struggle despite the fact that the US
government has poured billions into digitizing medical records in hopes of making them safer
over the past few decades.
“It’s hard to create a good alert. And it’s hard to get buy-in from doctors and nurses because it’s
‘just another thing’ to do,” Khurana, Banner’s director of health management, told BuzzFeed
News. “How do we keep that balance of not just expecting them to do more and more work, but
how do we make sure the patient is taken care of? … How good do the alerts need to be? …
Everybody in the health field is trying to figure out the answer to this.”
Moodboard / Getty Images
Banner Health started working on the alert in 2009; Khurana joined two years later. At first, they
looked at the common criteria for sepsis and organ dysfunction, like high breath and heart rates,
unusually high or low body temperature, and off-balance chemical levels in someone’s blood and
organs. Then they used these criteria to design an alert that continuously analyzed electronic
medical record data from medical device sensors and other sources. The alert would fire
whenever a patient showed two of four symptoms for sepsis and at least one of 14 symptoms for
organ dysfunction — if both of those things happened within eight hours of each other.
Khurana added the alert to Banner Health’s Cerner electronic medical record software, which,
like other programs, comes with its own built-in alerts (but did not at the time have a sepsis
alert). From April 2011 to June 2013, the sepsis algorithm monitored more than 312,000 patients
across the emergency, inpatient, and intensive care units of Banner Health’s 24 hospitals.
Weary, overworked staff are prone to ignore even
alerts that point out signs of danger.
Not everyone was thrilled, Khurana recalls. Some nurses and doctors complained that not every
patient flagged by the algorithm actually had sepsis — but the caregivers still had to evaluate the
patients, override the alerts, and document it. Those steps may take just a few minutes, but the
many false alarms made some staff members doubt if the algorithm was working at all.
A colleague who helped develop the alert, Dr. Nidhi Nikhanj, recalls similar sentiments. “There
was certainly was a lot of skepticism, especially from those who had to actually answer the
alerts, because of the extra workload it would bring on our shoulders,” he said.
These clinicians were grappling with a widespread phenomenon in health care dubbed “alarm
fatigue.” In a 2013 report, the Joint Commission, a health care accreditation nonprofit, found that
several hundred alerts can fire per patient per day, which amounts to tens of thousands of buzzes
or beeps throughout an entire hospital every day. But 85% to 99% of these warnings don’t
actually require clinicians to intervene, often because the alerts’ settings are too narrow or broad
to correctly identify patients in need. Weary, overworked staff are then prone to ignore even
alerts that point out signs of danger.
Alerts are best when they “continually tell physicians what they’re really not aware of,” said
Lorraine Possanza, a risk management analyst at the ECRI Institute, a nonprofit that studies
patient safety issues. “If you’re continuing to give them info they already know, the likelihood of
them bypassing that alert, or ignoring the alert, or becoming overwhelmed by the number of
alerts, is just much more likely.”
This May, nearly five years after the experiment started, Khurana’s team crunched the data and
published the results in the American Journal of Medicine. His colleagues’ complaints had been
partly accurate: The alert didn’t always flag patients with sepsis. More precisely, only about onequarter of patients the alert flagged had the condition.
The patients identified by the alert did turn out, however, to be much sicker than average in
general. This correlation wasn’t completely surprising, given how sepsis symptoms are known to
overlap with other severe illnesses.
The algorithm identified a small minority of
patients who accounted for nearly 90% of all deaths
in the hospital.
But Khurana was taken aback by just how sick this group was by virtually every measure. The
algorithm identified a small minority of patients — about one-fifth — who accounted for the
overwhelming majority — nearly 90% — of all deaths in the hospital. Compared with patients
who didn’t set off the alert, those who triggered it had four times the chance of dying the next
day. They were also more likely to suffer chronic medical conditions, such as chronic kidney
disease and obstructive pulmonary disease, and to stay in the hospital twice as long.
“We expected it would be sicker patients, and the rates would be higher, but not this high,”
Khurana said. In other words, the data showed that the alert had the potential to bring sick, inneed patients to clinicians’ attention — just not quite the patients who the Banner Health team
had first set out to find.
Since the initial data analysis of the alert in early 2014, clinicians at Banner Health have come to
perceive the algorithm in a new light, Khurana said. The question it used to prompt, as he put it,
was: “‘Does a patient have sepsis?’ If not, move on.”
Now, he said, the alert inspires clinicians to take a second look and ask themselves, “Is the
patient sicker than what I expected? Is there anything I can do to look at a patient’s care plan and
do things differently?” Khurana said those things include moving a patient to an intensive care
unit, checking in on them more frequently, and re-evaluating their diagnosis and treatment.
Brianajackson / Getty Images
The team hasn’t crunched the numbers yet to definitively know how, or if, these interventions
are improving patient health. But after seeing the first set of results, staff members are more
willing to embrace the algorithm’s potential. “Because of a new enthusiasm and renewed interest
in this, we were able to get a lot more buy-in,” Khurana said.
Electronic health record alerts are near-perpetual
works in progress — as unnerving as that might be
to hear.
While his team still wants to create a fully functioning sepsis alert, their main focus at the
moment is refining the original algorithm to better identify the sicker-than-average patients. One
insight from the first time around, for example, was that patients who triggered the alerts and had
elevated lactic acid levels were likelier to die than alert-triggering patients with normal levels.
(High levels can mean that the body is not getting enough blood supply.)
Taking this into account, their revamped alert doesn’t fire if a patient has normal lactic acid
levels and generally has stable vital signs. It’s too early to know if the tweak has made the
algorithm more accurate or helped save more lives; answers to those questions will be revealed
in future studies. But there are promising signs so far. “This has helped us filter out a lot of the
false negatives,” Nikhanj said.
What Banner learned is that electronic health record alerts are near-perpetual works in progress
— as unnerving as that may be for patients to hear. It’s likely that no one will ever come up with
a set of algorithms that saves patients’ lives 100% of the time, but clinicians and programmers
can’t stop trying to get there.
Depending entirely on algorithms was never the point, anyway. The goal, says John Gresham, a
vice president at Cerner, the company making Banner Health’s electronic health record software,
is to “guide the clinicians to make a different decision or to intervene more quickly. Not [to] take
care out of the hands of the physician, but guide them to make a better clinical outcome.”
Stephanie Lee is a senior technology reporter for BuzzFeed News and is based in San Francisco.
Contact Stephanie M. Lee at [email protected].