THE WALL STREET JOURNAL, July 6, 2012
HOW TO BE SURE YOU’VE
FOUND A HIGGS BOSON
BY CARL BIALIK
Other fields have different benchmarks because
less is at stake if a result proves faulty, or, as
with pharmaceutical studies, there is more upside to moving ahead quickly with a promising
result, statisticians say. "Drugs can be withdrawn, psychological experiments can be refuted, but nobody wants to see the laws of physics
proved wrong," says David Spiegelhalter, a statistician at the University of Cambridge.
Sigma is the Greek letter used as a symbol for
standard deviation—a measure of how far a
finding departs from the expected one. The
more sigmas attached to a result, the more likely
it is significant and not due to chance. Say a series of experiments on a randomly chosen coin
involves flipping it 1,000 times and then flipping
it 1,000 times again and again. The average
number of heads should be 500, but some experiments will yield more and some fewer. A fivesigma finding would be 590 heads.
The CERN finding, then, is the equivalent of getting a lot more heads than expected—which is
unlikely to occur by fluke rather than for some
systematic reason. The best explanation the
CERN physicists have for this excess signal is
the existence of the long-hunted Higgs boson.
They set such a high bar to rule out two other
possible explanations: Either their equivalent of
the coin is flawed in a way that tacks on extra
positive signals; or they've run the study enough
times and looked for anomalies in so many places in their data—the equivalent of running the
coin experiment over and over with different
coins—that they've stumbled upon a seemingly
unlikely result just by looking too hard for it.
Physicists call this the Look Elsewhere Effect, or
LEE, and try to account for it. One CERN group
The physicists who announced the likely discovery of the long-sought Higgs boson particle
this week were operating according to an extremely high standard of certainty. As was
widely reported, in order to achieve discovery
status their experiment had to clear a threshold
of "five sigmas" of statistical significance.
What precisely the five-sigma mark means,
however, wasn't always clearly explained in the
coverage of a ground-breaking development
that could explain how particles have mass and,
by extension, why planets and all other objects
exist at all. That is partly because the five-sigma
concept is somewhat counterintuitive. It has to
do with a one-in-3.5-million probability. That is
not the probability that the Higgs boson doesn't
exist. It is, rather, the inverse: If the particle
doesn't exist, one in 3.5 million is the chance an
experiment just like the one announced this
week would nevertheless come up with a result
appearing to confirm it does exist.
In other words, one in 3.5 million is the likelihood of finding a false positive—a fluke produced by random statistical fluctuation—that
seems as definitive as the findings released by
two teams of researchers at the CERN laboratory
in Geneva.
This is a very high burden of proof as far as science goes. For many medical experiments, researchers need merely to clear two sigmas. Sigmas don't scale in a linear way: A two-sigma result can have as much as a 5% chance of occurring as a false positive. Three sigmas, needed to
cite evidence—but not discovery—of a new particle in physics, correspond to a one-in-741
chance.
395
said its finding's significance falls to between 4.1
and 4.3 sigmas after accounting for LEE.
Another factor can pull in the opposite direction
from LEE: when another experiment finds the
same thing. That bolsters a finding's significance. The Higgs boson was found at the fivesigma level by two CERN experiments.
The five-sigma requirement also helps guard
against the equivalent of a faulty coin—some
kind of measurement error. That appears to be
the explanation for a finding last year that certain particles called neutrinos were traveling
faster than
the speed of
light. It now
appears to
be the result
of a flawed
cable.
Particle
physicists
cite
many
examples of
results that
cleared three
but not five
sigmas and
weren't replicated
by
follow-up
studies,
which
helped give
Part of the Large Hadron Collider in Geneva that scientists used to discover
rise to the
the Higgs boson particle.
five-sigma
rule. University of Pisa physicist Giovanni
Punzi, who has collaborated on a parallel hunt
for the Higgs boson at the Fermi National Accelerator Laboratory in Chicago, says physicists
have debated whether the rule is "exaggeratedly
high." It was devised decades ago to provide a
margin of safety in case calculations of statistics
for an experiment, done without today's computing power, were flawed.
But counterarguments prevailed: Modern computing power makes it easier to find false positives; and with so many particles discovered,
new ones better clear a high bar. Maybe the rule
is "not a bad idea, after all," Dr. Punzi says.
396