Trials factor in “bump” hunts
WOW! Green jelly beans cause acne* ...
Seek (enough) and ye shall find ...
Lies, damned lies and statistics ...
If at first you don't succeed, try, try again!
However, the focus of this talk is narrower. I look
only at how to estimate the number of effective
trials when you scan a distribution in small
steps for a narrow peak ...
The basic idea is to diagonalize the space of
scan points so a fast, uncorrelated “toy” Monte
Carlo can be done in the diagonalized space and
the need to refit all the scan points for each
“toy” experiment avoided
The “toy” directly generates significances at
each scan point ...
* Not completely implausible
Fri Jun 1 16:32:23 PDT 2012
A. E. Snyder, SLAC
1
What could go wrong?
RMP, 35, 314 (1963)
Lot's of exotic isospin 2
mesons “discovered”...
Where have the exotics all
gone?
I=2
I=2
Cupertino High School
circ 1963
Particle data group, March 1963
Fri Jun 1 16:32:23 PDT 2012

A. E. Snyder, SLAC
2
Indication of
experimenter bias ...
Scanning & Trials
Simultaneous fit
in ee,  and e modes
significance
limit
←significance scan→
Significance distribution
-- nothing there --
Fit, move signal position, fit, i.e., scan!
Scan points can be thought of as a vectors space with inner product defined as
correlation between points
● Diagonalize this space
● Throw "toys" in the diagonalized space where "axis" are orthogonal
● Transform back to real scan space
● Search for most significant "toy" scan point
● Count how often it exceeds your threshold of interest, e.g., an interestingly
significant point in the scan of the real data
●
Search for a low-mass Higgs boson in Y(3S) ---> gamma A0, A0 ---> tau+ tau- at BABAR; BABAR
Collaboration (Bernard Aubert et al.); BABAR-PUB-09-013, SLAC-PUB-13673, Jun 2009
Fri Jun 1 16:32:23 PDT 2012
A. E. Snyder, SLAC
3
Constructing the scan correlation matrix
Class buildScanMartrix.h,C
Use method of "virtual bins" to evaluate correlations between scan points. i.e.,
consider very small bins and evaluate averages of variations in bin contents
( ni) over infinite ensemble of hypothetical experiments ...
The likelihood is given by
where mi is the number of "events" in expected in bin i and ni is the number observed
where si is fraction of signal in each bin, bi is background fraction, and, S and B are
number of signal and background events, respectively
Fri Jun 1 16:32:23 PDT 2012
A. E. Snyder, SLAC
4
To compute correlations we need to look at the variations of S ( S) and B ( B) w.r.t. the
variations in the independent variables -- the events ni ( ni) at the maximum of the
likelihood.
Maximum here
 represents parameters
that control the shape of
the background
I assume signal shape is
known
At this point we need to
take  ni variations and
solve  S and  B
Method of “virtual bins” ...
Fri Jun 1 16:32:23 PDT 2012
A. E. Snyder, SLAC
5
Take variations  S,
 B and  's w.r.t.
“bin” contents ( ni ),
i.e., the independent
variables
Since we are interested in deviations ( ) from the null hypothesis (S=0) the sums (not involving  ni) can be
evaluated at this stage using the approximation ni=Bbi & mi=Bbi. This is what is done in buildScanMatrix. A
bunch for terms go to zero and 1 in this approx, e.g., only  S+ B survives on LHS of eqn. 8.
The results is the matrix equation for  S,  B and the  's in terms of sums over the  ni's. Solving we get an
equation of the can be expressed in the form
which is gives us the variations in signal for signal location loc. From  Sloc we can compute the correlation
between signals at different locations (< SA SB>) with the rule < ni nj>=ni ij and, thus, construct
correlation matrix
Fri Jun 1 16:32:23 PDT 2012
A. E. Snyder, SLAC
6
2D example
 =0.9
 =0.99
Above/right of dashed lines is  >3
Area corresponds to independent trials
Direction transverse to the red/blue lines
represents correlated scan
Above red ( =0.9) or blue (=0.99) line or
right of vertical dashed line corresponds to
reduced number of effective independent
trials when O1 and O2 are correlated
When  =1 trials factor is reduced to one
O2 vs O1
To get effective trials one needs to integrate
unit Gaussian over the desired area (e.g.,
 >3). I do this by Monte Carlo integration
(a.k.a., “toy” Monte Carlo)
Some method to do this by just counting
(PCA) might be possible
Fri Jun 1 16:32:23 PDT 2012
A. E. Snyder, SLAC
7
Scan Matrix
buildScanMatrix.C
doEigenMapToy.C
doGSO.C
The correlations between scan points A and B that make up the scan matrix
are defined by
where I've normalized  S distributions to have < S2>=1
This is encoded in the class BuildScanMatrix which constructs the matrix 
from simple sums over the signal and background PDFs (like above)
The matrix can then be fed to class eigenToy which will throw experiments
transform back to scan space, and histogram biggest significance
Fri Jun 1 16:32:23 PDT 2012
A. E. Snyder, SLAC
8
Correlations plots
plotMatrix.C
Correlation plots for a simple example:
Power law background f(E)=K*1/E with =2; signal fixed width Gaussian ( = 0.1)
200 scan point between 1 and 10 GeV; plot correlation  between scan points (x-axis)
Signal normalization S, background normalization B and background shape are "free"
E =1.180
E =2.080
E =4.500
<------1 -- 10 GeV --------->
E =7.75
Fri Jun 1 16:32:23 PDT 2012
E =8.875
A. E. Snyder, SLAC
E =9.550
9
doEigenMapToy.C
testScan.C
scanMatrix-5186.root
Toy
Based on "typical" Fermi dark matter line scan
Energy range 20-220 GeV
Energy resolution 10% (not so unrealistic)
Number of scan points 100; log(E) scan point locations
Fit to power law (index= -2) background + Gaussian
signal. Scan signal location in energy
Correlations with
scan point 25
  :scan point
100K toys generated
experiments
correlations
simulated!
Eigenvalues
one experiment
p-value
 max
Fri Jun 1 16:32:23 PDT 2012
pull:scan point
A. E. Snyder, SLAC
10
Common effective number of trials?
trials.C
doEigenMap.C
Can the toy distribution be characterized by a common effective number of trials?
experiments
10M experiments
red is "eigen" toy described above
blue is 22 trials drawn from unit width Gaussian
distribution
22 trials was chosen to match the means
~25 1 intervals fit between 20-220 GeV (rule of thumb)
 max
A single effective number trials does not fully capture the
distribution of the toys ...
Ratio of “eigen” to simple trials distributions
toy/simple trials
Simple trials adjusted to the mean underestimates the pvalues by ~1.3-1.5 above ~4
Indicates the number of effective trials needed increases
with n of interest ...
 max
Fri Jun 1 16:32:23 PDT 2012
While correlations reduce trials, it's not by am integer
number of trials ...
But p-value can just be read off directly toy distribution
A. E. Snyder, SLAC
11
trialGraph.C
Trials vs. scans
effective trials vs. number of scan points
trials
As it should trials saturates as number of
scan points becomes large and the
correlation between adjacent scan points
approaches one ...
scan points
FIT: K*(1-exp(-B*x))
NAME
1 K
2 B
VALUE
ERROR
2.22092e+01 6.01178e-01
6.95735e-02 5.06539e-03
Fri Jun 1 16:32:23 PDT 2012
At small numbers of scan points the
effective number of trials seem to be at bit
bigger than number of number of scan
points (e.g., 5 →6)
I think this is caused by the long range anticorrelations mediated by the background ...
A. E. Snyder, SLAC
12
130 GeV?
scanMatrix-5323.root
E.g.: A Tentative Gamma-Ray Line from Dark Matter Annihilation at the Fermi
Large Area Telescope
Christoph Weniger, Munich, Max Planck Inst.
e-Print: arXiv:1204.2797 [hep-ph]

Scans 20-300 GeV using Fermi energy resolution
I compute “trials” using triple Gaussian approx. to line shape due to Andrea Albert
and simple E background, i.e., nothing official for the moment
1M Experiments
It takes me ~45 trials to reproduce the mean of the
resulting distribution
Weniger's bootstrap estimate of effective trials is
12.7
Number of trials needed according to “1  ” rule-ofthumb is ~21 ...
Differences need investigating ... sensitivity to signal
shape? Correlations seem to be smaller for a given
RMS when signal PDF has “narrow” and wide
components ...
Fri Jun 1 16:32:23 PDT 2012
A. E. Snyder, SLAC
13
Comments
●
●
●
Orthogonalizing the “space” of scan points provides an fast, effective way to
estimate the impact of the look-elsewhere-effect
●
Note: flexible background shape allowed, complicated signal shapes accommodated
●
See Gross & Vitells for another approach using an extrapolation from lower significance [a]
Orthogonalization does not eliminate toy MC, but makes it simpler and avoids the
often expensive effort of fitting/scanning each toy experiment. Toys are thrown directly
in scan point space, making them simple and fast
It's not exactly a “trials” factor, but it handles the equivalent task ...
●
●
●
●
●
There does not appear to be a single common effective number of trials
An actual effective number of trials is not needed; the p-value correction (trials factor) can
be read directly off the n plot; an approximate number of trials can be constructed for
purposes of comparison
Comparison with full scan/fit toys should be done (I don't have machinery setup yet)
Generalizations to 2D or to varying signal strength models (e.g., models of dark matter
distribution on the sky) are possible by computing correlations from bin content
variations
The signal shape need not be a “lump”, e.g., two peak structure from dark->Z
[a] A viable/complementary alternative is provided by Gross and Vitells in EPJ (arXiv: 1005.1891)
Fri Jun 1 16:32:23 PDT 2012
A. E. Snyder, SLAC
14