From NHST to the New Statistics:
How do we get there?
APS Convention, Boston 26 May 2017
Geoff Cumming
La Trobe University, Australia [email protected]
In an Open Science world, why The New Statistics, and what are the
roadblocks?
Susan A. Nolan
Seton Hall University [email protected]
The Textbook Author’s Perspective: Roadblocks and Opportunities for
Incorporating The New Statistics Into Teaching Materials
Tamarah Smith Cabrini University [email protected]
& Robert J. Calin-Jageman Dominican University [email protected]
The Instructor’s Perspective: Moving Your Students (and Department)
Towards The New Statistics
D. Stephen Lindsay
Victoria University, Canada [email protected]
The Editor’s Perspective: What Happened When Psychological Science
Began Encouraging the Use of The New Statistics?
1
Why The New Statistics?
What are The New Statistics?
 Estimation
 Effect sizes
 Confidence intervals
 Meta-analysis
 …not NHST, p values
2
Why The New Statistics?
The best answer to a
research question is a
confidence interval
50
51
52
53
54
55
5
3
Why The New Statistics?
The best answer to a
research question is a
confidence interval
50
51
52
-50
-40
53
-30
-20
-10
54
0
10
20
55
30
5
40
.309
CIs are way more informative
than p values
Successive experiments
**.002
.104
.110
**? .005
.051
*.048
***.000
**.007
*.027
? .096
? .079
***.001
.327
? .067
.806
***.000
**.001
.303
**? .009
.092
.713
*.035
.456
*.013
H0
4
Why The New Statistics?
The best answer to a
research question is a
confidence interval
50
51
52
-50
-40
53
-30
-20
-10
54
0
10
20
55
30
5
40
.309
CIs are way more informative
than p values
Successive experiments
**.002
.104
.110
**? .005
.051
*.048
***.000
**.007
*.027
? .096
? .079
***.001
.327
? .067
.806
***.000
**.001
.303
**? .009
.092
.713
*.035
.456
*.013
Open Science needs replication,
which needs meta-analysis,
which needs estimation
-0.5
H0
0
0.5
d unbiased
1
1.5
2
5
The New Statistics:
Why has it still not happened?
NHST and p values simply persist
…but they are NUTS!
6
Here’s how we do it

Make an assumption, which

you believe, and desperately hope, is not true

is almost certainly not true
Here’s how we do it


Make an assumption, which

you believe, and desperately hope, is not true

is almost certainly not true
Calculate a strange quantity, which

most people don’t really understand

is only partly related to what matters
Here’s how we do it



Make an assumption, which

you believe, and desperately hope, is not true

is almost certainly not true
Calculate a strange quantity, which

most people don’t really understand

is only partly related to what matters
Compare that quantity with an arbitrary benchmark

if below, reject the assumption you didn’t believe, declare
statistical <misleading word> and enjoy the fruits of success

if not below, don’t reject the assumption, but despair, conjure up
ad hoc excuses, and re-consider your life goals
Here’s how we do it



Make an assumption, which
Null hypothesis, H0

you believe, and desperately hope, is not true

is almost certainly not true
Calculate a strange quantity, which

most people don’t really understand

is only partly related to what matters
p value
Compare that quantity with an arbitrary benchmark .05 (.01?)

if below, reject the assumption you didn’t believe, declare
statistical < significance > and enjoy the fruits of success

if not below, don’t reject the assumption, but despair, conjure up
ad hoc excuses, and re-consider your life goals
Nuts!
…fake statistics!
The NHST story in quotes

1960s “The traditional null-hypothesis significance-test method ... is here
vigorously excoriated for its inappropriateness” (Rozeboom, 1960).

1970s “I’m not making some nit-picking statistician’s correction. I am saying
that the whole business [NHST] is so radically defective as to be scientifically almost
pointless” (Meehl, 1978).

1980s “It is remarkable that despite two decades of … attacks, the mystifying
doctrine of null hypothesis testing is still today the Bible from which our future
research generation is taught” (Gigerenzer & Murray, 1987).

1990s “It is time to go beyond this institutionalized illusion [NHST]. We must
write new textbooks and change editorial practices” Gigerenzer (1993).

2000s “The current practice of focusing exclusively on a dichotomous rejectnonreject decision strategy of null hypothesis testing can actually impede scientific
progress. …The focus of research should be on … what data tell us about the
magnitude of effects, the practical significance of effects, and the steady
accumulation of knowledge” (Kirk, 2003).

2010s “In the post p < .05 era, scientific argumentation is not based on whether
a p value is small enough or not. Attention is paid to effect sizes and confidence
intervals” (Ron Wasserstein, ASA President, March 2016).
Valiant efforts at statistical reform
Ken Rothman


Since 1980s medicine has reported CIs… but still uses p values
Epidemiology (’90-’00) : NO p values!
Fidler, F., Thomason, N., Cumming, G., et al. (2004). Editors can lead researchers to confidence intervals,
but can’t make them think: Statistical reform lessons from medicine. Psychological Science, 15, 119-126.
Geoff Loftus

Memory & Cognition (’94-’97): Error bars, CIs, but still p values
Finch, S., Cumming, G., Williams, J., et al. (2004). Reform of statistical inference in psychology: The case
of Memory & Cognition. Behavior Research Methods, Instruments & Computers, 36, 312-324.
APA taskforce (TFSI), APA Manual

10 top journals (’98-’06): CIs 4% to 11%, but p values 98%
Cumming, G., Fidler, F., Leonard, M., Kalinowski, P., Christiansen, A., Kleinig, A., ... & Wilson, S. (2007).
Statistical reform in psychology: Is anything changing? Psychological Science, 18, 230-232.
Psychonomic Society

Plan N, report ESs, use CIs. (’13-’15): Small improvements only.
Morris, P.E. & Fritz, C.O. (2017). Meeting the challenge of the Psychonomic Society’s 2012 Guidelines on
Statistical Issues: Some success and some room for improvement. Psychonomic Bulletin & Review.
doi:10.3758/s13423-017-1267-y.
Psychological Science …
Valiant efforts at statistical reform
Encouraging signs

Effect size reporting is now common

CI use is increasing, but patchy

Open Science is advancing, but early days
BUT

CIs are rarely used for interpretation

NHST and p values are STILL close to universal 

Double nuts!
…fake statistics!
p values and NHST: WHY have they persisted?!
Psychologists… have, for the last 40 years or so, been almost wilfully
stupid. What explanations can be offered for their failure to acknowledge,
at a much earlier date, the cogency of these arguments [against NHST]?
(Oakes, 1986)
Roadblocks?

“The tyranny of the discontinuous mind” (Dawkins)


Yearning for certainty
NHST as security blanket? As addiction?

Deeply embedded in the researcher’s psyche
Fiona Fidler’s PhD thesis: tiny.cc/fionasphd
…watch for the book!
p values and NHST should quietly die
How to overcome the roadblocks?
Rational argument won’t suffice. Dramatic demonstrations?
Dance of the p values tiny.cc/dancepvals
Significance roulette
Search at
YouTube
tiny.cc/SigRoulette1
tiny.cc/SigRoulette2
Teach estimation and Open Science
Enjoy!
Learn from our students that p values are superfluous
The OLD: The .05 imperative
I must get under,
I must get under,
I must get under…
…that’s all that
really matters
http://imgur.com/r0PNlFK
18
The OLD: The .05 imperative
I must get under,
I must get under,
I must get under…
…that’s all that
really matters
http://imgur.com/r0PNlFK
The NEW: Open Science, The New Statistics
19
Video: Significance Roulette 2
tiny.cc/SigRoulette2
20
From NHST to the New Statistics:
How do we get there?
APS Convention, Boston 26 May 2017
Geoff Cumming
La Trobe University, Australia [email protected]
In an Open Science world, why The New Statistics, and what are the
roadblocks? www.thenewstatistics.com
Susan A. Nolan
Seton Hall University [email protected]
The Textbook Author’s Perspective: Roadblocks and Opportunities for
Incorporating The New Statistics Into Teaching Materials
Tamarah Smith Cabrini University [email protected]
& Robert J. Calin-Jageman Dominican University [email protected]
The Instructor’s Perspective: Moving Your Students (and Department)
Towards The New Statistics
D. Stephen Lindsay
Victoria University, Canada [email protected]
The Editor’s Perspective: What Happened When Psychological Science
Began Encouraging the Use of The New Statistics?
21
Introduction to the new statistics … and Open Science
tiny.cc/itnsroutledge
www.thenewstatistics.com
1. Research questions: Estimation, CIs, meta-analysis, Open Science
2. Research fundamentals: Don’t fool yourself. Inference, measurement, Open Science
3. Picturing data: Descriptive statistics, normal distribution
4. Sampling: Sampling distributions, SE, central limit theorem
5. Confidence intervals, effect sizes, interpretation
6. NHST: p values. Relation with CIs; p value red flags
7. Independent groups design. Cohen’s d. Dance of the p values
8. Paired design. Cohen’s d
9. Meta-analysis: Forest plots, meta-analytic thinking
10. Open Science and planning research. Precision, power
11. Correlation: Scatterplots, r, the CI on r
12. Simple linear regression, relationship with correlation
13. Categorical data: Frequencies, proportions, and risk
14. Extended designs: One IV, planned contrasts
15. Extended designs: Two IVs, interactions
16. Open Science and future directions
22