SHAPING JOURNAL EDITORIAL POLICY
TO INCREASE REPLICABILITY AND
TRANSPARENCY
D. Stephen Lindsay
Wi-Fi?
Transparency and Openness Promotion Guidelines
(757 journals and 63 Organizations)
https://cos.io/our-services/top-guidelines/
Eric Eich’s Interventions with Psych Science
• Badges to reward sharing of materials,
data, pre-registration.
Percentage of articles claiming to have posted data
Link to Feb TOC
From Kidwell et al., 2016, COS
http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002456
Eric Eich’s Interventions with Psych Science
• Badges to reward sharing of materials,
data, pre-registration.
• Requirements to justify N, report datacollection stopping rule, disclose COIs.
• Requirement to disclose all IVs and DVs
• Removed word-count limits on Method and
Results sections
• Encourage alternatives to NHST, require
measure of ES and precision (e.g., CIs).
Avoid publishing papers that report a single underpowered study with surprising findings.
Encourage preregistration.
Research Preregistration 101
D. STEPHEN LINDSAY, DANIEL J. SIMONS, SCOTT O. LILIENFELD
Preregistration = Date-stamped plan for study
making clear what (if any) analyses and outcomes
were planned and predicted.
Very detailed discussion of preregistration:
van ‘t Veer, A. E., & Giner-Sorolla, R. (2016). Pre-registration in social
psychology — A discussion and suggested template. Journal of
Experimental Social Psychology, 67, 2–12. doi:10.1016/j.jesp.2016.03.004
Avoid publishing papers that report a single underpowered study with surprising findings.
Encourage preregistration.
Invite revisions with preregistered replications.
Avoid publishing papers that report a single underpowered study with surprising findings.
Encourage preregistration.
Invite revisions with preregistered replications.
Encourage sharing data with reviewers (and more
widely after publication).
Require fine-grained graphical presentations of data.
Anscombe’s quartet
4 data sets, each
with the same mean
of X, the same SD of
X, the same mean of
Y, the same SD of Y,
the same correlation
between X and Y,
and the same linear
regression line.
A
B
C
D
C1=Congruent, C2 = Incongruent
O = bilinguals, X = monolinguals.
DV = RT
E. J. Wagenmakers
In each, congruency effect twice as big for mono as for bi, but whereas A and
C at least directionally support hypothesis of bilingual advantage in executive
control, B and D do not.
Avoid publishing papers that report a single underpowered study with surprising findings.
Encourage preregistration.
Invite revisions with preregistered replications.
Encourage sharing data with reviewers (and more
widely after publication)
Require fine-grained graphical presentations of data.
Require authors to address known/anticipated
constraints on generality.
Constraints on Generality Statements
(COGS)
Dan Simons, Yuichi Shoda, and I are revising a
paper promoting idea of requiring authors to address
the known/expected constraints on the generality of
their findings.
Dan Simons
Yuichi Shoda
Avoid publishing papers that report a single underpowered study with surprising findings.
Encourage preregistration.
Invite revisions with preregistered replications.
Encourage sharing data with reviewers (and more
widely after publication).
Require fine-grained graphical presentations of data.
Require authors to address known/anticipated
constraints on generality.
Use tools to reduce/detect errors.
StatCheck
Michele B. Nuijten
Tilberg University
Detects internal inconsistencies between elements of
widely used inferential statistical tests (e.g., DF, t, and p).
Richard and Candice Morey’s “Data
Buddies”
• Encourage your students to make deals with
students in other labs to attempt to reproduce one
another’s data analyses based on the posted info.
Psychological Science Statistical Advisors
Emilio Ferrer
(UC Davis)
Greg Francis
(Purdue)
Jennifer Krull
(UCLA)
Tessa West
NYU
Monica Melby-Lervǻg
U. Oslo
Richard Morey
Cardiff
Shift cultural expectations and incentive systems away
from encouraging maximizing the N of minimally
publishable papers.
Lief Nelson, Joe Simmons, Uri Simonsohn (2012)
http://urisohn.com/sohn_files/papers/fewer.pdf
Questions?