REPRODUCIBILITY
Or, “Who knew that replication could be so complicated?”
A LITTLE HISTORY FIRST…
“WHY MOST PUBLISHED FINDINGS ARE
FALSE”
• Ioannidis, 2005, biostatistician
• Clinical trials, epidemiological studies, molecular research
• Less likely to be true if
• Studies are smaller
• Effects are smaller
• More tests are done
• The more flexibility there is in design
• The more financial interests, etc. are involved
• The “hotter” the field
Ioannidis, J. P. A. (2005). Why most published findings are false. PLoS Med, 2, e124.
FALSE-POSITIVE PSYCHOLOGY
• Simmons, Nelson, & Simonsohn, 2011
• Listening to “When I’m Sixty-Four” can make
you younger!
song
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False positive psychology: Undisclosed flexibility in data
collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366.
CAN WE KNOW THINGS BEFORE THEY
HAPPEN?
Bem, D. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on
cognition and affect. Journal of Personality and Social Psychology, 100, 407-425.
BARGH, CHEN, & BURROWS, 1996
(STUDY 2)
• When you’re primed with “old people” words, you walk more slowly.
• Florida, old, lonely, grey, retired, bingo
Walking speed by condition
8.4
8.2
Time in sec
8
7.8
7.6
7.4
7.2
7
6.8
6.6
study 1
study 2
old age
neutral
Bargh, J. A., Chen, M., & Burrows, L. (1996). Automaticity of social behavior: Direct effects of
trait construct and stereotype-activation on action. Journal of Personality and Social Psychology,
71, 230-244.
DOYEN REPLICATION, 2012
• You only get the effects if you tell the experimenter that the person is supposed to
walk slowly.
Walking speed by condition
7.5
Time in sec
7
6.5
6
5.5
5
no expection
expect slow
old age
neutral
expect fast
Doyen, S., Klein, O., Pichon, C., & Cleeremans, A (2012).
Behavioral priming: It’s all in the mind, but whose mind? PLoS One,
7, e29081.
GENETICS STUDIES DON’T ALWAYS
REPLICATE.
• Only
44% of associations replicated
Lohmueller, K. E., Pearce, C. L., Pike, M., Lander, E. S., & Hirschhorn, J. N. (2003). Meta-analysis
of genetic association studies supports a contribution of common variants to susceptibility to
common disease. Natural Genetics, 33, 177-182.
ECONOMICS STUDIES DON’T ALWAYS
REPLICATE.
• Only
25-63% of studies replicated (using the same data!)
Dewald, W. G., Thursby, J. G., & Anderson, R. G. (1986). Replication in empirical economics: The
Journal of Money, Credit, and Banking Project. American Economic Review, 76, 587-603.
Chang, A. C., & Li, P. (2015). Is economics research replicable? Sixty published papers from
thirteen journals say “usually not.” Finance and Economics Discussion Series 2015-083.Washington:
Board of Governors of the Federal Reserve System.
OTHER EXAMPLES
• Cancer epidemiology
• Ecological and evolutionary biology
• Neuroscience
Boffetta, P., McLaughlin, J. K., La Vecchia, C., Tarone, R. E., Lipworth, L., Blot, W. J. (2008).
False positive results in cancer epidemiology: A plea for epistemological modesty. Journal of the
National Cancer Institute, 100, 988-995.
Jennions, M. D., & Moller, A. P. (2002). Relationships fade with time: A meta-analysis of temporal
trends in publication in ecology and evolution. Proceedings of the Royal Society of London, 269, 43-48.
Wager, T. D., Lindquist, M., & Kaplan, L. (2007). Meta-analysis of functional neuroimaging data: Current
and future directions. Social Cognitive and Affective Neuroscience, 2, 150-158.
IS THE FAILURE TO REPLICATE A
PROBLEM?
• Pashler & Harris, 2012
• Why aren’t these enough?
• Alpha = .05
• Conceptual replications
• The self-correcting nature of science
EXPERIMENTAL ECONOMICS REPLICATION
• Replications of 18 studies from 2 econ journals by 18 researchers
• 67% similar results
• 61% had significant results in the same direction
• Camerer, C. F. et al. (2016). Evaluating replicability of laboratory experiments in economics.
Science, Published online 10.1126/science.aaf0918
REPRODUCIBILITY PROJECT: CANCER
BIOLOGY
• 37 studies being replicated
• So far, 2 replicated, 2 were unclear, and 1 did not replicate
Errington, T. M., Iorns, E., Gunn, W., Fraser, E. T., Lomax, J. & Nosek, B. A. (2014). Science Forum: An open
investigation of the reproducibility of cancer biology research. eLife, 3:e04333.
•
OPEN SCIENCE COLLABORATION, 2015
• 100 replications by 270 authors from 2008 articles in
• Psychological Science
• JPSP
• JEP: LMC
• How did they choose which studies to replicate?
• How do you know if a study replicates?
• Table 1 methods
• Pros and cons of each method?
• What is related to whether something replicates?
• Table 2
• What does a failure to replicate mean?
WHY IS REPLICATION SO HARD?
• Maxwell, Lau, & Howard, 2015
• Patil, Peng, & Leek, 2016
• Replication studies are usually underpowered
• Need more than .80 power
• Overestimation of ES
• Sampling variability of ES
• Conditional vs. predictive power
• Confidence intervals might lead to different predictions.
ALTERNATE WAYS TO LOOK AT
REPLICATION
• Prediction intervals
• Equivalence region
• Bayesian
• ROPE
• Bayes factor
WAYS TO GET BETTER REPLICATIONS
• Use really big samples and good designs
• Cumulative meta-analyses (fixed vs. random)
• Crowdsourcing
IS THIS A SOCIAL PSYCH PROBLEM?
• https://replicationindex.wordpress.com/2017/03/01/3950/
• Replication best for cognitive, then (clinical, bio, developmental) then social
• Uses z-curve
Uli Schimmack (FB)
WHAT ABOUT CLINICAL PSYCH?
• Tackett et al., in press
• Implications for false positives
• ESTs
• http://eiko-fried.com/category/clinical-trials/
IS CLINICAL SCIENCE DIFFERENT?
• Bigger effects
• Correlational research
• No hypotheses
• Multiple measures
• Undetected moderators assumed
• Less emphasis on individual study findings
• Fewer QRPs
• Single study articles
• Moderators examined
GOALS (TACKETT ET AL., IN PRESS)
• Decrease QRPs
• Big studies and their effects
• Keep up-to-date on p-hacking, HARKing, QRPs
• Report all relevant DVs
• Preregister study hypotheses
• Think about particular clinical QRPs
• Pre-registration and open data
• Big datasets
• Procedures change
• Hypotheses change
• Sensitive data
• Document moderators
• Independent replication
• Expensive
• Changes in clinical diagnoses
• Replicate across datasets
• Build replication into studies
• Self-examination and replication
• Be wary of big ES
• Correlational research isn’t an issue?
• Think about sample size, publication bias
• Report failed replications
• Form groups to deal with all this
• Increase power
• Increasing sample size is hard
• Do power analysis
• Use multiple measures
• Share measures across studies
• Focus on dimensions instead of categories
THEIR CONCLUSIONS/SUGGESTIONS
• What do you think of their suggestions?
IMPEDIMENTS TO OPEN SCIENCE
• Reviewers and the review process
• Article I shared
• Others?
SO IS THERE A PROBLEM?
• And if so,
• How big is it?
• What are the best solutions?
HOW WOULD YOU RESPOND TO THE
PHYSICIST?
OTHER THOUGHTS FROM THE TALKS?
• http://osf.io/bqbt3
PRESENTATIONS
• Any questions?
• Week after spring break, plus 3 the next week
• Don’t forget to get your committees to me
PEER REVIEW
• Have a complete draft ready to give your partner.
• Email it to him/her by midnight on Monday, April 3.
• cc me on the email
• Provide comments throughout the paper
• Summarize with 2-3 strengths and 2-3 things to work on
• Meet and go over the comments with the person
• Email them the draft back by midnight on Wednesday, April 5 (no class that day—use that time
to meet)
• cc me on the email
• Final paper due one week later (along with validity and ethics appendices)
• Ellie – Kristin
• Toni – Tim
• Jerome – Hailey
• Alba to Emma
• Emma to Sage
• Sage to Alba
• You can also get feedback from your thesis supervisor, other people in class, your
mom, whomever.