Reproducibility Problem
in Cognitive Psychology
池田 功毅
JSPS PD
中京大学心理学研究科
Execuse
This talk provides neither convicing data nor precise
formalization, but try to deliver a bold idea nevertheless…
Overview
▣ Precision in Measurement
▣ Lakatos’s Research Programme
▣ Troubles in Cognitive Psychology
▣ Solution?
1.
Lakatos’s
Resarch
Programmes
‘’
Sir Karl R. Popper
“In so far as a scientific
statement speaks
about reality, it must be
falsifiable: and in so far
as it is not falsifiable, it
does not speak about
reality.”
Imre Lakatos
(1922 – 1974)
proposed the idea of
“Research Programme.”
Research Programme
Hard
Core
Auxiliary
Hypotheses
When Auxiliary Hyp. are well-defined…
(“Strong theory” by Eysenck)
Falsified!
🔨
Hard
Core
Auxiliary
Hypotheses
When Auxiliary Hyp. are well-defined…
(“Strong theory”)
Not falsified!
🔨
Hard
Core
Auxiliary
Hypotheses
When A. Hyp. are not well-defined…
(“Weak theory”)
🔨
🔨
Hard
Core
🔨Auxiliary
Hypotheses
When the test failes…
?
🔨
? ?
?
🔨
?
?Hard
?
? Core?
?
?
🔨Auxiliary ?
? Hypotheses
When the test succeeds…
🔨
🔨
Hard
Core
🔨Auxiliary
Hypotheses
UnderFailed
a weak
theory,
test
Very hard
to specify
reason.
publication
bias
couldthebe
beneficial.
Not so much informative…
Successful test
That specific research paradigm is correct.
Could be informative for the next research…
As far as the positive reports are not hacked…
2.
Troubles in
Cognitive
Psychology
=
Reproducibility
Direct replication
50%
Cognitive
Social
25%
A direct replication will test the entire
study paradigm.
Hard
Core
Auxiliary
Hypotheses
My claim:
Cognitive psychology might
have been relatively successful
in direct replication, but failing
to establish robust theorycores, because of the
paradigm-dependent research
strategy it employs.
‘’
▣“…once an experimental
paradigm has been established
as a method to investigate an
initial research question, the
following course of research
and theorizing is often limited
to the specific features of the
paradigm itself and to the
effects that the specific
paradigm elicits.”
(Meiser, 2011. Pers. Psychol. Sci.)
Paradigm-dependent research
Hard
Core
Auxiliary
Hypotheses
Paradigm-dependent research
Theory 1b
Theory 1a
Theory 1c
Paradigm-dependent inductive (adhoc) theorization
Only nominally or partially related
Theory 1b
Theory 1a
Theory 1c
Paradigms in executive functions
▣ Stroop
▣ Flanker
▣ Simon
▣ Visual search
▣ Dot-probe (Posner)
▣ Visual working memory
▣ Verbal working memory
▣ Task switching
etc.
What’s good about it?
▣ Given the relatively high success rate of
direct replication in cognitive psychology,
paradigm-dependent strategy is much more
promised than cross-paradigm study in order
to obtain positive results.
▣ Under the culture of “publish or perish,”
there’s no/little incentive to take the risk to
conduct a cross-paradigm study.
▣ However, a “paradigm-space” is inevitably
limited, so that you cannot write an infinite
number of papers based on one paradigm.
Used up!
Used up!
Used up!
Replication &
exploration within
the paradigm
Replication &
exploration within
the paradigm
Replication &
exploration within
the paradigm
Personal
Anecdotes
Quantitative
Evidence?
Publications in Cogn. Psychol.
Working
Memory
Cognitive
Control
My
Favorites
Log Publications in Cogn. Psychol.
Working
Memory
Visual
Search
My
Favorites
My Favorites
Task
Switching
Attentional
Blink
The Decline of AB?
Further research
required,
but I propose…
p hacking
p-value
hacking
paradigm
hacking
p hacking
Paradigm-hacking
▣ Paradigm-dependent ad-hoc theorization
▣ Little genralizability; neighboring theories are
only nominally or partially related to each
other.
▣ “Disposable” paradigms
Is it really a bad thing?
▣ Inefficient (if you don’t like it)
▣ We could be stuck in an endless loop of weak
theory and the status of “soft” science (if you
don’t like it)
▣ I don’t like them.
Irreproducible Theory
Cognitive
Social
Irreproducibile Study
Summary of my claim
▣ The major problem of reproducibility in
cognitive psychology might not be at the
level of individual studies (direct replication),
but theory building (conceptual replication).
▣ Both significantly hinder the advancement of
psychology; i.e. cognitive psychology looks a
bit nicer, but in fact no better than social
psychology.
What’s the
solution?
▣ Paradigm-hacking will go nowhere but to
rethink the strategy someday soon.
▣ But if you are too impatinet to wait (like me)…
How to stop paradigm hacking
▣ A finding must be tested in as many
paradigms as possible in order to prevent the
paradigm-dependent ad-hoc theorization.
▣ However, under the culture of “publish or
perish,” there’s no/little incentive to take the
risk to conduct a cross-paradigm study.
Pre-review & pre-registration
▣ Pre-review & registration could eliminate the
anxiety to obtain negative-results.
Big data cognitive psychology
▣ A bottom-up approach.
▣ Collecting data in a variety of different
paradigms (i.e. big data) might allow you to
model the paradigm-general and –specific
components of a psychological mechanism.
▣ I am not aware of any attempts in this
direction yet…
Adaptive computational model
▣ A top-dwon approach.
▣ Cognition must be adaptative in the timespan of biological evolution or individual
development.
▣ “The adapated mind” must be independent
from artificial paradigms.
▣ Domain-specificity (of evol. psychol.) is not a
necessary constraint.
Thanks!