Role of Peer Review in Solving the
Problem of Reproducibility
American Physiological Society 2015
Richard Nakamura, Ph.D.
Director
NIH Center for Scientific Review
NIH . . . Turning Discovery Into Health
NIH’s mission is to seek fundamental knowledge
about the nature and behavior of living systems
and the application of that knowledge to
enhance health, lengthen life, and reduce illness
and disability.
NIH achieves its mission largely through awarding
research grants based upon peer review of applications
from extramural scientists
CSR Peer Review – Fiscal Year 2014
• 86,000 applications received
• 16,000 reviewers
• 237 Scientific Review Officers
• 1,500 review meetings
Grant Success Rates
FY 1978-2013
45%
40%
35%
Success Rate
30%
25%
20%
15%
10%
5%
0%
* FY 2013 success rate is a preliminary estimate
Fiscal Year
NIH Program Level in Nominal Dollars and
Constant 1998 Dollars, FY1998 – FY2014
$35
(In Billions)
$30
$25
$20
$15
$10
$5
$0
1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Appropriation
Appropriation in 1998 Dollars
Number of Applications Received by Fiscal Year
We are caught in a vicious cycle of
dropping success rates (award
rates) and increasing applications
Scientists struggling to stay ahead
in an arms race of intense
competition are amplifying their
story lines and weakening their self
checks, while constantly searching
for more funding.
The number of applications and low
paylines has had a palpable effect
on the perceived quality of peer
review.
In this environment our science
cannot afford the economic waste
of irreproducible results, let alone
any loss of confidence in our
product by the nation.
How to intervene
• The most important intervention point is at journal review of
research reports.
• Here is where the exact study design and methods can be
strongly evaluated and where many editors have agreed to
better reporting of methods including reporting of statistical
power.
• Here is where electronic space can be created to allow
archival publication of methodologic details and data can be
shared so studies can be more easily reproduced.
Is the grant application an
appropriate intervention point?
The conservatism problem
• For decades we have assumed that funded scientists will
employ rigorous methods and that grant applications
(particularly in basic areas) will reflect work in progress and
general directions, rather than a fixed protocol.
• Funding institutes already complain about the conservatism of
peer review due to prioritization of approach.
• Further attending to rigor will increase conservatism.
• NIH has determined that until we collectively turn around the
problem of reproducibility, review conservatism will have to be
accepted.
• CSR will therefore place additional emphasis on premise and
design rigor until the lesson is established.
NIH has not yet finalized the
language providing guidance to
reviewers. This should be resolved
this month.
In the meantime, 1) please adopt
the best practices discussed here
in your experiments and 2) help
rally national understanding of a
science and technology sector
funding crisis.
Questions?
[email protected]