The Case for Standards in
Life Science Research
Seizing Opportunities at a Time of Critical Need
Research conducted by:
About GBSI
The Global Biological Standards Institute
(GBSI) advances life science standards to
enhance global health. Through policy
initiatives, thought leadership, and
education, GBSI seeks to expand the
development and adoption of standards
in support of credible, reproducible, and
translatable outcomes in life science
research. The Washington, D.C.-based
institute provides an independent forum
for government, non-governmental
organizations, industry, academia, and
other stakeholders to ensure that
biological standards development
is focused on priority needs.
Table of Contents
Contents
Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2
Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5
Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6
Current State of Standards in
Life Science Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7
Tackling Irreproducibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9
Negative Effects of Irreproducibility:
The Case for Standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20
Life Science Research Standards:
Adding Value and Accelerating Progress . . . . . . . . . . . . . . .26
Moving Forward to
Solve the Problem of Irreproducibility . . . . . . . . . . . . . . . . .33
Appendix I: Supporting Materials . . . . . . . . . . . . . . . . . . . . . . .34
Appendix II: List of Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . .41
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
1
Foreword
Reproducibility is the Foundation of Life Science Research.
Reproducibility self-corrects for random observations, unconscious experimental bias and selective
reporting, and poorly designed studies. Yet far too often, the inability to reproduce experimental data
in the life sciences has resulted in the invalidation of research breakthroughs, retraction of published
papers, abrupt discontinuation of studies, and reduced trust in the research enterprise. More alarming—
to the scientific community, research funders, patient advocates, and the public at large—is that
valuable time and resources are wasted by irreproducibility, and opportunities to enhance global
health are delayed or lost.
While the causes of irreproducibility are complex, we know from many sources, including this Report,
that many of the causes can be traced to the absence of a unifying standards framework. The concept of
standards is not new, nor specific to the life sciences. Standards have in fact served as the foundation
of progress throughout millennia. Many advancements that we take for granted every day, including
language, light bulbs, bridges, Wi-Fi, and the internet, would not be possible without the development
and adoption of global standards within their respective industries. But unlike other fields of endeavor,
life science research has very few broadly-implemented standards.
It’s Time to Act.
The global community can no longer afford the economic and intellectual drain that is caused by
irreproducible research. GBSI and several independent organizations, including prestigious journals
and the U.S. National Institutes of Health, have recognized the urgent need and are stepping up to
address the issue. These early efforts must now be augmented with a much larger collaborative and
unifying effort that engages and mobilizes all stakeholders across the life sciences to advance the
development and adoption of standards.
Everyone is affected by irreproducibility, from bench scientists to patients. Everyone is a stakeholder
in these efforts. Everyone should have a voice. Everyone includes you.
Together, let’s solve this urgent challenge.
For more information or to learn how to get involved in community-wide efforts to effect meaningful
change, I invite you to visit GBSI at www.gbsi.org.
Leonard P. Freedman, Ph.D.
President
Global Biological Standards Institute
2
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
Executive Summary
In recent years, tremendous advances in the life sciences have improved the lives of millions of people.
Breakthroughs in genomics and other technologies presage an unprecedented era of understanding
human biology and disease. Recently, however, a rising tide of questions about the quality and
reproducibility of research findings have arisen in scientific journals and in the lay press. Because the
quality of scientific research is critical to both the life science community and the general public, the
Global Biological Standards Institute (GBSI)—a multidisciplinary, non-profit organization dedicated to
the advancement of life science (also known as biological) standards—commissioned this independent
third-party evaluation of the life science research landscape. The objectives were to:
• Evaluate the current state of research quality;
• Identify areas of concern;
• Highlight how the development and widespread adoption of standards
could be beneficial to producing credible, reproducible, and translatable
outcomes in life science research; and
• Illuminate the case for standards.
This Report is based on the results of nearly 60 in-depth interviews with prominent life science research
stakeholders, including academic research leaders, biopharmaceutical R&D executives, government
agency personnel, editors of peer-reviewed journals, and leaders of professional societies. In addition,
an extensive literature review and an analysis of data from multiple sources, including the National
Institutes of Health, National Science Foundation, and PhRMA, were performed.
The Life Science Research Landscape Is Changing
Traditionally, the quality of life science research has been maintained through the peer review process
in the short-term and “self-correction” of erroneous hypotheses and results through additional studies
over the longer-term. Recent changes in the life science research environment are putting pressure on
these traditional systems and decreasing their effectiveness. These changes include:
• Increasing complexity associated with growth of high-throughput, specialized technologies
such as next generation sequencing and the emerging “omics”;
• Expanding volume of publications in a resource-strapped environment; and
• Growing focus on translating research findings into clinically-actionable results.
Irreproducibility Is a Pervasive, Systemic Problem
The evolution of the life science research landscape has led stakeholders to pay greater attention to the
quality of research. In particular, there is increased awareness of and concern about irreproducibility
of research findings. Studies in both commercial and academic settings show that irreproducibility is
ubiquitous across settings. Similar experiences were shared by the life science research stakeholders
interviewed for this Report.
All life science research stakeholders—including academic laboratories and institutions, industry and
industry associations, investors, government agencies, charitable foundations, journals, professional
societies, standards development organizations, and policy and patient advocacy organizations—
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
3
Executive Summary
need to confront the challenges that result from irreproducible research. Irreproducibility has extensive
negative and lasting effects, including:
• Inefficient use of time and resources across the life science research space,
with high profile instances costing millions of dollars;
• Damage to reputations of laboratories, institutions, journals, and funders;
• Impairment of industry-academic relationships; and
• Overall less favorable public opinion of life science research.
Irreproducibility is a pervasive, systemic problem that has multiple direct and systems-level causes.
These causes range from individual laboratory decisions, such as lack of an appropriate control for a
particular experiment; to variability in journal publication requirements; to lack of consistent research
quality systems. Fundamentally, irreproducibility stems from undefined variance in reagents, practices,
and assays between laboratories.
Standards Are the Solution
Irreproducibility can be traced to an absence of a unifying standards framework. Biological standards
are reference materials or consensus documents generated by professional communities to reduce
differences in practices. Hence, standards provide an effective and feasible solution to problems
stemming from uncontrolled variation. Standards have been successfully used in countless other
fields, including other laboratory-based disciplines, to successfully reduce variability and improve
quality and outcomes. However, few broadly-implemented standards exist in life science research.
Because irreproducibility is highly prevalent and its effects are profound and lasting, all stakeholder
groups interviewed for this Report agree that there is a need for additional standards in life science
research. Multiple standards can be envisioned; for example, standards to ensure quality can be
developed and implemented for reagents, assays, laboratory practices, and data analysis and reporting.
The development and implementation of standards, whether as adherence to written documents or
use of reference materials, requires community consensus and alignment around both the necessity
for standards and their content. Ultimately, expanded adoption of life science standards will require:
• Educational initiatives to raise stakeholder awareness of the purpose and benefits of
biological standards and understanding of the standards development process;
• Opportunities and forums for stakeholders to identify areas in the life sciences
where accelerated standards adoption could provide maximum benefit;
• Engagement of stakeholders with standards development organizations or material
reference providers in the development of specific standards; and
• Development of effective policies and practices within the life science research community
to ensure the proactive development and periodic updating of biological standards.
By increasing reproducibility, standards can benefit all stakeholders, counteract quality challenges,
and be a unifying driver of current and future life science research quality improvement efforts.
Uniting diverse life science research stakeholders in a standardization effort will be challenging.
Nonetheless, influential entities within the life sciences community can all join in a global initiative
to exchange ideas, identify areas of greatest need, and advance the development and adoption of
biological standards.
4
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
Introduction
“Standard” is an inflammatory word for many biological scientists, one that conjures up images of
bureaucracy, regulation, and paragraphs filled with obscure references and alphanumeric designations.
In reality, life science (also known as biological) standards are either highly characterized reagents or
written documents that outline community consensus around certain practices.1 Although government
agencies can adopt specific standards as regulations, adherence to a majority of standards is voluntary.2
Standards have extensively benefitted many fields, from the railroad industry to clinical diagnostics,
and standards underpin almost all the capabilities of modern life. However, although some life science
research standards, such as those dealing with animal welfare, are broadly used, overall the adoption
of standards in life science research has been limited. Are more life science standards necessary? The
project culminating in this Report was undertaken to answer this question.
Scientific research has entered a new realm of complexity. Researchers are discovering targets more
quickly, and the increasing understanding of pathway biology has necessitated multidisciplinary
approaches to “omics.” The environment is also competitive, with a strong drive to make and report
new discoveries and to do so in a cost-effective way. There is a paramount need for each breakthrough
to be a quality output that is also timely and economical.
Over the past several years, concerns about the quality of research have garnered the attention
of multiple stakeholders in the scientific community.3–6 Specifically, widespread and pervasive
irreproducibility of research findings has been documented. As a consequence, millions of dollars of
research funds have been wasted, and researchers must confront a small but growing negative public
perception of scientists and the scientific method. Periodically, the scientific community has voiced
concerns about irreproducibility, and some stakeholders have started to take action. However, thus
far there has been no coordinated global effort to ameliorate these problems.
The scientific community feels the profound effects of research irreproducibility and is ready to find a
unifying solution to this problem. Standards can be the solution. Standards are a vehicle for reducing
irreproducibility by aligning the community around consensus-based methods. Life science research
stakeholders have highlighted the need for research standards as a way to increase reproducibility of
research findings. Better understanding of where, how, and why standards can help the life science
community in the future is one of the first steps to meet this need.
The purpose of the research conducted to generate this Report is to evaluate the current state of
the quality of life science R&D methodologies, identify areas of concern, and highlight why the
development and widespread adoption of standards could be beneficial. The Report is a catalyst for
action to inform and mobilize stakeholders, and to establish recommendations for enabling change.
1. ISO/IEC Guide 2:2004, definition 3.2.
2. American National Standards Institute (ANSI). [Internet] [cited 2013 Sep 17]. Available from
http://www.ansi.org/about_ansi/faqs/faqs.aspx.
3. Ioannidis, J.P. PLOS Med. 2005; 2(8):e124.
4. Mobley, A. PLOS ONE. 2013; 8(5):e63221.
5. Westphal, S. “Unverified Science.” Boston Globe. 2013 Jul 8.
6. Jasny, B. et al. Science. 2011; 334(6060):1225.
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
5
Methodology
This summary Report, based on research commissioned by GBSI to an independent organization in
2013, reflects the opinions of multiple stakeholders and proposes broad-based solutions.
The Report is based on an extensive research program comprised of broad secondary research and
a detailed primary interview process. The secondary research program included a literature review
and synthesis of over 100 peer-reviewed and other references from sources such as government
agencies. Many of these are cited in the footnotes of this Report. Analysis of data from multiple
sources—including the National Institutes of Health (NIH), National Science Foundation (NSF),
and PhRMA—was performed. The primary interview component consisted of almost 60 in-depth,
individual discussions with life science research stakeholders, including:
• Academic research leaders;
• Academic and industry
researchers;
Figure 1. Stakeholder Interviews
N=57
• Biopharmaceutical company
60
research and development
Experts in Standards
(R&D) executives;
Experts in Quality Management
50
• Editors of peer-reviewed
Editors of Peer-Reviewed Journals
professional societies;
• Executives at charitable
foundations that fund basic
and translational research;
Number Interviewed
scientific and clinical journals;
• Leaders of scientific
Leaders of Scientific Professional Societies
40
Charitable Foundation Executives
Academic & Industry Researchers
30
Government Agency Personnel
Biopharmaceutical R&D Executives
20
• Government agency personnel;
• Experts in quality management;
10
Academic Research Leaders
and
• Experts in standards.
0
Interviews were based on discussion
guides developed for specific stakeholder
groups. Interviewees had no knowledge
Interviewed Stakeholders
Note that in addition to stakeholders with exclusively journal editorial or
professional society leadership responsibilities, multiple interviewees in
the academic research leader group also serve in these capacities
of the research sponsor during the
interviews,7 and all interviewee comments
remained anonymous. These candid discussions yielded insights that were analyzed both qualitatively
and quantitatively.
The findings of the secondary research program, analysis, and stakeholder feedback were assimilated
into this summary Report.
7. After the conclusion of the interview, some interviewees chose to release their names to the interviewer and the sponsor without
connection to any comments made during the interviews. In these cases, the sponsor’s name was revealed to the interviewee.
6
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
Current State of Standards in Life Science Research
Defining Standards
Before a community can internalize issues surrounding standards in life science research, it needs
to understand what “standards” are. Standards align people around a consensus-based method for
performing important functions. Two commonly described categories of standards are materials
standards and written consensus standards (Figure 2). Materials standards are highly characterized
physical substances, such as chemical or biological reagents (known as reference materials or
reference reagents), and are routinely used in several areas of life science research and healthcare.
Written consensus standards are documents that outline community consensus around certain
practices. Document (or “paper”) standards can cover a broad range of practices, such as designating
names, defining common sizes, outlining processes or procedures, and establishing quality systems.
Because irreproducibility is highly prevalent and its effects are profound and lasting, all stakeholder
groups interviewed for this Report agree that there is a need for additional standards in life science
research. Multiple standards can be envisioned; for example, standards to ensure quality can be
developed and implemented for reagents, assays, laboratory practices, and data analysis and reporting.
Figure 2. Definition and Types of Standards
Standard: A document or biological material that serves as a guideline
and is established through broad community consensus
Materials Standards: Well-characterized, purified
biological or chemical reference materials
Reference Reagents
Written Consensus Standards: Consensus
documents describing optimal practices
Specification Standards
Analytic Standards
• Used for assay
validation and
calibration or
directly in research
and development
• State an agreed
value for something
and may specify
materials standards
used for calibration
• Can specify assay
methodology, cutoff values, and/or
calibration reagents
• Describe optimal
procedures for
performing a
defined task
• Example: Standard
viral strains; size
standards for mass
spectrometry
• Example: Meter is
the length travelled
by light during
1/299,792,458 of
a second
• Example:
ANSI/ATCC ASN0001 standard
provides a uniform
methodology for
assaying activities
of anthrax toxins
• Example: MIAME
criteria for
microarray data
reporting; standards
for ethical treatment
of laboratory
animals
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
Procedural Standards
Systems Standards
• Establish systems to
enable consistency
of practices,
including use of
other standards
• Example: ASQ or
ISO 9000 quality
management
standards outline
effective quality
management
systems
7
Current State of Standards in Life Science Research
Although government agencies can adopt specific community standards as government regulations,
thereby making them mandatory, adherence to standards is most frequently voluntary. Adherence
to voluntary standards is commonly checked through accreditation or certification, a process during
which an organization is inspected by an independent body to ensure that it is adhering to standards.
Standards are broadly used in a variety of fields. Both materials and document standards have been
extremely effective in improving reproducibility and quality of laboratory-based fields, such as
blood banking and clinical laboratory diagnostics. The fact that standards have been a part of these
evolving fields for many years has allowed for documentation of benefits that can be achieved through
standards development and implementation over time.8
Current Standards in Life Science Research
In life science research, however, standards are not as prolific. The most well-developed standards
and accreditation system exist to protect laboratory animal welfare. In the United States, this system
is centered on the Guide for the Care and Use of Laboratory Animals, a collection of standards for
appropriate laboratory animal care, initially published
in 1963 and revised multiple times thereafter.
Adherence to the principles in the guide is required
“Very few standards apply to biological research. It can
be a significant problem, particularly for assays,
because everyone is doing something different and
it is difficult to compare results across studies.”
for institutions that receive Public Health Service9
support (including grants from the NIH and CDC10),
and is monitored internally by each institution through
the self-regulating Institutional Animal Care and
Use Committees.11
Academic Research Leader
Materials standards are also used in some research
communities, particularly in areas like vaccine and
gene therapy development and microbiology. Some
research communities have also developed standards that apply to specific fields, such as standards
for characterizing and identifying human embryonic stem cells.12 But other than in such highly defined
areas, few established and broadly-implemented standards exist in life science research. Is this a
concern? The feedback from the scientific community interviewed for this Report suggests that it is.
8. For more examples of clinical standards, please see Appendix I.
9. Committee for the Update of the Guide for the Care and Use of Laboratory Animals. Guide for the Care and Use of Laboratory Animals,
8th Ed., NRC 2011.
10. Centers for Disease Control and Prevention (CDC).
11. Institutional Animal Care and Use Committee (IACUC). [Internet] [cited 2013 Aug 18]. Available from http://www.iacuc.org/aboutus.htm.
12. Loring, M.F. and Rao, M.S. Stem Cells. 2006; 24:145–150.
8
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
Tackling Irreproducibility
Changes in the Life Science Research Landscape
Over the past 30 years, life science research advances have benefited the life and health of millions of
people. For example, with treatment, HIV patients in the United States and Canada can expect to live
an almost a normal life span of over 70 years.13 Routinely recommended genetic testing for diseases
such as cystic fibrosis (CF) allows detection and estimation of disease risk prior to pregnancy, enabling
couples to make informed decisions, and some CF-affected individuals to live until adulthood.14 These
advances are the result of translating basic research findings into clinical treatments or laboratory tests.
And importantly, over 50% of basic research is performed in the academic setting.15
The current practices used to maintain research quality are deeply entwined with the fundamental tenets
of academic research culture and have successfully produced most biological and treatment advances
that we enjoy today. However, changes in the life science landscape, including rising complexity,
competition, economic challenges, and translational focus, increase the challenges associated with
maintaining life science research quality through traditional systems.
The general flow of the current academic life science research process is summarized in Figure 3.
Figure 3. Academic Life Science Research Process
Replication by other laboratories
Peer review
Generation of derivative results
Pl review
NIH ethics regulations
Synthesis
of Results
Pl review
Publication
Manuscript
Preparation
Project
Idea
Grant review process
Replication
Obtain
Funding
Technology
Transfer
Pl review
Data
Analysis
Journal
requirements
Journal
requirements
Statistician
review
Experimental
Design
Quality Checkpoints
Institutional
policies
Broadly used
Moderately used
Rarely used
Record
Data
Pl review
Experimental
Protocols
Pl review
Intralaboratory replication
Pl review
Perform
Experiment
Equipment
Calibration
Institutional
review
(animal
experiments)
Pl or lab manager review
Reagent
Validation
Peer review
Green circles indicate common steps in the life science research process. Adjacent color-coded text describes current/traditional quality checkpoints.
13. Hogg, R. Increases in Life Expectancy among Treated HIV-Positive Individuals in the United States and Canada, 2000–2007. IAS.
2013:TUPE 260.
14. American Congress of Obstetricians and Gynecologists (ACOG). Update on Carrier Screening for Cystic Fibrosis. 2011.
15. National Science Board. Science and Engineering Indicators 2012.
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
9
Tackling Irreproducibility
Currently, the maintenance of life science research quality relies on a set of key checkpoints (Figure 4).
Although the process is iterative, there are currently few standards surrounding quality management
in life science research.
Figure 4. Common Academic Life Science Research Process Checkpoints
Quality
Checkpoint
Description
Research Findings
• Grant proposals describe purpose of research, preliminary data,
high level methodology for future experiments, and resources needed
Grant
Review
Intralab PI
Review
• Proposals reviewed by a panel of scientific experts recruited by the
funder using funder-defined criteria that may incorporate adherence
to guidelines or standards
• Informal one-on-one and group laboratory meetings
with trainees and staff
• Internal laboratory policies (generally informal)
• Experts selected by journal determine if paper results are based
on sound scientific practices and conclusions stem appropriately
Peer
from
the data
Review of
Manuscript • Journal editors ensure paper adheres to journal policies that may
incorporate guidelines or standards
Colleague
Replication
• Published results can be replicated by other labs using
methodology available in the publication
• Only a fraction of all work is replicated
• Conclusions of prior studies used to generate new hypotheses
SelfCorrection
• As new hypotheses are tested, the prior conclusions are either
validated or refuted
Scientific Knowledge
Legend:
High-Quality Work
Low-Quality Work
The funnel represents the overall research process, with circles representing high-quality (yellow) or low-quality (blue) work. As the
work flows though the checkpoints described on the right, most of the low-quality research is filtered out.
Lack of broadly-accepted standards results in extensive
variability in the ways these quality checkpoints are
“I guide my students but also try to give them freedom
in experimental design. We meet informally every week
to discuss the data; it’s an iterative process.”
Academic Research Leader
implemented. For example, academic research
leaders and researchers report large differences
between laboratories as to which steps of the research
process are routinely monitored by the principal
investigator (PI). Only a few steps, such as manuscript
preparation, are carefully checked for quality.
10
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
Tackling Irreproducibility
Peer review of manuscripts prior to publication is a particularly powerful checkpoint that many
stakeholders believe fundamentally protects research integrity. However, journals exhibit significant
diversity in requirements for publication as well as
in the criteria that different journals and reviewers
“In my laboratory, we don’t let results go to publication
use to evaluate work.16 Currently, few broadly
implemented standards exist to help journals align
without someone who was not involved in the
around requirements for publication. Although some
experiments replicating them. We have caught many
journals may incorporate guidelines and standards
errors this way over the years and experiments that were
into publication decisions, many do not.
just not robust enough to withstand scrutiny. However,
Publications release and disseminate results into the
I don’t see this being done in most laboratories.”
broader scientific community where findings and
Academic Research Leader
interpretation face additional scrutiny. The feedback
from the life science community suggests that
researchers believe that when the journal peer review
process fails and erroneous data or interpretations get published, highly important or controversial
results will be replicated and either confirmed
or disproved by other laboratories within a
few years. If findings are reproducible, other
“If one journal doesn’t accept a paper, there is always
laboratories will use them as the basis for
a different journal that you can send it to. Different
additional research, resulting in further
publications and the growth of the field. If
journals look at different things. Some look at how
findings are not reproducible, the situation is
well the work was done, others want great work
more variable. When the original hypothesis is
that will also attract a lot of publicity and attention.
of tremendous significance, the irreproducibility
of results is usually published, although this
Some will publish anything.”
process can take several years. For results of less
Academic Research Leader
significance, the inability to replicate findings is
frequently not discovered nor reported.
In some cases, reporting guidelines for certain types of studies or laboratory procedures may be
developed by a professional society, and adherence to guidelines may be required by its journal as a
condition for publication. For example, Cytometry A
and Cytometry B, the journals of the International
“Discrepancies between laboratories can arise because
Society for Advancement of Cytometry (ISAC), require
adherence to the MiFlowCyt standard for minimum
they may use different cut-offs for compensation, a data
analysis technique frequently used in flow cytometry.
Because publications rarely required reporting this level of
detail, there was no way for the replicating laboratory to
determine what values were used initially, resulting in
irreproducibility. The MIFlowCyt standard prevents this.”
Academic Research Leader and Journal Editor
information required to report the experimental details
of flow cytometry experiments.17 However, this type of
standard is rare in life science research.
The current research landscape is shaped by multiple
changes, including novel technologies, increasing
specialization, and imperatives to make a faster impact
on patient outcomes. These changes are exerting
pressure on traditional paradigms for maintaining
research quality (Figure 5).
16. Bohannon, J. Science. 2013; 342(6154):60–65.
17. Cytometry Author Guidelines. [Internet] [cited 2013 Aug 18].
Available from http://onlinelibrary.wiley.com/journal/10.1002/(ISSN)1552-4930/homepage/ForAuthors.html.
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
11
Tackling Irreproducibility
Figure 5. Pressures on Traditional Academic Life Science Research Process Checkpoints
Traditional Life Science Research Quality Check Points
Research Findings
Increasing
Complexity and
Collaboration
Grant
Review
Growing
Competition
Intralab
PI Review
Peer
Review of
Manuscript
Colleague
Replication
Economic
Challenges
SelfCorrection
Greater
Translational
Focus
Scientific Knowledge
Legend:
High-Quality Work
Low-Quality Work
The funnel represents the overall research process, with circles representing high-quality (yellow) or low-quality (blue) work. As the
work flows though the checkpoints described on the right, most of the low-quality research is filtered out. As pressures represented by
the blue arrows are applied to the process, the filters become more permeable, allowing more low-quality work to pass through.
Complex and high-throughput technologies and
“It used to be that only flow cytometrists did
flow cytometry. Now everyone wants to have a flow
figure in their paper. The researchers rely on the
expertise of our core facility to guide them through
the process. They have to take it on faith that
the core facility is doing things correctly.”
Academic Research Leader
disciplines, such as genomics, proteomics, cellomics,
metabolomics, and others, generate increasingly large
amounts of biological data.18 For example, the Human
Protein Reference Database grew from 2,750 proteins
in 200319 to over 15,000 today.20 The complexity and
breadth of techniques leads to more specialization
and a greater reliance on core facilities21 among
researchers as the intricacies of each area present
greater challenges and require more focused expertise.
18. Vucic, E. et al. Genome Res. 2012; 22(2):188–195.
19. Peri, S. et al. Genome Res. 2003; 13(10):2363–2371.
20. Human Proteinpedia. [Internet] [cited 2013 Jul 26]. Available from www.humanproteinpedia.org.
21. Core facilities are centralized, usually institutional, laboratories dedicated to one or more complex techniques such as flow cytometry or
microarrays, have the equipment necessary to perform these techniques, and employ experts in relevant methods and technologies.
12
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
Tackling Irreproducibility
Academic research leaders note that they
are relying more on collaboration with
Figure 6. Academic Research Funding and Output
experts in complex technologies as a
110
towards team science is encouraged by
100
academic biomedical research in the
United States.22,23,24 For example, the
2010 NIH Collaboration and Field Science:
A Field Guide notes that “Innovations
and advances that were not possible
within one laboratory working in isolation
are now emerging from collaborations
and research teams that have harnessed
techniques, approaches, and perspectives
from multiple scientific disciplines and
therapeutic areas.”
At the same time, multimodal approaches
and reliance on team science can undermine
the individual investigator’s ability to
100%
90
80%
80
70
60%
60
50
40%
40
% NIH Grant Applications
the NIH, which funds the majority of
Number of PubMed Citations (10,000s) NIH Funding (USD B)
routine part of their work. The trend
30
20%
20
10
0%
0
2005
2006
2007
2008
PubMed Citations (10000s)
control the research process and make
2009
2010
2011
2012
NIH Research Grant Funding
NIH Successful Grant Applications (%)
peer review more challenging. With use
of novel, complex technologies to address
Sequester-driven and other budget cuts in 2013 will cause the NIH to
lose ~$1.7B and the NSF to decrease new research grants by 1,000.
a broader scope of biological and clinical
problems through multimodal and team
Data from PubMed.gov and NIH Research Portfolio Online Reporting Tool (RePORT).
approaches, these scientists may not have
expertise in all the methods being used to
generate data for a particular project or paper.
Detailed knowledge of experimental modalities and
common pitfalls would allow the PI to maintain
“I have several postdocs in my lab that trained in
different fields than my expertise. They perform
experiments that merge our areas together and help us
work at the intersection. I can try to help them if the
experiments are not going well, but ultimately
I just don’t have the same level of understanding
in these areas as I do in my field.”
Academic Research Leader
research quality by rapidly identifying trainee errors
in experimental design or interpretation. However,
this checkpoint is weakened when PIs must evaluate
methods and technologies with which they are less
familiar, including interpretation of data obtained
from collaborators and core facilities or even routine
use of kits. Similarly, journal reviewers selected for
their expertise with a particular aspect of a paper
might not be familiar with the intricacies of all the
technologies used in the research, making it more
challenging for reviewers to detect errors and
inconsistencies.
22. NSF. Higher Education Research and Development: Fiscal Year 2011 Detailed Statistical Tables. NSF 13-325. 2013 Jul.
23. National Institutes of Health 2012 budget. [Internet] [cited 2013 Jul 26]. Available from http://www.nih.gov/about/budget.htm.
24. Federation of American Societies for Experimental Biology (FASEB). Federal Funding for Biomedical and Related Life Sciences Research,
Fiscal Year 2014.
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
13
Tackling Irreproducibility
The academic research environment has become even more competitive. The percentage of
successful NIH grant applications is declining (Figure 6) as NIH funding has flattened.25 That figure is
likely to decrease further in response to 2013 budget cuts of approximately $1.6 billion.26 The NSF also
plans to decrease new research grants by 1,000.27 Meanwhile, academic research output (e.g., the
number of PubMed citations), is steadily increasing.
Academic research leaders and researchers report
“We like to approach the problem through a variety
of methods and test the concept in multiple systems,
but in the current environment this is becoming
more difficult. I would like to wait and try the
pressure from their institutions and funding agencies
to produce more publications. These scientists feel that
productivity expectations for advancing professionally
and obtaining grants are increasing, particularly because
professional advancement decisions frequently take
concept in cells, several different animal models, and
the quantity of published work into consideration.
human materials and then publish a thorough paper
The dynamic of flattened funding with increasing
that really synthesizes the concept. Now, this would
have to be three papers. If the third one doesn’t
support the first, the first paper is still out there.”
Academic Research Leader
output leads to fewer available resources for each
published paper. Research leaders believe that this
resource scarcity leads to the publication of shorter,
simpler papers that can be published more rapidly but
may not thoroughly explore all aspects of a particular
scientific problem, leaving material for subsequent
publications. Additionally, the pressure to produce work
faster might lead to less thorough experimental design than the PI may otherwise want. These factors
can make PI control of research quality less effective.
Life science research is also becoming more
translational in nature. Although basic science
“Ten years ago, my laboratory performed mainly
research has the goal of advancing knowledge
basic research. Now, we don’t have any projects
about biologic processes, translational research
without at least some translational focus.”
focuses on producing knowledge and products—
Academic Research Leader
such as novel drugs or laboratory tests—that can
lead to better diagnosis, treatment, prevention,
and cures of human diseases. A growing emphasis on producing clinical results can shorten the time
between basic research discoveries and use of the discoveries in patients during clinical trials. The
increase in importance of translatability is driven in part by NIH funding priorities28 as well as
pressures from institutions, disease foundations, and industry. These pressures are strongly felt by
stakeholders. For example, research leaders report that many PIs are incorporating more translatable
projects into their laboratories in response to the current NIH funding environment, leading some
laboratories with primarily basic research experience to engage in translational research.
Increasing translational focus is also manifested through a growing number of academic-industry
collaborations,29 such as the California Institute for Biomedical Research (Calibr),30 Johnson & Johnson’s
Innovation Centers,31 and Novartis and University of Pennsylvania Center for Advanced Cellular Therapy.32
Several academic organizations have also established institutes focused on drug discovery such as the
25.
26.
27.
28.
29.
30.
31.
32.
14
NIH Research Portfolio Online Reporting Tools (RePORT). [Cited 2013 Jul 8].
NIH. Fact Sheet: Impact of Sequestration on the National Institutes of Health. 2013 Jun 3.
NSF. Notice No. 133. 2013 Feb 27.
Waldman, M. Nature. 2010; 468:877.
Schachter, B. Nat Biotechnol. 2012; 30(10):944–952.
Timmerman, L. “Merck Returns to SD, Pouring $90M into New Schultz-Led Institute.” 2012 Mar 15.
Johnson and Johnson. “Johnson & Johnson Announces Plans to Establish Innovation Centers.” 2012 Sep 18.
Novartis, “Novartis and University of Pennsylvania Form Broad-Based R&D Alliance to Advance Novel T-Cell
Immunotherapies to Treat Cancer.” 2012 6 Aug.
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
Tackling Irreproducibility
Vanderbilt Center for Neuroscience Drug Discovery. These institutes combine academic expertise with
structured industrial practices pertaining to processes and reagents to improve translatability and
accelerate drug development.
The growing translational focus of research makes it more challenging for sufficient time to pass for
“self-correction” to take place before research is adopted by industry or used as a basis for clinical trials
or off-label drug use. The shortening of the time available for self-correction as a result of institutional
and patient pressures is highlighted by a recent controversy surrounding the reproducibility of results
obtained when treating Alzheimer’s disease mouse models with a Food and Drug Administration (FDA)approved skin lymphoma drug Targretin® (bexarotene).33 Initial results released in Science in March of
201234 indicated that the drug might have promise in treating Alzheimer’s disease because it increases
clearance of soluble AB amyloid, decreases the area of AB amyloid plaques, and reverses cognitive
deficits in mouse models. A New England Journal of Medicine editorial35 at the time stated “A single
report of this kind of preliminary evidence will require confirmation before Alzheimer’s disease
investigators even consider launching clinical trials in humans,” and cautioned against off-label36
use of the drug in Alzheimer’s disease patients.
Over the next year, multiple laboratories attempted to replicate this high-profile research.37-40 These
studies, released in Science in May 2013, had more equivocal results. The data supported some but
not all of the findings of the original work,41 dampening the enthusiasm many researchers felt about
the drug’s promise.
However, during the year between the publication of the original paper and subsequent publications,
substantial activity to enable drug use in patients took place. A phase II clinical trial to study the use
of bexarotene in Alzheimer’s patients was initiated in January of 2013.42 In April of 2013, Case Western
Reserve University, where the original research was performed, granted an exclusive license for the
treatment strategy to a spinoff company, ReXceptor. The company was co-founded by two of the original
study authors and secured $1.4 million in funding from multiple sources, including Alzheimer's Drug
Discovery Foundation (ADDF) and BrightFocus Foundation, to support a phase Ib trial in healthy
volunteers.43 Furthermore, patient and family pressure to prescribe the drug off-label was significant,44
resulting in at least some use of a drug that has side effects but no proven benefit.45
The Targretin example illustrates the growing pressures that make it hard for scientific research to
self-correct. Although replication of key data and discussions about interpretation are a routine part of
the scientific process and enable self-correction of findings to take place, the time to allow this process
to proceed before a drug is administered to patients has shortened. The example also demonstrates
that the changes in the life science research environment are beginning to influence ways that research
quality has been traditionally maintained. As stakeholders are affected by these trends, they are
beginning to focus attention on quality indicators, such as the reproducibility of research findings.
33. Previously owned by Eisai, currently owned by Valeant Pharmaceuticals. Valeant Pharmaceuticals. “Valeant Pharmaceuticals Acquires
U.S. Rights to Targretin® from Eisai Inc.” 2013 Feb 21.
34. Cramer, P. et al. Science. 2012; 335(6075):1503–1506.
35. Lowenthal, J. et al. N Engl J Med. 2012; 367(6):488–490.
36. Off-label use is prescription of a drug that is FDA-approved for one indication (e.g., skin lymphoma) to patients with another disease (e.g.,
Alzheimer’s). This practice is legal in the United States and frequently used in disease areas where few explicitly approved treatment options exist.
37. Fitz, N. et al. Science. 2013; 340(6135):924-c.
38. Price, A. et al. Science. 2013; 340(6135):924-d.
39. Tesseur, I. et al. Science. 2013; 340(6135):924-e.
40. Veeraraghavalu, K. et al. Science. 2013; 340(6135):924-f.
41. Shen, H. “Studies Cast Doubt on Cancer Drug as Alzheimer’s Treatment.” Nature News. 2013 May 23.
42. Bexarotene Amyloid Treatment for Alzheimer’s Disease (BEAT-AD) [Internet]. ClinicalTrials.gov [cited 2013 Jul 31]. Available from
http://www.clinicaltrials.gov/ct2/archive/NCT01782742/.
43. PRNNewswire. “Research Foundations Collaborate to Fund Phase 1 Study of Cancer Drug in Alzheimer's Disease Patients.” 2013 Apr 24.
44. Wang, S. “Alzheimer’s Families Clamor for Drug.” The Wall Street Journal. 2012 Feb 11.
45. Koebler, J. “Study Finds ‘Serious Problems’ with 2012 Alzheimer’s Breakthrough.” U.S. News and World Report. 2013 May 23.
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
15
Tackling Irreproducibility
Reproducibility of Research Findings
Increasing pressures on traditional systems for
maintaining research quality are leading to an
“If there is one retraction or irreproducible paper,
increased awareness of irreproducibility of research
findings among stakeholders. Reproducibility and
the field forgives you. If there are retractions or
reputation are key criteria that stakeholders use to
irreproducible papers all the time, people will not
judge the quality of scientific work (Figure 7).
trust the journal. It means there is something wrong
Reproducibility can be demonstrated directly when
with your review process.”
results are replicated externally or expected because
Journal Editor
of the intrinsic qualities of a paper.46
Number of Times Listed as Top 3* Way to Assess Research Quality
Figure 7. Stakeholder Assessment of Quality
N=40
90
80
Other
70
Impact on Field
60
Reputation
50
40
Externally
Reproducible
30
Intrinsic Aspects
of Paper
Good
Reputation
• Internal
consistency
• Pl
• Institution
• Journal
• Consistency
across
experimental
systems
• Appropriate
data analytic
techniques
20
10
• Strong
experimental
design
Intrinsic
Aspects
of Paper
Externally
Reproducible
Impact
on Field
• Importance
• Novelty
• Non-selective
presentation
of results
0
Responses to Open-ended Question
“How do you define life science quality?”
Stakeholders commonly use these connected features to assess the quality of life science research. Strong experimental design
includes validation of reagents; appropriate selection of controls and experimental systems; blinded experimentation; and robust
number of replicates. Bar graph shows number of times interviewed stakeholders noted a feature of life science research as a top
3 way to assess quality (N=40). *Some stakeholders reported only one or two top ways to assess research quality.
Reputation is also viewed as a prominent indicator of
“When you work in the field, you know what labs
are good and which ones have data that you
need to take with a grain of salt because you
will not be able to replicate it.”
Academic Research Leader
quality. This criterion can include the reputation of
the PI or laboratory that has produced the work, the
institution where the research was performed, or the
journal in which the work was published. Over time,
the reputations of laboratories, journals, and PIs
are mainly derived from producing work that is
reproducible and stands the test of self-correction.
46. Begley, C.G. Nature. 2013; 497(7450):433–434.
16
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
Tackling Irreproducibility
Extent of Irreproducibility
Irreproducibility is a pervasive, systemic problem experienced by most researchers across life science
settings. Discussion about the irreproducibility of many studies, particularly those in genetics, has
been ongoing since the early 2000s. This issue was highlighted in a seminal 2005 essay by John
Ioannidis, a professor at the Stanford School of
Medicine, titled “Why Most Published Research
“We’ve had to address issues with replicating
published work many times. Sometimes it’s a technical
issue, sometimes a reagent issue, sometimes it’s that
the technique was not being used appropriately.”
Findings Are False.”47 In that paper, Ioannidis
performed mathematical modeling and suggested
that “for most study designs and settings, it is more
likely for a research claim to be false than true.” The
article postulated unconscious bias as a key cause.
Academic Research Leader
Several years later, two highly publicized experimental
studies showed a significant lack of reproducibility of
academic research findings in the industrial setting.
In a 2011 Nature Reviews Drug Discovery article,48 Prinz and colleagues from Bayer Healthcare reported
that in 67 projects over 4 years, published data was reproducible only 21% to 32% of the time, depending
on the definition of reproducibility (all data vs. main dataset) used. A subsequent report coauthored
by Begley and Ellis49 reported a similar experience at Amgen. Out of 53 “landmark” papers that the
hematology and oncology department attempted
to replicate over 10 years, only 11% were
reproducible. Industry stakeholders interviewed
“Our lab is focused on translating basic research
for this study also report extensive experiences
discoveries and we are only able to replicate 1 in 5
with lack of reproducibility.
concepts that have passed through our lab.”
Irreproducibility is not limited to the industrial
Government Research Leader
setting. A recent MD Anderson survey showed
50
that over 65% of senior academic faculty has had
the experience of being unable to reproduce a finding from a published paper. Most tried to contact
the original authors, but over 60% of the time they received an indifferent or negative response or no
response at all.
Among the stakeholders interviewed for this work, experiences with irreproducibility were reported by
most academic research leaders, government research leaders, industry leaders, and journal editors.
Over 80% of interviewed academic research leaders have had some experience with irreproducibility,
either when trying to reproduce the work of others or
having their own work reproduced (Figure 7).
“Because we carefully select the labs we work
with, in our hands, irreproducibility is lower
than the Begley [2012] study, maybe 30%, but
this is still very significant.”
Industry Research Executive
Causes of Irreproducibility
Irreproducibility is the result of differences in
performance of a particular experiment between
laboratories. These differences can occur when
one laboratory performs the experiment in a more
optimal way, or even when both laboratories perform
47.
48.
49.
50.
Ioannidis, J.P. PLOS Med. 2005; 2(8):e124.
Prinz, F. et al. Nat Rev Drug Discov. 2011; 10 (9):712.
Begley, C.G. and Ellis, L.M. Nature. 2012; 483(7391):531–533.
Mobley, A. et al. PLOS ONE. 2013; 8(5):e63221.
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
17
Tackling Irreproducibility
the experiment optimally but with inherent differences in methods or reagents. Multiple systemic
causes contribute to variability of both practices to ensure quality and performance of the specific
experiment in question, including absence of formal or consistent research quality systems, variable
education of staff, and differences in journal review and reporting policies (Figure 8). Many of these
causes of irreproducibility can ultimately be traced to an underlying lack of a standards framework.
Figure 8. Causes of Irreproducibility
Experimental
Irreproducibility
Academic Experience
Resource
Limitations
Reagent Variability/Lack of Validation
Experimental Bias
Increasing
Complexity
Differences in Analytic Techniques
Differences in Statistical Methods
No Quality
Systems
Unintentional Selective Reporting
Lack of Appropriate Controls
Variable
Education
Lack of Equipment Calibration
Variations in Interpretation
Scientific Misconduct
Variable
Journal
Policies
Legend:
➡
Pervasive
➡
Moderate
➡
➡
➡
➡
➡
➡
➡
➡
➡
➡
➡
% Senior Faculty
Incomplete Protocol Reporting
75%
50%
25%
0%
GBSI
Mobley
No Irreproducibility Experience
Experienced Irreproducibility
Industry Experience
100%
% Papers Examined
Lack of Standards
Pressure
to Publish
100%
75%
50%
25%
Rare
0%
Prinz
Begley
Results Confirmed
Results Not Confirmed
Academic experience refers to personal experience of academic senior faculty with irreproducibility. Industry experience refers
to percentage of papers studies reported being able to reproduce. Graphs generated based on responses from current study
interviews (labeled GBSI, N=16) and Mobley PLOS ONE (2013) (N=148). In industry experience, papers were considered
reproducible when literature data was in line with in-house data. Graphs generated based on Prinz et al. in Nature Reviews (2011)
and Begley and Ellis in Nature (2012). Causes of irreproducibility defined and ranked based on current interviews, Loscalzo in
Circulation (2012) and Begley in Nature (2013).
18
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
Tackling Irreproducibility
These systemic causes translate into a variety of immediate causes of irreproducibility, including the following:
• Incomplete protocol reporting; for example, when authors leave out certain details that they do not
believe to be crucial to the performance of an experiment or that were never recorded as a formal
part of the laboratory’s procedure for performing the experiment.
• Reagent variability, such as can occur when two laboratories use what they perceive to be the same
reagent, but are in fact two biologically-distinct forms.51 For example, significant differences can exist
between antibodies52 to the same antigen from different manufacturers or even from different lots
produced by the same manufacturer.
• Lack of reagent validation, such as using unauthenticated cell lines (see “Life Science Standards at
Work: Cell Line Authentication” box, page 28).
• Experimental bias,53 such as unconsciously and unintentionally measuring the size of a tumor in an
animal from the group an investigator knows to be treated as smaller than in the untreated group.
Unblinded experiments where an investigator knows which cell culture dish or group of animals has
received treatment are most prone to this type of bias.54
• Differences in analytic techniques; for example, using different assays (tests), imaging, or
measurement techniques to evaluate experimental results.
• Differences in statistical methods, including use of different mathematical approaches to analyze
data or use of statistical approaches that might not be optimal for the particular data type.
• Unintentional selective reporting, such as assuming that the experiment worked when results are
positive but did not work when results are negative.
• Lack of appropriate experimental controls, such as excluding a true negative control to understand
the specificity of an antibody or a primer for detecting the antigen or sequence of interest.
• Lack of equipment calibration leading to differences in measurements between laboratories.
• Variations in interpretation, such as interpretation of a very weak band on a gel as positive by one
laboratory and negative by another.
• Scientific misconduct.
Scientific misconduct has recently received attention in the scientific literature,55 the National Academy of
Sciences,56 the 2013 International Congress on Peer Review and Biomedical Publication (Peer Review
Congress),57 as well as in the popular press.58,59 Particular attention has been brought to the increasing
number of peer-reviewed paper retractions and the fact that the majority of retractions result from scientific
misconduct.60 Although there is limited overlap, irreproducibility is not equivalent to paper withdrawals or
scientific misconduct. Only 0.01% of papers are retracted, whereas irreproducibility is a pervasive, complex
issue that is far more prevalent than paper withdrawals and associated misconduct. Furthermore, most
irreproducible papers are never withdrawn. Although scientific misconduct is certainly an issue of grave
concern, most interviewed stakeholders view it as an uncommon cause of irreproducibility in the life sciences.
51. Vasilevsky, N. et al. PeerJ. 2013; 1:e148.
52. Antibodies are proteins produced by immune system cells that have a specific affinity to parts of other biologic molecules (antigens). Antibodies can be
produced commercially, labeled with dyes, and used to detect specific biologic molecules (e.g., a particular protein) in scientific experiments.
53. Loscalzo, J. Circulation. 2012; 125(10):1211–1214.
54. Vesterinen, H.M. et al. Mult Scler. 2010; 16(9):1044–1055.
55. Sliwa, J. “Has Modern Science Become Dysfunctional.” American Society of Microbiology, 2012 Mar 27.
56. Fang, F.C. Proc Natl Acad Sci USA. 2012; 109(42):17028–17033.
57. Peer Review Congress. [Internet] [cited 2013 Sep 15]. Available from http://www.peerreviewcongress.org/2013/Plenary-Session-Abstracts-9-8.pdf.
58. Zimmer, C. “Misconduct Widespread in Retracted Science Papers, Study Finds.” New York Times. 2012 Oct 1.
59. Johnson, C. “Harvard Investigation of Stem Cell Scientific Misconduct Provides Insight into Secretive Process.” Boston Globe. 2013 Apr 8.
60. Steen, R. J Med Ethics. 2011; 37:249–253.
61. Corbyn, Z. “Misconduct Is the Main Cause of Life-Sciences Retractions.” Nature News. 2012 Oct 1.
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
19
Negative Effects of Irreproducibility:
The Case for Standards
Irreproducibility Reflects the Need for Life Science Standards
In its essence, irreproducibility is the result of frequently unrecognized differences in the performance
of a particular experiment between laboratories or researchers. Solutions to this pervasive problem can
focus on two areas: (1) increasing recognition of intentional differences between experiments conducted
by different laboratories (e.g., better data reporting practices), and (2) reducing unintentional differences
between laboratories. Standards can effectively reduce differences in practices by aligning the community
around consensus-based methods, and are therefore an effective solution for irreproducibility. Multiple
laboratory-based fields, such as clinical diagnostics and blood banking, have been able to successfully
improve reproducibility and quality through standards. For example, standardization of HbA1c testing in
diabetes significantly decreased variability of results between laboratories62 and the AABB (previously
known as the American Association of Blood Banks) standard for bacterial testing of platelets resulted in a
70% decrease in transfusion-related sepsis63 (please see Appendix I for further details of both examples).
Because of the significant consequences of irreproducibility, all stakeholders report the need for more
life science standards (Figure 9).
Figure 9. Need for Life Science Standards
Need for Standards to:
Improve
Consistency of
Reagents,
Substrates, and
Analytic Methods
Improve
Consistency of
Daily Laboratory
Processes
Improve
Experimental
Design and
Analysis
Improve
Sharing of Data
Stakeholder Group
Academic
Researchers
Industry
Government
Agencies
Non-Profit
Funders
Journals and
Professional
Societies
Examples of
Possible Standards
Antibody specificity,
endotoxin positivity,
methods for analyte
measurement
Legend:
Laboratory data
recording,
maintenance of SOPs,
quality systems
Low Unmet Need
Better procedure
Minimum replicates,
reporting, compatibility
data analytic
of dataset formats
techniques, quality
and labeling
systems
Some Unmet Need
High Unmet Need
Rankings are based on qualitative interviewee responses (N=50). Orange circles indicate that the majority of interviewed
stakeholders reported no or very low need for additional biological standards. Light green circles indicate that the majority of
interviewed stakeholders felt there is some need for additional standards. Dark blue circles indicate that the majority of interviewed
stakeholders reported a prominent need for additional life science standards.
62. Little, R. Clin Chem. 2011; 57(2):205–214.
63. AABB. “Public Conference – Secondary Bacterial Screening of Platelet Components, 7/17/12.”
20
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
Negative Effects of Irreproducibility:
The Case for Standards
Effects of Irreproducibility
Research practice variability and quality challenges signified by irreproducibility can lead to inefficient
use of time and resources across the life science research space (Figure 10).
Figure 10. Effects of Irreproducibility
Irreproducibility
Academic Labs
& Institutions
• Increased time
and effort to
generate
results and
publications
• Reputation
impact
Industry &
Investors
• Target
validation and
development
failures
• Unpredictable
outcomes
• Suboptimal use
of funds
• Failed
investment
Funders
• Suboptimal
use of financial
resources
• Delays in
research
progress
• Reputation
impact
• Worse public
perception
Journals &
Professional
Societies
• Delays in
research
progress
• Reputation
impact
• Worse public
perception
Inefficient Use of Resources and Time
Impact on Reputation
Less Favorable Public Opinion of Research
Irreproducible Findings Waste Academic and Public Health Resources
Recent controversy surrounding the role of a retrovirus called xenotropic murine leukaemia virusrelated virus (XMRV) in causing prostate cancer and chronic fatigue syndrome (CFS) is an example of
how extensive resources may be needed to show that irreproducible results are invalid (see call-out
box on next page).
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
21
Negative Effects of Irreproducibility:
The Case for Standards
Irreproducible “Discovery” of XMRV
XMRV was initially identified in human prostate cancer samples in 2006.64 In 2009, another study
showed that XMRV infection is associated with higher grade of prostate cancer.65 Within a few months,
a widely publicized study published in Science demonstrated that XMRV could be identified in almost
70% of patients with chronic fatigue syndrome (CFS), but in less than 5% of healthy volunteers.66
Due to the presence of a virus that might be a cause of disease and potentially transmissible through
blood transfusion, the public health community began to voice serious concern about the need to protect
the blood supply from XMRV. The AABB and U.S. government (through a collaboration between the NIH,
FDA, CDC, and several academic groups)67 established groups to evaluate the potential risk to the blood
supply. Multiple blood banks around the world began to refuse blood donations from patients with CFS.68
Given the potential consequences to the blood supply, multiple attempts to replicate the findings were
made in Europe and the United States. Although one study supported the findings in CFS patients,69
multiple other studies did not find the virus either in CFS patients, prostate cancer patients, or in healthy
controls.70 In 2010 and 2011, contamination of samples and laboratory reagents (such as commercial
RT-PCR71 reagents) with XMRV derived from experimental mice or mouse cell lines was proposed as the
cause of the initial positive results. Many additional studies, including two large multi-center efforts72.73
and studies examining thousands of donor blood units performed between 2010 and 2012, did not identify
XMRV in human samples. It is now believed that the initial research findings were an erroneous result
of contamination that was not detected due to lack of appropriate controls.70,72
Over the course of almost four years, and through the collective efforts of multiple laboratories,
government agencies, blood transfusion services, and organizations around the world, the initial errors
were corrected. However, substantial costs were associated with these efforts. Although the total research
expenditure of dozens of laboratories and public health efforts by blood transfusion services around the
world have not been calculated, the cost of one of the multi-center studies alone was approximately $2.3
million,72 suggesting that the overall costs were very significant.
The potential public health implications make the XMRV example particularly dramatic, but even when
the consequences are less compelling than blood supply safety, significant resources can be invested
in trying to act on or replicate work that is inherently irreproducible. Academic laboratories can expend
significant time and funds attempting unsuccessfully to reproduce work that they want to use as the basis
for future experiments. This expense creates challenges for the laboratories and affects government
and non-profit research funders of the original or subsequent research. Moreover, limited financial
resources may be wasted on work that cannot lead to further advances.
64.
65.
66.
67.
68.
69.
70.
71.
72.
73.
22
Urisman, A. et al. PLOS Pathog. 2006; 2(3):e25.
Schlaberg, R. et al. Proc Natl Acad Sci USA 2009; 106:16351–16356.
Lombardi, V. et al. Science. 2009; 326:585–589.
Simmons, G. et al. Transfusion. 2011; 51(3):643–653.
Kakisi, O.K. et al. Transfusion Med. 2013; 23(3):142–151.
Lo, S.C. et al. Proc Natl Acad Sci USA. 2010; 107(36):15874–15879.
Groom, H.C. and Bishop, K.N. J Gen Virol. 2012; 93(Pt 5):915–924.
Reverse transcription polymerase chain reaction.
Simmons, G. et al. Science. 2011; 334:814–817.
Alter, H.J. et al. Mol Biol. 2012; 3:266–312.
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
Negative Effects of Irreproducibility:
The Case for Standards
A researcher recalls that as a graduate student he read a
paper in a prestigious journal showing that a particular
mutation could lead to protein interactions that cause
carcinogenesis in several cell lines. The researcher
generated several exciting hypotheses based on this
Irreproducibility Complicates
Academic-Industrial Relationships
The inability to reproduce results also presents a
significant challenge for industry. Biopharmaceutical
companies might waste resources in attempts to
optimize published academic procedures that are not
work. However, his principal investigator insisted that
replicable and therefore not translatable, as demonstrated
because the findings in the paper were novel, the
in the Amgen and Bayer experiences described above.
researcher should replicate several of the key
experiments before proceeding. The researcher
Irreproducibility contributes to the unpredictability
of development outcomes and failures in target
validation and development. While most companies
attempted to replicate the experiments for several
months. After encountering multiple failures, he
contacted the authors of the original paper. The authors
were not happy about his inquiries, but agreed to help
him. For several more months the researcher attempted
to replicate the experiments, finally traveling to the
original lab. When he was there, the authors could not
replicate their published experiments at their own
are highly protective of information regarding costs
associated with these failures, one example is
provided by Bruce Booth in his blog, LifeSciVC.com:
“The company spent $5 million or so trying to
validate a platform that didn’t exist. When they tried
to directly repeat the academic founder’s data, it
never worked.… Sadly this ‘failure to repeat’ happens
more often than we’d like to believe. It has happened
to us at Atlas several times in the past decade.”74
laboratory. The precise cause of irreproducibility was
never pinpointed, but the researcher believes it may
“Academic collaboration is the main way to reach key
have been due to over-passaging of the cell lines used
opinion leaders, particularly in areas where we don’t have
by the original laboratory. Investing time in trying to
internal expertise. Their experience and support is
replicate these results delayed the researcher’s
essential to our drug development efforts.”
graduation by six months and slowed down his lab’s
Biopharma R&D Executive
progress on several other projects. He states: “It would
have been wiser to quit earlier. But when the results are
profound, you want them to be true and you want to
make the next leap. Once you start trying to replicate
something, it is difficult to say ‘This is just not valid.
I will give up.’ You keep thinking that maybe it is
just something you are doing wrong.”
Academic Researcher
Irreproducibility can significantly complicate
academic-industry relationships at a time when both
sides are finding these collaborations increasingly
desirable.29 Government funding pressures are
increasing the importance and desirability of industry
collaborations for both institutions and individual
laboratories. Academic research leaders report that in
an environment of government funding challenges,
industry collaboration can be a highly valuable source
of funds and add stability to an otherwise volatile
“We spent several months trying to reproduce
findings in an animal model from another lab but
couldn’t. That time and money was just lost.”
Academic Researcher
funding environment. Industry is also steadily seeking
academic collaborations. Whereas early research and
development investment in many companies has
flattened, the number of commercial Investigational
New Drug (IND) applications has declined (Figure 11),
suggesting decreased R&D efficiency.
74. Booth, B. Academic Bias & Biotech Failures. LifeSciVC.com. 2011 Mar.
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
23
Negative Effects of Irreproducibility:
The Case for Standards
Figure 11. Industry Early R&D Spend and Output
To combat these trends, biopharmaceutical
companies are increasingly focused
$1.4
1,000
2005–2011 FL to I in industry
on improving the translatability of
$1.2
of otherwise unobtainable expertise and
biological models for biopharma, resulting
in a steady stream of collaborative activity.
Collaboration with academic laboratories
allows biopharmaceutical companies to
access leaders in particular scientific
disciplines and other experts that might
not be available internally. Academic
800
$1.0
$0.8
600
$0.6
400
$0.4
$0.2
Commercial IND Applications
opportunities. Academia provides a source
Representative Discovery and
Preclinical Research Spend (USD B)
discoveries and pursuing commercializable
200
$0.0
laboratories can also provide valuable
resources, such as cell and animal models
2005
approaches to quality can be challenging.
Generally, while academic laboratories are
2007
2008
2009
2010
2011
Commercial IND Applications
industry relationship is not without
between the academic and industrial
2006
Representative Discovery and Preclinical Research Spend
Although highly beneficial, the academicimpediments; mediating differences
0
$0.0
and active compounds.
Representative R&D spend based on top four pharmaceutical (J&J, Pfizer,
Novartis, Roche) and biotechnology (Novo Nordisk, Amgen, Gilead, Biogen)
companies by 2012 revenue estimated based on company 10-K R&D spend
and PhRMA annual member surveys, PhRMA Profile Biopharmaceutical
Research Industry 2007–2013. Commercial IND application numbers based on
Parexel 2012/2013.
focused more on obtaining publishable
results, industrial laboratories are more
concerned with discovery that occurs
Figure 12. Formal Quality Systems
within the bounds of replicable processes
100%
and can ultimately be transferred into the
Have Formal
Quality System
clinical setting. Whereas maintaining quality
in the academic setting is primarily an
informal process, many biopharmaceutical
75%
of ensuring quality in the basic research
and discovery space. These practices range
from basic document control to strict
process control systems resembling Good
% Respondents
companies have formalized some aspects
50%
No Formal
Quality System
Have Formal
Quality System
Laboratory Practices that the FDA mandates
in later stages of the drug development
25%
process.75 These industry practices can
include defined experimental replication
procedures or staff training requirements
to perform daily research functions. Many
academic laboratories and institutions,
however, do not have such formal quality
systems76 (Figure 12).
No Formal
Quality System
0%
Academia
Industry
Interview participants (N=30) were asked about whether their laboratory,
institution, department, or company had a formalized quality system
pertaining to basic and discovery research. “Formal” refers to presence of
policies outlined in writing that were in some way enforced.
75. Code of Federal Regulations, Title 21, Part 58.
76. A quality system is a formalized way to ensure that practices that lead to better outcomes are consistently
performed. ASQ Glossary. [Internet] [cited 2013 Jul 1]. Available from http://asq.org/glossary/.
24
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
Negative Effects of Irreproducibility:
The Case for Standards
Collaborations can rapidly break down when
academic results are not reproducible in
“If we had industry-wide standards for laboratory
industrial laboratories because, no matter how
processes or ensuring that everyone performing
promising academic published results might
research is actually qualified to perform the research
be, irreproducible results cannot be developed
they are doing, we could say to laboratories or
into commercializable products. Industry
stakeholders report that differences in quality
institutions ‘You have to follow these guidelines to
assurance practices, the quality of reagents,
work with us.’”
and in the degree of attention paid to the
Biopharma Industry Executive
reproducibility of processes can present
obstacles to translating academic laboratory
results into drug products.
Irreproducibility Damages Reputations
Irreproducible results can significantly damage the reputations of laboratories, institutions, journals,
and funders associated with them. Laboratories whose work cannot be reproduced face reputational
concerns that can adversely affect their ability to publish or obtain funding in the future.
Names of institutions where the irreproducible
research was produced can be connected
One academic research leader described an example of
irreproducible work that was performed in his
laboratory. The irreproducibility was due to a weak
signal on a gel in a published experiment that was
interpreted as positive in his laboratory and as negative
by others. Within a year, a competitive lab published a
paper with a better protocol showing a strong signal,
indefinitely to irreproducibility. For example,
a 2013 review article about XMRV59 describes
the investigators of the original 2006 study as
“researchers at the Cleveland Clinic and the
University of California,” as opposed to referring
to the PIs by name. Journals likewise face
negative effects on their reputations if a
significant amount of their published work is
shown to be irreproducible. As concerns about
and took credit for the discovery. They even included
a figure showing that the original results
were irreproducible. The academic research leader
suffered some embarrassment within his
community and believes that his laboratory lost
irreproducibility gather popular attention, being
associated with a publicized incident might
worsen public perception of the funding
organization or agency and the overall public
opinion of life science research.
some credibility as a result of this incident.
“I know the results were right, but now I insist that
everything we publish is reproduced by someone
in the lab who didn’t work on the project before
we submit it for publication.”
Academic Research Leader
“If we find out three years later that some of the
work we are funding was predicated on false or
inaccurate data, that’s a terrifying, expensive, and
dangerous problem.”
Disease Foundation Executive
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
25
Life Science Research Standards:
Adding Value and Accelerating Progress
Further Applications of Standards to Life Science Research
Standards can benefit life science research quality and reproducibility by decreasing unrecognized
variance between laboratories, and can serve as an effective tool for disseminating and encouraging
consensus-based research practices. As illustrated in Figure 13, it is possible to envision standards
pertaining to multiple aspects of life science research.
Figure 13. Standards Applications in Life Science Research
• Establishment of laboratory
or institutional systems that
ensure research quality
• Standardized
routine laboratory
processes such
as equipment
calibration
• Reference reagents
• Reagent validation
Quality
Systems
Reagents
Laboratory
Maintenance
Assays
• Consensus
assays
• Consensus
cut-off values
• Reference
reagents for
calibration
Life Science
Standards
• Consensus
reporting
frameworks for
materials, methods,
and procedures
• Consensus systems
for reporting
experimental data
and results
Legend:
Reporting
Protocols
Data
Analysis
Good
Experimental
Practices
• Consensus best
practices for data
analytic techniques
• Consensus on
best practices
for common
protocols
• Consensus best practices for
general concepts in experimental
design and performance
Materials & Document Standards
Primarily Document Standards
Standards can have broad applications in life science research. Blue indicates areas where both materials and document
standards may be applicable; green indicates areas where primarily document standards are applicable.
Standards related to reagents, assays, and laboratory maintenance increase quality and reproducibility
by decreasing unintentional (unrecognized) variation between laboratories. Standards related to
reagents can include both reference reagents and document standards that describe reagent validation
techniques. It is not uncommon for laboratories to use what they believe, based on name or vendor
description, to be the same antibody or cell line as another laboratory, only to find out later that the
particular lot or batch of the reagent had different biological qualities that lead to variability in results.
26
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
Life Science Research Standards:
Adding Value and Accelerating Progress
Materials standards for certain research reagents (e.g., antibodies, cell lines, animal models) can be used
by laboratories and vendors to decrease these unexpected differences. Document standards pertaining
to reagent validation can outline consensus-based
ways of ensuring that a particular reagent performs
“There is a lot of variability in reagents such as antibodies
as expected and, likewise, decrease unanticipated
variations that lead to irreproducibility.
between manufacturers and even between lots. It can take
Assays and measurements present another area
a significant amount of time to validate a new antibody
and sometimes we can never get it to work. Standards to
reduce this would be very helpful and save time.”
Academic Research Leader
where unintended variation can be reduced through
biological standards. In many life science research
areas, there is little consensus about which assays
researchers should use. There might not be reagents
available to calibrate the assays in order to ensure
that results obtained in one laboratory are
comparable to another. Furthermore, there might
be variability in cut-offs for considering results positive or negative. Document standards can enable
consensus around assays and measurements. For
example, a standard can represent the scientific
community consensus on which assay will be
used to measure activity of a particular bacterial
toxin and on the cut-offs for positive and negative
“There is a need to standardize assays used in
[Clostridium] difficile research. Papers use different
results. Additionally, the development of
techniques for measurement, and this makes it difficult
materials standards for assay calibration can
to compare and synthesize results across different
particularly benefit fields that use novel and
publications. I believe a consensus method would
emerging technologies, where such standards
are frequently not available.
increase the pace of research in this area.”
Academic Researcher
Standards surrounding laboratory maintenance
can define expectations for certain routine
“housekeeping” practices. These practices can include the accurate documentation of laboratory
protocols and procedures, data recording (e.g., maintenance of laboratory notebooks, deciding
how long original data is kept), equipment calibration (e.g., annually ensuring that all pipettes in a
laboratory are accurate), checking for reagent expiration, and not using expired reagents. Standards
can also outline good practices for common laboratory activities, such as cell or microorganism
culture and cell line or microorganism authentication.
A limited number of standards pertaining to reagents, assays, and laboratory maintenance are already
available. Some materials standards, primarily microorganisms, are available to researchers through
the World Health Organization (WHO), NIH, and private bioresource centers. Document standards are
gaining increasing adoption. One example is the ATCC cell line authentication standard that describes
a consensus procedure for unambiguous authentication and identification of human cell lines using
a specific type of analysis (short tandem repeat [STR] profiling).77 (See call-out box on next page.)
However, many opportunities for achieving community consensus and alignment remain and can
serve to reduce unintentional variability and improve reproducibility between laboratories.
77. ANSI/ATCC ASN-0002-2011. 2012. Authentication of Human Cell Lines: Standardization of STR Profiling.
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
27
Life Science Research Standards:
Adding Value and Accelerating Progress
Life Science Standards at Work: Cell Line Authentication
Cell lines are cells either from cancer or normal tissues that can proliferate under laboratory conditions.
Initial experiments on a particular tumor type (e.g., breast cancer) are usually performed using these
cells. A large number of cells lines representing a variety of tumor types exist. Laboratories depend on
visual examination of the cells and vial labels to ensure that they are performing each experiment on
the correct cell type. However, even for an expert, it can be difficult or impossible to determine with
certainty from which type of tumor a particular cell line originates.78 Because cells are repeatedly
grown, frozen, and stored by laboratories, cross-contamination and errors can occur, resulting in
experiments being performed on an incorrect cell type.
Examples of using incorrect cell lines in research have been reported for a variety of tumor types. A
database maintained by the International Cell Line Authentication Committee (ICLAC) lists over 350 cell
lines that have been misidentified or cross-contaminated and publications based on these cell lines.79
Although many of these have been contaminated by the HeLa cell line, the problem is even more pervasive.
For example, several common esophageal cell lines that have served as a basis for over 100 publications
and several clinical trials have been shown to be from other parts of the body (e.g., lung and colon).80 In
2011, a novel gastric MALT lymphoma cell line was described. It was considered groundbreaking because no
cell lines were available to model this disease. Recently, however, this cell line was shown to actually be
a misidentified, well-known cell line of a different lymphoma type.81
Misidentification of cell lines can be prevented by cell line authentication. Authentication is based on
determining the genetic signature of a particular cell line and comparing it to established databases to
ensure that the cell line used by a laboratory matches the expected signature.82 In 2011 and 2012, an
international group of scientists from academia, regulatory agencies, major cell repositories, government
agencies, and industry collaborated to develop a standard that describes optimal cell line authentication
practices, ANSI/ATCC ASN-0002-2011: Authentication of Human Cell Lines: Standardization of STR
Profiling.83 More recently, multiple journals including Nature,84 several American Association for
Cancer Research (AACR) journals,85 and the International Journal of Cancer86 now require or strongly
recommend cell line authentication.
In addition to decreasing inter-laboratory variation in reagents, assays, and daily laboratory operations,
standards can also help researchers reach voluntary consensus on general scientific practices and
disseminate these optimal practices. For example, standards can outline general good experimental
practices, such as a suggested number of replicates, blinded experimentation, appropriate selection of
controls, or selection of experimental systems; standards can also serve as a reminder and educational
tool about these practices. Standards can also be a vehicle for defining optimal protocols for common
78.
79.
80.
81.
82.
83.
84.
85.
Capes-Davis, A. et al. Int J Cancer. 2013; 132(11):2510–2519.
Capes-Davis, A. et al. Int J Cancer. 2010; 127(1):1–8.
Boonstra, J. et al. J Natl Cancer Inst. 2010; 102(4):271–274.
Capes-Davis, A. Genes Chromosomes Cancer. 2013; 52(10):986–988.
ANSI/ATCC ASN-0002-2011. 2012. Authentication of Human Cell Lines: Standardization of STR Profiling.
Kerrigan, E. “Working Together to Eradicate Cell Line Misidentification and Contamination: New Practices and Policy.” DDNews. 2012 Sep.
Nature Editorial Board. Nature. 2013; 496:398.
AACR Journals: Instructions for Authors. [Internet] [cited 2013 Aug 5]. Available from
http://cancerres.aacrjournals.org/site/misc/ifora.xhtml#celllineuse.
86. International Journal of Cancer Author Guidelines. [Internet] [cited 2013 Aug 5]. Available from
http://onlinelibrary.wiley.com/journal/10.1002/(ISSN)1097-0215/homepage/ForAuthors.html.
28
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
Life Science Research Standards:
Adding Value and Accelerating Progress
experimental procedures. Additionally, standards can apply to data analysis by outlining consensus
practices for the use of particular statistical methods, handling of outliers, and even the need for
biostatistician involvement.
Standards pertaining to submission of results to peer-reviewed journals (e.g., reporting guidelines)
or to separate databases can facilitate consistency and completeness of reporting for experimental
methods and results. Although some standards such as Minimal Information About a Microarray
Experiment (MIAME)87 already exist in this area, there is tremendous variance between journals and
even between reviewers in the same journal about the level of detail reported for many types of
experiments.88 Standards could help journals align around the types of data that are necessary to
describe the experimental conditions, analysis, and results.
Finally, standards can serve to establish systems that promote and ensure consistent quality and
the reproducibility of research results. For example, standards can outline quality systems for
performing life science research. Standards pertaining to quality systems (such as the ISO
[International Organization for Standardization] 9000 standards) generally do not explicitly state
what constitutes a high-quality outcome (e.g., that every experiment submitted for publication has
an appropriate control). Instead, these standards establish institutional frameworks for checking
quality consistently based on criteria that may be defined either by other standards or by each
institution or laboratory. For instance, such a standard may state that an institution or a laboratory
must have a documented internal process for ensuring that all experiments submitted for publication
have appropriate controls.
Quality system standards have been successfully and broadly adopted in other industries. For
example, over 1 million organizations adhere to quality standards and are ISO-9000 certified,89 and
over 7,000 laboratories in the United States and internationally adhere to the College of American
Pathology (CAP) standards.90 There has also been some progress toward developing these types of
standards for life science research. In particular, the American Society for Quality (ASQ) produced a
technical report, ASQ TR1-2012: Best Quality Practices for Biomedical Research in Drug Development,
in 2012. That document outlines a quality system aimed at non-regulated biomedical research, and
could serve as a foundation for standards in this area.
87. MIAME is a standard for reporting of microarray data developed in 2001. Please see Appendix I for additional information.
88. The variation in implementation and effectiveness reporting guidelines has been particularly well-documented in literature pertaining
to clinical trials. See Hirst, A. and Altman, D.G. PLOS ONE. 2012; 7(4):e35621. These findings are also summarized in Tugwell, P.A. et al.
J Clin Epidemiol. 2012; 65(3):231–233.
89. ASQ. “25 Years of ISO 9000.” [Internet] [cited 2013 Sep 16]. Available from http://asq.org/blog/2012/03/25-years-of-iso-9000/.
90. CAP. ”About the CAP: Laboratory Accreditation Program.” [Internet] [cited 2013 Sep 16]. Available from
http://www.cap.org/apps/cap.portal?_nfpb=true&_pageLabel=accreditation.
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
29
Life Science Research Standards:
Adding Value and Accelerating Progress
Systemic Benefits of Standards
Life science research standards can be a unifying driver of quality improvement efforts and have the
potential to benefit all stakeholders (Figure 14).
Figure 14. Benefits of Standards
Academic Labs & Institutions
Industry & Investors
• Ensure best practices incorporated into
research process
• Reduce target validation and
development failures
• Produce more reproducible results
• Improve translatability and commercialization
of discoveries
• Decrease misinformation
• Enhance collaboration with academia
• Protect reputation
• Enhance collaboration with
industry
Government & Non-Profit Funders
• Facilitate grant review process
• Improve return on investment
• Optimize use of funds
Standards
=
Reproducibility
Journals & Professional Societies
• Facilitate peer review process
• Optimize use of financial resources
• Protect reputation
• Protect reputation
• Decrease misinformation
• Improve public perception of life
science research
• Improve public perception of life
science research
More Efficient Use of Resources and Time
Protection of Reputation
More Favorable Public Opinion of Research
Standards can benefit the quality of the research process by strengthening current checkpoints and
counteracting forces that lead to quality challenges (Figure 15). Document and material standards
(e.g., reference reagents) can serve as tools to disseminate best practices to laboratories and
institutions, resulting in the overall elevation of minimum experimental quality and facilitate the
meaningful comparison of results between different laboratories.
Standards can also ease the evaluation of quality and experimental robustness during the grant
review and peer review processes by creating a unifying framework, developed and consented to by
experts in the field, and available to a broad range of stakeholders. A standards framework can enable
stakeholders to ascertain that best practices are being followed without delving into the details of how
they are followed. Over time, adoption of optimal, consensus-based practices and materials standards
can lead to increasingly reproducible results and lower reliance on time-consuming self-correction.
30
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
Life Science Research Standards:
Adding Value and Accelerating Progress
Figure 15. Standards Counteract Quality Challenges
Increasing
Quality Challenges
Improvement
Through Standards
Research Findings
Research Findings
• Decreasing grant
application page
number
➡
• Easier evaluation
of quality and
experimental robustness
Grant Review
• Less resources per
paper leads to less
robust experimental
design
➡
• Better minimum
experimental
design quality
• Decreased bias
and variability
Intralaboratory Pl Review
• Increasing complexity
• Inconsistent review
metrics
➡
• More effective peer
review and ability to
judge complex work
Peer Review
• No incentive for
replication (funding
or publication)
➡
• Decreased variability
• Easier to compare data
Colleague Replication
Scientific Knowledge
• Decreased time due
to increasingly
translational focus
➡
• Less reliance on
time-consuming
self-correction
Scientific Knowledge
Self-Correction
Legend:
High-Quality Work
Low-Quality Work
The funnels represent the research process, with circles representing high-quality (yellow) or low-quality (blue) work. The research
flows through the checkpoints described on the right and most low-quality work is filtered out. With increasing quality challenges, the
filters become more permeable and allow more low-quality work to pass through. Standards, represented by orange arrows, can
counteract the quality challenges and strengthen the current quality checkpoints (orange), resulting in less low-quality research.
By increasing reproducibility, standards can make research results more translatable and facilitate
collaboration between industry and academic laboratories. Industry stakeholders report that they
would prefer to collaborate with academic laboratories that follow standardized processes and use
reliable reagents because this would align these laboratories closer to internal quality processes
routinely practiced by industry (described above). Working with laboratories that adopt best practices
through consensus standards will increase the reproducibility of their research and reduce technology
and process transfer failures. Collaborating with these laboratories could also help to decrease
uncertainty of outcomes inherent in academic-industry research partnerships.
Finally, standards can unify and accelerate current and future efforts to improve life science research
quality. Stakeholders have started to address irreproducibility in multiple, frequently disjointed ways,
including action by journals, funding agencies, and independent organizations.91 For example, Nature
recently launched a Reproducibility Initiative,92 and the NIH has held several workshops to address the
91. Please see Appendix I for detailed descriptions of representative efforts to enhance life science research quality and reproducibility.
92. Nature Editorial Board. Nature. 2013; 496:398.
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
31
Life Science Research Standards:
Adding Value and Accelerating Progress
issue of irreproducibility.93 The 2013 Seventh
Peer Review Congress had several
Figure 16. Standards are a Unifying Driver of Quality
Improvement Efforts
presentations and posters dedicated to this
and related topics.94 Stakeholders interviewed
for this Report can also envision multiple other
ways of improving reproducibility such as
oved
Impr tion
a
docs
in
postEduc
improved education.
Current efforts to improve research quality
and reproducibility in the life sciences are
laudable and have the potential to be highly
Impro
v
Sharin ed
g of D
a ta
“C
Institutional Policies
g
ers
catin
arch
“Edu ior rese f what is
n
o
uce
or ju ncepts
prod
o
the c essary to le work
nec
ducib long
ro
p
a
re
ld go
wou
lder
keho
.”
way ustry Sta
Ind
effective. However, these efforts are highly
“Institutions could
implement programs to
improve quality and advertise
them to potential industry
partners.”
onsis
report tent
ing
c
used ancer sub of
would
ty
data
make pes
a lot
the
ea
comp
are a sier to
nd sy
nthes
Disea
ise.”
se Fo
“Com
unda
tion
mon
are n
d
ecess ata form
ats
a
datas
ets.” ry to share
A
Quality Expert
Journal Review
Policies
cadem
“Publication is the goal
of academic research, so
anything impacting that
will be powerful.”
ic Re
searc
h L ea
der
Academic Research Leader
dependent on the activities of individual
journals, institutions, or organizations,
and place a significant burden on these
stakeholders. Because these efforts are
disconnected, each stakeholder must invest
licies d
er Po
de
Fund NIH demanld do
u
e
“If th eople wo of the
d
p
ch
this, . So mu controlle
it
g is
ould
in
d
ere w .”
fun
th
,
e
em
tiv
by th r alterna h Leader
rc
othe
esea
be no ademic R
Ac
anew in identifying best practices and
developing its own definitions, guidelines, or
Quality Sy
stem
Greater Experimental
Robustness
“It is a matter of
applying the basic
principles of the scientific
method and doing it
consistently, like using
statistically valid sample
size and appropriate data
analysis techniques.”
s
“A formal
qu
is the best ality system
way of
reducing
uncontrolle
d
variables
and insur
ing
appropria
te re
keeping, eq corduipment
calibratio
n and
training.”
Standards
Expert
Industry Stakeholder
systems. By providing a common framework
of practices, biological materials, and methods
developed by or consented to by key
stakeholders in a particular area, standards
can facilitate existing efforts to improve life
science research quality and reproducibility.
Standards
“Over time, science self-corrects. But if you have
standards to improve quality of laboratory procedures, to
improve basic techniques that result in inconsistencies,
this process does not always need to happen.”
Standards Expert
Greater
Experimental
Robustness
• Raise awareness
of proven
methodologies and
lower uncontrolled
variation
Journal Review Policies
• Harmonize and simplify
review policies
Funder Policies
Overall, standards are broadly applicable
across life science research and can
significantly benefit the entire field. Standards
can help the life science research community
align around consensus-based best practices,
reduce uncontrolled variance, and improve
Quality Systems
Improved
Sharing of Data
• Enable common
reporting
frameworks
and formats
• Simplify quality
assessment
during grant
review process
• Enable
implementation
and accreditation
process
Improved
Education
Institutional Policies
• Provide consistent
framework for policies
• Serve as an
educational
tool and ensure
eduction
reproducibility, resulting in more efficient use
of resources, less misinformation, and a more
favorable public opinion of research.
93. Landis, S.C. et al. Nature. 2012; 490(7419):187–191.
94. Peer Review Congress. [Internet] [cited 2013 Sep 15]. Available from http://www.peerreviewcongress.org/2013/Final-Program.pdf.
32
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
Moving Forward to Solve Irreproducibility
While life science research has made tremendous advances over time, traditional methods for ensuring
quality are coming under increasing pressure as the field progresses and the funding environment
worsens. Irreproducibility of life science research findings is a pervasive, systemic, and expensive
problem that affects those directly involved in research and publication, as well as funders and the
general public. Key stakeholders—academic laboratories and institutions, industry and industry
associations, investors, government agencies, charitable foundations, journals and professional
societies, standards development organizations, and policy and patient advocacy organizations—
associate irreproducibility with poor quality, and are voicing significant concerns about the effects
of irreproducibility on their work, on the use of public and private resources, and on public perception
of life science research.
Thus far, however, there has been a chasm between the pervasiveness and seriousness of the
problem and efforts to find solutions. To date, there have been no coordinated endeavors with a
national or international scope to engage all stakeholder groups in order to find effective ways to
increase reproducibility across the life sciences research landscape. Emerging efforts, such as the
Nature Reproducibility Initiative, are important and will surely benefit the field. Nevertheless, the
prevalence and magnitude of this problem requires a systemic, unifying effort.
Life science standards can provide the basis for a collaborative, global movement to improve
reproducibility.
• Standards are an essential platform that can elevate the quality of the field by encouraging
stakeholders to identify and disseminate best practices and optimal materials.
• Standards can serve as a unifying driver for current and future efforts to improve reproducibility
by providing stakeholders with a common framework of expert, consensus-based opinions.
Uniting the diverse life science research community in a standardization effort will be challenging.
Nonetheless, multiple other fields have successfully incorporated standards to improve quality by
associating with core organizations. For example, over a million organizations are ISO 9000-certified and
thousands of laboratories adhere to the College of American Pathology standards. The development
and implementation of standards, whether as adherence to written documents or use of reference
materials, requires community consensus and alignment around both the necessity for standards and
their content. Ultimately, expanded adoption of life science standards will require:
• Educational initiatives to raise stakeholder awareness of the purpose and benefits of biological
standards and understanding of the standards development process;
• Opportunities and forums for stakeholders to identify areas in the life sciences where
accelerated standards adoption could provide maximum benefit;
• Engagement of stakeholders with standards development organizations or material reference
providers in the development of specific standards; and
• Development of effective policies and practices within the life science research community to
ensure the proactive development and periodic updating of biological standards.
The life science research community is creative, inventive, and innovative. It is a community that
has found cures for diseases and extended human life. This community can solve the problem of
irreproducibility, and a solid standards framework can unify and accelerate this global effort.
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
33
Appendix I: Supporting Materials
Description of Current Academic Life Science Research Process
Government research funders (e.g., NIH) and non-government funders, such as disease or other
charitable foundations, all use the peer review95 process to evaluate grant proposals. Multiple criteria,
such as the potential importance of work to the field and adherence to funders’ guidelines, are used.
This review typically forms the basis for deciding whether the funding will be granted.
After funding for a particular research project is obtained, the PI is responsible for controlling the quality
of the research performed and of any papers submitted for publication. He or she must balance
ensuring research quality with allowing sufficient freedom to promote innovation. The PI must also
maintain a suitable educational environment because academic research laboratories are primarily
staffed by graduate and post-graduate students.96
After a manuscript is prepared, it is submitted to a journal for publication. Checkpoints here are
particularly important in ensuring the quality of research because the researchers’ publication records,
including the number of publications and the reputation of the journal where the work is published,
are commonly used to determine whether a researcher is hired, promoted, or receives tenure.97 These
considerations may also be taken into account during the grant-funding process.
Journals use a combination of peer and editorial review to determine which papers will be accepted
for publication. Journals also take the novelty and importance of results to the field into account.
Extensive variability exists in requirements (e.g., reporting guidelines) for publication as well as
criteria used by different journals and reviewers to evaluate work.
Publications release and disseminate results into the broader scientific community where findings and
interpretation face additional scrutiny. Feedback from the life science community suggests that researchers
believe that when the journal peer review process fails and erroneous data or interpretations get
published, highly important or controversial results will be either replicated and confirmed or refuted
by other laboratories within a few years. If the findings are reproducible, other laboratories will often
use them as the basis for additional research, resulting in further publications and the growth of the
field. If findings are not reproducible, the situation becomes more variable. When the original hypothesis is
of tremendous significance, the irreproducibility of results is usually published, although this might
take several years. For results of less perceived significance, the inability to reproduce findings is
frequently not discovered nor reported.
Over the longer term, work that builds upon the conclusions of published research either confirms or
refutes them, resulting in a process of scientific “self-correction.” This process may take many years, is
not infallible, but acts to eliminate erroneous hypotheses and results from the scientific worldview.98,99
Academic researchers are typically comfortable with the concept and process of self-correction because it
stems from the basic trial-and-error approach inherent in the application of the scientific method.
Document Standards
The International Organization for Standardization (ISO) and American National Standards Institute
(ANSI) define a standard as “a document, established by consensus and approved by a recognized
body, that provides, for common and repeated use, rules, guidelines or characteristics for activities or
their results, aimed at the achievement of the optimum degree of order in a given context.”100
95. Peer review is a process in which the work of one scientist is evaluated by other scientists with expertise in the field,
generally recruited on a voluntary basis.
96. Rohn, J. Nature. 2011; 471:7.
97. Mann, S. AAMC Reporter. 2013 Jan.
98. Ioannidis, J.P. Persp Psych Sci. 2012; 7(6):645–654.
99. The Economist. [Internet] [cited 2013 Oct 21]. Available from
http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble.
100. ISO/IEC Guide 2:2004, definition 3.2.
34
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
Appendix I: Supporting Materials
“Approval by a recognized body” is the main factor that separates voluntary document standards
from consensus guidelines or best practices.101 Guidelines, recommendations, or best practices can
be developed by any group and do not necessarily need to engage all stakeholders that the guideline
may affect. In contrast, standards are commonly developed by standards development organizations
(SDOs) or consortia. SDOs are collaborative organizations that meet international requirements for
openness, balance, consensus, and due process, and provide all affected stakeholders with an
opportunity to participate in the development of standards.101 These requirements are outlined in a
system of standards pertaining to SDOs, and SDOs
are accredited to follow these standards by national
“Standards should be based on the consolidated results of
science, technology and experience, and aimed at the
promotion of optimum community benefits.”
International Organization for Standardization
in ISO/IEC Guide 2:2004, definition 3.2
and international standards bodies. ANSI is both the
U.S. national standards body and representative to
ISO, which is the worldwide federation of national
standards bodies. ISO, which was established in 1947
and is based in Geneva, Switzerland, draws members
from over 145 countries. There are currently over 200
SDOs in the United States participating in this system,
which helps ensure that all major perspectives in a particular field are elicited during the standard
development process, and that all the major stakeholders are engaged. Consortia are generally not
accredited, but follow the spirit of this system by engaging all key viewpoints in a particular field
during standards development.
Document standards have been adopted across multiple industries. For example, the National
Electrical Manufacturers Association (NEMA) publishes standards for electrical plugs and sockets in
the United States to ensure that all electrical devices intended for home use have similar plugs that are
compatible with commonly installed sockets.102 Standards describing systems that companies need
to put in place in order to produce high-quality products and services, such as ISO 9000 series or ASQ
Quality Standards, are broadly used in most industries to improve and maintain performance. College
of American Pathology (CAP) Standards outline processes and procedures necessary to maintain
quality in clinical diagnostic testing and are used by clinical laboratories throughout the United States.
Accreditation is one way to encourage compliance with voluntary standards. For example, a company
may choose to work only with suppliers that have ISO 9000- or ASQ-accredited quality practices
because this accreditation increases the likelihood that the supplier will deliver a high-quality product.
As a result of voluntary inducement and community expectations, over one million companies are
ISO 9000 series-certified.103 The Centers for Medicare and Medicaid (a government agency) uses
accreditation by Joint Commission or CAP (both of which are independent, non-profit organizations)
as a basis for determining whether Medicare will reimburse services from a particular clinical
laboratory,104 resulting in almost universal laboratory certification.
Materials Standards
Materials standards are highly characterized physical substances, such as chemical or biological
reagents (known as reference materials or reference reagents), and are routinely used in several areas
of life science research and healthcare. Examples of materials standards include reagents for biologics
and pharmaceutical development and production, such as well-characterized virus strains to be used
for vaccine production (e.g., annual flu vaccines) or in genetic therapy development. Materials
101.
102.
103.
104.
ANSI. [Internet] [cited 2013 Sep 17]. Available from http://www.ansi.org/about_ansi/faqs/faqs.aspx.
NEMA. NEMA Standards Publication. ANSI/NEMA WD 6-2002 (R2008).
ISO. “The ISO Survey of Management System Standard Certifications – 2011.”
CAP. About the CAP Laboratory Accreditation Program.
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
35
Appendix I: Supporting Materials
standards can also be used to test assays105 in order to ensure that assays provide accurate
measurements and work appropriately. For example, when a sample is tested for a specific infectious
organism (e.g., a bacterial pathogen) in a clinical laboratory, infectious organism reference materials
(or derivative commercial reagents) can be used to ensure that the assay is sufficiently sensitive and
specific to detect the microorganism.
Reference materials can be produced and distributed by several sources, depending on the type of
reference material. Sources may include the WHO; government agencies such as the NIH and the
National Institute of Standards and Technology (NIST); non-profit entities, including biological resource
centers like ATCC; and commercial companies. The exact source varies by the specific material.
Standards Example: HbA1c Testing
Materials and systems standards are effective in reducing inter-laboratory variability and irreproducibility, as
exemplified by the standardization of testing for glycated hemoglobin. Glycated hemoglobin is a marker of how
effectively blood sugar is controlled in diabetic patients. Although a direct blood glucose measurement
provides information about the patient’s blood sugar control at that moment, glycated hemoglobin
measurements provide information for how blood sugars have been controlled during the past month
or more. Hemoglobin A1c (HbA1c) is one type of glycated hemoglobin test. The importance of tightly
controlling blood sugar in diabetic patients and of HbA1c as a marker of control became broadly accepted
as a result of two large clinical trials.106,107 Both trials determined HbA1c values that were associated with a
lower risk of diabetic complications, and resulted in recommendations that patients’ blood glucose
should be controlled to a degree that would maintain the patient’s HbA1c below these target levels.
Figure A. Standardization of HbA1c Testing
8.0
7.5
7.0
% Hb A
6.5
6.0
5.5
5.0
4.5
4.0
3.5
Legend:
3.0
HbA1c
HbA1
Total GHb
DCCT Target
2.5
1993
1999
2004
2010
Standardization
Figure reproduced from Little (2011). Data from College of American Pathology surveys 1993–2010. Each circle, square, or rhombus
represents the mean value for testing a control material obtained through distinct method of glycated hemoglobin determination used
among survey participants. Bars show 2 standard deviations from the mean.
105. An assay is a biological, biochemical, or chemical way of measuring if and how much of a particular substance, biological molecule, or
organism is present. Assays are frequently used in research, public health, or as diagnostic tests.
106. The Diabetes Control and Complications Trial Research Group. N Engl J Med. 1993; 329:977–986.
107. U.K. Prospective Diabetes Study (UKPDS) Group. Lancet. 1998; 352:837–853.
36
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
Appendix I: Supporting Materials
At the time these trials came out, laboratories used multiple methods for measuring glycated hemoglobin
that produced results that were not directly comparable.108 For example, in 1993, only 50% of laboratories
were actually reporting results as HbA1c; the remaining were reporting different measurements such
as total glycated hemoglobin. Furthermore, even laboratories using the same methods had substantial
variability in results. In essence, the results were irreproducible from laboratory to laboratory. This
made it difficult or impossible for clinicians to compare the HbA1c levels obtained from their patients
to the target levels from the trials or even to compare the results for a single patient over time if they
were obtained in different laboratories (Figure A). For
example, if the same patient’s blood sample was sent
to two different laboratories, it would not be unusual
“HbA1c testing standardization is an example of a success
story. The community aligned behind this effort. The
clinical societies, laboratory societies, manufacturers, and
laboratories themselves – everyone came to understand
that the standardization was necessary to provide
evidence-based patient care.”
Standards Expert
for one laboratory to produce results that were below
the target level established in the clinical trials,
indicating that the patient’s treatment was working
appropriately, and for a different laboratory to produce
results significantly above the target level, indicating
that the treatment was not working and should be
increased or changed.
Due to the extent of this problem, two professional
societies, the American Association for Clinical
Chemistry (AACC) and International Federation of
Clinical Chemistry (IFCC), created committees to enable the standardization of testing methodologies.
These groups created two complementary systems that together serve to standardize manufactured
HbA1c assays. The IFCC created reference materials for HbA1c measurement that can be used across
different methodologies for calibration, and to ensure that findings obtained though all methodologies
are comparable. Manufacturers can obtain calibrators and controls with values assigned by the IFCC and
use them to determine how their assay methodology compares to the IFCC reference method. The AACC
effort developed a standard protocol109 and created the National Glycohemoglobin Standardization
Program (NGSP), a laboratory-based certification program where manufacturers of HBA1c tests can
exchange samples with special NGSP laboratories in order to certify that the results of the test are
consistent with the results obtained using the standard protocol. Thus, the IFCC program ensures that
each manufacturer can trace its assay to an accurate base through a reagent standard-based system,
whereas the NGSP program certifies that the assay accuracy is within an acceptable range of results
obtained using a standard protocol.
Together, these standardization systems significantly improved the accuracy, reproducibility, and
clinical value of HbA1c testing. As Figure A shows, in 2010 with over 3,000 laboratories participating
in the CAP survey for HbA1c testing, the variability between different testing methods and within
each testing method was significantly lower than in the years before standardization became common
(1993 and 1999). Today, clinicians can routinely use and meaningfully compare HbA1c values from
different laboratories and be assured that the values they are using to treat their patients are
accurately connected to clinical trial results.
108. Little, R. Clin Chem. 2011; 57(2):205–214.
109. Little, R. Clin Chem. 2001; 47(11):1985–1992.
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
37
Appendix I: Supporting Materials
Standards Example: Bacterial Testing of Platelets
The development and adoption of standards surrounding the bacterial testing of platelets is an example
of how the quality of laboratory practices can be improved by document (written) standards. Platelets are
a component of blood donations that are given to patients with low platelet counts in order to prevent
bleeding complications. Over 2 million units of platelets were administered to patients in the United
States in 2011.110 Because of their biological properties, platelets are stored at room temperature,
which encourages the growth of bacteria. If bacterially-contaminated platelets are given to a patient,
the patient might get ill and develop a systemic infection or sepsis.
Prior to 2004, some blood banks used technologies to detect bacterial contamination in platelets, believing
it to be necessary for patient protection, but many blood banks did not. As a result, bacterial contamination
of platelet products was acknowledged to be the most frequent infectious risk from transfusion in the
United States, and was considered the second most common cause of death resulting from transfusion.111
In response to increasing research pointing to bacterial contamination of platelets as a significant cause of
transfusion-related illness and death, in 2004 the AABB, a voluntary standards and accreditation body,
developed a standard for limiting and detecting bacterial contamination of platelets. The development
of that standard involved extensive input and
consensus from the blood banking community.
As a result of adoption of this standard by AABB-
“At the time when the standard was implemented,
accredited blood banks, septic transfusion
some people were resistant because they felt it
reactions per 100,000 transfusions had decreased
increased blood bank costs and disrupted operations.
by 70%. There was also a similar decrease in
deaths from transfusion-related sepsis.112 Overall,
Of course it wasn’t perfect, but looking back now,
the AABB standards surrounding bacterial testing
having the standard is a really good thing. It improved
of platelets resulted in better care and saved lives
transfusion safety and cut the risk associated with
of platelet recipients.
bacterial contamination of platelets more than in half.”
Blood Banking Expert
Standards Example:
Laboratory Animal Welfare
The most well-developed standards and accreditation system exists to protect laboratory animal
welfare. In the United States, this system is centered around the Guide for the Care and Use of
Laboratory Animals, a collection of detailed standards for appropriate laboratory animal care initially
published in 1963 and revised multiple times thereafter. The latest revision (the 8th edition) was
published in 2011 by the Committee for the Update of the Guide for the Care and Use of Laboratory
Animals,113 which was appointed in 2008 by the National Research Council.114 The Committee included
research scientists, veterinarians, and nonscientists representing biomedical ethics and the public’s
interest in animal welfare. Funding for development and publication of the Guide was received from
government agencies and non-profit organizations, as well as from the pharmaceutical industry.
Adherence to the principles in the guide is required for institutions that receive Public Health Service
support (including grants from NIH and CDC), and is monitored internally by each institution through
self-regulating Institutional Animal Care and Use Committees.115 Additionally, an independent voluntary
110.
111.
112.
113.
U.S. Department of Health and Human Services. The 2011 National Blood Collection and Utilization Survey Report.
Hillyer, C.D. Hematology Am Soc Hematol Educ Program. 2003:575–589.
AABB. Public Conference – Secondary Bacterial Screening of Platelet Components, 7/17/12.
Committee for the Update of the Guide for the Care and Use of Laboratory Animals. Guide for the Care and
Use of Laboratory Animals, 8th Ed., NRC 2011.
114. The National Research Council (NRC) is a group of non-profit institutions that include the National Academy of Sciences
and the Institute of Medicine.
115. IACUC. [Internet] [cited 2013 Aug 18]. Available from http://www.iacuc.org/aboutus.htm/.
38
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
Appendix I: Supporting Materials
accreditation system administered by the Association for Assessment and Accreditation of Laboratory
Animal Care International (AAALAC International) exists to ensure that institutional self-monitoring is
optimal. The AAALAC accreditation also requires adherence to two other standards when applicable:
Guide for the Care and Use of Agricultural Animals in Research and Teaching (Ag Guide)116 and the
European Convention for the Protection of Vertebrate Animals Used for Experimental and Other
Scientific Purposes (ETS 123).117
Standards Example: Data Reporting Frameworks
Voluntary standards that have been recently adopted
in life science research are focused on establishing
“Before the MIAME criteria, we found that even for the
data-reporting frameworks. Examples of these include
same experimental dataset, the analytic results could only
the Minimal Information About a Microarray Experiment
be reproduced approximately 10 to 30% of the time
(MIAME, developed in 2001),118 Minimum Information
because the information in the methods section was not
sufficient to ensure that two investigators were really
performing the analysis similarly. This is why our journal
adopted the standard as a condition for publication.”
Journal Editor
about a MARker gene Sequence (MIMARKS) and
Minimum Information about Any (x) Sequence (MIxS,
developed in 2011),119 Minimal Information About a
Proteomics Experiment (MIAPE, developed in 2007),120
and Minimum Information about a Flow Cytometry
Experiment (MIFlowCyt, developed in 2008).121 Multiple
other reporting guidelines for a broad range of
experimental techniques are also available.122
These standards, developed by consortia rather than SDOs, apply to high-throughput technologies
that generate large datasets of results that are then analyzed using complex bioinformatic techniques.
Multiple sources of variability exist in experimental techniques used to generate the data and in
assumptions that can be made during the data analysis process (e.g., where to set the cut-off for how
much signal intensity is necessary to consider a result positive). These standards encourage more
detailed reporting with a particular focus on steps that are common sources of inter-laboratory variability
and, as a result, these standards improve reproducibility between laboratories. The MIAME standard
has the broadest adoption, and adherence to this reporting framework is currently required by over 50
journals, including high-profile publications such as Cancer Research, Cell, The Lancet, and Nature.123
There are multiple ongoing standards efforts in this area, including Core Information for Metabolomics
Reporting (CIMR), a standard to specify minimal guidelines for reporting metabolomics data that is
being developed by the Metabolomics Standards Initiative (MSI),124 and the Standard Reference System
for Information on Bioinformatics Data Structures that is being developed by IEEE (Institute of Electrical
and Electronics Engineers) to help support exchange and comparison of biological data sets in the life
sciences.125 A recently-formed international initiative, the Global Alliance to Enable Responsible Sharing
of Genomic and Clinical Data, intends to develop technical and other standards to enable sharing of
human genome sequencing data for research purposes in a “secure, controlled and interpretable manner.”126
116,
117.
118.
119.
120.
121.
122.
123.
124.
125.
126.
Federation of Animal Science Societies (FASS). [Internet] [cited 2013 Oct 22]. Available from http://www.fass.org/docs/agguide3rd/Ag_Guide_3rd _ed.pdf.
European Treaty Series (ETS). [Internet] [cited 2013 Oct 22]. Available from http://conventions.coe.int/treaty/en/treaties/html/123.htm.
Brazma, A. Nature Genetics. 2001; 29:365–371.
Yilmaz, P. Nature Biotechnol. 2011; 29:415–420.
Taylor, C. Nature Biotechnol. 2007; 25:887–893.
International Society for Advancement of Cytometry (ISAC). Minimum Information about a Flow Cytometry Experiment: MIFlowCyt 1.0.
Minimum Information for Biological and Biomedical Investigations (MIBBI). [Internet] [cited 2013 Aug 18]. Available from
http://mibbi.sourceforge.net/portal.shtml.
Functional Genomics Data Society (FGED). [Internet]. [Cited 2013 Aug 18]. Available from http://www.mged.org/Workgroups/MIAME/journals.html.
Metabolomics Society. [Internet] [cited 2013 Aug 18]. Available from http://msi-workgroups.sourceforge.net/.
IEEE. [Internet] [cited 2013 Aug 18]. Available from http://standards.ieee.org/develop/wg/1953_WG.html.
Global Alliance to Enable Responsible Sharing of Genomic and Clinical Data. Creating a Global Alliance to Enable
Responsible Sharing of Genomic and Clinical Data. White Paper, 2013 Jun 3.
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
39
Appendix I: Supporting Materials
Initial Efforts to Increase Reproducibility
Journal publication requirements, particularly those established by prestigious and respected journals, are
a powerful incentive for scientists. Recently, several journals have made efforts to combat irreproducibility
of published findings, primarily by encouraging more detailed and complete reporting of methods and
procedures. The Nature Reproducibility Initiative,127 for example, centers on an 18-step checklist that
includes steps to ensure that authors disclose sufficient technical and statistical information. It is
particularly focused on experimental and analytical design elements that the editors consider “crucial
for the interpretation of research results but are often reported incompletely.” In conjunction with this
initiative, Nature and several associated journals also plan to abolish space restrictions on the methods
section in each paper.
The NIH has held several recent workshops to address issues of reproducibility. In 2012, NIH’s National
Institute of Neurological Disorders and Stroke (NINDS) convened major stakeholders to discuss
improving the methodological reporting of animal studies in grant applications and publications.128 The
workshop recommended “that at a minimum studies should report on sample-size estimation, whether
and how animals were randomized, whether investigators were blind to the treatment, and the handling
of data.” A National Cancer Institute (NCI) Board of Scientific Advisors meeting in March 2013 included
two presentations focused on reproducibility, including “Data Replication: How Reliable Are the Published
Results of NIH-Funded Research”129 by Dr. Harold Varmus, the Director of NCI. Further activity in this
area is expected as senior NIH officials assess input from the agency’s 27 institutes before conferring
with NIH director Francis Collins.130 The NIH is also considering a variety of proposals, and to date,
consensus on the best course of action does not appear to have been reached. Although NIH efforts to
consider reproducibility are in the early stages, they may have significant impact on the life science
research community in the future due to the potential connection of these initiatives to grant funding.
Independent organizations have also started efforts to improve research reproducibility. The Reproducibility
Initiative131 is a collaborative effort of several organizations including the Science Exchange, a website that
is dedicated to connecting scientists to services they may need; PLOS ONE, a major open-source journal;
Mendeley, a reference manager; and FigShare, a data storage and sharing service. To participate, investigators
can submit their publications to the website and pay for the independent replication of their research.
The Reproducibility Initiative has collected nearly 2,000 author consents to reproduce their work, and
recently announced that it had received $1.3 million from the Laura and John Arnold Foundation to assess
50 of the highest-impact cancer findings published between 2010 and 2012.132
The U.K.-based EQUATOR (Enhancing the QUAlity and Transparency Of health Research) Network,
formed in 2008, “seeks to improve the reliability and value of published health research literature by
promoting transparent and accurate reporting and wider use of robust reporting guidelines.”133 The
Network is directed by an international Steering Group that brings together researchers, medical journal
editors, peer reviewers, developers of reporting guidelines for health research, research funding
bodies, and other collaborators concerned with improving the quality of research publications and of
the research process itself. Although EQUATOR is primarily focused on reporting of research involving
human subjects, the organization’s scope also includes guidelines related to pre-clinical animal studies.134
127.
128.
129.
130.
131.
132.
Nature Editorial Board. Nature. 2013; 496:398.
Landis, S.C. et al. Nature. 2012; 490(7419):187–191.
[Internet] [cited 2013 Aug 8]. Available from http://deainfo.nci.nih.gov/advisory/bsa/bsa0313/.
Waldman, M. Nature. 2013; 500:14–16.
[Internet] [cited 2013 Aug 14]. Available from https://www.scienceexchange.com/reproducibility/.
[Internet] [cited 2013 Oct 21]. Available from http://blog.scienceexchange.com/2013/10/
reproducibility-initiative-receives-1-3m-grant-to-validate-50-landmark-cancer-studies/ [cited 2013 Oct 21].
133. [Internet] [cited 2013 Oct 8] Available from http://www.equator-network.org/.
134. [Internet] [cited 2013 Oct 9] Available from http://www.equator-network.org/?post_type=eq_guidelines&eq_guidelines_
study_design=animal-pre-clinical-research&eq_guidelines_clinical_specialty=0&eq_guidelines_report_section=0&s=.
40
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
Appendix II: List of Acronyms
AAALAC
AABB
AACC
AACR
ACOG
ADDF
ANSI
ASQ
ATCC
CAP
CDC
CF
CFS
CIMR
EQUATOR
ETS
FASEB
FASS
FDA
GBSI
HbA1c
HIV
IACUC
ICLAC
IEC
IEEE
IFCC
IND
ISAC
ISO
MIAME
MIAPE
MIBBI
MIFlowCyt
MIMARKS
MIxS
MSI
NCI
NEMA
NINDS
NGO
NGSP
NIH
NIST
NRC
NSF
PI
RePORT
SDO
STR
UKPDS
WHO
XMRV
Association for Assessment and Accreditation of Laboratory Animal Care
(previously known as American Association of Blood Banks)
American Association for Clinical Chemistry
American Association for Cancer Research
American Congress of Obstetricians and Gynecologists
Alzheimer's Drug Discovery Foundation
American National Standards Institute
American Society for Quality
American Type Culture Collection
College of American Pathology
Centers for Disease Control and Prevention
cystic fibrosis
chronic fatigue syndrome
Core Information for Metabolomics Reporting
Enhancing the QUAlity and Transparency Of health Research Network
European Treaty Series
Federation of American Societies for Experimental Biology
Federation of Animal Science Societies
Food and Drug Administration
Global Biological Standards Institute
hemoglobin A1c
human immunodeficiency virus
Institutional Animal Care and Use Committee
International Cell Line Authentication Committee
International Electrotechnical Commission
Institute of Electrical and Electronics Engineers
International Federation of Clinical Chemistry
Investigational New Drug
International Society for Advancement of Cytometry
International Organization for Standardization
Minimal Information About a Microarray Experiment
Minimal Information About a Proteomics Experiment
Minimum Information for Biological and Biomedical Investigations
Minimum Information about a Flow Cytometry Experiment
Minimal Information About a MARker gene Sequence
Minimum Information about Any (x) Sequence
Metabolomics Standards Initiative
National Cancer Institute
National Electrical Manufacturers Association
National Institute of Neurological Disorders and Stroke
non-governmental organization
National Glycohemoglobin Standardization Program
National Institutes of Health
National Institute of Standards and Technology
National Research Council
National Science Foundation
principal investigator
Research Portfolio Online Reporting Tool
standards development organization
short tandem repeat
U.K. Prospective Diabetes Study Group
World Health Organization
xenotropic murine leukaemia virus-related virus
THE CASE FOR STANDARDS IN LIFE SCIENCE RESEARCH
41
The Case for Standards in Life Science Research:
Seizing Opportunities at a Time of Critical Need
The GBSI trademark and trade name is a trademark
of Global Biological Standards Institute unless
indicated otherwise.
This study was commissioned by GBSI. The research
and the writing of the Report was completed by
Health Advances, a global healthcare strategy firm,
in collaboration with Feinstein Kean Healthcare,
a strategy and communications firm for life
sciences and healthcare.
For more information, please contact us at
www.gbsi.org
[email protected]
www.gbsi.org/blog
LinkedIn: GBSI
Follow us @GBSIorg
Our offices are located at:
1020 19th St., NW, Suite 550
Washington, DC 20036 USA
+ 1 (202) 772-0133