Biological and Life Sciences - Council of Graduate Schools

Appendix B. Biological and Life Sciences Module
Biological and Life Sciences
Introduction to Biological and Life Sciences
Welcome to the Biological and Life Sciences portion of the Research Integrity in International
Collaborative Contexts course. Please read this brief background as to the importance of this topic
before exploring the four areas of content.
Collaborative Research
Conflicts of Interest
Publication Practices and Responsible Authorship
Data Management: Access, Sharing, and Exchange
Additional Resources
1
Appendix B. Biological and life Sciences Module
Introduction to Biological and Life
Sciences
The Biological and life Sciences comprise the fields of science that involve the scientific study of living
organisms - such as microorganisms, plants, animals, and human beings - as well as related fields like
bioethics. While biology remains the centerpiece of the life sciences, technological advances in
biotechnology and molecular biology have led to a burgeoning of specializations and interdisciplinary
fields. The Biological and Life Sciences are helpful in improving the quality and standard of life. They
have applications in health, agriculture, medicine, and the pharmaceutical and food science industries.
Some life science fields focus on a specific type of life. For example, botany is the study of plants, while
zoology is the study of animals. Other life sciences focus on aspects common to all or many life forms,
such as anatomy and genetics. Other fields are interested in technological advances involving living
things, such as bio-engineering. Another major, though more specific, branch of life sciences involves
understanding the mind - neuroscience.
Ethics is the concept of balancing the benefits and risks of choices and decisions. The underlying
heritage of ethics is present in all cultures and religions in the world. Research integrity includes
(National Institutes of Health, 2013):
•
the use of honest and verifiable methods in proposing, performing, and evaluating research
•
reporting research results with particular attention to adherence to rules, regulations, and
guidelines, and
•
following commonly accepted professional codes or norms
Ethical considerations in international research contexts are vast and complex, especially in the
biological and life sciences. First, biological and life sciences research is multi-disciplinary and
interdisciplinary, encompassing different disciplines, countries, cultures, and social and political
identities and beliefs. Second, there are various stakeholders involved in biological and life science
research, including governmental agencies, healthcare providers, corporations, administrative
personnel, and the various communities of study. Scientific research should always encompass the
values of honesty, accuracy, efficiency, and objectivity. Researchers working in international contexts
and with international collaborators must understand the different research, cultural, and social norms
of that country, as well as the differences in guidelines, regulations, and policies.
This module focuses on national and international guidelines and the complex issues of conducting
research in cross-cultural and international contexts. In this module, you will examine
1.
2.
collaborative research and how it changes in international contexts,
how conflicts of interest can arise in research and suggestions for avoiding or resolving the
conflicts,
3. publication practices and responsible authorship, and
4. data management, including the access, sharing, and exchange of information
2
Appendix 8. Biological and Life Sciences Module
Collaborative Research
Goals
When you have finished reading the information in this section, you should be able to:
•
•
Understand recent trends in international collaboration in the life sciences
Describe some of the factors that have contributed to those trends
•
Understand the challenges of international collaboration in terms of rules and regulations
You will take a short quiz after completing the module to test your understanding of the content.
Collaborative Resea rch in the Biological and Life Sciences
Collaboration among life scientists is growing in part because of the increasing recognition that
environmental and human health issues are best understood through international and interdisciplinary
research. There are numerous international collaborative networks aimed at understanding global
environmental issues, including climate change and biodiversity loss. These co llaborations can involve
shared ideas and data but also access to complex equipment or other reso urces (e.g., environmental
samples). Collaborations aimed at understanding complex issues can be small in scale, involving just a
few researchers from different countries, or global in scope, involving researchers from around the
world, who provide data in specific formats for use by network members, and in many cases, the
general public.
Collaborations may also be increasing because it is easier for researchers to communicate and share
data through technological advances. Communication through email, phone, fax, and video chatting
ensures direct communication with your collaborators. Sample shipments and travel are also easier to
all parts of the world . In addition, funding for data management, including t raining for new researchers
is available for many collaborative networks, breaking down one of the major barriers of data sharing.
Trends in Life Science Collaboration
Scientific output, as measured by publications, is increasing, and data from the 2012 National Science
Foundation Scientific Indicator Report shows that a large component of this increase is attributed to
non-US institutions. In addition, international collaboration is increasing in nearly all field s of scie nce
and engineering research. Approximately 32% of U.S. articles in 2010 were internationally coauthored,
which is up from 23% in 2000 (National Science Foundation Indicator Report, 2012). Increased rates of
international co-authorship are evident among the countries of the European Union, w here large
Framework Research Programs encourage it. For example, Operational Eco logy (OPEC) consists of nine
partner institutions from across Europe who are developing forecast tools to assess and manage the
risks of human activities to the marine environment (Operational Ecology). In contrast, while t he
science and engineering output of China has increased dramatically in recent years, this increase is
largely the re sult of domestic research (National Science Foundation Ind icator Report, 2012).
3
Appendix B. Biological and life Sciences Module
While international collaborative opportunities are very exciting, different countries operate under
different regulations. In their 2012 publication, The Inter Academy Council uses the following example
to illustrate this point:
"In recent decades, many universities and other research institutions, scientific societies, and
national governments have developed rules, guidelines, institutions, and procedures to address
actions that damage the research enterprise. As this body of work has developed, significant
differences have emerged among countries. For example, the U.S. federal government defines
"research misconduct" as "fabrication, falsification, or plagiarism (FFP) in proposing, performing,
or reviewing research, or in reporting research results" (Office of Science and Technology Policy,
2000). By contrast, Finland defines "fabrication, misrepresentation, plagiarism and
misappropriation" as "fraud in science," and includes behavior such as " understatement of
other researchers' contribution to a publication and negligence in referring to earlier findings"
as "misconduct in science" (National Advisory Board on Research Ethics in Finland: TENK, 2002).
The Australian Code for the Responsible Conduct of Research (National Health and Medical
Research Council, 2007), includes "failure to declare or manage a serious conflict of interest,"
"avoidable failure to follow research proposals as approved by a research ethics committee,"
and "willful concealment or facilitation of research misconduct by others" in its definition of
research misconduct."
It is important to be aware of these differences and to draft research collaboration contracts with these
differences in mind.
Guidelines for Successful Collaboration
In order for collaboration to be effective at small or large scales, clear goals and communication about
how those goa ls will be achieved are critical. Problems among collaborators can be diverse, ranging
from a misunderstanding of project goals, to disagreements on the timing of publication, or decisions
about the extent of communication with the media. In th eir article, Gadlin and Jessar (2011) suggest
that collaborators address the following questions at the onset of their collaboration to avoid these
issues:
What are the scientific issues, goals, and anticipated outcomes or products of the
collaboration?
When is the project over?
Are all members of the research team on the same wavelength regarding these issues?
What are the expected contributions of each participant?
Who will write any progress reports and final reports?
How will you decide about redirecting the resea rch agenda as discoveries are made?
What will be your mechanism for routine communications among members of the research
team (to ensure that all appropriate members of the team are kept fully informed of relevant
issues)?
How will you negotiate the development of new collaborations and spin-off projects, if any?
How, and by whom, will personnel decisions be made? How and by whom will personn el be
supervised?
What will be the criteria and the process for assigning authorship and credit ?
4
Appendix B. Biological and Life Sciences Module
•
How will credit be attributed to each collaborator's institution for public presentations,
abstracts, and written articles?
How and by whom will public presentations be made?
How and by whom will media inquiries be handled?
When and how will you handle intellectual property and patent applications?
How and by whom will data be managed? How will access to data be managed? How will you
handle long-term storage and access to data after the project is complete?
In addition to the above questions, Shamoo and Resnik (2003) suggest that collaborators discuss
responsibility and accountability in the context of their project. Collaborations frequently bring together
individuals with different skills who might individually carry out only a small portion of a complex
project. Sha moo and Resnik (2003) indicate that "people are accountable for an action if they have an
obligation to give an account of the action, while people are responsible for an action if they deserve to
be praised or blamed for an action." They give the example of a boy who throws a rock through a
window - he is responsible for throwing the rock, but his parents may be accountable for his actions
(Sha moo & Resnik, 2003). How should responsibility and accountability be assigned in complex
collaborations?
Consider the list of questions above, as well as the issue of accountability versus responsibility in the
context of international collaborations. Are any of the questions more challenging to answer? Are the
contributions and timelines realistic for all participating countries the same considering their financia l,
personnel, and technical resources? Gaining permission to collect samples can be relatively quick and
easy in some countries but may require permission from multiple government agencies in
another. Differences in perspective ca n be more subtle. Promotion practices in some countries include
consideration of the h-index of the candidate, a measure which combines scientific productivity
(number of journal articles) with the impact of those articles (how frequently they are cited by other
researchers). How might decisions on when to publish be made if one collaborator is trying to boost
their h-index ahead of an application for promotion, while the h-index is irrelevant for another who
might want to wait to publish until multiple years of data have been collected?
It is important to understand the differences in scientific norms among nations, but it is also important
to be informed about potential cultural differences that could affect the success of collaboration,
including the degree of international communication, how experiences are influenced by different
genders and generations, and variations in the perceptions and expectations about the research
timeframe.
Are there other questions you and your collaborators should discuss when working
internationally?
Case Study
Regarding the issue of accountability and responsibility, rea d these short articles on the case of Dr.
Gerald Schatten (2006) and this cloning controversy (2009).
As you complete the readings above, think about the following questions, and be prepared to discuss
them when we meet in person.
s
Appendix B. Biological and Life Sciences Module
1.
Do you think that Dr. Schatten should have been held more accountable for the data in the
manuscript which he co-authored? Why or why not?
2. Do you think the fact that the researchers involved worked in different countries had
anything to do with the issue or that it could have happened just as easily to scientists
working in labs right next door to one another, or across the country? Why or why not?
Quiz - Collaboration in the Biological Sciences
(Answers are bolded)
1.
2.
3.
Collaborative research in the biological/life sciences has increased substantially in the last
decade?
a. True
b. False
Which of the following is a reason for increased international collaborations in biological/life
sciences?
a. Ease of communication via email and other electronic means
b. Greater realization that many biological issues/problems are international in scope and
can best be studied/solved by international cooperation
c. Ease of international travel and shipment of supplies, etc., around the world
d. All of the above
Is it best to wait until collaboration has been underway a while before discussing issues like data
storage, authorship on publications, or how to address questions from the media.
a. True
b. False
6
Appendix B. Biological and Life Sciences Module
Conflicts of Interest
Goals
After reading the information in this section, you should be able to:
•
Define conflict of interest and explain its prevalence in the biological sciences
•
Understand the importance of conflict of interest and how it might hinder scientific progress
and harm objectivity
•
Understand the guidelines around disclosing conflicts of interest and suggestions for addressing
conflicts of interest
You will read a case study and complete an exercise after reviewing this module.
Conflict of Interest
Researchers have a tradition of free inquiry and free exchange of ideas, united in the shared purpose to
create knowledge, critique existing knowledge, and disseminate knowledge. Trust, the core ethical value in
this issue, is essential in the scientific pursuit of the truth. A relationship based on trust is necessa ry with
colleagues, the government, research sponsors, and, of course, the public.
Collaborators must be clear, open, and honest regarding the purpose, methods, outcomes, and funding
sources for their research. Conflicts of interest, including competing obligations or interests, can lead to
competing responsibilities or bias. Competing obligations between colleagues, participants, students,
employers, and sponsors are always present in research (American Anthropological Association, 2014).
Conflicts of interest should be addressed as soon as they arise to ensu re they do not negatively affect the
research project.
Conflicts of interest are intrinsic to research in the biological and life science. Conflicts can harm the
research project, collaborators, and participants, but, on a larger sca le, a conflict of interest can damage an
entire research enterprise by reducing the trust and confidence that people generally have in resea rch .
What is Conflict of Interest?
A conflict of interest is any competing obligation or interest that could lead to competing
responsibilities or bias. There are two main types of conflicts of interest: financia l and non-financia l
(Antonuccio, Danton, & McClanahan, 2003; Greenwald, Poehlman, Uhlmann, & Banaji, 2009; Levinsky,
2002) . Financial conflicts of interest include cases in which a researcher serves as a consultant for a
company or receives funding from the industry to conduct research. Nonfinancial conflicts of interest
include cases where a researcher's political, religious, and theoretical beliefs influence publishing or
funding decisions. Although both of types are serious, major funding organizations and other scholarly
research focus on finan cial conflicts of interest.
7
Appendix B. Biological and Life Sciences Module
Prevalence of Conflicts of Interest
Studies on the extent and impact of financial interests in biological and life sciences research have fueled
concern. Financial interests between academic researchers and industry are common and correlate
directly with results that favor sponsors and increased secrecy-scientists refusing to share data with
colleagues, withholding negative data from publication, and delaying publication of research results .
Researchers often maintain relationships with corporations and industries. These relationships should be
disclosed to collaborators and research participants. Corporations and industries, especially those funding
research, may have certain expectations from the research. They may want certain results because it
supports their product and may not want to disclose results that would harm this outcome.
Consider the following examples of researcher-corporation/industry relationships:
•
•
•
Medical schools found that nearly 60% of respondents to a survey had a personal relationship with
industry (Lesser, Ebbeling, Goozner, Wypij, and Ludwig, 2007; Johnson, 2008).
Between 23-28% of academic investigators received research funding from industry, with over 40%
of those surveyed receiving research-related gifts, and about 33% maintaining personal financial
ties to industry sponsors (Bekelman, Yan, and Gross, 2003; Johnson, 2008).
The Association of University Technology Managers (AUTM; 1999) found that 68% of academic
research institutions held equity in companies that in turn sponsored research there (Johnson,
2008).
Guidelines and Policies
Institutions, government agencies, and funding agencies all have guidelines for identifying and disclosing
conflicts of interest. Always review your professional association's and conflict of interest regulations or
guidelines before beginning research. Here are a few examples of guidelines:
Conflict of Interest Guidelines and Policies
•
Conflicts of Interest Policy(NIH)
•
Conflicts of Interest Policies(NSF)
•
ABOR Conflict of Interest Policy (Ari zona Universities)
•
Conflicts of Interest (Elsevier Publishing)
Safeguarding Resea rch Quality and Trust
Currently, most conflict of interest policies are to some extent self-regu latory systems: while federal
regulations require that research institutions monitor, report, and sometimes resolve financial conflicts of
interest, the institutions are free to create their own policies to achieve these goals. Many biological and
life sciences journals require contributors to disclose some or all conflicts of interest, and professio nal
organizatio ns issue guidelines and recommendations. These policies generally rely on individuals to be
honest in their disclosures. A third party, such as university conflict of interest committees or scientific
journal editors, assesses the disclosures and decides whether to prohibit a financial interest, allow it, or
allow it subject to additional measures. Most of these policies apply only to financial interest s held by
individual researchers-few extend to financial interests held by institutions-and many appear to be
inadequately understood, followed, enforced, and assessed.
8
Appendix B. Biological and life Sciences Module
Nevertheless, most commentators-including several committees convened by professional
organizations-remain committed to self-regulation within the biological and life sciences research
community. For self-regulation to successfully safeguard the quality and trustworthiness of research,
conflict of interest policies must be improved. These important questions should be considered when
crafting conflict of interest policies:
•
•
•
•
Whose financial interests should be disclosed, and to whom?
Which interests pose a risk to research quality or trustworthiness?
What are the management options following initial disclosure?
Who decides and who enforces the rules?
Details of conflict of interest policies vary considerably among institutions, journals, professional societies,
and other organizations. Some policies require disclosure of financial interests over $10,000, while others
require disclosure of interests over $25,000. Some policies strongly recommend that a financially conflicted
individual not be involved in human subject research, while others are silent on that question. Some
policies urge public disclosure in all publications and presentations, but others do not specify if or when
public disclosure is necessary.
Addressing Conflicts of Interest with International
Collaborators
Researchers are obligated to identify and disclose all conflicts of interest to the different parties
participating in the research project. Researchers must identify potential and real conflicts of interest,
which may change throughout the course of the research project. Remember, not all countries define
conflicts of interest in the same way. International collaborators will need to discuss what constitutes a
conflict of interest in their home countries or for their funding agencies. Accounting for these
differences, each collaborating researcher will need to disclose and discuss their conflicts of interest at
the start of the project, as well as any that arise during the course of the research.
When conflicts between ethical standards and expectations arise, researchers must explicitly disclose
their ethical obligations and develop a plan to move forward. In their roles as researchers, teachers,
mentors, and experts in their field, researchers are responsible for conducting ethical and transparent
research that does not unfairly benefit one group at the cost of another.
By establishing the scope of the project and the rights and responsibilities of the researchers and
research subjects and disclosing any and all conflicts of interest, researchers can maintain an open and
honest relationship throughout the research project.
Case Study
You started collaborating with a prominent researcher in Canada. You both have contribute funding
from federal agencies in your home countries to the research project, which studies the effect of habitat
size and noise levels on the reproduction rates of an owl population in northern Canada.
Halfway through the data collection, you learn that one of your collaborator' s sponsor is an oil company
that would like to drill for oil in this same area.
9
Appendix B. Biological and Life Sciences Module
You finish data collection and begin to analyze the results. You question the results gathered by your
collaborator, as they differ significantly from the results your research team collected. You contact your
collaborator, and his graduate student sends you the raw data. You discover that some of the data was
not included in the analysis of his results.
Exercise
Write a short paper for discussion, considering the following questions:
•
•
What type of biases and conflicts of interest could your international collaborator have?
Discovering a possible discrepancy in the results and their interpretation, what are you obligated
to do? What should your colleague do?
•
How will you ensure that all conflicts of interest are disclosed and addressed in your upcoming
research project with this collaborator?
10
Appendix B. Biological and life Sciences Module
Data Management: Access, Sharing, and
Exchange
3 Goals
Once you have completed the data management and sharing section of this module, you should:
•
Understand the basics of sound data management and the growing challenges that large data
sets create for data management and archiving
•
Be able to describe the issues that can surround data ownership and how these issues can be
even more complicated in international collaborations
•
Describe several data sharing policies and how you would apply them to your own
collaborations
You will review a case study and answer a few questions after completing this module.
Introduction
The overview section of this course defines data and also outlines some of the major issues of data
management, including data acquisition, data protection, including archiving, and data sharing. In the
biological sciences, these issues are becoming increasingly important for several reasons. First, we
have entered a time of "big data," with increases in the variety of data and the volume and rate of data
acquisition. For example, sensor networks in many parts of the world record a range of environmental
measurements every hour, minute, or even second. Advances in DNA sequencing technology can
result in millions of sequences for a single project. Such " big data" presents numerous opportunities to
address problems at larger scales than previously possible, but it also presents numerous data
management challenges. Secondly, there is increasing pressure on researchers to make data collected
using public funds quickly accessible to other researchers and members of the general public. Meeting
these requirements is not trivial for scientists, and these requirements have led to debates about data
ownership and attribution, as well as more mundane, but no less important, concerns about the
mechanics of data sharing and its associated monetary cost. Large databases containing publicly
accessible data are now available and in use by individuals who played no role in collecting the
data. Lastly, new ways of communicating data can mean that large data sets may be shared and used
by others soon after their generation. Resea rchers can post large data sets on line, widely
disseminating them in the absence of traditional peer-review or the quality control procedures used by
large public databases. This rapid dissemination can provide tremendous opportunities for scientific
advancement, but new mechanisms fo r ensuring data quality may be necessary. International
differences in data policies can further complicate data management, particularly in an era of " big
data," and the opportunities it presents for synthesis and understanding across international
boundaries.
To understand the history of " big data" and the innovations because of it, review this Big Dat a website
that emphasizes how physicist s solved data management and sharing issues as they entered the world
of big data years ago.
11
Appendix B. Biological and Life Sciences Module
Recording and Maintaining Data
Let's think about large, complex data sets within the framework of established guidelines for
maintaining a laboratory or field notebook. These well-established guidelines generally stipulate that
notes on research activity and data will be kept in a waterproof laboratory or field notebook that
includes information on experimental design, data records, and any ensuing calculations or analyses.
Computer print-outs of statistical analyses are typically attached to the lab or field notebook. No
revisions of entries are permitted, although mistakes can be indicated along with a clear explanation in
a new entry. Only certain individuals, all with proper training, are allowed to enter data into a given
lab or field notebook, which tend to be project-based . Lab or field notebooks are copied regularly as
insurance against data loss. Field data sheets are printed on waterproof paper, photocopied, and
entered into an acceptable spreadsheet as soon as possi ble. Paper and digital data are stored in at
least two separate locations for secu rity and redundancy. Data are kept for a period of time; the
specific time frame is determined by the investigator or may be stipu lated by their funding agency. All
notebooks, data sheets, digital data, etc., are kept in the laboratory where they were produced. If an
investigator leaves an institution, copies of the data may be taken with him or her but arrangements
must be made with the institution . This process may be less straightforward if data are in the form of
specimens that have become part of the collections of the institution (e.g., plants, animals, tissues,
DNA) .
Now imagine a "big data" scenario with an international component. You are a researcher conducting
a study of the effects of oil spills on the microbial communities of estuarine environments. Your first
step in understanding the effects of the spill is to compare the microbial species composition of
estuarine environments impacted by oil spills to those nearby that are not impacted . You have the
opportunity to sample sites both internationally and in the United States. You can sample all of the
sites in the U.S., but will need the help of collaborators to obtain the international samples. You elect
to characterize the microbial communities using next generation DNA sequencing, the best new
method available, and have selected a sequencing platform that is most appropriate. As commonly
happens, however, the sequencing technology you need is not available at your institution, and you
must use a facility for which you pay for services by the sa mple. You set up sa mpling kits for your
collaborators with precise instructions, written in English, and all necessary materials. However, some
of the samples are returned in different vials than those you sent, without explanation. You send
samples to the sequencing fa cility, and data files that consist of hundreds of thousands of DNA
sequences of ~300 base pairs in length are se nt to you for ana lysis. The facility also offers data analysis
services for an additional charge, which you take advantage of because you have no experience with
extremely large data sets. Your output from the DNA sequencing facility now includes analyses and
graphs, along with the DNA sequences and a link to a website that describes the protocols used. As
your sa mples were taken from temperate and tropica l areas and estuarine environments that can vary
a lot in salinity, you want to include some environmental measurements, such as water temperature
and salinity in your analysis. You were able to make direct measurements at the sites you visited, but
rely on large government databases for data on the other sites.
Think about the differences betwee n th e standard laboratory notebook and this big data sce nario, as
well as th e potential implications of these differences t o data quality and accountability for the results
of a study.
12
Appendix B. Biological and Life Sciences Module
Archiving Data
As mentioned above, data is often retained for a period of years after collection. Ensuring that the
data remains valuable requires careful archiving. After spending considerable time and energy on a
project, it may seem like you will never forget methodological details. However, after even a few
months, these details can be difficult to recall precisely. Archiving data requires archiving the data
itself (primary data,) as well as information about the data (metadata or secondary data). Michener,
Brunt, Helly, Kirchner, & Stafford (1997) described what metadata for an experiment should include:
1.
2.
3.
4.
5.
6.
Data set descriptors (i.e. abstract, key words)
Research origin descriptors (i.e. Investigators involved, funding sources, objectives,
methods, experimental design)
Data set status and accessibility (i.e., date of last modifications, data use restrictions)
Data structure (i.e. file type, description of response variables, data type)
Supplemental descriptors (i.e. data forms, quality control procedures, publications)
Location(s) of physical specimens (preserved organisms, DNA, etc.)
A great deal of data now exists in digital form. Software changes dramatically over the years,
sometimes rendering data stored in an older format inaccessible in a newer format. Public data
repositories can help solve accessibility problems, but the review the reference below for ways to
effectively manage data for the long-term (Borer, Sea bloom, Jones, and Schildhauer, 2009):
•
Some Simple Guidelines of Effective Data Management
If you need general tips on spreadsheets, databases, etc., check out the Geek Girls website (2013).
Data Sharing and Reuse
The recent increases in our ability to generate large amounts of data in the biological sciences led to
greater expectations for data dissemination and sharing, particularly for research that is publicly funded
by the National Science Foundation or other agencies. An excerpt of NSF' s data dissemination and
sharing policies is shown below. The National Institutes of Health has similar guidelines but specifies a
timeline for sharing in relation to initial publication of research results. Many journals also require that
raw data that contributes to published graphs, tables, etc., be available publicly. Similar policies can be
found internationally, but they vary widely from country to country, with some countries having no
formal policies. Before starting any research in a foreign country or with foreign collaborators, be sure
to review their data dissemination and sharing policies.
From the NSF Award and Administration Guide {2013):
"Dissemination and Sharing of Research Results"
a.
Investigators are expected to promptly prepare and submit for publication, with
authorship that accurately reflects the contributions of those involved, all significant
findings from work conducted under NSF grants. Grantees are expected to permit and
encourage such publication by those actually performing that work, unless a grantee
intends to publish or disseminate such findings itself.
13
Appendix B. Biological and Life Sciences Module
b.
Investigators are expected to share with other researchers, at no more than incremental
cost and within a reasonable time, the primary data, samples, physical collections and
other supporting materials created or gathered in the course of work under NSF grants.
Grantees are expected to encourage and facilitate such sharing. Privileged or
confidential information should be released only in a form that protects the privacy of
individuals and subjects involved. General adjustments and, where essential, exceptions
to this sharing expectation may be specified by the funding NSF Program or
Division/Office for a particular field or discipline to safeguard the rights of individuals
and subjects, the validity of results, or the integrity of collections or to accommodate
the legitimate interest of investigators. A grantee or investigator also may request a
c.
d.
particular adjustment or exception from the cognizant NSF Program Officer.
Investigators and grantees are encouraged to share software and inventions created
under the grant or otherwise make them or their products widely available and usable.
NSF normally allows grantees to retain principal legal rights to intellectual property
developed under NSF grants to provide incentives for development and dissemination
of inventions, software and publications that can enhance their usefulness, accessibility
and upkeep. Such incentives do not, however, reduce the responsibility that
investigators and organizations have as members of the scientific and engineering
community, to make results, data and collections available to other researchers.
e.
NSF program management will implement these policies for dissemination and sharing
of research results, in ways appropriate to field and circumstances, through the
proposal review process; through award negotiations and conditions; and through
appropriate support and incentives for data cleanup, documentation, dissemination,
storage and the like."
Although there is not a unified, international data sharing code of conduct for science generally, several
organizations have drafted guidelines of their own. Read this short paper by Knoppers, Harris, Tasse,
Budin-Uose, Dechenes, and Sawati (2011) in Genome Medicine, in which they outline guidelines for data
sharing among genome researchers internationally.
Compare the guidelines from NSF to the guidelines articulated in the Knoppers et al. (2011) article and
think about questions found in the "Preparing for Our in Class Meeting" section of the module.
A Look at Some Public Data Repositories
Data sharing can be costly and time-consuming, which has led funding agencies to request data
management plans as part of grant proposals. Proposals are expected to include funding for data
management materials and personnel. There are also a growing number of international public
databases, which accept data and associated metadata to be archived and shared. Examples of these
databases include:
•
Dryad
•
GenBank
14
Appendix B. Biological and Life Sciences Module
Take a look at both of these databases and answer the questions about them in the " Preparing for Our
in Class Meeting" section of the module.
Data Reuse
Although many scientists have embraced the idea of data sharing, there is some resistance. Scientists
have objected to the cost in time and money of sharing data, criticisms which funding agencies and
scientific journals are addressing. Scientist s object to providing data to others because they might want
to use the data themselves in the future or because they believe they will not receive credit for their
efforts in data collection. In international collaborations, tensions may arise when data sharing is
required by the funding agencies of some collaborating countries, but not others. Read this article by
Duke and Porter (2013) that reviews issues regarding data sharing and drafts ethical considerat ions for
data reuse, outlining the types of acknowledgment data providers should receive for their
contributions.
What not to do!
Check out this short and humorous video describing a data management and sharing nightmare. See if you
ca n determine all of the data management errors made by the brown bear.
Case Study: Who Owns the Data?
As a student working hard to collect data to test a hypothesis, the answer to this question might seem to
be a simple one -you own the data. However, this might not be the case depending on your project's
funding or where it takes place. Please read this page from the United States Office of Research
Integrity and determine how it applies to your research project.
Now, think about additional complexities that might arise if your project takes place at an international
location or if you have international collaborators who have invested significantly in the project, either
in terms of personnel, funding or both.
Case Study
Student A just completed his PhD and is ready t o take a post-doctoral position at a universit y in another
country. Student A's research was funded by a National Science Foundation (NSF ) grant to his advisor,
Professor X. Professor X used preliminary data from Student A's research to renew her NSF
funding. Student A is excited to pursue a series of follow-up questions that arose out of his dissertation
research during his post-doc. The lab he will join has all of the equipment and software that he will
need to pursue these additional questions and his post -doc mentor is very excited about the
project. Student A is not aware that some of the ideas he would like to follow up on will be pursued in
the coming year by Professor X as part of the new funding she received. As he prepares to leave,
Student A removes his lab books from Professor X's lab and also makes digital copies of all of the data he
has collected. He arranges to have soil samples critica l to his dissertation research shipped t o his new
lab. As he does not have the necessary permits for this in his own name, he mentions his plans to
Professor X, and asks for permission to use her soil permits. Professor Xis very su rprised and tells
Student A that he cannot take the lab books or the soil samples. She reminds Student A that his
dissertation project was funded by the National Scie nce Foundation, and that the resulting data do not
15
Appendix B. Biological and Life Sciences Module
belong to him. Professor X also indicates that the data do not belong to her either. The grant was
awarded to the institution, under the direction of Professor X. Keep this example in mind as you answer
questions posed in the "Preparing for Our in Class Meeting" section of the module.
Preparing for Class
Questions on the data ownership case study:
a. Who owns Student A's data?
b. What can Professor X and Student A do that potentially benefits them both while also
following responsible conduct of science regulations?
2. Questions for discussion about the NSF and Genome Researchers Data Sharing Guidelines:
a. Do the two set s of guidelines have different emphases?
b. Is it easy to know if you are following the guidelines or not?
3. Questions about Dryad and Gen Bank data bases:
a. Do the data bases seem easy to use if you are contributing data?
b. Do the data bases seem easy to use if you are accessing data?
c. Both GenBank and Dryad are international data bases which should provide ways of data
archiving and sha ring for biologists around the world. Do you see any problems with the
way that the database websites are structured that might limit participation by some
nations?
4. What data management mistakes did the brown bear make?
1.
16
Appendix B. Biological and Life Sciences Module
Publication Practices and Responsible
Authorship
Goals
After completion of the section on publication practices and responsible authorship, you should be able
to:
•
Understand the importance of authorship and author order to researchers in the life sciences
•
Understand the criteria fo r authorship and how they might vary internationally
•
Identify strategies for dealing with authorship issues in the context of international collaboration
You will review a case study and complete a few brief exercises after completing this module and in
preparation for the in-person portion of the training.
Introduction
As described in the overview section of this course, the publication of research results is extremely
important. A researcher's published findings have generally gone through a rigorou s peer review
process designed to ensure that the study methods and results are carefully evaluated and critiqued and
provide a significant contribution to the discipline. Publication is also important to the researchers
themselves, as it provides them with acknowledgment from their peers, contributes to their ability to
gain new research funding, and is often necessary for ca reer advancement. The career trajectories of
researchers are assessed based on the number of articles/books they have published, the prestige of the
journals in which they have published, and the number oftimes their research has been cited in articles
written by other scientists. Given the growing trend towards multi-authored works in the biological
sciences, author contribution to articles with large numbers of collaborators is also often considered in
assessing a scientist's productivity. In many cases, the order in which the author's name appears
indicates the author's contribution to the publication . However, this is not always the case in
publications with international authors.
Criteria for Authorship
Individuals should be included as an author on a manuscript submitted for publication if they have made
a significant or substantial contribution to the research. However, significant or substantial
contributions have been defined in a number of ways. For example, a definition of authorship accepted
by many medical journals is that adopted by the Internation al Comm ittee of Medical Journal Editors
(ICMJE) (also known as the Vancouver Protocol). To be included as an author an individual must have
(International Committee of Medical Journal Ed itors, 2014):
1.
Contributed substantially to the conception and design; or acquisition of data, analysis, or
interpretation of data; AND
17
Appendix B. Biological and Life Sciences Module
2. Drafted the article or revised it critically for important intellectual content; AND
3. Approved of the final version to be published; AND
4. Agreed to be accountable for all aspects of the work in ensuring that questions related to
the accuracy or integrity of any part of the work are appropriately investigated and
resolved.
In contrast, the guidelines of Ecological Society of America (2000) are slightly less stringent, stating that
authorship may legitimately be claimed if researchers:
1.
2.
3.
4.
Conceived the ideas or experimental design;
Participated actively in execution of the study;
Analyzed and interpreted the data; OR
Wrote the manuscript
Increasingly, journals are requiring authors to list their contributions to articles. This allows a
researcher's work to be explicitly acknowledged but also facilitates assigning accountability for multiauthored works. Here is an example using initials of the authors:
•
AB, CD, FG and HI designed the study;
•
AB, CD, JK, and LM conducted the field work;
•
CD, JK, LM, and NO conducted the molecular analyses; and
•
AB analyzed the data and wrote the first draft of the manuscript;
•
All authors contributed to revisions.
Based on what you read above, who would be listed as an author on the paper?
Author Order and What It Means
The order in which an author's name appears in a scientific publication can also be important. In
general the first, or lead, author of a manuscript or a grant proposal generally contributes the most to
the work and therefore receives the most credit. In some fields, including the biomedical sciences and
increasingly the environmental and ecological disciplines, the last author, sometimes called the senior
author, also occupies a position of prestige, as they are thought to be critical to the intellectual
development of the research and securing financial support. However, this is not always the case, as
authorship order in multi-authored works can be decided based on relative contribution to the project
(first author contributing the most and last author contributing the least) or alphabetically. Cultures of
author order can vary within a discipline. For example, the position of last author might be a coveted
role for an academic researcher but indicate little contribution to the research for his/her collaborator
who works for a federal government agency. In other countries, the director of the agency or institution
receives the first author position solely based on his or her position, without actually contributing to the
research or the publication process.
Conflicts about author order arise in part because hiring, granting, and promotion decisions are
frequently based on publication productivity; reviewers can be skeptical of the contribution of
researchers whose names are in the middle of a long list of authors. Solutions to the problem of author
order include the development of metrics to objectively quantify author contributions, often providing
each author with a score based on their effort.
18
Appendix B. Biological and Life Sciences Module
To acquaint yourself with how these approaches work, please read the short article by Tscharnke et al.
(2007). This article describes and compares four approaches to assess author contributions and applies
the methods to their own multi-authored opinion piece.
These approaches incorporate the impact factor of the journal in which the article is published, a
measure of journal prestige that is based on the average of how many times articles published in a given
journal are cited by other researchers over a defined time period . To learn more about journal impact
factors, review this Thomson Reuters Impact Factor article (Web of Science, 2014) . To give you some
context, more focused journals are likely to have lower impact factors than journals with very broad
readership. For example, the journal Plant Ecology has a 2012 impact factor of 1.534, while the journal
Nature has a 2012 impact factor of 38.597.
Authorship Issues
Differences among lab groups, disciplines, and other participants in the interpretation of "significant
contributions" or in which formal authorship guidelines are adopted can lead to authorship disputes.
These disputes include those regarding who deserves authorship and the order in which author's names
appear on a manuscript or grant proposal. Some issues can result from two of the types of authorship
described in the Overview section of the course:
•
Gift authorship is the act of acknowledging someone as an author without the individual
actively contributing to the research project. This can be done to gain favor from the individual
or to honor him or her. If the individual is very well regarded in their field, gift authorship may
be given to improve the chances that a manuscript will have an easier time during peer review
or greater impact when it is published. The latter type of gift authorship is sometimes called
prestige authorship.
•
Ghost authorship occurs when an individual makes a substantial contribution to the research
but is not listed as an author. These individuals meet authorship criteria but are not included as
authors on submitted manuscripts.
Some individuals argue that the leaders of large life science laboratories are essentially "gift" authors
because their jobs may be focused on securing funding for the laboratory and representing the work of
the group in national and international venues. They might have little to do with the research on a dayto-day basis. Other individuals counter this argument by pointing out that grant proposals written by
laboratory leaders provide the intellectual framework upon which subsequent work in the laboratory is
based.
Authorship issues are common and not always easy to resolve, even if researchers come from disciplines
that traditionally have had similar authorship practices. Conflicts may arise among international
collaborators if they have no established authorship practices or widely divergent practices in their
home countries. Variation in the structure of labs can contribute to authorship issues. In some
countries, most biological research laboratories are very large, containing multiple researchers with
PhDs (faculty and post-doctoral researchers) along with graduate and undergraduate students,
technicians, and other staff. The lab is lead by a more senior investigator, who also has a PhD. In other
countries, many laboratories, particularly at universities, are smaller, consisting of a lead investigator,
graduate and undergraduate students, and a post-doc and/or technician. Consider the case study in the
next section, which highlights the tensions these kinds of differences can create.
19
Appendix B. Biological and Life Sciences Module
Case Study
At an international meeting, collaboration develops between two investigators, a research faculty
member (Prof X), who is one of several junior faculty in a large European laboratory and an assistant
professor (Prof Y), who is just starting a position at a medium-sized U.S. university. Prof Y sends his PhD
student (Student Z) to the lab of Prof X for three months to learn new laboratory techniques and apply
them to samples she brought with her. Student Z works mostly with Prof X, but also receives help from
others in the large laboratory, particularly Technician A who spends many hours working with Student Z
as she masters the techniques. Student Z attends weekly lab meetings, where she provides suggestions
to other investigators and also receives helpful recommendations on her project from several members
of the group, especially one other research faculty member and a post-doctoral researcher. She only
briefly meets the head of the laboratory, Prof AA, who is away during most of her visit. Student Z gets
regular input on the project by Skype from her mentor back home, Prof Y.
Student Z's visit is very successful and within two months of her return home, she has a manuscript
prepared describing her findings. Her laboratory follows the guidelines of the International Committee
of Medical Journal Editors (ICMJE) to assign authorship and follows the first-last author emphasis norm
for author order (see Tscharnke et al, 2007) . On the manuscript, Student Z lists herself as lead author
and Prof Y as last author, and includes Prof X as second author, and Technician A as third author. The
helpful post-doc and research faculty members are included in the acknowledgments section of the
paper. After receiving significant input from Prof Y, she sends the manuscript to Prof X for comments.
Prof X makes several valuable suggestions, but also adds three names to the author list - the post-doc
and research faculty member who provided helpful suggestions at lab meetings, and the head of the lab,
Prof AA. Technician A is deleted from the author list.
Student Z is perplexed and upset. She describes her concerns to Prof X, along with describing the
authorship policies of Prof Y's lab. Student Z is particularly upset that Prof AA was added as an author.
Prof Xis surprised at Student Z's attitude. Prof AA is always an author on Prof X's manuscripts because
she pioneered many of the laboratory techniques they use and she is the lab's main fund raiser.
Likewise, Technician A is rarely included on manuscripts because he is paid well for his job and is not
usually involved in project design and manuscript preparation.
3 Questions to Consider:
1.
Do you believe there are any gift or ghost authors in this example? If so, who do you believe is a
gift author and who is a ghost author?
2.
Who do you believe should be included as authors and why? In what order would you place
them? Why?
3.
What could the collaborating labs have done to avoid this authorship controversy?
Preparing for the In-class Meeting: A Possible Solution
In addition to answering the question above, read the Weltzin et al. article (2006) for more information
on authorship issues and potential. We will then discuss your answers and the Weltzin article.
20
Appendix B. Biological and Life Sciences Module
Additional Resources
References Cited
Int roduction
•
National Institutes of Health: Office of Extramural Research. (2013). Research Integrity.
Retrieved from : http:/ /grants.nih.gov/grants/ research integrity/whatis.htm.
Collaborative Research
•
•
•
•
•
•
•
•
•
•
•
•
Gadlin, H., & Jessar, K. (2011). Preempting discord: Prenuptial agreements for scientists.
Retrieved from: http://ori.dhhs.gov/preempting-discord-prenuptial-agreements-scientist s.
Holden, C. (2006). Panel clears Schatten of misconduct. Retrieved from
http://news.sciencemag.org/.
lnterAcademy Council. (2012). Responsible Conduct in the Global Research Enterprise .
Retrieved from : http://www.interacademycouncil.net/24026/GlobalReport/28257.aspx.
International long-term ecological research network. (n.d.). Organisation. Retrieved from:
http://www.ilternet.edu/about/ organisation-l.
National Health and Medical Research Council. (2014). National Statement on Ethical Conduct in
Human Research. Retrieved from: https://www.nhmrc.gov.au/book/national-statement-ethicalconduct-human-research.
National Science Foundation. (2012). Science and engineering indicators 2012. Retrieved from:
http:// www.n sf.gov/statistics/seind12/c5/c5s4. htm .
National Science Foundation. (2012). National science foundation scientific indicator report.
Retrieved from: http://www.nsf.gov/statistics/.
Office of Science and Technology Policy. (2000). Federal policy on research misconduct:
Notification of final policy. Federal register, (65)235, 76260-76264. Washington, DC. Retrieved
from: http://www.ostp.gov/html/001207 3. html.
Operational Ecology. (n.d.). Marine Ecosystem Forecasting. Retrieved from: http://marineopec.eu/ documents/kt/OPEC brochure.pdf.
Sang-Hun, C. (2009). Disgraced cloning expert convicted in South Korea. The New York Times.
Retrieved from : http://w ww.nytimes.com/ 2009/ 10/ 27 /world/asia/27clone.html?
ref=hwangwoosuk& r=l&.
Sha moo, A. E., & Resnik, D. B. (2003). Responsible Conduct of Research. New York: Oxford
University Press.
TENK (National Advisory Board on Research Ethics in Finland). (2002). Good scientific practice
and procedures for handling misconduct and fraud in science. Helsinki : TENK Tilburg University.
Retrieved from: http://www.tenk.fi/ en/ advice-publications.
Conflict s of Interest
•
Antonuccio, D.O., Danton, W .G., & Mcclanahan, T.M . (2003). Psychology in the prescriptions
era: Building a firewall between marketing and science. American Psychologist, 58, 1028-1043.
doi: 10.1037/0003-066X.58.12.1028.
21
Appendix B. Biological and Life Sciences Module
•
•
•
•
•
•
•
•
•
Arizona Board of Regents. (2011). Conflict of Interest. Retrieved from :
http:Unau.edu/uploadedFiles/Administrative/Folder Templates/ Forms/3-901-Conflict-ofInterest.pdf.
Bekelman, J.E., Li, Y., and Gross, C.P. (2003). Scope and Impact of Financial Conflicts of Interest
in Biomedical Research: A Systematic Review. Journal of American Medical Association. 289(4),
454-465. doi:l0.1001/jama.289.4.454.
Elsevier. (2014). Conflicts of Interest. Retrieved from: http://www.elsevier.com/journals/lifesciences/0024-3205/guide-for-authors#7001.
Greenwald, A.G., Poehlman, T.A., Uhlmann, E.L., & Banaji, M.R. (2009). Understanding And
Using The Implicit Association Test : Ill. Meta-analysis Of Predictive Validity. Journal of
Personality and Social Psychology._97(1):17-41. Print.
Levinsky, N.G., (2002). Nonfinancial conflicts of interest in research. New England Journal of
Medicine. 5: 759-761. doi: 10.1056/NEJM200212263472617. Retrieved from:
http://www.nejm.org/doi/full/10.1056/N EJM200212263472617.
Johnston, J. (2008) . Conflict of Interest in Biomedical Research. In M. Crowley (Ed), From Birth to
Death and Bench to Clinic: The Hastings Center Bioethics Briefing Book for Journalists,
Policymakers, and Campaigns (pp. 31-34). Garrison, NY: The Hastings Center. Retrieved from
http://www.thehastingscenter.org/uploadedFiles/Publications/Briefing Book/conflict%20of0Ai20
interest%20chapter.pdf.
Lesser, L.I., Ebbeling, C.B., Goozner, M., Wypij, D., and Ludwig, D.S. (2007). Relationship between
Funding Source and Conclusion among Nutrition-Related Scientific Articles. PLOS Medicine, 4(1):
eS. Doi: 10.1371/journal.pmed.0040005. Retrieved from: http:Uwww.plosmedicine.org.
National Institutes of Health. (2014). Financial Conflict of Interest. Retrieved from:
http://gra nts. nih .gov/grants/po Iicy/ co i/.
National Scie nce Foundation. (2005). Grant Policy Manual: Chapter V - Grantee Standards.
Retrieved from: http://www.nsf.gov/pubs/manuals/gpmOS 131/gpmS.jsp#SlO.
Publication Practices and Authorship
•
•
•
•
•
Ecological Society of America Code of Ethics (2000). In Ecological Society of America. Retrieved
from http:Uesapubs.org/esapubs/ethics.htm.
International Committee of Medical Journal Editors. (2014) . Defining the Roles of Authors and
Contributors. Retrieved from: http://www.icmje.org/recommendations/browse/roles-andres pons i bi lities/defining-the-role-of-authors-and-co ntri bu tors. htm I.
Reuters, T. (1994). The Thomas Reuters Impact Factor. The Institute for Scientific Information.
Retrieved from: http://wokinfo.com/essays/impact-factor/.
Tscharntke, T., Hochberg, M . E., Rand, T. A., Resh, V. H., & Krauss, J. (2007). Author Sequence
and Credit for Contributions in Multiauthored Publications. PLoS Biology, 5(1), 13-14.
doi:l0.1371/journal.pbio.0050018. Retrieved from
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1769438/pdf/pbio.0050018 .pdf
Weltzin, J.F., Belote, R.T., Williams, L.T., Keller, J.K.& Engel, E.C. (2006). Authorship in ecology:
attribution, accountability, and responsibility. Frontiers in Ecology and the Environment. 4(8):
435-441. DOI: 10.1890/1540-9295(2006)4[43S:AIEAAA]2.0.C0;2. Retrieved from:
http://www.esajournals.org/doi/pdf/10 .1890/15409295%282006%294%SB435%3AAIEAAA%5D2.0.C0%3B2.
22
Appendix B. Biological and Life Sciences Module
Data Management
•
Borer, E.T., Seabloom, E.W., Jones, M.B., & Schildhauer, M.(2009). Some Simple Guidelines for
Effective Data Management. Bulletin of the Ecological Society of America. 90: 205-214.
Retrieved from: http://www.esa jou rna Is.org/do i/pdf/10 .1890/0012-9623-90.2 .205.
•
Duke, C. S., & Porter, J. H. (2013). The Ethics of Data Sharing and Reuse in
Biology. Bioscience. 63(6), 483-489. Retrieved from:
https://www.dataone.org/sites/all/documents/DukePorter2013.pdf
Dryad. (n.d.). The Repository: Key Features. Retrieved from:
http://datadrvad.org/pages/repository.
Geek Girl's Plain English Computing. (2013). Retrieved from: http://geekgirls.com/.
GenBank. (n.d.). GenBack Overview. Retrieved from: http://www.ncbi.nlm .nih.gov/genbank/
Knoppers, B.M., Harris, J.R., Tasse, A.M ., Budin-Ljose, I., Deschenes, M., & Zawati, M.H. (2011).
Towards a data sharing Code of Conduct for international genomic research. Genome Medicine,
3 (46). Retrieved from: http://genomemedicine.com/content/pdf/gm262.pdf.
Michener, W. K., Brunt, J. W., Helly, J. J., Kirchner, T. B., & Stafford, S. G. (1997, February) . Nongeospatial Metadata for the Ecological Sciences. Ecological Applications, 7(1), 330-342.
Retrieved from http://www.resea rchgate. net/pu bl icatio n/222535033 Metainformation concepts for ecological data management/links/02e7e51e459d7a4380000000.
National Institutes of Health. (2007.) NIH Data Sharing Policy. Retrieved from:
•
•
•
•
•
•
http://gra nts. nih.gov/grants/pol icy/data sharing/.
•
National Science Foundation . (2013). Proposal and award policies and procedures guide.
Retrieved from : http://www.nsf.gov/pu blications/pub su mm.jsp ?ods key=aag.
•
•
NYU Health Sciences Library. (2012, December 19.) Data Sharing and Management Snafu in 3
Short Acts. Retrieved from: https://www.youtube.com/watch?v=N2zK3sAtr-4.
Office of Research Integrity. (2013). Data Ownership. Retrieved from :
http://ori.hhs.gov/Chapter-6-Data-Management-Practices-Data-ownership.
•
TED Ed. (n.d.). Big Data - Tim Smith. Retrieved from: http://ed.ted.com/lessons/explorationo n-the-big-data-fro ntie r-t im-sm ith#review.
Other Resources
Conflict of Interest
•
•
•
•
Christakis, D.A., & Zimmerman, F.J. (2013). Rethinking Reanalysis. Journal of American Medical
Association. 310(23): 2499-2500. doi : 10.0001/jama.2013.281337.
Korn, D. (2000). Conflicts of Interest in Biomedical Research. Journal of American Medical
Association. 284(17): 2234-2237. doi: 10.1001/jama.284.17.22334
Pisano, E.D., Golden, R.N., & Schweitzer, L. (2014). Confl ict of Interest Policies for Academic
Health Systems Leaders Who Work with Outside Corporations. Journal of American Medical
Association.311(11): 1111-1112. doi : 10.0001/jama.2014.788.
Rockey, S.J. & Collins, F.S. (2010). Managing Financial Conflict of Interest in Biomedical Research .
Journal of American Medical Association. 303(23): 2400-2402. doi: 10.001/jama.2010.774.
23