San Jose State University
SJSU ScholarWorks
Master's Theses
Master's Theses and Graduate Research
2013
Funding sources of impactful and transformative
research
Barrett Anderson
San Jose State University
Follow this and additional works at: http://scholarworks.sjsu.edu/etd_theses
Recommended Citation
Anderson, Barrett, "Funding sources of impactful and transformative research" (2013). Master's Theses. Paper 4374.
This Thesis is brought to you for free and open access by the Master's Theses and Graduate Research at SJSU ScholarWorks. It has been accepted for
inclusion in Master's Theses by an authorized administrator of SJSU ScholarWorks. For more information, please contact [email protected].
FUNDING SOURCES OF IMPACTFUL AND TRANSFORMATIVE RESEARCH
A Thesis
Presented to
The Faculty of the Department of Psychology
San José State University
In Partial Fulfillment
of the Requirements for the Degree
Master of Arts
by
Barrett R. Anderson
December 2013
© 2013
Barrett R. Anderson
ALL RIGHTS RESERVED
The Designated Thesis Committee Approves the Thesis Titled
FUNDING SOURCES OF IMPACTFUL AND TRANSFORMATIVE RESEARCH
by
Barrett R. Anderson
APPROVED FOR THE DEPARTMENT OF PSYCHOLOGY
SAN JOSÉ STATE UNIVERSITY
December 2013
Dr. Gregory J. Feist
Department of Psychology
Dr. Kevin P. Jordan
Department of Psychology
Dr. Mark Van Selst
Department of Psychology
ABSTRACT
FUNDING SOURCES OF IMPACTFUL AND TRANSFORMATIVE RESEARCH
by
Barrett R. Anderson
Understanding how the most important scientific articles have been funded can
help inform and improve future funding decisions. Importance is here defined as science
that, in the metaphor of the tree of knowledge, plays a structurally significant role (e.g.,
creates new branches of knowledge or transforms existing ones). The structural
significance of articles is broken down into two submeasures: citation count and
“generativity” (a novel measure defined as being highly cited and also leading to a
comparatively large number of other highly cited articles). Generativity is an attempt to
provide a quantitative operationalization that should correlate with transformativeness, a
concept that has been used as a funding criterion despite not being well defined. This
report identifies the most impactful and generative publications within a representative
sample of articles indexed in the subject area of psychology in the Thomson ISI Web of
Science in the year 2002. For each of these articles, the funding source was determined,
and comparisons were made between publications that report their funding sources vs.
those that do not, publications funded publicly vs. privately, and publications funded by
various agencies. Publications that reported funding sources were found to be more
generative than those that did not, and research that was privately funded was found to be
more generative than publically funded research. This is consistent with a common
assumption, that public funding agencies are less likely to fund transformative research.
This research is exploratory, and its intent is to lay the foundation for future empirical
investigations into the structure and nature of transformative science.
ACKNOWLEDGEMENTS
This thesis would not have been possible without the support and patience of many
individuals. I would like to thank psychology student Alisha Thomas, for assistance
collecting and coding articles, SJSU librarians Bernd Becker and Emily Chan, for helping
me acquire a better understanding of the Web of Science, and my friend and fellow
student Robert Kutner, for invaluable discussions and assistance at every stage of this
project. I am grateful for the supportive environment at SJSU, and for the dedication of
Dr. Greg Feist, Dr. Kevin Jordan, Dr. Ron Rogers, Dr. Howard Tokunaga, and Dr. Mark
Van Selst. I would not be sane, productive, or complete, without the forbearance and
understanding of Cybil Schroder, the unremitting encouragement of Scott Anderson, the
faith and positivity of Anne Anderson, or the steadfast confidence and support of my
mother, Linda Lees Dwyer. This work is dedicated to my grandfather, David Anderson.
Any curiosity, compassion, or perseverance that I have shown, I owe to his example.
v
TABLE OF CONTENTS
Introduction…………………………………………………………………………
1
Metasciences and Cladistics………………………………………………
2
Transformative Research…………………………………………………
3
Generativity………………………………………………………………..
5
Structural Significance……………………………………………………
6
Research Questions…………………………………………………………
9
Methods…………………………………………………………………………….. 10
Participants…………………………………………………………………
10
Design………………………………………………………………………
10
Procedure…………………………………………………………………… 11
Analysis…………………………………………………………………………….. 17
Descriptive Statistics of Sample……………………………………………. 17
Data Preparation……………………………………………………………. 21
Planned Comparisons………………………………………………………. 25
Exploratory Inferential Analysis……………………………………………
27
Sector and National Origin…………………………………………. 27
Number of Funding Sources…….………………………………….
28
Generativity and Journal Ranking………..………………………… 28
Discussion…………………………………………………………………………..
33
Generativity as a Bibliometric Indicator……………………………………
34
Limitations ……………………..…………………………………………..
36
Future Research…………………………………………………………….. 37
References…………………………………………………………………………..
vi
39
LIST OF TABLES
Tables
Page
1
Generativity citation count thresholds for second-level articles…..........
16
2
Number and of Funding Sources Per Article…………………………… 17
3
Individual Funding Sources Accounting for More Than One Half of
One Percent of the Sample……………………………………………...
19
4
Descriptive Statistics for Generativity and Times Cited………………..
21
5
Structural Significance and Funding Sources ………………………….. 28
6
Journal Ranking for Psychology in 2002 by Mean
Generativity……………………………………………………………..
7
30
Journal Ranking for Psychology in 2002 by Percentage of Generative
Articles...................................................................................................... 31
8
Journal Ranking for Psychology in 2002 by Impact
Factor……………………………………………………………………
vii
32
LIST OF FIGURES
Figures
Page
1
Example Cladogram of Psychology from 1875-Present………….. 3
2
Steps to Calculate Generativity Score……………………………
15
3
Categories of Impactful and Generative Science………………..
16
4
Article Funding Sources by Sector………………………………..
20
5
Article Funding Sources by National Origin……………………...
20
6
Distribution of Generativity before Normalization…...………......
22
7
Distribution of Generativity after Normalization…...…………….
22
Distribution of Times Cited before
23
8
Normalization………………...
9
Distribution of Times Cited after normalization………………...... 23
10
Correlation of Generativity and Times Cited …………………….
24
Correlation of Generativity log 10 (Glog10) and Times Cited log
11
10 (TClog10)……………………………………………………… 24
12
Generativity by Funding Source Sector…………………………... 26
13
Times Cited by Funding Source Sector…………………………...
viii
26
Introduction
Imagine the entire history of human knowledge taking the form of a great tree.
The roots of the tree are deep in the past, and its branches grow up into the distant future.
As the tree grows up through time, the trunk splits into various branches, each of which
defines a new field of knowledge. Rising up into the tree, each of these branches divides
again and again. At first, these changes are easy to follow: natural science splitting off
from philosophy, further divisions defining the early boundaries of physics, geology,
astronomy, and biology. But as times goes on the complexity increases. Sometimes
branches go nowhere (phrenology, astrology), sometimes they are very fruitful (natural
selection, relativity), and sometimes a branch that has long been dormant begins to grow
again (naturalistic decision making). Branches that have for some time grown apart from
each other may begin to grow together again in an unexpected way (astrobiology,
behavioral economics). This complex, fruitful, and many-splendored Tree of Knowledge
describes the history of science.
The tree also describes an ongoing conversation, where ideas combine and build
on those that came before them. The history of science is no less a history of the
individual personalities that contributed to it, but in a way that may be unique among
human endeavors it is possible in science to separate the thought from the thinker. It is
equally valid to describe the history of science as a history of ideas.1 From this
perspective, the body of the Tree of Knowledge is composed of various books,
1
The choice to focus on ideas should not be construed as denying the impact of the individual participants in shaping a particular
course – “Generic eventuality is not equivalent to specific inevitability” (Simonton, 2004).
1
monographs, notes, theses, dissertations, articles, discussions, symposia, conversations,
websites, and emails – all of the physical artifacts and ephemeral moments that the life of
an idea will flow through.
Metasciences and Cladistics
The whole of the tree is too much to take in at a glance. Any hope of
understanding even a small portion of its structure requires a systematic approach.
Depending on the specifics, such an approach might be part of one of the four
metasciences – the history, philosophy, psychology and sociology of science. Any study
of the physical or electronic artifacts that form the body of the tree is a form of
bibilometrics or scientometrics. Recently these fields have also gone by the names
informetrics, webometrics, or cybermetrics (Andrès, 2009; De Bellis, 2009). These
names evidence the increasing technological complexity of scientific communication, but
it would be a mistake to read this variation as reflecting a change in the fundamental
subject of study. This subject, the transmission and measurement of scientific
knowledge, remains the same.
Drawing from the analogical relationship between the Tree of Knowledge and the
biological Tree of Life, a tree that describes the evolutionary relationships between
species, the effort to characterize the structure of the tree can be described metaphorically
as a form of cladistic analysis (Rieppel, 2010). Cladistics is a method of classification
that divides organisms into groups based on common ancestry, called clades. These
clades are the branches of the Tree of Life, and a cladogram is a diagrammatic illustration
of these relationships. By analogy, a cladistic analysis of the Tree of Knowledge would
2
consider the transmission of concepts through communication rather than the
transmission of genes through species.2 An example of a cladogram of a small portion of
the Tree of Knowledge is provided in Figure 1. Such an analysis will be beyond the
scope of the present study, but the cladistic model provides the appropriate context in
which to consider measures of structural significance. These measures will allow us to
identify those important nodes that either begin new branches or transform existing ones.
Put another way, these measures allow us to identify those nodes that significantly impact
the structure of the tree.
Figure 1. Example Cladogram of Psychology from 1875-Present
2
One possible drawback of the tree of life metaphor is that it implicitly downplays the impact of interdisciplinary work. These
collaborations would be metaphorically equivalent to horizontal gene transfer, which in fact does occur in most branches (prokaryotes,
bacteria, and archea) of the tree of life.
3
Transformative Research
A National Science Foundation (NSF) workshop on the meaning and implications
of transformative research in took place in March 2012 (Frodeman & Holbrook, 2012).
Indeed, inspired by a call from US Science Advisor John H. Marburger III, an entire new
funding program began in 2006 at NSF—Science of Science and Innovation Policy
(SciSIP)—whose charge it was to fund science and innovation policy research. Such
research aims to deliver empirical information to policy makers (e.g., politicians, funding
agencies, and administrative scientists) in their effort to make more efficient and
informative decisions about funding science, especially transformative and innovative
science. Moreover, transformational research was added to the NSF merit review criteria
in 2009, but similar concepts (research that is potentially transformative, high-risk,
innovative, or that might in the most favorable cases lead to discoveries that extend to
other fields of science) have been identified as important funding criteria for at least the
last quarter century.
The definition of transformative research has generally been vague (to the point
that defining the term was identified as a goal in the H.R. 5116--111th Congress:
America COMPETES Reauthorization Act of 2010) but always implies the
intensification of change in science. I do not think it would be controversial to contend
that work that starts a new branch of science, or that fundamentally changes an existing
one, should be considered transformative.
Transformativeness as a funding criterion was originally inspired by the concept
of revolutionary science from Thomas Kuhn’s Structure of Scientific Revolutions (1962),
4
which discussed the role of paradigm shifts in scientific progress. In Kuhn’s model,
anomalies that emerge in the course of normal science eventually lead to a crisis, which
can only be resolved by revolutionary science. Revolutionary science defines a new
paradigm that incorporates the anomalies and provides a whole new set of questions for
normal science to ask. According to Kuhn, this revolutionary science is a necessary
consequence of the buildup of anomalies from normal “puzzle-solving” science. Using
Kuhn’s definition of revolutionary science, anything that promotes normal science also
promotes transformative science. One cannot selectively promote transformative science,
in Kuhn’s model, but his definition is not the only one possible. There are other ways to
conceptualize transformative science (as a disruptive innovation, or on a continuum with
normal science), and some of these other perspectives imply it is possible to take a more
interventionist role in its promotion.
Generativity
Regardless of the specifics of the definition, research that is transformative must
necessarily be highly cited. No matter how potentially transformative a work might be in
isolation, that actual transformation has to occur within the social activity of science.
Scientists collaborate, forming teams throughout the process of designing experiments,
conducting research, and presenting their findings. They constantly evaluate each other’s
work at conferences, in peer-reviewed papers, and in grant applications. All of these
interactions provide the context for a scientific culture. To be influential, a potentially
transformative idea has to successfully travel through this culture and take hold in the
minds of the scientists who are participating in it.
5
It is my belief that many of the articles that cite a work of transformative science
will also be highly cited themselves. The transformation of an entire field is more than a
single event, and I suspect that research that is transformative will also be highly
generative. While it may be that not all generative research will be transformative, I hope
that a new measure of generativity will provide a good approximation for objectively
quantifying transformativeness. I am proceeding in the present study under the
assumption that this will be true, with the caveat that at the time of writing a validation of
generativity has not been conducted. Such a validation would require resources beyond
those currently available.
Structural Significance
The purpose of the present study is to identify research that has been structurally
significant in the Tree of Knowledge (i.e., transformative), and to describe how this
research is being funded. Examining the funding of science in the recent past will give us
a sense of how diligent we have been in our custodianship of the tree, with a special focus
on those transformative moments of creativity in which new branches appear.
Understanding how science has been funded can help inform and improve future funding
decisions. The impact of these decisions is broader than just on those who desire a good
return on their investment in science - it also includes every person who lives in a world
that can be transformed by the next big idea. Discussions that will lead to better choices
about the near future of science necessarily begin with an understanding of the recent
past, and these conversations should take place in as empirically grounded a context as
possible.
6
For reasons of familiarity, and to keep the scope in check, the present study will
focus on a small section of the recent past in the field of psychology. This window of
time—in this case chosen to be 2002—should not be so far back that the decisions that
were made then are far from relevant to those being made today, and should not be so
close to the present that the available data is too inconsistent or incomplete. Research
that focuses on the value of science, and especially on creative productivity, tends to use
metrics based on individual publications – the least publishable unit (Simonton, 2004).
And yet, the analysis is generally at the level of the individual scientist. In some cases
the metric rises to the level of journal, institution, or even nation, especially among
sociologists of science. The present study will remain focused on the level of individual
publications. Starting at this lowest possible level avoids unnecessary computational
complexity, which simplifies data collection and analysis. More importantly, the lower
level of complexity prevents unnecessary confusion, providing the most straightforward
example of the novel measure.
Identifying structurally significant work is a substantial challenge. Even an expert
may not be able to immediately identify important work without the benefit of historical
context. While this would appear to argue for only considering older work that already
has a well established place in the history of science, that advantage has to be weighed
against the benefit of providing more current information. Presumably, information about
work that is closer to the present day would be more relevant and useful to a
contemporary decision maker. For this reason, we will choose to rely on imperfect
metrics to provide us with something akin to a first draft of the history of the funding
7
transformative science.
The focus of the present study will be on publications that are structurally
significant to the Tree of Knowledge. These publications are impactful or generative.
Information about references and citations will be necessary to operationalize these
measures of structural significance, and that information is both less ambiguous and more
easily traceable at the level of individual publications. A description of both types of
structural significance under consideration follows:
1. Impactful publications are those that have received a large number of
citations. Many researchers built on the ideas that impactful publications
communicated.
2. A generative publication is one that leads to a new branching point in the Tree
of Knowledge (see Figure 1). Identifying this specific structural impact
requires a broader view than the individual publication. The simplest
description of a generative publication has two requirements, (a) that the
publication is itself is highly cited, and (b) that a large number of those
publications that cite are it are also themselves highly cited.
We will be looking at the most structurally significant publications in the field of
psychology in the year 2002. Specifically, we will be looking at publications that are
more structurally significant than their peers, defining peers as other publications in the
same field, in the same year. This focus on peers is important because the number of
researchers varies between fields, as well as across time (Garfield, 2006; Radicchi,
Fortunato, & Castellano, 2008). It is possible that even with our sample limited to a
8
single field in a single year, more populated subfields will be overwhelmingly
represented simply due to a greater number of publications. If it becomes clear that this
is the case, then a more finely grained distinction between subfields will be called for,
and any analysis will require further subdivision or some form of normalization.
Research Questions
In the process of reviewing the most structurally significant publications for
information regarding their funding sources, it is possible that several comparisons will
present themselves. Two research questions are anticipated:
1. First, is research that reports its funding source more likely to be structurally
significant than research which does not?
There may not always be a straightforward
relationship between funding and quality, but it would be surprising to find anything
other than an overall positive effect of support. Ideally this comparison would be
between funded and unfunded publications, but the funding status of publications that do
not report their funding is necessarily ambiguous. Presumably any publications that do
not report their funding sources, but are structurally significant, are worthy of further
attention.
2. Second, is privately funded research more likely to be structurally significant
than publicly funded research? It may be that highly structurally significant science
(both highly impactful and highly generative) will be less likely to be funded by federal
sources than science with a medium structural significance but more likely than science
with a low structural significance. That is, there may be a curvilinear relationship
between structural significance and federal funding, with science with a medium
9
structural significance being more likely to be federally funded, compared to science with
a high and low structural significance. Within the NIH, transformative research has been
identified as “high risk, high reward research” (Austin, 2008), although there is some
dispute about whether those terms should be synonymous (Frodeman & Holbrook, 2012).
Method
Participants
As this is an archival study, it was not necessary to recruit participants.
Design
The design of this study is an archival one, in which the published literature in the
scientific databases was coded on two characteristics: structural significance and source
of funding. Structural significance is broken down into two quantitative submeasures,
times cited (impact) and generativity. Each article was coded for source of funding in
three ways: funded versus unfunded; public versus private funding entity; and if funded,
name of funding agency. During coding, an additional category for funding sources was
added: domestic (US) versus international. These codings provide categorical
independent variables. The design of the investigation is between subjects ANOVA, with
subjects being research articles from different categories. The dependent variables are
times cited and generativity, which are both continuous. When it is necessary in our
analysis to distinguish between the higher-level categories of funding sources, the public
versus private axis will be labeled Sector and the domestic (US) versus international axis
will be labeled National Origin.
10
Procedure
Thomson ISI Web of Science has been the traditional source for citation data
(Harzing, 2008; Norris & Oppenheim, 2007). Other potentially useful sources have
emerged recently (Meho & Yang, 2007), the most notable of which is Google Scholar.
Although Google Scholar has several advantages, including free availability, high speed,
and broad scope, it is in some ways less useful and less transparent than Thomson ISI.
Google Scholar does not provide (a) the ability to sort results by citation count (b) the
ability to export results, or (c) an API which would allow a researcher to easily develop
solutions to the previous limitations. Google Scholar also does not provide information
about how its database is put together. Although this is an understandable omission for a
proprietary tool, it makes it less useful for this type of study.
Other newer options, such as Altmetrics and Academia.edu, take a fundamentally
different approach to measuring impact, placing additional weight on online interactions.
While many powerful analyses can take advantage of this new type of scientometric data
(Bollen et al., 2009a), neither of these options provides another source of citation data.
The data collection portion of the study consisted of three phases:
Phase one consisted of collecting the top 10 % (by citation count) of the records in
the Thomson ISI Web of Science that match predetermined criteria. These four criteria
are language (English), publication type (peer-reviewed article), date of publication
(2002), and subject area (psychology3). This search resulted in 1774 records. Following
The ISI Web of Science uses two fields to categorize articles by subject, Subject Area and Web of Science Category. The Subject
Areas correspond to thesauri managed by the indexers and editorial staff of Thomson Reuters. Notes that clarify and define the scope
for the various subject areas, which are specific to each index, are available online (http://ip-science.thomsonreuters.com/mjl/scope/).
Web of Science categories are assigned at the journal level. These categories are assigned in the Thomson Reuters Journal Citation
Reports, and carry over to the Web of Science.
3
11
this, we selected a sample consisting of one half of the top 10% of the entire collection
(887 articles). To create this sample we sorted the records by citation count, randomly
selected odds or evens (by coin flip), and included every other article from (and
including) the starting point. Our intent here was to select a random sample in which the
distribution of citation counts very closely or exactly matched the distribution of citation
counts in the top 10 percent.
In phase two, we assigned each of the publications selected in the first phase two
structural significance scores, namely impact and generativity. Impact is simply the raw
citation count, which was already included in all records collected from the database.
Generativity required more effort and was only assigned to records in the sample.
Generative articles are those papers that (a) are highly cited (first order), and that (b)
incite a next generation of research that itself becomes highly cited (second order). More
concretely, generativity is a count of the number of high impact articles that cite a given
high impact article. The steps to calculate a generativity score are outlined in Figure 2
and are:
1. In the first step, a high impact threshold was defined. For the purpose of this
measure, high impact articles were defined as any article in the top 10 % by
citation count of articles published in the same language, the same year and
the same field (defined by Web of Science category).
2. The second step was identifying those first order articles that are above the
threshold defined in the first step. All of the first order articles (i.e., articles in
the sample) necessarily met this threshold. Importantly, this means that only
12
high impact articles (identified as A1 and A2 in the figure) will have any
generativity score at all.
3. The third step was to define a high impact threshold for the second order
articles (the citing articles).
In this case the peers are not the articles in the
initial sample, but other articles that were published in the same language,
year, and field. It is important here to note that the 88,691 second-level
articles ranged across 147 of the 250 Web of Science Categories, and in many
cases more than one category applied to a given article. Although
conceptually an ideal generativity score would include thresholds for all 147
categories, in practice this proved impractical. Fortunately, restricting the
analysis to categories that individually accounted for at least 1% of the sample
identified 13 categories (See Table 1) that together accounted for 80.98% of
the whole. (The initial generativity score, generated only from articles in the
Psychology category, correlated with the final combined generativity score
based on all 13 categories, r = .913, p <.001.)
4. The fourth step was identifying those second order articles that were above the
thresholds defined in the third step.
5. The fifth step was to convert second order articles to numerical values. Any
article that was identified as above the threshold in the previous step (for any
applicable category) should be counted as a one; any article below the
threshold (for all applicable categories) can be counted as a zero.
6. Finally, the numerical values from the previous step are summed for each
13
article, resulting in a positive integer for each high impact article in the
sample. This is the generativity score.
To provide a concrete example (with invented values), we will begin with the
article A1. We will assume that A1 has 268 citations in the Web of Science. A1 is
in our sample and therefore is a first order article. Each of those 268 articles that
cite A1, and all of the other articles that cite articles that are in our sample, are
second order articles (B1-Bmax). We will assume that for the field of psychology
in the year 2002 in the Web of Science that the articles in the top 10 % by citation
count have at least 50 citations. Since A1 has a number of citations equal to or
greater than 50, it does have a generativity score. Next, we generate thresholds
based on all of the second order articles (this will need to be per year and per Web
of Science category). The generativity score is the number of those 268 second
order articles that have citation counts above the appropriate threshold. Of the
268 articles that cite A1 16 have are in the top 10 percent of articles in their year
and in at least one of the categories that they belong to. Therefore A1 has a
generativity score of 16.
14
Figure 2. Steps to Calculate Generativity Score.
15
Table 1
Generativity citation count thresholds for second-level articles.
Year
2002
2003
2004
2005
2006
2007
2008
2009
Behavioral Sciences
54
55
48
43
36
32
25
19
Business
48
44
41
36
29
23
17
12
Economics
48
44
41
36
29
23
17
12
Education & Educational Research
28
26
24
23
20
16
12
10
Family Studies
38
36
38
31
28
23
16
11
Neurosciences
74
69
64
58
48
42
33
25
Pediatrics
42
39
37
33
30
24
19
15
Pharmacology & Pharmacy
50
66
44
40
35
30
25
19
Psychiatry
74
69
66
58
48
42
32
23
Psychology
54
54
50
42
36
30
23
22
Public, Environmental & Occupational Health
52
51
46
41
35
29
23
16
Rehabilitation
38
34
32
31
26
22
17
13
Substance Abuse
52
47
49
40
33
30
23
17
Web of Science Category
As previously mentioned, generativity scores apply only to high impact articles. The
case of a low impact article that is cited by a high impact article might be a case of latent
potential, but it is also possible that the initial article was of only auxiliary utility (See
Figure 3). Articles are cited for a variety of reasons (Bornmann & Daniel, 2008), and not
all citations are created equal.
First Order Impact
Second
Highly Cited
Order
Impact
Not Highly Cited
Highly Cited
Not Highly Cited
Transformative
Science
Latent Potential OR
Auxiliary Contribution
False Start
Ordinary Science
Figure 3. Categories of Impactful and Generative Science.
In phase three each publication in the sample that was collected in phase one was
briefly reviewed. This review served to identify whether a funding source has been
16
reported, and to record the identity of that source. Funding information was gathered
from the article itself. Individual funding sources were categorized as public, if they were
a government funded agency, or private, if not. During this process a second category of
interest emerged, domestic (US) and international funding sources. Each funding source
was also categorized on this criterion.
Analysis
Descriptive Statistics of Sample
The following figures characterize the entire sample, the top 10% of Englishlanguage articles published in Psychology in 2002 and indexed in the Web of Science.
The sample contains 1774 articles from 265 journals. The top 10 journals by count of
articles accounted for about a third of the sample (28.07%). More than half of the articles
(50.45%) were from the top 30 journals.
Out of the half of the sample reviewed for funding source (887 articles), 290
(32.69%) did not list any funding source. Considering only those articles that did list
funding sources, 63.71% listed a single source and 95.89% list 3 or fewer (See Table 2).
Table 2
Number and of Funding Sources Per Article
Number of Funding Sources
Count of Articles
Percentage
1
402
63.71%
2
161
25.52%
3
42
6.66%
4
16
2.54%
5
7
1.11%
6
3
0.48%
631
100.00%
Sum
17
Funding sources that accounted for more than one half of one percent of all funding
sources listed are listed in Table 3. In total, this accounts for slightly more than one half
(56.22%) of all funding sources. The NIH, including those organizations that operate
under it, accounted for 29.92% of the total.
18
Table 3
Individual Funding Sources Accounting for More Than One Half of One Percent of the Sample.
Count
Percentage
Parent Agency
Country
Public
The National Institute of Mental Health (NIMH)
128
13.73%
NIH
US
Public
1.082
3.719
National Institute of Health (NIH)
National Science Foundation (NSF)
62
54
6.65%
5.79%
NIH
US
US
Public
Public
1.050
2.367
1.080
2.282
National institute on Drug Abuse (NIDA)
33
3.54%
NIH
US
Public
Social Sciences and Humanities Research Council of Canada
Medical Research Council (UK)
23
21
2.47%
2.25%
Canada
UK
Public
Public
1.056
1.047
1.678
1.424
1.170
1.193
National Institute on Aging
18
1.93%
NIH
US
Public
0.872
1.504
National Institute of Child Health and Human Development (NICHD)
18
1.93%
NIH
US
Public
1.101
0.979
German Research Foundation (DFG)
17
1.82%
Germany
Private
1.012
1.000
Natural Sciences and Engineering Research Council of Canada
Wellcome Trust
15
15
1.61%
1.61%
Canada
UK
Public
Private
1.098
1.566
1.312
1.076
National Institute on Alcohol Abuse and Alcoholism (NIAAA)
14
1.50%
US
Public
0.986
1.049
WT Grant Foundation
12
1.29%
US
Public
0.953
0.775
Centers for Disease Control and Prevention (CDC)
9
0.97%
US
Public
The John D. and Catherine T. MacArthur Foundation
9
0.97%
US
Private
1.213
1.017
0.331
0.800
Maternal and Child Health Bureau (MCHB)
9
0.97%
US
Public
1.227
0.622
Australian Research Council
8
0.86%
Australia
Public
0.917
0.639
Economic and Social Research Council (UK)
8
0.86%
UK
Private
0.909
1.097
James S. McDonnell Foundation
8
0.86%
US
Private
0.832
1.154
Netherlands Organisation for Scientific Research (NWO)
8
0.86%
Netherlands
Public
1.128
0.245
United States Department of Veterans Affairs (VA)
7
0.75%
US
Public
0.877
0.493
Canadian Institutes of Health Research (CIHR)
National Institute of Neurological Disorders and Stroke (NINDS)
6
6
0.64%
0.64%
Canada
US
Public
Public
1.114
0.770
1.054
0.686
Spencer Foundation
6
0.64%
US
Private
1.243
0.576
Eli Lilly and Co.
5
0.54%
US
Private
0.911
0.313
Royal Netherlands Academy of Arts and Sciences (KNAW)
5
0.54%
Netherlands
Private
0.937
0.042
Total
56.22%
19
NIH
HHS
HHS
NIH
Mean
Generativity
SD
Individual articles with more than one funding source are in some cases funded by a mix
of public and private, or domestic and international sources (See Figures 3 and 4).
Figure 4. Article Funding Sources by Sector
Figure 5. Article Funding Sources by National Origin
20
Data Preparation
The highly skewed nature of citation data necessitated performing a log
transformation before conducting inferential tests (See Figures 5 and 6). Following
convention, base 10 was chosen because it is effective for normalizing skewed
distributions of continuous numerical data (Osborne, 2008). Visual inspection indicates
that normalization of Generativity was successful (Figures 6 and 7), whereas
normalization of Times Cited was more questionable (Figures 8 and 9). The raw values
for Times Cited and for Generativity were strongly and positively correlated (r = .870, p
<.001), as were their log transformations, Times Cited log 10 (TClog10) and Generativity
log 10 (Glog10)(r = .687, p <.001), See Table 4 and Figures 10 and 11.
Table 4
Descriptive Statistics for Generativity and Times Cited
Measure
Mean
Median
Mode
Times Cited
99.99
74
53
Generativity
13.14
10
TClog10
1.93
Glog10
1.03
SD
Skewness
Kurtosis
106.49
12.27
208.99
8
14.49
7.19
92.92
1.87
1.72
0.20
1.78
5.39
1.04
0.95
0.32
-0.12
0.96
Note: TClog 10 = Times Cited log 10, Glog10 = Generativity log 10
21
Figure 6. Distribution of Generativity before Normalization.
Figure 7. Distribution of Generativity after Normalization.
22
Figure 8. Distribution of Times Cited before Normalization.
Figure 9. Distribution of Times Cited after Normalization.
23
Figure 10. Correlation of Generativity and Times Cited
Figure 11. Correlation of Generativity log 10 (Glog10) and Times Cited log 10
(TClog10)
24
Planned Comparisons
Although the current research is exploratory, our inferential analysis was guided
by two research questions: First, is research that reports its funding source more likely to
be structurally significant than research which does not? Second, is privately funded
research more likely to be structurally significant than publicly funded research? (This
second question was simplified from our original intent, which was to determine if there
is a curvilinear relationship between structural significance and federal funding.)
To answer the first question, whether research which reports its funding source
more likely to be structurally significant than research which does not, we conducted an
ANOVA with Glog10 as the DV and Funding Source (Reported, Not Reported) as the
IV. We found that research which reported its funding source (M = 1.043, SD = .315)
was more generative than research which does not (M = .997, SD = .327), F (1,885) =
3.944, p<.05. We repeated this analysis with TClog10 as the DV. The difference was
much smaller, and was not statistically significant (Reported: M = 1.934, SD = .209, Not
Reported: M = 1.930, SD = .190, F (1,885) = .085, p = .771).
The second research question, whether privately funded research is more likely to
be structurally significant than publicly funded research, lead us to conduct an ANOVA
with Glog10 as the DV, and Funding Source (Not Reported, Public, Private, Combined)
as the IV. This analysis indicated that there was a significant effect of Funding Source on
Glog10, F (3,883) = 4.162, p <.01, partial eta2 = .014, See Figure 12. The same analysis,
replacing the DV with TClog10, did not indicate a significant effect, F (3,883) = 2.370, p
= .069, partial eta2 = .008, See Figure 13.
25
Post hoc comparisons with Glog10 indicated that Generativity was greater for
articles with a Private funding source (M=1.126, SD=.292) than for those with a public
funding source (M=1.020, SD=.334), p<.05, and greater for those with a private funding
source than for those whose funding source was not listed (M=.997, SD=.315), p<.01.
Figure 12. Generativity by Funding Source Sector
Figure 13. Times Cited by Funding Source Sector
26
Exploratory Inferential Statistics
Sector and National Origin. The addition of a National Origin categorization
for funding sources lead to the suggestion that there might be a difference in the
relationship between public and private funding between countries. We could expect
because the nature of public funding institutions, both structurally and culturally, might
vary between nations. Coding for National Origin allowed us to test for an interaction
between the effect of Public vs Private funding sources, and the effect of Domestic vs
International Funding sources, for Glog10 and for TClog10 (See Table 5). This was
followed by an ANOVA with Glog10 as the DV, and Funding Source category (Not
Listed, US, International, Combined) as the IV. As when comparing public and private
sources, this analysis was repeated, replacing the DV with TClog10. The first set of tests
did not indicate a significant interaction F (4,886) = 2.028, p = .088, partial eta2 = .009, a
significant main effect of Public vs Private Funding Source, F (2,886) = .870, p = .419,
Partial Eta Squared=.002, or a significant main effect of Domestic vs. International
Funding Source, F (2,886) = .388, p = .678, partial eta2 = .001. In short, in our
exploratory analyses we did not see any interaction, or any effect of Funding Source on
Generativity. Unsurprisingly, the second set of analyses, with TClog10 as the DV also
indicated no significant interaction F (4,886) = .303, p = .876, partial eta2 = .001, no
significant main effect of Public vs Private Funding Source, F (2,886) = 1.338, p = .263,
partial eta2 =.001, and no significant main effect of Domestic vs. International Funding
Source, F (2,886) = .294, p = .745, partial eta2 = .001.
27
Table 5
Structural Significance and Funding Sources
Funding Source
Mean Generativity
SD
SEM
Mean TC
SD
SEM
Not Listed
0.997
0.315
0.018
1.930
0.190
0.011
Public
1.020
0.334
0.016
1.921
0.207
0.010
Private
Combined Public/Private
1.126
0.292
0.032
1.979
0.214
0.024
1.076
0.309
0.034
1.956
0.209
0.023
US
1.030
0.333
0.016
1.937
0.226
0.011
International
Combined US/International
1.025
0.323
0.036
1.917
0.176
0.019
1.127
0.279
0.030
1.957
0.189
0.020
Sum
1.028
0.324
0.011
1.932
0.203
0.007
Number of Funding Sources. During the course of analysis it was suggested
that Generativity might vary based on the number of funding sources, because of the
cautious reception we might expect for transformative ideas from funding institutions.
We found no significant difference in Generativity (Glog10) between research supported
by multiple funding sources (M = 1.060, SD = .309) and research supported by a single
source (M = 1.015, SD = .338), F (1,629) = 2.654 p = .104. The second analysis, with
TClog10 as the DV, also indicated no significant effect (Single Source: M = 1.915, SD =
.217, Multiple Sources: M = 1.944, SD = .188, F (1,629) = 2.770, p = .097).
Generativity and Journal Ranking. Because Generativity is defined at the level
of individual articles, it is possible to create a derivative measure at a higher level, such
as researcher or journal. Simplified examples of such a ranking system, based on mean
Generativity (Table 6), or on the percentage4 of Generative articles (Table 7), are
4
Ideally the percentage used in this ranking would be equal to the number of Generative articles divided by the number of published
articles. In the present example (Table 7) the number of published articles only includes those collected in our sample.
28
provided. It is important to note (1) that these rankings are based only on those journals
that included at least one generative article, and (2) that the rankings are not weighted
based on the number of articles published in each journal. A table of 2002 psychology
journals ranked by impact factor (Table 8) is included for comparison
29
Table 6
Journal Ranking for Psychology in 2002 by Mean Generativity
Rank
Journal Title
1
PSYCHOLOGICAL METHODS
2
3
PERCEPTION
JOURNAL OF THE EXPERIMENTAL ANALYSIS OF BEHAVIOR
4
JOURNAL OF CHILD PSYCHOLOGY AND PSYCHIATRY
5
5
N
Articles
N Generative
Articles
Mean
Citations
Mean
Generativity
28
7
1028.86
116.00
108
64
5
1
103.20
179.00
41.00
40.00
10
2
120.00
33.00
ADVANCES IN EXPERIMENTAL SOCIAL PSYCHOLOGY
7
5
201.60
29.00
JOURNAL OF EXPERIMENTAL PSYCHOLOGY-APPLIED
21
3
155.33
29.00
6
JOURNAL OF ABNORMAL PSYCHOLOGY
65
21
153.95
26.85
7
PSYCHOLOGICAL REVIEW
20
7
161.86
25.50
8
GROUP DYNAMICS-THEORY RESEARCH AND PRACTICE
21
3
153.33
25.33
9
JOURNAL OF AUTISM AND DEVELOPMENTAL DISORDERS
51
14
109.14
25.17
10
JOURNAL OF RESEARCH IN PERSONALITY
39
2
127.50
25.00
11
JOURNAL OF COGNITIVE NEUROSCIENCE
102
48
136.92
23.40
12
DEVELOPMENT AND PSYCHOPATHOLOGY
35
14
109.43
22.40
13
NEUROBIOLOGY OF LEARNING AND MEMORY
66
14
124.64
21.30
14
BRITISH JOURNAL OF DEVELOPMENTAL PSYCHOLOGY
33
1
161.00
21.00
15
PSYCHOPHYSIOLOGY
91
14
108.00
20.50
16
NEUROPSYCHOLOGIA
234
45
98.38
20.05
17
INFANCY
27
1
90.00
20.00
18
PSYCHOLOGICAL ASSESSMENT
43
9
138.78
19.80
19
PSYCHOLOGY OF ADDICTIVE BEHAVIORS
53
5
78.60
19.00
19
PSYCHOTHERAPY AND PSYCHOSOMATICS
36
3
97.33
19.00
20
JOURNAL OF VOCATIONAL BEHAVIOR
45
7
165.86
18.60
30
%
Generative
0.39%
0.28%
0.06%
0.11%
0.28%
0.17%
1.18%
0.39%
0.17%
0.79%
0.11%
2.71%
0.79%
0.79%
0.06%
0.79%
2.54%
0.06%
0.51%
0.28%
0.17%
0.39%
Impact
Factor
1.315
1.314
1.579
2.514
4.7
1.58
3.215
6.75
0.17
2.142
0.905
6.096
4.121
2.417
1.041
2.674
3.184
N/A
2.041
1.432
3.188
1.99
Table 7
Journal Ranking for Psychology in 2002 by Percentage of Generative Articles
Rank
Journal Title
N
Articles
N Generative
Articles
Mean
Citations
Mean
Generativity
%
Generative
Impact
Factor
1
2
JOURNAL OF PERSONALITY AND SOCIAL PSYCHOLOGY
JOURNAL OF CLINICAL PSYCHIATRY
148
206
78
73
109.99
98.38
13.70
12.29
4.40%
4.11%
3.649
4.333
3
JOURNAL OF THE AMERICAN ACADEMY OF CHILD AND ADOLESCENT
PSYCHIATRY
170
57
97.67
16.31
3.21%
4
CHILD DEVELOPMENT
118
53
99.92
12.58
2.99%
5
JOURNAL OF COGNITIVE NEUROSCIENCE
102
48
136.92
23.40
2.71%
6
NEUROPSYCHOLOGIA
234
45
98.38
20.05
2.54%
7
JOURNAL OF APPLIED PSYCHOLOGY
101
43
103.51
16.24
2.42%
8
PERSONALITY AND SOCIAL PSYCHOLOGY BULLETIN
145
38
81.76
10.81
2.14%
9
PSYCHOLOGICAL SCIENCE
95
32
111.94
15.94
1.80%
10
JOURNAL OF CONSULTING AND CLINICAL PSYCHOLOGY
74
31
99.45
16.17
1.75%
11
DEVELOPMENTAL PSYCHOLOGY
73
30
93.97
10.08
1.69%
12
PHYSIOLOGY & BEHAVIOR
265
28
98.32
14.54
1.58%
13
PSYCHOLOGICAL MEDICINE
124
26
110.81
9.27
1.47%
14
HEALTH PSYCHOLOGY
73
23
86.61
9.33
1.30%
15
BEHAVIOUR RESEARCH AND THERAPY
101
22
90.09
14.83
1.24%
15
PSYCHOSOMATIC MEDICINE
81
22
120.95
12.58
1.24%
16
JOURNAL OF ABNORMAL PSYCHOLOGY
16
JOURNAL OF ADOLESCENT HEALTH
17
18
18
18
65
21
153.95
26.85
1.18%
140
21
95.71
9.82
1.18%
JOURNAL OF EDUCATIONAL PSYCHOLOGY
65
20
92.50
12.78
1.13%
COGNITION
67
19
95.63
17.33
1.07%
3.662
3.272
6.096
3.184
1.98
1.758
2.961
3.613
2.496
1.652
2.784
3.5
2.188
3.218
3.215
1.544
0.476
3.099
58
19
87.16
18.45
1.07%
2.514
90
19
95.58
10.90
1.07%
2.335
JOURNAL OF CHILD PSYCHOLOGY AND PSYCHIATRY AND ALLIED
DISCIPLINES
JOURNAL OF EXPERIMENTAL PSYCHOLOGY-HUMAN
PERCEPTION AND PERFORMANCE
31
Table 8
Journal Ranking for Psychology in 2002 by Impact Factor
Rank
Journal Title
N
Articles
N Generative
Articles
3
Mean
Citations
Mean
Generativity
%
Generative
Impact
Factor
1
BEHAVIORAL AND BRAIN SCIENCES
2
TRENDS IN COGNITIVE SCIENCE
N/A
1
N/A
67.00
N/A
2.00
N/A
0.06%
N/A
8.129
8.73
3
ANNUAL REVIEW OF PSYCHOLOGY
N/A
N/A
N/A
N/A
N/A
7.898
4
PSYCHOLOGICAL BULLETIN
13
6
140.17
13.00
0.34%
7.011
5
6
PSYCHOLOGICAL REVIEW
MONOGRAPHS OF THE SOCIETY FOR RESEARCH IN CHILD
DEVELOPMENT
20
N/A
7
N/A
161.86
N/A
25.50
N/A
0.39%
N/A
6.75
6.625
7
PSYCHOLOGICAL INQUIRY
N/A
N/A
N/A
N/A
N/A
6.25
8
JOURNAL OF COGNITIVE NEUROSCIENCE
102
48
136.92
23.40
2.71%
6.096
9
AMERICAN PSYCHOLOGIST
27
3
113.67
5.50
0.17%
5.981
7
5
201.60
29.00
0.28%
4.7
10
ADVANCES IN EXPERIMENTAL SOCIAL PSYCHOLOGY
11
JOURNAL OF CLINICAL PSYCHIATRY
206
73
98.38
12.29
4.11%
4.333
12
DEVELOPMENT AND PSYCHOPATHOLOGY
35
14
109.43
22.40
0.79%
4.121
13
19
6
102.83
15.67
0.34%
4.059
14
COGNITIVE PSYCHOLOGY
JOURNAL OF THE AMERICAN ACADEMY OF CHILD &
ADOLESCENT PSYCHIATRY
170
57
97.67
16.31
3.21%
3.662
15
JOURNAL OF PERSONALITY AND SOCIAL PSYCHOLOGY
148
78
109.99
13.70
4.40%
3.649
16
JOURNAL OF CONSULTING AND CLINICAL PSYCHOLOGY
74
31
99.45
16.17
1.75%
3.613
17
HEALTH PSYCHOLOGY
73
23
86.61
9.33
1.30%
3.5
18
COGNITIVE NEUROPSYCHOLOGY
29
1
55.00
1.00
0.06%
3.391
19
JOURNAL OF EXPERIMENTAL PSYCHOLOGY: GENERAL
32
13
88.15
10.60
0.73%
3.348
20
CHILD DEVELOPMENT
118
53
99.92
12.58
2.99%
3.272
32
Discussion
Generativity was created to be a measure of structural significance (i.e., its
relation to new branches of knowledge) and our findings are consistent with that being
the case. The NIH was, predictably, the largest single funding source for generative
research in psychology. However, almost half of the articles in our sample were funded
by sources that individually accounted for less than one half of one percent of the sample.
Whether those sources were public or private, we saw that a great deal of the funding for
generative research came from a large variety of smaller and more varied sources. When
we look at funding sources by Sector and by National Origin, we also see a great deal
more cooperation between the public and private sectors (and much of that within the
US) than we see between nations.
Our research questions were guided by the assumption that public funding
agencies are more conservative in their funding decisions, and therefore less likely to
fund transformative research. We saw that generativity was greater for those articles that
reported their funding sources than those that did not. The data support this assumption:
generativity varied based on funding source, and it was greater for privately funded
research. We also saw a difference in the same direction for citation count, but it was
smaller and was not statistically significant. This pattern is consistent with the idea that
both generativity and times cited are measures of structural significance but that
generativity is the more sensitive measure. Given our assumptions, our results are
consistent with (1) generativity containing information not provided by pure citation
count, and (2) private funding sources (at least in the US) recognizing and encouraging
33
more generative research than public sources. This should not be construed as implying
that privately funded science has more value than science that is publicly funded. It may
be that private sources are free to pursue riskier ideas only in a context where more basic
science (Kuhn's "puzzle solving" science) is publicly funded.
Generativity as a Bibliometric Indicator
Generativity offers a partial glimpse into the Tree of Knowledge, with unique
advantages over other bibliometric measures. Situated at a level between the immediacy
of pure citation count and the judgment of history, generativity balances the advantages
of perspective with the demands of relevance. In the context of the debate about the
nature of transformative research, a computational measure also has the advantage of
reducing ambiguity, and hopefully may encourage more clarity in the definition of
research that is (and is not) transformative.
The journal impact factor, one of the most widely-used bibliometric measures,
was created to help librarians prioritize journals to include in their collections (Garfield,
2006). Despite this original intent, the measure has since been used to influence
decisions about hiring, promotions, tenure, awarding grants (Meho, 2006; PLoS Medicine
Editors, 2006;), and in some cases even government funding (Adam, 2002; Ferreira,
Antoneli, & Briones, 2013). Journal impact factor is unsuitable for these roles, both
because of a lack of transparency in the measure (Thomson ISI, a private corporation,
alone decides which papers are “citable”), and because it applies at the level of journals
rather than individual contributions. Impact factor has also been criticized for the undue
influence of a small number of highly cited articles (or a large number of uncited
34
articles), the exaggerated impact of review articles, and the limited perspective of a two
year “citation window” (Meho, 2006). There is also reason to believe that reliance on
impact factor underestimates the impact of social science (Hegarty & Walton, 2012). In
the past year, researchers at American Society for Cell Biology published a declaration
decrying journal impact factor’s flaws and abuses, and calling for better research metrics
(San Francisco Declaration on Research Assessment, 2013). As of this writing the
declaration has more than 6000 signatures, but this is by no means the first time that
journal impact factor has been subject to these criticisms (Campbell, 2008; Kurmis, 2003;
Opthof, 1997; Largent & Lane, 2012; Seglen, 1996).
Perhaps because of their accessibility, and certainly in part due to a perception of
objectivity, quantitative measures can be misused. Julia Lane, the former program
director of SciSIP (Science of Science and Innovation Policy) at NSF wrote “Science
should learn lessons from the experiences of other fields, such as business. The
management literature is rich in sad examples of rewards tied to ill-conceived measures,
resulting in perverse outcomes.” (Lane, 2010) This is just as true for bibliometric
measures as it is for IQ, Body Mass Index (BMI), standardized testing scores, or the Dow
Jones Industrial Average. Often critics of these indicators argue that we should rely on
more narrative evaluations, but this is an insufficient response. No bibliometric measure
will ever be a substitute for expert judgment (and generativity is not an exception) but the
cost of obtaining expert evaluation can quickly become prohibitive. It does not scale
well, it is already strongly correlated with many bibliometric measures (Oppenheim,
1996), and despite being a “gold standard”, it is also worth considering that expert
35
judgment itself might require some form of validation (Harnad, 2008). One of the best
answers to abuses of a quantitative measure is to provide a better quantitative measure.
Many bibliometric measures, like the journal impact factor or the h-index
(Hirsch, 2005), are derived from citation data, but are able to achieve a greater degree of
nuance (Cronin & Meho, 2006). Generativity is one such measure, but given its relative
correspondence to raw citation count it may be possible, assuming enough time has
passed for collecting generativity to be feasible, that it could be substituted as the basis
for other citation derived metrics.
The form of generativity that we have explored here is in many ways an
incomplete approximation, limited by time and resources. Although a version of the
measure has been fully specified in this paper, it should not necessarily be understood as
definitive. The core of the concept of generativity – of examining the contribution of
individual articles by looking further down the branches of the tree – can be implemented
in a variety of ways. This could be as simple as varying the threshold for citation counts,
or as complex as basing the measure on the shape of the growth curve of citations. In
either case, the central concept is the same. Present evidence suggests that, on the
spectrum of bibliometric measures (Bollen et al, 2009b), generativity or a measure
derived from it will prove itself to occupy a novel and useful niche.
Limitations
We have only examined the top 10% of articles by citation count (published in
English, in Psychology, and in 2002). This does not provide a picture of the overall
funding situation. Although it may be that what we see at the top is a small-scale version
36
of the whole distribution it is important to recognize that we are looking at the “winners”,
and that greater context could change the interpretation of our findings. Additionally,
although we would argue that a more generative article is a more transformative one, we
recognize that the measure requires validation. Finally, whereas our statistical techniques
are robust against some degree of violation of normality (Howell, 1997), it is possible
that the skewed distribution of citation data renders some of our analysis suspect and in
need of replication.
Future Research
The present study suggests three kinds of future projects, (1) research that focuses
on validating generativity, (2) research that extends or improve on the quality of
generativity, and (3) the development of tools to increase the ease of use of the measure.
We suggest two complimentary approaches to validating generativity. First, if
expert ratings of transformativeness for a sample of articles (which have generativity
scores) could be collected, and compared to citation count, we would expect that
generativity scores would correlate more strongly with expert judgments than pure
citation count. The second validation study would require extending generativity to the
researcher level, so that it could be correlated against measures of lifetime achievement
(awards, honors, etc.). Here generativity could be compared against citation count as well
as a variety of other scientometric measures (h-index, creativity index, etc).
Other future projects could include development of automated tools to ease in the
collection of an even more complete generativity score, and research to fine tune the
measure (varying aspects of the measure such as citation count thresholds) and to extend
37
it to other levels (researcher, journal, society).
Ultimately the judgment of transformativeness belongs to the history of science,
but such a judgment requires a perspective far removed from funding decisions that are
being made today. It is our hope that generativity, or bibliometric measures derived from
it, might provide decision makers with more complete information in an appropriately
timely manner. We also hope that generativity might serve as a foundation for future
empirical investigations into the structure and nature of transformative science
38
References
Adam, D. (2002). Citation analysis: The counting house. Nature, 415(6873), 726-729.
Andrés, A. (2009). Measuring academic research: How to undertake a bibliometric
study. Oxford: Chandos Publishing.
Austin, F. C. (2008). High-risk high-reward research demonstration project. Presentation
given to the NIH Council of Councils, Bethesda, MD.
Bollen J., Van de Sompel, H. V. , Hagberg, A., Bettencourt, L., Chute, R., Rodriguez,
A.A., & Balakireva, L. (2009a). Clickstream data yields high-resolution maps of
science. PLoS ONE 4(3): e4803. doi: 10.1371/journal.pone.0004803.
Bollen, J., Van de Sompel, H.V., Hagberg, A., & Chute, R. (2009b). A principal
component analysis of 39 scientific impact measures. PLoS ONE, 4(6): e6022.
doi:10.1371/journal.pone.0006022
Bornmann, L., & Daniel, H. D. (2008). What do citation counts measure? A review of
studies on citing behavior. Journal of Documentation, 64(1), 45-80.
Bornmann, L., Mutz, R., Neuhaus, C., & Daniel, H.D. (2008). Use of citation counts for
research evaluation: standards of good practice for analyzing bibliometric data
and presenting and interpreting results. Ethics in Science and Environmental
Politics 8, 93–102.
Campbell, P. (2008). Escape from the impact factor. Ethics in science and environmental
politics 8(1), 5-7.
Cronin, B., & Meho, L. (2006). Using the h‐index to rank influential information
scientistss. Journal of the American Society for Information Science and
Technology, 57(9), 1275-1278.
De Bellis, N. (2009). Bibliometrics and citation analysis: from the Science Citation Index
to cybermetrics. Lanham, MD: Scarecrow Press.
Evans, J. A. & Foster, J. G. (2011). Metaknowledge. Science, vol. 331, no. 6018, pp.
721–725. doi: 10.1126/science.1201765.
Ferreira, R. C., Antoneli, F., & Briones, M. R. (2013). The hidden factors in impact
factors: A perspective from Brazilian science. Frontiers in Genetics 4, 130.
39
Frodeman, R. & Holbrook, J.B. (2012). The promise and perils of transformative
research. Report on the workshop: Transformative research: Ethical and societal
implications. Arlington, VA.
Garfield, E. (2006). The history and meaning of the journal impact factor. JAMA-Journal
of the American Medical Association, 295(1): 90-93.
H.R. 5116--111th Congress: America COMPETES Reauthorization Act of 2010. (2010).
Harnad, S. (2008). Validating research performance metrics against peer rankings. Ethics
in Science and Environmental Politics, 8(11).
Harzing, A. W. K., & van der Wal, R. (2008). Google Scholar as a new source for
citation analysis. Ethics in Science and Environmental Politics, 8(1), 62–71.
Hegarty, P., & Walton, Z. (2012). The consequences of predicting scientific impact in
psychology using journal impact factors. Perspectives on Psychological Science,
7(1), 72-78.
Hirsch, J. E. (2005). An index to quantify an individual's scientific research output.
Proceedings of the National academy of Sciences of the United States of America,
102(46), 16569.
Howell, D. C. (1997). Statistical methods for psychology. (4th ed.) Belmont, CA:
Duxbury Press.
Kuhn, T. (1962). The structure of scientific revolutions. Chicago, IL: The University of
Chicago Press.
Kurmis, A. P. (2003). Understanding the limitations of the journal impact factor. The
Journal of Bone & Joint Surgery, 85(12), 2449-2454.
Lane, Julia. "Let's make science metrics more scientific." Nature 464.7288 (2010): 488489.
Largent, M. A., & Lane, J. I. (2012). STAR METRICS and the Science of Science
Policy. Review of Policy Research, 29(3), 431-438.
Meho, L. I. (2006). The rise and rise of citation analysis. arXiv preprint physics/0701012.
Meho L.I., & Yang K. (2007). A New Era in Citation and Bibliometric Analyses: Web of
Science, Scopus, and Google Scholar. Journal of the American Society for
Information Science and Technology 58:1-21.
40
Norris, M., & Oppenheim, C. (2007). Comparing alternatives to the Web of Science for
coverage of the social sciences’ literature. Journal of Informetrics, 1(2), 161-169.
Oppenheim, C. (1996). Do citations count? Citation indexing and the Research
Assessment Exercise (RAE). Serials: The Journal for the Serials Community,
9(2), 155-161.
Opthof, T. (1997). Sense and nonsense about the impact factor. Cardiovascular research,
33(1), 1-7.
Osborne, J. (2002). Notes on the use of data transformations. Practical Assessment,
Research & Evaluation, 8(6).
The PLoS Medicine Editors (2006) The impact factor game. PLoS Med 3(6): e291. DOI:
10.1371/journal.pmed.0030291
Radicchi, F., Fortunato, S., & Castellano, C. (2008). Universality of citation distributions:
Toward an objective measure of scientific impact. Proceedings of the National
Academy of Sciences, 105(45), 17268-17272.
Rieppel, O. (2010). The series, the network, and the tree: changing metaphors of order in
nature. Biology & Philosophy 25:475-496
San Francisco Declaration on Research Assessment. (2013). Retrieved July 23, 2013
from http://am.ascb.org/dora/
Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating
research. BMJ: British Medical Journal, 314(7079), 498.
Simonton, D.K. (2004). Creativity in Science: Chance, Logic, Genius, and Zeitgeist.
Cambridge: Cambridge University Press
41