Scientometrics
DOI 10.1007/s11192-010-0333-2
Effect on the journal impact factor of the number
and document type of citing records: a wide-scale study
Juan Miguel Campanario • Jesús Carretero • Vera Marangon
Antonio Molina • Germán Ros
•
Received: 6 July 2010
Ó Akadémiai Kiadó, Budapest, Hungary 2010
Abstract We studied the effect on journal impact factors (JIF) of citations from documents labeled as articles and reviews (usually peer reviewed) versus citations coming from
other documents. In addition, we studied the effect on JIF of the number of citing records.
This number is usually different from the number of citations. We selected a set of 700
journals indexed in the SCI section of JCR that receive a low number of citations. The
reason for this choice is that in these instances some citations may have a greater impact on
the JIF than in more highly-cited journals. After excluding some journals for different
reasons, our sample consisted of 674 journals. We obtained data on citations that contributed to the JIF for the years 1998–2006. In general, we found that most journals
obtained citations that contribute to the impact factor from documents labeled as articles
and reviews. In addition, in most of journals the ratio between citations that contributed to
the impact factor and citing records was greater than 80% in all years. Thus, in general, we
did not find evidence that citations that contributed to the impact factor were dependent on
non-peer reviewed documents or only a few citing records.
Keywords
Journal impact factor Citations Citing records
Introduction
The journal impact factor (JIF) continues to be one of the most widely used scientometric
indicators. It is computed for each year (Y) according to the following equation (Glänzel
and Moed 2002):
J. M. Campanario (&) J. Carretero A. Molina G. Ros
Departamento de Fı́sica, Edificio de Ciencias, Universidad de Alcalá, 28871 Alcalá de Henares,
Madrid, Spain
e-mail: [email protected]
V. Marangon
Facultad de Ciencias Económicas, Universidad de Alcalá, 28807 Alcalá de Henares, Madrid, Spain
123
J. M. Campanario et al.
JIF ðYÞ ¼
Citations in Y to documents published in Y1 and Y2
Citable items published in Y1 and Y2
where Y1 and Y2 are the 2 years before Y. To calculate the numerator of the JIF, ISI
Thompson Reuters counts citations to all types of documents, whereas the denominator
includes only citable documents (Glänzel and Moed 2002; Golubic et al. 2008). A vast
literature exists on the JIF (see, for example, a recent review by Archambault and Larivière
(2009).
In previous studies we introduced the concept of structure of the JIF. By ‘‘structure’’ we
mean the different subsets used to classify citations (Campanario et al. 2006). The
numerator of the JIF equation is a set of citations which can be grouped in subsets
according to different criteria. In previous studies we examined the structure of the JIF of
academic journals from different perspectives. For example, we previously calculated the
percentage of citations that contributed to the JIF and that appeared in articles authored by
members of the journal’s editorial board (Campanario et al. 2006). In other studies we
analyzed the effect on JIFs of citations in documents labeled by Thompson Scientific as
‘‘editorial material’’ (Gonzalez and Campanario 2007; Campanario and Gonzalez 2006)
and the possible use of journal self-citations to increase JIFs (Campanario and Molina
2009; Andrade et al. 2009). In general, we did not find evidence of wide-scale use of
journal self-citations to increase the JIF of the journals we studied.
Here we turn our attention to other factors that could contribute to the structure of JIFs.
For example, consider the case of two journals that receive 10 citations that contribute to
the JIF. Now consider that all citations to the first journal come from a single article,
whereas citations to the second journal come from 10 different articles. We could think that
the true ‘‘impact’’ of the two journals differs. It could be argued that the second journal has
‘‘had an impact on’’ more articles than the first one. This reasoning makes it interesting to
explore differences among journals according to the number and type of citing records that
contribute to the JIF.
Consider now two journals that receive 10 citations that contribute to the JIF. Now
suppose that the first journal is cited only in documents labeled by ISI Thomson Reuters as
articles and reviews, whereas the second one is cited in documents labeled as editorial
material, book review, letter, correction or some other type of document. The citing
documents that cite the first journal are usually peer reviewed and are considered more
relevant than the citing documents that cite the second journal. In most journals, articles
and reviews are peer reviewed and published mainly because of their scientific merit.
However, other documents can be published according to other criteria and goals. Actually,
the idea of the JIF is closer to the notion of impact related to ‘‘scientific’’ origin. In fact, the
denominator of the equation used to compute the JIF includes only articles and reviews. It
seems to us that the contribution to the JIF of citations coming from documents other than
articles and reviews could be counted in a different way.
We should keep in mind that in journals whose JIF is based on a high number of
citations, the influence of any few citations in this scientometric indicator is small.
However, in journals that are not cited very much, the influence of individual citations in
the computation of the JIF may be much greater. For example, citations coming from only
one or two documents or citations coming from documents other than articles and reviews
could boost the JIF to an extent greater than what would be expected for journals that
receive many citations.
123
A wide-scale study
In light of the above considerations, the goals of this study were:
(a)
To introduce a new approach to the study of the structure of JIFs in a wide set of
journals.
(b) To study the relative contribution to the JIF of the number of citing records.
(c) To study the relative contributions to the JIF of citations coming from more researchoriented documents (articles and reviews) versus other types of citing documents.
Method
We selected 700 journals from the JCR database (SCI). Only journals listed (short name) in
JCR in all years from 1998 to 2006 were selected. For each journal, we calculated the sum
of citations received each year, according to data published in the JCR database. Journals
were ranked from lowest to highest based on the total number of citations received. Next,
the first 700 journals listed in the SCI database were selected.
For each journal, we searched all cited records with a publication year from 1996 to
2005. This range allowed us to study all JIFs from 1998 to 2006. We used the Web of
Science (WOS) web interface available to universities in Spain, with the default limits (all
years, all databases).
The WOS screen showing the result of the Cited Reference Search shows a number of
fields for each record. These fields are Select, Cited Author, Cited Work, Year, Volume,
Page, Article ID, Citing Articles and View Record (see Box 1). For each cited record, we
recorded all fields to obtain a Cited Reference field by combining the relevant fields (see
Box 1 for an example). Next, we obtained all citing record citations to each cited reference.
Records were downloaded from WOS without restrictions for year or any other characteristic. This allowed us to identify all citing records of each Cited Reference. We used a
computer program to construct a file in which each cited reference was linked to all its
citing records. We checked that for each Cited Reference, we obtained a number of citing
records equal to the number that appeared in the Citing Articles field. In addition, we
checked that each Cited Reference appeared as a text string in the CR field of their citing
records. The CR field includes all references cited in a given citing record. This method
allowed us to analyze all citations to any given cited reference. After we completed the
above checks, we discarded all citing records with publication years outside the range
studied here (1998–2006).
To avoid spurious results, we deleted from our list all journals with more than one
abbreviated title that could be used to search by cited records. For example, the journal
Box 1 Example of a cited reference search
Original record from the Cited Reference Search screen
Select Cited author
ABDULLAH
MZ
Cited work
Year Volume Page Article ID
T I MEAS
CONTROL
2005 27
65
Citing
articles
DOI 10.1191/ 2
0142331205
tm138oa
View record
View Record
Cited reference (constructed using relevant fields)
ABDULLAH MZ, 2005, T I MEAS CONTROL, V27, P65, DOI 10.1191/0142331205tm138oa
123
J. M. Campanario et al.
Acta Zoologica Academiae Scientiarum Hungaricae is identified by ISI-Thomson Reuters
both as ACTA ZOOL ACAD SCI H and ACTA ZOOL HUNG. The journals excluded for
of this reason were ACTA ZOOL ACAD SCI H, ADV IMAG ELECT PHYS, ADV
POWDER TECHNOL, ASHRAE J, COMPUTATION STAT, FOOD AGR IMMUNOL,
FOOD BIOTECHNOL, POSITIVITY, REAL-TIME IMAGING, STOCH ANAL APPL,
THEOR FOUND CHEM ENG and ZKG INT. A similar problem affected IEE REV, as
explained to us by ISI-Thomson Reuters technical support staff. Changes were made in the
abbreviations of this journal but the changes were not implemented in all records.
We also excluded other journals because of problems with the downloading process or
citation analysis: AUST NZ J STAT, MAR TECHNOL SOC J, ARCH FISH MAR RES,
ICHTHYOL RES, CIENC MAR (some citing records were empty); VERHALTENSTHERAPIE (this journal is included in both SCI and SSCI), FIBER INTEGRATED OPT,
NEURAL PROCESS LETT, ENVIRON ENG SCI, COMBUST THEOR MODEL; J
THEOR PROBAB (some cited records were cited 0 times); INDIAN J PURE AP MAT
(we were unable to process the downloads because of problems caused by citations to a
Cited Reference). For one journal we were unable to find citations that contributed to the
impact factor. This journal is SADHANA-ACAD P ENG S. After these journals were
excluded, the final sample consisted of 674 journals.
Our final file contained a long list of records that each included the data of one cited
reference and one citing document. We discarded all records in which the publication type
was ‘‘C’’ (Congress). In recent years, ISI-Thomson Reuters has started to include meeting
and congress proceedings as a source of records, but we were interested in data from citing
journals only. Citations indexed by ISI-Thomson Reuters as from a congress may not have
been counted in the calculation of the JIF. For example, if the proceedings of a given
congress were indexed recently, citations from these proceedings were probably not
counted in the computation of the JIF of cited journals. Moreover, documents published in
journals tend to be considered more valuable than items published in meeting and congress
proceedings. In addition, in most fields, the peer review process for journals is generally
more stringent that the peer review process for congresses and meetings. However, in some
fields (computer science and physics the peer review process can be also stringent for
congresses and meetings.
We then selected only records corresponding to citations that contributed to the JIF.
These are cited references for which the citing document was published in year Y (for
example, 2004) and the cited reference was published in year Y-1 (2003) or Y-2 (2002).
These records were classified in two categories:
(a)
Citing documents labeled by ISI-Thomson Reuters as an article or review (we labeled
these citations AR)
(b) Citing documents labeled by ISI-Thomson Reuters as other kinds of document (we
labeled these citations OT).
We did not try to reproduce the number of citations used by ISI-Thomson Reuters to
compute the JIF. It is well know that the number of citations used by ISI-Thomson Reuters
to compute the JIF does not always match the number of citations downloaded with the
WOS data. For example, ISI-Thomson Reuters staff told us that when computing the JIF,
they sometimes use manual procedures to delete or add citations (Campanario et al. 2006).
Other authors have noted that the total number of citations extracted from WOS data is
different from the number listed by the ISI in the Journal Citations Reports (Christensen
et al. 1997; Rossner et al. 2007, 2008). Thus, we consider our data as a proxy for the
number of citations that contributed to the JIF.
123
A wide-scale study
For each journal and year studied we calculated the following variables:
CIF: Number of citations that contributed to the JIF.
CIF-AR: Number of citations that contributed to the JIF coming from documents
labeled articles or reviews by ISI-Thomson Reuters.
CIF-OT: Citations that contributed to the JIF coming from other documents.
PCIF-AR and PCIF-OT: The variables CIF-AR and CIF-OT divided by CIF, with the
quotient expressed as a percentage.
CR/CIF: Number of citing records divided by citations that contributed to the JIF (CIF),
expressed as a percentage. This variable has a maximum value of 100 when the number of
citations and the number of citing documents are the same. If CIF is much greater than the
number of citing records, this would suggest that JIF is mainly dependent on citations
received from a small number of records.
These variables allowed us to study the distribution of journals according the contribution to the JIF of citations from articles and reviews versus citations from other documents. We also studied the year-to-year evolution of the contribution of citations from
other documents and the effect of the number of citing records on CIF.
We did not try to compare groups, so no statistical tests for significance were required.
Our basic approach was to study the relationships between changes in the variables.
Results
Distribution of journals in SCI groups
Journals are indexed in JCR in groups. Many journals are indexed in two or more groups.
Table 1 shows the groups that appear more frequently in the set of journals we studied.
Engineering and Biological Sciences are very common in the list.
Distribution of journals according the percentage of articles that they publish
Table 2 shows the distribution of journals according to the percentage of documents
labeled by ISI as ‘‘articles’’ that they publish. Most of journals studied are publishing
research articles.
Distribution of journals according to the contribution to JIF of citations from articles
and reviews and other documents
Table 3 show, for each year, the distribution of journals according the value of PCIF-AR.
In each set, distributions are very similar for all years studied. Citations from documents
other than articles or reviews made relevant contributions to the CIF in only a few journals.
Increase in PCIF-OT with time
We studied the increase in PCIF-OT from one year to the next. Table 4 shows the distribution of journals according to the number of increments in the variable PCIF-OT.
Almost 50% of journals showed only 1 or 2 yearly increases. About 35% of journals had 3
or more yearly increments. However, no clear trend was evident toward large, sustained
increases in the contribution of citations from other documents to the impact factor (CIF).
We found only 27 journals with three or more consecutive increments in PCIF-OT.
123
J. M. Campanario et al.
Table 1 JCR groups in which
journals are indexed
Only groups that appear more
than 10 times are included. Note
that some journals are indexed in
more than one group
Group
Times
Engineering, Electrical & Electronic
49
Veterinary Sciences
31
Materials Science, Multidisciplinary
29
Mathematics
26
Engineering, Mechanical
25
History & Philosophy of Science
21
Computer Science, Theory & Methods
20
Computer Science, Artificial Intelligence
20
Engineering, Multidisciplinary
19
Engineering, Civil
19
Plant Sciences
18
Engineering, Chemical
18
Zoology
16
Telecommunications
15
Food Science & Technology
15
Computer Science, Interdisciplinary Applications
15
Biotechnology & Applied Microbiology
15
Computer Science, Information Systems
15
Sport Sciences
14
Multidisciplinary Sciences
14
Mechanics
14
Mathematics, Applied
14
Computer Science, Hardware & Architecture
14
Neurosciences
13
Chemistry, Multidisciplinary
13
Biochemistry & Molecular Biology
12
Polymer Science
12
Operations Research & Management Science
12
Metallurgy & Metallurgical Engineering
11
Medicine, General & Internal
11
Instruments & Instrumentation
11
Entomology
11
Distribution of journals with values of PCIF-OT higher than 30%
Table 5 shows the distribution of journals with a value of PCIF-OT higher than 30% in
different years. In only a few journals the value of CIF was related to a significant percentage
of citations from documents other than articles or reviews in at least 3 years. In these journals
the JIF depended mostly on citations from documents that, in general, were not peer reviewed.
These journals are indexed in groups related mainly to engineering and medical sciences.
Distribution of journals according to the variable CR/CIF
Table 6 shows the distribution of journals according the value of CR/CIF in each year. The
distributions were similar for all years. In all years, CR/CIF was greater than 80% in about
123
A wide-scale study
Table 2 Distribution of journals
according the percentage of
document labeled by ISI as
‘‘articles’’ they publish
Percent of articles
N Journals
%
10
5
0.7
20
18
2.7
30
22
3.3
40
19
2.8
50
30
4.5
60
52
7.7
70
67
9.9
80
104
15.4
90
153
22.7
100
204
30.3
Total
674
100.0
Table 3 Distribution of journals according to the value of PCIF-AR
Percentage
Year
1998
1999
2000
2001
2002
2003
2004
2005
2006
0–10
0.5
0.2
0.2
0.0
0.6
0.0
0.6
0.0
0.0
11–20
0.0
0.0
0.0
0.2
0.0
0.0
0.0
0.0
0.0
21–30
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.2
31–40
0.2
0.0
0.3
0.0
0.2
0.2
0.0
0.0
0.0
41–50
0.8
0.6
0.6
0.5
0.5
0.8
0.6
0.3
0.3
51–60
0.8
0.5
0.5
0.3
0.3
0.0
0.2
0.6
0.3
61–70
1.5
1.2
1.8
1.2
1.7
2.9
1.5
0.8
0.6
71–80
3.7
2.3
3.3
4.0
4.0
2.4
3.5
2.6
3.2
81–90
10.8
10.4
10.9
9.0
9.6
10.6
9.4
8.6
8.3
91–100
81.7
84.9
82.5
84.9
83.3
83.3
84.3
87.2
87.2
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
Table 4 Distribution of journals
according the number of increments in the variable PCIF-OT
Number of increases
Number of journals
%
0
99
14.7
1
173
25.7
2
163
24.2
3
133
19.7
4
87
12.9
5
16
2.4
6
3
0.4
7
0
0.0
8
0
0.0
674
100.0
Total
123
J. M. Campanario et al.
Table 5 Distribution of journals
with a PCIF-OT higher than 30%
in different years
Number of years
Number of journals
0
570
84.6
1
74
11.0
2
19
2.8
3
6
0.9
4
2
0.3
5
3
0.4
6
0
0.0
7
0
0.0
8
0
0.0
674
100.0
Total
%
Table 6 Distribution of journals according to the value of CR/CIF
Percentage
Year
1998
1999
2000
2001
2002
2003
2004
2005
2006
0–10
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
11–20
0.2
0.0
0.0
0.0
0.0
0.3
0.0
0.0
0.2
21–30
0.2
0.0
0.0
0.2
0.0
0.0
0.2
0.2
0.3
31–40
0.2
0.2
0.3
0.3
0.6
0.6
1.2
0.3
0.5
41–50
0.6
0.6
1.2
0.9
1.4
0.5
0.9
1.2
0.6
51–60
2.2
2.1
1.7
3.2
2.3
0.9
1.5
1.5
1.8
61–70
5.1
6.3
2.0
5.1
5.3
4.7
3.2
4.2
5.0
71–80
10.9
11.2
11.7
10.9
12.0
11.0
10.8
13.3
9.4
81–90
22.2
21.7
26.5
21.1
22.0
22.1
22.7
23.8
24.9
91–100
58.5
57.9
56.7
58.3
56.4
59.9
59.5
55.5
57.5
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
90% of journals. This means that citations were not too concentrated in a given set of citing
records.
Changes in CR/CIF with time
We studied the decrease in CR/CIF from 1 year to the next. This variable represents an
increase in the number of citations that contributed to the impact factor (CIF) per citing
record. Table 7 shows the distribution of journals according to the number of year-to-year
decreases in CR/CIF. The most frequent number of yearly decreases was 3. There was no
clear trend toward large, sustained increases in CR/CIF with time.
Distribution of journals with values of CR/CIF lower than 70%
For each year, we identified journals that had a value of CR/CIF lower than 70%. Table 8
shows the distribution of journals with a given value of CR/CIF in different years. Only 26
123
A wide-scale study
Table 7 Distribution of journals
according the number of decreases in the variable CR/CIF
Number of decreases
Number of journals
0
11
1
61
9.1
2
126
18.7
3
208
30.9
4
182
27.0
5
72
10.7
6
14
2.1
7
0
0.0
8
Total
Table 8 Distribution of journals
in which the value of CR/CIF
was lower than 70% during 3 or
more years
%
1.6
0
0.0
674
100.0
Number of years
Number of journals
0
375
55.6
1
157
23.3
2
60
8.9
3
36
5.3
4
20
3.0
5
19
2.8
6
5
0.7
7
1
0.1
8
1
0.1
674
100.0
Total
%
or 674 journals had a CR/CIF value lower than 70% during 5 years or more. These journals
are indexed in groups related mainly to engineering and life sciences.
Conclusions
The JIF has a great influence on researchers’ publication strategies and decisions. The
study of the structure of JIF should be a strategic research area that could help us to
understand the nature and evolution of scientific impact. Our research suggests a new
perspective on the study of the JIF, and is potentially useful to study the dynamics of
science.
We found that in most journals, the JIF depended on citations coming from documents
labeled by ISI-Thomson Reuters as articles or reviews. As noted above, these documents
are usually peer reviewed. Our results suggest that the JIF was not greatly influenced by
documents that usually are not peer reviewed (for example, editorial material and other
article types). We detected no large changes with time in the sample of journals we studied.
In addition, we found that for most journals, the number of citing records that contributed to the JIF was similar to the number of citations that contributed to the impact
123
J. M. Campanario et al.
factor (CIF). There were no large changes in this pattern during the period from 1998 to
2006. Because our sample included journals that were not frequently cited, any change in
the origin of citations from a small pool of citing records would have been expected to
strongly influence the JIF.
Acknowledgments This work was supported by a grant from the Spanish Ministry of Science and
Technology (Dirección General de Investigación) and the European Regional Development Fund (ERDF/
FEDER, project SEJ2007-66236/SOCI). We thank K. Shashok for improving the use of English in the
manuscript and for suggestions about the content. We also thank an anonymous referee for suggestions.
References
Andrade, A., González-Jonte, R., & Campanario, J. M. (2009). Journals that increase their impact factor at
least fourfold in a few years: The role of journal self-citations. Scientometrics, 80(2), 515–528.
Archambault, E., & Larivière, V. (2009). History of the journal impact factor: Contingencies and consequences. Scientometrics, 79, 635–649.
Campanario, J. M., & Gonzalez, L. (2006). Journal self-citations that contribute to the impact factor:
Documents labeled ‘‘editorial material’’ in journals covered by the Science Citation Index. Scientometrics, 69, 365–386.
Campanario, J. M., Gonzalez, L., & Rodriguez, C. (2006). Structure of the impact factor of academic
journals in the field of Education and Educational Psychology: Citations from editorial board members.
Scientometrics, 69, 37–56.
Campanario, J. M., & Molina, A. (2009). Surviving bad times: The role of citations, self-citations and
numbers of citable items in recovery of the journal impact factor after at least four years of continuous
decreases. Scientometrics, 81(3), 859–864.
Christensen, F. H., Ingwersen, P., & Wormell, I. (1997). Online determination of the journal impact factor
and its international properties. Scientometrics, 40, 529–540.
Glänzel, W., & Moed, H. F. (2002). Journal impact measures in bibliometric research. Scientometrics, 53,
171–193.
Golubic, R., Rudes, M., Kovacic, N., Marusic, M., & Marusic, A. (2008). Calculating impact factor: How
bibliographical classification of journal items affects the impact factor of large and small journals.
Science and Engineering Ethics, 14, 41–49.
Gonzalez, L., & Campanario, J. M. (2007). Structure of the impact factor of journals included in the Social
Sciences Citation Index: Citations from documents labeled ‘‘editorial material’’. Journal of the
American Society for Information Science and Technology, 58, 252–262.
Rossner, M., van Epps, H., & Hill, E. (2007). Show me the data. Journal of Cell Biology, 179, 1091–1092.
Rossner, M., van Epps, H., & Hill, E. (2008). Irreproducible results: a response to Thomson Scientific.
Journal of Cell Biology, 180, 254–255.
123