Scientometrics (2011) 88:495–498
DOI 10.1007/s11192-011-0382-1
Severe language effect in university rankings:
particularly Germany and France are wronged
in citation-based rankings
Anthony F. J. van Raan • Thed N. van Leeuwen • Martijn S. Visser
Received: 25 December 2010 / Published online: 13 April 2011
Ó The Author(s) 2011. This article is published with open access at Springerlink.com
Abstract We applied a set of standard bibliometric indicators to monitor the scientific
state-of-arte of 500 universities worldwide and constructed a ranking on the basis of these
indicators (Leiden Ranking 2010). We find a dramatic and hitherto largely underestimated
language effect in the bibliometric, citation-based measurements of research performance
when comparing the ranking based on all Web of Science (WoS) covered publications and
on only English WoS covered publications, particularly for Germany and France.
Keywords
Bibliometric analysis University rankings Language of publication
The WoS data system covers a number of journals in non-English languages, particularly
in German and in French. Publications in these non-English language journals are counted
as part of the output of countries, but they generally have a very low impact because only a
few scientists outside Germany, Austria and Switzerland are able to read German; we
have a similar situation for French. Thus, these non-English publications will ‘dilute’
considerably the average impact of countries such as Germany, Austria and France. This is
particularly the case for the more application-oriented fields such as clinical medicine and
engineering, and also for the social sciences and humanities. Because clinical medicine
represents a considerable part of the entire scientific output of a country (particularly in
western countries), this language effect directly influences the overall impact position of
universities (van Leeuwen et al. 2001).
This effect has serious consequences for all citation-per-paper based rankings, but also
for citations per staff rankings. The most striking result is a too low position of most
German and France universities in these rankings, particularly universities with a medical
school. As citations play an important role in university rankings such as the Times Higher
Education, QS, Shanghai (ARWU) and the Leiden Ranking, all these rankings suffer from
this language effect.
A. F. J. van Raan (&) T. N. van Leeuwen M. S. Visser
Centre for Science and Technology Studies (CWTS), Leiden University, Leiden, The Netherlands
e-mail: [email protected]
123
496
A. F. J. van Raan et al.
Thus, for ranking and comparison with countries such as the US, UK and all countries
where almost all WoS-covered publications are written in English, calculation of bibliometric indicators based on only-English publications is a more fair procedure.
We show results in Fig. 1 for in total 69 German and French universities. We see that
the ‘English-only’ citation impact is systematically higher than the ‘all publications’
Citation Impact German and French universities
All versus English-only Publications
2,00
1,80
1,60
imp-eng
1,40
1,20
y = 1,08x
2
R = 0,81
1,00
0,80
0,60
0,40
0,20
0,00
0,00
0,20
0,40
0,60
0,80
1,00
1,20
1,40
1,60
1,80
2,00
imp-all
Citation Impact Ranking German and French universities
All versus English-only Publications
500
450
400
r-eng
350
300
y = 1,02x - 38,31
2
R = 0,83
250
200
150
100
50
0
0
50
100
150
200
250
300
350
400
450
500
r-all
Fig. 1 Language effect in citation impact of all German and French universities. The upper figure shows
the difference in impact (numerical values of the field-normalized citation score) in the case of all WoS
(imp-all, horizontal axis) and only-English WoS publications (imp-eng, vertical axis). The lower figure
shows the difference in citation impact ranking position in the case of all WoS (r-all, horizontal axis) and
only-English WoS publications (r-eng, vertical axis)
123
IV
H
EI
U
N DE
U IV L B
N
IV MU ER
F N G
U RA CH
N
IV NK EN
E FO
R
T RL
U U M AN T
N
G
I
U
E
V
U
N GO NC N
IV
H
D TT EN
U
KA
S S IN G
R
E EN
LS
U LD
N
R
IV OR
U
U
H
N
M F
A
M ER IV
E D I N A IN
AC Z
S
U
T
H
N
IV F T EN
U
N HA ECH
IV N
H
W NO N
U
U
M
R VER
BO
Z
LD U BU
T NIV RG
U
B
U NIV ON
N
IV B E N
FR R
EI LIN
B
U UR
N
IV G
U
K
U
N NIV IE
U IV P K L
N
O
IV OT LN
D
O SD
A
R
TM M
U
U UN
N
N
U
N IV I V D
IV
H
U
R A M LM
E
U
N GE BU
IV N R
G
U
N ST SBU
IV
U
SA TT RG
G
U AR A
R
N
IV LA T
TU ND
BI ES
N
G
EN
U
N
ranking
IV
H
E
U ID
N
E
U IV L B
N
IV MU ER
F N G
U RA CH
N
IV NK EN
E FO
R
T RL
U U M AN T
N
G
I
U
E
V
U
N GO NC N
IV
H
D TT EN
U
KA
S S IN G
E EN
R
LS
U LD
N
R
IV OR
U
U
H
N
M F
A
M ER IV
ED IN A IN
AC Z
S
U
T
H
N
IV F T EN
U
N HA ECH
IV N
H
W NO N
U
U
M
R VER
BO
Z
LD U BU
T NIV RG
U
B
U NIV ON
N
IV B E N
FR R
EI LIN
B
U UR
N
IV G
U
K
U
N NIV IE
U IV P K L
N
O
IV O T L N
D
O SD
A
R
TM M
U
U UN
N
N
U
N IV IV D
IV
H
U
R A M LM
E
U
N GE BU
IV N R
G
U
N ST SBU
IV
U
SA TT RG
G
U AR AR
N
IV LA T
TU ND
BI ES
N
G
EN
U
N
impact
Severe language effect in university rankings
1,40
497
1,60
Citation Impact 25 German universities
impact english only
impact all
1,20
1,00
0,80
0,60
0,40
0,20
0,00
350
Citation Impact Ranking 25 German universities
ranking english only
300
ranking all
250
200
150
100
50
0
Fig. 2 Language effect in citation impact of the 25 highest impact German universities. The upper figure
shows the difference in impact (numerical values of the field-normalized citation score) in the case of all
WoS and only-English WoS publications. The lower figure shows the difference in citation impact ranking
positions in the case of all WoS and only-English WoS publications
123
498
A. F. J. van Raan et al.
impact, and the ranking position based on ‘English-only’ citation impact is systematically
lower (thus, ‘better’) than based on ‘all publications’ impact, with a few exceptions due to
differences in inter-group changes of ranking positions. In Fig. 2 the language effect is
shown in a slightly different way for the 25 German universities with the highest citation
impact. A comparison of the rankings for the 500 largest universities in the world based on
all WoS publications as well as on English-only publications is given in our ranking
website (Leiden Ranking 2010).
In the above context, we focus on the language effect of, particularly European universities in the German and French language areas, but a similar effect will also be present
for Italian and Spanish universities. The issue of publication language is related to the
problem of ‘local journals’ (but covered in the Web of Science). These journals play only a
marginal role in the scientific communication process at an international scale. The publications in these journals mainly transfer knowledge to local or national end-users. For
instance, medical knowledge transferred to non-academic hospitals to improve clinical
practice. In the technical sciences we find transfer of new knowledge to the engineering
practice. As most fields contain local journals, these journals should ideally be excluded
from the sets forming the basis for bibliometric studies in international comparison. This is
not a simple task and it requires further bibliometric studies.
One can expect that similar language effects will occur when using Scopus data and that
this effect will seriously aggravate by the extension of the coverage of the WoS and Scopus
with national language journals.
Open Access This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium,
provided the original author(s) and source are credited.
References
Leiden Ranking (2010). http://www.socialsciences.leiden.edu/cwts/products-services/leiden-ranking-2010cwts.html.
van Leeuwen, T. N., Moed, H. F., Tijssen, R. J. W., Visser, M. S., & van Raan, A. F. J. (2001). Language
biases in the coverage of the science citation index and its consequences for international comparisons
of national research performance. Scientometrics, 51(1), 335–346.
123