Measuring Quality by Quantity
Comments from a
University
Prof. Dr. Hans Weder, President
University of Zurich
1
Comments from a University
The Academic Ranking of World Universities…
• is a good ranking;
• ranks research universities by their research
performance;
• is based on internationally comparable data
(not on mere estimations);
• is published on the web;
• its methodology is continuously to be
discussed.
2
Discussing the Ranking’s Methodology
The questions from which the discussions arise
•
Can the quality of universities be measured by
mere numbers?
•
What are the indicators, how are they defined,
what are the main problems with them, how
can they be improved?
3
Comments on the Indicators
Nobel Prizes
• How to count prizes if they are shared?
• In the case of a migrating scientist: To which
country shall the prize be given?
• Shall they be set in relation to the population of
a country as a whole?
• The rankings can diverge.
• There are more good scientists than prizes.
4
Comments on the Indicators
Fields Medals
• Only mathematics are incorporated.
• How about other fields and other core prizes
and medals?
• E. g., the Balzan Prize (humanities)
5
Comments on the Indicators
Articles published in Nature and Science
• Interesting, but highly selective results
• Expanding the list of core journals?
6
Comments on the Indicators
Science Citation Index-expanded and Social
Science Citation Index
• For many subjects (e. g., humanities) the
period of time taken into account is too short.
• The time window differs very much from the
time windows for the other indicators.
• It is used what is available.
7
Comments on the Indicators
How about peer review?
• It is mostly used for ex ante evaluation of
manuscripts.
• For university rankings: Quality can be
measured by combining bibliometric analyses
and peer review.
8
Bibliometric Analyses and Peer Review
Post-publication filtering and evaluation:
Faculty of 1000
• Focusing on the best papers regardless of
the journal in which they are published.
• Redefining core journals within faculties?
10
Faculty of 1000
Table 2 Redefining ‘core’ journals within Faculties?
Cell biology
Chemical
biology
Nat Struct
Biol
Chem Biol
Neurobiology
Plant
biology
Nat Neurosci
Plant Cell
Cell
Neuron
Plant J
Nat Struct Biol
Cell
Cell
Mol Cell
Nat Med
Nature
Cell
Science
Science
Nature
J Exp Med
Nat Med
Nature
Science
Nat Cell Biol
Development
J Clin Invest
Mol
Microbiol
J Neurosci
Plant
Physiol
Cell
Nature
Science
Nat
Biotechnol
Genetics
J Immunol
PNAS
Nat Genet
PNAS
EMBO J
J Cell Biol
Developmental
biology
Genomics &
Genetics
Immunology
Cell
Cell
Immunity
Dev Cell
Nat Genet
Development
Science
Nat
Immunol
J Exp Med
Nat Cell Biol
Nature
Plant Cell
Science
Nature
Mol Cell
Genes Dev
Genome Res
Plant Cell
1
Cell
2
Nat Cell Biol
3
Dev Cell
4
5
6
7
Nature
Science
Mol Cell
J Cell Biol
Nat
Biotechnol
Science
Nature
Cell
Mol Cell
8
Nat Immunol
Structure
Curr Biol
9
Immunity
PNAS
Genes Dev
10
Plant Cell
JACS
Nat Genet
Am J Hum
Genet
EMBO J
Microbiology
Nat
Immunol
Immunity
Genome
Res
Curr Biol
Structural
biology
Nat Cell Biol
Science
Nature
Nat Biotechnol
Chem Biol
Structure
EMBO J
Journals are ranked within eachFaculty according to the percentage of papers selectedfor Faculty of 1000relative to their total
conte (i.e. a ‘normalized’ perspective). Data calculated for papers added to the site between Aug 2001 and May 2003.
Source:
Wets / Weedon / Velterop. (2003) Post-publication filtering and
evaluation: Faculty of 1000. Learned Publishing 16(4), 249-258.
11
Examining quality at the University of Zurich
• Integrated evaluation approach
• Informed peer review
12
Examining Quality at the University of Zurich
Evaluation objectives
• Assess, assure, and improve the quality of
academic work in research, teaching, and
services as well as assure the quality of
management and administration
• Provide decision aids to support medium and
long-term strategic planning
• Report to the public (accountability)
13
What is Evaluated?
Units
• Academic area: Departments, degree programs,
institutes, sections, clinics, faculties
• Executive board of the university, university
administration
Fields of activity
• Research, teaching, services, promotion of young
academics / scientists, management and
administration
General conditions
• Structures, surrounding fields / cooperation (e.g., ETH
Zurich)
• Resources
14
Overview of the Evaluation Process
Informed
peer review
Follow-up / goal
agreement
(implementation of
results)
Monitoring of
implementation of
goal agreement
Re-evaluation
15
Informed peer review
Evaluation Procedures
Self-evaluation
• With report prepared by the unit under evaluation
Surveys / analyses
• Conducted by the evaluation office
External evaluation
• With site visit and report prepared by the team of experts
and statement of response by the unit under evaluation
Comprehensive report
• Prepared by the evaluation office; statement of response
by the unit under evaluation
Follow-up / goal agreements
• By the executive board of the university
Monitoring of implementation of goal agreements
• By the evaluation office
16
Thank you for your attention.
17