Measuring Research Output in
Communication Sciences
between international
benchmarks, cultural
differences and social relevance
SAGW-Jahresversammlung 2013
Qualitäts- und Leistungsbeurteilung in den Geistes- und Sozialwissenschaften
Konferenz der Präsidentinnen und Präsidenten
24 May, 2013. Bern
Prof. Dr. Diana Ingenhoff
1
The project on measuring research output in Swiss communication
sciences
  Developped an instrument for the measure of products in Swiss communication
sciences
  Taking into account cultural and linguistic differences, as well as the
diversity of types of products (academic vs. applied)
  Based on the concept of activity profiles as tool to represent
multidimensional patterns of activities  beyond unidimensional rankings
based only on international visibility
  A joint project between University of Fribourg (Communication sciences and
evaluation unit) and University of Lugano.
  In cooperation with the Swiss Communication Sciences Society
  Supported by three leading international experts (Tijssen, CWTS, van den
Besselaar, Rathenau Instituut, Larédo, Université de Paris Est).
Participation of units in communication to the project
University
Institute/Department
University of Bern
IKMB
University of Fribourg
MUKW
Università della Svizzera
italiana
Facoltà di Scienze della comunicazione: all Institutes
University of St.Gallen
MCM (3 chairs)
ZHAW Winterthur
IAM
University of Zurich
IPMZ (divisions Bonfadelli, Jarren, Siegert)
University of Neuchâtel
AJM
  Coverage of the field is very good at least in the university sector
  Resulting in 19 RU (plus three laboratories)
  Probably 80-90% of human resources
  Coverage is quite incomplete in the UAS sector
Media and Communication Sciences: A fragmented field
  highly fragmented and diversified field in terms of parent disciplines, linguistic
communities and research traditions
  Very few relationships between the linguistic regions in terms of careers and
cooperations
  A very careful and disaggregated view is required in order to take into account this
diversity
  The German-speaking universities (including FR)
  Orientation towards social sciences based « Publizistik » (especially ZH and FR)
  Aspects of economics are well represented (especially SG)
  Journalism is not any more central and largely concentrated in FH
  University of Lugano with the only whole Faculty in the field
  More oriented towards English-speaking countries and interpersonal
communication
  French-speaking universities play a rather minor role in the field
  On-going restructuring
  A strong focus on sociology especially in GE
4
The overall picture
  A very small scale-field (150 FTE as compared to 4’300 in SSH)
  Few professors (41 HC) and many PhD students (148 HC)
  High stability of professoral corps and high renewal rate of PhD students
  Very few new professors nominated after 2005
  Emerging stability of a substantial share of the intermediary corps
  Looking to career paths and publication activity
5
Goals of today’s presentation
  Discuss to which extent activity profiles might provide a tool for evaluation of
university institutional units
  discuss advantages and limitations in respect to other quantitative
evaluation tools like rankings
  Show how profiles can be operationalized using readily available indicators
  Present first experiences of usage of profiles by Swiss communication sciences
  Highlight a few implications for the CRUS initiative on measuring the quality of
research, as well as future priorities
Data collection
  From the RU themselves through a set of instruments
  Interviews to head of units
  Personnel survey and individual’s publication lists
  Factsheet with RU basic data and activities
  Analysis of media presence
  The data collected cover all RU activities
  Much more than just publications data
  The strenght of the project is to have collected a set of data on the whole set of
activities of RU:
  Which can be flexibly used to produce different types of analyses
  The price is a high level of complexity
  Both in the data collection and data management process
7
Basic concepts: Research Units and Activity Profiles
  Research as a professional activity related to different context of usage (academia,
society, economy)
  Each with different needs, regulation mechanisms and ways of producing
reputation and wealth
  No single context of usage can be defined a priori as the most important
  Need of context or field-specific measures by dimension
  Institutional units (institutes, chairs) as multi-activity and multi-product organizations
  Combining hererogeneous resources (personnel, infrastructures) to produce
different types of output
  Complex relationships of complementarity between activities
  Hence looking to a single dimension of activity is less meaningful
  Activity profiles are tools to measure and visualize the composition of output and
activities of institutional units
  Comparing them with a benchmark
  In our case the whole field of Swiss communication sciences
Profile of activity
  On average each professor had in 2009
  5 doctoral students
  367 teaching hours (mostly at bachelor and master level)
  25 undergraduate theses (mostly at bachelor and master level)
  9 peer-reviewed journal articles, 16 book chapters and 24 conference
presentations
  Transfer activities are substantial towards the public sector (e.g. 184 reports and 96
presentations), but low towards the private sector (and concentrated in very few RU)
  Data on share of time devoted to activities show that most individuals are engaged
in all activities at the same time (limited specialisation)
  A number of individuals with very low research activities
  Time devoted to services in much higher than in SFSO data and at the same
level as teaching
9
Examples of profiles
Wissenschaft
5.00
4.00
3.00
Transfer privat
2.00
Ausbildung BA
1.00
0.00
-1.00
A
-2.00
B
-3.00
C
Transfer öffentlich
Forschungsausbildung
Ausbildung MA
Weiterbildung
D
Participative design
  There is no «right» set of measures of output in science
  These are socially constructed and different communities will have different
measures depending on their interests and value choices
  the design of the instrument will depend on the objectives and reference
community
  Develop the instrument with the Swiss Society of Communication Sciences
(SGKM)
  Working closely with an expert groups composed by representatives of the
domain
  Validating all choices concerning the project design, dimensions, indicators
  Important for acceptance & cooperation of the units to be analysed
  Construction of indicator systems is both a technical and socio-political process
  Indicator designer as a social mediator
  But technical quality matters nevertheless
Dimensions and indicators
  Research training
  PhD students and theses
  Education
  theses, hours taught and organized by level of education (BA, MA)
  Scientific production
  Funds from public agencies
  Publications
  Scientific awards
  Public transfer
  Contracts
  Reports and presentations
  Media presence
  Private transfer
  Contracts
  Reports and presentations
12
Applications of profiles
  Tools for strategic decision-making for heads of institutional units
  Providing an overview of own position in the field
  Devising «strategies» and negotiating with faculty/university on future activities,
positions, etc.
  Project output: individual RU reports
  Analyzing the position of individual universities in the whole field
  Comparing and different indicators and identifying strenght and weaknesses
concerning the volume of activities
  Project output: university-level reports
  Look to internal differentiation in the field
  And relate it to disciplinary (sub)cultures as well as to different resource
acquisition strategies
  Understand the underlying «production mechanisms» of these units
  Project output: a Swiss-level report on communication sciences (public)
% of journal publications
Two publication cultures
Interpersonal
communication
Mass
communication
% of publications in English
14
Differentiation in funding acquisition strategies
Towards education vs. acquisition of external funds
Total number of educational hours
 
Third-party funding (mio. CHF)
15
Analyzing publication output
  Rather complete data on publication output from individual’s publication lists
  Will allow much for more fine-grained analyses in the next months
  1299 scientific publications (531 articles and 601 book chapters) and 1162
conference presentations in 2005-2009
  A divided patterns between journals and book publications
  With a corresponding pattern in the use of language (English vs. National
languages)
  A very large spread of publication media: 571 journal articles spread over 330
journals
  Can we provide some insights on the characteristics of these journals
16
Diversity in the use of language
17
Skewness of output
  As well-know from science studies most output is produced by few people
  13 out of 73 professors and post-docs account for half of the whole publication
output
  The distribution would probably be even more skewed if looking to international
publications
  Productive people are mostly professors and most RU have just a single highproductive persons (presumably the head of RU)
  14 individuals with more than 30 publications spread over 12 RU
  7 RU don’t have high productive researchers
  What does this mean for evaluation?
  Is RU productivity just determined by the choice of their head?
  Why there are almost no high-productive people at the intermediary level?
  Is is high productivity the result of quality or of exploitation of other’s work?
18
Number of publications 2005-2009 by individual
Profs
Intermediary
PhDs
19
Generalists vs. specialists
  When looking to profiles two broad groups emerge:
  «generalists»: 10 RU whose profile is near to the average of the whole field
  «specialist»: 9 RU which show a distinct specialization in a specific activity
The common core for all units are scientific production and research training (with
the exception of the only UAS-based unit).
  Distinguishing characteristics:
  Specialists are much larger than generalists (11.1 FTE vs. 4.7 FTEs)
  Most specialists in Lugano (5 out of 9), most generalists in Zurich and Sankt
Gallen (7 out of 10)
  Most generalist are in mass communication (7 out of 10), most specialists in
interpersonal communication (5 out of 9)
  Add a couple of examples
20
Generalist vs. Specialists (volume, non normalized)
21
Measures profiles I (volume-based)
22
Measuring profiles II
23
Discussion
 
Profiles provide a flexible tool to highlight the characteristics of institutional units
  Focusing on differences in composition of activities rather than on a single dimension
  Taking into account the specificities of fields and context of usage
  They shed light on interdependences and cumulative effects
 
Accordingly they are quite useful in situations where
  There are ambigous and/or conflicting goals (e.g. research vs. transfer)
  Differentiation and building on each own strength is more important than making the race
on international visibility
 
Profiles can support a dialogic culture inside universities
  Objectivizing the evaluative discussion, but explicitly allowing for differences and
confrontations on goals and strategic decision
  We argue that this is central for a successful university competing externally with other
universities but building on a common culture and cooperation inside faculties
 
The level of acceptance in the project was quite high
  The community in the field made a step towards a more structured evaluation culture
  And was helped to defined its now identify more precisely
Open perspectives and discussion issues
  Integrate in profiles some indicators of quality
  Working on the notion of publication profiles providing a basket of indicators on
different dimensions of the publication activity
  Developing a notion of quality in publications acceptable for SSH
  We will work in this direction in the second phase 2013-2016
  Discuss transferability to other fields
  The framework is well-suited for that, but details need to be worked out
  Support by the relevant stakeholders is required in order to achieve consensus
  How can this be made?
  Develop university central datasets in order to use them for profiling
  Issues of data quality, validation and attribution need to be solved
  Some data will have to be collected at the unit level, but reducing their amount
will be critical for the future of profiles
Thank you for your attention!