RCN 14 April 2008
Peer review of impact?
Options and challenges
Liv Langfeldt
NIFU STEP studies in Innovation, Research and Education
www.nifustep.no
Impact of research?
 Science impacts
 Organisation impacts
 Technology impacts
 Health impacts
 Economy impacts
 Environment impacts
 Culture impacts
 Symbolic impacts
 Society impacts
 Training impacts
 Policy impacts
(Godin and Doré 2006/OECD 2008)
NIFU STEP studies in Innovation, Research and Education
2
Challenges in assessing impact of research
 Extended effects/combined effects (no linear
model)
 Complex transfer mechanisms/interplay
 National, sectoral and disciplinary spillovers
 Time lag:
 Often both ex ante and ex post peer review assesses
potential, not attained, impact
 What can replace the linear model as bases for
measuring potential impact?
 Who are the peers or experts for assessing extrascientific impact?
NIFU STEP studies in Innovation, Research and Education
3
Impact focus may include a wide variety of initiatives
Examples from the Norwegian context
 Performance based budgeting in higher education and
hospital, to be introduced also for the research institutes
(“Tellekanter”)
 Programme development/formulation (ex ante)
 Project selection (ex ante)
 Programme evaluations
 Costumer surveys on user directed research
 Evaluation of research fields
 RCN projects’ final reports
 Communication of findings/results from projects (RCN
efforts in facilitating impact)
 Foresight studies (ex ante)
NIFU STEP studies in Innovation, Research and Education
4
Performance based budgeting in Norwegian higher education
 Education indicators: about 25 percent of core
funding
 Research indicators: about 15 percent of core
funding
 Doctoral candidates: weight 0.3
 EU research funding: weight 0.2
 RCN research funding: weight 0.2
 Scholarly publications: weight 0.3
 Includes all sources in all fields. Publication sources
divided into two levels giving different scores.
 Annual meetings in each discipline to determine and
revise “level 2” under the guidance of the Norwegian
Association for Higher Education Institutions.
NIFU STEP studies in Innovation, Research and Education
5
Project level impact: Ex ante
 Scientific impact: the typical peer review task
 Project selection: Guessing potential impact based
on past performance and project description
 Different peers may have different
assessments/opinions about impact:
 The likeliness/conditions for impact
 The importance of the potential impact (scientific
value/relevance)
 The relative societal importance of the research topic
(and the legitimacy of such concerns)
 If we do not believe in the linear model and are
unable to separate impacts of specific projects, on
project level ex ante estimates of extra-scientific
impacts are complex/less meaningful.
NIFU STEP studies in Innovation, Research and Education
6
Project level impact
 Links between ex post and ex ante assessments:
 Ex post assessments usually related to general
scientific merits of the group/PI, not the specific
projects.
 Ex post assessments of projects may have little
implications for the evaluees – only one of many
evaluations of their activity
 Publications: basis for assessing results and for impact
on career (awards, positions)
 Extra-scientific impacts have little effect on
academic careers?
 Options
 Link project selection more directly to former
projects’ final reports, programme and/or field
evaluations?
 Study how past extra-scientific impacts influence
project selection? (and vice versa)
NIFU STEP studies in Innovation, Research and Education
7
Programme level impact
 Ex ante impact assessments
 Negotiations including a wide set of stakeholders
defining the programme
 Ex post impact assessments: challenges
 Evaluation at the end of a programme is still mainly
ex ante with regard to impact
 Costumer surveys
 Impacts for a limited group
 Mainly ex ante/potential impact
NIFU STEP studies in Innovation, Research and Education
8
National level impact
 Ex ante assessments of relevance/impact
 Policy formulation
 Budget allocations
 Various stakeholders involved
 Ex post assessments
 Should evaluations of research fields include impact
assessments?
NIFU STEP studies in Innovation, Research and Education
9
Example: Evaluation of Norwegian development research
 ToR: Assess whether the research was utilized
 One of about 13 questions for the international
expert panel
 Sources on extra-scientific use/impact
 Self assessment: “Examples of results/impacts of
your development research”
 Interviews with users
 Conclusion
 The research has policy relevance and a high
proportion of the research is directed at user needs
 Mapped different ways of use
NIFU STEP studies in Innovation, Research and Education
10
Possible reactions to a demand for more impact measurements
 Reframe in impact terms what is already done
 Rethink timeframe for evaluations/the notion of an
evaluation
 Rethink the role of peer panels and other
stakeholders in evaluations
 Separate panels/assessments
 The relevant societal/non-academic impacts of the
research
 The scientific success and excellence of the research
 The connections still the most interesting?
NIFU STEP studies in Innovation, Research and Education
11
Models for performance based funding of research institutions
Panel evaluation,
ranking
Bibliometric models
All disciplines;
all publications
All disciplines;
selected
publications
AUS
GB
N
DK
SF
S
Selected
disciplines; all
publications
Selected
disciplines; ISI
data
B
NIFU STEP studies in Innovation, Research and Education
12