June 2010
Demonstrating the Impact of Research:
Institutional Responses to
Policy Change
Dr Andrew Walsh
Head of Research Policy
Why this new focus on impact?
• More than a decade of greatly enhanced public investment
in research
• UK science budget has doubled in real terms from £1.3
billion in 1997-98 to almost £4 billion by end of 2010-11
• Rationale? R&D as a public good
increasing the stock of useful knowledge;
2. training skilled graduates;
3. creating new scientific instrumentation and methodologies;
4. forming networks and stimulating social interaction;
5. increasing capacity for scientific and technological problemsolving;
6. creating new firms.
(The Relationship with Between Publicly Funded Basic Research and
Economic Performance: A SPRU Review, HM Treasury 1996)
1.
What do we mean by impact?
“A policy action has an economic impact when it affects
the welfare of consumers, the profits of firms or the
revenue of government… But the economic impacts of
science and innovation are much more extensive than
what can be captured by data on economic growth and
productivity. The welfare and quality of life for
consumers may be enhanced in a number of ways,
including improved health and longevity; improved
social outcomes; a clean, green and safe environment
which supports the essentials of life: air, land, water
and food; a safe and stable political environment and
the maintenance of national security.”
(Measuring economic impacts of investment in the
research base and innovation: a new framework for
measurement, DTI, 2007)
Breadth of impact
Economic
Social
Quality of life
Types of impact
Public policy &
services
Environment
Cultural
Health
Issues in Assessing Impact
• Definition – economic or wider impacts?
• Time-lags
• Nature of knowledge transfer – linear or otherwise? How
to measure HEI impact in a complex process?
• HEFCE – The proposals are not about:
–
–
–
–
–
–
Quantifying impact
Focusing narrowly on economic impact
Assessing impact of every researcher or output
Trying to predict future impact
Discouraging curiosity-driven research
Trading-off impact and excellence
• Internal impact data collection & case study development
Nature of the pilot
Five proposed REF UOAs were chosen in which to pilot the
assessment of impact information:
•
Clinical Medicine
•
Physics
•
Earth Systems and Environmental Sciences
•
Social Work and Social Policy & Administration
•
English Language & Literature
Twenty-nine HEIs from across the UK were selected to take part
in the pilot, to provide a spread of at least 10 institutions with
differing characteristics submitting to each pilot UOA.
Earth Systems and Environmental Sciences and English
Language and Literature were piloted in Manchester
Submission contents
For each UoA:
• overview information relating to the submitted unit as a whole
(narrative template plus excerpts from RAE RA4 and RA5)
• a number of (template-structured) case studies illustrating
specific impacts (one case study for every ten 2008 Cat A staff,
rounded up to the nearest 10)
Impacts that have occurred between Jan 2005 and Dec 2009
The underpinning research could date back to 1993 and should
be >2* in quality
Impact defined broadly as “any identifiable benefit to or positive
influence on the economy, society, public policy or services,
culture, the environment or quality of life.”
Does not include impacts within the academic sphere or the
advancement of scientific knowledge
Developing submissions
Research Director in each school co-ordinated submission
Full Impact assessment of school activities (via School Board,
email reminders, Research Group leaders and School
Research Committee). Minimal initial interest.
Input used to (a) write school overview and (b) identify potential
case studies.
Case studies discussed in detail with Research Group leaders
and School Research Committee and then drafted by relevant
academics.
Each school chose to submit one ‘supplementary’ case study in
addition to the required four.
Outcome expected in Autumn 2010
What have we learned - challenges?
HEFCE advice ambiguous in some areas at present. Difficult
always to distinguish fully between inputs (which “are not the
focus of assessment”) and interim or full outcomes (which are).
Templates sometimes unclear
Difficult to gauge how some activities will be assessed/weighted
by particular panels
Tendency, in case studies, for attribution of impacts to be
stressed more than their significance
Desire to avoid ‘mere’ knowledge transfer can lead to an overly
inhibited account of the contribution of our research to major
impacts
Relatively low level of appreciation of what counts as impact for
REF
What have we learned - challenges?
Tendency to focus on recent research activity by current
members of staff with strategic academic potential. Are we
missing profitable case studies relating to research areas that
may now be dormant?
Indicators of impact range and significance not readily available
or widely understood in the institution
For commercial activities (e.g. spinout SMEs) it is important to
have detailed and accurate assessment of real and potential
impact (long lead time for max impact)
Difficult to determine what some stakeholders do with data
provided by the institution or what access we might have to
impact indicators
How might we move forward?
Cover impact in staff performance and development review
Collection of impact statements and reports for all grants
Improve systems for reporting commercial funding (inc. PGR) and
patent/licence information
Maintain an accurate picture of where postgraduate alumni and
research staff go, especially if entering areas of national strategic
importance
Consider involvement of science journalists and other
professionals to help case studies communicate impact
Use of consultants to help with quantifying the commercial impact
of our work