NATHFE’s response to the
Joint consultation on the review of research assessment
2003/22
Introduction
NATFHE welcomed the review undertaken by the Roberts Committee and we
respond below to the consultation on the detail of that review, in the format
provided by HEFCE. However in our view it is not possible to make
judgements about matters of assessment processes separately from
judgements about the funding criteria that will inform the use of such
processes, and both have to be considered in the context of the current
political intention that research funding in the UK be yet further concentrated
and more selective.
In our response to an earlier phase of the joint review of research
assessment, in November 2002, we set out NATFHE’s priorities for research
assessment and funding. We have assessed the recommendations in the
Roberts’ Review report against these priorities, whilst recognising that the
review, and subsequent proposals from HEFCE on the research funding
method, are driven by government’s demands for greater concentration and
selectivity. In our view the misguided belief in the necessity for greater
concentration of research funding, in an already highly concentrated system,
has distorted the attempts to review research assessment and funding. This
will also impact differentially on different elements of a UK system. The
Funding Council for Wales has indicated that it is likely to resist moves
towards greater concentration for research yes, if it continues to use the
same assessment system as England, will find that under these proposals
that it was using a system devised for other policy objectives than its own.
Nonetheless we welcome the areas in which the Roberts Review has
addressed specific concerns of academics in relation to the operation of the
research assessment exercise.
NATFHE’s priorities for research assessment and funding
1. The retention of a higher education system in which all institutions
can provide– to a greater or lesser extent – a research environment,
within which the resources exist for research and scholarship to take
place both to support and inspire students, and those academic staff
whose own work may be teaching-focused
Given the intention that only very small, or non-existent, amounts of
funding for research will be available to all but a concentrated group
of institutions and departments, the proposals for a three tier system
of research assessment can only facilitate the reduction in research
funding to many. However the real threat to this priority is the
insistence on greater concentration of funding, rather than the
structural proposals that would both recognise and facilitate
concentration.
2. A system which encourages collaboration between groups of
researchers across institutional and disciplinary boundaries
Whilst elements of detail in the proposals for a new RQA seek to
address this priority, and are welcome, once again the overall context
is one of reducing the scope for research to flourish in a wide range of
departments and institutions, and this of reducing the scope for
collaboration.
3. A system that is sufficiently broad in scope to allow for funding a
context within which excellent research may arise – that can assess
and fund potential as well as achievement
The whole thrust of the existing level of concentration – let alone
further concentration – is to put all the funding eggs in the basket of
prior success and achievement, and to reduce the funding available to
develop capacity and recognise potential. The specific
recommendation by Roberts that institutions be allowed to identify
emerging units of research and to identify their capacity for
development, which we would support, has been deemed inappropriate
for consultation by the funding councils.
4. A system that is rigorous about testing out the need for the
appropriate degree of selectivity, and that does not apply selectivity
where it is not necessary
Once again where the Roberts report does attempt to engage with this
question – by suggesting that there might be different funding ratios
in different discipline areas, this has been deemed inappropriate for
consultation. The political context is currently hostile to any
questioning of the need for selectivity and we are left with a situation
where the research-concentration demands of a small – albeit very
important – number of discipline areas are driving the entire system.
5. A system that is not designed to mask deficiencies in funding, nor
weighted towards protecting the status quo
In one sense the Roberts recommendations do not mask deficiencies in
funding so much as make them explicit and seek to introduce a fairer
process, by acknowledging that if there is little or no money available
then institutions might as well not waste their time on a complex
assessment process. Nonetheless, the assessment process is driven
not only by the available funding, but also by pre-emptive decisions as
to the relative allocation of that funding.
6. An approach to peer and “expert” review that involves more of those
not part of the current research-elite establishment, including users of
research and scholarship from government, industry and teaching
We welcome aspects of the proposals in this respect, and in particular
the suggestion that panels and sub-panels be informed by those with a
multi-disciplinary background, and that assessment be made with
respect to the impact of research on professional practice, and of
practice-based research, which we assume would require
appropriately broad panel membership.
7. A system that encourages fruitful links between research, scholarship
and teaching
Whilst we welcome the proposal that an assessment of institutions’
research competencies should include an assessment of the links made
between research work and teaching and scholarship, the greater
threat to this relationship lies in an approach to sector-wide funding
that would lead to some institutions being largely starved of funding
for research.
Recommendation 1 (see paragraphs 113-116 of the review)
Any system of research assessment designed to identify the best research
must be based upon the judgement of experts who may, if they choose,
employ performance indicators to inform their judgement.
Do you agree or disagree with recommendation 1? Place a cross beside the
appropriate answer:
Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree
…
…x
…
…
…
Comments on recommendation 1:
Recommendation 2 (see paragraphs 117-126 of the review)
a. There should be a six-year cycle.
b. There should be a light-touch ‘mid-point monitoring’. This would be
designed only to highlight significant changes in the volume of activity in
each unit.
c. The next assessment process should take place in 2007-8.
Do you agree or disagree with recommendation 2? Please indicate your views
using the grid below:
Strongly
agree
Point a
Point b
Point c
Agree
Neither
agree nor
disagree
x
x
x
Disagree
Strongly
disagree
Comments on recommendation 2:
The timing of the next exercise should depend, in part, on the scale of
change envisaged at the end of the consultation process and any further
policy development
Recommendation 3 (see paragraphs 127-133 of the review)
a.
There should be an institution-level assessment of research
competences, undertaken approximately two years before the main
assessment.
b.
The competences to be assessed should be institutional research
strategy, development of researchers, equal opportunities, and dissemination
beyond the peer group.
c.
An institution failing its assessment against any one of the
competencies would be allowed to enter the next research assessment but
would not receive funding on the basis of its performance in that assessment
until it had demonstrated a satisfactory performance.
Do you agree or disagree with recommendation 3? Please indicate your views
using the grid below:
Strongly
agree
Point a
Point b
Point c
Agree
Neither
agree nor
disagree
Disagree
Strongly
disagree
x
x
x
Comments on recommendation 3:
Although we support the principle that there should be an institution-level
assessment of human resource and dissemination strategies in relation to
research, we would argue for this to be conducted in a holistic manner that
brought together equal opportunity policies, staff development policies and
teaching dissemination policies across institutions as a whole, and that
located the research elements within these.
We welcome the attention paid to the importance of the dissemination of
research beyond the academic peer group and in particular the
encouragement to institutions to demonstrate the ways in which the links
between research and teaching are fostered.
All institutions should be in a position to submit to the research competency
exercise, at the appropriate level of engagement and activity depending on
the volume and nature of their research. This in itself has funding
implications.
Recommendation 4 (see paragraphs 134-155 of the review)
a. There should, in principle, be a multi-track assessment enabling the
intensiveness of the assessment activity (and potentially the degree of
risk) to be proportionate to the likely benefit.
b. The least research intensive institutions should be considered
separately from the remainder of the HE sector.
c. The form of the assessment of the least research intensive institutions
would be a matter for the relevant funding council.
d. The less competitive work in the remainder of institutions should be
assessed by proxy measures against a threshold standard.
e. The most competitive work should be assessed using an expert review
assessment similar to the old Research Assessment Exercise.
Do you agree or disagree with recommendation 4? Please indicate your views
using the grid below:
Strongly
agree
Point
Point
Point
Point
Point
Agree
a
b
c
d
e
Neither
agree nor
disagree
x
Disagree
Strongly
disagree
x
x
x
x
Comments on recommendation 4:
Given the intention that only very small, or non-existent, amounts of funding
for research will be available to all but a concentrated group of institutions
and departments, the proposals for a three-tier system of research
assessment can only facilitate the reduction in research funding to many.
Having demonstrated research competence, all institutions should be free to
choose whether or not to make submissions to the RAE/RQA.
Nonetheless we agree that there should be clarity and transparency as to the
likely funding that would result. In the context of a less-selective funding
regime, where significant levels of funding could be made available to
support small-scale and emerging research, it might be appropriate for a
more light-touch system to then operate for those institutions/departments
not submitting to the RAE/RQA.
Recommendation 5 (see paragraphs 156-171 of the review)
a. The output of the Research Quality Assessment should be a ‘quality
profile’ indicating the quantum of ‘one star’, ‘two star’ and ‘three star’
research in each submission. It will not be the role of the assessment
to reduce this profile to summary metrics or grades.
b. As a matter of principle, star ratings would not be given to named
individuals, nor would the profile be published if the submission were
sufficiently small that individual performance could be inferred from
it.
c. Panels would be given guidelines on expected proportions of three
star, two star and one star ratings. These proportions should normally
be the same for each unit of assessment. If a panel awarded grades
which were more or less generous than anticipated in the guidelines,
these grades would have to be confirmed through moderation.1
Do you agree or disagree with recommendation 5? Please indicate your views
using the grid below:
Strongly
agree
Point a
Point b
Point c
Agree
Neither
agree nor
disagree
Disagree
Strongly
disagree
x
x
x
Comments on recommendation 5:
We regret that the part of the original recommendation dealing with the
potential for funding ratios to vary according to discipline, has been omitted
from this consultation. It is essential that the issue of whether or not the
same degree of selectivity is appropriate to every discipline area (in our view
not) is properly debated.
Recommendation 6 (see paragraphs 172-197 of the review)
a. There should be between 20 and 25 units of assessment panels
supported by around 60 sub-panels. Panels and sub-panels should be
supported by colleges of assessors with experience of working in
designated multidisciplinary ‘thematic’ areas.
b. Each panel should have a chair and a moderator. The role of the
moderator would be to ensure consistency of practice across the subpanels within the unit of assessment.
c. Each panel should include a number of non-UK based researchers with
experience of the UK research system.
d. The moderators of adjacent panels should meet in five or six ‘superpanels’ whose role would be to ensure consistency of practice between
panels. These ‘super-panels’ should be chaired by senior moderators
who would be individuals with extensive experience in research.
This consultation question reflects an edited version of recommendation 5. The recommendation in the
review report also states that ’the funding councils should provide institutions with details of the relative
value, in funding terms, of one star, two star, and three star research, and of research fundable through
the Research Capacity Assessment in advance of the assessment. These ratios might vary between
disciplines.’ In the event that the review recommendations are accepted, each funding council will develop
its own policies for reflecting the assessment results in funding, taking proper account of Sir Gareth’s
recommendation.
1
Do you agree or disagree with recommendation 6? Please indicate your views
using the grid below:
Strongly
agree
Point
Point
Point
Point
a
b
c
d
Agree
Neither
agree nor
disagree
x
Disagree
Strongly
disagree
x
x
x
Comments on recommendation 6:
It is important that whatever the precise number of units of assessment that
all areas of work can be properly recognized. At the moments areas like
gender and race studies often fall outside units of assessment, and it is also
very important that research into the pedagogy of subjects within higher
education gets full recognition – not least as part of promoting synergy
between teaching and research, and giving force to the oft-expressed desire
to see parity of esteem between the two.
In addition to recommendation c, that non-UK based researchers should sit
on panels, we would like to see more attention paid to the need to get a
broad base of expertise on panels, including those with the capacity to
make assessment in the areas of applicable research and impact on
professional practice.
Recommendation 7 (see paragraphs 198-204 of the review)
a. The rule that each researcher may only submit up to four items of
research output should be abolished. Research Quality Assessment
panels should have the freedom to define their own limits on the
number and/or size of research outputs associated with each
researcher or group.
b. Research Quality Assessment panels should ensure that their criteria
statements enable them to guarantee that practice-based and
applicable research are assessed according to criteria which reflect the
characteristics of excellence in those types of research in those
disciplines.
Do you agree or disagree with recommendation 7? Please indicate your views
using the grid below:
Strongly
agree
Point a
Point b
Agree
X
x
Neither
agree nor
disagree
Disagree
Strongly
disagree
Comments on recommendation 7:
Recommendation 8 (see paragraphs 205-213 of the review)
a. The funding councils should work alongside the subject communities
and the research councils to develop discipline-specific performance
indicators.
b. Performance against these indicators should be calculated a year prior
to the exercise, and institutions advised of their performance relative
to other institutions.
c. The weight placed upon these indicators as well as their nature should
be allowed to vary between panels.
Do you agree or disagree with recommendation 8? Please indicate your views
using the grid below:
Strongly
agree
Agree
Point a
Point b
Point c
Neither
agree nor
disagree
x
x
x
Disagree
Strongly
disagree
Comments on recommendation 8:
Whilst not in opposition to the panel’s work being informed by appropriate,
discipline-specific performance indicators – and whilst we would agree that
the weight placed in such indicators would need to vary between panels,
there would be a need to ensure consistency of the underpinning approach
used to make decisions about the use of PIs . We are not clear about the
intended use and value of the exercise whereby performance against PIs
would be calculated a year prior to research assessment and where this
would fit with the mid-cycle “light touch” review.
Recommendation 9 (see paragraphs 214-234 of the review)
a. Where an institution submits to Research Quality Assessment in a subunit of assessment all staff in that sub-unit should become ineligible
for the Research Capacity Assessment, even if they are not included in
the Research Quality Assessment submission.
b. The funding councils should establish and promote a facility for work
to be submitted as the output of a group rather than an individual
where appropriate.
c. The funding councils should consider what measures could be taken
to make joint submission more straightforward for institutions.
d. Where an institution submits a sub-unit of assessment for Research
Quality Assessment, no fewer than 80% of the qualified staff
contracted to undertake research within the sub-unit of assessment
must be included in the submission.
e. All staff eligible to apply for grants from the research councils should
be eligible for submission to Research Quality Assessment.
Do you agree or disagree with recommendation 9? Please indicate your views
using the grid below:
Strongly
agree
Point
Point
Point
Point
Point
a
b
c
d
e
Agree
Neither
agree nor
disagree
x
Disagree
Strongly
disagree
x
x
x
x
Comments on recommendation 9:
Question (a) is predicated on an agreement with the proposed structure of a
three tier assessment process
Recommendation 10 (see paragraphs 235-238 of the review)
Each panel should consider a research strategy statement outlining the
institution’s plans for research at unit level.
Do you agree or disagree with recommendation 10? Place a cross beside the
appropriate answer:
Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree
…
…x
…
…
…
Comments on recommendation 10:
Question 11 Burden for institutions
The review proposals have been designed to make the burden of assessment
proportionate with the possibility of financial reward. Do you agree that this
has been achieved? Place a cross by the appropriate answer:
Strongly agree
Agree
Neither agree nor disagree
Disagree
…
…
…x
…
Strongly disagree
…
Comments on question 11 – burden for institutions:
Arguably this is the case, but on the basis that it is right to have such
marked degrees of concentration in funding that a three-tier system is
appropriate – which we do not believe. However the approach taken in this
review does seek to recognize the burden involved for institutions that are
the capacity-building stage in some or all of their research. As we have said
in our comments on recommendation 4, in the context of a less-selective
funding regime, where significant levels of funding could be made available
to support small-scale and emerging research, it might be appropriate for a
more light-touch system to then operate for those institutions/departments
not submitting to the RAE/RQA.
Question 12 Value of research assessment
What value do you place on the research assessment if the financial reward is
likely to be small? Place a cross by the appropriate answer:
High
Medium
Low
…
…
…
Comments on question 12 – value of research assessment:
Question 13 Equality of opportunity for all groups of staff
The funding councils wish to promote equality of opportunity for all staff
regardless of age, sexual orientation, political belief, disability, gender, race
or religion and seek to ensure that its research assessment policies are
compatible with this objective. How successful do you consider that the
proposals of the research assessment review are in this respect? Place a cross
by the appropriate answer:
Very successful
Successful
Neither successful nor unsuccessful
Unsuccessful
Very unsuccessful
…
…
x
…
…
Comments on question 13 – equality of opportunity for all groups of staff:
It is hard to make a realistic response to this question. Of course there are
details of the proposals that seek to address issues of equal opportunities,
particularly in the proposals for a research competencies exercise that looks
at equal opportunity issues and development for researchers. However,
even here a lot will depend on the extent and seriousness of this exercise
and whether it is divorced from institution-wide work on equality of
opportunity or whether it scrutinizes the range of human resource policies
across the research/teaching divide, including institutional approaches to
recruitment and selection, and to the employment of part-time teaching
staff.
However at another level – the sectoral level – the exclusion of large
numbers of institutions and departments from research quality assessment
is bound to have a disproportionately negative impact on women and
minority ethnic staff, who are concentrated in the post-92 institutions.
Question 14 Overall approach of the review
Notwithstanding your views on any specific recommendations, and given the
responses to the earlier ‘Invitation to contribute’, do you agree or disagree
with the broad approach taken by the review to the question of research
assessment? Place a cross by the appropriate answer:
Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree
…
…
…
x
…
Comments on question 14 – overall approach of the review:
Please see our introductory statement
Question 15 Further comments
Question 15 – any further comments:
Please see our introductory statement. In addition we believe that any
review of research assessment must take greater account of potential
differences in parts of the UK. In Wales we hope to see significant emphasis
on collaboration between Welsh HEIs, and a greater linkage between
research assessed currently through QR assessment, and “third mission”
activity. Any system of assessment must facilitate funding decisions that
link research and institutional activities that might, under the Roberts
proposals, be dealt with entirely separately.