May 11, 2015
The evaluation of research
units at HCERES
Alain Cappy, coordinator for the S&T domain
1
Outline
1-HCERES présentation
Status and Organization
Founding Principles
Missions and Objectives
EU Recognition
Annual Work
IntegratedE
2-Evaluation of research units
Principles
Report
Syntheses
3- Conclusion
2
HCERES: Status and Organisation
Status: Independent Administrative Authority
Organization
•President, chairs the board
•Board of the Council: 30 members
•3 evaluation departments
•
•
•
Institutions
Research units
Programs and degrees
•Observatory of Science and Technologies ( data base, bibliometric analysis,
indicators…)
•Support service
3
HCERES: Status and Organisation
Scientific organization
• About 100 part-time scientific delegates
• 4,500 experts commited every year, including 20% outside France
Administrative organization
•About 100 full-time administrative and technical staff
Budget in 2015: about 15 M€
4
Founding Principles
Independence:
Status of independent administrative authority. HCERES to carry out its missions
without coming under pressure from:
•
government authorities,
•
Evaluated entities ( research units, Universities, research bodies),
•
or any other stakeholders
HCERES guarantees compliance with the rules of ethics, in particular those concerning
conflicts of interests.
5
Founding Principles
Competence of experts and transparency of procedures
Quality of evaluation rests on:
• the expert’s skills and correct matching his/her profile with the entity evaluated,
• a methodology and procedures guaranteeing equality of treatment,
• drawing up by HCERES of a quality management system.
Transparency of evaluation lies on publication (website) of :
• criteria for and methods of evaluation,
• the list of scientific delegates, experts and their positions,
• Summary of evaluation reports.
6
HCERES: Missions and Objectives
Evaluating:
•HE & R institutions (universities, schools and research bodies), taking account of all
their missions and activities.
•Research activity laboratories and research units of these institutions
• Education programs and degrees in the French higher education system (BMD)
Assisting evaluated entities in the continuous improvement of their practices
Providing clarification for decision-makers:
•The State (financing, recognition of qualifications)
•Research bodies (certification of laboratories)
•Others (Local Authorities, Companies, Families)
7
What HCERES Do and Do Not
HCEES evaluates and does NOT decide
Decisions about accreditation and funding belong to Universities,
Institutions, State,…
HCERES does NOT evaluate persons
But representatives of person evaluation bodies are included at
different levels :
o HCERES Council
o Visiting committees
8
European Recognition
HCERES is member of the European Association for Quality Assurance in Higher
Education (ENQA). ENQA promotes European co-operation in the field of quality
assurance in higher education .
HCERES is listed in EQAR, the European Quality Assurance Register for Higher Education.
9
Annual Work
Institutions:
• 60-70 universities and schools
• 3 research organisms
Research units:
•
600 evaluation committees
Training courses:
• 700 Bachelor degrees
• 300 Master degrees
• 50 doctoral schools
10
The Regional Breakdown of the Evaluation Campaigns
A 5-Year Period
11
Integrated Evaluation
For one institution ( University, Engineer School…)
Evaluation of research
units including on site
visit
Evaluation of training
programmes, ( Bachelors,
Masters, Doctoral Schools)
Evaluation
reports
Evaluation
reports
12
Evaluation of
institution
including on site
visit.
Document Management System
Web based evaluation process (GED)
• File and report deposition
• Evaluation committee creation
• Flags to follow the state and progress of a specific
unit evaluation
13
Document Management System (GED)
14
Evaluation of research units
15
Methodological Principles
The criteria defined and procedures implemented by the High Council take into
account:
•
•
•
the diversity of the natures and missions of the research units
the diversity of disciplinary fields ( Life Sciences, Social Sciences and Humanities,
Physical and Engineering Sciences)
based on peer-assessment (national and international experts)
Evaluation goals:
•
•
Oriented towards improvement of unit performance
Combines quantitative data ( scientific production, impact) with qualitative
information assessed by experts
16
Evaluation of Research Units: for Who ?
the recipients of the evaluation are:
•
•
•
•
•
Research units and researchers
research organisms, Universities, Grandes Ecoles
Students
Local Authorities (regions, etc.)
Ministry for Higher Education and Research and other ministries
17
Evaluation of Research Units
The annual task: evaluation of all State funded research units in specific
Regions ( France is divided in 5 evaluation campaigns)
• ~ 600 research units
• > 3000 experts committed every years for the peer review
Covering ALL research bodies supported by public funds, including
private universities, architectural and cultural research, and possibly
clinical research…
18
Research Unit Evaluation Process
Main steps
1. The research unit writes a report following a model. This report is
deposited on the HCERES document management system (GED).
2. A HCERES scientific officer is designed to manage the evaluation
3. An evaluation committee is appointed by HCERES
4. The committee prepares a draft of the report before the visit
5. A visit (1, 2 or 3 days ) of the research unit is organised. During the
visit, the unit presents its main results and strategy. The committee
has meeting with the unit personnel
19
Research Unit Evaluation Process
Main steps (cont’d)
6.After the visit the committee finalised the report
7.The report is controlled by HCERES to be sure that it follows HCERES
writing rules
8.The report is transmitted to the unit and its supervising institutions. A
summary of the report is published on the HCERES website.
20
The Evaluation Report
The report is the main output of the evaluation.
•
The report should be critical, not descriptive,
•
The report must contain evaluation details of the research unit at
a whole but also for each research team,
•
The report has to specify strengths, weaknesses and
recommendations.
21
Evaluation Report
• The evaluation report: the six evaluation items
•
•
•
•
•
•
Scientific outputs and quality of research
Academic reputation and appeal
Interactions with the social, economic and cultural environment
Organization and life of the unit
Involvement in training
Strategy and five-year plan
• These six items are used both for the research unit as a whole and for the
research groups
22
Evaluation of research units
Synthesis by geographical site:
In addition to the research units evaluation reports, HCERES writes research
syntheses :
Goal: To give a schematic view of the particularities of a geographical site based
on:
• The characterization of working forces in the three domains (Social sciences and
Humanities (SH), Life sciences (LS),Physical and Engineering Sciences (PE))
• The analysis of evaluation reports in each domain
The synthesis contains two parts
- the workforce
- the synthesis of the evaluation reports
Specific syntheses can be produced by order
23
Breakdown of workforce: Grenoble
1400
1200
1000
800
Ass. Prof/Prof
Researchers
600
Support
400
200
0
Life Sciences
Social
Sciences and
Humanities
24
Physiscal and
Engineering
Sciences
Breakdown of workforce: Grenoble
1000
900
800
700
600
500
Ass Prof.prof
400
Researchers
300
support
200
100
0
25
Open questions
Evaluation is a difficult task. Many questions are still open
• How to evaluate interdisciplinary research ?
• How to measure the impact of a research ?
• How to efficiently use bibliometric data
HCERES working groups and seminars are organized to tackle these
questions
26
Conclusion
• HCERES, a ‘High Council for Research and Higher
Education’ devoted in France to the evaluation of
Institutions, research units and training courses
• HCERES is an independent authority
• HCERES performs an integrated evaluation in each
site
• HCERES is member of ENQA and EQAR
• All the French research units and research groups
are evaluated every 5 years
27