HORIZON 2020
Evaluation Procedure
One Stage Topics
Content
• Subject of the evaluation: call and schedule
• Types of Action under evaluation
• Evaluation Process: Criteria, Scoring, Important Definitions, Horizontal
Issues
• Individual Evaluation
• How to get started IER?
• Drafting of Consensus Report
• How to get started CR?
• Consensus Meeting
• Who can support you?
• Logistics for Brussels
Call
Greening the
Economy (SC5)
•Climate Services
•Towards a low-carbon Europe
•Nature-based solutions for
territorial resilience
•Water
•Raw Materials
•Earth Observation
•Cultural heritage for
sustainable growth
•Support to policy and
innovation procurement
Evaluation Schedule
Submission
Procedure
Topics
Stage
Deadline
Info to
Applicants
Grant
Agreements
One Stage
20 topics and
sub-topics
(RIA, CSA,
ERA-NET,
PCP)
One Stage
(full
proposal)
07/03/2017
June 2017
Dec 2017
First Stage
(short
proposal)
07/03/2017
May 2017
-
Second Stage
(full proposal)
05/09/2017
Dec 2017
May 2018
Two Stage
7 topics and
sub-topics
(IA)
Type of action
Research and Innovation
Actions
Coordination and Support
Actions
Pre-commercial Procurement
Actions
ERA-NET Cofund actions
RIA
Action primarily consisting of activities to establish new knowledge
and/or explore feasibility of new or improved technology, product,
process, service or solution
• May include basic and applied research, technology development and
integration, testing and validation on small-scale prototype in laboratory or
simulated environment
• Projects may contain closely connected but limited demonstration or pilot
activities to show technical feasibility in a near to operational environment
CSA
Actions consisting primarily of accompanying measures such as
• standardisation, dissemination, awareness-raising and communication,
networking, coordination or support services, policy dialogues and mutual
learning exercises and studies, including design studies for new
infrastructure, and
• may also include complementary activities of strategic planning, networking
and coordination between programmes in different countries
ERA-NET
Supports public-public partnerships, including joint programming
initiatives between Member States, in their preparation, establishment
of networking structures, design, implementation and coordination of
joint activities as well as EU topping-up of trans-national call for
proposals
• The main activity is the implementation of the co-funded joint call for
proposals that leads to the funding of transnational research and/or
innovation projects. In addition, consortia may implement other joint
activities e.g. joint calls without EU co-funding
• May also, depending on the research area and the underlying national
programmes and their governing principles, target governmental research
organisations. The co-funded call will in these cases be based on in-kind
contributions from their institutional funding and the beneficiaries carry out
the transnational projects resulting from their call fully or partially
themselves
• The in-kind contributions are the resources allocated as direct expenditure in
the selected trans-national projects that are not reimbursed by the EU
contribution
PCP
Encourage public procurement of research, development and validation
of new solutions that can bring significant quality and efficiency
improvements in areas of public interest, while opening market
opportunities for industry and researchers
• Provides EU co-funding for group of procurers to undertake together one
joint PCP procurement, so that there is one joint call for tender, one joint
evaluation of offers, and a lead procurer awarding the R&D service contracts
in the name and on behalf of the group
• Each procurer contributes its individual financial contribution to the total
budget necessary to jointly finance the PCP, enabling the procurers to share
the costs of procuring R&D services from a number of providers and
comparing together the merits of alternative solutions paths to address the
common challenge
• The PCP shall explore alternative solution paths from a number of competing
providers to address one concrete procurement need that is identified as a
common challenge in the innovation plans of the procurers that requires new
R&D
• Cross-border PCP cooperation should better address issues of common
European interest, for example where interoperability and coherence of
solutions across borders is required
Evaluation
Experts
EASME
EASME
Receipt of
proposals
Individual
evaluation
Consensus
meeting
Panel
meeting
Finalisation
Eligibility check
Individual
Evaluation
Reports
(remote)
Consensus
Report
(central)
Cross reading
EC ranked list
Allocation of
proposals to
evaluators
Evaluation Summary Information sent to
Report
applicants
Panel report
Expert ranked list
Evaluation
Eligible proposal
Proposal
Expert
Expert
Individual
Evaluation
Report
Individual
Evaluation
Report
Expert
Expert
Individual
Evaluation
Report
Expert
Individual
Evaluation
Report
Individual
Evaluation
Report
Draft
Consensus
Report
Minimum 3 evaluators
Individual Evaluation
Reports (IER)
Draft Consensus
report (CR)
RAPPORTEUR
Consensus meeting
Consensus
meeting
(on-site)
SUPPRA-RAPPORTEUR*
Consensus report (CR)
Consensus
Report
Evaluation
The specific schedule for your topic and for the
different tasks will be set by your topic
moderator
Please comply with set deadlines
The dates in the contract and in SEP are
general for all actors and topics
Evaluation
There are three evaluation criteria
• Excellence
• Impact
• Implementation
• The criteria are adapted to each type of action, as specified in the WP
You give a score of between 0 and 5 to each criterion
• The whole range of scores should be used
• Half-marks can be used
• Scores must pass thresholds if a proposal is to be considered for funding
Thresholds
Excellence
3
Impact
3
Implementation
3
Overall
10
Proposal Scoring
0
The proposal fails to address the criterion or cannot be assessed due
to missing or incomplete information.
1
Poor. The criterion is inadequately addressed, or there are serious
inherent weaknesses.
2
Fair. The proposal broadly addresses the criterion, but there are
significant weaknesses.
3
Good. The proposal addresses the criterion well, but a number of
shortcomings are present.
4
Very Good. The proposal addresses the criterion very well, but a
small number of shortcomings are present.
5
Excellent. The proposal successfully addresses all relevant aspects
of the criterion. Any shortcomings are minor.
Criteria - Excellence
Excellence - RIA
• Clarity and pertinence of the
objectives;
• Soundness of the concept and
credibility of the proposed
methodology;
• Extent that the proposed work
is beyond the state of the art,
and demonstrates innovation
potential (e.g. groundbreaking objectives, novel
concepts and approaches, new
products, services or business
and organisational models)
• Appropriate consideration of
interdisciplinary approaches
and, where relevant, use of
stakeholder knowledge.
Excellence - CSA
• Clarity and pertinence of the
objectives;
• Soundness of the concept and
credibility of the proposed
methodology;
• Quality of the proposed
coordination and/or support
measures
Excellence - PCP
• Clarity and pertinence of the
objectives;
• Soundness of the concept and
credibility of the proposed
methodology;
• Progress beyond the state of the
art in terms of the degree of
innovation needed to satisfy the
procurement need
Excellence –
ERA-NET
• Clarity and pertinence of the
objectives;
• Soundness of the concept and
credibility of the proposed
methodology;
• Level of ambition in the
collaboration and commitment
of the participants in the
proposed ERA-NET action to
pool national resources and
coordinate their
national/regional research
programmes
Evaluation Criteria
Important Definitions
The term 'innovation' is used in the EU policy context and more widely to
mean the introduction in the market of new or improved products, services,
processes and solutions.
Co-design & Co-creation
• Innovation potential is the potential of a project to create useful novelties
beyond what already existing (RIA)
Responsible Research and Innovation
Societal actors work together to align research and
results with the values, needs and expectations of
society.
Public engagement
Iterative/participatory multi-actor dialogues to cocreate research and innovation outcomes and policy
agendas.
Trans-disciplinarity
Methodologies that integrate scientific disciplines,
and non-academic and non-formalized knowledge.
Criteria - Impact
Impact - RIA
• The extent to which the outputs
of the project would contribute to
each of the expected impacts
mentioned in the work
programme under the relevant
topic;
• Any substantial impacts not
mentioned in the work
programme, that would enhance
innovation capacity, create
new market opportunities,
strengthen competitiveness and
growth of companies, address
issues related to climate change
or the environment, or bring
other important benefits for
society;
• Quality of the proposed
measures to:
- Exploit and disseminate the
project results (including
management of IPR), and to
manage research data where
relevant.
- Communicate the project
activities to different target
audiences.
Impact - CSA
• The extent to which the outputs
of the project would contribute to
each of the expected impacts
mentioned in the work
programme under the relevant
topic;
• Quality of the proposed
measures to:
- Exploit and disseminate the
project results (including
management of IPR), and to
manage research data where
relevant.
- Communicate the project
activities to different target
audiences.
Criteria - Impact
Impact - PCP
Impact – ERA-NET
• The extent to which the outputs
of the project would contribute to
each of the expected impacts
mentioned in the work
programme under the relevant
topic;
• The extent to which the outputs
of the project would contribute to
each of the expected impacts
mentioned in the work
programme under the relevant
topic;
• Strengthening the
competitiveness and growth of
companies by developing
innovations meeting needs of
European and global
procurement markets
• Achievement of critical mass for
the funding of trans-national
projects by pooling of
national/regional and
contribution to establishing and
strengthening a durable
cooperation between the
partners and their
national/regional research
programmes
• Quality of the proposed
measures to:
- Exploit and disseminate the
project results (including
management of IPR), and to
manage research data where
relevant.
- Communicate the project
activities to different target
audiences.
• More forward looking
procurement approaches
reducing fragmentation of
demand for innovative solutions
• Quality of the proposed
measures to:
- Exploit and disseminate the
project results (including
management of IPR), and to
manage research data where
relevant.
- Communicate the project
activities to different target
audiences.
Evaluation Criteria
Important Definitions
• Innovation capacity is the capacity of the new or improved novelties to
create impact beyond their original purpose i.e. the capacity of inspiring or
having a knock on effect in another domains / sectors (RIA)
• Intellectual Property is the knowledge / results created by the project.
(RIA, CSA, ERA-NET, PCP)
• Intellectual Property Rights are the legal rights granting preferential use.
For example, Patents, Copyrights, Trade marks, Design rights, Database
rights, Plant Breeders rights, Utility models / petty patents, Confidentiality
agreements, Trade secrets, etc. (RIA, CSA, ERA-NET, PCP)
• Intellectual Property Management is the process of monitoring,
assessing, protecting, disseminating, exploiting (i.e. making usable
elsewhere) the knowledge created. (RIA, CSA, ERA-NET, PCP)
Evaluation Criteria
Important Definitions
• Exploitation is the use of the results during and after the project’s
implementation. It can be for commercial purposes but also for improving
policies, and for tackling economic and societal problems. (RIA, CSA, ERANET, PCP)
• Dissemination is making the project outputs available to various
stakeholder groups (like research peers, industry and other commercial
actors, professional organisations, policymakers) in a targeted way, to
enable them to use the results in their own work. (RIA, CSA, ERA-NET,
PCP)
• Communication is the process of promoting the action and its results via
strategic and targeted measures to a multitude of audiences, including the
media and the public, and possibly engaging in a two-way exchange. The
aim is to reach out to society as a whole and in particular to some specific
audiences while demonstrating how EU funding contributes to tackling
societal challenges. (RIA, CSA, ERA-NET, PCP)
Criteria - Implementation
Implementation
• Quality and effectiveness of
the work plan, including extent
to which the resources
assigned to work packages are
in line with their objectives and
deliverables;
• Appropriateness of the
management structures and
procedures, including risk and
innovation management;
• Complementarity of the
participants and extent to
which the consortium as whole
brings together the necessary
expertise;
• Appropriateness of the
allocation of tasks, ensuring
that all participants have a
valid role and adequate
resources in the project to fulfil
that role.
• Innovation management is the
management of the process to create
innovation (from idea to market
introduction). (RIA, CSA, ERA-NET, PCP)
Horizontal Issues
Topics flagged for Social Sciences
and Humanities (SSH)
• SC5-06-2016-2017
• SC5-15-2016-2017
• SC5-19-2017
• SC5-22-2017
• SC5-30-2017
• SC5-32-2017
• SC5-33-2017
Topics where International
Cooperation is required
• SC5-16-2016-2017
Topics where International
Cooperation is encouraged
• SC5-02-2016-2017
• SC5-06-2016-2017
• SC5-13-2016-2017
• SC5-15-2016-2017
• SC5-30-2017
• SC5-31-2017
• SC5-32-2017
• SC5-33-2017
Individual Evaluation
• You read the proposal and evaluate it against the evaluation criteria
• Without discussing it with anybody else
• As submitted - not on its potential if certain changes were to be made
• You disregard excess pages in section 1-3 of Part B (RIA – 70 pages,
CSA and ERA-NET – 50 pages, PCP – 90 pages)
• Moreover, you do not consider information included in parts originally
foreseen for other purposes (e.g. section 4-5 of Part B, Ethics Annexes,
etc.)
• In case of doubt please ask your topic moderator
• You look at the substance: Some proposals might be face language
difficulties, others are deceptively well written
• You explain shortcomings, but do not make recommendations
Individual Evaluation
If a proposal:
• Is only marginally relevant in terms of its scientific, technological or
innovation content relating to the topic addressed, you must reflect this in a
lower score for the Excellence criterion
•
No matter how excellent the science!
• Does not significantly contribute to the expected impacts as specified
in the WP for that call or topic, you must reflect this in a lower score for the
Impact criterion
• Would require substantial modifications in terms of implementation
(i.e. change of partners, additional work packages, significant budget or
resources cut…), you must reflect this in a lower score for the Quality and
efficiency of the Implementation criterion
• If cross-cutting issues are explicitly mentioned in the scope of the call or
topic, and not properly addressed (or their non-relevance justified), you
must reflect this in a lower score for the relevant criterion
Additional Questions
Out-of-Scope
• If you consider a proposal to not be in scope in regards to the topic
description please signal it in the appropriate box and provide a
justification of why you believe it is out-of-scope.
Operational capacity
• If you identify any partner that seems inadequate or fraudulent
partners on basis of the info provided please signal it in the
appropriate box. Your individual evaluation should progress
normally.
Exceptional funding of third country participants / international
organisations
• If you identify any partner that is not eligible for funding please signal
it in the appropriate box, as well as your justification of why or why
not the partner should be funded.
• Exceptional funding needs to be justified on the basis of:
Outstanding competence / expertise; Access to particular
geographical environments or research infrastructure or data.
How to get started - IER?
1.
Go to "SEP", the online system for proposal evaluation
2.
Study the briefing material sent to you by the topic moderator and if being
organised attend the topic web-briefing
3.
Screen your proposals - to check potential Conflict of Interest and have
an overview of all proposals
4.
Indicate to your topic moderator any potential CoI
5.
Evaluate a first proposal and save a draft report in SEP. You can ask your
topic moderator to provide you feedback
How to get started - IER?
6.
Read each proposal in detail. Go through each criterion and write your
comments for every aspect mentioned under the criterion. Score!
7.
Complete your IER. The quality of your IER is directly proportional to the
quality of the final Consensus Report
8.
After completing each IER – check that your scores match your
comments. Use the full scale of scores!
9.
Review your facts – when you cite facts, are they truly in the proposal and
will you be able to find them back during the Consensus Meetings?
10.After having completed all IERs – calibrate your IER (have you bveen fair
with every proposal?) and submit the reports by the deadline indicated to
you.
Preparation of draft CR
• The rapporteur is responsible for drafting the CR
• In some cases, the rapporteur is not one of the evaluators
• The quality of the CR is of utmost importance, the aim is to give:
• A clear assessment of the proposal based on its merit, with justification
•
Clear feedback on the proposal’s weaknesses and strengths, of an
adequate length, and in an appropriate tone
• The goal of this task is to facilitate the work during the consensus
meeting as the CR will be a compromise report that balances the views of
the different evaluators
How to get started - CR?
1.
Study the briefing material sent to you by the topic moderator (even if you
are not an evaluator) and attend the topic web-briefing (if organised)
2.
Go to "SEP", the online system for proposal evaluation
3.
If not an evaluator screen your proposals - to check potential Conflict of
Interest and indicate to your topic moderator any potential CoI
4.
Read all the Individual Evaluation Reports and identify points of
convergence and divergence. You can use the functions "Merge" and
"Initialise" of SEP to help you in your work.
How to get started - CR?
6.
Propose a consensus wording for the convergent points highlighting
strengths and weaknesses; and identify the divergences as points for
discussion with pros and cons to guide the consensus meeting
7.
If needed, refer to the proposal for checking factual references
8. Save the report in SEP without scores within the set deadline. You can
ask your topic moderator to provide you feedback
9.
During the consensus meeting review and update your draft CR to reflect
the outcome of the discussion
10. If you did not evaluate the proposal do not express or impose your own
view
Consensus meeting
• It involves a discussion on the basis of the individual evaluations
• The aim is to find agreement on comments and scores and finalise
the CR
•
Keep in mind that is normal for individual views to change after
arguments are exchanged and that “outlying” opinions need to be
explored as they might be as valid as others
•
Agree on comments before scoring!
• Moderated by Agency staff
•
Manages the evaluation, protects confidentiality and ensures fairness
•
Ensures objectivity and accuracy, all voices heard and points discussed
•
Helps the group keep to time and reach consensus
Who can support you?
• Your first point of contact is your topic moderator:
• For all clarifications on your assignement
• Backup arrangements: your Panel Moderator will communicate to you any
backup arrangements needed during holiday times
• [email protected]
• For SEP issues - please contact the IT helpdesk:
• [email protected]
• Phone: +32 2 29 92222
• Available on weekdays 9:00 -18:00 CET (Friday 17:00)
• For Expert Payment issues - please contact directly:
• [email protected]
Logistics
• Your central evaluation week will take place in the dates indicated by
your topic moderator
• Work usually starts at 9h00 and finishes at 17h30
• Please do not arrange any other travel plan without explicit consent of
your topic moderator
• SEP – the electronic system for the evaluation of proposals is available and
accessible via your ECAS password
• Please make sure you know your ECAS login and password
• Please bring your own laptop/tablet/notebook
• Reduction of paper copies
• A few printers are available in the evaluation building in Brussels
• Please bring your own paper copies of material you need to support your
work