ERA 2015 Peer Reviewer Handbook
1
ISBN: 978-0-9924254-7-0
© Commonwealth of Australia 2015
This publication is available for your use under a Creative Commons BY Attribution 3.0 Australia
license, with the exception of the Commonwealth Coat of Arms, the Australian Research Council
(ARC) logo, the Excellence in Research for Australia (ERA) logo, images, signatures and where
otherwise stated.
Use of ARC material under a Creative Commons BY Attribution 3.0 Australia license requires you to
attribute the work. Attribution is not to be done in any way that suggests that the ARC endorses you
or your use of the work. The ARC prefers the following attribution: Australian Research Council
material used ‘as supplied’.
Provided you have not modified or transformed ARC material in any way the following attribution is
preferred: Source: The Australian Research Council, Excellence in Research for Australia.
If you have modified, transformed, or derived new material from the ARC in any way, the ARC
prefers the following attribution: Based on Australian Research Council, Excellence in Research for
Australia data.
Requests and enquiries regarding this licence should be addressed to ARC Legal Services on +61 2
6287 6600.
Front Cover Image Credits:
Blue green wave stream texture background iStockphoto.com / © rionm
Way energy iStockphoto.com / © alengo
Water splash iStockphoto.com / © kirstypargeter
Green leaves iStockphoto.com / © Zaharov
1
Table of Contents
1.
Introduction .................................................................................................................................... 4
1.1 Overview of the Peer Reviewer Role ............................................................................................ 4
2.
Undertaking Peer Review ............................................................................................................... 5
2.1 Preliminary Considerations ........................................................................................................... 5
2.2 Reviewer Expertise........................................................................................................................ 5
2.3 Selecting Outputs for Review ........................................................................................................ 5
2.4 Writing the Report ........................................................................................................................ 7
2.4.1 Approach ................................................................................................................................ 8
2.4.2 Contribution ........................................................................................................................... 8
2.4.3 Quality Distribution ................................................................................................................ 9
2.5 What Makes a Good Report? ........................................................................................................ 9
3.
Peer Reviewer Assignment and Responsibilities .......................................................................... 11
3.1 Assignment of Peer Review Tasks ............................................................................................... 11
3.2 Confidentiality ............................................................................................................................. 11
3.3 Conflict of Interest (COI) ............................................................................................................. 12
3.4 Other sensitivities ....................................................................................................................... 12
3.4.1 Commercially sensitive research outputs ............................................................................ 12
3.4.2 Culturally sensitive research outputs................................................................................... 12
3.4.3 Australian Government Security Classified research outputs ............................................. 12
3.5 Copyright ..................................................................................................................................... 13
3.6 Research Integrity and Research Misconduct............................................................................. 13
4.
System to Evaluate the Excellence of Research (SEER) ................................................................ 14
4.1 Introduction to SEER ................................................................................................................... 14
4.2 Logging into SEER ........................................................................................................................ 14
4.2.1 Incorrect username or password entered ........................................................................... 15
4.2.2 Resetting your password before login ................................................................................. 15
4.2.3 Requesting your username before login.............................................................................. 16
4.2.4 Logging out ........................................................................................................................... 16
4.3 SEER Help .................................................................................................................................... 16
4.4 The Launch Pad ........................................................................................................................... 17
4.5 Conducting Peer Review in SEER ................................................................................................ 17
4.5.1 The ‘Task Overview’ screen ................................................................................................. 17
4.5.2 The Task List ......................................................................................................................... 18
2
4.5.3 View UoE .............................................................................................................................. 19
4.5.4 Evaluation Tab – Peer Review Report .................................................................................. 21
5.
ERA Background ............................................................................................................................ 23
5.1 Overview ..................................................................................................................................... 23
5.2 The ERA Reviewers ...................................................................................................................... 23
5.3 Definition of Research................................................................................................................. 23
5.4 Field of Research (FoR) Codes ..................................................................................................... 23
5.5 Unit of Evaluation ....................................................................................................................... 24
5.6 Research Output Types ............................................................................................................... 25
5.7 Research Statements .................................................................................................................. 25
Appendix 1: Eligible Institutions and ERA Institution Identifiers .......................................................... 26
Appendix 2: Fields of Research code summary .................................................................................... 28
Appendix 3: Research Statement for Non-Traditional Research Outputs ............................................ 32
Appendix 4: SEER Supported Browsers................................................................................................. 33
Appendix 5: IT Troubleshooting ............................................................................................................ 34
Appendix 6: Abbreviations .................................................................................................................... 37
Appendix 7: Glossary ............................................................................................................................ 38
3
1. Introduction
Thank you for agreeing to assist the Australian Research Council (ARC) with Excellence in Research
for Australia (ERA) in 2015. Peer judgement is central to the evaluation processes and the ARC is
grateful to the large number of reviewers, whose contributions are central to the success of ERA and
the continued confidence of the sector and others in the outcomes.
The ERA 2015 Peer Reviewer Handbook details the peer review methodology used by the ARC in ERA
2015. It should be interpreted as consistent with the ERA 2015 submission and evaluation
documentation available on the ARC’s ERA website.
1.1 Overview of the Peer Reviewer Role
Peer reviewers are assigned to review the submitted work (the pool of nominated peer review
outputs) for a Unit of Evaluation (UoE—see Section 5.5 for an explanation about UoEs), and to
report their judgement of the quality of that work to the Research Evaluation Committee (REC)
members.
The specific roles and responsibilities of a peer reviewer are to:
•
evaluate assigned material and provide a report using the peer review template
•
be diligent in completing tasks allocated to them
•
exercise due skill and care in the performance of their responsibilities
•
abide by confidentiality requirements
•
identify instances where they may have a conflict of interest (COI), raise these with the ARC prior
to conflict occurring and comply with the directions of the ARC relating to the management of COI
•
note (and appropriately manage with the ARC as necessary) any relevant sensitive research
outputs.
The peer review process opens on 9 June 2015 and closes on 27 July 2015. The total time available
for peer reviewers is approximately seven weeks although it is expected that the work will only
require a proportion of that time. All peer reviewers will need to review their assigned research
outputs and complete peer review reports within this timeframe. There is no possibility for
extension.
All communication by ERA peer reviewers on all aspects of ERA should be directed by email to
[email protected].
4
2. Undertaking Peer Review
2.1 Preliminary Considerations
•
Peer reviewers will be advised when the ERA IT System—System to Evaluate the Excellence of
Research (SEER)—is open and assigned tasks are available.
•
The ARC will advise you if a new task is assigned to you after the commencement of evaluation
(this could occur due to declared conflicts of interest, but is unlikely).
•
Peer reviewers should consider whether they may have an inability to assess any assigned
items due to a conflict of interest (see Section 3.3) and contact the ARC as soon as possible to
enable reassignment of the UoE, if required.
•
Peer reviewers should also check sensitivity handling notes (see Section 3.4) attached to their
assigned UoEs:
 If you identify research output materials that cause offence or serious sensitivity, you should
raise this with the ARC as soon as practicable to allow withdrawal and reassignment of the
task. An example of sensitivity may include personal beliefs offended by the research or
other sensitivities that may significantly affect your expert judgement of the peer review
outputs for a particular UoE.
•
Peer reviewers must submit all reports in SEER by 5pm Australian Eastern Standard Time on
27 July 2015. There is no possibility of an extension.
2.2 Reviewer Expertise
You have been assigned each UoE based on your six-digit expertise within the Field of Research (FoR)
(see Section 3.1 for details on peer reviewer assignment).
The peer review report form (see Section 4.5.4 for the full form) asks that you indicate your
expertise for each assigned UoE. You are asked to report on the level of your own expertise in terms
of the discipline (i.e. four-digit code) on a scale of one (lowest expertise) to five (highest expertise).
The assumption is that a peer reviewer who rates expertise at 5 is well-qualified in the discipline to
comment on the assigned work. This information will assist REC members to incorporate peer
reviewer reports into the overall evaluation of the UoE under consideration.
2.3 Selecting Outputs for Review
Peer reviewers will see the list of UoEs that they have been assigned after logging in to SEER.
•
Each assigned task involves the review and evaluation of a sample of nominated research
outputs associated with a particular UoE.
•
Each UoE includes a target number of outputs that you should aim to read as a part of your peer
review task:
5
 This number takes into account the expertise of the other peer reviewers assigned to a UoE,
as well as the breadth and size of the peer review sample;
 You should aim to read the target number of outputs. This may include outputs with which
you are already familiar, and which you may not have to re-read through SEER.
•
Each UoE will show as a list of items available for peer review for that UoE. Peer reviewers must
indicate each output they read (including those with which they are already familiar) by marking
that output as ‘Read’ in SEER.
By using the ‘Read’ column peer reviewers will be able to track their progress through the research
outputs they are sampling in the UoE. It is information that the REC members will use to determine
which outputs have contributed to a peer review report. It will also direct REC members towards any
additional reading that may be needed during subsequent stages of evaluation to ensure a broad
range of the outputs submitted for peer review have been read.
By ticking the box, the peer review report will auto populate as follows:
6
The peer review report includes a section for you to describe your sampling strategy. It is useful for
the REC members to have an indication of the range of output types you have read, to what extent
you were familiar with these outputs prior to your evaluation, and to what extent these outputs
were within your disciplinary expertise. This information will assist REC members in applying the
information in your report to their task of evaluating the UoE as a whole.
Where UoEs span a range of disciplines that an individual ERA peer reviewer could not be expected
to cover (particularly with the 2-digit UoEs), you should tailor your selection so that you are not
assessing material significantly outside of your expertise. Other reviewers will be assigned to ensure
appropriate expertise for this UoE.
In some instances you will be the only reviewer reporting on a ‘block’ of outputs. If this is the case,
your reading and analysis will be critical in ensuring that as much of the nominated sample of
outputs as possible is considered by the REC members. In some instances you will be reporting on a
‘block’ of outputs which has been read and reviewed by another reviewer. Multiple assessments of
the same work, where it occurs, are still significant as they provide useful information for the RECs
to calibrate the judgements of quality which are made by reviewers.
What you bring to the evaluation of the research outputs is your understanding of the discipline
nationally and internationally, disciplinary publishing practices (the best/most appropriate outlets),
the major research issues for the discipline and generally where the frontiers of knowledge are.
When you look at the sample, you will see some research outputs that you know well and other
outputs which you will know less well or not at all.
You are reading to make an informed judgement about the collective quality of the work to which
you have been assigned.
2.4 Writing the Report
You will write one report for each assigned UoE. Each UoE will present a range of research topics,
approaches, levels of contribution and quality. Your written statements should be couched in terms
of the overall standard of quality of outputs reviewed, and specifically about a notional majority of
the outputs reviewed.
The peer review report consists of a textual response on the quality of the sample of outputs that
you will review, against the broad criteria of approach and contribution. The report form has a
separate section for each of the criteria. The report form has a limit of up to a maximum of 10 000
7
characters per criterion with no formatting characters for each criterion. It is expected that the
overall length of a report across all criteria will be between about 1000 and 2000 words.
2.4.1 Approach
Responding to this criterion allows you the opportunity to provide important disciplinary context for
the benefit of REC members not directly involved in the discipline. Approach is described as the
approach taken in the group of outputs reviewed, and may include answers to some or all of the
following questions.

What contextual information and common disciplinary practice is relevant for reviewing the
quality of research outputs that were sampled?

Are the methodologies appropriately articulated for work in the discipline?

Are the methodologies appropriate for the research questions or not? If not, what are the
limitations/effects?

Are the venues/outlets appropriate for reporting the research in this discipline?

What does the overall approach taken demonstrate about the quality of the research sampled?
2.4.2 Contribution
This is the section where REC members will be looking for your judgement about the quality of the
work which has been reviewed by you. Contribution is described as the contribution of the group of
outputs reviewed to the field and/or practice. This may include reference to the following
questions.

Is the research timely?

Is the research (in terms of question and/or findings) significant?

What degree of originality and/or innovation is there in the research approach or research
questions?

What is the level of theoretical and conceptual rigour in the research?

Does the research demonstrate depth of discussion and analysis?

Has the research been subsequently used by others?

What is the contribution nationally and/or internationally to the further development of
knowledge and understanding?
8
2.4.3 Quality Distribution
The final task is to report how the quality of the work is distributed across the sampled reviewed.
The scale from 1 (the lowest quality) to 4 (the highest quality) is intended to be a banding rather
than a series of fixed points. What that means is that each tier allows for a range of performance.
The expectation is that written analysis in the report will align with and reflect the proportions of
quality you have recorded across the quality distribution scale.
The quality distributions will be used as a guide for RECs in determining ERA ratings.
2.5 What Makes a Good Report?
The peer review reports provide significant evidence for the RECs to determine ERA ratings. Each
UoE rating (in peer review disciplines) is a judgement of the committee based on the collection of
peer review reports for that UoE and the other indicators included in the ERA submission. Based on
feedback from REC members who participated in ERA 2010 and ERA 2012, the following guidance is
provided to assist peer reviewers in developing peer review reports that are useful for RECs in
determining ERA ratings:
1. Address the criteria: The report form includes the criteria (approach and contribution), which
REC members will expect to be addressed in your report. They are the criteria which REC
members find most helpful in making a determination about the quality of the work, which has
been presented for peer review. Address the criteria directly in your report and use the terms
(originality, significance of the research questions etc.) which are in the report template as far as
possible. This will allow REC members to more readily draw comparisons with the views of
others who are reviewing the same sample of outputs.
2. Provide supporting evidence and reasoning for your judgement: REC members will need to
understand how you have constructed your views in order to be confident about your
conclusions. The REC members may not be reading the same outputs as you and may not have
your expertise, so your report needs to have evidence which supports your conclusions. Avoid
brief unsupported summative statements. A typical report will be 1000 to 2000 words in total.
3. Draw your report to a clear conclusion: The overarching question which REC members will ask is
‘what is the judgement of this reviewer about the overall quality of the outputs reviewed’. Each
sample reviewed will have work which ranges in quality. The REC members are seeking your
conclusion about where on a scale of ‘lowest’ to ‘highest’ quality the majority of the work
reviewed by you is located both in a ‘distribution’ report and in the text of your report.
4. Write collectively about the work reviewed: There may be instances where reference to an
individual work or to the work of an individual researcher may help to explain a larger point
(such as where there are pockets of excellent work in an otherwise low quality sample), but the
REC members are most interested in judgements of overall quality.
5. Write to inform a broad academic audience: Your report is advice to the REC members, and so
inevitably you will be writing for an audience with a range of expertise and familiarity with the
work which is being reviewed. Your report will also be available to and used by other members
9
of the REC who may not be as familiar as you with the work, approaches, publishing practices
and so on.
6. Avoid referring to additional evidence outside of the peer review sample: Your report should be
based on your reading of the sample of research outputs within the context of the discipline.
While reference to disciplinary practices is important, you should avoid discussion of citation
information, journal rankings, and similar additional evidence as RECs will ignore this
information when considering the peer review reports.
7. Write within the scope of your expertise: Reviewers are given a target number of outputs to
review and are instructed to select work for review which is most relevant to their expertise.
The expectation of REC members in appointing you to peer review is that you are an expert and
able to provide informed comment on the work, which you have selected for review. You
should generally write within the scope of your expertise, but be explicit in your report where
your expertise ends so that REC members can contextualise your comments, and know where to
consult the reports of other reviewers.
8. Use the report only for its intended purpose: The report is only intended for the information of
the REC members about the sample of work which you have reviewed. Other issues reported by
you are likely to detract from the effect of your report. If you have questions or concerns you
should email the ARC directly at [email protected]—it is the responsibility of the ARC, not the peer
reviewers or REC members, to follow up on these questions/concerns.
9. Write as if your report is going to be made publicly available: Be as critical as is necessary, but
couch your criticisms in terms that you would be happy to have published, or that you might
expect of your work from others. The reports are developed as advice to the RECs and the ARC
does not intend to release the peer review reports either publicly or to the reviewed groups.
However, there may be circumstances where the ARC is required to release the reports to
others.
10
3. Peer Reviewer Assignment and Responsibilities
3.1 Assignment of Peer Review Tasks
ERA peer review adheres to the following principles:

in all UoEs where external peer review is required, at least two peer reviewers must be assigned
(as a minimum—more will be assigned in larger UoEs)

assignment of peer reviewers will account for sub-disciplines within each four-digit Field of
Research, and peer reviewers are identified for their sub-disciplinary (six-digit FoR) expertise

REC members assigned to a UoE will also be required to conduct peer review.
In total, a minimum of five individuals will undertake peer review of a four-digit UoE (at least two
external peer reviewers, as well as three assigned REC members)1. The mix of REC members and
external peer reviewers is intended to provide a breadth of perspectives, but include enough
expertise to ensure that standards reflect the views of experts in the field, and are commonly
understood and implemented.
The ERA evaluation processes have been designed to deliberately exclude reviewers from the
consideration of submissions where they have an identified COI as per Section 3.3.
You will be assigned UoEs and provided with a target number of outputs in each allocated UoE. A
target number for each reviewer normalises the workload and is designed to ensure that the
majority of the nominated peer review outputs are reviewed by at least one person in each UoE.
The maximum number of UoEs that you can be assigned to is restricted to 6 and the maximum
number of outputs that you can be assigned to is monitored in assignment.
You are likely to be assigned to multiple UoEs. This will enable you to include a degree of
comparison in your evaluations and develop a greater familiarity with the peer review criteria.
3.2 Confidentiality
Peer reviewers are required to comply with the ARC Conflict of Interest and Confidentiality Policy
which is available on the ARC website.
Peer reviewers are required to sign a confidentiality agreement with the ARC prior to their
participation in ERA. The agreement covers all aspects of their work with ERA, and the agreement
survives the conclusion of their engagement for the purposes of ERA. There is no restriction on peer
reviewers informing their institution that they are involved in ERA peer review, for example for the
purposes of workload management.
Peer reviewers may not contact researchers and/or institutions under any circumstances in relation
to material that has been submitted for evaluation in ERA, or seek additional information from any
sources. Peer reviewers may not reveal details about any evaluation at any time.
1
For four-digit UoEs, three REC members are assigned; in the case of two-digit UoEs, additional REC members
may be assigned.
11
3.3 Conflict of Interest (COI)
A COI is any situation where a peer reviewer has an interest which conflicts, might conflict, or may
be perceived to conflict with the interests of ERA. Examples of COI include:

being employed by, or holding an adjunct or honorary appointment at, the institution which has
made the submission which is being assigned

having a close personal relationship with someone whose work is significantly incorporated in
the UoE or peer review task being assigned for evaluation—this could include a partner, spouse,
family member or close friend. Included in this category is enmity

being a close collaborator with someone whose work is significantly incorporated in the UoE or
peer review task which is being assigned for evaluation—for example, where a peer reviewer is a
close collaborator with authors for 10% or more of the total outputs of a UoE, that would
constitute a potential COI

other conflicts that a peer reviewer will need to raise and have clarified, including financial
interests (for example holding a company directorship, stock ownership or options, patents,
royalties, consultancy or grant).
While most COIs will be determined before assignment of evaluation tasks occurs, peer reviewers
may encounter material with which they have a potential COI during Evaluation and are required to
declare any potential or actual COI as soon as practicable after it has been identified. In such
circumstances, the ARC will address each instance on a case by case basis, usually by reassigning the
material to another reviewer.
A peer reviewer will never be involved in considerations about UoEs in any discipline from their own
institution, or any institution with which they have a declared COI.
3.4 Other sensitivities
To be eligible for ERA, all research outputs must either be published or made publicly available in the
ERA reference period. However, if any research material causes offence or serious sensitivity to a
peer reviewer, they are asked to raise their concern with the ARC as soon as practicable. In this case
the UoE would normally be reassigned.
Universities have been requested to identify, and provide sensitivity handling notes, for any research
outputs considered sensitive based on the following characteristics.
3.4.1 Commercially sensitive research outputs
A research output that includes commercially sensitive information may be included as part of a
submission provided the necessary permissions have been obtained. This will be flagged to peer
reviewers.
3.4.2 Culturally sensitive research outputs
A research output that is culturally sensitive may be included as part of a submission provided that
the ARC has been appropriately advised of the sensitivities. This will be flagged to peer reviewers.
3.4.3 Australian Government Security Classified research outputs
A research output that includes information classified in line with the Australian Government
Protective Security Manual as either ‘In-Confidence’ or greater, or ‘Restricted’ or greater, cannot be
12
included in a submission (this also includes outputs subsequently classified as ‘Sensitive’, ‘For Official
Use Only’, or greater under the Australian Government Security Classification System).
3.5 Copyright
Peer reviewers have access to, and use of, relevant research outputs to conduct ERA peer review.
Acting under section 183(1) of the Copyright Act 1968 (Cth), the Commonwealth of Australia, as
represented by the ARC, has authorised each peer reviewer to do acts comprised in the copyright of
relevant material for the purposes of ERA. As a result, authorised peer reviewers may make all uses
of relevant material that are necessary or convenient to enable their participation in ERA. The
authorisation is strictly limited to their participation in ERA and will not extend to uses for any
purpose unrelated to participation in ERA.
Access to research outputs is provided strictly for the purposes of conducting evaluation for ERA.
Peer reviewers are not permitted to reproduce or distribute the outputs for any purpose other than
participation in ERA. To ensure appropriate protection of copyright material in ERA submissions,
peer reviewers must at all times comply with the authorisation.
3.6 Research Integrity and Research Misconduct
ERA peer reviewers are encouraged to read the ARC Research Integrity and Research Misconduct
Policy and report alleged breaches of research integrity or research misconduct issues identified in
relation to ARC funded business as per this policy to the ARC Research Integrity Officer.
The policy and contact details for the Research Integrity Officer are available on the ARC website.
The Research Integrity Officer will refer the allegation to the relevant institution for investigation in
accordance with the requirements of the Australian Code for the Responsible Code of Research.
Sufficient information should be provided to enable the institution to progress an investigation into
the allegation (if required).
13
4. System to Evaluate the Excellence of Research (SEER)
4.1 Introduction to SEER
Peer reviewers will conduct all of their work online via the ERA IT system, SEER.
SEER is the web-based environment within which peer reviewers access assigned UoEs, and the
sample of peer review outputs nominated for each UoE. Institutions nominate a sample of outputs
for each assessable UoE and provide those outputs electronically via an institutional repository.
SEER is the system where the assessment of outputs takes place and peer review assessments are
recorded.
Selecting an assigned UoE in SEER will provide access to the list of research outputs that have been
made available for peer review by the submitting institution. Clicking on the links will provide access
to individual outputs.
Peer reviewers are able to access the nominated outputs through SEER at any time during the
evaluation process. SEER ensures the anonymity of peer reviewers accessing institutional
repositories.
The minimum computer browser requirements you will need to access SEER are explained at
Appendices 4 and 5.
4.2 Logging into SEER
Peer reviewers have already been provided with a SEER username and password by the ARC at the
time of their initial online agreement. Access to SEER will be provided by the ARC as soon as the
evaluation phase of ERA opens. On the SEERLOGIN page, enter the username and password
provided to you. Your username is not case-sensitive; your password is.
14
4.2.1 Incorrect username or password entered
If an incorrect username or password is entered, the following failed verification message will
appear:
Please note: if the incorrect username or password is entered more than three times your account
will be locked and you will need to contact the ARC to unlock your user account.
4.2.2 Resetting your password before login
To reset your password select the Reset Password tab on the SEERLOGIN page and enter your
username. By clicking ’send’, an automated email will be sent to you with your new temporary
password, even if your account has been locked. You will then be required to enter in, and then
confirm, a new password. The password must have a minimum of 10 characters and contain at least
three of the following character sets: lowercase alphabetic characters (a–z), uppercase alphabetic
characters (A–Z), numeric characters (0–10) and special characters.
You can also reset your own password at any time within your individual SEERMANAGE USERS
system.
15
4.2.3 Requesting your username before login
If you have forgotten your username, you can request that it be sent to the email address you used
to register with the ARC. This will also reset your password. By clicking ‘send’, an automated email
will be sent to you with your username and new temporary password. You will then be required to
enter in, and then confirm a new password.
4.2.4 Logging out
The ‘Logout’ button is located in the top right-hand corner of the SEER Launchpad, as indicated by
the red arrow in the screen shot below.
Please note: the SEER system will automatically log you out after one hour of inactivity. SEER does
not “auto-save” work before the automatic logout and any changes made since you last clicked save
will be lost.
4.3 SEER Help
Once you have logged into the system, SEER provides a help document that can be accessed via the
help tab. This help button can be located on the top left-hand corner of the page as indicated by the
red arrow in the screen shot below.
16
The Help content includes step-by-step instructions which will assist you in navigating through the
ERA 2015 process within SEER. For all other inquiries, please consult this Handbook or email the peer
reviewer helpdesk during business hours (Australian Eastern Standard Time)—[email protected]. See
Appendix 5 for IT Troubleshooting.
4.4 The Launch Pad
On successful login you are directed to the SEERLAUNCH PAD. You are presented with the modules
that you are able to access.
The relevant modules are:

SEEREVALUATE—all four-digit (and any two-digit) UoEs assigned to you for peer review; and

SEERMANAGE USERS—the ability to manage your own SEER account.
4.5 Conducting Peer Review in SEER
4.5.1 The ‘Task Overview’ screen
On selecting SEEREVALUATE you will be taken to the evaluation tasks screen which will present the
screen shown below. This indicates the number and status of tasks to be completed at the four- and
two-digit level. A ‘task’ is defined as one UoE that you have been allocated to complete. Your work
is complete when the ‘Outstanding Tasks’ counter is at zero. All assigned tasks must be submitted by
the due date of 27 July 2015.
17
You will need to then click onto the ‘FOUR DIGIT’ tab which will display the four-digit UoEs (tasks)
that have been assigned to you. If you have been allocated any two-digit UoEs, you will also note the
presence of a ‘TWO DIGIT’ tab where you will access any assigned two-digit UoEs.
You can return to the ‘LaunchPad’ at any time by selecting the ‘Launchpad’ button located in the top
left hand corner of your screen.
4.5.2 The Task List
The task list will present the list of UoEs that have been assigned to you; the screen is shown below.
You can switch between the four-digit and two-digit codes by clicking on the relevant tab.
At the top right, a search functionality has been included to search by FoR code or UoE ID.
18
You are able to sort the task list by clicking on the headers on all columns apart from ‘View UoE’
column.
The button circled at the bottom right hand of the screen shot will allow you to ‘SUBMIT ALL’ UoEs.
This submits all the UoEs that are in the ‘Valid’ state. If you select to do this a message will prompt
you to confirm the action. Once submitted, you will no longer be able to revise your text for the
submitted UoEs.
Please note at the top right hand side of the screen under your name is an indicator of the closing
date for this stage. Please ensure you have completed your evaluations and that you submit those
evaluations prior to the closing date. An extension beyond the closing date is not possible.
You will be automatically logged out of SEER after 60 minutes of inactivity. If you have finished with
your session it is recommended you always use the logout button at the top right hand side of the
screen.
4.5.3 View UoE
When you select to view a UoE from the list of UoEs, you will be presented with the items available
to you for peer review for that UoE. You are able to sort this list by clicking on the headers on all
columns.
Column heading descriptions:

Authors—the author(s) of the item from the submitting institution

Title—the title of the output

Type—indicates the type of output eg: journal article, book, conference paper

Detail—any further detail on where or how the output is published
19

Year—year of publication

Apport.—apportionment; meaning the percentage an item has been apportioned to this Field of
Research

Sensitivity type—to identify if there is any sensitivity, for example, of a cultural or confidential
nature

Links—click on the link to gain access to the output in an online repository
–
an ‘Item’ link should link directly to a research output in the institutional repository
–
a ‘meta’ link will link to a metadata page within the institutional repository.

Research Statement—used for non-traditional research outputs and portfolios (see Appendix 3
for further information about Research Statements)

Read?—this is to indicate that you have read this output (including those outputs in your sample
with which you are already familiar). Please tick this box for each output which you have
reviewed—for selection of outputs to review, please refer to Section 2.3.
20
4.5.4 Evaluation Tab – Peer Review Report
By clicking on the evaluation tab, you will find the peer review report template with check boxes to
indicate your expertise in the UoE under consideration and text fields for your comments under the
various report headings.
21
Each comment can be up to a maximum of 10 000 characters (there is no formatting in SEER),
although it is not expected that you will use the maximum character limit.
You will also note the ‘number of outputs reviewed’ table which populates from the items you have
ticked as ‘read’ for this UoE (see Section 2.3 for further information). This indicates to REC members
the extent of your review.
You are able to ‘save’ or ’save and submit’ your expertise and comments. When you ‘save’ an entry
and/or comment in the form, the details are saved and the status potentially changes on the ‘task
overview’ screen.
Only when an evaluation has both an expertise indicated and a comment, i.e. is ‘valid’, are you able
to ‘save and submit’. You will be asked to confirm your request when you select to ’save and
submit’.
Once an evaluation is submitted you are no longer able to edit it. If you ‘save and submit’ but have
not completed your evaluation, please contact the ARC at [email protected] to request the evaluation
be re-opened.
22
5. ERA Background
5.1 Overview
ERA is an independent assessment of the quality of the research conducted within Australian higher
education institutions2 on a discipline-by-discipline basis. ERA aims to identify and promote
excellence across the full spectrum of research activity.
The objectives of ERA are to:
1. Establish an evaluation framework that gives government, industry, business and the wider
community assurance of the excellence of research conducted in Australia’s higher education
institutions.
2. Provide a national stocktake of discipline-level areas of research strength and areas where there
is opportunity for development in Australia’s higher education institutions.
3. Identify excellence across the full spectrum of research performance.
4. Identify emerging research areas and opportunities for further development.
5. Allow for comparisons of Australia’s research nationally and internationally for all discipline
areas.
5.2 The ERA Reviewers
Evaluations in ERA are undertaken by Research Evaluation Committees (RECs) comprised of
experienced, internationally-recognised experts. A suite of discipline-specific indicators informs the
RECs’ evaluations. There are eight RECs in total, and each REC consists of a Chair and around 15-25
expert reviewers who are the REC members. A list of members for ERA 2015 is available on the ARC
website.
In addition, the ARC engages peer reviewers to assist with the process of evaluation to ensure that a
breadth of relevant expertise, experience and perspective contribute to each evaluation outcome.
5.3 Definition of Research
For the purposes of ERA, research is defined as the creation of new knowledge and/or the use of
existing knowledge in a new and creative way so as to generate new concepts, methodologies,
inventions and understandings. This could include synthesis and analysis of previous research to the
extent that it is new and creative.
Institutions must ensure that all research outputs submitted to ERA meet this definition of research.
Outputs that do not meet this definition may be excluded from submissions during the ERA
submission process or, where they are not excluded from submissions, their inclusion may adversely
affect the quality rating assigned by RECs during the evaluation process.
5.4 Field of Research (FoR) Codes
For the purposes of ERA, disciplines are defined as four-digit and two-digit FoRs as identified in the
Australian and New Zealand Standard Research Classification (ANZSRC). The ANZSRC provides 22
two-digit FoR codes, 157 four-digit FoR codes, and an extensive range of
2
For a full list of eligible institutions and the institutional identification codes used for ERA purposes, see
Appendix 1.
23
six-digit codes. The ANZSRC was released in 2008 by the Australian Bureau of Statistics (ABS) and
Statistics New Zealand. The ANZSRC is available in full from the ABS website. The structure of FoRs is
as follows:

Two-digit FoR Code: This is the highest level of the ANZSRC hierarchy. A two-digit FoR code
relates to a broad discipline field. A two-digit FoR code consists of a collection of related fourdigit FoR codes.

Four-digit FoR Code: This is the second level of the ANZSRC hierarchy. A four-digit FoR code is a
specific discipline field of a two-digit FoR code. A four-digit FoR code consists of a collection of
related six-digit FoR codes.

Six-digit FoR Code: This is the lowest level of the ANZSRC hierarchy. A six-digit code is a further
breakdown of a four-digit FoR code.
Example of the FoR Hierarchy
5.5 Unit of Evaluation
ERA evaluation occurs at both the four-digit and two-digit FoR code levels. A UoE for ERA is the
research discipline for an eligible institution, defined by four-digit and two-digit FoR codes.
Data for ERA is submitted by institutions at the four-digit FoR code level, and is aggregated to create
four-digit and two-digit UoEs. Four-digit and two-digit UoEs are only assessed where there is a
meaningful level of data to be evaluated. For disciplines where peer review is used, no evaluation
will be conducted for the FoR at a given institution where, over the six year reference period, there
are fewer than the equivalent of 50 submitted research outputs. Books are given an effective
weighting of 5:1 compared with other research outputs for the purposes of determining the low
volume threshold in these disciplines; for other purposes in ERA, including peer review, they are
counted as a single research output.
Peer reviewers in ERA are engaged for their disciplinary expertise and are normally only assigned to
peer review work at the four-digit FoR code level. There may be some instances, however, where
work at the two-digit level is also assigned to peer reviewers. For example for some FoRs at some
institutions, there may be insufficient research volume to undertake a valid analysis at the four-digit
24
level, but sufficient research volume at the two-digit level. In these instances, evaluation will take
place at the two-digit FoR code level only.
Six-digit FoR data is not collected in ERA and evaluation is not conducted at this level. However, for
the purpose of identifying sub-disciplinary expertise within a UoE, the ARC requests this information
from peer reviewers. This assists in assigning work appropriately from a UoE for consideration by
peer reviewers.
5.6 Research Output Types
The research outputs which will be available for peer review include the traditional range of
academic outputs including journal articles, books, book chapters, and conference publications. As
well, ERA includes a range of non-traditional research outputs (NTROs) for some disciplines (see the
ERA 2015 Discipline Matrix for details). This takes account of, for example, research in the creative
arts which ranges from the experimental, involving the production of creative works, through to the
analytical, involving the study of particular subjects.
The NTRO sub-categories include:

Original Creative Works

Live Performance of Creative Works

Recorded/Rendered Creative Works

Curated or Produced Substantial Public Exhibitions and Events

Research Reports for an External Body.
These outputs may be submitted as individual items or, where individual works are derived from the
same underlying research endeavour but do not in themselves constitute research, they may be
submitted as a ‘portfolio’, which in ERA constitutes a single non-traditional research output. A
portfolio submitted in ERA has to demonstrate coherent research content.
5.7 Research Statements
When an NTRO is submitted to ERA for evaluation, the research component of the work may not be
immediately clear to peer reviewers. To assist peer reviewers institutions are asked to write a
research statement describing the research component of the output. The Research Statement will
include reference to the following:

research background—field, context and research question

research contribution—innovation/new knowledge

research significance—evidence of excellence.
Further details on the requirements for research statements are included in Appendix 3.
25
Appendix 1: Eligible Institutions and ERA Institution Identifiers
University
Acronym
Australian Catholic University
ACU
Batchelor Institute of Indigenous Tertiary Education
BAT
Bond University
BON
Central Queensland University
CQU
Charles Darwin University
CDU
Charles Sturt University
CSU
Curtin University of Technology
CUT
Deakin University
DKN
Edith Cowan University
ECU
Federation University Australia
FED
Flinders University
FLN
Griffith University
GRF
James Cook University
JCU
La Trobe University
LTU
Macquarie University
MQU
Monash University
MON
Murdoch University
MUR
Queensland University of Technology
QUT
RMIT University
RMT
Southern Cross University
SCU
Swinburne University of Technology
SWN
The Australian National University
ANU
The University of Adelaide
ADE
The University of Melbourne
MEL
26
University
Acronym
The University of New England
UNE
The University of New South Wales
NSW
The University of Newcastle
NEW
The University of Notre Dame Australia
NDA
The University of Queensland
QLD
The University of Sydney
SYD
The University of the Sunshine Coast
USC
The University of Western Australia
UWA
University of Canberra
CAN
University of Divinity
DIV
University of South Australia
USA
University of Southern Queensland
USQ
University of Tasmania (incorporating Australian Maritime College)
TAS
University of Technology, Sydney
UTS
University of Western Sydney
UWS
University of Wollongong
WOL
Victoria University
VIC
27
Appendix 2: Fields of Research code summary
This Appendix of the Handbook provides an overview of the FoR codes which have peer review as an
identified indicator, organised by the two-digit FoR codes. The ERA 2015 Discipline Matrix shows the
complete set of indicators which are relevant for each FoR code.
FoR
Description
01
Mathematical Sciences
0101
Pure Mathematics
08
Information and Computing Sciences
0801
Artificial Intelligence and Image Processing
0802
Computation Theory and Mathematics
0803
Computer Software
0804
Data Format
0805
Distributed Computing
0806
Information Systems
0807
Library and Information Studies
0899
Other Information and Computing Sciences
10
Technology
1005
Communications Technologies
1006
Computer Hardware
12
Built Environment and Design
1201
Architecture
1202
Building
1203
Design Practice and Management
1204
Engineering Design
1205
Urban and Regional Planning
1299
Other Built Environment and Design
28
FoR
Description
13
Education
1301
Education Systems
1302
Curriculum and Pedagogy
1303
Specialist Studies in Education
1399
Other Education
14
Economics
1401
Economic Theory
1402
Applied Economics
1403
Econometrics
1499
Other Economics
15
Commerce, Management, Tourism and Services
1501
Accounting, Auditing and Accountability
1502
Banking, Finance and Investment
1503
Business and Management
1504
Commercial Services
1505
Marketing
1506
Tourism
1507
Transportation and Freight Services
1599
Other Commerce, Management, Tourism and Services
16
Studies in Human Society
1601
Anthropology
1602
Criminology
1603
Demography
1604
Human Geography
1605
Policy and Administration
29
FoR
Description
1606
Political Science
1607
Social Work
1608
Sociology
1699
Other Studies in Human Society
18
Law and Legal Studies
1801
Law
1802
Maori Law
1899
Other Law and Legal Studies
19
Studies in Creative Arts and Writing
1901
Art Theory and Criticism
1902
Film, Television and Digital Media
1903
Journalism and Professional Writing
1904
Performing Arts and Creative Writing
1905
Visual Arts and Crafts
1999
Other Studies in Creative Arts and Writing
20
Language, Communication and Culture
2001
Communication and Media Studies
2002
Cultural Studies
2003
Language Studies
2004
Linguistics
2005
Literary Studies
2099
Other Language, Communication and Culture
21
History and Archaeology
2101
Archaeology
2102
Curatorial and Related Studies
30
FoR
Description
2103
Historical Studies
2199
Other History and Archaeology
22
Philosophy and Religious Studies
2201
Applied Ethics
2202
History and Philosophy of Specific Fields
2203
Philosophy
2204
Religion and Religious Studies
2299
Other Philosophy and Religious Studies
31
Appendix 3: Research Statement for Non-Traditional Research
Outputs
For non-traditional research outputs which are nominated for ERA peer review, a statement
identifying the research component of the output must be provided as part of the submission of an
institution. The statement must be no more than 2000 characters including spaces (around 250
words) and cannot contain any embedded links. Research statements should address the
following categories:
1. Research Background
 Field
 Context
 Research Question
2. Research Contribution
 Innovation
 New Knowledge
3. Research Significance
 Evidence of Excellence
The following is an example of an acceptable visual arts research statement:
Research Background
Current international developments in painting have identified the need to establish complex
forms for representing identity in terms of facial expression. While this research recognises
the significance of facial expression, it has overlooked the unstable nature of identity itself.
Research Contribution
The paintings Multiple Perspectives by Y address the question of the unstable nature of
identity as expressed in painterly terms through a study in unstable facial phenomenon using
the philosophical concept of ‘becoming’. In doing so it arrives at a new benchmark for the field
of research in understanding visual identity, namely that identity is not bound to stable facial
phenomena but, like other forms of meaning, is constantly undergoing change.
Research Significance
The significance of this research is that it overcomes barriers for visually understanding the
complex nature of identity and its expressive painterly possibilities. Its value is attested to by
the following indicators: selection of the painting for inclusion in the international exhibition
Documenta, Kassel, Germany; its inclusion as a case study in the renowned Courtauld
Institute, University of London, Issues in Contemporary Art graduate seminar series; its being
the subject of a chapter in the book Identity Reframed published by Thames and Hudson and
authored by the renowned art historian Z; its forming part of a competitively funded ARC
project.
REC members and ERA peer reviewers will evaluate non-traditional research outputs selected for
ERA peer review in the context of the research component as identified in the research statement.
32
Appendix 4: SEER Supported Browsers
Browser
Version
Platform
Level of Support
Internet
Explorer
Current and
the previous
release.
Any version of Windows compatible
with the stated Internet Explorer
versions.
Fully Supported
Google
Chrome
Current
release automatic
update must
not be
disabled.
Any operating system compatible with
Chrome.
Fully Supported
All other
browsers
and
versions
ARC applications should work in any browser or version not fully
supported, however as testing is restricted to those above, they will not be
guaranteed to function.
Mobile
Devices
(tablets,
mobile
phones,
etc.)
While ARC applications are not supported on mobile devices they should
work if the device has internet connectivity and is capable of running a
browser.
Web browsers must be configured to enable JavaScript, cookies and pop-up windows when
accessing SEER.
33
Appendix 5: IT Troubleshooting
The ARC works closely with institutions to ensure that links to outputs in a repository work and that
it is possible to download the outputs which have been nominated for peer review. There may be
instances where the link is broken or the output is not available, but this will be rare.
The most common download problems are with the reviewer’s internet browser settings, and most
can therefore be resolved quickly and easily by reviewers. The most commonly reported problems
are with pop-up blockers, Adobe, and downloads to temporary folders which are not immediately
apparent. The fix for common browsers for each of these problems is as follows:
1. You may not be prompted to either open or save the download
Internet Explorer
1) Open Internet Explorer
2) Go to the menu item Tools > Internet Options
3) Select the Security Tab
4) Click on Internet
5) Click on Custom Level button
6) Under the Downloads section, change “Automatic prompting for file downloads” to Enable and
click OK
If this does not assist, as a backup plan IE stores all downloads to a temporary folder
(C:\Users\username\AppData\Local\Microsoft\Temporary Internet Files)—this may be where
downloads are going.
Google Chrome
1) Open Google Chrome
2) Click on the Chrome menu on the browser toolbar
3) Select Settings
4) Select +Show advanced settings
5) Go to Downloads> Check Ask where to save each file before downloading
If you want the pdfs to always open after they have been downloaded, click the arrow next to the
file button in the downloads bar and select Always open files of this type
34
The Downloads page displays a chronological list of all the files you’ve downloaded. Follow these
steps to open the Downloads page: Click on the Chrome menu on the browser toolbar > Select
Downloads. To open a file, click the filename. To see where the file is located on your computer,
click Show in Folder.
Here is where you can find the temporary file storage location for a few common operating systems:
Windows Vista: C:\Users\[USERNAME]\AppData\Local\Google\Chrome\
Windows XP: C:\Documents and Settings\[USERNAME]\Local Settings\Application
Data\Google\Chrome\
Windows 7/8: C:\Users\[USERNAME]\AppData\Local\Google\Chrome\User Data\Default\Cache
Mac OS X: /Users/[USERNAME]/Library/Caches/Google/Chrome
2. Your Internet browser's pop-up blocker may be blocking the download. If you can disable it, this
may solve the problem.
Internet Explorer
A. Open Internet Explorer, and then click Internet Options on the Tools menu to open the Internet
Properties dialog box.
B. Click the Privacy tab, and then click to clear Block pop-ups to turn Pop-up Blocker off.
Google Chrome
A. Click on the Chrome menu on the browser toolbar
B. Select Settings
C. Click Show advanced settings
D. In the ‘Privacy’ section, click Content settings button
E. In the ‘Pop-ups’ section, select “Allow all sites to show pop-ups”. Customise permissions for
specific websites by clicking Manage exceptions
3. Your Internet browser may not recognise that the file the system is trying to send you is a PDF—
usually because an Adobe PDF plugin for your browser is absent or not working correctly. You
may need to update your PDF plugin. To install the latest update of Reader or Acrobat:
Internet Explorer /Chrome
A. Open Reader or Acrobat
B. Choose Help>Check for Updates. If a new update is available, it installs automatically.
35
4. The Adobe PDF plugin in your browser may be disabled.
Internet Explorer
The steps to enable the Adobe PDF plug-in vary depending on your version of Internet Explorer.
A. Select Tools > Manage Add-ons. (Or choose Tools>Internet Options, click the Programs tab, then
click Manage Add-ons)
B. In the Show pop-up menu, select All Add-ons. (In some versions the option is Add-ons That Have
Been Used by Internet Explorer)
C. Scroll to the section of add-ons for Adobe Systems, and select Abode PDF Reader
D. If the status of Adobe PDF Reader is set to Disabled, click on the Enable button.
Google Chrome
A. In the Chrome address bar, type chrome://plugins
B. Disable the Chrome PDF Viewer
C. Enable the Adobe Acrobat or Adobe Reader plug-in
D. Close the Plug-ins tab and restart Chrome
36
Appendix 6: Abbreviations
Acronym
Full Text
ARC
Australian Research Council
ANZSRC
Australian and New Zealand Standard Research Classification
COI
Conflict of Interest
ERA
Excellence in Research for Australia
FoR
Fields of Research (ANZSRC)
NTRO
Non-traditional Research Output
REC
Research Evaluation Committee
SEER
System to Evaluate the Excellence of Research
UoE
Unit of Evaluation

37
Appendix 7: Glossary
Phrase
Description
Discipline
For the purposes of ERA, ‘disciplines’ are defined as four- or two-digit FoR
codes as identified in the ANZSRC.
Discipline Matrix
Specification of which ERA indicators will be applied to which disciplines.
The ERA 2015 Discipline Matrix is available on the ARC website and made
available as part of the ERA-SEER 2015 Technology pack.
ERA peer review
Review conducted of a sample of research outputs by RECs and peer
reviewers as part of the ERA evaluation process.
ERA peer reviewer
Independent expert who undertakes ERA peer review of a sample of
research outputs as part of the ERA evaluation process.
Fields of Research
(FoR)
A hierarchical classification of fields of research set out in the ANZSRC. The
term ‘Fields of Research’ or ‘FoR’ applies to all three ANZSRC levels (twodigit, four-digit and six-digit).
Four-digit FoR
The middle level of the three hierarchical levels within ANZSRC Fields of
Research. An example of a four-digit FoR code is ‘0206-Quantum Physics’.
Within the ANZSRC classification, this level is referred to as a ‘Group’.
Institution
Higher education provider eligible to participate in ERA. For peer review
disciplines the low volume threshold is the equivalent of 50 submitted
apportioned outputs.
Low volume threshold A low volume threshold exists for each UoE in ERA.
Non-traditional
research output
(NTRO)
Peer review
Research outputs which do not take the form of published books, book
chapters, journal articles or conference publications.
Published
Published (in the case of traditional research outputs such as publications)
or made publicly available (in the case of non-traditional research outputs).
Reference periods
The periods during which research outputs must have been published,
research income reported under HERDC etc.; in order for associated data to
be included in ERA submissions. ERA reference periods vary according to the
research item.
Research Evaluation
Committees (REC)
The discipline grouping-specific committees which will undertake ERA
evaluations. Each such committee will include internationally-recognised
members with expertise in research evaluation and broad discipline
expertise.
For the purpose of ERA, an acceptable peer review process is one that
involves an assessment or review, before publication, of the research output
in its entirety by independent, qualified experts. Independent in this context
means independent of the author. Note that ‘ERA peer review’ has a
different meaning (see ‘ERA peer review’).
38
Phrase
Description
Research statement
For each NTRO or portfolio nominated for ERA peer review, institutions
must submit a research statement of 250 words identifying the research
component of the research output (ie. how the output meets the definition
of ‘research’).
Refers broadly to the higher education community and those individuals and
organisations who consider themselves affiliated the higher education
community.
Sector
Two-digit FoR
The highest of the three hierarchical levels within ANZSRC Fields of
Research. An example if ‘02 Physical Sciences’. Within the ANZSRC
classification, this level is referred to as a ‘Division’.
Unit of Evaluation
A discipline for a specific institution. In some contexts, the term refers to the
package of associated ERA information (including submission data,
indicators and evaluation outcomes). While all ERA data collection will be at
the four-digit FoR level for a specific institution, the UoE will be either at the
four-digit or two-digit FoR for an institution.
39