Guidelines for Best Practice in
Unsupervised Online Assessment
Introduction
CEB recognises that on-demand online solutions are now the reality in the assessment
of people at work. During 2005 alone, We administered over one million assessments
online. Organisations large and small are alert to the massive cost benefits of testing
candidates unsupervised online for selection, particularly when screening large numbers
of applicants during a recruitment programme. In this context, ‘unsupervised’ means
that there is no person present with the test taker to authenticate their identity and to
oversee them taking the test.
This unsupervised online environment has created a new landscape in which new issues
must be addressed to preserve the integrity of the test and the accuracy of the outcome.
Part of the benefit for test takers is that they take tests in their chosen environment – at
home, in an Internet café, or at work. Part of the benefit for organisations is the saving
on costs associated with traditional supervised assessment. However, it is difficult to
confirm an applicant’s true identity, or determine whether they have had assistance
when tests are completed without supervision.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
1
Organisations large and small are
alert to the massive cost benefits
of testing candidates unsupervised
online for selection.
The emergence of online testing has led to a debate about what is good practice (i.e. what
are the minimum standards and safeguards one should adopt) and what is ‘best practice’
(i.e. what should one aim for) when dealing with unsupervised assessment in high stakes
settings. The challenge faced by unsupervised testing online is achieving the correct
balance between minimising risk and maximising value.
The use of objective assessments must be properly managed in order to reap
the anticipated benefits. This becomes more important when tests are delivered
unsupervised over the Internet. If inappropriate tests are used, if the information from
tests is misunderstood, or if the process is not secure, the benefits can be quickly eroded.
Feedback from SHL clients and test users has highlighted the need to help practitioners
to do things better - help them to move in the right direction, to assess the risks and
mitigate them where possible. These guidelines provide a pragmatic, stepby-step set of
recommendations that fit into the reality of the world in which today’s organisations
operate.
These guidelines are not built on desk or academic research alone. Interviews with users
of assessments around the globe, including professional psychologists, consultants in the
recruitment and coaching markets, HR and line managers in both large and medium-size
organisations in the private and public sectors all contributed to the development of this
Guide.
Who Are These Guidelines For?
These guidelines are designed to assist HR practitioners who are developing
unsupervised online assessment programmes for the first time and practitioners who
already use online tests but are expanding or evaluating their use. In addition, these
guidelines are a useful reference for IT professionals who need to understand the
broader context around the introduction of an online testing programme.
Appropriate Use and More Information
This Guide highlights the areas that should be considered when designing online
assessment processes. This document, alongside the White Paper on this subject, should
be considered as a framework rather than an absolute reference containing all the
answers. A reference list of additional sources on the subject of online testing is in the
Appendix. More information can also be found at www.ceb.shl.com. CEB can supply
expert consultancy services if further guidance is needed.
In all cases, readers should ensure that the use of these guidelines and the development
of any procedures or policies comply with local legislation and/or professional practices.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
2
Structure of the Guide
The document takes a step-by-step approach to the development of unsupervised online
assessment programmes:
Lay the Foundations
Step 1 Evaluate if unsupervised testing online is right for your organisation
Step 2 Determine the objectives, stakes and scope of your programme
Step 3 Identify where your process needs to change and the impact of the change
Develop Policy and Procedures for Unsupervised Testing Online
Step 4 The four Better Practice principles for unsupervised testing online:
■■
Manage the security of the assessments
■■
Defend against cheating
■■
Build in verification procedures
■■
Establish and communicate an online assessment contract with the candidate
Step 5 Define new policies & procedures for unsupervised test usage in your
organisation
Implement the New Process
Step 6 How to apply the Better Practice principles:
a. Choosing a test provider
b. Things to look out for and avoid
c. Developing a new policy for unsupervised testing online
d. Responsibility towards test takers before, during and after the online test session
e. Guidelines on using cut scores
f. Cross checking results with other information
g. Retesting and verification testing
h. Supervising the verification test
i. Handling verification discrepancies
j. Monitoring test usage and results
k. Confidentiality, storage of results and access to test materials
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
3
Lay the Foundations –
Companies can realise many
benefits from online testing but
they must be carefully weighed
against the potential risks.
Step 1: Evaluate if unsupervised testing online is right for your organisation
The foundation of an effective unsupervised testing online policy is being clear on
whether or not it is appropriate for your organisation. Companies can realise many
benefits from online testing but they must be carefully weighed against the potential
risks.
Benefits
Earlier Decision-Making
Unsupervised online processes mean tests can be delivered to applicants during
preliminary stages, which means that information can be used for screening and
decision-making much earlier.
Saves Time
By identifying the best talent earlier and progressing them faster, the length of hiring
processes can be reduced. Online testing processes also save time by removing the need
for staff to physically administer tests until a later stage, and then for a smaller pool of
candidates.
Saves Money
Unsupervised testing online enables HR teams to do more hiring with the same HR staff.
There is no need to purchase, store and transport materials to assessment centres. Costs
are also saved through better hiring decisions, which reduce staff turnover and improve
performance.
Administrative & Practical Efficiency
The benefits of unsupervised testing online are particularly strong in high volume
recruitment scenarios. The process of accessing candidates in different locations and in
different languages is far easier. It requires little administrative effort and results can be
accessed faster.
A 3rd party workflow or Software-on-Demand provider can provide the online
platform to enable organisations to adopt this approach. (In an online testing context, a
‘Software-on-Demand’ provider owns and operates a software application that facilitates
the delivery and scoring of an assessment. The provider maintains and services the
application and makes the online assessments available to users via a browser, or within
a client application.)
Positive Candidate Experience
Applicants complete tests when it is convenient for them, in their preferred language,
with 24 hour accessibility 7 days a week. As a highly convenient, speedy and modern
process, online testing delivers a positive candidate experience, which can reinforce an
employer brand.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
4
International Consistency and Consistency Across Sites
Unsupervised online testing offers organisations, especially those with multiple
locations, the opportunity to standardise their approach to assessment. The availability
of the same tests in multiple languages enables multinational organisations to create a
consistent hiring process globally.
Centralised Data Storage and Reporting
A significant benefit of online testing, especially through a ‘Software-on-Demand’ model,
is that all recruitment data is held in a single, globally accessible system (subject of
course to access authorisations and data protection regulations).
Risks
Psychometric Integrity
When objective assessments (and in particular ability tests) are delivered online, the
impact on the psychometric properties behind the test is significant. Organisations need
to ensure that the underlying science of their online test is robust. The test provider
should ensure that security and protection methods are in place to protect the test
content, and also that candidates cannot preview and practice the test covertly.
Technology Issues
Ensuring that an online testing system performs reliably and provides a smooth
experience to users is a key requirement. There are steps that can be taken to manage
the risks. Putting the IT department in contact with the test provider at the outset will
ensure that any possible issues can be foreseen and overcome before an online testing
programme is launched.
Reputational Issues
Moving to unsupervised online test delivery reduces the level of control over the test
environment and reduces the opportunity to engage in a dialogue with the candidate
at the time of the test. It is important to ensure that the candidate can obtain help and
support if they need it, and that reassurance is provided to candidates that the testing
process is fair to all and not open to abuse.
Legal Implications
It is critical that a clear assessment policy is in place that outlines how the assessment
process is appropriate for the business need and meets the requirements of employment
law. The policy should also address assessment integrity, security and cheating issues,
which are specific to the unsupervised online testing context
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
5
Fairness and Quality
Data presented at the Association
of Test Publishers conference in
February 2006 showed that the
number of distractions experienced
with online tests was no greater
than those experienced in
traditional proctored or classroom
testing.
There is mounting evidence to remove the concerns around online testing in areas such
as gender and age bias. Examples of such evidence include the following:
In early 2006, the Irish Civil Service reported that online job applicants had a broader
age range and reflected the same proportion of women compared to offline applicants.
In addition, online applicants outperformed offline applicants on selection tests (source:
www.management-issues.com). Data presented at the Association of Test Publishers
conference in February 2006 showed that the number of distractions experienced with
online tests was no greater than those experienced in traditional proctored or classroom
testing.
In 2005, we surveyed 1,300 people participating in test development trials. This sample
covered potential job applicants with an equal gender split, age range from 18 to 70 and a
wide educational range. 63% of respondents preferred to complete assessments online.
Lay the Foundations –
Step 2: Determine the objectives, stakes and scope of your programme
Regardless of the size or scope of an assessment programme, most value is gained when
organisations adopt a strategic approach and link online assessment programmes into
their overall talent management strategy. Key steps can then be followed to define the
objectives, stakes and scope of a specific project.
Case study: Online testing achieves business needs for an investment bank
With thousands of graduate applicants across several countries, this organisation
looked to unsupervised ability testing online to meet three objectives:
1. Improve the quality of candidates attending the assessment centre.
2. Speed up the process of identifying those who should be called forward.
3. Reduce the overall cost-to-hire of graduate entrants to the organisation.
Introducing unsupervised online ability tests online resulted in a 40% improvement
in the success rate of those attending the assessment centre stage, enabled the
organisation to identify talent at a much earlier stage in the process and enabled the
organisation to reduce its cost by 37% from US$3,200 to US$2,000 per graduate hire.
1. Specify the business need
Articulate the business need that the assessment programme will address, including
how unsupervised online testing will meet that need and what metrics will be used to
measure success. For example, online assessment may enable faster and wider access for
candidates and may improve the organisation’s ability to collect and link assessment data
with performance data.
2. Define the key stakeholders
For small-scale assessment tasks, it is critical that the internal client is clearly identified
and requirements are agreed as early as possible. For larger programmes, the range of
stakeholders can be fairly large and involve a number of disciplines and backgrounds.
For example, where complex technical requirements such as integration with other
applications are involved, then an IT contact engaged from the start is critical.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
6
3. Define the needs of the intended users
Users of online assessments include those responsible for managing the administration
of assessments, the candidates, and those who will make decisions based on assessment
results. Reports summarising the results of an assessment may also need to support
different users, such as the HR function and the candidate’s future manager.
4. Clarify the project scope and investment choices to be made
Scope will dictate complexity, time and cost of a project and will also highlight who
needs to be involved in requirements and decision-making, how suppliers will be
selected and managed, and how internal ambitions and expectations will be managed
once the assessment is deployed.
Key questions to ask that can help clarify the scope of an unsupervised online testing
project include:
What Levels of Volume and Scalability Will be Needed?
■■
■■
■■
How many roles and candidates does the system need to cater for?
Are there peak periods - if so, what volume must the system handle at any one time?
What levels of security are required?
What kind of ‘future proofing’ is required to ensure the system can evolve?
How Much Functionality is Required?
■■
■■
■■
■■
■■
■■
■■
Is there one stage of assessment or several?
What types of assessment tools are required - a sifting tool to screen candidates, or a
richer assessment against core competencies?
How are the assessment stages organised – sequentially, or in parallel?
What navigation does the user need to access assessments and reports?
Does the assessment need to integrate with other applications or offline processes?
What types of reports are required for who and when?
What level of training is required for users of the assessments?
What Other On-Going Costs are Likely to be Involved?
■■
In addition to the costs associated with functional requirements (system, training
etc.), other costs should also be considered, such as staffing and infrastructure to
support on-going use of the assessments.
Buy, Build or Lease?
■■
Will the assessment and delivery technology require a custom-built system? Will
this be undertaken internally or through an external contractor (such as a Software
on Demand provider)?
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
7
Case study: New legislation leads to the introduction of online testing
What is score aberrance?Aberrance is observed when a
subject answers the test questions
in a manner that is inconsistent
with demonstrated knowledge
and behaviour. Examples are
inconsistencies in the amount
of time taken to respond to test
items, and answer selections that
are inconsistent with a student’s
demonstrated ability on other test
items.
Source: Caveon Test Security
New European Union legislation which required a demonstration of competence in
technical sales staff, led this multinational organisation to undertake a review of the
key competencies set out in the legislation against those used in its hiring, training
and promotion processes for key sales roles, including front-line sales, account
management and customer care. They identified that a number of softer skills relating
to communication and customer care were not sufficiently covered at any of these key
employment stages, and the links between recruitment, training and development
needed to be strengthened and more clearly documented. As such, the organisation
felt it was exposed to possible regulatory challenge.
These competencies were translated into a suite of assessment tools that were
delivered online across all of its European geographies and in local languages.
This was further strengthened by documented evidence showing how the online
assessments, which included screening tools for hiring as well as 360-degree tools for
development and performance management, were related to business metrics ranging
from sales revenues, through to customer satisfaction.
Lay the Foundations –
Step 3: Identify where your process needs to change and the impact of the change
For most organisations, the integration of an online testing programme represents more
than a straightforward addition of ‘psychometric testing’ to an existing recruitment
or development process. In reality, it often provides an opportunity to re-engineer
the entire process. Embarking on the journey with this expectation up front, and by
approaching it as a change programme, will lead to greater success.
Even if an organisation is already testing candidates online under supervised conditions,
introducing unsupervised online testing brings with it significant challenges. Along with
the obvious introduction of a new technology, it will also create different demands on
staff, which leads to training and potentially different skill-set requirements.
It is critical to outline both the current process and the new process to identify what
needs to change, and also the impact the changes will have.
Map out current process – who is involved, at what stage, and how?
Map out the new process – who is involved, at what stage, and how?
Put a plan in place to manage the changes
Compare the existing processes with the new, and identify areas that will change
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
8
When comparing the old and new process, consider each of the following areas and how
the requirements around each will change with the introduction of the new process:
■■
Confidentiality and storage of results.
■■
Test taker responsibilities.
■■
System administrator responsibilities.
■■
Communications to candidates and users of the system.
■■
Delivery of tests at the test session and troubleshooting.
■■
Scoring and reporting.
■■
Using results for decision-making.
■■
Feedback and communication after the test session.
■■
Communicating follow-up tests if applicable.
■■
Handling verification result discrepancies.
■■
Monitoring and identity authentication.
■■
Training, access and usage requirements.
■■
Who will access and use the tests and results?
■■
What are the new training requirements for users of the system/reports?
Reading the application notes in Step 6 (‘How to Apply the Better Practice Principles’) in
this document will provide valuable guidance on these areas.
Develop Policy and Procedures –
Step 4: The four Better Practice principles for unsupervised testing online
As already outlined, the delivery of tests in an unsupervised online environment is not
without its risks. As a globally-recognised, independent advisor in the field of objective
assessment, the International Test Commission (ITC) has developed a set of guidelines
for promoting good practice when testing on computer and online in their International
Guidelines on Computer-based and Internet-delivered Testing (2006). We strongly
recommends the ITC guidelines as a key reference for any organisation about to
implement online testing.
Consistent with the ITC’s work, CEB has provided additional Better Practice
recommendations to preserve the integrity of unsupervised testing online. These take
the shape of four principles that can be followed to overcome the risks and realise the
benefits from unsupervised testing online:
■■
Better Practice principle 1 – Manage the security of assessments
■■
Better Practice Principle 2 – Defend against cheating
■■
Better Practice Principle 3 – Build in verification procedures
■■
Better Practice Principle 4 – Establish and communicate an online assessment
contract with the candidate
These four Better Practice principles have been taken from the SHL Better Practice
for Unsupervised Online Assessment White Paper, in which can be found more detail
supporting each of these principles. By following these principles, an organisation
should gain a higher degree of confidence that the tests and test results are fair, objective
and accurate measures of performance potential.
Following the ITC guidelines for unsupervised assessment online, the basic
requirements that we recommend:
1. The candidate is registered for the assessment.
2. The candidate is issued with an ID and password.
3. The candidate is required to accept terms and conditions for the assessment, which
includes an ‘honesty contract’ and also their consent to a verification of their test
performance at a later date (see Principles 3 and 4).
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
9
What is collusion? “Collusion
occurs when examinees share
answers for the test items either
during or before the test. It also
manifests itself when an educator
provides the same answers to
test items to multiple students.
Statistically, collusion indicates
that the tests are not being taken
independently.”
Caveon Test Security talking about
educational testing
ITC Guidelines (2006) cover four areas relating to good practice in the delivery of
tests via computer or Internet:
1. Give due regard to technological issues
This area advises on hardware, software considerations, the robustness of the test,
human factor issues around presentation of tests on-screen and adjustments for
testing candidates with disabilities.
2. Attend to quality issues
This area covers issues to do with psychometric qualities of online tests, evidence of
equivalence, scoring, result interpretation, feedback and equality of access.
3. Provide appropriate levels of control
This area outlines different levels of control over test conditions, supervision,
controlling prior practice and item exposure, test taker authenticity and cheating.
4. Provisions for security and safeguarding privacy
This area guides on security of test materials, security of test taker data and
confidentiality of results.
ITC guidelines can be downloaded from www.intestcom.org
Better Practice Principle 1:
Manage the Security of the Assessments
1. Create and communicate an assessment policy.
The policy must set out the objectives and scope of the assessment, stating the reasons
for such a policy and the people, materials and systems that are affected by the policy.
The objectives and scope will normally make reference to other policies developed by
the organisation, such as employment contracts and employee handbooks that set out
employee responsibilities for intellectual property. There will also usually be reference
to trademarked/copyright information and products, as well as information technology
security policies and plans.
■■
Security and protection of intellectual property and data. The policy should
encompass assessment content, any preparation materials provided to candidates,
the security of the technologies that will be used to deliver and score the
assessments, as well as the technologies that will be used to store assessments,
scores and reports.
■■
Individuals responsible for implementing and maintaining assessment security.
The policy should identify those who have operational responsibility for the
deployment and use of assessments and those who advise on assessment use, as well
as those who will have responsibility for data security. The assessment policy should
be communicated to all relevant parties and the consequences of violating the policy
should also be stated.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
10
2. Implement active processes for security audits and specify how security breaches
will be dealt with.
Better Practice around assessment security should include active processes for auditing
security and procedures for dealing with suspected or confirmed security breaches. This
can be done by the organisation itself, or can be formally delegated to a supplier. Key
audits that should be applied in securing assessments are:
■■
Data forensic audit. This involves analysing sets of assessment data for the
occurrence of unusual or “aberrant” patterns of scores. Essentially, a data forensics
audit uses a number of sophisticated indices to analyse if response patterns to an
assessment are statistically different to normal response patterns that would be
expected if the assessment was taken honestly.
■■
Web monitoring. This involves searching the Internet to identify web sites and
discussion forums where information on assessment is being exchanged. This
includes advertising or requests for proxy assessment takers, meeting boards of
people seeking to collude and where content such as questions and answers may
be offered by pirates. For example, among the sites that CEB monitors in order to
protect tests are :
www.doctorjob.com/testingzone/test.asp
www.ebay.com
www.vault.com
www.assessmentday.co.uk
(The fact that CEB monitors these sites does not mean or imply that these sites are
sources of pirated information, or that they actively promote cheating.)
■■
Security audits. These involve regular checks of the security policy to confirm that
the list of those responsible for security is current, that recent threats to security
have been logged and the policy amended accordingly. Checks should also be made
to ensure that a plan is in place to respond to security breaches – for example
the use of alternate versions of the assessment, re-assessment of candidates and
replacement of compromised content.
Data forensic auditing and web patrolling is routinely conducted on behalf of clients to
safeguard CEB online assessments. Presently, these audits are carried out on aggregated
data from assessments taking place all over the world and individual organisations may
not have access to data in relation to their specific organisation. However, we believe the
trend is moving towards a more granular review of assessment data and it is anticipated
that, in the future, organisations will be able to access data forensics for their own online
candidates.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
11
Better Practice Principle 2:
“Any Internet test that administers
the same set of items to all
examinees is asking to be
compromised. At the very least,
the items should be administered
in a randomised order. It would be
better yet to sample items from
a reasonably large item pool.”
Source: Unproctored Internet
testing in employment settings.
Personnel Psychology, 59, 189-225
(2006)
Defend Against Cheating
Cheats can be very innovative, as our partners (Caveon Test Security) illustrate on their
website www.caveon.com. Caveon’s work with clients internationally in the fields of
educational and skills testing shows that 5% - 20% of people attending a supervised,
paper-based assessment are expected to engage in some form of cheating. These figures
suggest that it is debatable whether supervised assessments are more secure than
unsupervised ones, especially when the right safeguards are built into an unsupervised
online assessment.
Evidence suggests that the majority of people are in fact honest. The key issue is how to
minimise the risk of cheating to ensure that an assessment is as fair as possible to honest
candidates and supports valid decision-making. Organisations and practitioners in the
field of objective assessment can act to manage the risks of cheating and the good news is
that both the science and practice of assessment security are now very well advanced.
These advances have been incorporated in these Better Practice principles, which act to
prevent cheating by tackling the motivation, means and opportunity to cheat.
Actions to Defend Against Cheating
1. Use cheat-resistant assessment tools.
To qualify as cheat-resistant, an assessment must:
■■
minimise the potential for faking answers if the assessment is a self-report
questionnaire.
■■
minimise the opportunity to obtain questions and/or answers in advance if the
assessment is an ability test.
2. Adopt checks and rules in the assessment process.
■■
Design the assessment process to incorporate process checks on the validity of
scores and the authenticity of the candidates (see Principle 3).
■■
Set out clear rules for the candidate about what is expected of them (see Principle 4).
■■
Communicate the process and the checks clearly to candidates, which act as
tangible incentives for candidates not to cheat (see Principle 4).
3. Implement security procedures to defend the integrity of the assessment.
In partnership with your assessment provider:
■■
conduct on-going forensic audits of assessment data (see Principle 1).
■■
check on the activities of online pirates and other information sources to ensure
that assessment content has not been compromised (see Principle 1). Knowledge
that these cheat-resistant features of assessments exist, and are backed up by well
managed test security policies, will act to challenge the motivation for candidates to
cheat. It will also communicate to potential cheats that the means and opportunities
to cheat are being minimised and monitored. A key part of that communication is
the use of verification procedures.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
12
Better Practice Principle 3:
Building in Verification Procedures
A verification procedure is a check on the integrity of a score, or other candidate
information, that may be used to make a decision. For example, background checks
are often used to check the integrity of information in resumés and CVs. Verification
procedures in an unsupervised online testing context follow a similar principle by using
other sources of data to check the integrity of a test result that was gained from an
unsupervised online test.
Depending on your process, organisational policies and the stakes surrounding the
assessment, a number of verification methods could be employed, including:
■■
The use of psychometric verifications, such as in-built consistency checks within
questionnaires like the SHL Occupational Personality Questionnaire (OPQ), or the
use of specific verification tests, such as those in the SHL Verify Range of Ability
Tests. In the case of assessing ability, this method is the strongest and most reliable
form of verification.
■■
Using other assessments to check the candidate’s potential and fit to the job.
Information from ability tests, personality questionnaires, assessment centre
exercises and a well-structured interview can be used to contribute to an overall
assessment of potential and fit. Ability and personality assessments administered
unsupervised online are often used to provide data on whether the candidate should
be progressed to a second stage of the process, and if there are areas that should
be probed in more depth before an appointment decision can be made. Crossreferencing information from several assessments to focus on areas of inconsistency
is another form of verification.
■■
The use of other verifiable information related to the assessment. For example,
if the assessment is numerical reasoning, then information on the candidate’s
educational performance related to numerate subjects would be relevant and could
be verified.
As well as deciding what types of verification procedures will be used, it is also
important to decide how a verification procedure will be carried out, and at which point
in the process it is undertaken. A number of possible models exist:
■■
No verification at all
■■
Random verification – where some candidates are randomly selected for verification
procedures
■■
Verification is always conducted – across all scenarios and for all candidates
■■
Mixed models – where verification might always be conducted in some scenarios
(for example in high stakes recruitment contexts) and not in others
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
13
SHL continues to recommend that
if unsupervised ability test results
are being relied upon when making
‘yes’ or ‘no’ decisions, then some
form of verification should always
be carried out.
Providing prescriptive guidance on the process that should be implemented is not
straightforward, as much depends on the context of the organisation, including:
organisational policies; an organisation’s acceptance of risk; stakes of the assessment;
legal considerations and regulations in that country; and presence of other checks in the
process.
We continue to recommend that if unsupervised ability test results are being relied upon
when making ‘yes’ or ‘no’ decisions, then some form of verification should always be
carried out. The importance of verification increases when the stakes of the assessment
increase (i.e. in high stakes recruitment scenarios).
Verification procedures need to be handled sensitively. It should be noted that if a score
from an unsupervised online test appears to be inconsistent with a verification check,
this does not automatically mean that the candidate has cheated. What it does mean
is that the score from the unsupervised assessment may not be a true measure of the
candidate’s ability and competencies, and needs to be followed up.
The effectiveness of verification procedures also relies on how well the process has been
communicated to the candidate and whether they have bought into an online assessment
‘contract’ with you. The use of an honesty policy and statements around verification
processes are covered in Principle #4.
ITC Guideline 45 addresses the need for test users to confirm the results from
unsupervised tests and to ensure that test takers know this will be required.
For moderate and high stakes assessment (e.g. job recruitment and selection), where
individuals are permitted to take a test in controlled mode (i.e. in which, although no
direct human supervision is required, the test is made available only to known test
takers), those obtaining qualifying scores should be required to take a supervised test
to confirm their scores.
Procedures should be used to check whether the test taker’s original responses are
consistent with the responses from the confirmation test.
SHL continues to recommend that if unsupervised ability test results are being relied
upon when making ‘yes’ or ‘no’ decisions, then some form of verification should always
be carried out.
Test takers should be informed in advance of these procedures and asked to confirm
that they will complete the tests according to instructions given (e.g. not seek
assistance, not collude with others etc). This agreement may be represented in the
form of an explicit honesty policy, which the test taker is required to accept.
Provide test takers with a list of expectations and consequences for fraudulent
test taking practices, and require test takers to accept or sign the agreement form
indicating their commitment.
Source: ITC 2006
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
14
Clearly this kind of process fBetter
Practice Principle 4:
Establish and Communicate an Online Assessment ‘Contract’ with
the Candidate
Having a technically valid online assessment supported by legitimate and defensible
processes is one part of ensuring the relevance of an online assessment. Another significant
part depends on how the objectives and decision rules governing an assessment are
communicated to candidates. Managing expectations is critical to motivating candidates to
engage positively with the assessment and to obtaining accurate and valid scores.
Far from providing a distant and clinical means of assessing candidates, online
assessment can provide an active means of managing perceptions of fairness and
relevance. In effect, the stronger the social contract established with the candidate, the
greater control the organisation will have over managing the assessment process and
dealing with issues such as faking and cheating. Whether it is through a web site, web
pages that precede and follow the assessment, or through information provided by email
prior to an assessment, the key points to be covered in the communications supporting
an assessment are outlined below. These points build on the American Psychological
Association’s (APA) guidelines on the rights and responsibilities of test takers, which can
be found at http://www.apa.org/science/ttrr.html.
■■
Describe what the objectives of the assessment are. This should be an explanation
and not a justification or defence, and should cover the relevance of the assessment
to the job, role or position.
■■
Explain why these assessments have been selected. Reiterate the relevance of the
assessments to the job, role or position.
■■
Explain the process. Communicate what will happen once the candidate has
registered, completed the assessment, and what will happen once their results have
been received. Also explain what the ‘authentication policy’ is. While verification
is about the validity of scores, authentication is about validating the identity of the
candidate. Many organisations have now adopted authentication as an essential part
of their selection processes, for example requiring the candidate to provide some
form of identification, preferably a legal document containing photographic ID,
such as a passport or driver’s licence.
■■
Explain the rules for the assessment. This should include confidentiality of
assessment results and reports, when and in what form feedback or decisions will
be communicated to them, and what they are signing up to, including what the
consequences will be of fraudulent test taking or cheating.
■■
Include an honesty agreement. A key step in ensuring that candidates understand
the rules for the assessment is requiring them to agree to a set of conditions before
they proceed with the assessment. This process can be described as signing up to an
‘honesty agreement’.
The conditions should include:
■■
The expectation that the candidate will take the assessment honestly and without
the help of others
■■
The expectation that the candidate will respect the confidentiality of the
assessment and not share its contents with others
■■
The expectation that the candidate will respect copyright, trademarks and other
legal rights over the content of the assessment
■■
That the candidate accepts that verification will be undertaken and that they may
be required to sit a verification assessment at some point in the process
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
15
Managing expectations is critical to
motivating candidates to engage
positively with the assessment and
to obtaining accurate and valid
scores.
Please see the box below for an example ‘honesty agreement’ from the SHL Talent
Measurement online system, which is used for all our online unsupervised tests.
“You are about to begin a critical reasoning assessment. You will receive detailed
instructions and a practice test before starting the real test. You can exit this process
at any time before the real test begins. Please read and accept these conditions before
proceeding:
■■
I will take this assessment honestly and without assistance from others.
■■
I will not copy or distribute information from this assessment in any form.
■■
If requested, I agree to complete a further supervised assessment to verify my
results of this assessment.”
Source: Taken from the SHL Online Assessment Platform
Also, explain to the candidate where he or she can get further information on the
assessments and on where he or she can try out practice assessments. Tell the candidate
what to do and who to notify if they experience problems undertaking the assessment.
Having followed the four step-by-step Better Practice principles set out above,
organisations will safeguard the integrity of the assessment process and be best
positioned to overcome the risks and realise the benefits from online unsupervised
testing.
Develop Policy and Procedures –
Step 5: Develop new policies and procedures for unsupervised test usage in your
organisation
Having worked through Steps 1 to 4 and assessed the stakes, policies, culture and
current processes, organisations should now be better placed to decide how the risks of
unsupervised testing can be best managed. As a result, the next step will be to map out
and document an updated set of policy and procedures for the organisation, specifically
dealing with unsupervised testing online.
Probably the most important changes in policy will involve decisions around:
■■
Will a verification process be introduced, or amended if such a process is already in
place? If so, what will it look like?
■■
How will verification results be managed? How will discrepancies be handled?
■■
What is the impact on disability and adverse impact policies / procedures?
■■
What is the impact on data storage, confidentiality and access of results?
The updated/new policies and procedures should be documented and reviewed to
ensure alignment with local employment legislations and legal requirements.
Practical guidance on the development of policies based on the Better Practice
principles, is provided in Step 6 – ‘How to apply the Better Practice principles’.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
16
Implement the New Process –
Step 6: How to apply the Better Practice principles
In summary, these guidelines have provided the framework to assist you to:
■■
balance the pros and cons of unsupervised online testing.
■■
evaluate the objectives, stakes and scope as they relate specifically to your
organisation and assessment programme(s).
■■
identify where your current processes need to change and the impact of the
changes.
■■
update relevant policy and procedures.
SHL has also introduced four Better Practice principles to ensure any new process
retains integrity and delivers maximum value back to the organisation.
The purpose of the Better Practice Application Notes is to provide guidance on the
following key areas, which must be considered when implementing an unsupervised
online testing programme. They have particular relevance to unsupervised ability testing
online:
A. Choosing a test provider
B. Things to look out for and avoid
C. Developing a new policy for unsupervised testing online
D.Responsibility towards test takers before, during and after the online test session
E. Guidelines on using cut scores
F. Cross checking results with other information
G.Retesting and verification testing
H. Supervising the verification test
I. Handling verification discrepancies
J. Monitoring test usage and results
K. Confidentiality, storage of results and access to test materials
These key subject areas will continually evolve and, as such, each step will be updated
regularly and be available for individual download from: www.shl.com/.
It is also important to note that these guidelines have been updated to reflect the
latest knowledge around the management and delivery of psychometric tests based
on established best practice. These updates deal only with areas where online testing
requires the adaptation or expansion of best practice.
Readers unfamiliar with standard best practices for the management and delivery of
psychometric tests should also refer to existing best practice guidelines, which can be
found in www.shl.com.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
17
Breached tests rarely impact
the test provider, but can often
impact the clients that use them,
for example resulting in mis-hires,
fairness claims, etc.
How to apply the Better Practice
principles
A: Choosing a Test Provider
When objective assessments (and in particular, ability tests) are delivered online, the impact
on the psychometric science behind the test is significant. A new set of requirements emerge
and, in effect, the science behind an online test needs to be even better. While the important
questions around test reliability and validity remain, organisations need to ask new questions
to ensure that the underlying science of the online test is robust.
A thorough evaluation of the test provider’s capabilities will be critical, paying particular
attention to:
■■
the appropriateness and adequacy of tests and test content.
■■
the availability of tests appropriate for unsupervised online usage that defend
against cheating and piracy.
■■
the presence of security measures to protect the integrity of the testing process (e.g.
item banks, secure technology, response pattern monitoring, etc.).
The checklist below provides some tips and questions to guide you when evaluating a
prospective test provider.
What should you Look for When Selecting an Unsupervised Online
Test Provider?
Tools that are cheat and fake-resistant and specifically designed for unsupervised
online testing scenarios. Look for:
Different test versions generated for each candidate. DO NOT use fixed item tests
unsupervised online, or even tests with small numbers of parallel forms as these are
highly susceptible to piracy and cheating.
Evidence of equating the scores candidates get if they are all getting different
tests. The psychometric technology used to equate different tests is Item Response
Theory (IRT). Check that the tests for each candidate have been created from a bank
of items using IRT to ensure that each candidate gets an equivalent tests and that the
scores produced are comparable.
Security and data forensic monitoring ideally by an independent agency. This provides
employers with reassurance about the security and validity of their testing process.
Built-in verification processes which ensure the integrity of unsupervised test results
can be confirmed accurately. This is particularly important in high stakes recruitment
contexts.
Technological reliability. Ensure the test provider can provide evidence that their
infrastructure is resilient, has high availability and has the connectivity, concurrency
and scalability to handle assessment test loads.
Test timing mechanisms that are independent from Internet connections. This
ensures candidates using slower machines are not disadvantaged and also guards
against the loss of data caused by connection dropouts.
Technical information about test reliability, evidence for validity and suitability for
online delivery. Psychometric properties for online tests should be evaluated in the
same manner as traditional tests.
Mechanisms for assessing people with a disability. Ensure that when required,
adjustments can be made to the online testing system when testing people with
a disability. Also ensure that the on-screen presentation of test items will not
disadvantage candidates with colour blindness.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
18
Choosing a Workflow or Software-on-Demand Provider
In an online testing context, a Software-on-Demand provider owns and operates a
software application that facilitates the delivery and scoring of an assessment. The
provider maintains and services the application, and makes the online assessments
available to users via a browser or within a client application.
If the decision is made to partner with either a Workflow or Software-on-Demand
provider, essentially a trade-off decision has been made between a high level of direct
control at a high price and a lower level of direct control at a lower price. The need to
have skilled IT and application support staff in the organisation is removed, as is the cost
associated with the required hardware platforms. However, it is critical that the partner
chosen can deliver the level of service that is required.
In addition, since the users in the organisation and also the candidates will be accessing
the application across the Internet, there are a number of connectivity requirements
that must be checked. With a desktop application such as Microsoft Office, as long as
the PC is running, the application will be available for use. With a Software-on-Demand
provider, there are several things that determine ability to use the application at will. For
example, is the application always running and can the application be accessed when
needed?
Tips when selecting a Software-on-Demand provider…
Check the access settings in your organisation
Many IT departments in organisations block access to certain Internet sites, block
certain types of content and also block certain types of common Internet application
such as Macromedia Flash. Speak with your IT department and the Software-onDemand provider prior to signing any agreement, to ensure that connection and use of
the necessary application/s is possible.
Check the bandwidth settings in your organisation
It is important to understand what bandwidth requirements are necessary to use the
application and ensure that these are in line with the bandwidth that an organisation
has available. Failure to do this is likely to result in poor user experience and a failed
implementation.
Consider the candidate’s accessibility
Remember that many candidates will be accessing an application from what might
be considered to be an unusual connection or configuration. It is important to both
minimise the bandwidth required and the level of technology required to run the
candidate part of the application suitably.
Evaluate data protection and back-ups
Partnering with a third party Software-on-Demand also means that the data is hosted
by someone else and is not under direct control. Determine whether the provider has
backup procedures that will protect data to ensure compliance with data protection
and privacy regulations around the world.
Ensure a high degree of system security
With the Internet becoming an increasingly risky environment, ensure that the
Software-on-Demand provider has appropriate safeguards in place to stop hackers
getting access to data or tampering with an application. This is increasingly important
as the stakes of the application and testing rise.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
19
B: Things to Look Out For and Avoid
Unsupervised testing online offers rewards for both candidates and organisations.
However, because it is a new approach to testing, guidance around, for example, test
provider selection and test use is not readily available.
The checklist that follows below can help organisations avoid potentially costly mistakes
in relation to unsupervised ability testing online. Look out for and avoid:
■■
Fixed item tests online that are promoted as safe or secure forms of testing. Test
providers who promote such tests for unsupervised use offer little or no protection
over their test content.
■■
Test providers who place fixed item tests online for unsupervised use and claim that
no evidence exists that indicates their test content has been breached.
If problems were discovered indicating that items or tests were breached, the damage
has already been done at the expense of clients. Breached tests rarely impact the test
provider, but can often impact the clients that use them, for example resulting in mishires, fairness claims, etc.
■■
Parallel tests that are promoted under the description ‘randomised testing’.
This is certainly an improvement on fixed item testing online, but parallel forms of
testing still offer limited protection from integrity breaches.
■■
The promotion of ‘unlimited’ testing volumes.
This is a good indication that the test provider has no experience of real-world,
high volume testing situations. Credible providers understand that the Internet has
limits and will provide appropriate guidance on volume testing.
■■
Test providers who claim that no training is required to use their tests.
Specialised or technical training on test use and interpretation may be unnecessary,
but professional organisations will suggest or provide training to ensure users do not
inappropriately use or interpret the results, no matter how simple the outputs are.
■■
Tests that have no associated documentation detailing psychometric properties
or effectiveness.
A lack of documentation or research proving its effectiveness in workplace settings
should raise serious questions about the validity of that psychometric test.
■■
Online unsupervised tests that use another full test as an independent retest.
Using two separate tests that provide two separate scores raises many questions
about how to equate those scores appropriately, and how to determine which score
is more valid for use. Instead, look for tests that provide built-in verification tools
that determine the confidence of the original ability score.
■■
Test providers who rely on the prosecution of individuals who steal test content
as their primary protection of test integrity.
This is a protection for the test provider, not the client using the tests. Once a test
has been breached, the integrity of the test has been compromised, regardless of
whether someone is prosecuted or not.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
20
C: Developing a New Policy for Unsupervised Testing Online
Psychometric tests are powerful tools used by organisations for the selection,
development and management of people. Tests can be used to enhance the decisionmaking process throughout the employee lifecycle process. Ability tests
are a particularly useful tool for screening and selection because they are the single most
effective predictor of potential. In general, occupational tests have consistently been
shown to be better predictors of job success than interviews.
Although highly predictive, many of the tests in use today were developed for traditional
paper and pencil administration. The rapid growth of the Internet for recruitment and
selection purposes has placed significant pressure on organisations and test providers
to offer their tests online. Deploying tests online opens up a host of new questions and
concerns around transferring existing best practice in the use of employment tests.
Who Should use Tests?
Knowledge and experience are required to use psychometric tests effectively. It is
recognised throughout the world that psychometric instruments are potentially
dangerous in the wrong hands. Indeed in many countries, only qualified psychologists
are allowed to use them.
For example, existing best practice in the UK requires that reputable test providers
maintain a register of qualified people to whom test material may be supplied. To access
such material requires both general training in test use and often familiarisation training
for the specific instruments in question. Publishers usually give some recognition to
training from other providers. Current guidelines suggest several training levels for users
of tests, including Test Administration training and BPS Level A Occupational Testing
Guidelines for the use of online tests.
Better Practice still recommends that whenever possible, all test users follow existing
best practice guidelines. However, new technologies and capabilities of test publishers
have opened up possibilities to provide self-guided interpretation tools and outputs that
can be safely used by others within an organisation. This becomes particularly relevant
with online testing, as it allows tests to be tightly embedded in organisation-wide
systems that enable greater access to decision makers.
In all cases where traditional tests and psychologically rich outputs are used online,
users of these outputs should be fully trained in their use and interpretation. If the
selection of online test has been made by a qualified user and there is sufficient guidance
and interpretation within the output, it is feasible to allow minimally trained users to
safely use psychometric test outputs.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
21
Ability tests are a particularly
useful tool for screening and
selection because they are the
single most effective predictor of
potential.
Provided below is a checklist to help determine if a test is appropriate for minimally
trained users:
If you can answer “yes” to all of these questions, the test is likely to be
appropriate for minimally trained users
Is test selection governed by the supplier and/or a trained user within the
organisation?
Is the bulk of the test administration automated and managed by the system? Tests for
minimally trained users should be automated so that users do not need to set up and
manage test administration sessions.
If the test is timed, does the system automatically enforce timing? If not, a qualified
test user should administer the test.
Have test cut-scores already been set by a trained user? If cut-scores are easily
changed or set, the test should be restricted to users trained in this area.
Do the test outputs provide interpreted narratives and supporting guidance? If the
test outputs require psychometric knowledge, interpretation, or other specialised
knowledge about ability tests to interpret the results, proper training in the use of the
tests should be enforced.
If tests are designed for use with minimal training, does the provider offer basic
training for use? Regardless of the simplicity of the test and outputs, some training is
strongly advised for all users about appropriate use of the results.
Organisations seeking to expand the use of tests within the organisation should also
refer to existing best practices and other guidelines, such as those provided by the
International Test Commission, that provide guidance on the training required for test
users of online tests.
Case study: Understanding new training requirements for test administrators
1. It could substantially reduce costs to both candidates and the organisation by
eliminating the need for candidates to be screened and tested centrally;
2. It improved and expanded the applicant pool by making it easier and faster for
more graduates to apply for positions;
3. It provided more substantive and direct involvement of the local branch HR
personnel sooner in the selection process, a proven critical step in making the
right hiring decisions.
To realise the benefits of decentralised testing, the organisation had to involve their
branch managers in some parts of the testing process. Candidates would complete the
tests online and then go straight into the closest branch for next steps and interviews
rather than have central testing sessions at the corporate facility. This presented a
new challenge – how to train the staff on the interpretation and use of the test results.
Rather than focusing on technical training around the interpretation of traditional
test scores, the organisation chose test outputs that provided interpreted results. They
also provided structured guidance for users who did not have specialised training.
They selected an online test that did not require branch managers to reconcile two
test scores when they retested candidates.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
22
The organisation felt that the controls and the structure of the new online testing
process eliminated much of the training normally required of test administrators.
Branch managers did not need to take over the central test administrator’s role of
coordinating testing sessions because the testing process was automated. Instead of
training branch managers on how to administer tests, the organisation planned to use
the time saved to help managers with:
■■
structured interviewing techniques.
■■
appropriate use and interpretation of test results.
■■
how to verify and confirm online unsupervised test results. The organisation
decided that unsupervised, online testing supported at the branch
The organisation decided that unsupervised, online testing supported at the branch
level represented a good balance between risks and rewards without requiring
advanced training for branch managers or HR staff.
D: Responsibility Towards Online Test Takers
Current best practice specifies that whenever tests are used, testers should be honest
and open with candidates about why the test is being used and what will happen to
the results in some countries (e.g. the UK), guidance also recommends that candidates
should be offered feedback of their results.
In general, these practices still apply, but adaptations to these guidelines may be
appropriate for online testing due to several factors. Online testing can substantially
increase the volume and scale of test use in the recruitment process. It may no longer be
reasonable to provide feedback to all candidates who complete the test. Better Practice
recommends that organisations should still make feedback available to candidates who
proceed to a shortlist, or final testing stage. Organisations may choose not to offer personal
feedback to candidates in the screening, or sifting stage. However, it is recommended that
individual requests for candidate feedback are honoured whenever reasonable.
The automation of online tests often allows test user organisations to explain the
purpose, intent and process of testing via standardised online communications. Better
Practice for online testing allows for this to replace the traditional test administrator
activities and scripts. Depending on the online process, users may need to be trained on
specific areas of test administration, such as retest procedures and communications.
(a) Before the online test session.
Since the test takers are now responsible for their own testing environment, it is
important that the online test screens provide suggestions on creating a suitable
environment before the test begins.
In supervised test sessions, test administrators are normally responsible for
communicating information about the tests and the process. They are also responsible
for establishing a serious but sympathetic atmosphere. As with current guidelines,
providing this structure can help candidates relax, feel confident that the process is fair,
and reduce the need for support or complaints as a result of an inappropriate testing
environment. Guidelines for establishing the proper test setting are below.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
23
■■
Online tests often change the
feedback process for candidates
from facetoface experience to an
automated process.
■■
■■
■■
■■
■■
Online tests should provide for practice tests and example items. The use of
practice tests and items should be encouraged, and should not only allow test takers
to explore both the test content and style, but also become familiar with the format
and technology required to successfully complete the tests. It is also important to
consider how supplemental information is communicated online, such as:
■■
how candidates can get help
■■
what to do if there is a technology problem
■■
what to do if disabilities may prevent successful completion of the test
■■
tips and techniques on when/where to take the test
Online tests should establish the seriousness of the session. Candidates must be
informed about their responsibilities and the stakes involved with the test. Honesty
agreements, statements of test taker responsibilities or similar instructions are
important to ensure test takers understand the seriousness of the test.
Instruction text and test guidance must sufficiently guide test takers to avoid
interruptions and distractions. For example, candidates should be reminded to:
■■
turn off mobile phones or other devices that might interrupt them
■■
avoid working in a busy or noisy environment
Options should be provided for the candidate to leave the process and return
later. Before the test begins, candidates should have the option to delay starting the
real test until they are in a comfortable environment.
Information should be made available on the use and protection of data and
results. Online tests should provide links to policies that explain who will have
access to test data and what the test results will be used for.
Use of calculators and paper for notes and calculations. The online test
instructions should advise candidates on the use of calculators and having paper
available for rough calculations.
b) After the online test session.
The use of online tests may result in changes to how candidates are managed after
the testing session is complete. Online tests often change the feedback process for
candidates from face-to-face experience to an automated process.
This requires special consideration to ensure that the candidate experience remains
positive and productive. In general, the better the candidate experience is throughout
the entire testing and feedback process, the more likely it is that candidates will perceive
the process as fair and effective. Below are some guidelines for managing the post-test
process and communications:
■■
Provide a method for candidates to contact the organisation if they wish
to request feedback. If tests are used within an automated process, individual
feedback may not be available. If feedback is not provided to candidates, or if
feedback is automated, candidates should still have the option to request it. This
transparency helps to establish confidence in candidates that the entire process is
fair and considerate.
■■
Ensure that feedback and responses to candidates are timely. Results of an online
test should be provided as soon as is reasonable. In many situations, candidates will
perceive the fairness and legitimacy of a recruitment process as much by the overall
experience as by the results of a test. Timeliness and appropriateness of feedback on
tests is an important part of the candidate experience.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
24
■■
■■
Consider how soon automated “rejections” are sent to candidates when tests are
used for sifting. In situations where automated decision rules are used based on test
scores, consider timing of the candidate feedback, particularly for candidates who are
not successful in reaching a cut-off minimum level of ability. If “rejection” notices are
sent in real time, candidates may react negatively to the automated decision. Delaying
the results 12 to 24 hours can improve candidate reaction to the process and provide
time for hiring managers to review decisions, if this is part of the process.
Ensure that all test users in the organisation are trained on next steps. The advent
of online tests often means other people within the organisation, besides the
traditional test administrator, will interface with candidates who have taken tests.
In this case, such people must have the basic training and knowledge to answer
candidate questions about where and how to get feedback on their results. This
may be as simple as ensuring hiring managers know how to contact a qualified user
within their organisation to request feedback.
E: Guidelines on Using Cut Scores
bility tests are a useful method of screening large volumes of applicants fairly,
efficiently, and consistently. The use of ability tests often involves a cut score. Cut scores
are minimum thresholds of performance on a test that are based on the minimum
requirements of a job/position.
In situations where an ability test is used as a sole criterion for sifting (screening)
purposes, we recommend using a cut score no greater than the 30th percentile (however,
due diligence is required to ensure cut scores are set at an appropriate level). Using
a single criterion to screen out greater numbers increases the risk of screening out
candidates who are suitable for the job, as well as potentially increasing the risks that
there will be negative impacts on protected demographic groups.
If an organisation desires to screen out a greater number of applicants using a cut score
approach, a multi-faceted method is recommended. This involves either setting cut
scores based on multiple criteria, such as gross negative disqualifiers, ability, competency
questionnaires and/or background and skills qualifications, or creating composite
scores from these sources. The former approach is complex because it involves setting a
number of cut-scores that together have to produce the requisite overall pass rate. The
latter approach is less complex but assumes that one can let high scores on one attribute
compensate for low scores on another. Where it is possible to define a criterion-related
minimum qualification level for one attribute, a cut-score should be used. In other
situations, compensatory composite score models work well and are simpler to manage.
If using unsupervised ability tests with a cut score, it is important that the cut score is
not adjusted to try to compensate for cheating, unless technical advice is obtained from
your test provider.
For example, if a cut score is set at the 70th percentile, which results in 45% of applicants
passing (instead of the expected 30%), this may be caused by frequent incidents of
cheating (assuming the norm is appropriate for the applicants). Instead of increasing the
cut score upwards, suitable verification techniques should be used to identify individuals
who cheated from the short list of candidates.
There may be an inclination to push the cut up (e.g. to the 80th percentile) to attempt
to get back to the intended pass rate (i.e. 30% instead of 45% passing). Using a higher
cut score has two adverse effects. First, it may increase the incentive to cheat (the
higher the barrier, the higher the incentive, other things being equal). This will
increase the proportion of people in the shortlist who have cheated, ultimately being
counterproductive for the organisation. Second, it is unfair because it excludes
candidates who would legitimately meet the original cut score. Cut scores should be set
on the basis of job requirements, not just to reduce numbers.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
25
F: Cross Checking Results with Other Information
Cut scores should be set on the
basis of job requirements, not just
to reduce numbers.
Consider information from other competency assessments
Use results from other assessments that relate to the competencies and/or skills
important for performance in the job to evaluate the person’s actual ability level. (E.g.
work simulations, assessment centres, etc.)
This technique is useful to confirm that an individual has the required level of ability for
a specific role independent of a test result without having to administer a follow-up test.
Advantages of this method include low cost, flexibility, no requirement to have testing
materials/equipment, and an independent evaluation of the ability level using means
relevant to the job.
Disadvantages include the lack of consistency, more subjectivity in evaluating the actual
ability level, increased documentation requirements and the potential for variability in
decision-making.
This can be a useful alternative to evaluate an individual’s approximate ability level.
It can be used in situations where trained test administrators are not available to
administer and proctor follow-up tests, or in situations where test materials and
equipment are not readily available. In addition, it often fits in well with a normal
interviewing process and can reduce negative impacts on the candidate experience.
Competencies (as defined in the SHL Universal Competency Framework) related to
cognitive ability include:
■■
Presenting & Communicating Information.
■■
Writing & Reporting.
■■
Applying Expertise & Technology.
■■
Analysing.
■■
Learning & Researching.
■■
Creating & Innovating.
■■
Formulating Strategies & Concepts.
There are known relationships between ability and measures of competency potential
based on personality data in the above areas. Competency ratings from Assessment
Centres in the above UCF areas are also known to have relationships with ability.
Guidelines for evaluating an individual’s level of ability using other competency
assessments:
Using simulations that mirror the challenges presented within cognitive ability tests,
i.e. providing significant amounts of data and information (balancing strategic and
operational information as well as important and distractor information) which need to
be analysed within a set timeframe.
For example, to simulate numerical and verbal reasoning analysis, presentations and in
trays will assess:
■■
The breadth and depth of an individual’s analysis of a complex business situation.
■■
Their ability to challenge current thinking and identify new and innovative ways of
approaching business challenges.
■■
Their ability to balance operational with strategic issues and think broadly about
business issues and/or set strategic direction for the organisation.
■■
The effectiveness of their communicate style (written or oral) and the strength of
■■
their arguments.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
26
Use information from other sources
Results from examinations, qualifications, grades and other attainment tests that are
appropriate measures of a person’s verbal and/or numerical ability may help to evaluate
the person’s actual ability level.
This technique uses other information available about an individual to independently
confirm the approximate ability level. It is recommended when a follow-up test is not
possible and/or when additional competency assessments are not used.
Advantages to this approach include the ease of obtaining information, low cost and
convenience.
Disadvantages include inconsistency in the measures evaluated, the subjectivity of the
evaluations, increased documentation requirements and the difficulty in corroborating
the information.
Guidelines for evaluating information from other sources include the following:
■■
Do not attempt to directly equate information, such as grades or skills qualifications
to ability level. Use them as a general evaluation of potential.
■■
Use the information to help determine the individual’s approximate level of ability
in comparison with other candidates. Avoid trying to approximate a specific ability
score or level using this information.
■■
Grades or educational qualifications are usually a broad-based measure of
capabilities. There are typically moderate positive correlations between ability tests
and school or university grades, but they should not be used as a direct replacement.
■■
Other sources should be viewed as corroborating evidence supporting (or denying)
a specific ability score.
■■
Use sources of information that can be independently verified. For example, grades
or qualifications from institutions can usually be verified using official transcripts.
Other information may be harder to confirm.
■■
Look for evidence of training success or job performance from previous
employment. This can be corroborative of ability level, because ability levels
are often used to predict training success or job performance. Again, try to use
information that can be independently verified.
■■
Avoid using certifications or awards where little is known about the purpose or
content used to achieve success. These can be difficult to verify, and may not be an
appropriate corroboration of ability.
Use structured interviewing techniques to probe related competencies
As noted above, competencies (as defined in the SHL Universal Competency
Framework) related to cognitive ability include:
■■
Presenting & Communicating Information.
■■
Writing & Reporting.
■■
Applying Expertise & Technology.
■■
Analysing.
■■
Learning & Researching.
■■
Creating & Innovating.
■■
Formulating Strategies & Concepts.
Structured interviewing around competencies that are known to relate to ability level
can provide useful supportive evidence to corroborate ability levels.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
27
Structured interviewing around
competencies that are known to
relate to ability level can provide
useful supportive evidence to
corroborate ability levels.
Advantages of this approach include:
■■
Practical and easy to integrate into most existing hiring processes, since interviews
are a ubiquitous hiring tool.
■■
Structured interviewing is accepted in many organisations, and there is often
adequate training amongst existing staff on how to do effective structured
interviews.
■■
No additional hurdles for candidates (retests) means minimal disadvantage to the
candidate experience.
Disadvantages include:
■■
Subjectivity in evaluating approximate ability levels.
■■
Requires interviewers to be trained and listen for specific cues from the interviews
that can corroborate ability levels. Guidelines for using structured interview
techniques relating these specific competencies are available from SHL.
G: Retesting and Verification Testing
Current best practice addresses the issue of retesting and whether an applicant may
reapply or (be allowed to) retake a test. Online unsupervised ability testing adds another
retesting element to the mix – the option for organisations to retest candidates at a
future point in the process to verify unsupervised online test results.
In the context of retesting an applicant who is reapplying for a position, there is no
hard and fast rule on the policy an organisation should adopt. However the organisation
should ensure that it applies a consistent policy on retesting for applicants who are
reapplying. As with existing best practice guidelines for ability tests, it is still not
desirable to allow candidates to be retested regularly unless alternate forms of the
test are available. Whenever possible, candidates should be retested with a parallel
or alternate form of the test, or ideally with a uniquely generated form such as that
available from the SHL Verify ability tests.
It is normally reasonable to allow an applicant to be retested where there is evidence
that he or she might have under-performed the first time (e.g. due to illness).
Personality questionnaires with in-built consistency checks do not require verification
testing to confirm results. In the context of retesting for ability verification purposes,
it becomes more critical that alternate test forms are available. Verification testing is
used to confirm the result of an unsupervised test and typically is administered within
a relatively short time-frame compared to situations where candidates are reapplying.
Candidates are therefore more likely to remember questions. Organisations that use
verification testing should document the procedures around verification testing to
ensure that consistency and fairness is maintained. Verification testing policies should
include:
■■
Frequency – Consistency of the verification testing model is important to retain
fairness and credibility in the testing process. Is everyone retested, a random
sample, or only candidates where a specific concern is raised? Depending upon
the stakes, local regulations and organisation policy, any of these models may be
appropriate.
■■
Process – If only a subset of candidates is retested, how are the candidates selected?
Criteria should be clear and consistent.
■■
Supervision – Verification tests should be supervised using existing best practice
governing supervised testing.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
28
■■
■■
■■
Score comparisons – How will the organisation compare test scores? If a different
test is used for verification, the scores should be equivalent to the original test to
ensure comparisons can be made.
Acceptance criteria – What are the acceptance criteria for verification test scores?
How do test users determine significant differences between scores? If a test does
not automatically equate test/retest scores for the user, clear and specific guidance
must be provided to inform users when a retest score is statistically different from
the original test score. The organisation may wish to seek advice from their test
provider on this complex area.
Negative verification results – how should the organisation manage a candidate
who yields a verification test result that is significantly different to the original test
score? If a verification test yields a score that is significantly (statistically) lower
than an original test, organisations should provide specific guidance to test users on
how to investigate this difference and determine what to do next.
Choosing the appropriate Verification Testing Model
There are different methods organisations may use to verify the results of an
unsupervised test. Four models are presented here that encompass many of the options:
1. No verification testing.
2. Random verification testing – where some candidates are randomly selected for
verification procedures.
3. Verification testing is always conducted – across all scenarios and for all candidates.
4. Mixed models – where verification might always be conducted in some scenarios
(for example in high stakes recruitment contexts) and not in others.
In situations where high candidate volumes result in a large number of short-listed
candidates, companies may decide to use the verification test at a later stage, verifying
only the results of candidates who are being seriously considered for a final selection
decision.
Choosing the method most appropriate for an organisation can be a complex task.
Examples of each scenario are presented below.
This information should be used only as guidance. Organisations should carefully
evaluate their needs before making a final decision.
1. No Verification Testing
There are two common “no verification testing” models: a) No explicit retest used to
verify an unsupervised test score, but other techniques are used to confirm the results; b)
No verification step at all.
Option b), no verification at all, should only be considered in low stakes situations
(e.g. career guidance) where there are no other alternatives available. Typically
there will always be some low-cost, simple methods available to verify the result of
an unsupervised test even if another test cannot be used. These methods include
competency interviews, using other information to rationally evaluate an individual’s
ability level, or other similar techniques.
Using no verification at all is a risky proposition and may ultimately be
counterproductive.
Option a) is typically what companies choose as a “no verification testing” model. This
option uses some other technique to verify the results of an unsupervised ability test
other than a retest. This approach has the advantages of low cost and great flexibility
while still providing some protection against the damaging effects of cheating.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
29
“No verification testing” models may be more attractive in certain organisational
situations, such as:
■■
No verification testing environment is available for a retest, such as manufacturing
environments, field locations, mobile recruiting, or mass hire situations. These types
of environments often have logistical challenges that make it difficult or impossible
to administer a retest in a controlled environment necessary to provide an effective
retest.
■■
Remote or multiple location recruiting that is geographically dispersed. Companies
often process candidate applications and screen them centrally through an
online process, but send candidates to local offices for the next steps. Often, these
local offices are not equipped or trained to process verification tests. Therefore,
alternative verification methods are common due to the convenience and flexibility
they offer.
■■
Remote screening and candidate qualification followed by a group interview or
assessment centre process. If candidates convene in a central location for group
assessment centres or interviews, group testing can be cost prohibitive as well as
time consuming. Clients may find it more convenient to use alternative verification
methods in the interviews or assessment centres that can verify ability test results.
2. Random Verification Testing
Random verification testing uses sampling techniques to ‘spot check’ the candidate
pool for potential discrepancies in unsupervised testing. The premise is one similar to
random security spot checks in airport screening lines; some individuals are selected
for a thorough screening while the remaining individuals are screened using minimum
techniques. SHL recommends between 25% and 50% of candidates be retested if this is
to be a meaningful disincentive to cheating.
The advantages of this model are threefold:
■■
The random spot checks help to discourage cheating because candidates know
there is a chance they will be selected for verification testing.
■■
Verification test data can be examined periodically to spot cheating trends and
address potential issues related to these trends.
■■
It provides a method to verify some short-listed candidate scores directly without
substantial time or cost.
Disadvantages include:
■■
Cost and time to randomly retest may not be substantially less than other retest
models.
■■
Random retest models can be harder to manage for consistency and compliance.
■■
It may impact the candidate experience as some candidates will be retested and
others not.
■■
It requires regular review of group data to monitor for potential problems.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
30
Random check verification testing models may be more appropriate for:
■■
■■
■■
Companies that use a multi-faceted approach to screening (e.g. use multiple criteria
to short-list candidates). This reduces the emphasis on a single ability test criterion
as the screening tool. The combined criteria can a) reduce the stakes of the ability
test, reducing the desire to cheat among the candidate pool, and b) mitigate the
effects of cheating by considering other forms of qualifications.
Companies that use assessment centres to qualify short listed candidates for next
steps. Assessment centres often include work simulations, competency-based
interviews or other structured methods to determine candidate qualifications.
The assessment centres can help to verify unsupervised ability results without a
verification test for all candidates.
Group or mass hire situations where a limited number of computer or testing
facilities are available. Random testing can act as a disincentive to candidates to
cheat and provide adequate checks in the testing process to monitor for cheating
trends or aberrations.
3. Verification Testing Always Conducted (Retest)
Administering a verification test to all individuals considered for next steps is the most
secure method of verifying ability results obtained from unsupervised tests. This is an
ideal approach to testing, as it represents the most secure method of verifying ability
scores. It is more important in high stakes situations or situations where ability scores
are a key selection criteria. However, where there are a large number of candidates and
several subsequent selection stages, companies may decide that it is more efficient to
conduct verification testing only for candidates that are being seriously considered in a
final selection decision.
The advantages of this method are:
■■
Highest level of confidence that the testing process is providing accurate test results
for all candidates.
■■
Greatest likelihood of catching potential cheats.
■■
Consistent and objective process for all candidates, establishing fairness and
removing potential points of subjectivity from decisions based on test results.
■■
Greater defensibility in situations where results may be challenged.
Disadvantages include:
■■
Not always achievable or appropriate due to cost, time, and resource limitations.
■■
Can reduce flexibility in the interviewing and hiring process.
■■
May be difficult to manage and control in remote or distributed organisational
models.
■■
Increases candidate testing time and can have disadvantages for the overall
candidate experience.
This verification model is appropriate for:
■■
Any organisation that requires strict and consistent use of tests and results.
■■
Organisations that use testing for high-stakes situations. The higher the stakes, the
more likely bad hiring decisions will have negative impacts on an organisation.
■■
Companies that rely on ability test results as a single selection criterion, or as one
of few key criteria. If ability is substantially weighed in a selection decision, the
importance of an accurate and secure measure is paramount.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
31
■■
■■
Companies that have adequate facilities and capabilities to administer a verification
test. If companies have these capabilities, and if there are no obvious barriers such
as substantially increased costs, inconvenience, or other barriers, there is little
reason not to administer a verification test. The level of confidence and integrity
the verification test can provide will often outweigh the disadvantages and risks
associated with limited or no verification testing.
Companies that do only limited interviews, assessment centres, or other additional
assessments on candidates. If there are few other in-depth evaluations of candidates
before making hiring decisions, the potential for unqualified candidates getting
through the process is greater. It may be easier and more cost-effective for
companies in this situation to administer a verification test (and other structured
assessments) to mitigate risks caused by the absence of other interviewing activities.
4. Mixed Model Verification
Mixed model verification may be appropriate for companies where a single model is not
appropriate or cannot be applied consistently. Mixed models blend different verification
methods based on situational factors, or differences between the types (and stakes)
of selecting for different roles. For example, mixed models can vary the methods of
verification used for different roles, or they may vary the points at which verification
tests are used, and therefore how many candidates are verification tested.
(Please also refer to Application Note F: ‘Cross checking results with other information’.)
Advantages of mixed models include:
■■
Highly adaptable to different organisational structures and different recruitment/
hiring models.
■■
Greater flexibility to adapt to changing needs and conditions.
■■
Useful approach to use when balancing costs and resources is important.
■■
Adaptable to changing staff requirements and internal testing expertise/resources.
Disadvantages include:
■■
Less consistency and therefore more challenging to monitor effectiveness of the
verification procedures.
■■
More requirements for documentation and training to ensure individuals involved
with the testing and verification process understand their roles and the process that
should be used.
■■
Potentially more difficult to defend to fairness challenges. If companies use different
approaches in different situations, it can place a greater burden on the organisation
to defend their approach.
Mixed models may be appropriate for:
■■
Companies that hire for many different levels and positions. There are different
stakes involved for different positions that may warrant different verification
models. For example, in hiring key management staff (high stakes) verification may
be used in all cases. In contrast, hiring for a high volume call centre may mean lower
stakes and a more cost-effective or practical approach might be appropriate.
■■
Companies that use ability testing differently in their evaluation of candidates for
positions. For example, some positions may rely on ability as a single criterion or
one of few key criteria for decision-making, whereas others may use ability more
generally. This can influence the level of verification used for different positions.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
32
Case study: How one company determined the ideal verification method
This case study explores how one company decided to use a no retest verification
model. The company used a thorough review and analysis of the many different steps
in the selection process to help determine the potential risk caused by eliminating
the retest. Ultimately the company decided that, due to the many other checks used
throughout the interviewing process, a retest would add very limited additional
security to an already strict process.
A large multinational organisation was using critical reasoning tests as a key part
of their corporate leadership programme. The organisation screened thousands of
applications down to a few hundred candidates using a consistent but varied mix
of screening and sifting criteria. They used the results of a critical reasoning test to
determine the smaller group of finalists to proceed to in-depth and lengthy assessment
centres and on-site interviews. The organisation had strict and documented criteria
that established a minimum threshold of performance on the critical reasoning test.
The HR team ultimately determined that using an online, unsupervised test without
a mandated retest was in the best interests of the candidates, the programme and the
organisation. The decision not to retest was supported by a number of specific criteria
in three areas: test design and security, pre-test activities and post-test activities.
Test design and security criteria included:
■■
The decision to use a fully randomised test rather than a fixed item or parallel form
test to mitigate cheating risks.
■■
The use of test(s) that were as short as possible while still providing appropriate
accuracy (reliability). The team felt that the shorter tests mitigated a candidate’s
attempt to use assistance (e.g. get someone to help them). The team felt that most
candidates who would be tempted to take the test with assistance would think twice
about it as it would likely take as much or more time to coordinate the assistance
and could easily result in a lower score.
■■
Selection of a testing vendor that monitored content for security breaches, such
as the sale or exchange of test items and answers, or other inappropriate or illegal
activities.
Pre-test criteria included:
■■
The use of the critical reasoning test which was one part of a multi-faceted
screening and selection process. The team used a wide range of other criteria, at
both the screening and selection stages, to determine eligibility.
■■
The organisation carefully screened the candidates invited to complete the tests
well in advance of the invitations. The screenings included a thorough review of
qualifications, experience and references.
■■
The candidates were invited to complete the tests online only after an in-depth
telephone interview. These interviews were rarely less than 30 minutes and firmly
established the organisation’s legitimate interest in the candidate as part of the
selection process.
■■
The telephone conversations provided HR staff with the opportunity to explain
the purpose and expectations of the online test to each candidate personally.
They clearly communicated the candidate’s responsibilities and the organisation’s
expectations in advance.
■■
The invitations sent to the candidates emphasised the requirement to complete the
test unaided/unassisted.
■■
The tests required candidates to complete an online ‘honesty contract’.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
33
Test administrators are responsible
for setting up the verification
test and providing instructions to
candidates about the process.
Post-test criteria included:
■■
Consistency of cut-off scores. As long as candidates scored above a specific cut-off
score, they were eligible for further consideration.
■■
Use of the result as one small component of the overall decision. The organisation
decided that the ability test result was only a small part of the overall decision
criteria.
■■
Candidates were subjected to an in-depth, 3-day assessment centre process that
included simulation and scenario-based workshops designed to evaluate reasoning
and decision-making capabilities. These workshops provided an effective picture
of each candidate’s reasoning abilities and were used to confirm candidates’ test
scores.
■■
Trained interviewers that understood the links between the results of the critical
reasoning tests and candidate performance in structured competency-based
interviews. This allowed them to pick up on inconsistencies between test scores and
actual candidate performance in the interviews.
■■
The hiring decisions were not made until candidates were referred to the specific
division managers and had completed several other on-site interviews within the
division. These managers did not see or use the test scores in their decision-making
process.
Finally, the organisation clearly communicated the option to retest candidates at any
point in the process. The team decided that, if other data consistently indicated the
test scores did not represent other criteria used to verify the scores, candidates could
be asked to sit another test.
Combined, all of these measures built the confidence of the corporate leadership
team, the stakeholders within the organisation and the candidates themselves, that
the process was extraordinarily thorough and secure, and that an integrated retest
process was not warranted.
H: Supervising the Verification Test
Test administrators are responsible for setting up the verification test and providing
instructions to candidates about the process. Well-designed online tests should provide
most of the guidance and instructions to candidates on screen. Test administrators
are also responsible for candidate authentication, controlling, monitoring, and
communicating certain elements of a verification test, including:
■■
Ensuring the testing environment is quiet and free from interruption.
■■
Monitoring test takers for inappropriate behaviours, such as collusion or use of
disallowed assistance (e.g. mobile phones).
■■
Knowing how to activate or start the test for candidates.
■■
Providing sufficient guidance to candidates if they have questions about the
instructions, example questions, or testing process.
■■
Knowing what to do in the case of test interruption or technical problem with the
computer system.
■■
Providing calculators and paper for notes/calculations when appropriate. Some
tests allow test takers to use calculators and/or paper for calculations or notes.
Provide ready access to these materials for test takers or clearly inform them who to
contact to get these materials before they begin the test.
■■
Collecting paper used for notes/calculations after each test taker finishes. It is
important to collect and destroy paper used by test takers for notes and calculations
to prevent other candidates from using these materials.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
34
Even though verification testing is used to confirm the results of an unsupervised
online test completed earlier, companies still hold the responsibility of maintaining the
appropriate test environment for the verification tests. Test administrators responsible
for verification tests should follow most existing guidelines for test administration.
Depending upon the form of verification testing, test administrators may be required to
explain details of the test itself and how the results will be used. SHL also recommends
that test administrators who are coordinating verification tests, understand the
organisation’s policies on the frequency of verification testing and the selection process
used. For example, if the organisation uses random selection to test only a subset
of candidates using a verification test, candidates selected should be told that their
selection is random.
As with current best practice, it is important that test administrators have access to
a trained and/or qualified professional who understands the technical details and
interpretation of the test(s). Depending upon corporate guidelines or local legislation,
this qualified user may need to be within the organisation. In other situations, the test
publisher or agent may be able to provide this expertise.
I: Handling Verification Discrepancies
Verification procedures should always be handled sensitively. If the results of any
supervised online test are not verified by a verification test or other measure, this does
not automatically mean that a candidate has cheated or acted dishonestly. A discrepancy
highlighted by the verification procedure should not by itself cause an individual to
be eliminated from consideration, or removed from the process. The cause of the
discrepancy should be investigated with the individual.
Communicating the result of a verification discrepancy to the individual.
Before the feedback session begins:
■■
Do not assume the discrepancy is a result of a candidate’s dishonesty (cheating).
There can be other reasons for discrepancies between test results.
■■
If the feedback provider did not supervise the test, he or she should ask the test
supervisor if any technical or environmental factors during the verification test
might have resulted in a person getting an abnormally low score.
■■
Ensure that the feedback provider has access to other candidate information that
may be helpful to provide independent confirmation of a candidate’s ability level,
such as CV/resumé information, other assessment results, etc.
■■
Ensure that the company’s verification procedures were appropriately followed.
■■
Verify that the feedback session with the candidate can occur in a private location,
away from other candidates or distractions.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
35
A discrepancy highlighted by the
verification procedure should not
by itself cause an individual to be
eliminated from consideration, or
removed from the process.
Initiating the feedback session:
■■
Feedback providers should set the candidate at ease.
■■
Explain to the candidate that there is a significant discrepancy between the
candidate’s original ability test results and the results of the verification test.
■■
Explain that the verification tests are a standard part of the testing process and that
it is not uncommon for discrepancies to occur.
■■
Explain that the verification tests are used to ensure that the process is fair and
transparent and it is not directed at specific individuals.
■■
Explain that there maybe legitimate reasons for large discrepancies and that this
feedback session is intended to resolve any questions about this discrepancy.
■■
Explain that the discrepancy, by itself, does not mean the candidate is being
eliminated for consideration.
■■
Explain that the feedback session will help the company (test administrator)
determine if there were any specific issues that could have influenced the
candidate’s poorer performance in the verification test.
■■
Ask the candidate if they had any questions or concerns about the verification
process at this time. Once the introduction to the session is conducted, explore the
results with the candidate.
Explore the result with the individual.
A discussion with the individual can provide insight into why the two results may be
different. There are several factors that may have influenced an individual’s performance
and caused inconsistent test results, including:
■■
the individual’s emotional or physical well being during the tests.
■■
test anxiety, unease, or other conditions.
Explore these factors with the individual to determine if they influenced their
performance. Specific prompt questions (such as the ones below) can help structure a
discussion with the individual.
■■
What was his or her strategy or approach to the verification test regarding speed
and accuracy?
■■
Did the candidate understand the test was timed?
■■
Did the candidate focus on answering all the questions, or was he or she more
focused on accuracy of the items answered?
■■
How did he or she feel about completing the second (supervised) test compared to
the first test?
■■
Was the test taker significantly more nervous during the verification test?
■■
Did something happen during the verification test, such as technical problems,
disruptions, etc. that may have impacted the candidate’s performance?
■■
How seriously did he or she take the verification test?
■■
Is there any explanation they can suggest as to why their performance on the
verification test was significantly worse than on the original unsupervised test?
■■
Any emotional, physical, environmental or other condition that could have affected
their performance? (Note: appropriate caution should be taken when questioning
candidates that may involve disclosure of disabilities. Not all jurisdictions allow for
this form of questioning. Consult with a local qualified test administrator.)
■■
Did the candidate complete the verification test using appropriate assistance
devices, e.g. screen reader for vision disabilities?
■■
Did the candidate experience any illness, condition, or other physical distress
while completing the verification test that may have impacted the results?
Other techniques for managing feedback with candidates can be found in many
occupational testing manuals and guides.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
36
Decide on the most appropriate course of action.
If there is sufficient evidence that suggests that legitimate factors may have been
involved in a discrepancy, the following two options are open:
■■
If the discussion indicates the unsupervised test scores are likely to be an accurate
measure of the person’s true ability, the existing scores can be used as normal.
■■
If the individual can offer no good reason for the discrepancy, administering another
ability test in a supervised setting is recommended. The result can be used in place
of the unsupervised ability test to determine the person’s ability level.
The advantages of using a follow-up test to resolve any discrepancies include consistency
of measurement, candidate perception of fairness and objectivity of the measure.
Disadvantages include additional time and cost to administer another test, difficulty in
comparing multiple test results and the disadvantages to the candidate experience as a
result of taking another test.
Guidelines for using an additional ability test to resolve any discrepancies include:
■■
Administer the follow-up test in a supervised setting.
■■
This helps to resolve any concerns around environmental factors interfering with a
candidate’s performance.
■■
Ideally use a test that is administered in the same format (e.g. computer-based test)
as the unsupervised test.
■■
This reduces the risk of test results caused by the administration format. This can
make comparisons between results more consistent.
■■
If a different ability test is used for the follow-up test, it is important to understand
how to equate the results for comparison purposes. It is essential that any norms
used be as close to the original unsupervised ability test’s comparison group as
possible.
■■
In all cases, the results of the follow-up test should be used as a replacement for the
unsupervised test result.
We recommend this method as the ideal way to resolve discrepancies for companies
that have the ability to conveniently administer a test. This method is often the most
consistent and objective method to resolve discrepancies.
The SHL Verify ability tests allow for the administration of another randomised test in a
supervised setting. This simplifies the comparisons between test scores because the tests
used in both instances are directly comparable and use the same comparison group data.
The final decision on how to confirm and use the person’s test results should follow
internal policies and guidelines. Organisations should evaluate the risks involved, their
own corporate policy/governance, the use of other screening and selection tools, time,
cost and other factors. All of these may be important when deciding how to proceed with
discrepant results.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
37
J: Monitoring Test Usage and Results
Existing best practice requires that the use of tests and other psychometric instruments
be continually monitored to ensure continued appropriateness and effectiveness. Online
unsupervised testing makes test monitoring even more critical. Since online tests often
have content exposed publicly, the risks that the test content will become compromised,
and therefore compromise the results seen from the test, are greater.
Monitoring of online unsupervised tests should include the normal checks, including:
■■
ensuring that the tests remain relevant to the job.
■■
ensuring that test versions and norms remain up to date.
■■
monitoring of scores at regular intervals.
■■
monitoring by gender, ethnic group and other protected groups to identify any
adverse impact.
Online testing should additionally include:
■■
monitoring of test response patterns for isolated or wide-scale aberrant or deviant
patterns.
■■
monitoring sources and websites for possible selling or posting of test content.
Greater vigilance of online test content is required to ensure that negative impacts
of a compromised test are caught as early as possible. Test providers should provide
monitoring services, or other methods, to protect content from inappropriate use,
theft, or distribution.
K: Confidentiality, Storage of Results and Access to Test Materials
Data and information protection has become more complex with the introduction of
Internet-based systems and new regulations governing the storage and use of personal
data. However, best practices that currently govern the storage of results and access to
testing materials still apply to online tests.
Generally, test results should be treated as personal data and stored with due regard
to confidentiality. Access should be restricted to those with a need to know and in
accordance with what has been agreed with the respondent during administration and
feedback. Persons who are untrained should not be allowed access to raw data from tests,
but only to clearly described interpretations. Special considerations that organisations
may need to consider for online tests include:
■■
Control or restrict access to test scores in online recruitment or talent management
systems.
■■
Ensure that data storage systems (internal and vendor systems) allow for the proper
deletion of data as governed by local legislation governing personal data.
■■
Ensure that data storage systems provide adequate protection from intentional, or
unintentional disclosure, or theft.
■■
Ensure that any data transferred across legislative borders are governed by Safe
■■
Harbour or similar Data Protection Acts.
Best practices governing the security and storage of test materials for online tests are
different than those governing fixed or paper/pencil tests. Policies should still exist
that govern who can buy materials, where they are stored and who has access, but the
availability of the tests and results online may expand access to other users inside the
organisation that would not normally have had access to test materials in the past.
For most online tests today, scoring keys or other sensitive scoring information are no
longer available in a distributable form. Therefore, organisations are unlikely to have
scoring keys on site that must be protected and kept under lock and key. If scoring keys
are available, all existing best practice should apply to the protection of these materials.
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
38
Because of the potential for wider
use and access to tests and test
materials from online distribution,
it is critical that organisations
select tests that are developed and
designed for online use.
If online tests are fixed tests, appropriate due diligence and policy should be developed
internally to ensure that test items and content are not inappropriately copied and
distributed for unauthorised use. Whenever possible, randomised tests should be used to
manage the potential risk of fixed test items being inappropriately copied or disclosed.
Most reputable test providers have updated their guidance to allow for the access and
use of online tests and materials by minimally trained users, as long as the tests are
designed for online use and have been selected and implemented under the advice of a
qualified test user.
Because of the potential for wider use and access to tests and test materials from online
distribution, it is critical that organisations select tests that are developed and designed
for online use. This includes selecting tests that provide results or outputs that would be
appropriate for minimally trained users to use safely.
Conclusion and Summary
We believe that unsupervised testing online with the safeguards described does
represent a step change in the science and practice of objective assessment.
The steps outlined in the Guide mean that unsupervised online assessment is backed up
by sound science and good practice. By following the steps in this guide, organisations
will have created a roadmap for using unsupervised testing.
Our commitment to Better Practice
This Guide is not the only way in which we demonstrates its commitment to promoting
Better Practice.
The purpose of this Guide has been to look at the current state-of-play in terms of
what is happening to organisations and their employment processes, the challenges
that organisational requirements are presenting to the existing science and practice
of assessment, and to describe how practical steps can be taken by organisations to
secure the validity of their assessments as they move them increasingly online and
unsupervised.
But, Better Practice will develop through learning and the sharing of experiences. we
therefore provided an email address: [email protected] through which comments
on the Guide can be fed back so that improvements can continually be made. Views, case
histories as well as suggestions, are encouraged, so that learning and experience can be
shared across the assessment community.
This Guide can be downloaded from www.ceb.shl.com/betterpractice
The application notes in step 6 will continually evolve and be updated regularly - these
will also be available for individual download from the same site
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
39
Appendix
References
i. Bartram, D. & Hambleton, R.K. (Eds) (2006): ‘Computer-Based Testing and the
Internet: Issues and Advances’. Chichester, UK: Wiley.
ii. Tippins, N.T., Beaty, J.,Drasgow, F., Gibson, W.M., Pearlman, K., Segall, D.O. &
Shepherd, W. (2006) ‘Unproctored Internet testing in employment settings’
Personnel Psychology, 59, 189-225.
iii.Weiner, J. & Reynolds, D. (2006): ‘Issues in Unproctored Online Testing’ (paper
presented at the Association of Test Publishers Conference, Orlando, Florida,
February 2006).
iv.Burke, E. (2006): ‘Better Practice for Unsupervised Online Assessment’ White
Paper. Thames Ditton, UK. SHL Group.
v. International Test Commission (2006): ‘International Guidelines on Computerbased and Internet-delivered Testing’. International Journal of Testing, 6, 143-172
Resources and Additional Information
The American Psychological Association (APA) - based in Washington, DC, the APA is
a scientific and professional organisation that represents psychology in the United States.
With 150,000 members, APA is the largest association of psychologists worldwide. For
further information, go to www.apa.org
The British Psychology Society (BPS) is the representative body for psychologists and
psychology in the UK. The BPS aims to encourage the development of psychology as a
scientific discipline and an applied profession, to raise standards of training and practice
in the application of psychology, to raise public awareness of psychology and increase the
influence of psychological practice in the society. For further information, go to www.
bps.org.uk. For information on testing go to www.psychtesting.org.uk
The Society for Industrial and Organizational Psychology (SIOP) aims to enhance
human well-being and performance in organisational and work settings by promoting
the science, practice, and teaching of industrial-organisational psychology. For further
information, go to: www.siop.org
The International Test Commission (ITC) is an association of national psychological
associations, test commissions, publishers and other organisations committed to
promoting effective testing and assessment policies and to the proper development,
evaluation and uses of educational and psychological instruments. Its membership
covers most of the Western and Eastern European Countries and North America, as well
as some countries in the Middle and Far East, South America and Africa. For further
information and access to ITC Guidelines, go to www.intestcom.org
Caveon Test Security were the first end-to-end test security services firm to offer
protection against cheating and test fraud with detection services to identify breaches,
remediation to leverage legal process to halt abuses of sensitive test information, and
prevention to secure tests from compromise. For further information, go to:
www.caveon.com
Contact Us to Learn More
[email protected]
www.ceb.shl.com
www.ceb.shl.com
UNSUPERVISED ONLINE ASSESSMENT BEST PRACTICE GUIDE
© 2013 SHL, a part of CEB. All rights reserved.
UOABPG-102013-UKeng-GLO
40
Registered office: The Pavilion, 1 Atwell Place, Thames Ditton, Surrey, KT7 0NE, UK