Volume 19 No. 4 August 2003
ISSN 0963-2638
&&
Selection
Selection
Development
Development
Review
Review
PUBLISHED BY THE BRITISH PSYCHOLOGICAL SOCIETY
SDR
Editorial
C O N T E N T S
Survey of Good Practice Issues:
Internet Delivered Personality Testing
in the UK
Iain Coyne, Dave Bartram &
Penelope Smith-Lee Chong
3
A review of selected tests from the
7
Vienna Test System
Allan Whiteside, Giselle Parker & Richard Snodgrass
Five reasons to be optimistic about the
future of personality assessment as a
selection tool
Stephen A. Woods
12
The Big Five revisted: Where are we now?
Wendy Lord & John Rust
15
Performance Coaching – Spin or Win?
Francesca Buck
19
Selection & Development Review editorial team
Dr John Boddy
16 Tarrws Close, Wenvoe, Cardiff CF5 6BT.
Tel: 029 2059 9233 Fax: 029 2059 7399
E-mail: [email protected]
Stuart Duff, Stephan Lucks & Ceri Roderick
Pearn Kandola Occupational Psychologists,
76 Banbury Road, Oxford OX2 6JT.
Tel: 01865 516202 Fax: 01865 510182
E-mail: [email protected]
Philippa Hain
98 Plymouth Road, Penarth CF64 5DL.
Tel: 07816 919857 E-mail: [email protected]
Dr Tuvia Melamed
The Quo Group Ltd, 2 The Oaks, Clews Road,
Redditch B98 7ST. Tel: 01527 540057
Fax: 01527 547527 Mobile: 0777 5643128.
E-mail: [email protected]
Consulting Editors: Dr S. Blinkhorn;
Professor V. Dulewicz; Professor N. Anderson
Published six times a year by The British Psychological Society,
St Andrews House, 48 Princess Rd East, Leicester LE1 7DR at £37
(US $50 overseas) a year. Tel: 0116 254 9568. Fax: 0116 247 0787.
E-mail: [email protected]. ISSN 0963-2638
SDR is a review of current issues in selection, assessment and
development. It is not intended to be an academic journal and
while papers are subject to editorial review not all are referred to
independent referees. Any material intended for publication
should be sent by e-mail if possible. The guideline maximum
length is 2000 words.
The views expressed in articles do not necessarily represent
those of the Editorial team or The British Psychological Society.
2
WELCOME TO THE August issue of SDR. Personality testing is
the main theme this month with various articles debating
the importance of the big five model, examining different
psychometric instruments and exploring good practice
issues.
We start with Iain Coyne, Dave Bartram and Penelope
Smith-Lee Chong who look at good practice issues
involved in the delivery of personality testing on the
internet within the UK. Iain, Dave and Penelope draw our
attention to the fact that this is a growing industry which
undoubtedly requires a great deal of scrutiny and research
to ensure that the pro’s and con’s of using this form of
technology are clear to all who get involved. They
conclude from their research that the test publishers
involved in their sample were adhering to good practice
issues but as they point out, it is the test user that we need
to educate to ensure that they fully understand the issues.
While on the subject of using new technology in
psychometric testing, Allan Whiteside, Giselle Parker and
Richard Snodgrass have reviewed a selection of computeraided tests from the Vienna Series over a period of three
months.They are interested in those tests that have some
application within occupational psychology and, in particular, feel that they could be used within those areas of
recruitment, training and rehabilitation.
Assessing personality traits through psychometric
assessment for the use of both selection and development
has attracted much research attention and is a growing
industry for those in practice. Steve Woods expands on this
and gives us ‘five reasons to be optimistic about the future
of personality assessment as a selection tool’. Steve
examines the use of the Five Factor Model, explores its
history in the area of personality assessment and explains
how it provides clear benefits for application in occupational settings. He concludes that this area has flourished,
and should continue to do so, because ‘intuitively, personality matters’.
Continuing this theme, Wendy Lord and John Rust
argue that the Five Factor Model has become ‘the linchpin
that holds work-based personality assessment together’.
Wendy and John examine the relevance of the Five Factor
Model for occupational assessment. They have concentrated more particularly on looking at the NEO family of
personality inventories which ‘have become the international gold standard for direct measurement of the Five
Factor Model’, and those which are due to be standardised
in the UK shortly.
In our final article, Francesca Buck looks at the issue of
performance coaching. She argues that not enough work
has been done in evaluating and supporting the claims
about the benefits that coaching can deliver. In light of
this, Francesca carried out a thorough and comprehensive
evaluation of a two-day coaching programme within a
large UK retailer. The training programme in question
covers practical sessions alongside some of the key
aspects of coaching such as giving feedback, development
planning and goal setting.
Many thanks go to all those who have contributed to
this issue of SDR. As usual, we would welcome any
comments, responses or new papers around these or any
other issues within selection and development. Until the
next time, I wish you a warm and happy summer.
Philippa Hain
On behalf of the SDR editorial team
Selection & Development Review, Vol. 19 No. 4, August 2003
SDR
Survey of Good Practice
Issues: Internet Delivered
Personality Testing in the UK
Introduction
AS THE MARKET for Internet-delivered computerbased testing develops, and as the technological
sophistication of the products increases, so the
issue of ensuring those using such assessment
tools follow good practice will increase in importance. Given this, and following on from the
successful design, development and formal adoption of the International Guidelines for Test Use
(ITC, 2001), the International Test Commission
(ITC) decided to examine the issue of developing
guidelines for computer-based/Internet-based
testing (see http://www.intestcom.org/). Initial
research on this project identified that there was a
need for a more systematic survey to be undertaken of internet-based testing sites in relation to
good practice issues. The ITC, therefore, decided
to investigate good practice issues in relation to
Internet testing focusing initially on personality
testing. The initial focus was on personality tests
because they are the most widely available from
the UK test publishing companies.
Method
The sample consisted of test publishers whose
personality test was reviewed in the most recent
British Psychological Society Review (Lindley,
2000) and, to increase sample size, test publishers
whose tests were not in the review but who
offered internet-based personality testing. The
sample size consisted of 12 responses from the 14
test publishers contacted (an 85.7 per cent
response rate) and most of the well-known
personality tests in the UK were represented.The
length of time the participating test publishers
had been conducting internet-based personality
testing ranged from three months to seven years.
Iain Coyne, Dave Bartram &
Penelope Smith-Lee Chong
Data was collected via the use of a combination of
an initial survey questionnaire complemented by
follow-up interviews.The survey was divided into
five parts consisting of questions relating to Good
Practice, Performance, Test Administration,
Security and Fairness. On receipt of the questionnaire from participants, interview prompts were
developed and telephone interview sessions were
then arranged to clarify responses further.
Results
Issues of Good Practice
This section related to whether the test publishers
had a policy on good practice issues and what
issues were either contained within the policy or
should be considered important good practice
issues. The majority of the test publishers who
responded (83 per cent) had a policy regarding
good practice and for most of these (70 per cent)
it was an internal policy. Some of the participants
(40 per cent) reported they had faced problems
with putting such a policy into practice, with a
notable issue being commercial considerations
which may create pressures that run counter to
elements of good practice. Another issue raised
was the suggestion that a policy would need to be
more prescriptive for the internet because of the
different skills required by test users in internet
testing.
From the policy statements that the researchers
obtained and from concerns from the other
participants, a number of issues were covered:
Selection & Development Review, Vol. 19 No. 4, August 2003
3
●
●
●
Technical issues – understanding the technology, reliability of the technology, simplicity
of access and use by test takers.
Good practice issues in the use of testing –
briefing candidates before testing, providing
feedback at a distance (and the impact this may
have on training courses for potential test
users) and making the operation from administration to feedback appear ‘seamless’.
Ethical test use – policy on who can use the test
and under what conditions (level of supervision) the test should be used, the security and
confidentiality of materials and candidate data,
how to balance the ‘cold’ medium of internet
testing with the ‘human’ activity of personality
testing.
Issues of Performance
This section was designed to assess the performance of the internet as a test delivery medium in
terms of the timing of delivery and the robustness
of the test. Some technological limitations were
indicated and these included browser problems
and poor Internet Service Providers (ISP’s). As
well as technology issues, there were also
problems in relation to test taker and test user
knowledge and performance. These included the
candidate’s and user’s knowledge of basic
computer functions; training users in correctly
administering online testing; ensuring test takers
correctly enter access codes; and the need for IT
technical knowledge by support staff.
In terms of handling connection failures midtest, candidates could either re-start the test
completely or re-start the test at the point at
which the connection failed. Some organisations
used an applet approach and hence connection
failures mid-test were not a particular problem.
Similarly, a number of approaches were used to
handle slow connections. Internet Explorer and
Netscape were the browsers commonly used by
the test publishers but differences were seen in
the versions used. Most participants (75 per cent)
stated that they did test for the consistency of
appearance and operation of the test across
browsers.
4
Issues of Test Administration
Bartram (2001) has defined a number of modes of
test administration:
● Open – no human supervisor nor pre-identification.
● Controlled unsupervised – no human supervision but test-taker requires a log-in name and
password.
● Controlled and supervised – a level of human
supervision where test administrator logs-in a
candidate and confirms the testing was
completed correctly.
● Managed – high level of human supervision and
control.
Over 90 per cent of the respondents used the
controlled unsupervised method for internet
delivered personality testing. Only 33 per cent
used open mode, 66 per cent controlled and
supervised, and 42 per cent used managed (participants could choose more than one option).Twothirds of participants stated that they would not
vary the mode of administration for different
testing scenarios.
Four of the test publishers use test-taker honesty
policies where the test taker agrees that he/she has
given correct information and no cheating has
occurred. In terms of managing cheating, test
publishers recommend that information obtained
is corroborated and validated in feedback or an
interview context. Also, tests variously include
validity or lie scales as well as using ipsative item
formats that restrict response bias.
All respondents stated that feedback is
provided to the candidate, but differences
emerged in how it is given and who actually gives
it. Generally, it is the test user or client that is
responsible for giving the feedback to the test
taker and this is achieved by face-to-face feedback,
e-mail or mailed report to the test taker and telephone feedback (combinations of these were
used). Reports tended to be tailored to different
audiences such as a test taker, test user (e.g. line
manager, HR professional or other person) or a
third party recipient.
All of the tests surveyed apart from one were
standardised tests. For these tests, the majority
used the same norms as used in the paper and
pencil assessment (only two had specific internet
Selection & Development Review, Vol. 19 No. 4, August 2003
norms) and all but one of the participants stated
that they had undertaken an equivalency study to
ensure the Internet delivered test is equivalent to
the paper version.
Issues of Security
This section was concerned with how secure the
internet is for testing in terms of the test itself, the
test takers identity and the test results. All test
publishers surveyed stated that they had a policy
on how test-taker assessment data is kept. Aspects
detailed within this policy included: information
on how long data should be kept; who should
have access to the data; what the data can and
cannot be used for; how the policy relates to EU
data protection laws; and good practice requirements for clients on keeping data. In maintaining
the security of the test, publishers use secure
server sites, encryption devices, embedded
security devices in the software, password protection and firewalls. Further, scoring algorithms and
norms are held on the server and not held within
the pages the applicant completes.
Issues of Fairness
This issue revolves around the notion of a digital
divide between those who have access to the
internet and those who do not. Just under a half of
the test publishers surveyed collect data on the
number of people from protected minority groups
who take their internet test but 90 per cent felt
that there was unlikely to be a problem of differential access if the test was only offered on the
internet.
Conclusions
Although this study is based on only a small
sample (restricted to personality testing and to the
UK), the results suggest that test publishers have
considered and taken steps to prevent problems
relating to good practice, performance, test administration, security and fairness in Internet testing.
However, a number of key issues emerged.
Commercial pressures always have to be
balanced against good practice issues. New technology raises the risks of this balance being
shifted in ways that may unnecessarily compromise commercial considerations on the one hand
or expose users and test takers to avoidable risks
on the other.The question of to what extent do, or
will, commercial needs compromise adherence to
good practice guidelines remains unanswered, as
the answer to it depends on assessment of the
risks associated with new practices (such as unsupervised administration of personality questionnaires in high stakes assessment conditions). The
risks can be empirically quantified through
research, and we would encourage such research
to be carried out and placed in the public domain.
In practice, it is test users not publishers who
use tests. So the major issue for ensuring good
practice lies in the education of test users. Here
publishers have a responsibility towards their
clients to ensure that they understand good
practice issues. The internet has not changed the
fact that the burden of responsibility for use lies
with the test user. However, the perception may
change when the test-taker never meets the test
user and is directed to a web site that is branded
as the publisher’s site. The perception from the
test-taker could be that the publisher is responsible for the whole testing process rather than the
test user.
The need for different training and knowledge
in users and test takers was a further point to
emerge from this survey. Not only will users
require training in good test practice they may
also require training in some of the technological
aspects of internet testing. Further, it appears that
the provision of feedback may become much
more of an important issue within internet
testing. It is at this stage that ‘faceless’ internet
testing would become more ‘human’. Perhaps
training requirements for test users need to
concentrate less on the psychometric aspects of
testing (especially if the internet test operates as a
‘black box’) and more on the understanding of
reports and feedback.This does not mean that the
psychometrics qualities of the test are less important, but rather that the user does not have to
know how to work ‘under the bonnet’ and so
more effort can be focused on using the information the test provides.
Only a half of organisations did collect data on
fairness issues. In reality, this is considered a
client’s responsibility. However, it may be worth
Selection & Development Review, Vol. 19 No. 4, August 2003
5
publishers taking a more proactive role in encouraging clients to collect such data as a matter of
course, as this could be mutually beneficial.
An overall theme that did emerge from the
research was whether some of these issues are
actually internet-testing specific. Granted some of
the technological factors (such as slow connections and connection failures) are internetspecific, but others are just as much an issue for
computer-based or paper and pencil testing. For
example, the security of test taker data or security
of the test items and scoring keys are as much an
issue with computer-based testing as with paper
and pencil testing. Perhaps, there is a false impression that the internet throws up a whole new
range of new testing issues when in fact they are
just variations on old themes.
Finally, some of the issues may not necessarily
be relevant to personality testing or may be more
of concern in other countries. Some of the technology problems, issues to do with cheating and
problems of establishing equivalency may be
more of a concern for internet based ability and
aptitude testing. Access and good practice may
not necessarily be as good in other countries as
within the UK. Further research looking at these
two issues would be useful.
6
References
Bartram, D. (2001).The impact of the internet on
testing: Issues that need to be addressed by a
Code of Good Practice. Report for the BPS
Steering Committee on Test Standards.
International Test Commission (2001).
International Guidelines for Test Use.
International Journal of Testing, 1, 93–114.
Lindley, P. (Ed.) (2000). Review of personality
assessment instruments (Level B) for use in
occupational settings. Leicester: BPS Books.
Dr Iain Coyne – Department of Psychology,
University of Hull, HU6 7RX.
Tel: (01482) 465592.
E-mail: [email protected]
Professor Dave Bartram – SHL Group plc,
1 Atwell Place, Thames Ditton, Surrey.
Tel: 020 8335 8211.
E-mail: [email protected]
Penny Smith-Lee Chong – University of Hull.
E-mail: [email protected].
Selection & Development Review, Vol. 19 No. 4, August 2003
SDR
A review of selected tests
from the Vienna Test System*
This article reviews selected tests from the
Vienna Series. The tests were chosen for their
potential inclusion in the battery of computerised tests used by Jobcentre Plus Occupational
Psychologists in employment assessments. These
tests were trialled over a three-month period.
An evaluation of their practical effectiveness for
Occupational Psychologists follows. Although
this battery was piloted for specific use in
helping disabled people back into work, the
authors feel that it has wider application for
Occupational Psychologists in other fields.
Introduction
THE VIENNA SYSTEM contains over 60 computeraided tests, the majority of which are tools
designed for clinical or research purposes. There
are also personality tests, intelligence tests and a
facility to design and customise tests.
The tests that may be of use for Occupational
Psychologists are centred around memory, attention, concentration, fatigue, distractibility and
general cognitive ability.
This article reviews the effectiveness of eight
psychometric tests, selected from the Vienna
Series (see also Whiteside, 2003).
These tests are:
● Continuous Visual Recognition Task;
● Attitudes to Work;
● Continuous Attention;
● Cognitrone;
● Work Performance Series;
● Mechanical-Technical Perceptive Ability;
● Visual Memory Test;
● Verbal Learning Test.
Allan Whiteside, Giselle Parker &
Richard Snodgrass
Format
All the identified tests are computerised, designed
to be self-administered and at a ‘work at your own
pace’ level. Each test contains a practice session
and allows the individual to acquaint themselves
with the use of a PC. The tests are very user
friendly, relatively short and contain manuals,
selected norms and theoretical and practical information for the administrator. Some of the tests are
designed to be run with panels, lightpens/touchscreens and other specially designed input
devices (not reported here).
Cost of materials
All tests come with a seven-year licence and prices
are competitive compared to pencil and paper
tests, typically averaging £400, plus cost of the
basic software £560.16. Licences are available at
lower cost and discounts are available when five
systems or more are purchased.
Purpose of the selected tests
The eight tests selected assess different aspects of
memory, attention, concentration, fatigue,
distractibility and general cognitive ability.
Continuous Visual Recognition Task
This exercise examines memory performance for
normal and head injured clients. Depending on
the test form, words, objects, numbers, meaningless symbols, letter-number combinations, or
* This article represents the views of the Authors and does not necessarily represent the views and policies of the Dept. of Work
and Pensions Jobcentre Plus.
Selection & Development Review, Vol. 19 No. 4, August 2003
7
difficult to name items, appear in sequence. The
client has to decide whether an item is being
shown for the first time or is being repeated on
screen.The approximate administration time is 15
minutes and in conjunction with other tests in the
battery may replace the Wechsler Memory Scales
(WMS 111) (Psychological Corporation) and the
Rivermead Behavioural Memory Test (Wilson,
Cockburn & Baddely, 1987).
Attitudes Towards Work
This test is designed to look at personnel selection, and career and educational counselling. It
involves comparing area size, coding symbols and
differentiating between shapes.The test examines
‘frustration tolerance’, ‘performance motivation’
and ‘impulsiveness/reflexivity’. This offers a
unique addition to currently available tests. The
total administration time is 45 minutes.
Continuous Attention
This provides a measurement of ‘long term attention and concentration performance’ and is particularly useful in identifying difficulties along the
autism spectrum axis.The client is presented with
triangles arranged in a row under time-critical
conditions; the individual then has to continually
decide on the number of triangles pointing down.
Administration time is around 20 minutes. This
measure could be incorporated into an assessment for Attention Deficit Hyperactivity Disorder
screening in young adults.
Work Performance Series
This is a general performance test for the assessment of concentration ability, mental saturation
and fatigue in the workplace.The individual has to
solve simple addition over a period of 20 minutes
thus placing demands on short-term memory.The
exercise also looks at distractibility and sustained
attention to detail. This series could be utilised
with other tests in the battery to look at memory;
concentration and attention related difficulties.
Mechanical-Technical Perceptive Ability
This aptitude test looks at engineering and technical skills and is similar in format to the Saville &
Holdsworth Technical Test Battery (Edgar & Tuton,
8
1987), however the client has the advantage of
actually seeing the working parts of the diagram.
The exercise can aid in advising people who are
interested in pursuing a career in mechanical-technical occupations and those looking to re-train in
this field of work. This test could be used in
conjunction with more portable work samples
such as the Minnesota Rate of Manipulation Test
and Crawfords Small Parts Dexterity Test (Bartram,
Lindley & Foster, 1990) when looking at technical
aptitude and hand skills.The administration time is
around 25 minutes; this exercise could replace
current engineering aptitude tests that involve
time consuming marking procedures.
Visual Memory Test
The Visual Memory Test is a one dimensional, adaptive test assessing memory ability in perceiving
and reproducing visual information. The client is
shown a map of a city on which typical places are
marked.The person then has to recall the correct
locations of a series of facilities. The test takes
around 20 minutes and could form part of a
battery to replace current diagnostic memory
tests.
Verbal Learning Test
The VLT examines individual learning ability for
verbal memory material. The client is asked to
recognise and respond to the repetition of meaningless words. It can also be used to assess specific
disorders of memory functions occurring after
brain injury.The exercise is around 15 minutes in
duration and, once again, could form part of a test
battery that may be quicker and more accurate
than psychometric tools in present circulation.
Cognitrone
Cognitrone is a general performance test for the
registration of attention and concentration. It
examines the effort, quality and level of attention
applied to a visual information-processing task.
The client has to compare figures against a
standard grid; this test assesses speed, acuity and
constancy of attentiveness. The total administration time is approximately 15 minutes. This is an
alternative to WMS 111.
Selection & Development Review, Vol. 19 No. 4, August 2003
What occupations are they
relevant for?
Although these tests do not provide specific information on occupational area of use, they can be
applied to the assessment of clients in any occupational sphere where demands on memory, attention, concentration, alertness and non-distractibility
are likely to play a significant role in their job
performance.
Reliability
Depending on the test form and comparative
samples, coefficients for ‘internal consistency’
(Cronbach’s Alpha) have been found between
0.77 and 0.96.
Validity
A large number of studies for individual validity
concepts are available in the corresponding test
manuals. Criterion-related validity, construct
validity, content validity, discriminant validity and
extreme-group validity are all addressed in great
detail, the results of which indicate that the tests
are valid measurement tools.
Norms
The norm groups vary according to the particular
tests. In general, sample sizes are quoted between
110 and 2820 (n=111 and n=2819). Norm groups
are taken from all over Europe, including the UK.
Once again dependent on the test, differing occupational and educational standards are noted,
along with general samples and head injury
comparisons.
Restrictions on usage
The tests are designed for use by ‘suitably qualified
professionals’ including: Occupational, Clinical and
Educational Psychologists. No particular level of
qualification is quoted by the publishers, however
the authors believe that a Level A standard should
be the appropriate baseline for usage (Schuhfried
also gives training and certification that the correct
level of competency has been reached).
and as such can be used with a range of
individuals.
Throughout the three-month period these tests
have been ‘trialled’ in an employment assessment
situation with disabled people.
The majority of these tests, by nature of their
properties, were administrated to assess specific
aspects of cognitive functioning in clients.
For example, AL presented with a number of
cognitive problems following a head injury. These
included short-term memory, concentration and
attention difficulties. As part of the test battery
used with AL, four tests from the Vienna Series
were selected: Visual Memory Test, Continuous
Visual Recognition Task, Cognitrone and
Attitude Towards Work. These tests were implemented to assess the visual and recognition aspect
of short-term memory, concentration, frustration
tolerance, performance motivation and impulsiveness/reflexivity.
In order to complete these tests AL was
required to follow the on-screen instructions.
These were read aloud to AL in order to ensure his
understanding. AL was then required to either
operate the mouse or specifically coloured keys to
proceed with the test. Although AL had limited
computer skills he was able to complete the test
with few (operational) difficulties.
The results obtained indicated significant difficulties with visual memory and concentration.
These results were confirmed by reported difficulties and inferences drawn from the administration of further tests, i.e.WAIS III.
The specific assessment benefits, in this
instance, were the provision of independent
measurements of concentration, frustration tolerance, performance motivation and impulsiveness/reflexivity. Usually these aspects tend not to
be tested directly, but rather as an overall observation of the assessment process. The use of the
Vienna tests allowed for a direct measure of these
facets of cognitive functioning.
Advantages of using this battery
●
Disability
The Vienna Series was designed for use in many
aspects, including clinical and research purposes,
●
●
The battery is self-contained on a PC and, therefore, portable.
Most tests are less than 30 minutes in duration.
Test set-up time takes a minute or so.
Selection & Development Review, Vol. 19 No. 4, August 2003
9
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Self administration frees up valuable time.
A choice of norm groups is available for
selected clients.
A detailed print out can be accessed immediately.
Practice tests ensure client familiarity.
Test manuals are in built into the system.
Reliability and validity levels are extremely
favourable.
The tests are relatively inexpensive compared
to pencil and paper tests.
Manual storing, re-ordering and general paperwork is alleviated.
The greatest possible objectivity, accuracy, and
reliability.
Comprehension is ensured by means of tracked
instruction.
There is a high acceptance rate among users.
Time measurements are down to milliseconds.
There is a quick, error-free evaluation immediately following administration.
A database can be established for statistical
analysis.
Systems can also be networked.
All data is protected by password access.
Print out reports are easily generated.Additional
results and automatic reporting can be configured into the system. There is also room for
additional comments.
Can these tests make a significant
contribution to Occupational
Psychologists?
Through personal use of the Vienna Series it may
be concluded that these tests could make a significant contribution to Occupational Psychologists
involved in recruitment, training and rehabilitation.
However, throughout the three-month trial it
should be noted that, certain tests were used on a
more frequent basis. These were the Visual
Memory Test, Continuous Visual Recognition
Task, Cognitrone and Attitude Towards Work,
Verbal Learning Test, Work Performance Test.
Both the Mechanical-Technical Ability Test and
Continuous Attention were seldom or never
used. Of these tests a case could be made for the
use of Continuous Attention, although on the
10
occasion this was administered the client was
unable to complete the exercise due to reported
‘eye strain’. (This test requires the participant to
respond to stimuli over an approximately 30
minute period.)
When deciding whether these tests would
make a useful contribution, the consideration of
whether the abilities measured by these test could
be obtained more effectively using current test
batteries should be made. This case could be
argued, especially for the assessment of memory.
The Visual Memory Test and the Continuous
Visual Recognition Task does assess these cognitive functions effectively, quickly and to a significant degree to allow for the consideration of
support needs in the workplace.
What tests can it replace?
Potentially, a selection of time-consuming psychometric diagnostic tests of memory, attention,
concentration, cognitive ability, mechanical
aptitude, mental stamina and fatigue may no
longer be necessary.
Conclusion
In conclusion, the Vienna Test System may well
become a very useful accessory for Occupational
Psychologists working in employment, training or
rehabilitation settings.
Following many years of psychometric testing,
many will agree that the after care service of some
test distributors is far from adequate. After three
months of working and liasing with the
Distributor the authors feel that the quality of
backup, guidance and assistance they provide is of
the highest standard
The Authors are in the process of reviewing
other tests from this system and will report on
those, which they believe, have potential for
Occupational Psychology.
References
Bartram, D., Lindley, P.A. & Foster, J. (1990).
Crawfords Small Parts Dexterity Tests (CSPD)
Rev 4.7 Review of Psychometric Tests for
Assessment in Vocational Training. Produced
by Newland Park Associates for the
Employment Department Group.
Selection & Development Review, Vol. 19 No. 4, August 2003
Bartram, D. (Ed.) (1997). Edgar, S. & Tuton, C. –
Rev 2.9. Technical Test Battery (TTB-2)
Review of Ability and Aptitude Tests
(Level A). Leicester: BPS Books.
WMS 111 Administration and Scoring Manual.
The Psychological Corporation, 1990
Wilson, B.A., Cockburn, J. & Baddely,A. (1985).
Rivermead Behavioural Memory Test. Thames
Valley Test Company.
Whiteside,A.A (2003). Synopsis of the Vienna Test
System:A computer-aided psychological
diagnosis. Journal of Occupational
Psychology Employment and Disability, 5(1),
41–50.
Test publisher for the Vienna Series: Schuhfried –
Hyrtlestrasse, 45 Moedeling, a-2340,Austria.
Test Distributors: Layfayette Instrument Europe,
4 Park Road, Sileby, Leics. LE12 7TJ.
Allan Whiteside, Giselle Parker &
Richard Snodgrass, Jobcentre Plus North West
Work Psychology Service.
[email protected]
[email protected]
[email protected]
Selection & Development Review, Vol. 19 No. 4, August 2003
11
SDR
Five reasons to be
optimistic about the future
of personality assessment
as a selection tool
RECENT ARTICLES IN SDR have addressed current
issues in personality and work psychology
research (McCredie, April, 2003; October/
December 2002; Gray, February 2003).This article
continues the discussion by highlighting some
areas that give both researchers and practitioners
who use personality measures cause for considerable optimism. These five contemporary issues
reflect the resurgence in personality assessment
for selection and development, and attempt to
identify some of its often over-looked advantages
and strengths.
The Big Five
The rise of personality assessment for selection
from past unpopularity has undoubtedly been
catalysed by the emergence of the five-factor
model of personality. The Big Five personality
factors (Extraversion, Agreeableness, Emotional
Stability, Conscientiousness and Openness)
emerged from the factor analytic tradition in
personality research from two sources. From the
lexical tradition (the study of personality trait in
everyday language), five-factor solutions are
routinely extracted from English trait sets
(Goldberg, 1992; 1993) and five factor, or highly
similar solutions, have been extracted from trait
adjective sets from numerous other languages
(Saucier et al.,2000).The second source is from the
questionnaire studies of Costa and McCrae using
their NEO-PI-R questionnaire (Costa & McCrae,
1992). In cross-cultural studies, they too have
extracted five-factor solutions from numerous
different languages (McCrae & Costa, 1997).
12
Stephen A. Woods MSc
The five-factor model presents three clear
benefits for application in work settings. Firstly, it
is a model of the structure of normal personality,
in contrast to past models, which were often
constructed for use with a clinical population.The
second advantage is that it offers an elusive property in psychology, namely consensus and a largely
agreed terminology. Small differences in representations of the Big Five by different researchers are
greatly outweighed by their underlying
consensus. Agreed terminology means that both
researchers and practitioners understand the
kinds of behavioural criteria that are described
under the term Extraversion, for example.
The third benefit has been the most useful in
occupational psychology. The Big Five offer a
simple and quantitative predictive framework.
Studies of the effect of personality on work performance are generally of the kind testing the association between personality and criteria, and the Big
Five offer a convenient and theoretically sound
way of filling the ‘personality’ side of such studies.
Furthermore, the ability of the Big Five to explain
previous models of personality has allowed earlier
studies to be reclassified so that the predictive
framework of the five factors can be applied retrospectively to this research. This has allowed
researchers to conduct meta-analyses using the
Big Five as a framework, producing the results that
have indicated the validity of personality measures
as selection tools.
Selection & Development Review, Vol. 19 No. 4, August 2003
Interpreting validity coefficients
Managers tell us personality matters
In his response to Gray (February, 2003), McCredie
(April, 2003) comments that validity values
reported by Barrick, Mount and Judge (2001)
rarely reach more than 0.3, values which he
describes as modest. Hogan and Roberts (2001)
comments that interpretations of validity coefficients in selection research can sometimes be
misleading. Job performance is multi-faceted and
extremely difficult to define even in terms of
multi-variate frameworks. It is therefore unrealistic
to view the ‘ceiling’ of meta-analysis validity as 1.
No selection method is ever likely to predict
performance criteria perfectly. Hogan and Roberts
comment that the ceiling of observed coefficients
for single predictors of performance might be best
viewed at about 0.5. When compared to this, 0.3
or the higher values associated with tests of
integrity actually seem reasonably positive and
indeed respectably high.
Reading critiques of both personality theory and
the use of personality assessment for selection
from the late 60s and early 70s would leave both
researchers and practitioners in the field feeling
quite low about their work.Why is it then that the
field of assessment has flourished in the last 15
years to become a growing and indeed highly
profitable industry? One reason is reasonably clear
in the context of work psychology. Intuitively,
personality matters. In my experience, asking
managers to describe their staff and their
strengths and weaknesses, invariably leads them to
refer to personality type characteristics.
Descriptions such as ‘they are a responsible
employee, focused and organised’ or ‘they can
handle criticism well’ are such examples. In a
study of managerial perceptions, Dunn et al.
(1995) reported that managers describe conscientiousness as being as important as factors related
to General Mental Ability (GMA) when they rated
interviewees as employable. Moreover, managers
identified conscientiousness, agreeableness, and
emotional stability as being most important for
judging potential counter-productivity.
Changing organisations
Worker autonomy is an important factor in determining whether the demonstration of certain
behaviours can be expected. For example, jobs on
a factory production line are highly structured,
and in such cases the opportunity to
autonomously demonstrate individually-different
behaviours is limited. In these instances, personality has little opportunity to be expressed and to
have any influence on work outcomes, which are
to an extent, predetermined. It is therefore reasonable to assume that the amount of autonomy a
worker has, moderates the strength of the personality – performance relationship. Barrick and
Mount (1993) supported this, finding that jobautonomy moderated the predictive validity of
conscientiousness, emotional stability and agreeableness.Witt et al. (2002) point out that modern
industry and organisations are becoming more
flexible and autonomous, and that therefore the
importance of personality in these new organisational cultures is increasing. Moreover, a case can
be made that the increased autonomy associated
with management jobs, makes personality assessment particularly relevant.
Equal opportunities
The current employment law climate has rightly
added the weight of potential litigation against
employers in an effort to ensure that equal opportunities are promoted at every stage of business in
UK organisations. Whilst meta-analyses have
consistently suggested that GMA is the strongest
predictor of performance, concerns remain
regarding the adverse impact caused by GMA
testing for minority-ethnic groups and to a lesser
extent for women.This is not the case for personality assessment. Studies of both US and UK
samples have highlighted that no such systematic
variance exists for personality assessment data
across specific groups (Ones & Anderson, 2002;
Ones & Viswesvaran, 1998).
In summary
The highlighted issues have illustrated that the
field of personality assessment for selection and
development is healthy, is attracting considerable
research attention, and also generating business
Selection & Development Review, Vol. 19 No. 4, August 2003
13
for consultants. The issues demonstrate that
personality assessment is a selection tool that is
underpinned by consensus regarding the structure of personality, and is also perceived as relevant by managers. Additionally, personality
assessment for selection is fair to candidates, and
is contemporary in terms of its relevance to organisational cultures. Most importantly, personality
does predict performance and with the outlined
advantages added to this, researchers and practitioners can be, and in most cases are, optimistic
about the future of the field.
References
Barrick, M.R. & Mount, M.K. (1993). Autonomy as
a moderator of the relationships between the
Big Five personality dimensions and job
performance. Journal of Applied Psychology,
78, 111–118.
Barrick, M.R., Mount, M.K. & Judge,T.A. (2001).
Personality and performance at the beginning
of the new millennium:What do we know and
where do we go next? International Journal
of Selection and Assessment, 9 (1/2), 9–30.
Costa, P.T., Jr., & McCrae, R.R. (1992). Revised
NEO Personality Inventory (NEO-PI-R) and
NEO Five-Factor Inventory (NEO-FFI)
professional manual. Odessa, FL:
Psychological Assessment Resources.
Dunn,W.S., Mount, M.K., Barrick, M.R. & Ones,
D.S. (1995). Relative importance of personality
and general mental ability in managers’
judgments of applicant qualifications.
Journal of Applied Psychology, 80, 500–509.
Goldberg, L.R. (1992).The development of
markers for the Big-Five factor structure.
Psychological Assessment, 4, 26–42.
14
Goldberg, L.R. (1993).The structure of
phenotypic personality traits. American
Psychologist, 48, 26–34.
Hogan, R.T. & Roberts, B.W. (2001). Introduction:
Personality and industrial and organisational
psychology. In B.W. Roberts & R.T. Hogan
(Eds.), Personality psychology in the
workplace. Washington:APA.
McCrae, R.R. & Costa, P.T. (1997). Personality trait
structure as a human universal. American
Psychologist, 52, 509–516.
Ones, D.S. & Anderson, N. (2002). Gender and
ethnic group differences on personality scales
in selection: Some British data. Journal of
Occupational and Organisational
Psychology, 75(3), 255–276.
Ones, D.S., & Viswesvaran, C. (1998). Gender age
and race differences on overt integrity tests:
Analyses across four large-scale applicant data
sets. Journal of Applied Psychology, 83,
35–42.
Saucier, G., Hampson, S.E., & Goldberg, L.R.
(2000). Cross-language studies of lexical
personality factors. In S.E. Hampson (Ed.),
Advances in Personality Psychology:Vol. 1.
Hove, East Sussex: Psychology Press.
Witt, L.A., Burke, L.A., Barrick, M.R. & Mount, M.K.
(2002).The interactive effects of
conscientiousness and agreeableness on job
performance. Journal of Applied Psychology,
87, 164–169.
Stephen A. Woods MSc, University of Surrey.
E-mail: [email protected]
Selection & Development Review, Vol. 19 No. 4, August 2003
SDR
The Big Five revisited:
Where are we now?
A brief review of the relevance of the
Big Five for Occupational Assessment
Introduction
ANYONE WHO HAS attended a Level B course will be
aware of the abundance of personality theories
upon which an even greater profusion of personality assessment systems are based. Most occupational practitioners will use instruments based on
a Trait or a Type approach to personality. The
reasons for this are:
● The function of personality tools in an occupational context is to enrich our understanding of
what the personality is like rather than why it
is as it is.Trait and type approaches are the most
face valid and straightforward ways to achieve
this aim.
● In an occupational context we are interested in
understanding how various personality traits
and types relate to the effective performance of
various job requirements in a range of job environments. Instruments based on trait and type
approaches lend themselves to the collection of
objective predictive validity data.
While it may be easy to make the case for the
use of trait and type-based instruments as the
instruments of choice for HR professionals, there
is an enormous number of such instruments from
which to choose. One key issue that applies to
them all is their relationship to the Five Factor
model and its ‘Big Five’ personality factors. The
British Psychological Society stresses the importance of an understanding of this model for all
Level B test users. Today, reputable developers of
personality tests for the occupational market will,
as a matter of course, publish data on the relationship of their instruments to the Five Factor model,
usually through correlational studies with one
Wendy Lord & John Rust
form or other of the NEO Personality Inventory
(Costa & McCrae, 1992), by far the most widely
used measure of the Big Five. Basing this review
on the NEO instruments is thus timely, particularly
given the imminent UK standardisation of the
NEO test family, about to commence under the
auspices of the Psychometrics Research Centre at
City University, London.
One frequently asked question is ‘If the Five
Factor model and the NEO have such a pivotal role
in personality assessment, why bother with
anything else?’The customary answer is that other
personality instruments have been constructed
specifically with occupational settings in mind
and, therefore, have greater utility for the occupational test user. But is this actually true? The total
amount of recent available data from high level
academic journals concerning the work of Costa
and McCrae and the NEO is staggering. More
recent occupationally specific personality questionnaires such as Orpheus (1996) and BIP (1998)
have made the Big Five their starting point and
directly related it to the work context. Developers
of earlier work-based instruments have derived
their frameworks from other models of personality and subsequently investigated the correspondence to the Five Factor model.
The purpose of this paper is to revisit the relevance of the Big Five for the understanding and
prediction of behaviour in an occupational
context.The paper is necessarily brief but aims to
open up discussion about a framework that has
Selection & Development Review, Vol. 19 No. 4, August 2003
15
stood the test of time and that, perhaps, has even
more mileage than has been previously thought.
A note about terminology
The idea that, in factor analytic terms, measurable
personality traits might number five goes back to
Thurstone and Cattell in the 40s. It received
further impetus 50 years ago when first Fiske, and
then Tupes and Christal factor-analysed the results
from self ratings, peer ratings, supervisors ratings,
teacher ratings and clinical assessments together,
and found that a five factor solution generated
strong and recurrent factors across all these types
of assessment. From the 60s onwards, our understanding was advanced further when Norman and
then Goldberg and others showed that this result
could be replicated in a large number of different
types of respondent (e.g. students, workers, and
the military) and in a variety of settings (e.g.
education, recruitment, selection and appraisal).
However, it was not until the prolific work of
Costa and MaCrae in the 80s that the five factor
model began to attain its current status. It was
they that coined the term ‘Big Five’ and gave it the
prominence it receives today. Following the
seminal studies carried out by Barrick and
colleagues in the early 90s, the Big Five became
firmly established as the standard against which
other personality models should be compared.
For the purposes of this paper, the names used
for the Big Five are those with which most readers
will be familiar: ‘Openness’, ‘Conscientiousness’,
‘Extraversion’,‘Agreeableness’ and ‘Neuroticism’. It
should be remembered, however, that other
names are also in frequent use, particularly within
the work-based context.The trait of ‘Neuroticism’,
for example, is normally referred to as
‘Emotionality’, or sometimes ‘Negative Emotionality’ within the occupational market (Howard &
Howard, 1996), this being a more appropriate way
to describe the scale within a work setting.
‘Agreeableness’, when reversed, is frequently
called ‘Toughmindedness’, a more informative
label that does not carry the connotation that all
tough-minded
people
are
disagreeable!
‘Openness’, actually shortened from ‘Openness to
Experience’, is again often more easily understood
in terms of its opposite,‘Conformity’.
16
As well as researching myriad aspects of the Big
Five, Costa and McCrae constructed the first
psychometric instrument specifically designed to
measure them. In doing so they used a procedure
that identified six facets for each of the five principle factors. The Big Five trait of
Conscientiousness, for example, contains the
facets (subscales) of Competence, Order,
Dutifulness, Achievement Striving, Self-Discipline
and Deliberation. This faceted structure became
known as the Five Factor Model and is a particular
characteristic of the NEO. Many developers of
personality tests for the occupational market
investigate the extent to which their instruments
map onto the facets of the Five Factor Model since
they expect their scales to be measuring some
facets of the Big Five rather than the Big Five
factors in their entirety.
Today, Costa and McCrae’s NEO is a family of
tests that includes the NEO FFI, the NEO P-IR and
the NEO-4.The NEO FFI provides the most direct
measure of the Big Five themselves.The NEO P-IR,
is a comprehensive assessment of the Five Factor
model including all the facet scores, and the
NEO-4 is a short version of the NEO that excludes
Neuroticism.
There is now very extensive evidence available
in the literature concerning the reliability and
validity of the NEO and the Five Factor Model in a
wide variety of settings. In the work context
alone, the Five Factor Model has been used to
inform decisions concerning working style, leadership style, training needs, conflict resolution,
decision-making style, and stress management.We
now briefly summarise some of these areas.
Personality at work: Interaction
with the work environment
Work design research identifies three styles that
reflect the way in which an individual interacts
with his or her work environment.These are:
● The style in which lateral relationships with
peers and co-workers are managed.
● The style in which vertical relationships with
superiors and subordinates are managed.
● The style in which actual tasks are managed.
The first two of these are referred to as
‘Interpersonal Style’ and the third as ‘Task-Focused
Selection & Development Review, Vol. 19 No. 4, August 2003
Style’. Interpersonal style is said to be influenced
by two motivational orientations:
● ‘Communion Striving’, where the goal is
obtaining acceptance in personal relationships
and maintaining harmonious relations with
others.
● ‘Status Striving’, where the goal is obtaining
power and dominance within a status
hierarchy.
The Big Five traits of Extraversion and
Agreeableness are central to understanding interpersonal style, while for task-focused style the
assessment of Conscientiousness is crucial.
Neuroticism, although its use in work settings has
been more controversial, is also relevant to taskfocused style. Some human resource professionals
prefer to leave out the assessment of this trait, a
facility that NEO-4 provides. Others, recognising
its importance in situations where work may be
stressful, prefer to retain the trait under a
different name.
Personality at work: Individual
development
The Big Five trait of ‘Openness’ informs our understanding of how receptive the individual will be to
learning and change, but research using the NEO
has also shown that various combinations of the
Big Five factors enable us to predict and to
discriminate between many other important
characteristics that relate to individual development. For example:
● Leadership style; Seven styles of leadership
have been derived from combinations of the
Big Five factors.
● Identification of training needs; Big Five
profiles have been able to discriminate
between those individuals who might benefit
from advanced training in the development of
managerial competencies and those who might
not. The underlying implication that not all
competencies are trainable for all individuals is
a useful one, and being able to use a psychometric assessment system to distinguish what is
and what is not trainable can be very valuable.
● Style of dealing with conflict; Four different
styles of dealing with conflict have been identified with various combinations of the Big Five.
●
●
●
●
Decision-making style; Eight different styles
have been discerned under the Big Five framework, and these have been divided into four
styles of ‘approaching problems’ and four styles
of ‘implementing decisions’.
Learning style; Four learning styles have been
related to the Big Five profile
Management of pressure; The way in which
an individual’s interpretation of and attitude
towards events and people help or hinder their
management of stress has also been related to
the Big Five.
Occupational theme; The Big Five have been
mapped onto Holland’s occupational themes,
and thus have a role in career counselling and
guidance.
Personality at work:
Team development
One problem faced by the HR professional in
applying multi-trait measures to team development
is the difficulty of summarising the information in a
way that can help a team of individuals to understand and value their differences. The MBTI has
generally been the instrument of choice for personality-based team development interventions.
However, a disadvantage of the MBTI is that it
merely describes preferences rather than predicts
actual behaviour. Of all the multi-trait measures on
the market, the NEO (particularly the NEO-4 which
excludes measurement of Neuroticism) provides
the most effective way of succinctly summarising
likely team behaviour in a manner that is easily
understood by participants in a team development
situation. It is as user friendly as the MBTI. In fact
McCrae and Costa (1989) suggest that the NEO is
superior to the MBTI in this respect.The NEO also
overcomes the measurement limitations of a typebased instrument, such as the high risk of misclassification for respondents at or near the mid-point
of a scale. A particularly useful output from the
NEO-4 is a series of ‘style graphs’ that can be used
with individuals or teams to distinguish and easily
communicate individual differences in style.
Conclusions
Today, a vast amount of data has been accumulated
on how scores on the Big Five relate to behaviour
Selection & Development Review, Vol. 19 No. 4, August 2003
17
at work. Within the past ten years it has become
routine that the onus should be on the creators of
other instruments to show how their assessment
relates to the Big Five. For the test user as well, the
ability to understand how their instrument of
choice relates to the Big Five is becoming increasingly essential. Indeed, the five factor model has
become the linchpin that holds personality assessment together, at least within the work context.
Without it, how would we generalise with confidence from the validity of one work based instrument to that of another? Furthermore, the model
links the study of assessment instruments within
the HR field to research in personality and related
areas carried out within clinical and mainstream
psychology, fields in which The Big Five have also
become dominant. Among all the Big Five instruments that are currently on the market, Costa and
McCrae’s NEO family of tests are now so firmly
established as to have become the gold standard
against which other instruments are judged. The
forthcoming UK standardisation of these tests is
thus a particularly welcome development.
References
Barrick, M.R. & Mount, K.M. (1991).The Big Five
Personality Dimensions and Job Performance:
A Meta-analysis. Personnel Psychology, 44, 1–25
BIP (1998). Published by the Hogrefe Group;
currently being adapted for UK usage.To be
distributed in the UK by The Test Agency Ltd.
from September 2004.
Cattell, R.B. (1946). The description and
measurement of personality. New York:
World.
Fiske, D.W. (1949). Consistency of the factorial
structure of personality ratings from different
sources. Journal of Abnormal Social
Psychology, 44, 329–344.
Goldberg, L.R. (1983). The magical number five,
plus or minus two: Some considerations on
the dimensionality of personality descriptors.
Research Paper, Gerontology Research Center,
Baltimore.
Howard, J.P. & Howard, J.M. (1996). A roadmap
for individual and team interpretation of
scores on the FFM of personality. Centre for
Applied Cognitive Studies, Charlotte N.C.
18
McCrae, R.R. & Costa, P.T. Jr. (1989).The structure
of interpersonal traits:Wiggin’s circumplex and
the five-factor model. Journal of Personality
and Social Psychology, 56, 586–595.
McCrae, R.R. & Costa, P.T. Jr. (1989). Reinterpreting
the Myers-Briggs Type indicator from the
perspective of the five-factor model of
personality. Journal of Personality, 57, 17–40.
McCrae, R.R. & Costa, P.T. Jr. (1992). Discriminant
validity of NEO PI-R facet scales. Educational
and Psychological Measurement, 52, 229–237.
Norman,W.T. (1963).Toward an adequate
taxonomy of personality attributes. Replicated
factor structure in peer nomination personality
ratings. Journal of Abnormal and Social
Psychology, 66, 574–583.
Norman,W.T. (1967). 2800 Personality Trait
descriptors: Normative operating
characteristics for a university population.
Ann Arbor: University of Michigan, Department
of Psychology.
Orpheus (1996). Published by The Psychological
Corporation. Distributed by The Psychological
Corporation and The Test Agency Ltd.
Thurstone, L.L. (1947). Multiple Factor Analysis:
A development and expansion of vectors of
the mind. Chicago: University of Chicago Press.
Tupes, E.C. & Christal, R.E. (1961). Recurrent
personality factors based on trait ratings.
(ASD-TR-61-97) Lackland Air Force Base,TX:
Aeronautical Systems Division, Personnel
Laboratory
For a full list of references to data underpinning
the NEO family of tests, contact [email protected] or call Wendy Lord on 01491 413413.
Wendy Lord is Chief Psychologist at the Test
Agency Ltd, which has recently been acquired by
the Hogrefe Group of Companies.
John Rust is Professor of Psychometrics and
Director of the Psychometric Research Centre at
City University, London. John is also Course
Director for the City University MSc in
Psychometrics, which is specifically designed for
HR professionals. For details of the MSc course
contact John Rust at [email protected].
Selection & Development Review, Vol. 19 No. 4, August 2003
SDR
Performance Coaching –
Spin or Win?
An evaluation of the effectiveness of a
Coaching Training Programme in a retail
organisation
Introduction
Much has been written in the popular literature
about the benefits of coaching. Books, articles and
training courses all extol its virtues. There has
been, however, little substantive research to
support the assertions that have been made. The
majority of studies that have been conducted have
concentrated on the field of sporting performance. As business psychologists, we were
afforded the opportunity to investigate coaching
in a business context.
We were able to evaluate both a training
programme that managers undertook, and the
impact of the managers’ coaching on their direct
reports.
Coaching has been defined as ‘the provision of
support and guidance for people to use their
existing knowledge and skills more effectively’
(Bentley, 1995).
Additionally, ‘coaching aims at organisational
excellence through the effective use of everyone’s
abilities and potential, in a way that allows growth
in knowledge and experience’ (Thomas, 1995).
At a more practical level, coaching in the
current study is defined as ‘an ongoing process for
working with people to improve their performance and develop their potential’.
Our investigations evaluated the effectiveness
of a coaching skills training programme for
managers in terms of whether it was successful as
a training event and whether the training led to an
improvement in managers’ performance as
coaches in their day to day work.The question of
Francesca Buck
whether coaching ultimately affects business
performance was also explored.
The Training Programme
The purpose of the training programme was to
provide managers with the skills needed to
conduct brief, ‘two way’ coaching discussions,
focused on improving current performance, with
these discussions taking place on an ‘as and when’
basis as an integral part of a manager’s job. The
training lasted two days.The first day, conducted in
groups of up to 30 participants, covered the
following learning points:
● Giving feedback: which included the underlying psychology shown to be important for
providing effective feedback.There is considerable evidence that performance feedback, if
given appropriately, can lead to substantial
improvements in future performance (Guzzo,
Jette & Katzell, 1985).
● Planning improvement: to ensure participants
work with their direct reports to form a development plan. Through adopting a problem-solving
approach, focused on performance, the manager
and individual work together to find solutions to
current work problems.It has been suggested that
such an approach may help to reduce defensiveness (Greenberg,1986) and is one of the means by
which the manager demonstrates supportiveness
(Whetten, Cameron & Woods, 1994).
Selection & Development Review, Vol. 19 No. 4, August 2003
19
Setting clear expectations for future performance: to make sure that direct reports know
exactly what they are expected to achieve and
what is unacceptable. Goal setting has strong
theoretical underpinnings and there is considerable evidence that it has a positive impact on
employees’ subsequent performance (Locke &
Latham, 1990). Additionally, if people are clear
about what the boundaries of performance are,
they can shape their behaviour to ensure that
they do not under perform.
The second day of the training programme,
conducted in groups of three, was practical in
nature. Each participant took part in two ‘live’
coaching discussions with direct reports in the workplace,observed by a trainer and two colleagues,from
whom they received feedback on their coaching
skills.The second day took place between one week
and one month after the first day.
●
Research objective
The main research question addressed by this
study was: ‘How successful was the training
programme?’The key measures of success were:
● Manager (i.e. trainee) reactions to the training
programme.
● Post-course attainment of coaching skills.
● Staff perceptions of management behaviour
post course.
Method
Samples
The overall sample consisted of managers working
within a large British retail organisation. We used
two different samples for the study.
In order to examine Manager reactions and
Post-course skills attainment, we used a sample of
237 managers who had been trained as part of the
first phase of the coaching training programme as
it was rolled out nationwide.
In order to examine the effect of the coaching
training programme on Staff perception of
management behaviour we used a sample from 12
stores where the coaching training had been
rolled out to 132 first line managers (the experimental group) and a matched sample from 12
stores where 134 first line managers had received
no training in coaching (the control group).
20
Evaluation measures
Most training conducted in organisations is only
evaluated at the ‘reaction’ level i.e. the extent to
which the participants enjoyed the course, usually
measured by a ‘happy sheet’ filled in at the end of
the programme. In this study we had the opportunity to evaluate the effect of the coaching training
programme using a range of qualitative and quantitative sources. The first two measures evaluated
the coaching training programme directly, the
third measure evaluated the impact of the training
programme indirectly, by examining the perceptions of staff in relation to their managers’
coaching skills before and after the training
programme.
Manager reaction measure
All participants on the programme were asked to
complete an evaluation sheet immediately
following the programme. Respondents were
asked to indicate, by marking a percentage on a
six-point Likert-like scale, their perceptions of the:
● Applicability of the content of the programme
to their role;
● Satisfaction with the quality of the material;
● Extent to which the programme met personal
objectives;
● Extent to which the programme introduced
new skills.
Participants were also asked for qualitative feedback on the programme.
Skill attainment measure
At the end of the coaching programme, the participants were assessed, by the facilitator using a
structured rating form, on their coaching skills.
Each participant was given one of three ratings:
‘showing exceptional coaching skills’, ‘at the
standard expected’, or ‘further training needed’.
Staff perception measure
The organisation runs an employee survey twice a
year, measuring 30 factors which are important for
people to be effective in their work. We selected,
as the measure of Staff perception of management
behaviour, eight of these factors. Employees are
asked to respond to a question on each of the key
factors, giving ‘marks out of 10’ on a scale from
Selection & Development Review, Vol. 19 No. 4, August 2003
‘Definitely No’ to ‘Definitely Yes’.The eight factors
selected were the ones most likely to be affected
by the skills being developed in the coaching
training programme:
● Clarity about job responsibilities;
● Clarity about expected results;
● Feedback on performance;
● Praise for performance;
● Credit for achievement;
● Feedback on development needs;
● Encouragement to take on new challenges;
● Encouragement to develop.
To assess whether the coaching training
programme had an impact on the managers’
behaviour, the responses of the managers’ direct
reports on these factors were analysed from the
survey before the training and the survey four
months after the training.
Results
Manager reaction results
● Applicability of the course content was rated at
100 per cent.
● Satisfaction with the quality of material was
rated at 95 per cent.
● The extent to which the programme met
personal objectives was rated at 96 per cent.
● The extent to which the programme introduced new skills was rated at 78 per cent.
The two main themes reported in the analysis
of the qualitative comments were that participants found the programme ‘intense, but worthwhile’ and that a certain degree of anxiety was
expressed concerning the second part of the
programme: the ‘live’ coaching.
Skill attainment results
Ninety-two per cent of the 237 participants on the
programme reached or exceeded the standard of
coaching expected. The remaining participants
were provided with further training at a later date
to bring them up to the standard required.
Staff perception of management behaviour
results
The detailed results are contained in the table
below. A Mann Whitney U test was carried out on
the employee survey data, comparing changes in
direct reports’ perceptions between the first and
second surveys for the experimental and control
groups. A significant difference was identified
between the groups; with the staff perception in
the experimental group reporting significantly
higher levels of change, on six of the eight
selected factors.
S1 = Survey 1; S2 = Survey 2; D = Change
Experimental
group scores –
mean
S1
S2
D
Control
group scores –
mean
S1
S2
D
Feedback on performance
Praise for performance
Credit for achievement
Feedback on development needs
Encouragement to take on new challenges
Encouragement to develop
Clarity about job responsibilities
Clarity about expected results
39
45
41
44
48
47
68
66
40
44
41
46
46
46
67
64
47
52
48
53
54
54
75
73
+8
+7
+7
+9
+6
+7
+7
+7
Selection & Development Review, Vol. 19 No. 4, August 2003
44
48
45
51
51
50
73
70
+4
+4
+4
+5
+5
+4
+6
+6
Mann
Whitney
U
Sig
.014
.024
.039
.012
.033
.010
.143
.160
.05
.05
.05
.05
.05
.05
NS
NS
21
Discussion
Overall, the results show clearly that the coaching
training programme has been a success. It is
pleasing that the Manager reaction results are
overwhelmingly positive, but we would argue that
they are not enough in themselves. Research
shows a lack of relationship between positive or
negative affect on subsequent behaviour (Alliger
& Janak, 1989). The fact that the most common
theme reported in the qualitative data was that the
programme was ‘intense but worthwhile’ is,
however, of importance. Such a reaction has been
shown to have a positive impact on subsequent
job performance (Alliger, Tannenbaum, Bennett,
Traver & Shotland, 1997).
The Skill attainment results also appear very
positive. Again we would say, however, that they
are not compelling evidence on their own of the
success of the training programme, particularly as
in this initiative it was not possible to conduct preprogramme skills assessment. This focus on posttraining attainment is not unusual, since many
organisations are only interested in assessing
whether their employees reach a certain level of
competence, rather than the amount they have
learned during training.
The Staff perception of management
behaviour results are, however, extremely interesting.They suggest that the managers have been
successful in transferring the coaching skills
covered in the programme to the workplace, and
that the training programme has been successful
in improving managers’ performance as coaches
on a day to day basis. The findings show that the
training programme has had a significant impact
on people in terms of ‘improving their performance and developing their potential’. On closer
analysis the lack of impact of the coaching
training on people’s clarity about what they are
expected to do and the results they have to
achieve might be explained, at least in part, by the
high level of clarity reported by direct reports in
the experimental group in the first survey on
these areas.
Coaching and business performance
So the coaching training programme has been a
success. But what is the evidence that coaching
22
delivers improved business performance? We have
just started to look for this client at the relationship between employee perceptions and
measures of business success.
Preliminary analysis of the data indicates that
there is a strong predictive relationship between
‘Praise for performance’ and store sales per square
foot.This analysis is currently in progress, and will
be the subject of a paper at a later date, but is an
exciting avenue for further exploration.
Conclusions
Overall the results of our investigations examining
the effects of coaching and coaching training
using a variety of measures have been encouraging. Our findings indicate that:
● Coaching does seem to have a positive impact
in a business context. There is significant
evidence that training in coaching can change
manager behaviour and improve employee
perceptions about how they are managed.
● The indications are that managers who coach
and specifically who give praise to their direct
reports as part of their everyday activities can
improve employee perception and ultimately
leverage commercial performance.
References
Alliger, G.M. & Janak, E.A. (1989). Kirkpatrick’s
levels of training criteria:Thirty years later.
Personnel Psychology, 42, 331–342.
Alliger, G.M.,Tannenvaum, S.I., Bennett,W.,Traver,
H. & Shotland,A. (1997).A meta-analysis of the
relations among training criteria. Personnel
Psychology, 50, 341–358.
Bentley,T. (1995). Performance Coaching.
Training Officer, 31(2), 36–38.
Greenberg, J. (1986). Determinants of perceived
fairness of performance evaluations. Journal
of Applied Psychology, 71(2), 340–342.
Guzzo, R.A., Jette, R.D. & Katzell, R.A. (1985).
The effects of psychologically-based
intervention programmes on worker
productivity:A meta-analysis. Personnel
Psychology, 38, 275–291.
Locke, E.A. & Latham, P. (1990). A theory of goal
setting and task performance. Englewood
Cliffs, NJ: Prentice Hall.
Selection & Development Review, Vol. 19 No. 4, August 2003
Thomas,A.M. (1995). Coaching for staff
development. Leicester: The British
Psychological Society.
Whetten, D., Cameron, & Woods, M. (1994).
Developing management skills for Europe.
London: HarperCollins.
Francesca Buck is a consultant at
Kaisen Consulting Ltd.
E-mail: [email protected]
Website: www.Kaisen.co.uk
DIVISION OF OCCUPATIONAL PSYCHOLOGY
Annual Occupational Psychology Conference 2004
CALL FOR SUBMISSIONS
7–9 January 2004, Stratford Moat House, Stratford-upon-Avon.
Submissions are invited from academics and practitioners addressing any topic within the
broad area of occupational and
organisational psychology.
As per last year, work may be submitted under the following styles:
Full Papers, Focused Papers and Professional Forums, in addition to posters, workshops and
symposia. Further details on submission guidelines can be obtained from the Society’s
Leicester Office or at www.bps.org.uk. Submissions will not be reviewed unless they
comply with the guidelines.
The deadline for all submissions is Monday 4 August 2003.
All enquiries and submissions should be addressed to the
BPS Conference Office.
Tel: (0116) 252 9555 E-mail: [email protected].
Selection & Development Review, Vol. 19 No. 4, August 2003
23
The British Psychological Society
St Andrews House
48 Princess Road East
Leicester LE1 7DR
24