Rapid Evidence Assessment of
Performance and Development
Review (PDR) systems
Summary report
Matrix Evidence and NPIA Research,
Analysis and Information (RAI) Unit
May 2011
Contents
Page
1.
Summary of main findings
3
2.
Background
3
3.
Methods
3
4.
Main findings
4
4.1
The purpose of PDR systems
4
4.2
Outcomes of PDR systems
4
4.3
Features of successful PDR systems
5
4.4
Barriers to success
7
5.
Key implications for policy and practice
8
6.
References
9
6.1
Studies included in the REA
9
6.2
Other references
11
Appendix A
Research methods and limitations
12
Appendix B
Overview of research studies
14
2
1. Summary of main findings
Organisations generally implement ‘performance and development review’ PDR systems
(1) to review performance to determine salaries, promotion, redundancies, or poor
performance and/or (2) to promote professional development. The evidence as to the
extent to which PDR systems consistently achieve either of these two objectives or other
wider organisational outcomes is inconclusive.
The research reviewed in this rapid evidence assessment of research literature, which
draws on the results of a systematic search of electronic databases and topic specialists’
suggestions, primarily focuses on the perceptions of PDR systems from an employee
perspective. There was less research evidence found on how PDRs may contribute to
organisational performance. Features of PDR systems that may increase employees’
satisfaction with PDRs include:
• Implementing PDR systems with the explicit aim of promoting employees’
development
• Effectively communicating with employees about the PDR process and involving
employees in their own individual PDRs
• Providing training to appraisers and employees about the PDR process
• Implementing a PDR system that is perceived as fair, unbiased and accurate
• Ensuring PDR systems are relevant to job roles but not over-complex,
bureaucratic or time-consuming.
2. Background
The term ‘performance and development review’ (PDR) encompasses a range of
processes including the appraisal or rating of the performance of individual employees in
an organisation on a regular basis, either by their immediate superiors in the
organisational hierarchy and/or by their peers or subordinates. Such appraisal may form
part of a broader training and development plan for individual employees; it can also be
used for other purposes, including informing decisions about promotions and salary
incentives.
During 2009 and 2010, the National Policing Improvement Agency (NPIA) has
undertaken a review of the PDR process for police forces in England and Wales. The PDR
process has a total of nine aims and objectives, including improving performance,
efficiency and quality; increasing staff morale; and promoting communication between
employees and managers (Home Office Circular 014/2003). The competency domains
measured have been based on the police service’s Integrated Competency Framework.
On each measure, individual officers are rated as ‘exceptional’, ‘competent’ or ‘not yet
competent’. Formal PDR interviews between officers and their managers are required
once a year, with ratings based on evidence collected throughout the year.
3. Methods
Matrix Evidence was commissioned to conduct this rapid evidence assessment (REA) in
order to inform the NPIA PDR review and to obtain an overview of relevant research
findings. The REA involved a systematic search for and assessment of international
research literature on PDR outcomes and processes from across a range of
organisations, both public and private. The search for evidence was highly restricted by
date and focussed on studies published between 2004 and 2009, due to the time
constraints within which the REA was conducted. Studies were initially quality assessed
3
only by considering criteria measuring the completeness of the reporting. Following peer
review, these searches were supplemented by topic specialists supplying details of
further key references published since 1990. In addition, the standard of evidence of all
the studies cited in this REA was assessed using an instrument developed by the NPIA
RAI Unit. The process found a lack of high quality research evidence that explores the
effectiveness of PDR systems. Further details of the research methods used for the REA
are provided in Appendix 1 and details of each of the studies included are shown in
Appendix 2.
The rapid evidence assessment focussed on the three questions set out below.
1. What is the purpose and intended outcomes of PDR systems?
a. To what extent do these systems achieve their intended outcomes?
b. Are there any unintended outcomes?
2. What works in terms of ensuring PDR systems are implemented effectively and
achieve their intended outcomes?
3. What has been found to reduce the effectiveness of the PDR in achieving the
intended outcomes?
4. Main findings
This summary summarises the main findings of the REA, addressing these three
questions, and the implications for policy.
4.1. The purpose of PDR systems
In general the ultimate aim of PDR, along with other human resource management
(HRM) practices, appears to be to improve organisational functioning and performance
and to increase the skills of the workforce. However, the majority of the studies included
in this REA focussed on more immediate goals. Broadly these goals can be divided into
two types:
(1) to review performance to determine salaries, promotion, redundancies, or poor
performance and/or
(2) to promote professional development.
The most commonly cited aims of PDR systems cited in a survey of public and private
sector organisations were: performance feedback; recognising individual performance;
evaluating the achievement of goals at the organisational level; and identifying individual
strengths and weaknesses and poor performance. The use of PDR to determine salaries,
promotion and redundancies was found to be more widespread in the private sector than
in public sector organisations (Abu-Doleh and Weir 2007).
4.2. Outcomes of PDR systems
Limited data was found as a result of literature searches for this REA that directly
measured the achievement of organisational outcomes such as improving organisational
and financial performance. Some surveys of employers have identified an association
between the use and quality of PDR systems and organisational performance outcomes
such as patient mortality, workforce skills and financial performance. However, it cannot
be concluded from these studies that it is the PDR system that is causing the
improvement, and it is possible the causal relationship could be in reverse, ie higher
performing employers may be more likely to implement PDR systems. Furthermore other
4
factors may be involved in this relationship between PDR systems and organisational
performance, such as the integration of the PDR system with other human resource
management (HRM) practices and the engagement of employees in management
practices (Wright et al 1999, West et al 2006, Huang 2000, Huselid 1995, Coombs
2006).
The evidence found regarding success of PDR in promoting employee development and
performance is also mixed and inconclusive. For example:
• some studies indicate that the introduction of PDR systems has been successful in
meeting employees’ development needs, in particular a 360 degree feedback process
(Mabey 2001), but other studies suggest more limited usefulness of PDR in this
respect (Levy and Williams 2004).
• One study that used a randomised control trial design showed an improvement in
individual performance of teachers resulting from PDR. However, the PDR system
investigated in this study was very different from most others investigated (Wang
2007).
• Another study found that where the perceived aim of the PDR system was to clarify
individuals’ roles there was an association with reduction in role ambiguity
(Yangcourt et al 2007).
• Some evidence provides reasons to question whether PDR systems are sufficiently
accurate and fair in the ratings they produce to make them a basis for determining
salaries and promotion (Farh et al 1991 and Levy and Williams 2004).
Hence, this REA does not support strong and generalisable conclusions regarding the
effectiveness of PDR in improving individual or organisational performance.
Unintended outcomes
A range of possible unintended outcomes of PDR systems have been identified in the
research literature as potential risks. The principal risk is that if PDR systems are
perceived to be time-consuming, irrelevant and ineffective, they may contribute to
employee dissatisfaction with PDR and disengagement from the process (Narcisse &
Harcourt 2008. In addition, one study of a policing organisation suggests that linking
PDR to pay may have led to an increase in employee complaints regarding the process
(Catano et al 2007). However, it cannot be concluded from this research to what extent
PDR systems linked to salary are actually associated with complaints or any other
unintended outcome.
4.3. Features of successful PDR systems
Most of the evidence relating to PDR success factors included in the review concerns
employee perceptions of and satisfaction with the PDR system. As noted above few
studies measured success in terms of outcomes such as the effectiveness of PDR in
improving organisational and job performance, or in delivering fair and transparent
methods of determining pay and promotion. Therefore the evidence summarised below
mostly identifies factors that increase employees’ satisfaction with PDR.
Integration with other HRM practices
Studies that have identified a positive relationship between PDR systems and
organisational performance suggest that higher performing organisations integrate PDR
with a complementary set of HRM practices such as investment in training and
development, recruitment and programmes involving employees in management
practices (Wright et al 1999, West et al 2006, Huang 2000, Huselid 1995, Coombs
2006).
5
Purpose of PDR is to promote development and provision of feedback
The purpose of PDR may have an impact on employee engagement with the PDR
process. There is some evidence, including one study of police forces that suggests that
PDR systems with the explicit aim of promoting employees’ development are associated
with satisfaction with the system and commitment to the organisation (Lilley and Hinduja
2007 and Yangcourt et al 2007). Evidence from policing organisations also suggests that
PDR assessments that incorporate constructive feedback on the individual’s performance
and link to an individual’s development plan are perceived to be effective (Lilley and
Hinduja 2007).
Effective communication and collaboration
Employees tend to be more positive about PDR systems where the structure and purpose
of the process has been well communicated and employees also have active input into
their own individual PDR assessments (Cawley et al 1998, Narcisse and Harcourt 2008,
Kavanagh et al 2007, Mamatoglu 2008 and Kelly et al 2008).
The nature of the organisation: culture and sector
The effectiveness of PDR may depend on factors relating to the organisational culture,
industrial sector and occupational groups. However, limited evidence was found in
relation to this in this REA. Some of the employers’ surveys included priority given to
PDR, team working and employee involvement as other factors that may contribute to
the positive relationship between PDR and organisational outcomes (Wright et al 1999,
West et al 2006, and Huang 2000). Evidence on perceptions of PDR systems as overly
bureaucratic amongst professional groups is described in the section on barriers to
success below.
Training and guidance
Some evidence including research in policing organisations suggests that training for
both appraisers and employees on how to complete the PDR process may result in higher
levels of satisfaction with and acceptance of the process (Catano et al 2007, Lilley and
Hinduja 2007 and Harcourt 2008).
Fair, unbiased and accurate
A number of studies have identified that perceived fairness of the PDR system is
generally linked to satisfaction with PDR (Steensma and Visser 2007, Jawaha 2007 and
Kelly et al 2008). One of these studies also found fairness correlated with organisational
commitment and motivation (Steensma and Visser 2007).
Factors that may contribute to perceived fairness include trust in and perceived
neutrality of the appraiser, the appraisee’s participation in the PDR process and
knowledge of the PDR process (Steensma and Visser 2007 and Kavanagh et al 2007).
Appraisees may also be more likely to perceive the system as fair when assessment
includes work-based observations and an element of 360 degree feedback (Spence and
Wood 2007 and Mabey 2001).
Anchored rating systems
Evidence from police forces suggests that ‘anchored’ rating systems are to some extent
associated with overall satisfaction with the PDR process (Lilley and Hinduja 2007).
‘Anchored’ rating systems are where rating scores are supported by detailed behavioural
examples and guidance setting out examples of how to get a particular score on the
competency domain.
6
Simplicity
Employees prefer PDR systems that are simpler and less time-consuming to complete.
One robust evaluation found positive effects in terms of performance and employee
relationships from a very simple system comprising peer observation and brief feedback
(Wang 2007).
Timeliness
Participants in the PDR process have indicated that assessment needs to be timely to be
perceived as effective, i.e. assessment should be conducted as quickly as possible after
the end of the period under review (Narcisse and Harcourt 2008). An argument made in
one of the studies (McGivern & Ferlie 2007) was that appraisal and development should
be a continuous process rooted in the day-to-day fulfilment of individuals’
responsibilities.
4.4. Barriers to success
Dissatisfaction with PDR systems amongst employees is likely to be a barrier to effective
implementation. Features of PDR systems that have been identified in the research
literature as contributing to negative perceptions about PDR systems are set out below.
Over-complex PDR processes and bureaucracy
Research studies have found that many employees have strongly negative reactions to
PDR systems perceived to be excessively bureaucratic, time-consuming and ‘pointless
rituals’ (Pretorius and Ngwenya 2008, Catano et al 2007, McGivern and Ferlie 2007).
These studies identify a number of potential negative consequences which may arise
from such procedures, including: excessive demands on staff time, resulting in reduced
time to meet job requirements; problems arising from inaccurate assessments of
performance due to an exclusive focus on tangible or measurable outcomes; the loss of
trust and flexibility in employee relationships; and scepticism and disengagement in the
PDR process as a whole. Some studies suggest that highly skilled professional
employees, in particular, may resist the introduction of PDR schemes perceived as
bureaucratic and irrelevant (Pretorius and Ngwenya 2008, Catano et al 2007, McGivern
and Ferlie 2007). Furthermore a study of a policing organisation found that an
individual’s score across different competencies are usually highly correlated, even when
distinct competencies have been carefully defined and distinguished. Elaborate systems
that use multiple competencies therefore may not provide any more information than
simpler formats (Catano et al 2007).
Lack of job relevance
The perception that the competency domains or measures included as part of PDR are
not relevant to employees’ jobs is likely to increase dissatisfaction with the process.
Some study participants have expressed concern that they were evaluated on irrelevant
criteria or aspects of performance not reflected in their actual work (Narcisse & Harcourt
2008). Where criteria are perceived as irrelevant, employees may simply re-describe
existing activities to fit into the competency domains specified by the PDR form
(McGivern & Ferlie 2007), leading to inaccurate and unhelpful results. Comprehensive
competency frameworks and functional job analyses may help to ensure relevance;
however, they may also increase bureaucracy and complexity.
National frameworks
None of the studies included in this REA directly compared systems that were based on a
nationally determined framework within large organisations as compared to where there
was more local latitude in implementation of PDR. However, some evidence suggests
that perceptions of PDR may be more negative when they are based on national
7
frameworks or introduced to comply with external requirements (McGivern and Ferlie
2007 and Bryman et al 1994).
5. Key implications for policy and practice
Many of the review implications are reflected, at least to some extent, in policy guidance
on PDRs for the police service at the time the review was conducted (as reported in
Home Office 2003, Skills for Justice 2006 and HMIC 2008). There may yet be scope to
reduce staff time and paperwork by considering simpler PDR systems. The main
implications of this REA are:
•
•
•
•
•
•
•
PDR systems should be implemented in a way which is perceived by employees as
fair and unbiased.
PDR should feed into ongoing training and development and appears to be more
effective when integrated with a complementary set of HRM practices. PDR should be
designed to inform the future development of individuals and organisations, as well
as assessing past performance.
PDR systems need to be well-constructed to meet individuals’ and organisations’
needs.
Employees should be actively involved in the PDR process and the purpose of PDR
should be well communicated to employees.
The competency domains included in formal PDR frameworks should be relevant to
individuals’ roles (to the extent that this is possible). Complex multi-competency
frameworks within PDRs are likely to deliver little additional information compared to
simpler systems.
PDR should not be excessively bureaucratic and time-consuming.
Adequate training and guidance on PDR should be provided for managers and
employees involved in the process.
While the evidence suggests that it may be possible to achieve the goals of PDR using
relatively simpler systems, a challenge in developing effective PDR is to reconcile the
need for consistency and fairness with a system that is straightforward and relevant to
individuals. A further potential difficulty is that PDR systems may be expected to perform
two kinds of tasks: providing a measure of employees’ performance and to promote
development. There may be tensions between these two requirements; although, this
was not an issue that was explored in depth in any of the studies reviewed for this REA.
These points indicate the need for PDR to be as flexible as possible in responding to
individuals’ and local organisations’ needs but also based on consistent frameworks.
8
6. References
6.1 Studies included in the REA
Abu-Doleh, J., & Weir, D. (2007). Dimensions of performance appraisal systems in
Jordanian private and public organisations. International Journal of Human
Resource Management, 18(1), 75-84.
Bryman, A., Haslam, C., and Webb, A. (1994). ‘Performance appraisal in UK universities:
a case of procedural compliance?’ Assessment and Evaluation in Higher
Education, Vol 19 (3), pp 175-187.
Catano, V. M., Darr, W., & Campbell, C. A. (2007). Performance appraisal of behaviorbased competencies: a reliable and valid procedure. Personnel Psychology, 60(1),
201-230.
Cawley, B, Keeping L, Levy, P (1998) ‘ Participation in the Performance Appraisal Process
and Employee Reactions: A Meta-Analytic Review of Field Investigations, Journal
of Applied Psychology, Vol 83, No. 4
Combs, James, Yongmei Liu, Angela Hall and David Kecthan. 2006. ‘How Much Do HighPerformance Work Practices Matter? A Meta-Analysis of their Effects on
Organisational Performance.’ Personnel Psychology. Autumn: 501–528.
Farh J L, Cannelle Jr A A, Bedeian A G (1991) ‘Peer ratings: The impact of purpose on
rating quality and user acceptance’, Group & Organisation Studies. December Vol
16, No. 4
Huang T C (2000), ‘Are the Human Resource Practices of Effective Firms Distinctly
Different from Those of Poorly Performing Ones? Evidence from Taiwanese
enterprises’, International Journal of Human Resource Management, Vol. 11, No.
2, pp. 436-451
Huselid, M.A. (1995). The impact of Human Resource Management Practices on
turnover, productivity and corporate financial performance. Academy of
Management Journal, 40, 171-188.
Jawahar, I. (2007). The influence of perceptions of fairness on performance appraisal
reactions. Journal of Labor Research, 28(4), 735-754.
Kavanagh, P., Benson, J., & Brown, M. (2007). Understanding performance appraisal
fairness. Asia Pacific Journal of Human Resources, 45(2), 132-150.
Kelly, K. O., Ang, S. Y. A., Chong, W. L., & Hu, W. S. (2008). Teacher appraisal and its
outcomes in Singapore primary schools. Journal of Educational Administration,
46(1), 39-54.
Kuvaas, B. (2007). Different relationships between perceptions of developmental
performance appraisal and work performance. Personnel Review, 36(3), 378 397.
Levy, P. E., & Williams, J. R. (2004). The social context of performance appraisal: A
review and framework for the future. Journal of Management, 30(6), 881-905.
9
Lilley, D., & Hinduja, S. (2007). Police officer performance appraisal and overall
satisfaction. Journal of Criminal Justice, 35(2), 137-150.
Mabey, C. (2001) Closing the circle: participant views of a 360 degree feedback
programme Human Resource Management Journal, 11, (1), 41-53
Mamatoglu, N. (2008). Effects on organisational context (culture and climate) from
implementing a 360-degree feedback system: The case of Arcelik. European
Journal of Work and Organizational Psychology, 17(4), 426-449.
McGivern, G., & Ferlie, E. (2007). Playing tick-box games: Interrelating defences in
professional appraisal. Human Relations, 60(9), 1361-1385.
Narcisse, S., & Harcourt, M. (2008). Employee fairness perceptions of performance
appraisal: A Saint Lucian case study. International Journal of Human Resource
Management, 19(6), 1152-1169.
Pretorius, S. G. (2008). Teachers’ perceptions of and attitudes towards performance
appraisal in Zimbabwean schools. Africa Education Review, 5(1), 144-164.
Sanwong, K. (2008). The development of a 360-degree performance appraisal system: A
university case study. International Journal of Management, 25(1), 16-22.
Spence, D. G., & Wood, E. E. (2007). Registered nurse participation in performance
appraisal interviews. Journal of Professional Nursing, 23(1), 55-59.
Steensma, H., & Visser, E. (2007). Procedural justice and supervisors' personal power
bases: Effects on employees' perceptions of performance appraisal sessions,
commitment, and motivation. Journal of Collective Negotiations, 31(2), 101-118.
Taormina, R. J., & Gao, J. H. (2009). Identifying acceptable performance appraisal
criteria: An international perspective. Asia Pacific Journal of Human Resources,
47(1), 102-125.
Turk, K. (2008). Performance appraisal and the compensation of academic staff in the
University of Tartu. Baltic Journal of Management, 3(1), 40-54.
Wang, W. (2007). Evaluation of 2+2 alternative teacher performance appraisal program
in Shanxi, People's Republic of China. Teaching and Teacher Education: An
International Journal of Research and Studies, 23(7), 1012-1023.
West M A, Guthrie J P, Dawson JF, Borrill C S, Carter M (2006), ‘Reducing patient
mortality in hospitals: the role of human resource management’, Journal of
Organisational Behaviour, Vol.27(7), pp. 983-1002
Wright P, McCormick B, Sherman W, McMahan G (1999), ‘The Role of Human Resource
Practices in Petro Chemical Refinery Performance’, The International Journal of
Human Resource Management, Vol. 10, No. 4, pp. 551 571
Youngcourt, S. S., Leiva, P. I., & Jones, R. G. (2007). Perceived purposes of
performance appraisal: Correlates of individual- and position-focused purposes on
attitudinal outcomes. Human Resource Development Quarterly, 18(3), 315-343.
10
6.2 Other references
HMIC (2008). Leading from the Frontline: HMIC Thematic Report 2008. Downloaded 23
April 2009, from
http://inspectorates.justice.gov.uk/hmic/inspections/thematic/Frontline/.
Home Office. (2003). Circular 014 / 2003: Performance and Development Reviews.
Downloaded 23 April 2009, from http://www.homeoffice.gov.uk/aboutus/publications/home-office-circulars/circulars-2003/014-2003/.
Skills for Justice. (2006). PDR: A Guide to Successful Implementation and Improvement.
Downloaded 23 April 2009, from
http://www.skillsforjustice.com/websitefiles/PDRguide.pdf.
11
APPENDIX A
Research Methods and Limitations
A Rapid Evidence Assessment (REA) method was used to search for, review and
synthesise research literature. The guiding principles of an REA are that methods should
be transparent and explicit, and applied in a standardised and systematic way1. The REA
process comprises four stages:
•
•
•
•
Searching using electronic databases and other sources of literature;
Screening the available literature for relevance, using inclusion criteria which are
clearly defined and set in advance;
Assessing the literature for methodological quality and extracting data using
standardised forms;
Synthesising the data to provide an overview of research findings.
An overview of the methods employed is provided below.
Searching
For this REA four electronic databases were searched, using two types of search strategy
for each: one strategy focussed on reviews and was limited to the years 2004-2009, and
the other was unlimited by study design and was limited to the years 2007-2009. The
restricted date ranges were employed because the very wide scope of the REA,
combined with the very short timescale of the project (less than two months in total),
meant that comprehensive searching would not have been practicable.
In addition to the electronic database searches, references were supplied by experts at
the NPIA, and by Professor Rob Briner, the topic specialist for the REA. However,
concerns were raised by one of the peer reviewers of this REA in relation to the
restricted publication dates of included studies. In response to this concern, NPIA asked
two further topic specialists to provide details of key research on PDRs that they were
aware of that had been published since 1990.
Citations from the included studies were not followed up.
Screening
Studies were included in this REA if they provided substantial data relating to workrelevant organisational factors relating to any type of performance development and
review system. Studies not published in English were excluded. No attempt was made to
retrieve books or dissertations.
Data extraction and quality assessment
In this REA, the following data were extracted:
•
•
•
contextual data relating to the type of organisation studied, the content of the
PDR system, and other relevant background information;
findings relating to processes and outcomes;
methodological data on the aims and design of the study; and
1
Further details about rapid evidence assessment methods are provided in a Rapid
Evidence Assessment toolkit available from http://www.civilservice.gov.uk/my-civilservice/networks/professional/gsr/resources/gsr-rapid-evidence-assessment-toolkit.aspx
12
•
information on the reporting of key methodological aspects of the study, used as
a measure of study quality.
The quality of the studies was assessed firstly using an assessment of the completeness
of the reporting of methods. Following peer review feedback, all studies were assessed
using a Standard of Evidence instrument2 developed by the NPIA Research Analysis and
Information Unit. The Standard of Evidence tool assesses the literature against a set of
standards for both qualitative and quantitative research, and can be used on primary
research (new evidence) and secondary sources (literature that uses data collected by
others). The criteria by which evidence is assessed are whether the piece explains the
research study's purpose and context; describes the sample studied; outlines the
research methods and analytic procedures in sufficient detail to allow replication; and
whether it provides a detailed description of the results and their implications for
research, policy and practice. Studies are assessed on their own terms and the
instrument does not attempt to assess the methods of research used. Studies were only
included in the findings in this summary if they reached the required standard as set out
in this Standard of Evidence tool.
Data synthesis
In the REA a framework analysis method was used to categorise the findings from the
studies and bring them together in a narrative synthesis. This involved grouping the
findings into the themes of: aims, process, implementation and outcomes of PDR
systems.
Limitations of the REA
There are a number of important limitations in relation to the methods used for this REA:
• The REA was completed in a short time. The literature was not searched in a
comprehensive fashion. In order to address the broad scope of the research
questions, a narrow restriction in terms of publication date was used, only a limited
number of databases were searched, and a restricted set of search terms was used.
As a result, it cannot be said that this REA represents an exhaustive synthesis of the
available literature.
• The majority of the included studies used cross-sectional survey methods and from
these studies it is only possible to draw conclusions about associations between PDR
systems and possible outcomes at a single point in time. Only one of the included
studies evaluated outcomes of a PDR system pre and post-implementation with a
control group (Wang 2007). Given the limitations of the research studies included in
this REA, it is not possible to derive ‘what works’ conclusions about PDR systems.
• Many of the studies reviewed in this REA were surveys of employees’ perceptions
about PDR systems. Therefore the REA mostly focuses on factors that increase
employees’ satisfaction with PDR, not factors that increase overall effectiveness.
None of the included studies surveyed managers’ perceptions of PDR systems and
this may be an important evidence gap.
• It is unclear to what extent the findings of included studies are generalisable.
Included studies may be subject to sampling bias, in the sense that organisations
which are more committed to PDR may be more likely to be sampled for research
studies.
2
The Standard of Evidence tool is available from the NPIA Research Analysis and
Information Unit on request
13
APPENDIX B
Overview of Research Studies
Reference
Study
design
Country
Industrial
sector
studied
Abu-Doleh &
Weir (2007)
Crosssectional
employer
survey
Jordan
Various
(public and
private
sector)
Bryman et al
(1994)
Crosssectional
employee
survey and
qualitative
interviews
UK
Education
3
Primary
stated
purpose
of PDR
system
(see
note)
3
3
Main findings
This study focussed specifically on examining the aims of
performance appraisal3 (PA). The most commonly cited aims were:
performance feedback; recognising individual performance;
evaluating the achievement of goals at th organisational level; and
identifying individual strengths and weaknesses and poor
performance. The use of PDR to determine salaries, promotion and
redundancies was widespread in private organisations, but much
less so in the public sector. Public organisations were more likely
to utilise PDR to meet legal requirements than private companies.
Following the introduction of performance appraisal within two
universities in order to comply with external requirements
(procedural compliance) there was a common view amongst staff
that formal appraisal was unnecessary and there was a lack of
organisational support for it. There was scepticism around the
purpose of the appraisal system (ie a perception that the purpose
was to enhance the image of staff and universities).
Many of the studies reviewed for this rapid evidence assessment use the term ‘performance appraisal’ not ‘performance, development and review’.
14
Catano et al.
(2007)
Mixed
methods
Canada
Police
1
Cawley et al
1998
Systematic
literature
review
Internation
al
All
6
Coombs et al
(2006)
Metaanalysis
Internation
al
Private sector
manufacturin
g and service
1
Farh et al
(1991)
Quasiexperiment
al design
USA
Education
(Students)
3
A new performance appraisal system, developed for promotions in
the Royal Canadian Mounted Police was found to fairly differentiate
among candidates and predict promotion. Both candidates and
supervisors supported the system. The system utilised eight
competency domains: leadership; service orientation and delivery;
thinking skills; personal effectiveness and flexibility; organisation
and planning; interpersonal relations; communication; and
motivation. It was found that individuals’ scores on each domain
were significantly and quite highly correlated with scores on all the
other domains.
Employee participation was positively related to employee
satisfaction with the appraisal session, the appraisal system,
perceived utility of the appraisal, motivation of employees to
improve performance, and perceived fairness of the system.
Consistent evidence that value-expressive participation is more
strongly related to a number of employee reactions to the
appraisal than instrumental participation.
High performance work practices (HPWP) which include
performance appraisal enhanced organisational performance
however, other factors may influence this relationship. There was
found to be a stronger effect where there was a complementary
system of high performance work practices and performance
appraisal was found to have no significant impact on performance
outcomes on its own.
The quality of peer rating are highly susceptible to the influence of
rating contexts, peer ratings of performance were more lenient
and less reliable when students were told the purpose of ratings
was evaluative and would affect grades than when they were told
it was for developmental feedback.
15
Huang
(2000)
Crosssectional
employer
survey
Taiwan
Business
6
Huselid
(1995)
Crosssectional
employer
survey
USA
Private sector
firms
6
Jawahar
(2007)
Crosssectional
employee
survey
USA
Retail
5
Kavanagh et
al. (2007)
Crosssectional
employee
survey
Australia
Research
1
The degree of integration between performance appraisal and
other HR practices is positively related to a firm's performance.
There was a positive relationship between using long-term
appraisal criteria and firm effectiveness. Outstanding firms were
more likely to use group rather than individual performance
measures than firms with poorer performance.
High performance work practices (HPWPs) within firms, including
performance appraisals found to be significantly correlated with
productivity, turnover and corporate financial performance. The
study does not individually test different systems of HPWPs, so
does not provide individual results for appraisals
The success of appraisal systems may well depend on ratees’
perceptions of fairness and reactions to important aspects of the
appraisal process. Findings indicated that distributive justice
influenced satisfaction with performance ratings and procedural
justice influenced satisfaction with the appraisal system.
Procedural justice refers to the perceived fairness of procedures
used to determine the appraisal rating. Distributive justice refers
to the perceived fairness of an actual performance rating.
Participation in performance appraisal, attitudes towards the
supervisor, and knowledge of the performance appraisal process
are all positively and significantly associated with employee
perceptions of performance appraisal fairness.
16
Kelly et al.
(2008)
Crosssectional
employee
survey
Singapore
Education
5
Kuvaas
(2007)
Crosssectional
employee
survey
Norway
Finance
5
Lilley &
Hinduja
(2007)
Crosssectional
survey
USA
Police
2
The findings indicate that fairness of the performance appraisal
system and clarity of appraisal criteria are related to greater
satisfaction with the appraisal system, more positive attitudes
towards performance bonus, and higher job satisfaction and
motivation. Using appraisal criteria that are controllable is
associated with greater satisfaction with the appraisal system, less
stress experienced with the appraisal system, and higher job
satisfaction and motivation. Finally, teachers who report greater
trust in their appraiser and more positive assessment of their
appraiser’s credibility also report more cooperativeness amongst
teachers in their school.
The relationship between perceptions of developmental
performance appraisal and self-reported work performance was
mediated by employees’ intrinsic motivation, and strongly
moderated by their autonomy orientation.
For employees with a weak autonomy orientation, the relationship
was positive, but for those with a strong autonomy orientation, the
relationship was negative.
Results indicated that police organisations with a high level of
community policing differed from their more traditional
counterparts with regard to performance appraisal evaluation
procedure in that they provided substantially more training to
raters, emphasised the use of performance appraisal for officer
development, and evaluated a broader range of performance
criteria.
17
Mabey
(2001)
Crosssectional
employee
survey and
qualitative
research
UK
Education
2
Mamatoglu
(2008)
Longitudina
l study
Turkey
Manufacturin
g
5
McGivern &
Ferlie (2007)
Qualitative
research
UK
Healthcare
3
360 degree feedback appears to have a positive impact on those
involved in the process. Participants receiving the 360 degree
feedback register significantly higher scores for the dimensions of
training and development tested than those managers who have
not been involved in the programme. Participation also appears to
increase ability to exercise discretion in developing one's career
and generally participants rated their employer significantly higher.
Participants also had significantly higher ratings for satisfaction
with quality and opportunities of training and making best use of
their skills.
Participation in a 360 degree feedback system found to have
effects on organisational context (e.g. employees’ perceptions
regarding support and achievement culture within the
organisation). The results have also revealed some significant
effects on the participants’ perception of communication and the
performance appraisal
system in the organisation.
We here examine the introduction of appraisal for senior medical
professionals. Our recent qualitative field research found four main
experiences of appraisal (developmental, disappointed reflection,
defensive assessment and cynical dismissal of appraisal as a waste
of time), which we developed into a typology. We argue many
professionals 'play tick-box games' to give the impression of
auditable practice while continuing to practise in a traditional way,
We develop existing theory on the 'audit society', social defences
and 'mock bureaucracy' to explain interrelating defences which
occur in appraisal as a reaction to the risks and conflict
experienced in professional regulation.
18
Narcisse &
Harcourt
(2008)
Qualitative
research
St Lucia
NS (public
sector)
3
Pretorius
(2008)
Crosssectional
employee
survey
Zimbabwe
Education
2
Sanwong
(2008)
Longitudina
l study
Thailand
Education
1
Spence &
Wood (2007)
Qualitative
research
New
Zealand
Healthcare
5
Results show that distributive, procedural, and interactional justice
factors identified in the existing literature influence employee
perceptions of fairness in their appraisals. Distributive justice
refers to the perceived fairness of an actual performance rating.
Procedural justice refers to the perceived fairness of procedures
used to determine the appraisal rating. Interactional justice refers
to the perceived fairness of the rater's interpersonal treatment of
the ratee during the appraisal process.
A survey of teachers revealed that on the whole teachers are
positively motivated by staff supervision models which seek to
develop their pedagogical skills and therefore tend to enhance
their performance with a view to improving education and
attaining educational goals. The research revealed that
performance appraisal should be collaborative, transparent,
dialogical and accountable. Models which are judgmental and call
for close and constant supervision are unpopular.
This study examines the functioning of an innovative 360-degree
performance appraisal system within a university. While all
employees were satisfied with the system, support and clerical
staff in the university were more satisfied than academic staff.
Data from qualitative research with nine New Zealand registered
nurses employed by a single district health board suggest nurses
can be disappointed by the process of performance appraisal.
Although they may believe in the potential value of performance
appraisal interviews, they may not experience the feedback,
direction, and encouragement necessary for an effective appraisal
process.
19
Steensma &
Visser
(2007)
Crosssectional
employee
survey
Netherlands
Civil service
5
Taormina &
Gao (2009)
Crosssectional
employee
survey
China
Various
5
Turk (2008)
Crosssectional
employee
survey
Interventio
n study
Estonia
Education
1
China
Education
2
Wang (2007)
Neutrality, standing, trust, and accuracy of information correlated
positively with perceived procedural justice of performance
appraisal sessions. Quality of outcomes of PA sessions also
correlated with perceived fairness. Moreover, personal power
bases of supervisors (expert power, referent power) contributed to
employees' procedural justice perceptions. Perceived procedural
justice correlated, as predicted, positively with three outcome
variables: satisfaction with performance appraisal session;
organisational commitment; and motivation.
Results confirmed that it is possible to identify performance
appraisal criteria that are generally acceptable to employees from
a variety of industries, and that organisational socialisation (such
as training, understanding, co-worker support, and future
prospects) can influence the acceptance of performance appraisal
criteria.
Results suggest that the performance appraisal and compensation
system (pay-for-performance system) may have led to a highly
motivated core of staff
A cluster randomised control design was used to evaluate the
effectiveness of a PDR system for teachers. The system involved
regular observations of teachers by peers and assessment by
external evaluators. The results showed that the performance
appraisal system evaluated significantly improved teachers'
professional performance, enhanced teachers' collaboration, and
increased the feedback between the peers.
20
West et al
(2006)
Crosssectional
employer
survey
UK
Health
6
Wright et al
(1999)
Crosssectional
employer
survey
USA
Petrochemicals
6
Youngcourt
et al. (2007)
Crosssectional
employee
survey
USA
Retail
5
After controlling for other factors affecting patient mortality,
greater use of a complementary set of human resource
management (HRM) practices has a statistically significant
relationship with patient mortality. Although the study suggests
that the focus should be on a bundle of complementary HRM
practices rather than the individual affect of performance
appraisal, it found that higher quality appraisal systems within
hospitals were associated with reduced patient mortality.
The greater the use of performance appraisal at US petrochemical
refineries was strongly correlated with workforce skills. The use of
performance appraisal systems is also significantly positively
related to refinery performance but only where there was a high
degree of employee participation for example in problem solving
and planning.
Results suggested the purpose of the performance appraisal may
influence ratees' perceptions of the performance appraisal system
and attitudes toward their jobs.
Note: Codes for ‘purpose’ column are as follows:
1 = review performance to determine salaries, promotion, poor performance
2 = promote development
3 = both
4 = other
5 = unclear
6 = multiple employers with range of purposes
21