A descriptive analysis of the use of workplace-based assessments
in UK surgical training
Joseph Shalhoub1, Cristel Santos2, Maria Bussey2, Ian Eardley2, William Allum2
1
Department of Surgery & Cancer, Imperial College London, UK
2
Joint Committee on Surgical Training, UK
For Correspondence
Mr William Allum
ISCP Surgical Director
Joint Committee on Surgical Training
35 - 43 Lincoln’s Inn Fields
London WC2A 3PE, UK
E-mail:
[email protected]
Tel:
+44 (0) 20 8661 3982
Word Count
3,266 Words
1
Abstract
Background
Workplace-based assessments (WBAs) were formally introduced in the UK in 2007.
The study aim was to describe use of WBAs by UK surgical trainees and examine
variations by training region, specialty or level of training.
Methods
The database of the Intercollegiate Surgical Curriculum Programme (ISCP) was
interrogated for WBAs between August 2007 and July 2013, with in-depth analysis of
two periods: August 2011 to July 2012, and August 2012 to July 2013.
Results
Numbers of validated WBAs per trainee per year increased more than 7-fold, from
median 6 per trainee in 2007-2008, to 39 in 2011-2012, and 44 in 2012-2013. In
2011-2012, 58.4% of core trainees completed the recommended 40 WBAs; only
38.1% of specialty trainees achieving 40 validated WBAs. In 2012-2013, these
proportions increased to 67.7% and 57.0% for core and specialty trainees,
respectively. Core trainees completed more WBAs per year than specialty trainees in
the same training region. London core trainees completed the highest numbers of
WBAs in both 2011-2012 (median 67) and 2012-2013 (median 74). There was a
peak in WBAs completed by London specialty trainees in 2012-2013 (median 63).
The most validated WBAs were completed by ST1/CT1, with a gradual fall in median
WBAs to ST4, followed by a plateau; in 2012-2013 there was an increase in WBAs at
ST8. Core surgical trainees complete ~50% ‘operative’ (PBA/DOPS) and ~50% ‘nonoperative’ assessments (CBD/CEX). During specialty training, PBAs represented
~46% of WBAs, DOPS 11.2%, CBD ~23% and CEX ~15%.
Conclusions
UK surgical trainees are, on average, undertaking one WBA per week. Variation
exists in use of WBAs between training regions. Core trainees tend to use the
spectrum of WBAs more frequently than their senior colleagues. Further work is
required to examine the role of WBAs in assessment, and engagement and training
of trainers in processes and validation of WBAs.
2
Key Words
Workplace-based assessment, surgical training, Intercollegiate Surgical Curriculum
Programme
3
Introduction
Workplace-based assessments (WBAs) were formally introduced in the UK in 2007
to accompany the competency-based Modernising Medical Careers postgraduate
training programme 1. WBAs are delivered as part of the Intercollegiate Surgical
Curriculum Programme (ISCP), a joint activity by the four surgical Royal Colleges 2.
The ISCP has provided each surgical specialty with a comprehensive syllabus, a
teaching and learning framework, an assessment system, a repository for evidence
on individual trainee progress and an interactive web platform.
The use of WBAs is a mandatory component of each surgical trainee’s portfolio
through both core surgical training and specialty training. The main WBA assessment
tools include procedure-based assessments (PBA), direct observation of procedural
skills (DOPS), clinical evaluation exercise (CEX) and case-based discussion (CBD),
as well as multi-source feedback (MSF). A glossary of terms used is presented in the
Appendix.
PBA involves direct observation of an index procedure or operation with comments
given on important steps, tasks or skills at the pre-, intra- and post-operative stages
of the procedure, considered to be essential for its safe and successful completion.
The assessment form for a PBA is specific to the procedure or operation being
assessed. Similar to PBA, DOPS are more generic and relate to either less complex
procedures or parts of a larger procedure. The domains of assessment in DOPS are
generic and fixed and considered to be transferable across this spectrum of tasks.
CEX is an assessment of a trainee’s clinical skills, for example history taking, clinical
examination, and information giving. CBD refers to a formalized discussion related to
the care of a patient with a focus on knowledge and attitude. Each of these WBAs is
completed by both trainee and trainer, validated by the trainer and is recorded within
ISCP. MSF is an opportunity for members spread across the multidisciplinary health
care and administrative team to offer anonymous feedback on the trainee. The
results are then discussed at a meeting with the assigned educational supervisor and
the trainee’s annual review of competence progression (ARCP) 3.
At present, the Joint Committee on Surgical Training (JCST) recommends that each
trainee completes a minimum of 40 WBAs per year of training. This approximately
4
equates to one WBA per week, although trainees and trainers are encouraged to
complete more with a spread throughout a training post to demonstrate progression
of clinical and technical skills 5.
The purpose of this study is to report on the use by surgical trainees in the UK of the
largest platform for competence-based training worldwide. We aim to describe the
use of WBAs by surgical trainees and to examine any variations observed by training
region, surgical specialty or level of training.
5
Methods
Every assessment recorded in trainees’ portfolios is recorded with background
information about the individual trainee. These data were collated to facilitate
analysis to quantify the use of the WBAs. The ISCP database was interrogated for
WBAs linked to trainees’ portfolios between August 2007 and July 2013, with an indepth analysis of two periods: August 2011 to July 2012, and August 2012 to July
2013. These two time periods represent the most recent two years for which, at the
time of data analysis, complete data was available. Trainees in Core Surgery and all
but one of the surgical specialties were included from 2007. Vascular surgery only
became a separate specialty in 2012 but has been included for the second time
period. Trauma and Orthopaedics trainees were variably included in the first years of
ISCP as they were also able to use the Orthopaedic Competence Assessment
Project (OCAP) system; all had transferred to ISCP by 2012/13.
Medical graduates in the UK commence postgraduate clinical training with a 2-year
generic ‘Foundation Programme’, from which candidates apply through a competitive
national selection system for core surgical training (CST) programmes – previously
termed the senior house officer (SHO) grade. Following successful completion of
CST and the Intercollegiate Membership of the Royal College of Surgeons
examinations, trainees apply through competitive national selection for higher
surgical training (registrar grade) in one of the 10 surgical specialities 6.
The main WBA assessment tools included in the analysis were DOPS, PBA, CEX
and CBD for all trainees in both core and higher specialty training, although there
was some data available for assessment of audit, teaching and multi-source
feedback. All trainees with an ISCP appointment type of core surgery (CT), fixedterm specialty training appointments (FTSTA), locum appointments for training (LAT),
SHO, specialist registrars (SpR), specialty registrars (StR), and StRs converted from
SpR were included in the analysis. CT1 and CT2, as well as (for ‘run-through’
specialties/programmes) ST1 and ST2 were combined for analysis for consistency of
reporting on the early years of training.
Only WBAs which had been trainer validated were included. Entries with missing
variables were excluded. Data were anonymised and allocated a unique identifier to
avoid duplicate counting, and analysed by training region (Local Education and
Training Board [LETB] in England and Deanery in Wales, Scotland and Northern
6
Ireland), surgical specialty, and level of training at the time of the assessment.
Records without appointment type, training region, specialty or training level were
excluded from further analysis. After data cleaning, data comparison and summary
statistics were performed using STATA version 11 (StataCorp LP, Texas, USA). For
the analysis, medians and percentiles were used to represent the data as these are
not affected by extreme values.
7
Results
Records without appointment type, training region, specialty or training level totalled
1,433 for 2011-2012 and 1,355 for 2012-2013; these records were excluded from
further analysis. A total of 754,165 ISCP WBAs were validated by UK surgical
trainees between August 2007 and July 2013 (Table 1). Approximately two thirds of
trainees included are male (Table 1). There has been an increase in both the number
of WBAs validated using the ISCP and the number of surgical trainees using the
ISCP for WBAs year on year between 2007 and 2013. The number of validated
WBAs per trainee has increased more than 7-fold, from a median of 6 per trainee in
2007-2008, to 39 per trainee in 2011-2012, and 44 per trainee in 2012-2013.
In 2011-2012, 55.5% of core surgical trainees had completed the JCSTrecommended 40 WBAs, with only 35% of specialty trainees in surgery achieving 40
validated WBAs. In 2012-2013, these proportions increased to 65.5% and 53.6% for
core and specialty trainees, respectively Figure 1 illustrates the number of WBAs
being completed by core and specialty trainees for these two time periods.
WBAs across training regions
The use of WBAs across training regions for 2011-2012 and 2012-2013 can be seen
in Table 2. For the year 2011-2012, the median number of WBAs undertaken by
trainees varied from 30 to 50 (Figure 2a). For 2012-2013 this variation was between
39 and 67, with the peak number undertaken by London trainees. After separating
core and specialty trainees (Figures 2b and 2c), core trainees tended to complete
more WBAs per year than specialty trainees in the same training region. London core
surgical trainees completed the highest numbers of WBAs in both 2011-2012
(median 67) and 2012-2013 (median 74). There was also a peak in WBAs completed
by London specialty trainees in 2012-2013 (median 63).
WBAs across core surgical training and surgical specialties
The use of WBAs across core surgical training and individual surgical specialties for
2011-2012 and 2012-2013 can be seen in Table 3. The median number of WBAs
completed by core surgical trainees increased from 43 in 2011-2012 to 48 in 20122013. Excluding vascular surgery, which became a recognised surgical specialty in
2012, the median number of WBAs completed by specialty trainees in 2011-2012
ranged from 21 in trauma and orthopaedic surgery, to 41 in neurosurgery, oral and
maxillofacial surgery, and otolaryngology. By 2012-2013, this range increased to
8
between 39 in paediatric surgery, and 45 in otolaryngology and vascular surgery. For
trauma and orthopaedic surgery, the median number of validated WBAs almost
doubled from 21 in 2011-2013 to 41 in 2012-2013 reflecting the change from OCAP
to ISCP.
WBAs across surgical training levels
The use of WBAs across surgical training levels for 2011-2012 and 2012-2013 can
be seen in Table 4. The most validated WBAs were completed by ST1/CT1 trainees,
with a gradual fall in the median number of WBAs to ST4. This is followed by a
plateau, however in 2012-2013 there was an increase in WBA numbers at the ST8
training level.
Types of WBAs being used
Analysis of the type of WBA shows core surgical trainees complete approximately
50% ‘operative’ or interventional (PBA and DOPS) and 50% ‘non-operative’
assessments (CBD and CEX), with the remainder (namely assessment of audit,
observation of teaching and multi-source feedback) contributing a small proportion
(Figure 3a). During core training, PBAs contributed approximately 14% of validated
WBAs, whilst DOPS represented about 31%; CBDs and CEXs were equally split at
about a quarter of validated WBAs each. During specialty training, PBAs represented
approximately 46% of WBAs, DOPS 11.2%, CBD about 23% and CEX approximately
15%; the remaining WBA types again contributing a small proportion (Figure 3b). The
proportion of WBA types remained relatively stable from 2011-2012 to 2012-2013.
Variability in the type of WBAs performed across the specialties was observed.
9
Discussion
The ISCP has offered a curriculum for surgical training and a platform for monitoring
of the competency-based training that has accompanied the Modernising Medical
Careers era of postgraduate medical training 1. At present, competence in this
system is being assessed primarily through the use of WBAs. Few would disagree
that the introduction of WBAs to support competency-based training is one of the
most significant changes to educational policy in medical and surgical training in
recent years.
The ISCP is designed to allow the trainee to demonstrate progression in knowledge
and clinical and technical skills, as well as achieving professional capabilities using
WBAs as formative tools. Engagement with this approach to training has required
faculty and trainee reinforcement of its principles and, while acknowledging a specific
number should not be an absolute, the figure of 40 has been selected as it
approximates to a minimum of one WBA per week. For each training year, there are
expected levels of competence defined within the curriculum which are reviewed on
an annual basis by a team comprising a programme director, independent
educational supervisors, the postgraduate Dean and external specialty advisors. At
this review the formative assessments undertaken by clinical supervisors, together
with an overview report from an assigned educational supervisor, are considered in a
summative way to confirm the appropriateness of a trainee proceeding onto the next
year of training.
Whilst WBAs have been a part of surgical training since 2007, from the 2012-2013
academic year the American Board of Surgery has required the completion of inprogram assessments 4. These are broadly divided into operative performance
assessments and clinical performance assessments. At the time of implementation in
2012-2013, two of each of these assessments was required, increasing to six of each
from the 2015-2016 academic year 4. The Operative Performance Rating System
(OPRS) is employed for operative performance assessment. OPRS WBAs are
broadly similar to PBAs in that they are operation-specific, examine and assess
defined steps of a procedure, with each step scored against a 5-point Likert (as
compared with ‘satisfactory’ and ‘needs development’ used for PBAs). For clinical
performance assessment, similar to CEX in the UK, the American Board of Surgery
10
uses mini-CEX and Clinical Assessment and Management Exam – Outpatient
(CAMEO) for the direct observation of clinical assessment of patients.
The purpose of this study was to examine the use of WBAs across a number of
denominators in UK surgical training. This is the first time that data describing the
use of WBAs and their validation has been released by the ISCP. The data reflects
that there has been a good uptake with regards the use of WBAs since the
commencement of their use in 2007, with a steady rise in WBAs per trainee per year
from 2007 to 2013.
However, the proportions of trainees who are meeting the JCST’s set minimum for
validated WBAs of 40 per annum – although rising – remains below 70% and 60%
for core and specialty trainees, respectively, for 2012-2013. It may be that these
trainees are completing 40 WBAs, but these are not being validated. At present there
is a perceived emphasis on the numbers of WBAs completed, as compared with their
content; this may be driven, at least in part, by the minimum numbers requirement.
While this may show engagement with training it is essential that quantity does not
trump quality 7.
One of the implications of this study is to highlight that additional support is
necessary for trainees to facilitate their use of WBAs, but importantly there is
undoubtedly a need to ensure that trainers are engaged, trained and supported by
Deaneries / LETBs in the processes related to WBA completion and validation.
Mechanisms for this include through accredited courses, for example Training the
Trainers (TtT), Training and Assessment in Practice (TAiP) and Training and
Assessment in the Clinical Enviroment (TrACE), and through the Faculty of Surgical
Trainers (affiliated with the Royal College of Surgeons of Edinburgh).
Core trainees are completing more WBAs than specialty trainees but it is expected
that as trainees who have ‘grown up’ with WBAs become the majority within specialty
training, this difference will be minimised. There is no clear difference with regards
the uptake of WBAs by different specialties, in particular the numbers for trauma and
orthopaedic surgery have come up to the level of other surgical specialties in 20122013 following the transition onto ISCP from OCAP.
11
The variation in completed WBAs by training region highlights differences in
approach across the UK by Deanery / LETB. For example in London there was a
requirement in February 2012 for trainees to complete 80 per annum 8. This
requirement has resulted in a peak in WBAs in London from both core and specialty
trainees in 2012-2013. Despite this peak, the medians remained below 80 per annum
at a median of 74 and 63 per annum for core and specialty trainees in surgery,
respectively. Powell and colleagues specifically evaluated trainee and trainer opinion
with regards increasing the number of WBAs per trainee per year 7. Several concerns
were highlighted including a reduction in efficacy with increasing number, an
increased burden on trainers and the development of trainees focussed on
quantitative rather than qualitative outcomes. There is however no clear guidance in
the optimum number and further evaluation is required.
The dominance of ‘operative’ or interventional WBAs, particularly PBAs, amongst
specialty trainees is perhaps unsurprising. Work examining WBAs undertaken in
Wales emphasised that usage amongst specialty trainees is skewed towards PBAs,
with the authors concluding that the annual incremental uptake of WBAs between
2007 and 2013 “is reflective of the acceptability and reliability of the PBA and the
increasing confidence of trainers and trainees with WBAs” 9. However WBAs should
also be used to evaluate non-operative skills such that the more equitable proportion
of the different WBA tools seen in Core training should be re-established in Specialty
training.
The use of WBAs in surgical training is an evolutionary process. Although the
educational value of PBAs is supported by an evidence base
about the reliability of some of the other tools
3,11
there is uncertainty
3 10
. The existing evidence base for
WBAs is related to their formative rather than their summative use, and trainee and
trainer survey data underlines the desire at present that their use reflects this7. The
approach to the introduction of WBAs has somewhat inadvertently resulted in them
being used as assessments of learning. If WBAs are used as assessments for
learning additional work is required to support the educational value of the other key
WBAs, namely DOPS, CEX and CBD. Indeed the GMC has highlighted the need to
separate the use of WBAs into Supervised Learning Events (SLEs, an assessment
for learning, formative) and Assessment of Practice (AoP, an assessment of learning,
summative) 11.
12
Study limitations
This work represents a preliminary, purely descriptive analysis, which will form the
basis for further analyses of the rich dataset. The analysis looks at number of WBAs
completed, not how WBAs are being used nor the content or outcomes of the
assessment, i.e. whether trainees are performing appropriately for their stage in
training or if they are showing an improvement. Furthermore, no qualitative data has
been analysed to date. Given this analysis focuses on validated WBAs, it is
questionable that the mechanisms that trainers have for validating WBAs are
universally uniform.
Future work
Future work will aim to address the shortcomings of this present analysis, including
an examination of the qualitative elements within WBAs, how WBA numbers relate to
outcomes in terms of WBA final ratings and ARCP outcomes, and an analysis of
index surgical procedures numbers against PBAs through integration of the elogbook. With regards examining trainers’ input, an analysis of the quality of
feedback is planned as well as a review of individual trainers’ training activities.
There is an aim to introduce an automated report within ISCP, which may act as a
data source on training for all stakeholders.
13
Conclusions
This study confirms quantitatively the uptake of WBAs in surgical training in the UK. It
identified that surgical trainees are on average undertaking one WBA per week. The
majority of WBAs have assessed operative or interventional skills with lesser
emphasis on clinic-based interactions.
This study has shown some variation in the use of WBAs between the training
regions. Core trainees tend to use the spectrum of WBAs more frequently than their
senior colleagues. As trainees progress through training, operative or interventional
assessments predominate with limited use of non-operative assessments. Further
work is required to examine the role of WBAs in assessment, and the engagement
and training of trainers in the processes and validation of WBAs.
14
References
1. Health Do. Modernising Medical Careers, 2003.
2. McKee RF. The Intercollegiate Surgical Curriculum Programme (ISCP). Surger
(Oxford) 2008;26(10):411-16.
3. Shalhoub J, Vesey AT, Fitzgerald JE. What Evidence is There for the Use of
Workplace-Based Assessment in Surgical Training? J Surg Educ
2014;71(6):906-15.
4. Surgery ABo. Training & Certification - Resident Performace Assessments.
Secondary Training & Certification - Resident Performace Assessments.
http://www.absurgery.org/default.jsp?certgsqe_resassess.
5. Marriott J, Purdie H, Crossley J, et al. Evaluation of procedure-based assessment
for assessing trainees' skills in the operating theatre. Br J Surg
2011;98(3):450-7.
6. Fitzgerald JE, Giddings CE, Khera G, et al. Improving the future of surgical
training and education: Consensus recommendations from the Association of
Surgeons in Training. Int J Surg 2012;10(8):389-92.
7. Powell HRF, DiMarco AN, Saeed SR. Trainee and trainer opinion on increasing
numbers of workplace-based assessments. Bulletin of the Royal College of
Surgeons of England 2014;96(5):160-62.
8. Khera G, Baird E. Annual Workplace-Based Assessments. In: Standfield NJ, ed.
London, 2012.
9. Fishpool SJC, Stew B, Roberts C. Otolaryngology WBAs in the Wales Deanery:
the first six years. Bulletin of the Royal College of Surgeons of England
2014;96(5):164-66.
10. O'Kane R. Workplace Based Assessments: The Association of Surgeons in
Training, 2010.
11. Council GM. Learning and assessment in the clinical environment: the way
forward: General Medical Council, 2011.
15
Tables
Table 1
The adoption of workplace-based assessment (WBA) tools. Data represent trainer-validated WBAs between 2007 and 2013.
2007-2008
2008-2009
WBAs total
11,153
41,685
Number of trainees
1,549
3,042
Female
Male
No information
Median WBAs per
trainee
273.8%
2009-2010
82,627
98.2%
3,851
2010-2011
141,708
71.5%
4,705
2011-2012
210,249
48.4%
5,459
2012-2013
266,743
26.9%
5,808
415
26.8%
773
25.4%
1,037
26.9%
1,258
26.7%
1,497
27.4%
1,618
27.9%
1,121
72.4%
2,249
73.9%
2,799
72.7%
3,437
73.0%
3,943
72.2%
4,172
71.8%
13
0.8%
20
0.7%
15
0.4%
10
0.2%
19
0.3%
18
0.3%
6
13
20
29
16
39
44
TOTAL 2007-2013
754,165
Table 2
Workplace-based assessments by training region. HEE, Health Education England; WBA, workplace-based assessment.
Deanery / Local Education & Training
Board
Mean number of
WBAs per trainee
2011-12 2012-13
Median number of
WBAs per trainee
2011-12 2012-13 Change
Inter-quartile range
2011-12
2012-13
East Midlands HEE
35.0
38.0
35.0
41.0
6
22-45
23
27-48
21
East of England HEE
33.4
37.9
34.0
39.0
5
17-47
30
24-51
27
KSS HEE
44.5
47.8
49.5
52.0
3
24-65
41
28-66
38
London HEEs
45.3
61.3
40.0
67.0
27
20-68
48
37-84
47
North West (Mersey Sector) HEE
33.2
41.1
34.0
43.0
9
22-44
22
30-51
21
North West (North West Sector) HEE
38.8
44.2
41.0
45.0
4
34-48
14
39-54
15
Northern East HEE
38.9
41.2
40.0
41.0
1
31-48
17
33-48
15
Northern Ireland
34.9
40.2
38.0
41.0
3
23-43
20
36-47
11
Scotland
32.9
37.2
38.0
39.0
1
16-45
29
26-46
20
South West HEE
32.2
36.9
33.0
39.0
6
17-45
28
23-48.5
26
Thames Valley HEE
36.7
38.7
40.0
42.0
2
17-50
33
27.5-49
22
Wales
30.9
37.1
30.0
40.0
10
19-42
23
19-52
33
Wessex HEE
33.5
37.4
37.0
39.0
2
17-47
30
26-47
21
West Midlands HEE
41.3
47.1
42.0
48.0
6
22-55
33
34-61
27
Yorkshire and the Humber HEE
31.0
37.5
32.0
39.0
7
17-44
27
24-48
24
17
Table 3
Workplace-based assessments by surgical specialty. Core surgical training takes into account all trainees at core level (CT1/ST1 and CT2/ST2)
including neurosurgery, trauma and orthopaedic surgery, and Scottish trainees. The deadline for trauma and orthopaedic trainees to register
with ISCP was the 4th of April 2011. WBA, workplace-based assessment.
Specialty
Mean number of
WBAs per trainee
2011-12
2012-13
Median number of
WBAs per trainee
2011-12 2012-13 Change
Inter-quartile range
2011-12
2012-13
Core Surgical Training
43.0
47.8
43.0
48.0
5
28-56
28
32-63
31
Cardiothoracic Surgery
36.8
43.5
34.0
41.0
7
20-47
27
26-54
28
General Surgery
37.3
44.4
38.0
42.0
4
25-46
21
31-57
26
Neurosurgery
43.0
45.8
41.0
42.0
1
28-52
24
32-59
27
Oral and Maxillofacial Surgery
41.0
45.8
41.0
42.0
1
30-50
20
33-58
25
Otolaryngology
41.1
46.6
41.0
45.0
4
26-54
28
35-58
23
Paediatric Surgery
34.2
34.1
37.0
39.0
2
23-47
24
20.5-47
27
Plastic Surgery
38.0
44.4
35.0
42.0
7
21.5-48.5
27
28.5-56
28
Trauma and Orthopaedic Surgery
Urology
Vascular Surgery
24.6
33.8
-
41.9
38.9
48.6
21.0
36.0
-
41.0
39.0
45.0
20
3
45
12-35
23-43
-
23
20
-
28-54
27-48
34-59.5
26
21
26
18
Table 4
Workplace-based assessments by level of training. ST8 level does not include oral and maxillo-facial surgery, and urology as trainees in these
specialties finish their training at ST7. It is also noted that the first cohort of Modernising Medial Careers trainees are finishing their training in
August 2013. CT, core surgical training; ST, specialty training in surgery; WBA, workplace-based assessment.
Training level
Mean number of
WBAs per trainee
2011-12 2012-13
Median number of
WBAs per trainee
2011-12 2012-13 Change
Inter-quartile range
2011-12
2012-13
ST1/CT1
41.7
47.8
44.0
49.0
5
21-57
36
26-65
39
ST2/CT2
37.5
40.8
38.0
43.0
5
17-51
34
20-57
37
ST3
26.8
33.3
23.0
32.0
9
10-41
31
12-48
36
ST4
24.9
32.0
21.0
31.0
10
9-38
29
13-45
32
ST5
25.6
30.6
23.0
29.0
6
9-39
30
10-44
34
ST6
24.6
30.4
21.0
28.0
7
9-38
29
12-44
32
ST7
27.4
30.3
25.0
28.0
3
13-39
26
12-43
31
ST8
26.7
35.5
21.5
33.0
12
5-40
35
20-45
25
19
Figure Legends
Figure 1
Distribution of frequency of validated workplace-based assessments (WBAs). (a)
Core surgical trainees 2011-2012. (b) Specialty trainees in surgery 2011-2012. (c)
Core surgical trainees 2012-2013. (d) Specialty trainees in surgery 2012-2013.
Figure 2
Median numbers of validated workplace-based assessments by training region. (a)
For all trainees 2011-2012 (dark blue) and 2012-2013 (light blue). (b) For core
surgical trainees 2011-2012 (blue) and 2012-2013 (red). (c) For specialty trainees in
surgery 2011-2012 (blue) and 2012-2013 (red).
Figure 3
Break down of number of workplace-based assessment per type. (b) For core
surgical trainees 2011-2012 (pink) and 2012-2013 (red). (c) For specialty trainees in
surgery 2011-2012 (light blue) and 2012-2013 (dark blue). PBA, procedure-based
assessment; DOPS, direct observation of procedural skill; CBD, case-based
discussion; CEX, clinical evaluation exercise; OT, observation of teaching; MSF,
multi-source feedback.
20
Appendix
Glossary of terms presented in alphabetical order. Adapted from 3.
AoP
ARCP
CAMEO
CBD
CEX
CST
CT
DOPS
FTSTA
GMC
HEE
ISCP
JCST
LAT
LETB
MSF
OCAP
OPRS
OT
PBA
SHO
SLE
ST
SpR
StR
TAiP
TrACE
TtT
WBA
Assessment of Practice
Annual Review of Competence Progression
Clinic Assessment and Management Examination - Outpatient
Case-Based Discussion
Formalized discussion related to the care of a patient with a focus on
knowledge and attitude
Clinical Evaluation Exercise
An assessment of a trainee’s clinical skills (e.g., history, examination,
and information giving)
Core Surgical Training
Core Trainee
Direct Observation of Procedural Skill
Similar to PBA, but more generic and relate to either less complex
procedures or parts of a larger procedure. The domains of
assessment in DOPS are generic and fixed and considered to be
transferable across this spectrum of tasks
Fixed-Term Specialty Training Appointment
General Medical Council
Health Education England
Intercollegiate Surgical Curriculum Programme
Joint Committee on Surgical Training
Locum Appointment to Training
Local Education and Training Board
Multi-Source Feedback
An opportunity for members spread across the multidisciplinary health
care and administrative team to offer anonymous feedback on the
trainee. The results are then discussed at a meeting with the assigned
educational supervisor and the ARCP
Orthopaedic Competence Assessment Project
Operative Performance Rating System
Observation of Teaching
Workplace-based assessment permitting the documentation of an
observed teaching episode carried out by the trainee
Procedure-Based Assessment
Direct observation of an index procedure or operation with comments
given on important steps, tasks or skills at the pre-, intra- and postoperative stages of the procedure, considered to be essential for its
safe and successful completion
Senior House Officer
Supervised Learning Event
Specialty Trainee
Specialist Registrar
Specialty Registrar
Training and Assessment in Practice
Training and Assessment in the Clinical Environment
Training the Trainers
Workplace-Based Assessment
21
22