Xerte
Standard Setting
in
Psychiatry Undergraduate OSCE
Dr Ben di Mambro
Clinical Lecturer in Psychiatry
Psychiatry Examination

80 EMIs



1 hour 15 minutes
Standard set
2 OSCE stations



Each station 10 minutes
3 tasks based on competencies
% given for each task
May OSCE
Station
Max
Min
All
A
60.1
(14.3)
B
60.8
(6.6)
46.4
(5.6)
48.2
(4.4)
49.0
(9.7)
47.2
(4.1)
54.5
(10.3)
55.5
(9.0)
56.2
(9.8)
54.3
(10.8)
C
D
66.1
(8.5)
67.7
(12.8)
Psychiatry OSCE

Changes to the OSCE


Criterion marking scheme
Global clinical assessment
Honours
 Pass
 Borderline
 Fail



2 examiners were placed in each room
Standard set pass mark for each task
Task
Develop CAL package:
 Train the examiners in the new marking
scheme
 Standardise marking scheme


To allow for multiple sites to accommodate
increasing numbers of students
Improve inter-rater reliability
Development of Package

Scripted 10x 2-3 minute OSCE scenarios



Borderline/Pass
Pass/Honours
2 hours filming in my office


2 lecturers
1 bullied student
Creation of Media



wmv files converted to flv using Flash
Macromedia
Images created utilising Flash Macromedia
MP3 creation using Audacity
Xerte


Template ‘ratings’ created by Julian
Tenney
Media was imported into Xerte through
the Wizard
The Package



An introduction to the new marking
scheme
Example mark sheets
Video of example students



Rated on criteria
Global impression given
User then offered feedback
Using the Wizard

Once Wizard create – efficient method of
presenting content




Define scenario
Define marking sheet, including number of
criteria
Define fail/borderline/pass/honours
Import media
Sample of Package
Distribution of Package


Letters and emails were sent to all
examining consultants with following link
www.nottingham.ac.uk/~mczbjd/OSCE.htm
Package ran for ~20 minutes
Result of Package Introduction


30% examiners accessed the package
prior to the OSCE
Remainder underwent abridged version
before the OSCE
Analysis

ANOVAs undertaken on awarded scores by
the different examiners on the same
station
December OSCE
Station
A
B
C
D
Max
Min
All
70.3
(5.0)
72.6
(9.1)
66.1
(5.1)
66.1
(56.7)
53.4
(8.6)
52.1
(8.85)
51.0
(7.6)
56.6
(5.0)
62.3
(8.6)
63.7
(11.5)
56.9
(8.6)
60.7
(7.4)
May
December
OSCE 1
54.5**
(10.3)
62.3**
(8.6)
OSCE 2
56.2**
(9.77)
63.7**
(11.5)
OSCE 3
55.5**
(9.02)
56.9**
(8.6)
OSCE 4
54.3**
(10.8)
60.7
(7.4)
**p<0.001 *p<0.05
Variation
Station
May
Dec
A
13.7
16.9
B
12.6
20.5
C
17.1
15.0
D
20.5
10.5
Limitations





Flash Player 7 not installed as standard on
healthcare trusts’ computers
Minor problems with Xerte - since resolved
Delay on slow broadband connections
?Representative videos
Any improvement likely to be due to the
new marking scheme
Xerte
Once familiar:
 Easy to use
 Multimedia packages can be create in
minutes
 Easily distributed
 Much more complicated to create bespoke
packages
Future





Improve IT access in the healthcare trusts
Reduce bitrate
Produce improved examples of borderline
students
Extend the package to attempt to
standardise the Formative assessments
?Make available to students