I Think I Should Have an
OSCE…What Should I Consider?
It’s much easier to grant a “license”
than take it away…
Learning Resource Centre (LRC)
is a clinic-like
environment to learn
procedural, diagnostic
and communication
skills
provides expertise on
competence assessment
and simulation in
learning…
Is the OSCE the Best Way to Assess Your Candidates?
Validity
Reliability
Feasibility
Impact
About this Presentation
Less of a 20 minute primer on how to develop an OSCE
More of considerations before you meet with OSCE
“experts”
Two Questions:
Why are you thinking about an OSCE in your organization?
2. What 2 questions would you like me to address in next ½
hour?
1.
Competence or Performance Assessment
Competence
Measures what “doctors”
in a controlled environment
representing a sample of professional practice
Performance
Measures what “doctors” do in their practice environment
(workplace)
A Simple Model of Competence
Does
Behaviour
Shows how
Knows how
Knows
Cognition
Validity
Validity is the degree to which a test measures what
we intend it to measure
Validity may be considered in terms of
Content
Relationship to other scores
(e.g., “gold standard”)
Ability to predict performance
Reliability
The degree to which the scores obtained on one
administration of the test would be consistent with
those obtained on a second administration, using
the same or similar group
Reliabilities range from .0 to 1.0
Reliabilities of >0.70 preferred
Thinking about an OSCE
Determine your budget
Prepare a blueprint
Determine assessment criteria
Determine and develop representative cases
Recruit and train standardized patients
Recruit and orient examiners
Recruit and train OSCE staff
Administer then debrief the OSCE
Budgets
Administrative staff
Developers standardized patient educators
Exam day administrative staff
Standardized patients (clients)
Training + exam day time/hour/rate
Examiner honoraria + expenses
Supplies
Space rental
Exam day catering
Data Analysis consultation
Prepare a Blueprint
Make your purpose explicit
Clinical clerks readiness to start PGY1
Will guide level of difficulty for case development and
examiner ratings
Based on objectives
Determine domains to be assessed
Must be feasible for simulation
Determine the proportion of cases for each domain
Length of time per case and time between cases
Estimate length of your exam
Assessment Criteria
Develop rating instruments
Checklists
- Typically “yes/no” items
Global ratings on domains
- Behavioural anchors
- Inferior, Poor, Borderline, Satisfactory, Very Good, Excellent
Overall Global Rating
Determine “success” criteria in advance
Borderline method, for example.
Representative Cases
Congruent with professional practice
Content and time in the exam
Develop a bank of cases equal in length of time
Focus on assessing application of learning
Develop templates and directions for case authors
Cases developed in groups is best
Include detailed patient scripts
Will examiners ask candidates questions to be scored
Pilot test the cases
Recruit and Train SPs
Must have: experienced SP Educator
2+ hours training patients/case
SPs carefully selected for appropriate “fit”
Remember to recruit and train spare SPs/case
“dry run” the case with case expert
“dry run” again on exam day with examiner
Recruit and Orient Examiners
One examiner/case plus spares for exam day
Be aware of conflicts of interest
Clarify rating criteria
Examiners must follow the criteria
Ask for case input post-exam
Provide them feedback on their performance
“hawk” or “dove”
Recruit and Train OSCE Staff
Staff must understand purpose of OSCE
OSCEs require delegation of responsibility
Detailed planning and implementation is critical
Attention to detail is paramount
Orient them to the site
Explicit directions for each staff person role
Administer then Debrief the Exam
High stakes exams require a “secure” exam site
Develop protocols, publish and follow them; such as
candidate late arrival
appeals processes
Monitor your plan, make notes for next administration
Some Last Thoughts
Appoint a chief examiner: a must for QA
Candidates will be very nervous…
Orient them to content and process
Confidentiality of the exam
Staff, SPs, examiners
Leave your site/space as you found it
Do not leave until you have all paperwork
Summary
"If I had asked people what they wanted, they would
have said faster horses.”
Henry Ford
“We will be creative thinkers and doers as we plan
medical and health profession education. Simulation
in learning may be, in the next 5-7 years, what problem
based learning was in the early 1990s.”
Bruce Holmes
Executive Director, Learning Resource Centre
I Think I Should Have an
OSCE…What Should I Consider?
Bruce Holmes
Assistant Professor
Executive Director, Learning Resource Centre
Faculty of Medicine
[email protected]
902.494.3783