Clinicians’ Perspectives on the
Challenges of Standard Setting for
Objective Structured Clinical
Examination (OSCE)
Estebes Hernandez, M.D., Daniel Kahn, MD, Edward Ha, MD, Christina Harris, MD
Monica Plesa, MD, Rachel Brook, MD, Ming Lee, PhD
2
Outline
• The Observed Structured Clinical Exam
• Standard Setting
• The Process
• The Faculty
• Survey of Faculty
3
The Observed Structured Clinical Exam
• 3rd
year medical students at the David Geffen School
of Medicine at UCLA
•8
SP encounters lasting 15 minutes each
• Test
Administered by 8 California Medical Schools
• Graded
on checklist items (History, Physical, Patient
Physician Interaction)
4
Norm vs. Criterion
Norm-Referenced Standard: A relative standard based
on the performance of a comparative group of testtakers
Setting the pass/fail cut-score at 1 standard
deviation below the mean of the class
Criterion-Referenced Standard: An absolute standard
based on an expected minimal amount of competency
on the test-takers by a panel of content experts
Modified
Angoff Method
5
Modified Angoff Method
• Estimates by faculty
• Round of Discussion
• Reality Check
• New Estimates
• Final Round of Discussion (if major discrepancy)
6
The Faculty (7 total)
• Professor of Education
• Assistant Dean of Curricular Affairs
• 2 Assistant Program Directors of Residency Programs
• 2 Chairs of Colleges at the Medical School
• 6 Clinician Educators
•3
Hospitalists, 2 General Internal Medicine, 1 Family Medicine
7
Survey of Faculty
• E-mail survey of clinicians who participated
• Responses 6/6
• Questions
• What
were some of the challenges encountered by clinician
educators who participated in standard setting process for the
OSCE?
• How
were these challenges overcome?
• What
could have been done differently?
• Would
the clinician educators use this method for other tests in
the future in other settings?
8
Results of Survey
• What were some of the challenges?
• Determining
how often a minimally competent third year medical
student would get an item correct (4/6)
• Scheduling
a time when all clinicians could meet (3/6)
• Not
allowing clinicians experience to influence their decision
(2/6)
9
Results of Survey
• Overcoming Challenges - Defining Minimally Competent
• Material Covered in Curriculum up to MS3 year
(3/4)
• Group Discussions
• Basic
Question/Maneuver?
• Critical Question/Maneuver?
• Advanced Question/Maneuver?
• Done in Practice of Residents/Faculty?
• Does the Item Affect Medical Decision Making?
• Reality
Check
10
Results of Survey
• What could have been done differently?
• Video Review of a Minimally Competent Student (2/6)
• Video Review of someone else performing Angoff method
• Review of curriculum material
• Use in Future Settings?
• Yes
(6/6)
• All
concerned though regarding logistical issues (time
consuming, lack of resources
11
•Questions?
12