Standard Setting for a
Performance-Based Examination
for Medical Licensure
Sydney M. Smee
Medical Council of Canada
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
MCC Qualifying Examination Part II
OSCE format - 12 short stations
● 5 or 10 minutes per patient encounter
● Physicians observe and score performance
Required for medical licensure in Canada
Prerequisites
● Passed MCCQE Part I (Knowledge & Clinical reasoning)
● Completed 12 months of post-graduate clinical training
Pass/Fail criterion-referenced examination
Multi-site administration - twice per year
Overall fail rate 10%-30%
Implemented 1992
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Why do it?
Requested by licensing authorities, largely
in response to two issues:
● Increase in complaints, many centered around
communication skills.
● Public accountability - OSCE to serve as an
“audit” of training of all candidates seeking
licensure in Canada.
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Blueprint Considerations
Four domains
*History-taking
*Patient Interaction
*Physical Examination *Management
Multi-disciplinary / multi-system content
Patient demographics
Two formats
*5+5 couplets & 10 minute
Each case based on an MCC Objective
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Standard for MCCQE Part II
Acceptably competent for entry to independent
practice
Conjunctive standard
● Pass by total score
AND
● Pass by minimum number of stations
High performance in a few stations does not
compensate for overall poor performance
Just passing enough stations does not compensate
for overall poor performance
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Translating a Standard to a Pass Mark
Pilot exam: Ebel method
● Items rated for relevance and importance
● Pass based on most relevant and important items
● Failed 40%
First two administrations: Angoff method
●
●
●
●
Estimated score for the minimally competent candidate
Pass based on average of estimates per instrument
Pass marks varied more than the test committee liked
Test committee did not like the task
1994: Adopted borderline group method
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Physicians as Scorers
Three Assumptions:
Clinicians do not require training to judge
candidate behaviour according to checklists
for basic clinical skills
Most clinicians can make expert judgments
about candidate performance
Being judged by clinicians is vital for a
high-stakes examination
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Physicians as Standard Setters
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Global Rating Question
Did the candidate respond satisfactorily to
the needs/problems presented by this patient?
•Borderline Satisfactory
•Good
•Excellent
•Borderline Unsatisfactory
•Unsatisfactory
•Inferior
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Numbers....
1,000-2,200 candidates per administration
Examiners each observe 16-32 candidates
20-60 examiners per case
Number of candidates identified as
borderline per case ranges from 150-500
Collect >99% of data for global rating item
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Modified
Borderline Group Method
Examiners (content experts) identify borderline
candidates based on the 6-point scale
Scores of borderline candidates define
performance that “describes” the pass standard
Examiner judgments are translated into a pass
mark by taking the mean score for the borderline
candidates for each case
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Pass Marks by Case Across Exams
Challenge to assess pass marks over
multiple administrations
● Scoring instruments are revised post-exam
● Rating scale items have been revised
● Rating scale items have been added to cases
As competency and difficulty of cases
changes, so do cut scores
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Setting Total Exam Pass Mark
Pass marks for cases are summed
Add one standard error of measure (3.2% )
Pass mark falls between 1 to 1.5 SD below
mean score
Station performance is reviewed by Central
Examination Committee
● Then the standard for the number of stations
passed is set
Standard has been 8/12 since 2000
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Outcomes
15,331 candidates became eligible in 2000 – 2005
● 6,099 have yet to attempt MCCQE Part II
● 8,514 have passed
● 718 or 7.7% failed
2,243 candidates were eligible prior to 2000 and
also took MCCQE Part II in 2000 – 2005
● 2,166 have passed
● 77 or 3.4% failed and are likely out of the system
Fail rates do not reflect impact on repeat takers
● Focused hundreds of candidates on remediation
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Limitation
Current approach is easy to implement but it
relies upon
● Large number of standard setters per case
● Large number of test takers in borderline group
Smaller numbers would lead to more effort
● Increase training of examiners
● Impose stricter selection criteria on standard
setters
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
What’s ahead?
Increasing number of candidates to be
assessed each year
● Modifications to the administration are needed
• Predictive validity study currently in progress
● Use non-physician examiners?
• Which type of cases, who sets standard?
● Add more administrations?
• Case development / challenge of piloting content
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Medical Council of Canada
Ottawa
Sydney M. Smee, M.Ed.
Manager, MCCQE Part II
www.mcc.ca
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona