Documenting the Cognitive Performance of
Family Medicine Practicing Outpatient Medicine
Author: Allen F. Shaughnessy, PharmD, MMedEd
Date: 27 September 2013
I do not have an affiliation (financial or otherwise)
with a pharmaceutical, medical device or
communications organization.
Je n’ai aucune affiliation (financière ou autre)
avec une entreprise pharmaceutique, un fabricant
d’appareils médicaux ou un cabinet de communication.
Author: Allen F. Shaughnessy, PharmD, MMedEd
Date: 28 September 2013
Competency-based medical Education: Documenting Cognitive Performance| Shaughnessy
Introduction
• Physicians think for a living
• Development of cognitive skills, therefore, is a focus of residency
training
• Part of competency assessment involves assessing developing
cognitive performance over time in all areas of practice to
assure that graduates are competent.
The International Conference on Residency Education
|
La conférence internationale sur la formation des résidents
3
Competency-based medical Education: Documenting Cognitive Performance| Shaughnessy
Introduction
• This assessment occurs during precepting though it is not
often documented
• To capture preceptors’ assessments of the residents’
cognitive performance we developed the Resident
Cognitive Skills Documentation Form (CogDoc)
• Goals of study
» Develop a tool for capturing preceptors’ impressions of
residents’ performance in the care of individual patients
» Describe our experience with its use
» Determine its reliability
» Describe residents’ use of the formative feedback
The International Conference on Residency Education
|
La conférence internationale sur la formation des résidents
4
Competency-based medical Education: Documenting Cognitive Performance| Shaughnessy
The International Conference on Residency Education
|
La conférence internationale sur la formation des résidents
5
Competency-based medical Education: Documenting Cognitive Performance| Shaughnessy
Tool Development
• Collapsed Bloom’s 6 cognitive domains into 3 dimensions
» Application
» Understanding
» Medical Knowledge
Bloom, Benjamin S. & David R. Krathwohl. (1956). Taxonomy of educational objectives: The classification of educational
goals, by a committee of college and university examiners. Handbook 1: Cognitive domain. New York , Longmans.
The International Conference on Residency Education
|
La conférence internationale sur la formation des résidents
6
Competency-based medical Education: Documenting Cognitive Performance| Shaughnessy
Scale
• Three descriptive levels parallel to the Dreyfus model
» Below competent (“advanced beginner”)
» Competent (“competent”)
» Above competent (“proficient”)
• Criterion-referenced based on
descriptions
• Linked to entrustable professional activities
» Based on symptoms, not diagnoses
» Bounded knowledge and performance criteria
The International Conference on Residency Education
|
La conférence internationale sur la formation des résidents
7
Competency-based medical Education: Documenting Cognitive Performance| Shaughnessy
The International Conference on Residency Education
|
La conférence internationale sur la formation des résidents
8
Competency-based medical Education: Documenting Cognitive Performance| Shaughnessy
Scale
• Linked to entrustable professional activities
» Based on symptoms, not diagnoses
» Bounded knowledge and performance criteria
The International Conference on Residency Education
|
La conférence internationale sur la formation des résidents
9
Competency-based medical Education: Documenting Cognitive Performance| Shaughnessy
The International Conference on Residency Education
|
La conférence internationale sur la formation des résidents
10
Competency-based medical Education: Documenting Cognitive Performance| Shaughnessy
Method
• Capture all scores since inception (Oct 2008 – June 2011)
• Compare rates of completion by year of training
• Determine internal consistency
» Compared scores given to final-year residents for their
top 5 diagnoses
» Assumed that performance would be high and thus
differences would be due to differences among raters
• Survey of residents to determine their use of the reports
The International Conference on Residency Education
|
La conférence internationale sur la formation des résidents
11
Competency-based medical Education: Documenting Cognitive Performance| Shaughnessy
Results
• 5,330 visits documented for 33 residents by 38 faculty
members
» 27.2% of all visits
• PGY-1
37.5%
• PGY-2
30.1%
• PGY-3
26.7%
» Extrapolates over 3 years of training to ~600
assessments per resident
• Reliability via internal consistency (Chronbach’s alpha)
» Medical knowledge: .92
» Understanding: .94
» Application .94
The International Conference on Residency Education
|
La conférence internationale sur la formation des résidents
12
Competency-based medical Education: Documenting Cognitive Performance| Shaughnessy
Results
• Resident survey
» 30 of 33 (90.9%) reported looking at individual
assessment forms
• 58.6% read “most of the forms”
• 34.5% read “only forms documenting below competence”
» 32 of 33 (97%) looked at the graphical summary
» 27 of 33 (81.8%) agreed that the assessment “usually
reflected their performance”
The International Conference on Residency Education
|
La conférence internationale sur la formation des résidents
13
Competency-based medical Education: Documenting Cognitive Performance| Shaughnessy
Limitations
• Only 1-in-4 patient visits documented
• Risk of
» Documentation of exceptional cases vs. central tendence
effect
» Halo effect: score on one dimension influences the score of
another dimension
» Takes an experienced preceptor < 10 seconds to complete
• Has the potential to collect 1500 – 2000 data points
documenting the competence of each resident in outpatient
care
• Assessment is at the “Does” level of Miller’s pyramid*
• Validity has not yet been assessed
* Miller G. Acad Med. 1990;1990(65 (Suppl)):S63-S67.
The International Conference on Residency Education
|
La conférence internationale sur la formation des résidents
14
Competency-based medical Education: Documenting Cognitive Performance| Shaughnessy
Benefits
• Takes an experienced preceptor <10 seconds to complete
• Has the potential to collect 1500 – 2000 data points
documenting the competence of each resident in
outpatient care
• Assessment is at the “Does” level of Miller’s pyramid*
• Little risk of memory loss by preceptors since the
information is collected in the moment
• Immediate feedback to residents
* Miller G. Acad Med. 1990;1990(65 (Suppl)):S63-S67.
The International Conference on Residency Education
|
La conférence internationale sur la formation des résidents
15
Help us improve.
Aidons-nous à nous améliorer.
Your input matters.
Votre opinion compte.
• Download the ICRE App,
• Téléchargez l’application de la CIFR
• Visit the evaluation area in the
North Building, Upper Level, near
registration, or
• Visitez la zone d’évaluation, au
bâtiment nord, niveau supérieur,
près du kiosque d’inscription, ou
• Go to: http://www.royalcollege.ca/
icreevaluations to complete the
session evaluation.
• Visitez le http://www/collegeroyal
.ca/evaluationscifr afin de remplir
une évaluation de la séance.
You could be entered
to win 1 of 3
$100 gift cards.
Vous courrez la chance de gagner
l’un des trois chèques-cadeaux
d’une valeur de 100 $.