Selection for Family Medicine Residency
training: is it time to sharpen our tools?
A study of the consistency of selection
processes across Programs in Canada.
Keith Wycliffe-Jones BSc(Med Sci) MBChB FRCGP CCFP University of Calgary
Shirley Schipper MD CCFP University of Alberta
Maureen Topps MB ChB CCFP FCFP PBDM University of Calgary
Jeanine Robinson BA MPA University of Calgary
Kent G.Hecker PhD University of Calgary
Alianu Kingsly Akawung MSc Applied Statistics, BSc Biomed University of Calgary
ICRE October 2015
I do not have an affiliation (financial or otherwise)
with a pharmaceutical, medical device or
communications organization.
Je n’ai aucune affiliation (financière ou autre)
avec une entreprise pharmaceutique, un fabricant
d’appareils médicaux ou un cabinet de communication.
Author: Keith Wycliffe-Jones
Date: October 25th 2015
Selection for FM Residency Training
Objective
 Present initial results that address the question:
Do Family Medicine Residency Programs
rank the same applicants consistently?
Selection for FM Residency Training
Background
2013 Canadian Residency Matching Service(CaRMS) 1st iteration
 2695 Canadian Medical Graduate participants1
 1763 applied to Family Medicine(FM) for 1268 positions
 Canada - 17 schools; 106 sites
 979 applicants put FM as 1st choice
UK experience - General Practice selection
- no level of consistency (kappa coefficient -0.3) in ranking by
different Deaneries2
1.Canadian Resident Matching Service(CaRMS). https://www.carms.ca/en/r-1-match-reports-2013. (accessed April 23rd 2014)
2.Plint,S. A study of the outcomes of simultaneous multiple applications to different deaneries in the UK for vocational training in general practice - is there
a case for standardization of selection and processes. Dundee; 2005.
Selection for FM Residency Training
Background
 High stakes assessment process
 Canadian FM Programs more heterogeneous than UK
 Expected outcome - Family Physician who can practice
anywhere in Canada.
 Organizational Justice3,4,5
i) Distributive Justice - do applicants receive the selection
outcomes they deserve?
ii) Procedural Justice - fairness of procedure itself
3.Gilliband SW. The perceived fairness of selection systems: an organizational justice perspective. Academy of Management Review 1993; 18:694-734.
4.Gilliland SW. Effects of procedural and distributive justice on reactions to a selection system. Journal of applied psychology 1994; 79(5): 691-701.
5.Patterson F, Zibarras L, Carr V, Irish B, Gregory S. Evaluating candidate reactions to selection practices using organizational justice theory. Med Educ 2011
Mar; 45(3): 289-297.
Selection for FM Residency Training
Background
Distributive Justice
 Equity – “applicants should receive rewards that are
consistent with their inputs (self-conceptions of ability or
qualifications for job), relative to a referent comparison”.
 Equality - equal opportunity to apply
 Needs - rewards distributed based on individual needs.
Gilliland SW. The perceived fairness of selection systems: an organizational justice perspective. Academy of Management Review 1993; 18:694-734.
Gilliland SW. Effects of procedural and distributive justice on reactions to a selection system. Journal of applied psychology 1994; 79(5): 691-701.
Patterson F, Zibarras L, Carr V, Irish B, Gregory S. Evaluating candidate reactions to selection practices using organizational justice theory. Med Educ 2011 Mar;
45(3): 289-297.
Selection for FM Residency Training
Hypothesis (based on distributive justice)
 There will be a high-degree of consistency in how CMG’s
who apply to more than one FM Residency Program are
ranked
— particularly those Programs/Sites seeking similar
attributes
Selection for FM Residency Training
Methodology
Program ranking data for all FM Programs/Sites 2007-2013
Student ranking data 2007-2013
Multi – level modelling
 All students are ranked by some Programs (all to which they
have applied)
 Programs are nested within students – interested in the
variation in school rank within student
 Calculate intra-class correlation coefficient (ICC) as a measure
of consistency in ranking of students across
— All Programs and Sites
The Ottawa Conference 2014
Selection for FM Residency Training
Analysis - example grid of Program rank order data
Student ID
Program A
ID #156678
Program B
42
ID #219076
ID #133487
Program D
Program E
23
74
29
92
26
ID #542980
ID #701462
Program C
27
22
14
31
18
Selection for FM Residency Training
Analysis - Preliminary results
1. Across all 17 FM Residency Programs (2013)
— Within student variability = 3117.17 (high)
— ICC (intra class correlation) = 0.22 (low)
 low consistency of ranking of the same students
between Programs
Selection for FM Residency Training
Analysis - Preliminary results
2. ICC - all 17 FM Residency Schools (2007-2013)
0.4
0.35
Year
ICC
0.3
2007
0.34
0.25
2008
0.269
2009
0.216
2010
0.209
0.1
2011
0.239
0.05
2012
0.214
2013
0.225
0.2
ICC
0.15
0
2006
2007
2008
2009
2010
2011
2012
2013
2014
Selection for FM Residency Training
Analysis-sub-sets Preliminary results
3. Subset analyses
a) 4 rural FM Programs/Sites in same Western Province
— ICC = 0.66
b) 15 Western Canada rural FM Programs/Sites
— ICC = 0.34
c) 2 urban FM Programs in same Western Province
— ICC = 0.02
Selection for FM Residency Training
Analysis-sub-sets Preliminary results
Subset analyses(contd.)
d) 19 Eastern Canada rural FM Programs/Sites
-ICC = 0.50
e) 17 larger urban-based* Programs/Sites
-ICC= 0.19
(*>100,000 population, urban-based programs/sites)
Selection for FM Residency Training
Summary of preliminary results
 Suggest a very low to low level of consistency in
ranking of the same students between programs/sites
 Evidence of higher level of consistency in the ranking of
the same students between rural programs/sites
Selection for FM Residency Training
Questions?
Keith Wycliffe-Jones [email protected]
Help us improve.
Aidons-nous à nous améliorer.
Your input matters.
Votre opinion compte.
 Download the ICRE App,
 Téléchargez l’application de la CIFR
 Visit the evaluation area in
Pre-function Hall B, near
Registration, or
 Visitez la zone d’évaluation, au
vestibule de la salle B, près du
kiosque d’inscription, ou
 Go to: http://www.royalcollege.ca/
icreevaluations to complete the
session evaluation.
 Visitez le http://www.collegeroyal
.ca/evaluationscifr afin de remplir
une évaluation de la séance.
You could be entered
to win 1 of 3
$100 gift cards.
Vous courrez la chance de gagner
l’un des trois chèques-cadeaux
d’une valeur de 100 $.