15 July 2013
Strategy and Policy Board
5
To consider
Student selection next steps
Issue
1.
While we do not have a formal role in student selection, we do require in
Tomorrow’s Doctors that processes for selection must be open, objective, and fair.
2.
We also have a clear interest in the selection of individuals who will a few
years later join the medical register as licenced doctors.
3.
In addition, there is sustained political pressure on medicine as a profession
to open itself up to talented individuals from lower socioeconomic backgrounds.
4.
The Medical Schools Council has set up the Selecting for excellence group to
take forward work on student selection. Within this context we have identified next
steps on student selection for agreement by the Strategy and Policy Board.
Recommendations
5.
The Strategy and Policy Board is asked to:
a.
Agree to focus our influence on student selection through the Selecting
for excellence group.
b.
Discuss whether we have identified the right priority issues that we
would like to see the group address.
c.
Agree that we should continue to collect data on student selection
through the Medical School Annual Return and to analyse the responses to
the socioeconomic status questions in this year’s National Training Survey.
Issue
Our role and interest
6.
While we do not have a formal role in student selection, we do require in
Tomorrow’s Doctors that processes for selection must be open, objective, and fair.
7.
The 33 UK medical schools are responsible for putting in place processes that
meet our standard and for individual selection decisions.
8.
We also have a clear interest in the selection of individuals who will a few
years later join the medical register as licenced doctors. To start with, our concern
was that there is significant variation, and in some cases inconsistency (e.g. some
make use of personal statements while others consider these unreliable), between
medical schools’ selection processes without apparent justification. The extent of
variation is illustrated in the flow charts at Annex A.
9.
Traditionally, medical schools have exercised autonomy in delivering medical
education, including in selecting students. Generally, they tend to be sensitive to
moves, or perceived moves, by the GMC or other national bodies to exert greater
influence on selection processes.
External factors
10.
In May 2012, Alan Miburn, now chair of the Social Mobility and Child Poverty
Commission, published Fair access to professional careers: a progress update. The
report focused on four professions including medicine and concluded:
‘Medicine has a long way to go when it comes to making access fairer,
diversifying its workforce and raising social mobility... Its success in recruiting
more female doctors and doctors from black and minority ethnic backgrounds
indicates that with the right level of intentionality the medical profession can
also throw open its doors to a far broader social intake than it does at present
... Overall, medicine has made far too little progress and shown far too little
interest in the issue of fair access. It needs a step change in approach.
11.
The Commission have told us informally that medicine remains one of their
top priorities in terms of fair access to professional careers and that this will be
reflected in their annual report, due to be published in September 2013.
12.
Government is also putting pressure on medicine to widen access. While
Deputy Prime Minister Nick Clegg is the Government lead on social mobility,
Dan Poulter, Health Minister, has shown an interest in medicine, meeting with Peter
Rubin and Niall Dickson earlier this year before hosting a seminar on widening
access to medicine in March 2013. David Willets, Universities and Science Minister,
has also shown an interest in his role as chair of the Gateways to the Professions
Collaborative Forum Executive Group. Both ministers sit on a cross-ministerial group
on social mobility that coordinates the Government’s action in response to Milburn’s
reports.
2
13.
Health Education England (HEE) has a role to play in student selection as set
out in its mandate. In particular, HEE has an interest in values based selection, the
use of aptitude tests to allow a broader approach to selection, and also working with
local schools through its regional boards so that a wide range of students are
encouraged to consider a career in health as a rewarding option and are supported
into relevant work experience.
What we have done so far
14.
In June 2011 we held a seminar with key interests on the role of the regulator
in student selection.
15.
We co-funded and contributed to a Social Mobility Toolkit which was
published by the Professions for good coalition in March 2012.
16.
In February 2013 we published a literature review titled Identifying best
practice in the selection of medical students. We commissioned the research to
summarise evidence on student selection and widening access and to make
recommendations. The report suggested that the evidence base is weakened by a
lack of agreement as to the purpose of student selection – should it be to identify
good students or good doctors of the future? That said, the evidence for some
selection methods – ie multiple mini interviews (MMIs), aptitude testing, situational
judgement tests (SJTs), and selection centres – is better overall than for some
methods such as traditional interviews, references and autobiographic submissions
(eg personal statements). The evidence specifically on widening access was even
more limited. But what evidence there is suggests that some pre-entry activities are
effective, but more data in this area is needed, especially in terms of support for
successful applicants from lower socioeconomic backgrounds once they are studying
and their future career pathways. A summary of the research is at Annex B.
17.
Since the literature review was published, we have discussed its findings with
the Medical Schools Council (MSC). It was decided that the MSC should set up a
working group take action on the findings of our literature review and Alan Milburn’s
reports. The Selecting for excellence group has now been established. It is chaired
by Tony Weetman, current MSC chair. We are on the group along with the HEE, the
Department of Health England, and others. Dan Poulter has been invited to an event
to launch the group in July 2013. See Annex C for the group’s terms of reference.
18.
Since the literature review was published, we have also participated in a
seminar on widening access to medicine hosted by Dan Poulter, which came about
following his meeting Peter Rubin and Niall Dickson in March 2013. We have also
met with the Social Mobility and Child Poverty Commission. We continue to attend
the Gateways to the Professions Collaborative Forum Executive Group and the most
recent meeting was on 5 June 2013.
Next steps
We propose to focus our influence on student selection through the Selecting
for excellence group. Medical schools are responsible for the selection of medical
19.
3
students and so it is important that they – through the MSC as their umbrella body –
take the lead. We are also mindful of our limited role in selection and the
sensitivities noted at paragraph 9.
20.
There is no agreed resource commitment associated with our participation in
the group apart from staff time. We may be asked to contribute resources on a case
by case basis, e.g. funds to commission research, space to host an event or extra
staff time to gather and analyse information.
Although we plan to focus our influence through the Selecting for excellence
group, we will also discuss student selection and widening access bilaterally with
organisations such as the Social Mobility and Child Poverty Commission where
appropriate.
21.
22.
From our regulatory perspective we have identified several priority issues that
we would like to see the group address.
a.
The variation of selection processes used by medical schools without
apparent justification.
b.
The use of aptitude tests such as the UK Clinical Aptitude Test in the
light of evidence but also in light of continuing perceptions that they are
susceptible to coaching.
c.
Whether the use of non-academic components of personal statements
is defensible in light of the evidence.
d.
Whether all schools should move towards more, rather than less,
structured interviews, eg multiple mini interviews, in light of the evidence.
e.
How to select students with the values to be good doctors.
f.
The need for evaluation of initiatives to widen access to medicine.
g.
Access to work experience for applicants from lower socioeconomic
groups and how work experience is evaluated as part of selection processes.
h.
The use of contextual data as part of selection processes.
23.
We will also continue to gather information on current selection processes
through the Medical School Annual return and feed this into the work of the group.
24.
We also asked doctors in training questions about their socioeconomic status
as part of the National Training Survey for 2013. This has been a pilot and we will
consider whether to repeat it in 2014. Analysis of the answers to these questions is
currently underway. This will be the first time we have considered socioeconomic
status at the postgraduate level and over the next year we plan to combine the data
with other sources of data (eg relating to recruitment) to look at whether
socioeconomic status is related to career progression.
4
Supporting Information
How this issue relates to the corporate strategy and business plan
25.
Corporate strategy strategic aim 3: To provide an integrated approach to the
regulation of medical education and training through all stages of a doctor’s career.
26.
Business plan theme: Meeting future needs and changing demands of
doctors.
How the issues differ across the four UK countries
27.
HEE’s mandate gives it a role in student selection policy within England, but
not the other three countries.
What equality and diversity considerations relate to this issue
28.
This paper focuses on widening access to medicine in terms of socioeconomic
status, which is not a protected characteristic under the Equality Act. Medicine has
made progress on widening access in terms of gender and black and minority ethnic
groups, although it should be added that there is still more to be done, e.g. black
men are known to be under represented. There are also issues around disabled
applicants to medical school which are picked up in our separate work on health and
disability in medical education and training.
If you have any questions about this paper please contact: Martin Hart,
Assistant Director, Education and Standards, [email protected],
020 7189 5408.
5
5 – Student selection next steps
Annex A
Flow charts summarising variation in student selection
processes used by different medical schools
A1
Full 5/6 year programmes
A2
A3
A4
A5
A6
A7
4 year graduate entry programmes
A8
A9
A10
A11
A12
5 – Student selection next steps
Annex B
Summary of literature review report: I dentifying good
practice in the selection of m edical students
Introduction
1.
The review set out to identify and review evidence on the effectiveness of
methods used by medical schools to select students, and to collate evidence on the
effectiveness of widening access initiatives to promote fair access to medicine, both
in the UK and internationally.
2.
The study comprises a rapid review of literature (adopting a ‘realist’
approach) and a telephone survey of admissions deans. The search was limited to
the last 15 years and generated 150 papers for student selection and 44 for
widening access.
General findings and observations
3.
Existing selection processes (use of academic records, personal statements,
references and in some cases, performance on an aptitude test) have been criticised
for lacking reliability and incremental validity (the extent to which each component
adds weight to the predictive power of the selection decision) as well as inviting
bias.
4.
Assessing the validity of selection methods is hampered by ongoing confusion
as to what medical schools are selecting for - should it be a good medical student or
a good doctor?
5.
Studies in this field therefore face two key challenges:
a.
defining and accessing relevant outcome data to validate selection (this
has typically focused on medical school examinations to date)
b.
and related to this, that the criteria used to measure job performance
might not match the criteria used for selection.
B1
6.
In general, the evidence for multiple mini interviews (MMIs), aptitude testing,
situational judgement tests (SJTs), selection centres and academic record is better
overall than for traditional interviews, references and autobiographical submissions
(eg personal statements) – however, the evidence base is mixed, and where
evidence of predictive validity is presented, this typically relates to medical school
performance.
7.
The evidence base for widening access is weak and restricted to either
studies that are small scale in nature (and therefore lacking generalisability) or lack
robust evaluation (relying more on perceptions of utility rather than data on
performance).
8.
Student selection approaches, and the weighting used for each component
within these, is likely to impact upon widening access – exactly how needs to be
further explored but initial findings suggest that a greater focus on aptitude tests (eg
UKCAT) may have a positive effect.
9.
Comparatively fewer interventions tend to focus on supporting widening
access students once selected. Further research is required to determine how such
individuals can be best supported (whilst avoiding any sense of stigma).
Evidence – student selection
Aptitude tests
10.
There is some evidence to suggest that aptitude tests (eg UKCAT) have
incremental value over A-levels, however reliability data is lacking and suggests the
need for further research. Cost and lack of robust predictive validity data have led to
questions over its utility in some quarters, although this is countered by evidence
which suggests its use can have a positive impact on widening access.
Academic records
11.
Whilst academic records (eg A-levels) provide a stronger indication of medical
school performance than aptitude tests, and tend also to correlate more strongly
with drop-out rates, career progression and postgraduate examination, there is only
a weak association with subsequent performance in medical practice. Furthermore,
evidence suggests that the use of such records introduces a significant socioeconomic class bias.
Personal statements and references
12.
Personal statements and references are deemed to have only limited use in
predicting performance (due to the lack of standardisation and, in some cases,
limited objectivity)
B2
Situational judgement tests
13.
If constructed in accordance with principles of best practice in psychometrics,
SJTs offer a reliable and robust model for selection with some evidence of predictive
validity for medical school performance – with this effect increasing in strength as
individuals progress through their training. SJTs are also thought to have a lower
level of adverse impact on minority groups (compared to tests of cognitive ability eg
aptitude tests).
Personality assessments
14.
There is some evidence to suggest that behavioural traits may correlate with
medical school performance (eg conscientiousness). Whilst there is clear value in
developing a means of measuring desirable personal and interpersonal skills in
medical applicants, initial studies on the use of emotional intelligence tests are
inconclusive (and such testing may reduce the diversity of medical student
personalities which in turn may influence specialty distribution).
Traditional interviews and multiple mini-interviews
15.
Only limited evidence is available to support the use of traditional interviews
in student selection – one of the key concerns being the lack of reliability, and also
lack of evidence for predictive validity – primarily due to the lack of consensus over
which non-academic domains are most critical for a medical student (or future
doctor) and how such domains should be assessed. However, it should be noted that
interviews also have a potential role in screening out potential ‘undesirables’.
16.
The specific skills and attributes deemed appropriate for students and doctors
is much debated – and therefore the researchers assert that different schools tend
to focus on different characteristics. A job analysis may address this issue but given
the diverse range of medical careers, creating a core set of attributes is not without
challenge.
17.
The researchers were unable to find any study looking at the relationship
between either traditional interviews or MMIs and professional outcomes. However,
some limited evidence was presented of predictive validity for MMIs (for which they
were deemed the ‘best’ predictor of OSCE performance, clerkship ratings and clinical
aspects of the Canadian Licensure Examination). Furthermore, MMIs have the
benefit of an increased level of reliability over traditional interviews.
Evidence - widening access
The evidence base for this area is limited – with very few robust studies
(capable of generalisation) assessing the effectiveness of specific interventions. As
such, there is no direct indication of the value-added to the medical profession or
patient care of widening access initiatives – and given the importance attached to
performance in league tables for academic institutions (which may be brought down
by admitting students with lower academic achievement), incentives for widening
access remain few in number.
18.
B3
19.
The majority of interventions tend to focus on preparing individuals (through
outreach and mentoring) for medical school and supporting them in their application
– with relatively fewer focusing on the actual student selection process and the
provision of support to individuals throughout their education and training.
20.
More generally, how medical schools weight their selection process is likely to
influence which applicants get into each school, and may influence which schools
individuals apply to. At present, existing weighting systems lack transparency with
individuals often unaware of how their performance at each stage of the process is
likely to contribute to their chance of success.
21.
There is some evidence to support the use of aptitude tests (eg UKCAT) to
promote widening access through using the test to allocate interviews (as opposed
to academic records).
22.
Contextual data has the potential to facilitate widening access (although there
is a paucity of published academic literature evaluating outcomes of its use) –
however, many schools are struggling to know how to best use this, in part a
reflection of continuing confusion over the purpose of selection (is it to identify the
best doctor, the best student or to screen out undesirables?).
23.
The researchers identified some evidence (although this was limited) that
non-traditional students may struggle with both financial issues and medical student
socialisation as a result of divergence from their familial and social norms.
24.
A small scale study of six medical schools indicated that WA students were
more likely to perform better in those schools with a friendly institutional climate,
with support provided to first year students, an emphasis on formative assessment,
recognition that learning has a social dimension, and a student-centred attitude.
25.
The researchers conclude that the needs of widening access students are
likely to be different to more traditional students and therefore further research is
required to explore how these students can be best supported.
B4
5 – Student selection next steps
Annex C
Selecting for excellence group terms of reference
Purpose
The primary motivation of the group is to widen participation to medicine and to
promote excellence in selection.
It will be established by the Medical Schools Council to make rapid progress in
supporting aspiration amongst the lower socio-economic groups and increasing the
number of successful applicants from such groups on to medical degree
programmes. It will also provide an evidence based overview on issues surrounding
the selection of medical students. Thus it will ensure that medical schools choose
those people from the widest possible spectrum of society with the potential to
become the very best doctors of the future.
Work streams
The group will need to look at the recommendations from the GMC’s Selection
Literature Review as well as the two reports on widening participation from Alan
Milburn and evidence from the Shape of Training Review. Some of the specific points
for consideration are;
•
Widening participation – how can medical schools ensure they pick the
best candidates from a range of different socioeconomic backgrounds? How
can medical schools encourage and support applicants from lower socioeconomic groups to apply to medicine? Do different selection methods have
an impact on the success rates of candidates from lower socio-economic
backgrounds? Is there scope to develop a national programme of outreach
activities to encourage pupils from lower socio-economic backgrounds to
apply to medical school?
•
The role of the doctor – when medical schools are selecting potential
students they need to know what they are selecting for. What is a good
doctor and how do you select people with the potential to become one? What
C1
do patients expect of doctors now and what will their needs be in the future?
What are the values and attributes of a good doctor? How can Good Medical
Practise be used in selection?
•
Selection methods – the GMC’s literature review looked at the evidence
base for different methods of selection. Does more research need to take
place to validate different methods? What are the pros and cons of medical
schools using the same techniques to select students? Should medical schools
co-operate more on selection?
•
Evidence base – how do we evaluate selection methods? Is there good
practice outside the UK we might learn from? What longitudinal evidence can
be utilised to help decide which selection methods are effective? Is there a
minimum academic standard for medicine and how can this be tested? How
do you evaluate the effectiveness of widening participation activities?
These work streams all relate to each other and the group will need to draw
together the connections between them. Widening participation will be a central
consideration.
Working methods
This group will be chaired by Professor Tony Weetman. The MSC will provide the
administrative support for the meetings. Medical Schools will provide a full time
member of staff for 12 months. The group will meet at least three monthly for the
first year and after that a view will be taken by group members as to the frequency
of future meetings. The group will meet in person at the MSC offices in London and
in the DAs as deemed beneficial.
The group will be able to set up sub groups to look in more detail at specific issues
when needed. These sub groups will report to the main working group who will
maintain an oversight of all the different work streams.
The working group will be able to commission research further to develop
the evidence base on selection. It will also be able to commission the MSC to run
evidence gathering exercises such as focus groups and surveys.
The working group will make regular reports to MSC Council who will also sign off on
any recommendations the group makes.
Membership
Core membership will be drawn from the following organisations:
•
MSC
•
GMC
•
HEE
C2
•
Department of Health
•
Department for Education
•
HEFCE
•
BIS
•
Commission for Social Mobility and Child Poverty
•
BMA Medical Students Committee
•
NHS Employers
•
Office for Fair Access
•
Brightside Trust.
In addition the group will want to seek specific advice and input from other bodies,
including the following groups:
•
QAA
•
UKFPO
•
COPMeD
•
AoMRC
•
UKCAT
•
BMAT
•
GAMSAT
•
UCAS
•
OFSTED
•
•
National Association of Schoolmasters Union of Women Teachers
(NASUWT)
•
National Association of Head Teachers
•
National Association of Careers and Guidance Teachers
•
National Union of Teachers
C3
•
National Union of Students
•
Mumsnet
•
Royal College of Nursing and other representatives of the allied health
professions
•
Careers England
•
LETBs
•
Law Society
•
Royal College of Veterinary Science.
C4