Key Literature in Medical Education Podcast Review
Reviewer: Jonathan Sherbino
Date broadcast:
The overall impact of testing on medical student learning: quantitative estimation of consequential validity
Reference:
Kreiter CD1 2, Green J1, Lenoch S2, Saiki T3. The overall impact of testing on medical student learning:
quantitative estimation of consequential validity. Advances in Health Sciences Education; Epub (ahead of
print):1-10.
Author Institutions:
1
University of Tokyo, Bunkyo, Japan
2
University of Iowa, Iowa City, IA, USA.
3
Gifu University, Gifu, Japan.
PubMed URL
Tags
Clinical domain
Medical Expert
Educational domain
Assessment
Background
The 2010 Ottawa conference recommended criteria for good assessment
(PMID: 21345060) Included among the consensus statements was the concept that
assessment has both a formative effect (e.g. feedback as a result of testing
influences future learning) and educational effect (e.g. anticipation of testing
promotes learning) on learners. Certainly assessment is an key element of many
curricula if adjudicated on the basis of the frequency of journal manuscripts,
conference presentations and textbook chapters on the topic. However, what is the
actual evidence of the effect of testing on learning?
Purpose
This study attempts to systematically analyze the effect of testing on learning in
three domains (described below).
Type of paper
Meta-analysis
Key Points on the Methods
This is a “quasi” systematic review. There is a minimal description of the search
strategy, article selection, quality scoring etc. None of the PRISMA standards are
adequately addressed. Generalization of results from undergraduate studies to
medical education may be inappropriate.
In addition, to determine a “total” effect size, an ad hoc approach of converting
individual effect sizes (d scores) into an additive form (eta-squared) and then
reconverted to a “total” effect size was used. The statistical implications of this
approach are beyond the scope of this review.
Key Outcomes
•
•
•
•
Direct effect of testing: (a test’s intrinsic influence on students long-term
retention of tested material)
o In 2 medical education studies there was a large effect size that
favoured study + testing versus study alone; while in a third
medical education study there was non statistically significant
difference.
o In 5 undergraduate studies the effect size ranged from -0.13 to
3.08, with an average effect size of .99.
Indirect effect of testing: (a test’s extrinsic influence on students and an
education system to promote accountability mechanisms that provide high
quality education experiences)
• There are no (quantitative) medical education studies that
determine the effect of summative assessments on student
learning. There are a number of self-reported qualitative studies.
• Two studies of undergraduate students revealed moderate to large
effect sizes.
Selection effect: (an admission test’s extrinsic influence on an education
system to predict which applicants are most likely to excel)
• Two meta-analyses (52 studies in total) demonstrated a large
effect size of the MCAT on learning.
• One study of a national sample of medical schools also
demonstrated a large effect size of the MCAT on learning.
• A fourth study using a written admission test (before-after design)
shows, again, a large effect size for admission tests on learning.
Overall effect:
• Using a complex statistical process, the authors suggest that the
summative effect of all elements of testing on medical student
learning is large (d = 2.61; i.e. greater than 2.5 SD)
Key Conclusions
The authors conclude…
“The literature is far from complete, but the available evidence strongly suggests that each of these effects is large and the net cumulative impact on learning in medical education is over two standard deviations. While additional evidence is required, the current literature shows that testing within medical education makes a strong positive contribution to learning.”
Spare Keys – other take home points for clinician
educators
This paper reminds us that medical education is best served by analyzing (to support
or refute) the core tenets of the field. It may be surprising to many listeners that it
took until 2012 for an article to be published that addressed the evidence for the
effect of testing on learning.