THE POWER OF QUIZZING
Mark A. McDaniel
Washington University
NCAT Meeting, March 24, 2009
Kaplan Test Prep and Admissions
Survey
Eat Alaskan salmon for breakfast
Watch back-to-back episodes of ER
Wear Superman underwear
• Laboratory research: Taking a test can serve as a
learning opportunity, and enhance memory retention
even to a greater extent than additional studying…
the “testing effect”
• Central implication for course redesigns:
In college classes, low or no stakes quizzes (tests) can
be used to enhance student learning (quizzes not just a
tool to reveal student mastery)
Wildman, McDaniel & Anderson
(2009)
In WEB-based Brain and Behavior university course, each week
students covered one chapter (40 pages)
14 central facts from chapter presented to students in one of
three ways (as an assignment):
Read the facts to review
Multiple choice quiz on the facts
Short answer (fill in blank) quiz on the facts
14 other central facts not presented
Summative Assessments (course exams): 84 Item Multiple
Choice Exams over Three Chapters
Experiment 1
1.
Summative tests manipulated match between quiz stem and test stem:
Quiz Stem: Touch receptors (for nocioception and hapsis) on the skin have
their cell bodies in ____________ and connect in the spinal cord to
ascending spinothalamic cells.
Exam Stem:
Same Touch receptors (for nocioception and hapsis) on the skin have
their cell bodies in ____________ and connect in the spinal cord to
ascending spinothalamic cells.
A. dorsal root ganglia, B…., C…, D…..
Different Touch receptors (for nocioception and hapsis) on the skin have
their cell bodies in the dorsal root ganglia and connect in the spinal cord
to ___________.
A……, B……,C……, D. ascending spinothalamic cells
2. Quizzes presented on web
Ungraded
Correct answer feedback provided
3. Repeated quizzing: students required to take quizzes
four times. (Maximum credit for quiz taking was earned
only by taking quizzes four times)
Unit Test Performance
SA = Short answer quiz, MC = Multiple choice quiz,
QR = reading, NX = no-exposure
Experim ent 1
Different Stem s
n = 16
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0.76
0.76
0.71
0.68
SA
MC
QR
NX
SA = Short answer quiz, MC = Multiple choice quiz,
QR = reading, NX = no-exposure
Experim ent 1
Sam e Stem s
n = 16
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0.87
SA
0.85
MC
0.71
0.68
QR
NX
Experiment 2
Can These Robust Test-Enhanced Learning Effects Be
Replicated and Extended to condensed course format?
• Condensed summer-school course (7 weeks)
Individual students could start late and further condense the course
(essentially complete course in 3-4 weeks)
• Similar to Experiment 1
Summative tests manipulated match between quiz stem and test stem:
Different or Same
Students required to take quizzes four times (Maximum credit for quiz
taking was earned only by taking quizzes four times)
Unit Test Performance
SA = Short answer quiz, MC = Multiple choice quiz,
QR = reading, NX = no-exposure
Experim ent 2
Different Stem s
n = 27
1
0.8
0.6
0.4
0.75
0.75
0.76
SA
MC
QR
0.65
0.2
0
NX
SA = Short answer quiz, MC = Multiple choice quiz,
QR = reading, NX = no-exposure
Experim ent 2
Sam e Stem s
n= 27
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0.90
SA
0.92
0.76
MC
QR
0.65
NX
Summary
• Repeated quizzing produces robust learning
effects on classroom summative assessments
• Multiple choice quizzes are as effective as
short-answer quizzes, when summative
assessment is multiple choice and quizzes
repeated
• With same stems on quizzes and final test,
quizzing produces better performance than
presenting target content for additional study
• Can quizzing enhance richer learning of
conceptual information (application of
content)?
• Will application quiz items promote
learning of related factual content?
Short-answer quizzes (web-based course)
Factual
What is the most complex human behavior?
Applied
If an otherwise normal child does not start
talking until after his peers, what do we
expect about how long it might take this child
to learn to speak fluently?
Performance on Exam Multiple Choice Questions
as a Function of Quiz Type
1
Exam Question Applied
Proportion Correct at Exam
0.9
Exam Question Factual
0.8
0.7
0.6
0.5
0.4
0.84
0.74
0.65
0.3
0.64
0.2
0.1
0
Applied
Factual
Quiz Type Question
None
Performance on Exam Multiple Choice Questions
as a Function of Quiz Type
1
Exam Question Applied
Proportion Correct at Exam
0.9
Exam Question Factual
0.8
0.7
0.6
0.5
0.4
0.84
0.74
0.81
0.72
0.3
0.65
0.64
0.2
0.1
0
Applied
Factual
Quiz Type Question
None
Summary
• Quizzing (short-answer) can promote
transfer—it enhances performance on
related questions
Does Test-Enhanced Learning
Extend to Typical Lecture Courses
and Very Able Learners?
• Washington University Introductory Psych
Class (summer)
• Intervening quizzing:
short answer quizzed
not quizzed
• Exam question types: Multiple Choice,
Short Answer
McDaniel and Sun
(2008)
Daily
QUIZ SENT BY
E-Mail (2/week)
Class
lecture;
12 Short
answer items
reading
assignment
(12 Non-quiz
items)
N=21
UNIT EXAM
End of
Week
Mult. choice (16): 8 same wording and 8
different wording from quiz
Short answer (8): 4 same wording and 4
different wording from quiz
Proportion Correctly
Recalled
Proportion correct on exams as function of
quizzing, exam question type, and match of quiz
and exam question stems
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Quizzed
Not Quizzed
Same
Different
MC
Same
SA
Different
IS TEST-ENHANCED LEARNING
PERSISTENT OVER LONG DELAYS?
Columbia Middle School project
• 6th, 7th, and 8th grade science classes
• Some facts on each lesson are quizzed 3
times; comparable facts are not quizzed
• Consider recall on tests at ends of
chapters, end of the semester and
end of the year
COLUMBIA MIDDLE SCHOOL PROJECT
8TH GRADE SCIENCE
1
Tested
Non-Tested
0.91
Proportion Recalled
0.9
0.8
0.76
0.74
0.69
0.7
0.68
0.61
0.6
0.5
Chapter Exams
End of the Semester
End of the School Year
• Can learner initiated quizzes (retrieval)
enhance learning (when course
constraints preclude any quizzing)?
Learner controlled testing
• Read (study)
• Recite (recall material to oneself)
• Review (reread the material)
McDaniel, Howard, & Einstein
(2009, Exp. 2)
• Three study conditions:
– Read, recite, review
– Note-taking
– Read-twice control
• Each participant read two complex, ~900
word articles
– Brakes & Pumps, as in Mayer & Gallini (1990)
• Students were tested on one of the articles
immediately
• After a one-week delay, students were
tested on both articles
• Testing consisted of:
– Free recall
– Multiple choice (4 alternative)
– Short-answer inference questions
**
**
**
**
*
*
*
*
Summary
• Participants who used the 3R strategy
performed significantly better on a range
of classroom type tests relative to reading
twice or note-taking (excluding immediate
inference performance)
• 3R more efficient (takes less time) than
note-taking for enhancing problem
solving/inference
Summary of major findings
• Basic test-enhanced learning
demonstrated in college courses and
classrooms
• Found even when learning at high
levels.
• Similar learning benefits were not
always found from additional exposure
per se
• Self-testing is effective for learning
Many reasons to expect tests to confer learning
benefits in educational contexts
Retrieval itself produces retrieval practice
Test itself requires student to engage in the
material (Sikorsky et al., 2001)
Signals information to target for study.
(improves metamemory—both accuracy of
what you know and control of study)
Leads to more (and spaced) studying—keep up
with the material better (Johnson &
Kiviniemi, 2009)
Acknowledgements
Funding
Institute of Education Sciences
Grant #R305H030339
James S McDonnell Collaborative Activity
Grant