Overview of Assessment
ESME 2006
Stewart Mennin, PhD
 Mennin Consulting, 2006
Objectives





Make more informed decisions about
assessment
Apply information to your teaching
Describe the relationship between
assessment & learning
Discuss criteria for effective assessment
Define & apply basic terminology
Mennin Consulting, 2006
ESME
Objectives




Describe selection, supply & performance
methods
Link Miller’s pyramid to assessment methods
Match assessment methods to your
outcomes
Draft assessment plan (blueprint)for your
teaching
Mennin Consulting, 2006
ESME
GOALS OF ASSESSMENT





Enhance student learning
Guide faculty teaching efforts
Inform curricular development & revision
Fulfill responsibilities to students, institution &
society
Contribute to research
Mennin Consulting, 2006
ESME
FEATURES OF ASSESSMENT
 Assessment
is more a curriculum design
problem than a measurement problem
 Part of learning & can not be separate from it
 Assessment is about the present, past and
future of learning
 Assessment in not a collection of instruments
 Assessment should be authentic, grounded in
what is real
Mennin Consulting, 2006
ESME
EIGHT CRITERIA: EFFECTIVE
ASSESSMENT
1.
2.
3.
4.
Articulation of a clear conception of
intended learning outcomes
Use variety of assessment
procedures/methods
Close agreement between intended
outcomes of instruction, learning tasks, &
the assessment procedures/methods
Adequate sampling of student performance
Gronlund, 2000
Mennin Consulting, 2006
ESME
EIGHT CRITERIA: EFFECTIVE
ASSESSMENT
5.
6.
7.
8.
Procedures that are equitable for all
participants
Explicit, specific criteria for use in judging
successful performance
Timely feedback that emphasizes strengths of
performance & focuses attention on specific
areas in need of improvement
A grading & reporting system consistent with
the rest of the learning & assessment process
Mennin Consulting, 2006
ESME
Key Terminology




Formative – Summative
Reliability & Validity
Criterion Referenced --- Norm referenced
Progress test
Sequential test
Mennin Consulting, 2006
ESME
Assessment As Learning
Formative
Purpose Improvement
Timing Frequent & throughout
Evaluator Teacher & learner
Use Feedback to student & learner
Find misconceptions
Identify strengths &
weaknesses
Change future teaching
Develop learning plan
Mennin Consulting, 2006
Summative
Decisions / Judgment
Milestones & end
Teacher
Assign grades
Judge competence
Certify mastery
Promotion
ESME
Reliability


Consistency of results
Reliability is matter of sampling
–
Across content
Mennin Consulting, 2006
ESME
Domain of Interest
Test Sample
Test Sample
Mennin Consulting, 2006


ESME
Competent
Assessment of Clinical
Competence is a Issue of
Specificity
Mennin Consulting, 2006
ESME
Reliability

Practical suggestions:
–
Do not rely on short tests
–
Sample broadly (content, in time, examiners, patients)
–
Consider efficiency in:

selection of test format

construction test items
Mennin Consulting, 2006
ESME
Reliability

Adequate reliability requires substantial
sampling (therefore: resources, testing time,
examiners, patients, etc.)
– Be efficient
Mennin Consulting, 2006
ESME
Professional authenticity
Validity
Does
Shows how
Knows how
Knows
Miller GE. The assessment of clinical skills/competence/performance.
Academic Medicine (Supplement) 1990; 65: S63-S7.
Mennin Consulting, 2006
ESME
Professional authenticity
Validity
Does
Performance
or hands on
assessment
Shows how
Knows how
Knows
Written,
Oral or
Computer
based
assessment
Miller GE. The assessment of clinical skills/competence/performance.
Academic Medicine (Supplement) 1990; 65: S63-S7.
Mennin Consulting, 2006
ESME
Valid Inferences

Validity is a single concept with different
forms of evidence
–
–
–
Content-related evidence
Criterion-related evidence
Construct-related evidence
Mennin Consulting, 2006
ESME
ASSESSMENT METHODS

All assessment methods have
limitations and no one method can
assess all types of knowledge & skill

Multiple methods are recommended
Mennin Consulting, 2006
ESME
Some Assessment Options:

Selection Item
–
–
–

Supply Items
–
–
–

Multiple Choice Question
One-Best Answer Questions
Extended-Matching Questions
Free Response Questions
Short Answer Questions
Essay Questions
Performance Examinations
–
–
–
Laboratory Practical Examinations (Identification and Concepts)
Clinical Skills (e.g., OSCE)
Other
Mennin Consulting, 2006
ESME
Selection Methods





MCQ
Matching
Extended Matching
True False
Multiple True False
Does
Shows how
Knows how
Knows
Mennin Consulting, 2006
ESME
Does
Knows/Knows how
Shows how
Knows how
Knows
Knows

The stimulus format is more
important than the response
format
Mennin Consulting, 2006
ESME
How Well Written?
Stem
a. Poor
b. Good
Mennin Consulting, 2006
ESME
How Well Written?
Stem
a.
b.
c.
d.
a. Poor
b. Good
Mennin Consulting, 2006
ESME
How Well Written?
30-year-old man has loss of pain and temperature
sensation from the neck down on the right side of the
body and on the left side of the face; partial paralysis of
the soft palate, larynx, and pharynx on the left; and
ataxia on the left. This syndrome is most likely to result
from thrombosis of which of the following arteries?
A Basilar
B Right posterior inferior cerebellar
C Left posterior inferior cerebellar
D Right superior cerebellar
E Left superior cerebellar
Mennin Consulting, 2006
ESME
Extended Matching
Authors on Education:
a.
D. J. Boorsten
e.
Lao-Tzu
b.
Camus
f.
S. Scott Obenshain
c.
Albert Einstein
g.
John Pieper
d.
Thomas Harris
h.
Mao Tse-Tung
For each quotation, select the most likely author.
1. I never teach my pupils. I only attempt to provide the conditions under
which they can learn.
2. All genuine knowledge originates in direct experience.
3. To know one’s ignorance is the best part of knowledge.
Mennin Consulting, 2006
ESME
Extended Matching Answers
Authors on Education:
a. D. J. Boorsten
e.
b. Camus
f.
c. Albert Einstein
g.
d. Thomas Harris
h.
Lao-Tzu
S. Scott Obenshain
John Pieper
Mao Tse-Tung
1. I never teach my pupils. I only attempt to provide the conditions under which
they can learn. (c)
2. All genuine knowledge originates in direct experience. (h)
3. To know one’s ignorance is the best part of knowledge. (e)
Mennin Consulting, 2006
ESME
Supply Methods





MEQ
Essay (long & short)
Practical
Short Answer
Written simulations
Does
Shows how
Knows how
Knows
Mennin Consulting, 2006
ESME
Performance Assessment









Procedures – skills lab
OSCE (with skills stations)
Mini CEX
Long Case
Logs
Portfolio
Clinical Notes
Observation
Other
Mennin Consulting, 2006
Does
Shows how
Knows how
Knows
ESME
Interviewing & Communication Skills
Communication Skills
Mennin Consulting, 2006
ESME
Essential Elements of Communication
Skills







Open the Discussion
Build a Relationship
Gather Information
Understand the Patient’s Perspective
Share Information
Reach Agreement
Provide Closure
Acad Med 2001;76(4):390-393
Mennin Consulting, 2006
ESME
Organigramtitel
Clinical
Clinical
Problem
Scenario
Key Feature Approach
Testing time
Test using
integral cases
Test using
key features
Case
2
Case
1
Case Case
1
2
Case
3
Case
4
Case
5
Case
6
Case
7
Does
Shows How: OSCE
Shows how
Knows how
Knows



Make stations as clinically
authentic as possible
Global judgments do well in
OSCEs
Content specificity is the problem
Mennin Consulting, 2006
ESME
Checklist/Global Rating Scale
Comparison

Checklists
–
–
Rating Scales
Dichotomous scores
(yes/no)
Relatively objective
–
Record rather than
interpret
–

–

–
Ordinal level measures
(poor, good, excellent)
Relatively subjective
Station independent
Station specific
Mennin Consulting, 2006
ESME
Does
Does: Practice Assessment
Shows how
Knows how
Knows

Indirect methods
–
–
–

Chart audit
Practice audits
Prescription data………
Direct methods
–
–
–
–
–
–
–
Clinical (supervisor) ratings
Undercover standardized patients
Video assessment
Patient reports
Peer reports
Clinical work samples
Mini-CEX………
Mennin Consulting, 2006
ESME
Does
Clinical Work Samples
Shows how
Knows how
Knows

Repeated direct observations of clinical
performance in practice using (generic)
evaluation forms, completed by any
significant observer (clinician, nurse, peer…..)

Illustration
Mennin Consulting, 2006
ESME
Does
Mini Clinical Examination
(Norcini, 1995) Shows how
Knows how
Knows





Short observation during clinical patient
contact (10-20 minutes)
Oral evaluation
Generic evaluation forms completed
Repeated at least 4 times by different
examiners
(cf. http://www.abim.org/minicex/)
Mennin Consulting, 2006
ESME
NEW SKILLS EMPHASIZED





Learning how to learn
Self-assessment
Humanization
Leadership
Team skills
Mennin Consulting, 2006




Skills of expression
Professional behavior
Reflection
Permanent education
ESME
Methods & Knowledge
Written Tests
Written Exercises
Patient Exercises
Observations
Essay
MCQ
Application Test
MEQ
Practical
OSCE
SPs
Triple Jump
Mini CEX
Long Case
Direct Observation
Med Record Review
Oral Examination
Peer Evaluation
Rating Scale
Mennin Consulting, 2006
Knows
A
A
B
B
Knows How
A
A
A
A
B
B
B
B
A = Most effective
B = Less effective
Shows How
Does
A
A
A
A
A
A
B
B
B
B
B
B
B
A
A
A
A
A
ESME
Task

Use the table from effective teaching and
match assessment methods to your outcome
competencies
Mennin Consulting, 2006
ESME
Faculty of Medicine Maastricht University
Progress Test
Scores on one progress test
Mennin Consulting, 2006
ESME
Faculty of Medicine Maastricht University
Progress Test
Maastricht Test Results ‘77 - ‘88
Mean correct scores on the Progress Test from a number of Maastricht student
cohorts (black lines) and reference group of family physicians in training (blue line)
Mennin Consulting, 2006
ESME
Planning: Blueprinting




Competencies
Time
Teaching methods
In class -- out of class
Mennin Consulting, 2006
ESME
Methods
Select
Competencies
Clinical
Skills
Critical
Int Know
Comm
Human
Self
Assess
Profess
Supply
Perform-1
Competent
Perform-2
Capable
Conclusions

Assessment is less of a psychometric problem, but
rather an educational design problem (i.e. how to use
assessment strategically for its educational effects)


Assessment requires careful planning and monitoring
To effective with the entire competence pyramid
–
a mix of methods that depends on the context; not a single method
is best or bad
Cees van der Vleuten
Mennin Consulting, 2006
ESME
Acknowledgements





Cees van der Vleuten - Maastrict
Deana Richter – New Mexico
Teresita McCarty - New Mexico
Nancy Sinclair - New Mexico
Regina Petroni Mennin – UNIFESP
Mennin Consulting, 2006
ESME
Presentations Available at:
www.menninconsulting.com
[email protected]
[email protected]
Mennin Consulting, 2006
ESME