6/24/2013
Test Item Writing Basics
Judy Westman MD
May 16, 2013
Objectives • Compare and contrast test item stems by vignette format, shapeliness, and level of focus.
• Identify the characteristics of choices which Identify the characteristics of choices which
promote assessment versus reading skill and which reduce the likelihood of answer cueing.
• Discuss the use of item analysis in improving assessments and items.
Primary sources
NBME: Constructing Written Test Questions, 3rd ed.. 2002
http://www.nbme.org/PDF/ItemWriting_2003/2003I
WGwhole.pdf
www.nbme.org
Online item writing tutorial
http://download.usmle.org/IWTutorial/intro.htm
1
6/24/2013
Purpose of testing
• Communicate to students what material is important
• Motivate students to study
Identify areas of deficiency in need of
• Identify areas of deficiency in need of remediation or further learning
• Determine final grades or make promotion decisions
• Identify areas where the course/curriculum is weak
NBME: Constructing Written Test Questions, 2002
What should be tested?
• Exam content should match course/clerkship objectives
• Important topics should be weighted more heavily than less important topics
heavily than less important topics
• The testing time devoted to each topic should reflect the relative importance of the topic
• The sample of items should be representative of the instructional goals
NBME: Constructing Written Test Questions, 2002
Desired structure of MCQ (USMLE)
• Stem based on clinical or research vignette
• Clear focus on an important concept
• Promote critical reasoning rather than simple recall of facts
• Stem ends with a question
S
d ih
i
• Potential to answer question without seeing choices
• Choices alphabetized to enable rapid finding of answer if known
• Relatively short choices; plausible, homogeneous
2
6/24/2013
Undesirable MCQs: banned at USMLE and OSU
• “Except” or “not” questions
– “Which of the following is NOT correct?”
• “None of the above”
• “All of the above”
• Key questions: Key questions:
–
–
–
–
“1,2,3 are correct”
“2 and 4 are correct”
“1 and 3 are correct”
“only 4 is correct”
• Guess what the faculty is thinking questions
– “Calcium signaling is:” followed by lengthy choices
Less desired features may still be present at OSU (banned at USMLE):
• Multiple T/F format
– “Which of the following is a correct statement about cystic fibrosis?”
• Factoids
acto ds
– “Triglycerides are broken down into free fatty acids and which one of the following?”
• USMLE concentrates on critical reasoning
• OSU needs to assure mastery level of some facts as well as increasing degrees of critical reasoning
Vignette
• Helps focus on important concept
• Helps assess application of knowledge, not recall of an isolated fact
– Two step thought process
Two step thought process
• Write clinical vignettes as a clinical history: presenting problem, history, physical findings, diagnostic studies, initial treatment, course, etc
– Do not have to include every portion of history
3
6/24/2013
“Shapely” item
• Stem should generally be long and choices should be short (one line). STEM: Clinical or research vignette
Lead in: Which of the following metabolic pathways is most likely defective? Choice A
Choice B
Choice C
Choice D
Choice E
Cover/uncover test
• Can you answer the question by covering up the choices?
• Try these:
1. True statements about cystic fibrosis include:
1
1.
Choices
Choices…
2. In the clinical assessment of chronic pain,
1.
Choices…
3. A 58‐year‐old man with a history of heavy alcohol use and previous psychiatric hospitalization is confused and agitated. He speaks of experiencing the world as unreal. What is this symptom called?
1.
Choices…
One‐best‐answer vs multiple T/F
One‐best‐answer
• May satisfy cover/uncover test
• May have focused lead in
• Tolerates shades of gray in responses
• “Most likely diagnosis”
• Ranking best to worst answer
• Options must be homogeneous in nature or absolutely 100% T or F
• More valid assessment
Multiple T/F
• Do not satisfy cover/uncover
• Unfocused lead in
• Options must be completely O ti
tb
l t l
true or false; no shades of gray permissible
• Avoid “is associated with”, “is useful for”, “is important”, “may”, “could be”, “usually”, “frequently”
4
6/24/2013
Homogeneous options
• All in the same category as the correct answer
• Avoid using double options unless the correct answer and all distractors are double options
• All should be plausible
ll h ld b l ibl
• All should be grammatically consistent
• All should be logically compatible
• All should be of the same relative length as the correct answer
Example
• Who was the primary author of the Declaration of Independence?
– Abraham Lincoln
– Thomas Jefferson
Thomas Jefferson
– Franklin Roosevelt
– King George II
– Catherine the Great
Example
• Who was the primary author of the Declaration of Independence?
– George Washington
– Thomas Jefferson
Thomas Jefferson
– Alexander Hamilton
– Benjamin Franklin
– James Madison
5
6/24/2013
Student approach to one‐best‐answer
• Read the last sentence of the stem first. That’s where the question is.
• Once you know what is being asked, then read the vignette
the vignette.
– Reduces distraction by “red herrings”
– If you can answer the question without reading the vignette, you’ve saved valuable time.
Common cues for testwise students
• Absolute terms (always, never) in a choice
• The longest answer is the correct one
– Faculty are teachers and write long correct answers that include additional instructional
answers that include additional instructional material
• Word included both in the stem and the correct choice
Avoid Irrelevant Difficulty
• The item should test content knowledge, not the learner’s reading skills or speed.
• Use parallel options
Use logical order
• Use logical order
• Alphabetize choices
– Speeds up learners’ ability to find correct answer if they know it
– Prevents item writer bias for placement of correct answer
6
6/24/2013
Example
• Peer review committees in HMOs may move to take action against a physician’s credentials to care for participants of the HMO. There is an associated requirement to assure that the physician receives due process in the course of these activities. Due process must include which of the following?
1.
2.
3.
Notice, an impartial forum, council, a chance to hear and confront evidence against him/her
evidence against him/her
Proper notice, a tribunal empowered to make the decision, a chance to confront witnesses against him/her, and a chance to present evidence in defense
Reasonable and timely notice, impartial panel empowered to make a decision, a chance to hear evidence against himself/herself and to confront witnesses, and the ability to present evidence in defense
120 words  79 words without stem
Other issues
• Avoid 2nd person in stem, lead in, and choices: “What would you do?” – The student can defend any answer given.
– Desired testing measure is “What is the correct g
response?” not what would this specific student do
• Make sure there are no typographical, grammatical, or spelling errors.
– Students see it as an issue of faculty competence
HELP WITH REAL ITEMS
7
6/24/2013
COM uses MCQ item stats
• Differential (p) = % of class who answered correctly
• Compare top quartile of class (as determined by the exam) vs bottom quartile of class
• Discrimination index = % difference between top Discrimination index = % difference between top
and bottom
• Compare responses to different choices
• Used to determine validity of MCQ – Poor stats will cause item to be thrown out
– Use stats to tweak item for future exams
8