Test Construction: Using Best Practices to Construct Quality Items
AACN Faculty Development Conference 2009
Tommie L. Norris, DNS, RN
1
Dance Away Test Construction Blues
Tommie L. Norris, DNS, RN
2
Tommie L. Norris, DNS, RN
3
Teacher Made Exams: Is it Worth the Time?
• Guides remediation
• Validates learning
– Unit exams (formative)
– Final exams (summative if comprehensive)
• Directs Curriculum revision
• Familiarizes learners with nursing exams
• 2008 NCLEX Pass Rates
Tommie L. Norris, DNS, RN
4
Evaluation of Student’s Knowledge
• Most recognized source of assessment is a “test”
– Students demonstrate learning
– Teachers evaluate student performance
• Must be accurate and unbiased
• Remember, tests only provide one measure and only one domain of learning
– Selective (can we test everything we teach?)
Tommie L. Norris, DNS, RN
5
Domains of Learning
• Cognitive
– Most commonly assessed with exam (Bloom’s)
• Presentations; discussion boards
• Psychomotor
– Skills lab; clinical; simulation
• Affective
– “Affective learning outcomes involve attitudes, motivation, and values” (Smith & Ragan, 1999). – Affective has strong influence on cognitive & psychomotor
Tommie L. Norris, DNS, RN
6
Tommie L. Norris, DNS, RN
7
Teacher Made Exams
• Teacher made vs. Standardized
– Keyed to specific objectives
– ?Less expensive
Curriculum Course Objectives Unit Objectives Evaluation Tommie L. Norris, DNS, RN
8
2007 NCLEX‐RN Testplan
• NCSBN
“All registered nurse candidates must answer a minimum of 75 items. The maximum number of items that the candidate may answer is 265 during the allotted six‐hour time period. Examination instructions and all rest breaks are included in the measurement of the time allowed for a candidate to complete the examination”. (www.ncsbn.org.)
Tommie L. Norris, DNS, RN
9
4 Major Client Needs Categories
• Safe & Effective Care Environment
– Management of care 13‐19%
– Safety & Infection control 8‐14%
• Health Promotion and Maintenance 6‐12%
• Psychosocial integrity 6‐12%
• Physiological integrity –
–
–
–
Basic care and comfort 6‐12%
Pharmacological and parental therapies 13‐19%
Reduction of risk potential 13‐19%
Physiological adaptation 11‐17%
Source: 2007 NCLEX‐RN Test Plan NCSBN
Tommie L. Norris, DNS, RN
10
Integrated Processes
• Fundamental to practice of nursing
• Integrated throughout “client needs”
categories/subcategories
– Nursing process
– Caring
– Communication and documentation
– Teaching/learning Source: 2007 NCLEX‐RN Test Plan NCSBN
Tommie L. Norris, DNS, RN
11
Bloom’s Taxonomy: Classification of cognitive levels
• Knowledge‐requires only memorization
– Define, list, match, name, recall (Focus on facts)
• Comprehension‐lowest level of learning or understanding
– Paraphrases
• Identify, restate
• Application‐places in new situation
– Demonstrate, illustrate, predict, use
• Analysis‐break into pieces
– Calculate, contrast, diagram, differentiate
• Synthesis
• Evaluation
Tommie L. Norris, DNS, RN
12
Sample Questions at Cognitive Levels
Tommie L. Norris, DNS, RN
13
Levels
• Knowledge
– The normal pulse rate for an adult ranges from:
• Comprehension
– The nurse checks the patient radial pulse and notes 100 beats per minute (bmp). The nurse realizes this represent which cardiac rhythm?
• Application
– The nurse is having difficulty heart the apical pulse. Using the diagram, click on the spot where the stethoscope should be placed.
• Analysis
– A patient is exhibiting the above rhythm. What is the nurse’s first action?
Tommie L. Norris, DNS, RN
14
Suggestions
• Write questions as you prepare lecture
– Be certain objectives are evaluated
– Level according to Bloom’s taxonomy
– Classify according to nursing process
– Carefully combine questions from previous tests
– Provide rationales as you write questions
– Site references
Tommie L. Norris, DNS, RN
15
Blueprints
• Level of questions=Level of course
• Number of questions based on amount of time spent covering content
• Based on course objectives/outcomes
Tommie L. Norris, DNS, RN
16
Format Items Used on NCLEX
•
•
•
•
•
•
Multiple choice (4 options)
Fill‐in‐the‐blank (dosage)
Multiple‐Multiple or Multiple Response
“Hot Spot” Illustrations
Drag‐and‐Drop (Prioritization/ordered)
Chart/exhibit
Tommie L. Norris, DNS, RN
17
Fill‐In‐The‐Blank
ORDER: dilantin (Phenytoin) 200 mg per PEG every 8 hours. Label: dilantin (Phenytoin) Suspension 125 mg/5mL
How many mL should the nurse administer per dose? ___8__mL/per dose
Tommie L. Norris, DNS, RN
18
Hot Spot
The nurse is caring for a patient with
infarction of the atrioventricular node (AV
node).Identify the area of the conduction
cycle of the heart where the blockage
has occurred.
Tommie L. Norris, DNS, RN
19
Multiple Multiple
• Students must select more than one correct answer and all correct answers must be selected to receive credit.
– Minimum of 5 – Minimum of 2 must be correct
Tommie L. Norris, DNS, RN
20
Example
The nurse caring for a patient with right‐sided heart failure would assess for which of the following expected clinical manifestations?
(Select all that apply)
a.
b.
c.
d.
e.
f.
Paroxysmal nocturnal dyspnea
Productive cough
Jugular venous distension
Weight gain
Hemoptysis
Right upper quadrant pain
Tommie L. Norris, DNS, RN
21
Drag and Drop
• Performing a skill, drag and drop each step in order
• Nurse receives shift report, place order patient would be initially assessed
Tommie L. Norris, DNS, RN
22
Chart Review
Questions shows different sections of chart on same page or by selecting tabs. Student must then determine nursing intervention or diagnosis. May also be asked what information is needed next.
Tommie L. Norris, DNS, RN
23
Multiple Choice Format
Multiple Choice: Stem & Distracters
Advantages
• Tests all cognitive levels
• Helps prepare for standardized test
• Allows evaluation of more content
• Easily scored
Tommie L. Norris, DNS, RN
24
Multiple Choice Format
Disadvantages
– Time consuming to construct
– Must generate several “plausible” alternatives
• Why “plausible”?
Tommie L. Norris, DNS, RN
25
Multiple Choice????
Tommie L. Norris, DNS, RN
26
Include Directions Directions: Unless otherwise indicated, choose the ONE best answer for the following questions. Alternate formatted questions are indicated. Mark your answer on the Scantron
in pencil. All questions are worth 2 points. You have ONE hour to complete this exam. Tommie L. Norris, DNS, RN
27
Writing the “Stem”
–Let’s look at the “stem” first
• Clearly written
• Place test taker in situation
• Include “Nurse”
• Bold key words first, most important, priority
–Require all distracters to be correct
• End in ? Or :
Tommie L. Norris, DNS, RN
28
Writing the Stem
• Write out all abbreviations first time (unless testing)
–Board of Nursing (BON)
• Is task within scope of practice of entry level nurse?
• Length of stem
• Label all values Potassium=5.5 mEq/l
Tommie L. Norris, DNS, RN
29
Writing the Stem –Avoid negatives “Except” “Not”
–Avoid “all of the above” or “none of the above”
–Do not teach in stem
Tommie L. Norris, DNS, RN
30
Writing the Stem –Do not provide clues
–Avoid gender unless gender specific –Avoid scenarios
–Avoid giving patient a name
Tommie L. Norris, DNS, RN
31
Writing the Stem –Avoid “you” change to “nurse” write in 3rd person
–Avoid age unless developmental question
–Avoid stereotyping
Tommie L. Norris, DNS, RN
32
Now the Distracters
• “alternatives” or “distracters”
• Distracters should match the ending of the stem
:
– If stem ends ? distracters are upper case with no period (unless – If stem ends distracters are lower case and end in a period ●
complete sentences)
The “P” wave in the electrocardiogram (EKG) represents:
a. ventricular repolorization.
b. the firing of the atrioventricular node (AV).
c. atrial depolorization.
d. stimulation of the purkinje fibers.
Tommie L. Norris, DNS, RN
33
Grammatically Correct
• Stem should grammatically match all distracters and correct answer/s
George Washington was a:
a. President.
b. Electrician.
c. Acrobat.
d. Ombudsman. Can you identify the problems?
Tommie L. Norris, DNS, RN
34
Distracters Best Practices • Keep distracters the same length
• Don’t use pattern of correct answer “c”
• Don’t sequence content
Tommie L. Norris, DNS, RN
35
Distracters
• Avoid Jargon
• Use both generic and trade name of all medications
Tommie L. Norris, DNS, RN
36
If numerical:
– Place in ascending or descending order
– Keep “ranges” equal
– Don’t overlap values
What is the normal range of serum sodium in the adult?
a. 90‐100 mEq/l
b. 120‐130 mEq/l
c. 135‐145 mEq/l
d. 150‐160 mEq/l
Tommie L. Norris, DNS, RN
37
Opposites & “Alikes”
Do not have distracters the opposite of the correct answer.
–
If two opposite, one must be right
A patient’s heart rate is counted as 140 beats per minute (bpm). This finding would be charted as patient is demonstrating:
a.
b.
c.
d.
Bradycardia.
Tachycardia.
A slow heart rate.
Palpatations.
Tommie L. Norris, DNS, RN
38
Familiar with all Distracters
• When testing “can the student define an item”, be certain they are familiar with all distracters
A patient asks, “Why does cancer seem to grow in some organs like the lungs but not in others such as the heart?”
The nurse bases her answer on the knowledge of:
– a. intravasion.
– b. extravasion.
– c. diapedesis.
– *d. tropism
Tommie L. Norris, DNS, RN
39
Timely Preparation for NCLEX
• Time Limit
– 1 question per minute
• 50 per hour
• 100 per final in two hours
– What about Multiple Choice questions that require calculations?
Tommie L. Norris, DNS, RN
40
Other Types of Questions?
Tommie L. Norris, DNS, RN
41
Created Response Items
• Essay
• Short answers
• Advantages
– Easy to write
– Allow student to demonstrate learning at all cognitive levels
– Encourage students to organize thoughts
– Student’s can’t guess answer
Tommie L. Norris, DNS, RN
42
Matching Items
•
•
•
•
•
•
Series of items using same list of distracters
Longer statements on left
Typically measure knowledge or comprehension Easy to construct and score
All distracters and descriptors should be on one page
In nursing, used as quizzes or “check yourself”
Tommie L. Norris, DNS, RN
43
True‐False
•
•
•
•
Easy to construct and score
Most are knowledge level
50% chance of guessing
Research does indicate T‐F testing is reliable and valid for periodic use
• Require students to correct false statements to increase level
• Avoid all, never, always (usually wrong) or sometimes, usually, typically (usually correct).
• Use in conjunction with graphs, etc.
Tommie L. Norris, DNS, RN
44
How Hard Should the Test Be?
• The “sweet spot” is 50‐70% – <30 too hard
– >90 too easy
Tommie L. Norris, DNS, RN
45
Exams are Valuable
• Replace at least 25% of exam questions when re‐administering
• Form your own course test bank from question faculty write
• Should I use a test bank that accompanies textbook?
Tommie L. Norris, DNS, RN
46
Test Security
• How do you know you have all exams returned?
• Do not save on hard drive
• Students can reconstruct entire exam
• Instructor manual
• What about review books?
• Form A & B
Tommie L. Norris, DNS, RN
47
Test Review
• To Do or Not to Do, that is the question?
– No, how is it done is the question
Tommie L. Norris, DNS, RN
48
Evaluate the Test: Validity
• Content validity‐opinion of experts
• Discrimination Index‐did students who scored high get the question correct?
• Concurrent validity‐degree to which a test correlates with another test testing the same content
• Test‐retest reliability
• Kuder Richardson is a statistic that measures reliability
Tommie L. Norris, DNS, RN
49
Let’s Get Started
• Revise previous questions
– Trend statistical information on each question/distracter – Replace any distracter that “0” chose as correct
– Look closely at any question that 50% of the class missed
• Was it a discriminator?
– Look closely at questions that ½ of class choose one distracter that was incorrect
Tommie L. Norris, DNS, RN
50
Tommie L. Norris, DNS, RN
51
Rewrite using “Best Practices of Test Construction”
A 45-year old woman was involved in a rear end
automobile collision on her way home from work. Having
sustained multiple fractures from hitting the steering
wheel, the paramedics initiated intravenous fluids at the
site and transported her to the Emergency Department.
Upon arrival her family was contacted to obtain a
medical history including all current medications and
allergies. What assessment will you make in the
emergency
department?
a. Level of consciousness
b. Blood pressure
c. Respiratory rate
d. Urinary output
Tommie L. Norris, DNS, RN
52