Physics with Technology Sample
Test Question Development
Edson Barton
Precision Exams
Wednesday, 15 June 2016
1:00-3:00 pm
© Copyright Precision Exams 2016
Today’s Agenda
• Foundation Discussion
•
Importance of Standards
•
Importance of Item Writing
• Writing Great Items
• Analyze Items as a Group
• Item Writing as a Group
• Individual/Small Group Item
Writing
• Group Review of New Items
© Copyright Precision Exams 2016
•
Over 4,000,000 exams delivered
•
Develop exams for other organizations
•
16 years
o
•
Team has helped develop, administer, market, and
manage some of the worlds most used certifications and
assessments
•
Focused on CTE education
•
© Copyright Precision Exams 2016
ProCert Labs > Precision Exams & Surveys > Precision Exams (2006)
o
Standards development, validation, content mapping
o
Exam development, psychometrics, delivery and support
Bridge gap between “certification” testing and
“standardized assessment” testing in school systems
Why discuss item writing?
Because it’s difficult . . .
to write good questions
•
Distractors – “they need a gimme”
Because it’s easy . . .
to write bad questions
•
Simple rules of test-taking
Because it’s critical . . .
to good instruction!
•
Foundation for student and selfimprovement
© Copyright Precision Exams 2016
What is an Item?
In test development vocabulary:
• “Item” is the complete
question
• “Stem” is the body of a
question or statement
• “Options/Responses” are the
answer choices
•
Key is the one and only correct
answer(s)
•
Distracters are the plausible, yet
wrong options
© Copyright Precision Exams 2016
Item Writing Training
Standards
They are the
FOUNDATION
to great
education!
© Copyright Precision Exams 2016
What is the most
important thing to
writing great
items?
What is a standard?
• The individual bit of
knowledge that needs to
be transferred
• Knowledge or parameters
that everyone knows and
can rely upon
© Copyright Precision Exams 2016
Standards Are “Managers”
Good Managers:
•
Direct to specific outcomes
•
Demand quality – Keep the bar high
•
Provide all tools/knowledge needed to
complete tasks
•
Direct effort, but leave room for
individuality
Poor Managers:
•
Provide vague guidance
•
Are confusing, inconsistent, or tricky
•
Apathetic (don’t expect too much)
© Copyright Precision Exams 2016
Education Pyramid
4 Fundamental Education Questions:
1.
What do I need to teach today?
2.
How am I going to teach it?
3.
How do I know students learned it?
4.
What do I teach next? (Do I
remediate or move on?)
Test
Evaluate if
students, educators,
and materials met the goals.
Teach
Educators give and focus learning based on a
set of outcomes they expect students to master.
Instructional Materials
Materials are built from the guidance given by the standards.
Standards
Standards are the foundation for all aspects of education!
Standards set the goal.
© Copyright Precision Exams 2016
Usable Standards Development
Hardest thing about writing good standards . . .
deciding what should and should NOT be included
© Copyright Precision Exams 2016
Critical Components of Education Standards
1. Content Coverage
•
Identify the cumulative bits of
knowledge to be transferred for the
intended outcome/purpose and
candidate
2. Content Depth
•
Identify the appropriate depth of
knowledge for the intended
outcome/purpose and candidate
3. Timeframe
•
© Copyright Precision Exams 2016
Identify the period of time needed to
transfer the intended content
knowledge at the intended content
depth
Standards Writing Training
Two Primary Indicators of Good
Standards
1. If the standards were handed to an
educator, could they reasonably
understand what needs to be
taught?
2. If the standards were handed to a
student, could they reasonably
learn the information “on their
own?”
© Copyright Precision Exams 2016
Standards Writing: Identify and Practice
•
Objective: Format text.
•
Objective: Develop positive assertion skills to be used in conflict resolution.
•
•
Objective: Identify various types of communication styles.
•
•
•
•
Identify positive and negative methods of conflict resolution.
Identify types of destructive communication (blaming, interrupting, endless fighting,
character assassination, calling in reinforcements, and withdrawal).
Objective 1: Analyze ways economic, social, cultural, education and political
conditions can affect income and career potential.
Objective 2: Compare and contrast wage, gift, rent, interest, dividend, capital
gain, tip, commission, and business profit as sources of personal income.
Objective 1: Demonstrate understanding of computer hardware, peripherals
and troubleshooting.
•
•
•
Explore various computers available in current technology.
Identify and understand components and peripherals, including input-output devices:
cameras, printers, scanners, game-controllers, etc.
Explore and demonstrate knowledge of how to maintain computer equipment and solve
common hardware problems.
© Copyright Precision Exams 2016
Item Writing: Extract Essential Knowledge
Goals of Proper Item Writing
•
•
•
Evaluate student’s true
knowledge
Evaluate student’s depth of
knowledge
Improve instruction
evaluation
synthesis
analysis
application
Ask yourself two questions…
•
•
What is the essential knowledge
you want to extract?
If you had any means to evaluate
the student, how would you do it?
© Copyright Precision Exams 2016
comprehension
knowledge
Level 1: Knowledge
• Knowledge – exhibiting previously
learned material by recalling facts,
terms, basic concepts, and answers.
• Key words: who, what, why, when,
omit, where, which, choose, find,
how, define, label, show, spell, list,
match, name, relate, tell, recall,
select.
evaluation
synthesis
analysis
application
comprehension
knowledge
Example Question Cues
• What is…? How is…?
• Where is…? When did _________ happen?
• How did _______ happen?
• Why did …? Can you select…?
• How would you show…?
© Copyright Precision Exams 2016
Level 2: Comprehension
• Comprehension – demonstrating
understanding of facts and ideas by
organizing, comparing, translating,
interpreting, giving descriptions, and
stating main ideas.
• Key words: compare, contrast,
demonstrate, interpret, explain,
extend, illustrate, infer, outline, relate,
rephrase, translate, summarize, show,
classify.
Example Question Cues
• How would you classify the type of…?
• How would you compare…? contrast…?
• What facts or ideas show…?
• What is the main idea of…?
• Which statements support…?
© Copyright Precision Exams 2016
evaluation
synthesis
analysis
application
comprehension
knowledge
Level 3: Application
• Application – solving problems by
applying acquired knowledge, facts,
techniques, and rules in a different
way.
• Key words: apply, build, choose,
construct, develop, interview, make
use of, organize, experiment with,
plan, select, solve, utilize, model,
identify.
evaluation
synthesis
analysis
application
comprehension
knowledge
Example Question Cues
• How would you use…?
• How would you solve ______ using what you
have learned?
• How would you organize ______ to show?
• What approach would you use to…?
• What elements would you choose to
© Copyright Precision
Exams 2016
change…?
Level 4: Analysis
• Analysis – examining and breaking
information into parts by identifying
motives or causes; making
inferences and finding evidence to
support generalizations.
• Key words: analyze, categorize,
classify, compare, contrast,
discover, dissect, divide, examine,
inspect.
Example Question Cues
• How is ______ related to…?
• What inference can you make…?
• What conclusions can you draw…?
• How would you categorize…?
• What is the relationship between…?
© Copyright Precision Exams 2016
evaluation
synthesis
analysis
application
comprehension
knowledge
Level 5: Synthesis
• Synthesis – compiling information
together in a different way by
combining elements in a new
pattern of proposing alternative
solutions.
• Key words: build, choose, combine,
compile compose, construct, create,
design, develop, estimate,
formulate, imagine, invent.
Example Question Cues
• What changes would you make to solve…?
• How would you improve…?
• What way would you design…?
• How would you test…?
• Can you think of an original way for the…?
© Copyright Precision Exams 2016
evaluation
synthesis
analysis
application
comprehension
knowledge
Level 6: Evaluation
• Evaluation – presenting and defending
opinions by making judgments about
information; validity of ideas or quality
of work based on a set of criteria.
• Key words: choose, conclude, criticize,
decide, defend, determine, dispute,
evaluate, judge, justify, measure,
compare.
Example Question Cues
• How would you prove…? Disprove…?
• Would it be better if…?
• How would you evaluate…?
• How would you determine…?
• Based on what you know, how would you
explain…?
© Copyright Precision Exams 2016
evaluation
synthesis
analysis
application
comprehension
knowledge
Achieve Higher Cognitive Levels
Level:
Key Words:
1. Knowledge
Who, what, why, when, omit, where,
which, choose, find, how, define, label,
show, spell, list, match, name, relate,
tell, recall, select




What is…?
How is…?
Where is…?
Can you select…?
Compare, contrast, demonstrate,
interpret, explain, extend, illustrate, infer,
outline, relate, rephrase, translate,
summarize, show, classify




How would one compare..? Contrast..? classify..?
What facts or ideas show…?
What is the main idea of…?
Which statements support…?
Exhibit previously learned material by recalling
facts, terms, basic concepts, and answers.
2. Comprehension
Demonstrate understanding of facts and ideas by
organizing, comparing, translating, interpreting,
giving descriptions, and stating main ideas.
3. Application
Solve problems by applying acquired knowledge,
facts, techniques, and rules in a different way.
4. Analysis
Examine and break information into parts by
identifying motives or causes; making inferences
and finding evidence to support generalizations.
5. Synthesis
Compile information together in a different way by
combining elements in a new pattern of proposing
alternative solutions.
6. Evaluation
Present and defend opinions by making
judgments about information; validate ideas or
quality of work based on a set of criteria.
© Copyright Precision Exams 2016
Example Question Cues:
Apply, build, choose, construct, develop, 
interview, make use of, organize,


experiment with, plan, select, solve,
utilize, model
How would one use…?
What approach would one apply to…?
What elements would one choose to change…?
Analyze, categorize, classify, compare,
contrast, discover, dissect, divide,
examine, inspect





How is ______ related to…?
What inference can you make…?
What conclusions can you draw…?
How would you categorize…?
What is the relationship between…?
Build, choose, combine, compile
compose, construct, create, design,
develop, estimate, formulate, imagine,
invent





Given the tool set, design a system to solve…
How would you improve…?
What way would one design…?
How would one test…?
Given the entity’s situation, how would you…?
Choose, conclude, criticize, decide,
defend, determine, dispute, evaluate,
judge, justify, measure, compare





How would one prove…? Disprove…?
Would it be better if…?
How would one evaluate…?
How would one determine…?
How would one explain…?
Enhanced Question Types
•
Multiple Choice, Drag & Drop, Labeling, Grouping, Ordering, Multiple Select, Hot Spot
© Copyright Precision Exams 2016
Elements of Great Test Items
• Best Written Items
•
•
•
•
•
•
Accurately and only evaluate the standards
Appropriately difficult – built to minimally competent
candidate
Tests knowledge not test-taking skills
Clarity – clear, simple, expresses one complete thought,
includes all information necessary to answer the
problem, and uses proper EMPHASIS
Avoid:
• (except, not) (always, never) (best, recommended)
(all of the above, none of the above)
• cultural, ethnic, and/or gender specifics or
insensitivities
• true or false construction
• measuring trivia, “fluff” or not relevant knowledge,
or knowledge that could easily become outdated
If the stem is a question, end it with a question mark
and use proper sentence construction with each option
© Copyright Precision Exams 2016
Elements of Great Test Items
• Options – Key and Distractors
•
•
•
•
•
Key should have verifiable sources (readily
available and current info)
Key should not be based on an opinion
All options must be consistent with stem and
other responses (grammatically and
contextually), and not be noticeably shorter,
longer, or different
All options should be written in parallel style
Distractors must be plausible (but not possible
– there must be only one correct answer)
• Writing good distractors is the most
difficult part of creating good items
© Copyright Precision Exams 2016
Writing in Parallel Style
• Consistent Parallel Style
In the story Little Red Riding Hood, what type
of animal ate grandma?
A. Snake
B. Wolf
C. Bear
D. Badger
• Two Pairs of Parallel Style
What type of water is hazardous to your
health?
A. Filtered
B. Boiled
C. Contains contaminates
D. Contains fluoride
© Copyright Precision Exams 2016
Working with Scenarios
Crafting a solid scenario
presents a bit of a
challenge compared with a
straight-forward item.
• A scenario is meant to give
information from which a
question can be drawn.
• It is not about setting a “mood.”
• Length does not matter: it
should be long enough to
present all the facts necessary,
without giving non-relevant
details to the reader.
© Copyright Precision Exams 2016
Example Scenario
Company A and Company B merge to create a new organization, Company ABC. Both companies operated
strategic business units and employed full-time project managers. Although both companies were composite
matrix management organizations, their corporate and project management cultures and organizational
structures differed. Company A’s project management organizations tailored their processes and tools to their
assigned strategic business units; Company B’s project management organizations centralized the
development of processes and tools for corporate-wide adoption. The new organization, Company ABC,
retained the strategic business units, composite matrix management organization, and full-time project
managers. It has a single project organization that aligns project managers with strategic units and has
separate headquarters in the United States and the United Kingdom to oversee its North American and
European regions, respectively. In so doing, cultural diversity is recognized and accepted.
What kind of project management behavior culture would MOST likely have developed at Company A?
A.
Isolated
B.
Fragmented
C.
Non-cooperative
D.
Competitive
Which cultural characteristics are MOST likely to be evident at the North American headquarters?
A.
Communicate formally and respect tradition.
B.
Focus on task accomplishments and reward individualism.
C.
Value history, hierarchy, individualism and loyalty.
D.
Communicate indirectly and emphasize hard work and success.
© Copyright Precision Exams 2016
Example Scenario
As Program Manager you have just finished gathering the program requirements and have
defined its deliverables. Now, you are set to build the program's WBS (PWBS).
What should you do FIRST?
A.
Check similar programs in the organization or industry and start from there.
B.
Ask your project managers to build a WBS for their respective projects and
combine them (bottom-up).
C.
Build the program's WBS first and then build the individual project's WBS (topdown).
D.
Identify the deliverables at the project level.
What level of detail should the WBS (PWBS) include?
A.
Program-level deliverables, first to second level of each project's WBS
B.
Program- and project-level deliverables
C.
Program level only
D.
Program-level WBS and project-level WBS's that are required for control.
© Copyright Precision Exams 2016
Analyze Items as a Group
What type of water is the best kind to drink?
A.
B.
C.
D.
Filtered
Contaminated
Frozen
Distilled
• Not good: A reasonable candidate would
immediately know that “frozen” and
“contaminated” could not be correct because you
cannot drink frozen water and contaminated has a
generally negative association.
• “Best” is an opinion in this context.
© Copyright Precision Exams 2016
Analyze Items as a Group
What type of water is hazardous to your health?
A. Filtered water, because it does not contain minerals.
B. Boiled water, because it kills all the bacteria that are
good for your digestive system.
C. Contaminated, because it contains bacteria and other
elements that can make you sick.
D. Fluoride water.
• Not good: Because “fluoride water” is noticeably shorter
than the other responses it signals that it is either correct
or incorrect.
• “Contaminated” is the only response that doesn’t include
“water” in the first part of the option.
© Copyright Precision Exams 2016
Analyze Items as a Group
Which of the following is NOT a Web browser?
A.
B.
C.
D.
E.
Microsoft Internet Explorer
Adobe Acrobat
Opera
Netscape Navigator
Lynx
• Avoid NOT - Write the stem in positive form. The
only time a negative question is warranted is when
it is important to test what NOT to do.
© Copyright Precision Exams 2016
Analyze Items as a Group
Which part of the following URL is the domain name?
URL = http://testdev.org/item/class2.htm
A.
B.
C.
D.
http://
Testdev.org/
access/
class2.htm
• Bad – because “Testdev.org” is capitalized. Avoid giving
hints like this that savvy test-takers may easily pick apart.
Even if they didn’t know that testdev.org/ was the correct
choice, they might be inclined to choose B.
© Copyright Precision Exams 2016
Analyze Items as a Group
Which one of the following statements is TRUE?
A. A packet is a complete message sent over the
Internet.
B. All browsers display information in exactly the same
way.
C. URL stands for “Ultimate Resource Location.”
D. A browser is used to read FTP messages.
E. The Web was created by American physicians.
• Poor-quality: Make all distractors plausible - All
distractors should look, feel, and sound like they
could be the correct answer.
© Copyright Precision Exams 2016
Analyze Items as a Group
A Web page has a file size of 4 kb (including just text and html code). A photo
with 23 kb and a button-bar with 14 kb are also placed on the page. An
additional navigational element with 1.5 kb is used four times on the page.
What amount of data must a visitor’s browser load to show this page with all
graphics?
A.
B.
C.
D.
•
4 kb
42.5 kb
47 kb
60 kb
Good: This item tests understanding and application. The response options
are logical.
© Copyright Precision Exams 2016
Analyze Items as a Group
Networks allow the connected computers to share which of
the following?
A. Files and resources
B. Resources and programs
C. Files and programs
D. Files only
E. Files, resources, and programs
• Bad: Answer choices should be independent and mutually
exclusive. E is the correct answer, which makes A, B, and C
correct as well. These are “possible” answers.
© Copyright Precision Exams 2016
Analyze Items as a Group
HTML stands for:
A. high text master language
B. hypertext markup language
C. hypertext methodology
D. high tech machine language
• Definition Item: This item tests recall. Although recall
items are easy to design and use, a high-quality multiplechoice item should test understanding, critical thinking,
and/or problem solving.
• Should phrase the stem as a question.
© Copyright Precision Exams 2016
Analyze Items as a Group
Given the HTML code fragment:
<P><IMG SRC="image.gif" BORDER=2>
Which one of the following is a correct reformulation of the
HTML code into XHTML?
• A.
<P><IMG SRC="image.gif" BORDER="2"/>
• B.
<p><img src="image.gif" border=2></p>
• C.
<P><IMG SRC="image.gif" BORDER="2">
• D.
<p><img src="image.gif" border="2" /></p>}
• Good: This item tests understanding and application. Also,
this would be an appropriate performance-based item.
© Copyright Precision Exams 2016
Analyze Items as a Group
Which one of the following <list> attributes is NOT
deprecated in HTML 4.0?
A. type
B. start
C. style
D. All of the above attributes are deprecated.
E. None of the above attributes are deprecated.
• Bad: Avoid All of the Above and None of the Above –
Using these options draws students into test-taking
strategies and we are more concerned about testing
knowledge.
© Copyright Precision Exams 2016
Hour 1 Complete
© Copyright Precision Exams 2016
Item Writing Practice as a Group
5 minutes:
Create an item from your standards. Let’s start with standard 1.
4.1
• Calculate the kinetic energy of an object with a mass of 900kg
moving at a velocity of 4 m/s.
•
7200 J PE_KEY
•
3600 J
•
1800 J
•
14400 J
© Copyright Precision Exams 2016
Item Writing Practice as a Group
5 minutes:
Create an item from your standards.
4.1
• What would happen to the kinetic energy of marble if its
velocity doubles?
•
Increase by a factor of 4
•
Decreases by a factor of 4
•
Increase by a factor of 2
•
Decreases by a factor of 2
© Copyright Precision Exams 2016
Item Writing Practice as a Group
5 minutes:
Create an item from your standards.
• [stem]
•
[key]
•
[distractor]
•
[distractor]
•
[distractor]
© Copyright Precision Exams 2016
Item Writing Practice as a Group
5 minutes:
Create an item from your standards.
• [stem]
•
[key]
•
[distractor]
•
[distractor]
•
[distractor]
© Copyright Precision Exams 2016
Individual/Small Group Item Writing Practice
20 minutes:
Write several questions to
present back to the group for
review and possible inclusion
to state skills exams.
© Copyright Precision Exams 2016
Group Review of Individually Written Items
20 minutes:
As a group, review the items
that were created
individually by participants.
© Copyright Precision Exams 2016
Congratulations!
You’re now item writing experts!
801.653.9356
[email protected]
www.precisionexams.com
© Copyright Precision Exams 2016