Leading the Way
to Formative Assessment
with the KCAS
August 1st
Introductions
• CCMS Faculty
• Dr. Sue Cain, CPE
• University of Louisville and Bellarmine Faculty
Agenda
 Housekeeping
 Norms
 Surveys
 On Demand
Writing/Constructed
Responses, Vocabulary
 September and October dates
 Knowledge pre-assessment
 Website
 Materials/resources
 Where have we been?
 Lesson Plan guide
 FRAME
 Concept Mastery Routine
 Where are we going?
 Classroom assessment
 Formative assessment
 Creating higher order
multiple choice, short
answer and, essay questions
 In-class questioning
 Question Answer
Relationships (QAR): A
strategy for students
 Survey
 September and October
dates
 Knowledge post-assessment
 Evaluation
Norm!
 Honor start/stop times
 Lunch for today
 Silence electronic equipment
 Participate actively
 Avoid sidebar conversations
 Celebrate!
Surveys
• Future PD topics
– On Demand Writing /Constructed Responses
– Vocabulary
• September and October PD dates
– Provide your school name
– Provide two or more dates for each month
• Knowledge pre-assessment
Website
• Currently housed at
http://www.bellarmine.edu/education/klandr
um.aspx
– Ppts
– Links
• Will move to UofL soon
– Ppts
– Links
– Videos
Materials and Resources
• Additional copies of information text
– Copies
– Texts and Lessons for Content-Area Reading
Common Core Standard(s)
Lesson Plan Guide
Instructional Strategy:
Key Ideas and Details:
Common Assessment (Description):
Assessment
Formative
difference
is
USE
of
DATA
Summative
DeSales 2/23/2011 Corrie Orthober, Ph.D.
How Do I Create Good SelectResponse and Supply
Response Items?
Types of Selects Response Items
 Multiple Choice
Types of Supply Response Items
 Short Answer
 Essay
Multiple Choice Questions:
Some Guidelines to Consider
• Item Stem
– *State the item stem in the positive whenever
possible.
– *Ask a complete question to get the item started.
– *Make sure the item stem does not give away the
correct answer.
– Avoid grammatical clues in an item stem such as plurals
or articles
– Emphasize qualifiers such as most likely and best
in the item stem.
If Possible, State the Item Stem in the
Positive
Poor Item
Better Item
Which of these is not a main idea of the
first paragraph?
Which of the following choices best
identifies the main idea implied in the
passage?
A. Abraham Lincoln was elected president
in 1860.
A. Abraham Lincoln was elected
president in 1860.
B. The issue of slavery troubled the nation B. The issue of slavery troubled the
since its birth.
nation since 1776.
C. Slaveholding states resolved to fight
over slavery.
C. The outbreak of the civil war occurred
in 1861.
What are some issues that you have had
with stating questions in the negative?
Prentice Hall Literature, 2007
Standard
Objective
Instructional
Strategy
Sample Multiple
Choice Assessment
Item
RI.6.2. Determine a
central idea of a
text and how it is
conveyed through
particular details…
Students will
differentiate the
central idea in a
piece of text from
the supporting
details.
FRAME
Which of the
following choices
best identifies the
main idea implied
in the first
paragraph?
A. Abraham
Lincoln was
elected
president in
1860.
B. B. The issue of
slavery troubled
the nation since
1776.
C. C. The outbreak
of the civil war
occurred in
1861.
Ask Complete Questions
Poor Item
Better Item
One key detail
A detail that contributes to the main idea
is that libraries are most important to
A. is that libraries are important to
imagination.
A. imagination
B. is that libraries are important to the
poor.
B. the poor
C. is that libraries are important to
children.
C. children
D. is that libraries are important to
D. citizens
imagination citizens.
What benefits does a complete versus
incomplete question create for students?
– from “Libraries Face Sad Chapter,” by Pete Hamill
Adapted from Prentice Hall Literature, 2007
Standard
Objective
Instructional
Strategy
Sample Multiple
Choice Assessment
Item
RL.6.2. Determine a
theme or central
idea of a text and
how it is conveyed
through particular
details…
Students will
differentiate the
central idea in a
piece of text from
the supporting
details.
FRAME
A detail that
contributes to the
main idea is that
libraries are most
important to
A. imagination
B. the poor
C. children
D. citizens
Make Sure the Item Stem Does Not
Give Away the Correct Answer
Poor Item
Better Item
The author introduces the concept of creed
in the passage, which of the following are
other words that have the same or similar
meaning?
The author introduces the concept of creed
in the passage, which of the following is a
word that has the same or similar meaning?
A. protest
A. protest
B. belief and faith
B. belief
C. recantation
C. recantation
What is poor about this item?
Adapted from Prentice Hall Literature, 2007
Standard
Objective
Instructional
Strategy
Sample Multiple
Choice Assessment
Item
RL.6.4. Determine
the meaning of
words and phrases
as they are used in
a text…
Students will
determine word
meanings through
the context
provided in a given
passage.
Think aloud,
modeling
(definition,
synonym, antonym,
etc) context clue
strategy.
The author
introduces the
concept of creed in
the passage, which
of the following is a
word that has the
same or similar
meaning?
A. protest
B. belief
C. recantation
Multiple Choice Questions:
Some Guidelines to Consider
• Answer Choices
– *Develop answer choices that are parallel in
grammar and in length
• Avoid clues in the answer choices that give away the
correct response
– *Make sure that the correct response is the only
correct response.
– Make sure the answer choices are plausible
Develop Answer Choices That Are
Parallel in Grammar and Length
Poor Answer Choices
Which of the following statements
represents the best paraphrase of the first
stanza
Better Answer
Choices
Which of the following choices best
identifies the main idea implied in the
passage?
A. Clouds and daffodils are part of nature,
unlike human beings.
A. Clouds and daffodils are part of nature,
unlike human beings.
B. I was walking by myself and I was
amazed to see a huge field of daffodils.
B. Clouds and fields of daffodils should be
treasured.
C. I was by myself when I saw a field of
daffodils so I felt lonely.
C. Clouds and fields of daffodils should be
shared.
D. It was a windy day but I did not mind
because the daffodils swayed in the wind.
D. Clouds and fields of daffodils are
beautiful.
What is poor about these answer
items?
Adapted from Prentice Hall Literature, 2007
Standard
Objective
Instructional
Strategy
Sample Multiple
Choice Assessment
Item
RL.7.4. Determine
the meaning of
words and phrases
as they are used in
a text, including
figurative and
connotative
meanings, analyze
the impact of
rhymes and other
repetitions of
sounds( e.g.
alliteration) on a
specific verse or
stanza of a poem or
section of a story or
drama.
Students will
determine the
meaning of phrases
in poems.
Modeling, think
alouds
Which of the
following choices
best identifies the
main idea implied
in the passage?
A. Clouds and
daffodils are part of
nature, unlike
human beings.
B. Clouds and fields
of daffodils should
be treasured.
C. Clouds and fields
of daffodils should
be shared.
D. Clouds and fields
of daffodils are
beautiful.
Make sure that the correct response is the only
correct response
Poor Answer Choices
Better Answer Choices
The author introduces the concept of creed The author introduces the concept of creed
in the passage, which of the following is a
in the passage, which of the following is a
word that has the same or similar meaning? word that has the same or similar meaning?
A. protest
A. protest
B. belief
B. belief
C. recantation
C. recantation
D. faith
D. race
What is poor about these answer
items?
Standard
Objective
Instructional
Strategy
Sample Multiple
Choice Assessment
Item
RL.6.4. Determine
the meaning of
words and phrases
as they are used in
a text…
Students will
determine word
meanings through
the context
provided in a given
passage.
Think aloud,
modeling
(definition,
synonym, antonym,
etc) context clue
strategy.
The author
introduces the
concept of creed in
the passage, which
of the following is a
word that has the
same or similar
meaning?
A. protest
B. belief
C. recantation
D. race
Test of Franzipanics
An illustration of the impact of poorly
written test questions
Stiggens, Arter, Chappuis, & Chappuis
Test of Franzipanics
The purpose of the cluss
in furmpaling is to
remove
a.
cluss-prags
b.
c.
d.
tremalis
cloughs
plumots
– State the item stem in
the positive whenever
possible
– Ask a complete
questions to get the
item started
– Make sure the item
stem does not give away
the correct answer.
– Avoid grammatical clues
in an item stem such as
plurals or articles
Stiggens, Arter, Chappuis, & Chappuis
Test of Franzipanics
Trasssig is true when
a. lusp trasses the vom
b. the viskal flans, if the
viskal is donwil or zortil
c. the belgo frulls
d. dissles lisk easily
– Make sure the answer
choices are plausible
– Develop answer choices
that are parallel in
grammar and in length
– Avoid clues in the
answer choices that give
away the correct
response
– Make sure that the
correct response is the
only correct response.
Stiggens, Arter, Chappuis, & Chappuis
Test of Franzipanics
The fribbled breg will minter best with an
a. derst
b. morst
c. sorter
d. ignu
Use the guidelines provided earlier to
answer the above question
Stiggens, Arter, Chappuis, & Chappuis
Your Turn!
Evaluate and Revise Test Items
•
Examine Your Item Stems
– Do you state them in the
positive?
– Do you ask complete
questions to get the item
started?
– Have you made sure the
item stem does not give
away the correct answer?
– Do you avoid grammatical
clues such as plurals or
articles?
– Do you emphasize qualifiers
such as most likely and best
in the item stem?
•
Examine answer choices
– Do you make sure the
answer choices are
plausible?
– Are your answer choices
parallel in grammar and in
length?
– Do you avoid clues in the
answer choices that give
away the correct response?
– Do you make sure that the
correct response is the only
correct response?
Multiple Choice Items Lend Well to
Recall and Procedural Type
Questions But … Multiple Choice
Items Can Also Tap Higher Order
Thinking
Multiple Choice Items Can Also Tap
Higher Order Thinking
• A multiple choice item developed to assess higher-order
thinking usually provides a situation/scenario or
graphic/chart followed by several questions that require
students to use or apply what they have learned to provide
the correct answer.
• Avoid questions, statements or examples used in class or
readings for the class. If students can recognize something
mentioned in class then they can answer the question
correctly simply by memorizing such statements, facts or
examples.
• The key to preparing memory-plus application questions is
to place the concept in a life situation or context that
requires the student to first recall the facts and then apply
or transfer the application of those facts into a situation.
http://www.schreyerinstitute.psu.edu/MultipleChoiceItems/
Multiple Choice Items Can Also
Encourage Higher Order Thinking
• Sample Item
Change
The main idea of the passage is
a. Brooklyn and Queens are important
to immigrants.
b. There are many new immigrants in
Brooklyn and Queens.
c. Libraries in Brooklyn and Queens are
crowded places.
d. Many immigrants in Brooklyn and
Queens go to libraries.
To
The author states that “libraries of
Brooklyn and Queens are jammed with
the new immigrants.” Which of the
following does this detail best
support?
a. Brooklyn and Queens are important
to immigrants.
b. There are many new immigrants in
Brooklyn and Queens.
c. Libraries in Brooklyn and Queens are
crowded places.
d. Many immigrants in Brooklyn and
Queens go to libraries.
– from “Libraries Face Sad Chapter,” by Pete Hamill
Prentice Hall Literature, 2007
Standard
Objective
Instructional
Strategy
Sample Multiple Choice
Assessment Item
RL.6.2. Determine a
theme or central
idea of a text and
how it is conveyed
through particular
details…
Students will
differentiate the
central idea in a
piece of text from
the supporting
details.
FRAME
The author states that
“libraries of Brooklyn and
Queens are jammed with
the new immigrants.”
Which of the following
does this detail best
support?
a. Brooklyn and Queens
are important to
immigrants.
b. There are many new
immigrants in Brooklyn
and Queens.
c. Libraries in Brooklyn
and Queens are crowded
places.
d. Many immigrants in
Brooklyn and Queens go
Techniques for Writing MultipleChoice Items that Demand Critical
Thinking
• Premise – Consequence
• Analogy
• Case Study
Techniques for Writing Multiple-Choice
Items that Demand Critical Thinking
Premise - Consequence
• Students must identify the correct outcome of a given
circumstance.
• Example If the libraries in Brooklyn and Queens were to
close then the people most affected would be:
• a) imaginative people
b) poor people
c) young children
d) American citizens
Note: To increase the difficulty, provide more than one
premise.
– from “Libraries Face Sad Chapter,” by Pete Hamill
http://tep.uoregon.edu/resources/assessment/multiplechoicequestions/sometechniques.html
Standard
Objective
Instructional
Strategy
Sample Multiple
Choice Assessment
Item
RI.7.3. Analyze the
interactions
between
individuals, events,
and ideas in a text
(e.g., how ideas
influence
individuals or
events, or how
individuals
influence ideas or
events).
Students will
Think aloud,
determine the
modeling.
impact of people on
events in a piece of
text.
If the libraries in
Brooklyn and
Queens were to
close then the
people most
affected would
be:
a) imaginative
people
b) poor people
c) young children
d) American
citizens
Techniques for Writing Multiple-Choice
Items that Demand Critical Thinking
Analogy
• Students must map the relationship between
two items into a different context:
• Example: Creed is to disbelief as contribute is
to
• A). provide
B). neglect
C). sacrifice
D). enrich
http://tep.uoregon.edu/resources/assessment/multiplechoicequestions/sometechniques.html
Standard
Objective
Instructional
Strategy
Sample Multiple
Choice Assessment
Item
RI.7.4. Determine
the meaning of
words and phrases
as they are used in
a text, including
figurative,
connotative, and
technical meanings;
analyze the impact
of a specific word
choice on meaning
and tone.
Students will
determine word
meanings through
the context
provided in a given
passage.
Think aloud,
modeling
(definition,
synonym, antonym,
etc) context clue
strategy.
The author
introduces the
concept of creed in
the passage, which
of the following is a
word that has the
same or similar
meaning?
A. protest
B. belief
C. recantation
D. race
Techniques for Writing Multiple-Choice
Items that Demand Critical Thinking
Case Study
• A single, well-written paragraph can provide material for several follow-up questions.
Example:
The tragedy of Romeo and Juliet is a very famous and simple love story. Written by William
Shakespeare in the fifteenth century, the play continues to be enjoyed by audiences more
than 500 years later. The reason Romeo and Juliet enjoy such enduring popularity is because
a) Shakespeare is the author
b) It is a great love story
c) Tragedies are popular
d) It is a famous text
Which phrase in the text helps the reader determine the meaning of the word enduring?
a) “written by William Shakespeare”
b) “very famous love story”
c) “more than 500 years”
d) “enjoyed by audiences”
Prentice Hall Literature, 2007
http://tep.uoregon.edu/resources/assessment/multiplechoicequestions/sometechniques.html
Standard
Objective
Instructional
Strategy
Sample Multiple
Choice Assessment
Item
RI.8.1. Cite the
textual evidence
that most strongly
supports an
analysis of what the
text says explicitly
as well as
inferences drawn
from the text.
Students will be
Think aloud,
able to make
modeling
inferences based on
text and
background
knowledge.
The reason Romeo
and Juliet enjoys
such enduring
popularity is
because
a) Shakespeare is
the author
b) It is a great love
story
c) Tragedies are
popular
d) It is a famous
text
Test Item Quality Checklist
Guidelines for Multiple Choice Items
____State whole question in item stem.
____Eliminate repetition of material in response
options.
____Be sure there is only one correct or best answer.
____Keep response options brief and parallel.
____Make all response options the same length.
____Limit use of all or none of the above.
____Use “always” and “never” with caution.
Stiggins et al., 2006
Student Documentation of Selected
Response
Items
As you answer each question on the test, decide whether you feel confident in your
answer or are unsure about it, and mark the corresponding box.
Problem
#
Learning
Target #
Confident
Unsure
Right
Wrong
Simple
Mistake
Further
Study
1. After your test has been corrected, identify which problems you got right and
which you got wrong by putting Xs in the “Right” and “Wrong” columns.
2. Of the problems you got wrong, decide which ones were due to simple mistakes
and mark the “Simple Mistake” column.
3. For all the remaining wrong answers, mark the “Further Study” column.
Stiggins et al., 2006
Time Permitting
• Transform a recall multiple choice question
into a higher order multiple choice question