What do you believe is the most
powerful use of assessment? Why?
A. To measure
Asking the Right
Questions:
Question Complexity
student learning.
C. To identify student
weaknesses.
B. To give student
grades.
D. To provide
feedback to students.
E. A Different use…
June 15, 2016
Workshop Facilitator
Anne Weerda, M.S.Ed., M.Ed.
Anne is an assessment specialist focusing on the
effective implementation of student growth measures in
school districts. She is the founder of Kids At The Core,
specializing in developing assessment and data literacy in
schools around student growth measurements. She has
experience at the District Office level, as a school
improvement assessment specialist, a high school
administrator, department chair and a classroom teacher.
@StudentGrowth
What is the range of complexity
in classroom instruction?
Kids At The Core
Determine the Spectrum of
Complexity
Acknowledging the various levels of cognitive demand within each standard will
help teachers write questions at a consistent cognitive level across the
assessment sets thus allowing the sets to mirror in complexity.
Bloom’s Taxonomy
(Revised)
Marzano's Taxonomy
Remembering
Level 1: Retrieval
Understanding
Level 2:
Comprehension
Applying
Level 3: Analysis
Analyzing
Level 4: Knowledge
Utilization
Evaluating
Level 5: Metacognition
Creating
Level 6: Self-System
Thinking
Webb's Depth of
Knowledge
Recall and reproduction
(DOK1)
Skills and concepts
(DOK2)
Strategic thinking/complex
reasoning
(DOK3)
Extended
thinking/reasoning
(DOK4)
Blooms Taxonomy
(Revised)
Marzano's Taxonomy
Remembering
Level 1: Retrieval
Understanding
Level 2: Comprehension
Webb's Depth of
Knowledge
Recall and reproduction
(DOK1)
Blooms Taxonomy
(Revised)
Marzano's Taxonomy
Remembering
Level 1: Retrieval
Understanding
Level 2: Comprehension
Define the following:
Define the following:
Alliteration: ____________________
Setting: _______________________
Foreshadowing: ________________
Personification: ________________
Alliteration: ____________________
Setting: _______________________
Foreshadowing: ________________
Personification: ________________
Who was the main character in the
story...
Who was the main character in the
story...
Blooms Taxonomy
(Revised)
Marzano's Taxonomy
Remembering
Level 1: Retrieval
Understanding
Level 2: Comprehension
Webb's Depth of
Knowledge
Recall and reproduction
(DOK1)
Blooms Taxonomy
(Revised)
Webb's Depth of
Knowledge
Recall and reproduction
(DOK1)
CAUTION: This requires
a rubric. What is the
value of an open ended
question here?
Marzano's Taxonomy
Applying
Level 3: Analysis
Analyzing
Level 4: Knowledge
Utilization
Webb's Depth of
Knowledge
Skills and Concepts
(DOK2)
Interpret the following quote and
support your analysis with
evidence from the text...
Define Active Therapy.
Blooms Taxonomy
(Revised)
Marzano's Taxonomy
Applying
Level 3: Analysis
Analyzing
Level 4: Knowledge
Utilization
Why is my cake not as moist as the sample?
Webb's Depth of
Knowledge
Skills and Concepts
(DOK2)
Procedural Knowledge Is:
●
“Action sequences for solving problems” (RittleJohnson & Wagner, 1999)
●
“Like a toolbox, it includes facts, skills,
procedures, algorithms or methods” (Barr, Doyle
et. al., 2003)
●
“Learning that involves only memorizing
operations with no understanding of
underlying meanings.” (Arslan, 2010)
Procedural Knowledge Is:
Barr Doyle et. al. (2003) finds that when students
are asked to perform a procedure such as solving
an equation, students can often follow an example
and get a correct answer without understanding
how or why the process works.
Procedural Knowledge Is:
EXAMPLE:
Procedural questions highlight student ability to
demonstrate process, but do not shed light on the
meaning of why.
Conceptual Knowledge is:
●
●
“Explicit or implicit understanding of the
principles that govern a domain and the
interactions between pieces of knowledge in
a domain,” (Rittle-Johnson & Wagner, 1999)
“Ideas, relationships, connections...”
What they Look Like...
Procedural
Examine a dough sample and
determine if it needs more flour or
more water based on the desired
texture.
Perform CPR
Apply knowledge based on a given
scenario.
How could failing to check an
airway prevent success when
using CPR?
List developmental milestones
Diagnose a child or student’s
potential disability based on a
scenario of behaviors and
developmental milestones.
(Barr, Doyle
et. al., 2003)
●
“Learning that involves understanding and
interpreting concepts and the relations
between concepts” (Arslan, 2010)
Measuring Procedural Knowledge:
● Familiar tasks--problem types students have solved before
● Transfer problems--problems with an unfamiliar problem
feature that requires recognition that a known procedure is
relevant OR small adaptations of a known procedure (Renkl,
et. a., 1998)
● Timed tasks--to measure automatization of procedural
knowledge
Conceptual
Follow a recipe to make pizza
dough
Measuring Procedural Knowledge:
Measuring Procedural Knowledge:
Measuring Procedural Knowledge:
Measuring Conceptual Knowledge:
Measuring Conceptual Knowledge:
● Explicit Measures--providing definitions and explanations
(ex: explaining why a procedure works)
● Unfamiliar Tasks-- (student derives answer from conceptual
knowledge rather than implementation of a known procedure)
Note: children develop procedures through repeated practice of
the same problem type.
● Measures stronger if use multiple tasks
● Reduce influence of task-specific characteristics
(Schneider & Stern, 2010)
● Can measure multidimensional construct involving more
than one concept
Trouble at Level Two?
Analyze assessment data….where
is student stuck/falling apart?
Trouble at Level Two?
Analyze assessment data….where
is student stuck/falling apart?
A = 5, C = 13 B = ?
Find B
a) 65 (used 13 as b)
b) 12 *
c) 144 (didn’t take root)
d) 8 (didn’t square)
Blooms Taxonomy
(Revised)
Marzano's Taxonomy
Evaluating
Level 5: Metacognition
Creating
Level 6: Self-System
Thinking
Webb's Depth of
Knowledge
Strategic
Thinking/Complex
Reasoning (DOK3)
Consider the question: Should Students be able to grade
their teachers?
Read the following excerpt, and in a 3 paragraph essay, state
your opinion and support with specific details and evidence.
How do you vary question complexity
on your assessments?
A. Not varied much. They
are at the DOK 1 level,
mostly recall.
C. Mostly at the DOK3/4
level. Most questions are
Analytical and Evaluative.
B. Question complexity is
balanced & even.
D. It depends on the unit.
E. Unsure...
Blooms Taxonomy
(Revised)
Marzano's Taxonomy
Evaluating
Level 5: Metacognition
Creating
Level 6: Self-System
Thinking
Webb's Depth of
Knowledge
Strategic
Thinking/Complex
Reasoning (DOK3 +
DOK4)
Create a logo that
represents the key
benefits, feelings and
ideas of a brand.
Design a questionnaire to gather
information about your target
audience, analyze results and create
an ad campaign around what
appeals to your target audience.
Critique a 3D model of a
part using specific
content specific
vocabulary.
Evaluate a given recipe for common
dietary restrictions and suggest
modifications to the ingredients.
Question Stems. Free Resource.
Standard
Standard: Apply & Analyze
Expanded: Evaluate & Create
RL 4.1
&
RI 4.1
Identify explicit information:
What does the author mean by:
“quote”...?
What is the purpose of this...?
Basic: Remember & Understand
Analyze explicit information; making
inferences:
What inferences can you make
about...?
Which of these examples tells us
why...?
Evaluate explicit information and
inferences:
(Defend a position)Why do you
believe...?
Is there a better solution to the
character’s problem...?
RL 4.2
&
RI 4.2
Identify Theme/Idea:
What is the theme of this story
(text)...?
What is the message of this text
(poem, story, etc.)...?
Analyze Theme/Idea:
How do the character’s action support
the theme...?
What are the most important events in
the story...? (RL)
Which of these is a good summary
sentence...?
Evaluate Theme/Idea:
Which of these details does not
support the main idea
(message)...?
RL 4.3
&
RI 4.3
Identify elements
What is the setting of the story...?
Which describes character x...?
Which describes the setting (time,
place, social environment)...?
Which of these details (quotes)
describes character x...? (RL)
Analyze Elements
Why might ____ have happened...?
(RL)
What is the first (second) step in the
procedure...? (RI)
What was the effect of _____’s idea...?
(RI)
Evaluate Elements
Did the environment affect the
outcome of the story...?
Create a scenario: How would you
imagine the events from the text
effecting you today..?
www.KidsAtTheCore.com
Mirrored Assessments are
Not Identical!
Comparable
Assessments:
Measuring Growth
Mirroring refers to assessments created in
parallel, not exact copies.
And Why Not?
Testing the exact question twice is
unintentionally assessing recall rather than
discrete skills.
Mirrored Assessments are
Not Identical!
The Practice Effect
Mirroring refers to assessments created in
parallel, not exact copies.
And Why Not?
Testing the exact question twice is
unintentionally assessing recall rather than
discrete skills.
After Repeated Exposure to a single task:
1. Improvement in scores due to memorization &
practice on exact items(s).
2. Decrease in scores due to fatigue of test taker.
3. Decrease in scores due to boredom on the part of
the test taker.
Mirrored Assessments are
Not Identical!
Mirrored Assessments are
Not Identical!
1) Students will remember seeing the same
questions twice.
1) Students will remember seeing the same
questions twice.
2) The Teachable Moment
Lets not underestimate our students.
Kids have good memories.
Even a kindergartner will recognize the same question repeated.
3) Validity of the Data
Mirrored Assessments are
Not Identical!
Mirrored Assessments are
Not Identical!
2) The Teachable Moment
3) Validity of the Data
Piaget: The moment of questioning and disequilibrium is the moment of the
student’s desire to create new learning.
Students’ answer choices may be influenced by their previous selection.
It is our job as teachers to create and capitalize on these moments.
Student effort will be potentially affected.
Valid Results: Assessment measures
what you intended to measure Students may “look up” or “talk about” questions
understanding
of a skill or topic
Mirrored Assessments are
Not Identical!
Mirrored Assessments are
Not Identical!
Are there cases where a question COULD be
identical?
3) Validity of the Data
Recall or Memorization
Student effort will be potentially affected.
Students’ answer choices may be influenced by their previous selection.
Students may “look up” or “talk about” questions
-Math Facts
-Vocabulary
-Sight Words
For Example?
What is the main idea
of this passage?
What is the main idea
of this passage?
For Example?
What is the main idea
of this passage?
For Example?
What is the main idea
of this passage?
For Example?
What is the main idea
of this passage?
For Example?
For Example?
Clap this rhythm...
What is the main idea
of this passage?
What is the main idea
of this passage?
For Example?
Clap this rhythm...
For Example?
Clap this rhythm...
Clap this rhythm...
Clap this rhythm...
For Example?
For Example?
Clap this rhythm...
Clap this rhythm...
Analyze this data set...
For Example?
For Example?
Analyze this data set...
Analyze this data set...
Analyze this data set...
For Example?
Analyze this data set...
Question Complexity is Key
Analyze this data set...
Analyze this data set...
For Example?
A
Define these terms...
Name This Note...
B
Define these terms...
Label this map...
Define these terms...
Label the parts of this cell...
Can we show growth from A to B?
A
B
A
B
A
Name this note...
Define these terms...
Define these terms...
Define these terms...
B
Name this note...
Define these terms...
Can we show growth from A to B?
Can we show growth from A to B?
A
B
A
Name this note...
Define these terms...
B
A
B
Name this note...
Name this note...
Define these terms...
Define these terms...
A
B
Name this note...
Define these terms...
Can we show growth from A to B?
A
B
Can we show growth from A to B?
Label the
mitochondria
Label the endoplasmic
reticulum
For Growth in MEMORIZATION
A
B
A
Name this note...
Define these terms...
B
A
B
A
B
Name this note... Name this note...
Name this note...
Define these terms...
Can we show growth from A to B?
Define these terms...
A
Label the
mitochondria
B
Label the endoplasmic
reticulum
A
Label the mitochondria &
endoplasmic reticulum
Define these terms...
Name this note... Name this note...
B
Label the mitochondria &
endoplasmic reticulum
The Balanced Assessment
The Balanced Assessment
No Assessment should be 100% Recall
A balanced assessment has appropriate amounts of questions at each
level of cognitive demand. It is reflective of the range of cognitive
demand required in the class it measures.
Great Multiple Choice
Improving Validity
What do you think about this
Improving Validity
example?
A. Answer choice length?
A. Answer choice length?
Who is Kate?
B. Homogeneous choices?
D.
Another
thought...
C.
Plausible?
a friend of Ben’s
Ben’s little sister
cousin of Ben’s
older sister of Ben
C.
Plausible?
B. Homogeneous choices?
C.
Plausible?
What do you think about this
Improving Validity
example?
Who is Kate?
a.
b.
c.
d.
a friend of Ben’s
Ben’s little sister
cousin of Ben’s
older sister of Ben
D.
Let’s make it better!
B. Homogeneous choices?
Another
thought...
A. Answer choice length?
a.
b.
c.
d.
a friend of Ben’s
Ben’s little sister
cousin of Ben’s
older sister of Ben
D.
What do you think about this
Improving Validity
example?
Who is Kate?
a.
b.
c.
d.
Another
thought...
Let’s make it better!
How can we make
the choices more
consistent?
What do you think about this
Improving Validity
example?
After reading Passage 1: Excerpt from Dr. Martin Luther King Jr.’s
Letter from Birmingham City Jail and Passage 2: Excerpt from “The
Life of Martin Luther King, Jr.” (Nobel Peace Lectures)
Who is Kate?
a.
b.
c.
d.
Ben’s friend
Ben’s little sister
Ben’s cousin
Ben’s older sister
What do you think about this
Improving Validity
example?
How can we make
the choices more
consistent?
Let’s make it better!
What do you think about this
Improving Validity
example?
A.
How does the presentation of the same person (Dr. King)
compare in these two sources?
a. Both passages describe the assassination of Dr. King.
b. Both passages describe Dr. King as an intelligent,
educated and passionate civil rights leader.
c. Passage 1 describes the life of Dr. King, but Passage 2
describes one event.
d. Passage 1 describes Dr. King is as a nonviolent leader,
but Passage 2 describes Dr. King as an aggressive, angry
leader.
What do you think about this
Improving Validity
example?
choiceKing
length?
After reading Passage 1: Excerpt from Dr.Answer
Martin Luther
Jr.’s
Letter from Birmingham City Jail and Passage 2: Excerpt from “The
Life of Martin Luther King, Jr.” (Nobel Peace Lectures)
After reading Passage 1: Excerpt from Dr. Martin Luther King Jr.’s
Letter from Birmingham City Jail and Passage 2: Excerpt from “The
Life of Martin Luther King, Jr.” (Nobel Peace Lectures)
How does the presentation of the same person (Dr. King)
compare in these two sources?B. Homogeneous choices?
a. Both passages describe the assassination of Dr. King.
b. Both passages describe Dr. King as an intelligent,
C. Plausible?
educated and passionate civil rights
leader.
c. Passage 1 describes the life of Dr. King, but Passage 2
describes one event.
D. Another
d. Passage 1 describes Dr. King is as a nonviolent leader,
thought...
but Passage 2 describes Dr. King as an aggressive, angry
leader.
How does
presentation
the same
person
(Dr.
Is itthe
Plausible
that an of
author
can write
about
hisKing)
compare
in these
two sources?
own
assassination?
a. Both passages describe the assassination of Dr. King.
b. Both Is
passages
describe
Dr.Nobel
King Peace
as an intelligent,
it Plausible
that the
lecture is
educated
rights leader.
aboutand
his passionate
aggressive civil
leadership?
c. Passage 1 describes the life of Dr. King, but Passage 2
describes one event.
d. Passage 1 describes Dr. King is as a nonviolent leader,
Let’s
make2itdescribes
better!Dr. King as an aggressive, angry
but Passage
leader.
What do you think about this
Improving Validity
example?
What do you think about this
Improving Validity
example?
After reading Passage 1: Excerpt from Dr. Martin Luther King Jr.’s Letter
from Birmingham City Jail and Passage 2: Excerpt from “The Life of
Martin Luther King, Jr.” (Nobel Peace Lectures)
After reading Passage 1: Excerpt from Dr. Martin Luther King Jr.’s Letter
from Birmingham City Jail and Passage 2: Excerpt from “The Life of
Martin Luther King, Jr.” (Nobel Peace Lectures)
How does the presentation of the same person (Dr. King)
compare in these two sources?
a. Both passages describe the assassination of Dr. King.
b. Both passages describe Dr. King as an intelligent,
educated and passionate civil rights leader.
c. Passage 1 describes the life of Dr. King, but Passage 2
describes one event.
d. Passage 1 describes Dr. King is as a nonviolent leader,
but Passage 2 describes Dr. King as an aggressive,
angry leader.
How does the presentation of the same person (Dr. King)
compare in these two sources?
a. Both passages describe the importance of Dr. King.
b. Both passages describe Dr. King as an intelligent,
educated and passionate civil rights leader.
c. Passage 1 describes the life of Dr. King, but Passage 2
describes one event.
d. Passage 1 describes Dr. King’s wrongful imprisonment,
but Passage 2 describes Dr. King leadership style.
What do you think about this
Improving Validity
example?
What do you think about this
Improving Validity
A. Answer choice length?
example?
B. Homogeneous choices?
Who was the president when WWII ended in
1945?
Who was the president when WWII ended in
1945?
a.
b.
c.
d.
a.
b.
c.
d.
Franklin D. Roosevelt
Ralph Teetot
Harry S. Truman
Percy Spenser
What do you think about this
Improving Validity
example?
Who was the president when WWII ended in
1945?
C. Plausible? They
a. Franklin D. Roosevelt are not all presidents...
b. Ralph Teetot
c. Harry S. Truman
d. Percy Spenser
Franklin D. Roosevelt
Ralph Teetot
Harry S. Truman
Percy Spenser
C.
Plausible?
D.
Another
thought...
What do you think about this
Improving Validity
example?
Who was the president when WWII ended in
1945?
C. Plausible?
a.
b.
c.
d.
Franklin D. Roosevelt
Harry S. Truman
John Kennedy
Thomas Woodrow Wilson
Let’s make it better!
What do you think about this
Improving Validity
A. Answer choice length?
example?
What do you think about this
Improving Validity
example?
B. Homogeneous choices?
Avery bought a popsicle that costs $1.25. He
gave the clerk $5.00. How much change did he
get back?
a.
b.
c.
d.
$3.25
$2.75
$3.75
$2.50
C.
Plausible?
D.
Another
thought...
Avery bought a popsicle that costs $1.25. He
gave the clerk $5.00. How much change did he
get back?
a.
b.
c.
d.
$3.25
$2.75
$3.75
$2.50
Is this a Logical
Order?
What do you think about this
Improving Validity
example?
Great Constructed Response.
Improving Validity
B. Is prompt
A. Clearly Stated?
Avery bought a popsicle that costs $1.25. He
gave the clerk $5.00. How much change did he
get back?
a.
b.
c.
d.
$2.50
$2.75
$3.25
$3.75
novel?
C.
Too many options?
D.
Measurable
Responses?
E.
Another
thought...
What do you think about this
Improving Validity
example?
What do you think about this
Improving Validity
example?
Read: All About Ants
Constructed Response (2 pts each)
In complete sentences, tell me about ants.
____________________________________
____________________________________
____________________________________
What do you think about this
Improving Validity
A. Clearly Stated?
example?
What do you think about this
Improving Validity
A. Clearly Stated?
example?
C.
B. Is prompt novel?
What
are we trying to measure?
D.
Too many
options?
Read: All About Ants
Constructed Response (2 pts each)
Read: All About Ants
Constructed Response (2 pts each)
In complete sentences, tell me about ants.
____________________________________
C. Too many
____________________________________
options?
____________________________________
In complete sentences, tell me about ants.
____________________________________
____________________________________
____________________________________
Measurable
Responses?
E.
Another
thought...
Let’s make it better!
What do you think about this
Improving Validity
A. Clearly Stated?
example?
What do you think about this
Improving Validity
example?
C. Too many
Read: All About Ants
Constructed Response (2 pts each) options?
What are 2 things that make ants amazing?
Remember to use evidence from the text.
____________________________________
____________________________________
____________________________________
Simplify:
Explain how you found your answer.
Let’s make it better!
What do you think about this
Improving Validity
A. Clearly Stated?
example?
What do you think about this
Improving Validity
example?
B. Is prompt novel?
A. Clearly Stated?
What
are we trying to measure?
Simplify:
Simplify:
Explain how you found your answer.
D.
Measurable
Responses?
C.
Too many
options?
E.
Another
thought...
What do you think about this
Improving Validity
example?
Explain how you found your answer.
Let’s make it better!
What do you think about this
Improving Validity
example?
A. Clearly Stated?
Simplify Completely:
Read the story “Powder” by Tobias Wolff. In a
multi-paragraph essay, analyze the author’s work.
Explain how you found your answer, using words
and numbers, and label each step.
Let’s make it better!
What do you think about this
Improving Validity
A. Clearly Stated?
example?
B. Is prompt novel?
What
are we trying to measure?
C.
Too many options?
Read the story “Powder” by Tobias Wolff. In a
multi-paragraph essay, analyze the author’s work.
D.
Measurable
Responses?
E.
Another
thought...
What do you think about this
Improving Validity
example?
What do you think about this
Improving Validity
A. Clearly Stated?
example?
B. Is prompt novel?
What
are we trying to measure?
C.
Too many options?
Read the story “Powder” by Tobias Wolff. In a
multi-paragraph essay, analyze the author’s work.
D.
Let’s make it better!
Measurable
Responses?
What do you think about this
Improving Validity
example?
A. Clearly Stated?
C.
Read the story “Powder” by Tobias Wolff. In a
multi-paragraph essay, analyze the development of
the theme over the course of the story, including
how it emerges, is developed, and is supported by
specific details.
Too many options?
Read the story “Powder” by Tobias Wolff. In a
multi-paragraph essay, analyze the development of
the theme over the course of the story, including
how it emerges, is developed, and is supported by
specific details.
Let’s make it better!
Let’s make it better!
What do you think about this
Improving Validity
example?
What do you think about this
Improving Validity
example?
Read the story “Powder” by Tobias Wolff. In a
multi-paragraph essay, analyze the development of
the theme over the course of the story, including
how it emerges, is developed, and is supported by
specific details.
D.
Let’s make it better!
Measurable
Responses?
Write a rhythm in the space below. Make your
rhythm 4 measures long. Be creative.
What do you think about this
Improving Validity
A. Clearly Stated?
example?
B. Is prompt novel?
What
are we trying to measure?
C.
Measurable
Responses?
C.
Too many options?
Write a rhythm in the space below. Make your
rhythm 4 measures long. Be creative.
D.
What do you think about this
Improving Validity
A. Clearly Stated?
example?
E.
Another
thought...
Too many options?
Write a rhythm in the space below. Make your
rhythm 4 measures long. Be creative.
Let’s make it better!
What do you think about this
Improving Validity
example?
Write a rhythm in the space below. Begin with a
time signature and end with either a double bar line
or a repeat sign. The rhythm should be at least 4
measures long. The composition must include at
least 4 of the rhythms from the box below.
Question
Complexity
Question Writing Activity
Elementary Example
Standard: RL 2.1 Ask and answer such questions as who, what, where, when, why, and how to
demonstrate understanding of key details in a text.
And so the group of toys decided to put on
the play. Everyone learned their part.
Elephant was very scared he would forget
his lines, so he practiced and practiced.
Finally, the day of the performance arrived.
Sam opened the curtain. Sam’s and his
friends were worried. Where was everyone?
Had nobody come to watch?
Remember & Understand
Apply & Analyze
Evaluate & Create
Let’s make it better!
And so the group of toys
decided to put on the play.
Everyone learned their part.
Where Can I Get Tools?
Elephant was very scared
he would forget his lines, so
he practiced and practiced.
Finally, the day of the performance arrived.
Sam opened the curtain. Sam and his friends were
worried. Where was everyone? Had nobody come to
watch?
Remember & Understand
RL 2.1
&
RI 2.1
Apply & Analyze
Evaluate & Create
Identify explicit information: Analyze explicit information
Evaluate explicit information
Who did this (action)
first…?
Where does the story
take place…?
(Defend a position)Why
do you believe...?
Is there a better solution
to the character’s
problem...?
What does the author
mean by: “quote”...?
Which quote best
supports the reason for
______...?
Questions?
Twitter
@StudentGrowth
Facebook
Kids At The Core
Twitter
@StudentGrowth
Twitter
@StudentGrowth
Please Share Articles,
Ideas With Us!
Facebook
Kids At The Core
Facebook
Kids At The Core