Webb’s Depth of
Knowledge
June 25, 2015
IU 13 E-Learning Conference
Cindy H. Goldsworthy, Ed. D.
Evidence to Action: K-12 Consulting Services
[email protected]
Opening Activity: The Gettysburg Address
• Four score and seven years ago our fathers brought forth on this continent, a
new nation, conceived in Liberty, and dedicated to the proposition that all men
are created equal. Now we are engaged in a great civil war, testing whether that
nation, or any nation so conceived and dedicated, can long endure. We are met
on a great battle-field of that war. We have come to dedicate a portion of that
field, as a final resting place for those who here gave their lives that that nation
might live. It is altogether fitting and proper that we should do this. But, in a
larger sense, we can not dedicate -- we can not consecrate -- we can not hallow - this ground. The brave men, living and dead, who struggled here, have
consecrated it, far above our poor power to add or detract. The world will little
note, nor long remember what we say here, but it can never forget what they did
here. It is for us the living, rather, to be dedicated here to the unfinished work
which they who fought here have thus far so nobly advanced. It is rather for us to
be here dedicated to the great task remaining before us -- that from these
honored dead we take increased devotion to that cause for which they gave the
last full measure of devotion -- that we here highly resolve that these dead shall
not have died in vain -- that this nation, under God, shall have a new birth of
freedom -- and that government of the people, by the people, for the people,
shall not perish from the earth.
Welcome and Introductions
• What do you know already about
Webb’s Depth of Knowledge?
• What do you hope to learn as a result
of participation today’s session?
Today’s Planned Outcomes
• Define Rigor
• Develop understanding of what it means to
apply rigor to curriculum, instruction, &
assessment
• Explain Webb’s DOK
• Apply Webb’s Depth of Knowledge (DOK) in
the design of curriculum, instruction, &
assessment
But, I Know This Already!
“The mind is not a vessel to be
filled but a fire to be kindled.”
-On Listening to Lectures (Plutarch)
Is the need for “higher-order thinking
skills” really new?
PA Core Standards
• What’s changed?
• Increased rigor
• What does that mean?
• Focus on “college and career” ready
• What else?
• How does that impact student
outcomes?
So, What is Rigor?
•Write your definition of
“cognitive rigor” as it
applies to schools &
classrooms.
•One minute write (Collins,
Type 1 writing)
What Rigor is NOT!
• NOT just about asking higher level
questions
• NOT just about writing higher level
questions on quizzes and tests
Rigor C, A, & I!
• Rigor in curriculum, instruction, and
assessment is critical
• planning & developing curriculum
• planning & delivery of instruction
• Planning & developing assessments that
require higher levels of thinking
Disconnect…..
Instruction
Curriculum
Assessment
Measures of Rigor
• There are different models to address
cognitive rigor. They each measure
something different.
–Bloom: Hierarchy based on verbs
–Bloom Revised: Some verb changes
–Webb: How complex is the content? How
deeply do you have to understand the
content to successfully interact with it?
Bloom’s Taxonomy [1956 ] &
Bloom’s Cognitive Process Dimensions [2005]
Knowledge—Define, duplicate, label, list, name, order,
recognize, relate, recall
Remember—Retrieve knowledge from long-term
memory, recognize, recall, locate, identify
Comprehension—Classify, describe, discuss, explain,
express, identify, indicate, locate recognize, report,
review, select, translate
Understand—Construct meaning, clarify, paraphrase,
represent, translate, illustrate, give examples, classify,
categorize, summarize, generalize, predict…
Application—Apply, choose, demonstrate, dramatize,
employ, illustrate, interpret, practice, write
Apply—Carry out or use a procedure in a given situation;
carry out or use/apply to an unfamiliar task
Analysis—Analyze, appraise, explain, calculate,
categorize, compare, criticize discriminate, examine
Analyze—Break into constituent parts, determine how
parts relate
Synthesis—Rearrange, assemble, collect, compose,
create, design, develop, formulate, manage, write
Evaluate—Make judgments based on criteria, check,
detect inconsistencies/fallacies, critique
Evaluation—Appraise, argue, assess, choose, compare,
defend, estimate, explain, judge, predict, rate, core,
select, support, value
Create—Put elements together to form a coherent
whole, reorganize elements into new patterns/structures
Developing the Cognitive
Rigor Matrix (Hess)
Different states/schools/teachers use different models
to describe cognitive rigor. Each addresses
something different.
• Bloom –What type of thinking (verbs) is
needed to complete a task?
• Webb –How deeply do you have to
understand the content to successfully
interact with it? How complex is the content?
BLOOM’S TAXONOMY
KNOWLEDGE
“The recall of specifics and universals, involving
little more than bringing to mind the appropriate
material”
COMPREHENSION
“Ability to process knowledge on a low level such
that the knowledge can be reproduced or
communicated without a verbatim repetition.”
WEBB’S DOK
RECALL
Recall of a fact, information, or procedure (e.g.,
What are 3 critical skill cues for the overhand
throw?)
SKILL/CONCEPT
APPLICATION
“The use of abstractions in concrete situations.”
ANALYSIS
“The breakdown of a situation into its component
parts.”
SYNTHESIS AND EVALUATION
“Putting together elements & parts to form a
whole, then making value judgments
about the method.”
Use of information, conceptual knowledge,
procedures, two or more steps, etc.
STRATEGIC THINKING
Requires reasoning, developing a plan or
sequence of steps; has some complexity; more
than one possible answer
EXTENDED THINKING
Requires an investigation; time to think and
process multiple conditions of the problem
or task.
Wyoming School Health and Physical Education Network (2001). Standards,
Assessment, and Beyond. Retrieved May 25, 2006, from
http://www.uwyo.edu/wyhpenet
The Hess Cognitive Rigor Matrix: Applies Webb’s DOK to Bloom’s
Cognitive Process Dimensions
Depth +
thinking
Level 1
Recall & Reproduction
Level 2
Skills & Concepts
Level 3
Strategic Thinking/
Reasoning
Level 4
Extended Thinking
Remember
-Recall, locate basic facts,
details, events
Understand
-Select appropriate words to
use when intended meaning
is clearly evident
-Specify or explain relationships
-summarize
-identify central idea
-Explain, generalize, or
connect ideas using
supporting evidence (quote,
example…)
-Explain how concepts or
ideas specifically relate to
other content domains or
concepts
-Use language structure
(pre/suffix) or word
relationships
(synonym/antonym) to
determine meaning
-Use context to identify meaning
of word
-Obtain and interpret
information using text features
-Use concepts to solve nonroutine problems
-Devise an approach
among many alternatives to
research a novel problem
-Identify whether
information is contained in a
graph, table, etc.
-Compare literary elements,
terms, facts, events
-analyze format, organization, &
text structures
-Analyze or interpret author’s
craft (literary devices,
viewpoint, or potential bias) to
critique a text
-Analyze multiple sources
-Analyze complex/abstract
themes
-Cite evidence and develop a
logical argument for
conjectures
-Evaluate relevancy,
accuracy, & completeness
of information
-Synthesize information within
one source or text
-Synthesize information
across multiple sources or
texts
Apply
Analyze
Not appropriate at this level
Evaluate
Create
-Brainstorm ideas about a
topic
-Generate conjectures based on
observations or prior knowledge
Where did DOK come from?
• Developed by Dr. Norman Webb,
senior research scientist at the
National Institute for Science
Education. Several other states (at
least 20) use DOK to evaluate the
rigor of their state assessments.
Webb’s DOK
Depth of Knowledge is broken into 4 levels.
Level One is the most basic level, essentially
the “definition” stage. As the levels
increase, students must demonstrate
increasingly complex mental strategies.
Higher levels of DOK require that students
solve problems in new and creative ways,
and allow for multiple solutions to solve
those problems.
Levels of DOK
• LEVEL ONE - RECALL
• Recall of a fact, information, or procedure
• LEVEL TWO – SKILL/CONCEPT
• Use information or conceptual knowledge
• LEVEL THREE – STRATEGIC THINKING
• Reasoning, developing a plan
• LEVEL FOUR – EXTENDED THINKING
• Requires an investigation, collection of data and analysis of
results
Webb’s Depth of Knowledge (DOK)
DOK Level 1:
Recall and Reproduction
• Requires recall of
information, such as a fact,
definition, term, or
performance of a simple
process or procedure
• Answering a Level 1 item can
involve following a simple,
well-known procedure or
formula
Recall and Reproduction
DOK Level 1: Examples
• List animals that survive by eating other
•
•
•
•
•
animals
Locate or recall facts found in text
Describe physical features of places
Determine the perimeter or area of
rectangles given a drawing or label
Identify elements of music using music
terminology
Identify basic rules for participating in
simple games and activities
Skills/Concepts: DOK Level 2
• Includes
the engagement of some mental
processing beyond recalling or reproducing
a response
• Items require students to make some
decisions as to how to approach the
question or problem
• Actions imply more than one mental or
cognitive process/step
Skills/Concepts:
DOK 2 Examples
• Compare desert and tropical environments
• Identify and summarize the major events, problems,
solutions, conflicts in literary text
• Explain the cause-effect of historical events
• Predict a logical outcome based on information in a
reading selection
• Explain how good work habits are important at
home, school, and on the job
• Classify plane and three dimensional figures
• Describe various styles of music
Strategic Thinking: Level 3
• Requires deep understanding exhibited
through planning, using evidence, and
more demanding cognitive reasoning
• The cognitive demands are complex and
abstract
• An assessment item that has more than
one possible answer and requires students
to justify the response would most likely be
a Level 3
DOK Level 3: Strategic Thinking
Examples:
• Compare
consumer actions and analyze how
these actions impact the environment
• Analyze or evaluate the effectiveness of
literary elements (e.g., characterization, setting,
point of view, conflict and resolution, plot
structures)
• Solve a multiple-step problem and provide
support with a mathematical explanation that
justifies the answer
DOK Level 3 Examples
Develop a scientific model for a complex
idea
• Propose and evaluate solutions for an
economic problem
• Explain, generalize or connect ideas, using
supporting evidence from a text or source
• Create a dance that represents the
characteristics of a culture
•
•
Extended Thinking: Level 4
•Requires high cognitive demand and is
very complex
• Students are expected to make
connections, relate ideas within the
content or among content areas, and
select or devise one approach among
many alternatives on how the situation
can be solved
• Due to the complexity of cognitive
demand, DOK 4 often requires an
extended period of time
“Extending the length of an activity
alone does not necessarily create
rigor!”
Some Rules of Thumb…
• One correct answer – DOK 1
• DOK 1: Either you know it, or you don’t
• Apply one concept, then apply second
concept – DOK 2
• If more than one solution/approach,
requiring evidence, it is DOK 3 or 4
• DOK 3: Must provide supporting evidence and
reasoning (not just HOW solved, but WHY – explain
reasoning)
• DOK 4: all of “3” + use of multiple sources or texts
Gettysburg Address
• DOK 1: What date is Lincoln referring to as “four score
and seven years ago”?
• DOK 2: What’s the connection between four score
reference (the date) and the argument Lincoln is
making?
• DOK 3: Lincoln’s address lays out the thesis and
argument in support of equality. Analyze his
reasoning and the evidence he uses to advance his
point of view.
• DOK 4: Draw from additional related services to write
a research paper to discuss the role war plays in
nation building.
http://www.stancoe.org/SCOE/iss/common_c
ore/overview/overview_depth_of_knowledge.
htm
Revisiting YOUR Question
• What DOK level is your question?
• Write a second question at a
higher level (or lower level).
It’s NOT about the verb...
The Depth of Knowledge is
NOT determined by the verb
(Bloom’s Taxonomy), but by the
context in which the verb is
used and the depth of thinking
required.
Verbs do not always indicate level
of complexity!
Words like explain or analyze have to
be considered in context.
• “Explain to me where you live” does
not raise the DOK of a simple rote
response.
• Even if the student has to use
addresses or landmarks, the student is
doing nothing more than recalling and
reciting.
DOK is about what follows the verb...
What comes after the verb is more
important than the verb itself.
“Analyze this sentence to decide if the
commas have been used correctly” does
not meet the criteria for high cognitive
processing.
The student who has been taught the rule
for using commas is merely using the rule.
Same Verb—Three Different DOK Levels
DOK 1- Describe three characteristics of
metamorphic rocks. (Requires simple recall)
DOK 2- Describe the difference between
metamorphic and igneous rocks. (Requires
cognitive processing to determine the differences
in the two rock types)
DOK 3- Describe a model that you might use to
represent the relationships that exist within the
rock cycle. (Requires deep understanding of rock
cycle and a determination of how best to represent
it)
DOK is about intended outcome, not difficulty
DOK is a reference to the complexity of mental
processing that must occur to answer a question,
perform a task, or generate a product.
• Adding is a mental process.
• Knowing the rule for adding is the intended
outcome that influences the DOK.
• Once someone learns the “rule” of how to add, 4
+ 4 is DOK 1 and is also easy.
• Adding 4,678,895 + 9,578,885 is still a DOK 1
but may be more “difficult.”
DOK is not about difficulty...
•Difficulty is a reference to how many students answer a
question correctly.
“How many of you know the definition of exaggerate?”
DOK 1 – recall
If all of you know the definition, this question is an easy
question.
“How many of you know the definition of prescient?”
DOK 1 – recall
If most of you do not know the definition, this question is a
difficult question.
DOK is about complexity
• The
intended student learning
outcome determines the DOK level.
• Instruction and classroom
assessments must reflect the DOK
level of the learning target(s).
Quick Quiz
11) Give an example of a statement that uses a verb
that “sounds” like a high DOK but is used
inappropriately.
2) Make this statement True: What comes before
the verb is more important than the verb itself in
determining the DOK level.
3) What is the difference between difficulty and
complexity?
4) What really determines the DOK level?
• If 10-20% of the questions on “assessment
X” are low Level of Complexity...How much
class time would we devote to DOK Level 1
thinking?
• If 80% of the question on
PSSA/Keystone/Final require moderate to high
levels of complexity....What are we doing to
promote these complex levels of higher order
thinking?
Remember DOK is...
…descriptive
…focuses on how deeply a
student has to know the
content in order to
respond
…NOT the same as difficulty.
…NOT the same as Bloom’s
Taxonomy
Building Classroom Assessments
• DOK 2-4 objectives require assessments
with DOK 2-4 items.
• If a student makes 80% on a classroom test,
what do we have to know about that
assessment before we can make a
judgment about the student’s
performance?
• If the test has 10 items and only 3 are DOK
2-4 items, what do we know about the
student’s performance?
Take-Away Message: Some Implications
• Assessing only at the highest DOK level will miss
opportunities to know what students do &
don’t know – go for a range; end “high” in
selected/prioritized content
• Performance assessments can offer varying
levels of DOK embedded in a larger, more
complex task
• Planned formative assessment strategies and
tools can focus on differing DOK levels
Related Resources
• Hess (2004). Applying Webb’s Depth-of-Knowledge (DOK) Levels in reading,
writing, math, science, social studies, science [online] available:
• http://www.nciea.org/publications/DOKreading_KH08.pdf
http://www.nciea.org/publications/DOKsocialstudies_KH08.pdf
http://www.nciea.org/publications/DOKwriting_KH08.pdf
http://www.nciea.org/publications/DOKscience_KH08.pdf
http://www.nciea.org/publications/DOKmath_KH08.pdf
• Hess & Biggam (2004). A Discussion of "Increasing Text Complexity“[online]
available: http://www.nciea.org/publications/TextComplexity_KH05.pdf
• Hess (2006). Linking Formative Assessment Approaches to Instructional
Decisions http://www.nciea.org/publications/RILS_KH06.pdf
• Hess (2008). Teaching and Assessing Understanding of Text Structures across
Grades. [online] available:
http://www.nciea.org/publications/TextStructures_KH08.pdf
Thank-you!
Cindy H. Goldsworthy, Ed. D.
Evidence to Action: K-12 Consulting Services
[email protected]
717 649-8139