CELEBRATE THE STRUGGLE:
…it is the Essential Component of RIGOR!
R
Rigor
4
Relevance
Results
Relationships
Making a Difference:
A Vertical Articulation of Thinking
“The difference between who you are and what you want
to be is what you do.”
~Bill Phillips
PROVIDING EDUCATIONAL EXCELLENCE FOR ALL STUDENTS
Begin with the Rigor!
Prepared for the Professional Learning Network of
NEWTON COUNTY SCHOOLS
by Dan Mulligan, Ed. D., flexiblecreativity.com
JANUARY 2016
RIGOR = TYPE of THINKING + DEPTH of THINKING
DEPTH OF KNOWLEDGE
Depth of THINKING
This body of work offers the Depth of Knowledge (DOK) model employed to analyze the
cognitive expectation demanded by standards, curricular activities, and assessment
tasks (Webb, 1997). The model is based upon the assumption that curricular elements
may all be categorized based upon the cognitive demands required to produce an
acceptable response. Each grouping of tasks reflects a different level of cognitive
expectation, or depth of knowledge, required to complete the task. It should be noted that
the term knowledge, as it is used here, is intended to broadly encompass all forms of
knowledge (i.e. procedural, declarative, etc.). The following table reflects an adapted
version of the model.
DOK Level
Title of Level
1
Recall & Reproduction
2
Basic Skills & Concepts
3
Strategic Thinking & Reasoning
4
Extended Thinking
As an administrator/teacher reflect on the level of thinking experienced by
students by the ‘conclusion’ of most unit learning in your school. Where are
you currently? Where do we want to be? How will you get there?
2
 RECALL AND REPRODUCTION
Curricular elements that fall into this category involve basic tasks that require students to recall or
reproduce knowledge and/or skills. The subject matter content at this particular level usually involves
working with facts, terms and/or properties of objects. It may also involve use of simple procedures
and/or formulas. There is little transformation or extended processing of the target knowledge
required by the tasks that fall into this category. Key words that often denote this particular level
include: list, identify and define. A student answering a Level 1 item either knows the answer or does
not; that is, the answer does not need to be “figured out” or “solved.”
POSSIBLE PRODUCTS
QUIZ
DEFINITION
FACT
WORKSHEET
TEST
LABEL
LIST
COLLECTION
PODCAST
WORKBOOK
EXPLANATION
REPRODUCTION
SHOW AND TELL
CATEGORIZING/
TAGGING
COMMENTING
VOCABULARY QUIZ
OUTLINE
BULLETING
RECITATION
BLOG
HIGHLIGHTING
EXAMPLE
WIKI
SOCIAL
NETWORKING
SOCIAL
BOOKMARKING
SEARCHING
GOOGLING
ROLES
STUDENT
TEACHER
DIRECTS
TELLS
RESPONDS
ABSORBS
SHOWS
EXAMINES
REMEMBERS
RECOGNIZES
QUESTIONS
EVALUATES
MEMORIZES
DESCRIBES
DEMONSTRATES
LISTENS
EXPLAINS
TRANSLATES
COMPARES
CONTRASTS
RESTATES
DEMONSTRATES
EXAMINES
INTERPRETS
NOTES TO SELF:
3
 BASIC SKILLS AND CONCEPTS
This level generally requires students to contrast or compare people, places, events and concepts;
convert information from one form to another; classify or sort items into meaningful categories ;
describe or explain issues and problems, patterns , cause and effect, significance or impact,
relationships, points of view or processes. A Level 2 “describe or explain” would require students to
go beyond a description or explanation of recalled information to describe or explain a result or “how”
or “why.” The learner should make use of information in a context different from the one in which it
was learned.
POSSIBLE PRODUCTS
PHOTOGRAPH
INTERVIEW
LINKING
SOLVE
MIND MAPS
PRESENTATION
BLOG COMMENTING
ILLUSTRATION
CRACKING CODES
REVERSEENGINEERING
BLOG REFLECTING
SIMULATION
PERFORMANCE
MODERATING
SCULPTURE
DEMONSTRATION
EXPLAIN
CATEGORIZE
GENERALIZE
TRANSLATE
ESTIMATE
JOURNAL
ROLES
STUDENT
TEACHER
SHOWS
FACILITATES
SOLVE PROBLEMS
OBSERVES
EVALUATES
CALCULATES
DEMONSTRATES USE OF
KNOWLEDGE
COMPILES
ORGANIZES
QUESTIONS
COMPLETES
ILLUSTRATES
CONSTRUCTS
NOTES TO SELF:
4
 STRATEGIC THINKING & REASONING
Items falling into this category demand a short-term use of higher order thinking processes, such as
analysis and evaluation, to solve real-world problems with predictable outcomes. Stating one’s
reasoning is a key marker of tasks that fall into this particular category. The expectation established
for tasks at this level tends to require coordination of knowledge and skill from multiple subject-matter
areas to carry out processes and reach a solution in a project-based setting. Key processes that
often denote this particular level include: analyze, explain and support with evidence, generalize, and
create.
POSSIBLE PRODUCTS
GRAPH
SPREADSHEET
CHECKLIST
EVALUATING
SURVEY
DEBATE
CONCLUDE
PODCAST
DATABASE
PANEL
PROGRAM
PUBLISHING
REPORT
FILM
CHART
ABSTRACT
ANIMATION
OUTLINE
INVESTIGATE
VIDEO CAST
ROLES
STUDENT
TEACHER
PROBES
GUIDES
DISCUSSES
UNCOVERS
OBSERVES
EVALUATES
DEBATES
THINKS DEEPLY
ACTS AS A RESOURCE
QUESTIONS
EXAMINES
QUESTIONS
ORGANIZES
DISSECTS
JUDGES
DISPUTES
CLARIFIES
ACCEPTS
ASSESSES
DECIDES
JUSTIFIES
TESTS
SELECTS
COMPARES
GUIDES
NOTES TO SELF:
5
 EXTENDED THINKING
Curricular elements assigned to this level demand extended use of higher order thinking processes
such as synthesis, reflection, assessment and adjustment of plans over time. Students are engaged
in conducting investigations to solve real-world problems with unpredictable outcomes. Employing
and sustaining strategic thinking processes over a longer period of time to solve the problem is a key
feature of curricular objectives that are assigned to this level. Key strategic thinking processes that
denote this particular level include: synthesize, reflect, conduct, and manage.
POSSIBLE PRODUCTS
FILM
PLAN
PROJECT
NEW GAME
NEWSPAPER
SONG
MEDIA PRODUCT
STORY
ROLES
STUDENT
TEACHER
FACILITATES
EXTENDS
DESIGNS
FORMULATES
REFLECTS
ANALYZES
TAKES RISKS
MODIFIES
PROPOSES
PLANS
EVALUATES
CREATES
NOTES TO SELF:
6
7
8
9
Rigor-Infused Math Inquiry: FACILITATING MATH-TALK
Questions I currently use to guide and encourage student thinking:
Questions I can add to my toolkit:
What then is effective feedback?
“The most powerful single
moderator that enhances
achievement is feedback”
Quality feedback should:
• focus on the learning intention of the task
John Hattie. University of Auckland. 1999 Inaugural
• occur as the students are doing the learning
• provide information on how and why the student understands and misunderstands
• provides strategies to help the student to improve
• assist the student to understand the goals of the learning
Key Research Findings







When feedback is corrective in nature—that is, it explains where and why students
have made errors--significant increases in student learning occur (Lysakowski &
Walberg, 1981, 1982; Walberg, 1999; Tennenbaum & Goldring, 1989).
Feedback has been shown to be one of the most significant activities a teacher can
engage in to improve student achievement (Hattie, 1992).
Asking students to continue working on a task until it is completed and accurate (until
the standard is met) enhances student achievement (Marzano, Pickering, & Pollock, 2001).
Effective feedback is timely. Delay in providing students feedback diminishes its value
for learning (Banger-Drowns, Kulik, Kulik, & Morgan, 1991).
Administer tests to optimize learning. Giving tests a day after a learning experience is
better than testing immediately after a learning experience (Bangert-Downs, Kulik,
Kulik, & Morgan, 1991).
Rubrics provide students with helpful criteria for success, making desired learning
outcomes clearer to them. Criterion-referenced feedback provides the right kind of
guidance for improving student understanding (Crooks, 1988; Wilburn & Felps, 1983).
Effective learning results from students providing their own feedback, monitoring their
work against established criteria (Trammel, Schloss, & Alper, 1994; Wiggins, 1993).
Additional Resources
RubiStar is a free online tool that teachers can use to make and save rubrics. Developed by the High Plains
Regional Technology in Education Consortium, RubiStar includes a tutorial for new users and a feature that
enables teachers to analyze student data and identify areas for focusing additional instruction.
http://rubistar.4teachers.org/index.php
The National Center for Research on Cultural Diversity and Second Language Learning has published an article
called The Instructional Conversation: Teaching and Learning in Social Activity. The authors discuss the use of
modeling, providing feedback, contingency management, directing, questioning, explaining, and task structuring in
classroom activity settings.http://www.ncela.gwu.edu/pubs/ncrcdsll/rr2.htm
11
12
14
Depth of THINKING
Depth of Knowledge (DOK) Levels for MATHEMATICS
DOK Level 1 (Recall) includes the recall of information such as fact, definition,
term, or a simple procedure, as well as performing a simple algorithm or applying a
formula. That is, in mathematics a one-step, well-defined, and straight algorithmic
procedure should be included at this lowest level. Other key words that signify a Level
1 include "identify," "recall," "recognize," "use," and "measure." Verbs such as
"describe" and "explain" could be classified at different levels depending on what is to
be described and explained.
Level 2 (Skill/Concept) includes the engagement of some mental processing
beyond a habitual response. A Level 2 assessment item requires students to make
some decisions as to how to approach the problem or activity, whereas Level 1
requires students to demonstrate a rote response, perform a well-known algorithm,
follow a set procedure (like a recipe), or perform a clearly defined series of steps.
Keywords that generally distinguish a Level 2 item include "classify," "organize,"
"estimate," "make observations," "collect and display data," and "compare data." These
actions imply more than one step. For example, to compare data requires first
identifying characteristics of the objects or phenomenon and then grouping or ordering
the objects. Some action verbs, such as "explain," "describe," or "interpret" could be
classified at different levels depending on the object of the action. For example, if an
item required students to explain how light affects mass by indicating there is a
relationship between light and heat, this is considered a Level 2. Interpreting
information from a simple graph, requiring reading information from the graph, also is a
Level 2.
Interpreting information from a complex graph that requires some decisions on what
features of the graph need to be considered and how information from the graph can
be aggregated is a Level 3. Caution is warranted in interpreting Level 2 as only skills
because some reviewers will interpret skills very narrowly, as primarily numerical skills,
and such interpretation excludes from this level other skills such as visualization skills
and probability skills, which may be more complex simply because they are less
common. Other Level 2 activities include explaining the purpose and use of
experimental procedures; carrying out experimental procedures; making observations
and collecting data; classifying, organizing, and comparing data; and organizing and
displaying data in tables, graphs, and charts.
16
Level 3 (Strategic Thinking) requires reasoning, planning, using evidence,
and a higher level of thinking than the previous two levels. In most instances, requiring
students to explain their thinking is a Level 3. Activities that require students to make
conjectures are also at this level. The cognitive demands at Level 3 are complex and
abstract. The complexity does not result from the fact that there are multiple answers,
a possibility for both Levels 1 and 2, but because the task requires more demanding
reasoning. An activity, however, that has more than one possible answer and requires
students to justify the response they give would most likely be a Level 3. Other Level 3
activities include drawing conclusions from observations; citing evidence and
developing a logical argument for concepts; explaining phenomena in terms of
concepts; and using concepts to solve problems.
Level 4 (Extended Thinking) requires complex reasoning, planning,
developing, and thinking most likely over an extended period of time. The extended
time period is not a distinguishing factor if the required work is only repetitive and does
not require applying significant conceptual understanding and higher-order thinking.
For example, if a student has to take the water temperature from a river each day for a
month and then construct a graph, this would be classified as a Level 2. However, if
the student is to conduct a river study that requires taking into consideration a number
of variables, this would be a Level 4. At Level 4, the cognitive demands of the task
should be high and the work should be very complex. Students should be required to
make several connections-relate ideas within the content area or among content
areas-and have to select one approach among many alternatives on how the situation
should be solved, in order to be at this highest level. Level 4 activities include
designing and conducting experiments; making connections between a finding and
related concepts and phenomena; combining and synthesizing ideas into new
concepts; and critiquing experimental designs.
17
Mathematics Item Depth of Knowledge Indicators
DOK 1
RECALL
Examples represent, but do not
constitute all Level 1 mathematics
performances:
 Recall or recognize a fact, ,definitions,
or term
 Apply a well-known algorithm
 Apply a formula
 Determine the area or perimeter of
rectangles or triangles given a drawing
and labels
 Identify a plane or three dimensional
figure
 Measure a length
 Perform a specified or routine
procedure
 Evaluate an expression
 Solve a one-step word problem
 Retrieve information from a table or
graph
 Recall, identify, or make conversions
between and among representations or
numbers (fractions, decimals, and
percents), or within and between
customary and metric measures
 Locate numbers on a number line, or
points on a coordinate grid
 Solves linear equations
 Represent math relationships in words,
pictures, or symbols
DOK 2
SKILL/CONCEPTS
DOK 3
STRATEGIC THINKING
DOK 4
EXTENDED THINKING
Examples represent, but do not
Examples represent, but do not
constitute all Level 2 mathematics constitute all Level 3
performances:
mathematics performances:
Examples represent, but do not
constitute all Level 4
mathematics performances:
 Classify plane and three
dimensional figures
 Interpret information from a
simple graph
 Use models to represent
mathematical concepts
 Solve a routine problem
requiring multiple steps, or the
application of multiple
 concepts
 Compare figures or statements
 Compare and contrast figures
 Provide justifications for steps
in a solution process
 Extend a pattern
 Retrieve information from a
table, graph, or figure and use
it solve a problem requiring
multiple steps
 Translate between tables,
graphs, words and symbolic
 notation
 Select a procedure according
to criteria and perform it
 Relate mathematical concepts
to other content areas
 Relate mathematical concepts
to real-world applications in
new situations
 Apply a mathematical model
to illuminate a problem,
situation
 Conduct a project that
specifies a problem, identifies
solution paths, solves the
problem, and reports results
 Design a mathematical model
to inform and solve a practical
or abstract situation
NOTE: Level 4 requires
applying one approach among
many to solve problems.
Involves complex restructuring
of data, establishing and
evaluating criteria to solve
problems.
 Interpret information from a
complex graph
 Explain thinking when more
than one response is possible
 Make and/or justify
conjectures
 Develop logical arguments for
a concept
 Use concepts to solve
problems
 Perform procedure with
multiple steps and multiple
decision points
 Generalize a pattern
 Describe, compare, and
contrast solution methods
 Formulate a mathematical
model for a complex situation
 Provide mathematical
justifications
 Solve a multiple- step
problem, supported with a
mathematical explanation that
justifies the answer
 Formulate an original
problem, given a situation
Using Depth of Knowledge
Following are some questions to consider when analyzing your curriculum tasks for DOK.
1. What level of work are the students most commonly required to perform?
2. What is the complexity of the task rather than the difficulty?
3. What are all the skills and knowledge scaffolding that the students will have already needed to build
to complete the task?
Examples of what Depth of Knowledge ‘sound-like’ in the classroom
Content Area
Elementary
Language Arts
Level 1
Sort known
words as quickly
as possible
Elementary
Math
Collect data on
the number of
teeth lost by
students in one
2nd grade
classroom.
Elementary
Interdisciplinary
List the
ingredients of a
peanut butter
and jelly
sandwich.
Level 2
Find words in text
that illustrate a
defined pattern.
Level 3
Create an open sort
and define the rule
and explain.
Organize these
Using the graph,
data using a grph
predict how many
or chart (e.g., a line teeth would be lost by
plot
all the 2nd grade
classes in the school
and justify your
answer
Collect the
Investigate how many
ingredients for a
people are coming to
peanut butter and
dinner and formulate
jelly sandwich and
the appropriate
write the recipe.
amounts of
ingredients for that
many people.
Level 4
Illustrate through authentic
writing stability in pattern and
content of identified stages.
Create a model to estimate
how many teeth are lost be 2nd
grade students in the United
States in one year. Include the
type of data needed to collect
and explain how your model
works.
Research and design a system
to lift heavy objects using the
conversion of electrical energy
to mechanical energy. Build a
prototype of the system using
materials found in the
classroom.
19
21
22
24
25
26
27
28
29
30
31
32
Personal Reflections
Dan Mulligan
Flexiblecreativity.com
33
34
My Reflections
35