Thinking more deeply or just working harder? Infusing Rigor With High Quality Assessment WELCOME! Dr. Karin Hess
Adjunct Faculty
[email protected]
www.karin-hess.com
Follow us on Twitter!
@KevinEBaird
Contact: [email protected]
Request Information for Free Programs
[email protected]
www.CollegeCareer.org
Session Overview •  Develop a shared understanding of the concept of cogni=ve rigor •  Dispel some common DOK misconcep=ons •  Use DOK & the Hess Cogni=ve Rigor Matrices to: –  Examine what rigor/DOK looks like in ac=on •  Sample videos, assessments/tasks, & rubrics •  Karin’s Coaching Tips: Analyzing assessment tasks/
rubrics & instruc=on in light of current research •  Provide tools & strategies for current & future work Track your reflec=ons as we work… Ways I am refining my thinking about DOK/
rigor… •  ? •  ? Scaffolding strategies for geVng students to deeper thinking… •  ? •  ? Before we begin… •  Take a minute to jot down 2-­‐3 words/
phrases that come to mind when you think of “cogni=ve rigor” as it relates to instruc=on, learning, and/or assessment Let’s apply your rigor defini=ons Your class has just read some version of Li#le Red Riding Hood. •  What is a basic comprehension ques=on you might ask? •  What is a more rigorous ques=on you might ask? What must you consider when developing each type of ques=on???? 6 The Hess Cogni=ve Rigor Matrix integrates Bloom + Webb Different states/schools/teachers use different models to describe cogni=ve rigor. Each addresses something different. •  Bloom – What type of thinking (verbs) is needed to complete a task? •  Webb – How deeply do you have to understand the content to successfully interact with it? How complex is the content? 7 Bloom’s Taxonomy [1956] & Bloom’s CogniKve Process Dimensions [2001] Knowledge -- Define, duplicate,
label, list, name, order, recognize,
relate, recall
Remember Retrieve knowledge from
long-term memory, recognize, recall,
locate, identify
Comprehension -- Classify, describe,
discuss, explain, express, identify,
indicate, locate, recognize, report,
review, select, translate
Understand -- Construct meaning,
clarify, paraphrase, represent,
translate, illustrate, give examples,
classify, categorize, summarize,
generalize, predict…
Application -- Apply, choose,
demonstrate, dramatize, employ,
illustrate, interpret, practice, write
Apply -- Carry out or use a procedure
in a given situation; carry out or use /
Analysis -- Analyze, appraise, explain
calculate, categorize, compare,
criticize, discriminate, examine
Analyze -- Break into constituent
Synthesis -- Rearrange, assemble,
collect, compose, create, design,
develop, formulate, manage, write
Evaluate -- Make judgments based
on criteria, check, detect
inconsistencies/fallacies, critique
Evaluation -- Appraise, argue,
assess, choose, compare, defend,
estimate, explain, judge, predict, rate,
core, select, support, value
Create -- Put elements together to
form a coherent whole, reorganize
elements into new patterns/
structures
apply to an unfamiliar task
parts, determine how parts relate
8 Webb’s Depth-­‐of-­‐Knowledge Levels •  DOK-­‐1 – Recall & ReproducKon -­‐ Recall of a fact, term, principle, concept, or perform a rou=ne procedure •  DOK-­‐2 -­‐ Basic ApplicaKon of Skills/Concepts -­‐ Use of informa=on, conceptual knowledge, select appropriate procedures for a task, two or more steps with decision points along the way, rou=ne problems applying 2+ concepts, organize/display data, interpret/use simple graphs •  DOK-­‐3 -­‐ Strategic Thinking -­‐ Requires reasoning, developing a plan or sequence of steps to approach problem; requires some decision making and jusKficaKon; abstract, complex, or non-­‐
rou=ne; oien more than one possible answer or approach •  DOK-­‐4 -­‐ Extended Thinking -­‐ An original inves=ga=on or applica=on to real world; requires =me to research, problem solve, and process mulKple condi=ons of the problem or task; non-­‐rou=ne manipula=ons, across disciplines/content areas/
mul=ple sources 9 DOK Misconcep=on #1: All kids can’t do this; or Kids don’t need scaffolding to get “up” there. Engaging in “a complex task” with supports/ scaffolding is an essen=al step along the way to proficiency (Vygotsky’s ZPD) –  Do it with others first; DOK 3 and 4 are not meant to only be done alone/independently, especially at first –  Oral language & meaningful discourse support deeper thinking and increase ini=al exposures to the content and student engagement. This is NOT chea=ng! –  One strategy: Plan ques=oning & forma=ve probes from DOK 1-­‐2-­‐3-­‐4 over the course of a lesson or unit of study. Consider all DOK levels in your planning. 10 11 Vygotsky: Zone of Proximal Development
(What a child can do with assistance today)
What a child can
do independently
now: “ENTRY”
Actual
Development
Area
The
ZONE
What a child can
do independently
tomorrow/future
Potential
Development
Area
LEARNING PROGRESSIONS ZONE:
Dynamic area
Causes development to move forward
Social interaction essential (scaffolding)
Karin Hess (2008). Using learning progressions as a schema to monitor progress across grades. 12 DOK Misconcep=on #2:
Webb’s DOK model is a taxonomy •  Bloom’s is a taxonomy, intended to be a hierarchy •  Primary Weaknesses of Bloom: generic verbs (void of content); some of the same verbs at different levels •  Webb’s DOK model is nomina8ve: –  It names how you interact with content; is content-­‐
specific –  It differen=ates varying levels of engagement with content and suggests what tasks might look like –  DOK 4 is not berer than DOK 3, or DOK 2, or DOK 1 13 DOK Misconcep=on #3: Bloom verbs & levels = Webb DOK The DOK “Wheel of Misfortune” implies that a DOK level is
indicated by a particular verb or set of verbs.
Norman Webb, “It’s what comes a`er the verb, that indicates the complexity of a task (content).” 14 What does (cogni=ve) rigor mean? 15 DOK Misconcep=on #4: DOK is about difficulty •  The intended student learning outcome (not grade level) determines the DOK level. What mental processing must occur? DOK = Complexity, not difficulty! •  While verbs may appear to point to a DOK level, it is what comes aier the verb that is the best indicator of the rigor/
DOK level and complexity of the task. –  Describe the informa=on contained in graphics or data tables in the text; or the rule for rounding a number –  Describe how the two story characters are alike and different –  Describe the data or text evidence that supports your solu=on, reasoning, or conclusions –  Describe varying perspec=ves on global climate change using suppor=ng scien=fic evidence, and iden=fy the most significant effects it might have on the planet in 100 yrs 16 DOK 2 Skills and Concepts DOK LEVEL 1 Recall and ReproducKon Remember Recall, locate basic facts, definiKons, Understand
details and events Apply DOK LEVEL 3 Reasoning Analyze Evaluate DOK 4 Extended Thinking Create Explain relaKonships Summarize Central ideas Explain, generalize or connect ideas using supporKng evidence (quote, text, evidence, data, etc.) Explain how concepts relate to other content domains Use language structure or word relaKonships (synonyms/ antonyms) Use context to find meaning Obtain and use informaKon in text features Use concepts to solve non-­‐rouKne problems and jusKfy soluKons with evidence Devise an approach among alternaKves to research a novel problem IdenKfy informaKon in a graphic, table, visual, etc. Compare literary elements, facts, terms and events Analyze format, organizaKon & text structures Analyze or interpret author’s cra` (e.g., literary devices, viewpoint, or potenKal bias) to criKque a text Analyze mulKple sources or texts Analyze complex abstract themes Cite evidence and develop a logical argument for conjectures based on one text or problem Evaluate relevancy, accuracy and completeness of informaKon Develop a complex model or approach for a given situaKon Develop an alternaKve soluKon Synthesize informaKon across mulKple sources ArKculate a new voice, theme, perspecKve Select appropriate word when intended meaning is clear “UGs” Brainstorm ideas, concepts, problems, or perspecKves related to a topic Generate conjectures or hypotheses based on observaKons or prior knowledge Bloom’s Taxonomy + Webb’s DOK = the Hess CRM 17 17 The Hess Cogni=ve Rigor Matrix Applies Webb’s DOK to Bloom’s Cogni=ve Process Dimensions Depth +
Thinking
Level 1
Recall &
Reproduction
Level 2
Skills &
Concepts
Level 3
Strategic
Thinking
Level 4
Extended
Thinking
Remember
- Recall, locate basic
facts, details, events
Understand
- Select appropriate
words to use when
intended meaning is
clearly evident
- Specify, explain
relationships
- summarize
– identify main ideas
- Explain, generalize,
or connect ideas using
supporting evidence
(quote, example,
data …)
- Explain how
concepts or ideas
specifically relate to
other content
domains or concepts
Apply
- Use language structure
(pre/suffix) or word
relationships (synonym/
antonym) to determine
meaning
– Use context to identify
meaning of word
- Obtain and interpret
information using text
features
- Use concepts to
solve non-routine
problems
- Devise an approach
among many
alternatives to
research a novel
problem
Analyze
- Identify whether
information is contained
in a graph, table, text
feature, etc.
– Compare literary
elements, terms, facts,
events
– Analyze format,
organization, & text
structures
- Analyze or interpret
author’s craft (literary
devices, viewpoint, or
potential bias) to
critique a text
– Analyze multiple
sources
- Analyze complex/
abstract themes
– Cite evidence and
develop a logical
argument for
conjectures
- Evaluate relevancy,
accuracy, &
completeness of
information
- Synthesize
information within
one source, data set,
or text
- Synthesize
information across
multiple sources or
texts
Evaluate
Create
“UG” = unsubstantiated
generalization
- Brainstorm ideas about
a topic
- Generate conjectures
based on observations or
prior knowledge
18 DOK Misconcep=on #5: All DOK levels can be assessed with a mul=ple choice ques=on •  That’s just dumb! •  “Weak” DOK 3 mul=ple choice items are possible; but does selec=ng the best op=on (e.g., locate suppor=ng evidence for a theme) provide as much insight as seeing HOW a student formulates and reveals thinking? •  By their nature, DOK 3 and 4 ques=ons/tasks are more open-­‐ended, generally take longer to respond to/solve, and may have more than one “appropriate right answer” 19 2. The DOK
Instruction
& Assessment
Matrix Instructional
Decisions…
Paths
Selected Response Each standard has an assigned Depth
Performance of Knowledge.
Constructed Response Tasks/Projects DOK 2 DOK 1 Skills and Concepts Recall and ReproducKon Remember Recall, locate basic facts, definiKons, details, events Understand Select appropriate words for use when intended meaning is clearly evident. The DOK determines the cognitive
level of instruction.
DOK 4 DOK 3 Extended Thinking Reasoning and Thinking Explain relaKonships Summarize State central idea Explain, generalize or connect ideas using supporKng evidence (quote, evidence, data) Use context for word meanings Use informaKon using text features Apply Analyze . Analyze or interpret author’s cra` (e.g., literary devices, viewpoint, or potenKal bias) to criKque a text Cite evidence and develop a logical argument for conjectures based on one text or problem Evaluate Create Use concepts to solve non-­‐rouKne problems and jusKfy . Develop a complex model or approach for a given situaKon Develop an alternaKve soluKon -­‐Explain how concepts or ideas specifically relate to other content domains. Devise an approach among many alternaKves to research a novel problem Analyze mulKple sources or mulKple text Analyze complex abstract themes Evaluate relevancy, accuracy and completeness of informaKon across texts or sources Synthesize across mulKple sources/ texts ArKculate a new voice, theme, or perspecKve 20 20 Back to Li#le Red Riding Hood … using the CRM •  Your sample quesKons – a basic and more rigorous ques=on •  Linking Research with PracKce, Tool #1: CRM template for Close Reading [www.karin-­‐hess.com] 21 Level 1 Level 2 Recall & ReproducKon Skills & Concepts Depth + Thinking Remember What color was Red’s cape? Who is this story about? Understand Who are the main characters? What was the story’s seVng? Retell or summarize the story in your own words. Iden=fy words/phrases that helped you to know the sequence of events in the story. Is this a realis=c or fantasy story? Compare the wolf character to the character of Red. How are they alike-­‐
different? Apply Analyze Evaluate Create Write text messages between Red & her mother explaining the wolf incident. Level 3 Strategic Thinking/ Reasoning Level 4 Extended Thinking What is the author’s message or theme? Jus=fy your interpreta=on using text evidence. Given when this was wriren, what social context is applied by the author? Is this a realis=c or fantasy story? Jus=fy your interpreta=on using text evidence. Are all wolves (in literature) like the wolf in this story? Support your response using evidence from this and other texts. What is your opinion about the cleverness of the wolf? Jus=fy your opinion using text evidence. Which version’s ending has the most emoKonal impact? (establish criteria first, then locate evidence) 22 Some other content examples… •  Your class will be learning about… –  FracKons/decimals; data use –  Data use; scienKfic invesKgaKon –  Elements of art & principles of design •  Come up with a basic understanding and more rigorous ques=on you could pose •  Use a CRM Template to Plan Units 23 Depth +
Thinking
Level 1
Recall &
Reproduction
Remember
What is slope? What is negaKve space? Understand
Read, write, and represent these fracKons Describe why negaKve space is used. Apply Convert fracKon to a decimal Add these fracKons Analyze
What kind of graph or model is this? Which data point shows ____? Is this a … Level 2
Skills & Concepts
Level 3
Explain how you
solved this problem.
Why control
variables in the
investigation?
Find examples… Construct a math argument to show equivalence using area, set, and linear models Interpret the theme. Organize these data
to support your
solution (e.g., graph,
diagram)
Conduct the invesKgaKon, interpret results, and support conclusions with data Design & conduct an invesKgaKon, based on a new quesKon raised. Compare these methods/mediums. Which graph shows how the data would be displayed? Interpret what was happening in the event? JusKfy your interpretaKon using what you know about slope. Are there flaws the design of the invesKgaKon? Analyze more than one product (same Kme period, medium, theme drawing from mul+ple contexts source materials for the analyses
How would you rank these ___? JusKfy your rankings using data that supports your criteria. Some say the NFL sevlement for player brain injury is not adequate. Evaluate both sides using data to determine the validity of this claim. Evaluate UG -­‐ Which team, ar=st, play, etc. … is the best? Create
How would you demonstrate this technique? Create a card game using fracKons. Create scenario explained by a data display. Strategic Thinking/
Reasoning
Level 4
Extended Thinking
Integrate mulKple source materials with intent to develop a novel product 24 DOK Misconcep=on #6:
Higher order thinking = deeper learning •  What we have thought of as “higher order” (analysis, evalua=on, crea=ve) thinking might only be engaging or fun…and not always deeper •  Many cri=cal thinking examples do not go deep or ever get to DOK 3 or 4 (e.g., interpret/solve and jus=fy) •  Shii our thinking from “higher order” to deeper learning, and that can mean: –  deeper understanding –  deeper applica=on –  deeper analysis, etc. The Hess CRM illustrates this
shift
25 Some general rules of thumb… •  If there is one correct answer, it is probably level DOK 1 or DOK 2 –  DOK 1: you either know it (can recall it, locate it, do it) or you don’t know it –  DOK 2 (conceptual): apply one concept, then make a decision before going on applying a second concept; express rela=onship (if-­‐then; cause-­‐effect), making connec=ons; HOW I did it •  If more than one answer/approach, requiring evidence, it is DOK 3 or 4 –  DOK 3: Must interpret, provide suppor=ng evidence and reasoning (not just HOW solved, but WHY it works– explain reasoning for each step/decision made) –  DOK 4: all of “3” + use of mul=ple sources/data/ texts; ini=ate & complete an inves=ga=on 26 DOK Misconcep=on #7: Mul=-­‐step or longer tasks, mul=ple texts, or complex texts always means deeper thinking •  DOK 2 is not simply more than one step, it’s applying more than one concept; DOK 2 is s=ll rou=ne/typical (main idea, word problems, etc.) •  Simply reading more complex texts, but NOT delving deeply into the text’s meaning/style etc., is likely to s=ll be DOK 1 or 2 •  DOK 3 requires some aspect of open-­‐
endedness and interpreta=on with jus=fica=on or support; DOK 4 = interpreta=on, draws from mul=ple sources •  DOK is not cumula=ve! More = Deep 27 Time to Reflect & Connect… Ways I am refining my thinking about DOK/
rigor… •  ? •  ? Scaffolding strategies for geVng students to deeper thinking… •  ? •  ? How does your current understanding relate to: Level of Thinking? Level of Performance? Level of Autonomy? 28 Where is more-­‐less rigor implied in any set of curriculum standards? 29 Reading & Wri=ng Depth +
Thinking
Level 1
Recall &
Reproduction
Level 2
Skills &
Concepts
Level 3
Strategic
Thinking
Level 4
Extended
Thinking
Remember
KEY DETAILS MAKE LISTS Understand
KEY DETAILS WORD MEANINGS-­‐ fill in WRITE SENTENCES, FACT STATEMENTS CENTRAL IDEAS SUMMARIZE predict, infer REASONING & SUPPORT – DEVELOP THEME, OR PT OF VIEW/ PERSPECTIVE TOPIC REASONING & SUPPORT – USE MULTIPLE TEXTS -­‐ COMPARE OR ELABORATE WORD STRUCTURE RELATIONSHIPS EDIT/CLARIFY USE TECHNOLOGY CITE SOURCES WORD MEANINGS-­‐
LANGUAGE USE USE OF TEXT STRUCTURES & FEATURES INTEGRATE TEXT STRUCTURES & FEATURES INTO COMPOSITIONS REASONING & SUPPORT USE of TEXT STRUCTURES or FEATURES LANGUAGE USE ANALYSIS & REASONING WITHIN TEXTS – RESEARCH ANALYSIS & REASONING ACROSS TEXTS FOR WRITING DEVELOP REASONING RESEARCH FOR WRITING; COMPARING THEMES AUTHOR’s CRAFT WITHIN A TEXT EVALUATE AUTHOR’s PURPOSE or CRAFT ACROSS TEXTS COMPOSE /REVISE FULL TEXTS COMPOSE FULL TEXTS-­‐SOURCES Apply
Analyze Evaluate Create
EVALUATE CREDIBILITY OF SOURCES WRITE/EDIT BRIEF TEXTS 30 Math Content Standards vs. Math Prac=ces Depth +
Thinking
Level 1
Recall &
Reproduction
Level 2
Skills & Concepts
Level 3
Strategic Thinking
(support with data, equaKons, models, etc.)
Level 4
Extended
Thinking (cross
domains)
Remember Know math facts, terms, principles Understand Avend to precision Evaluate expressions, locate/plot points Represent math relaKonships Apply Calculate, measure, make conversions Use formulas Analyze IdenKfy a pavern Locate informaKon in table, diagram Evaluate Create
Model with mathemaKcs Construct viable arguments Integrate concepts across domains or content areas Make sense of rouKne problems Make sense of non-­‐
rouKne problems Design & conduct a project Use tools strategically Classify, organize data, extend a pavern Reason abstractly Generalize a pavern Collect and analyze mulKple sources of data/ evidence EsKmate, predict, observe, explain relaKonships Geometry proof CriKque the reasoning of others Design a model for a new perspecKve Design a complex model with mulKple constraints 31 For each assessment task (and rubric)…ask •  What is its purpose? (What understandings -­‐ content/skill-­‐are being assessed? is there a ‘right’ answer?) •  What is the implied/intended rigor? (What mental processing would you expect students to engage in? Use the CRM Tools to find descriptors.) •  Which standards does it REALLY elicit and assess? (content + intended rigor) •  Does the scoring guide/rubric match content + intended rigor? •  What would student responses tell a teacher if students could/
could not do all or part of this task? (open-­‐ended tasks, reasoning used) – next instruc=onal decisions are clear Hess FA Tool: What will this assessment uncover? Hess Tool 9A: Assessment ValidaKon Protocol Hess Tool: Student Work Analysis Hess Tool: Looking for Rigor – ObservaKon & ReflecKon 32 Take-­‐Away Message: Cogni=ve Rigor & Some Implica=ons for Assessment •  Begin with daily DOK3 classroom discourse! •  Assessing only at the highest DOK level (the “ceiling”) will miss opportuni=es to know what students do & don’t know – go for a range; end “high” in selected/priori=zed content •  Performance assessments can offer varying levels of DOK embedded in a larger, more complex (“rich”) tasks •  Planned forma=ve assessment strategies and tools can/should focus on differing DOK levels 33 Some resources worth exploring… • 
• 
• 
• 
• 
• 
• 
• 
hrp://schools.nyc.gov/Academics/CommonCoreLibrary/default.htm -­‐ New York City Department of Ed website – Common Core library: preK-­‐12 assessments in ELA and mathema=cs, some of which Karin helped to develop and pilot www.nms.org – na=onal math science ini=a=ve – gr 3 – HS; also has ELA and SS performance tasks with DOK designa=ons www.readworks.org – gr k-­‐8; short literary & informa=onal texts with CC Qs hrp://www.louisianabelieves.com/resources/library/common-­‐core-­‐state-­‐standards -­‐ LA gr 3-­‐HS math & ELA tasks (some math samples are weak and not all math DOK levels are correct, but generally good examples) hrp://ccgpsmathema=csk-­‐5.wikispaces.com/K-­‐5+Forma=ve+Assessment+Lessons+
%28FALs%29-­‐ GA gr K-­‐5 math www.exemplars.com– sample math PAs for k-­‐12; science PAs for k-­‐8 Dan Meyer blog – math PAs for MS-­‐HS; kids have to build the problems by deciding what’s needed to solve them – good strategic thinking required Links and resources on www.karin-­‐hess.com: –  Norman Webb video clip – on how he developed the DOK model –  Karin’s YouTube video – “Text Complexity Tools” (qualita=ve text analysis) –  Karin’s blog, newslerer, classroom poster set -­‐ “Rock the Rigor” & other many other resources 34 THANK YOU! Dr. Karin Hess
Adjunct Faculty
[email protected]
www.karin-hess.com
Follow us on Twitter!
@KevinEBaird
Contact: [email protected]
Request Information for Free Programs
[email protected]
www.CollegeCareer.org