Content Analysis Workshop
Minneapolis, MN
Nov 7-9, 2007
Describing the Content of
Standards & Assessments
John L. Smithson, Director, Measures of the Enacted Curriculum
Alissa Minor, Projects Manager, Measures of the Enacted Curriculum
Wisconsin Center for Education Research, University of Wisconsin-Madison
[email protected]
The Goal
To render quantitative descriptions of
instruction, standards, and assessments using a
common language in order to facilitate
comparisons and analyses of these three
domains of a standards-based approach to
education reform and their relationship to one
another.
The Goal
To render quantitative descriptions of
instruction, standards, and assessments using a
common language in order to facilitate
comparisons and analyses of these three
domains of a standards-based approach to
education reform and their relationship to one
another.
Teacher
Reports
SEC
Taxonomy
Content
Descriptions
Needs
Assessment
Curriculum
Management
Alignment
Analyses
Monitoring
Change
Content
Analyses
Content
Descriptions
Program
Evaluation
Uses of Content Analysis Results
Descriptive: (Tile Charts and Content Maps)
• Visual, curriculum-based descriptions of
Instructional Targets for teacher reflection,
discussion and planning.
Analytic: (Alignment)
• Predict student achievement gains
• Control for content to examine other factors
• As an outcome measure for change over time
• Examine alignment of Standards & Assessments
To Describe Instructional Content
SEC utilizes a two-dimensional taxonomy based on:
1
Topic
2
3
by
4
5
Cognitive Demand
B
C
D
E
F
The Content Matrix
Categories of Cognitive Demand
Memorize
Topics
Nature of
Science
Science &
Technology
Science,
Health, Env.
Meas. &
Calc. In Sci.
Comp. Of
Living
Systems
Botany
Conduct
Communicate
Analyze
Apply / Make
Investigations Understanding Information Connections
… adding levels of relative emphasis yields a 3-D construct
Categories of Cognitive Demand
Memorize
Topics
Nature of
Science
Science &
Technology
Science,
Health, Env.
Meas. &
Calc. In Sci.
Comp. Of
Living
Systems
Botany
Conduct
Communicate
Analyze
Apply / Make
Investigations Understanding Information Connections
Content Map Data Displays
State J Grade 8
Mathematics Assessment
Number Sense
Operations
Measurement
Algebraic Concepts
Geometric Concepts
Data Analysis
0.1
0.08
0.06
0.04
0.02
0
Instructional Technology
Memorize Communicate Connect
Perform
Conjecture
Memorize
Communicate
Connect
Perform
Conjecture
To Facilitate Comparisons
Uses of Content Analysis Results
Descriptive: (Tile Charts and Content Maps)
• Visual, curriculum-based descriptions of
Instructional Targets for teacher reflection,
discussion and planning.
Analytic: (Alignment)
• Predict student achievement gains
• Control for content to examine other factors
• As an outcome measure for change over time
• Examine alignment of Standards & Assessments
Alignment as a Quantity
Aligning Tests to Standards
0.27 (Avg. Alignment: Test to Standard)
Range of Alignment: Test to Standard)
0.14
0.00
0.25
0.40
0.50
0.75
1.00
State U Grade 8 Mathematics Alignment: Test to Standard (0.23)
(Based on results for 10 states, across grades 4, 6 and 8: SEC Collaborative 2003)
Instructional Alignment
Instruction to Standards & Assessments
Fine Grain
Standards
0.05
Min.
0.00
0.03
Min.
0.17
Avg.
0.29
Max.
0.25
0.19
Avg.
0.50
0.75
1.00
0.31
Max.
Assessments
Based upon results for 168 teachers, across 3 states: MSP PD Study 2004
Explaining variation in student learning gains
Learning Gains by Course Type
From:
Upgrading High School Mathematics Instruction ,
(Gamoran, Porter, Smithson, & White, 1997),
EEPAv19n4
12
11.5
11
10.5
Learning Gains Controlling for Content
10
12
9.5
11.5
Time 0
Time 1
Time 2
11
Regents
Algebra
Stretch Regents
Math A/B/UCSMP
Gen. Mth. / Pre-alg.
10.5
10
9.5
Time 0
Time 1
Time 2
Alignment Analyses for School Improvement
Using alignment as an outcome measure
Alignment Index:
Instruction to Standards
Mathematics
Across 4 Districts
Counts
Treatment
99
Control
124
Leaders
16
(Measuring change in alignment over time)
Content Analysis Procedures
Exploring the
Dimensions of Content
But first….
Let’s take a 10 minute break!
The Two Dimensions Of Content
What students should know
[Topics]
And…
Be Able to Do
[Expectations for student performance]
Describing the Cognitive Domain
How Many Categories?
3
SCASS Science
4
DOK (Webb)
5
SEC
6
Bloom’s
Dimensions of Knowing & Inquiry
Acquire
Use
Extend
(From: Dimensions of Knowing and Inquiring about Science, State Collaborative on Assessments & Student Standards
Science Project, Council of Chief State School Officers, 1997)
Depth of Knowledge
Level
1
Recall
2
Skill/Concept
Strategic
Thinking
3
Extended
Thinking
* Webb, N. 1999. Alignment of Science and Mathematics Standards in Four State.
NISE Research Monograph #18. Madison:Wisconsin Center for Education Research.
4
Exploring Cognitive Demand
Acquire
Recall
Skill/Concept
Use
Extend
Strategic
Thinking
Extended
Thinking
Cognitive Demand (or Expectations for Student Performance)
Acquire
Recall
Use
Skill/Concept
Extend
Strategic
Thinking
Extended
Thinking
Memorize
Perform
Procedures
Demonstrate
Understanding
Conjecture,
Generalize
Prove
Solve nonroutine/ make
connections
Memorize
Conduct
Investigations
Communicate
Understanding
Analyze
Information
Apply concepts
/make
connections
Recall
Perform
Procedures
Generate
/Demonstrate
Analyze/
Investigate
Evaluate
Cognitive Demand (or Expectations for Student Performance)
Acquire
Recall
Memorize
Recall
Use
Skill/Concept
Perform
Procedures
Understanding
Extend
Strategic
Thinking
Demonstrate
Understanding
Application
Extended
Thinking
Analyze
Information
Analyze
Evaluate/Apply
Evaluate
Create
Exploring Cognitive Demand
CgD Immersion Activity
• Organize into Groups/Tables
• Each Table w/ CgD Pie
• Each Person w/ Cgd Descriptors
Step 1: Place CgD cards on Pie Slices face-down
Step 2: Turn cards over: ID agreements
e.g. 2 cards w/ same descriptor in same slice
if Group Agrees ... discuss key words
if not … Discuss … operational definition to distinguish
Step 3: Discuss disagreements
if consensus reached put in envelope / if not, set aside
Content Analysis Materials
• Cognitive Demand List
• Topics Lists
• Comments & Suggestions Worksheet
• Coding Forms
• Documents to be analyzed
Cognitive Demand Lists
• Five categories of cognitive demand
• Slightly different for each subject
• Each category is defined by a list of descriptors
• The list of descriptors are not exhaustive
• Each category stands on its own
• Each category has an associated letter (B-F)
Dimension A: Content Topics
Topics List (In your packet of material)
Organized at two levels:
Content Areas
(16 for Mathematics)
(27 for Science)
(14 for ELAR)
Topics (identified by number)
(182 Mathematics Topics)
(211 Science Topics)
(114 ELAR Topics)
Plus: non-specific & other
Topics Lists
• Topics Lists
• Mathematics
• Science
• English Language Arts & Reading
• Cover grades K-12
• Organized into Content Areas
• Topics & Content Areas have an associated #
Comments & Suggestion Worksheet
• One for each reviewer - more available
• Use to:
• Record coding conventions/decision rules
• Suggest/identify additional topics not listed
• Suggest/identify additional CGD descriptors
• Provide other comments & suggestions
• Be sure to turn in at end of workshop (and with
mail-in materials, as necessary).
Coding Forms
• Assessment Coding Forms
• Benchmark Coding Forms
• Each is used to record content descriptions
• Each content description consists of
• A topic number
• A cognitive demand category letter
Assessment Coding Forms
Rater: ________
Subject:__NC Math Gr 3 Benchmark Test (SF)_
Itm. Desig./Nbr. Content Code 1 Content Code 2 Content Code 3
TPC1 CGD1
TPC2 CGD2
TPC3 CGD3
1
40
2
41
3
42
4
43
5
44
6
45
7
46
8
47
9
48
10
49
11
50
12
51
13
52
14
53
15
54
16
55
17
56
18
57
19
58
20
59
21
60
22
61
23
62
24
63
25
64
26
65
27
66
28
67
29
68
30
69
31
70
32
71
33
72
34
73
35
74
36
75
37
76
38
39
77
78
Form: Test 1
Page
of
Itm. Desig./Nbr. Content Code 1 Content Code 2 Content Code 3
TPC1 CGD1
TPC2 CGD2
TPC3 CGD3
Standards Coding Forms
Rater: ________
Itm. Desig./Nbr.
Number Properties
Number Sense
1a
Document:__NAEP Grade 8____
Content Code 1 Content Code 2 Content Code 3
TPC1 CGD1
TPC2 CGD2
TPC3 CGD3
Itm. Desig./Nbr.
Ratios & Proportional
Reasoning
4a
1b
4b
1d
4c
1e
4d
1f
1g
Properties of Number &
Operations
5a
1i
5b
1j
5c
Estimation
2a
TPC1 CGD1
TPC2 CGD2
TPC3 CGD3
Content Code 1 Content Code 2 Content Code 3
TPC1 CGD1
TPC2 CGD2
TPC3 CGD3
TPC1 CGD1
TPC2 CGD2
TPC3 CGD3
TPC1 CGD1
TPC2 CGD2
TPC3 CGD3
5d
5e
2b
5f
2c
Measurement
Measuring Physical Attributes
1b
2d
Number Operations
3a
TPC1 CGD1
TPC2 CGD2
TPC3 CGD3
1c
3d
1g
3e
1h
3f
1j
3g
1k
Standards Coding Forms
Practice Coding Exercise
Content Analyzing Assessments
(Three code maximum)
Independent Coding
TPC1 CGD1 TPC2 CGD2 TPC3 CGD3
Practice Coding Exercise
Content Analyzing Standards
(Six code maximum)
Independent Coding
TPC1 CGD1 TPC2 CGD2 TPC3 CGD3
The Content Analysis Process
‘Coding’ Teams of 4-5 Content Experts
Independent Coding by each Analyst w/ Group Discussion
Should not be necessary to discuss every item – select by team
Goal for Process: Generalizability not Inter-rater Reliability
Pick-up and return documents / coding sheets to Alissa
Sign & return to Alissa non-disclosure forms
Content Analysis Workshop
The intended
curriculum:
State content
standards—
What students
should learn
The enacted
curriculum:
What teachers
teach
A neutral content language
Topics by Cognitive Demand
The assessed
curriculum:
State (and other)
assessments—
tested learning
The learned
curriculum:
Student outcomes
based on school
learning