Methodological Issues in
Developing a Learning Progressionbased Assessment System
Jennifer Doherty, Karen Draney
and Andy Anderson
Michigan State University
BEAR Center, UC Berkeley
NARST 2012
Research Goal
Develop a valid and reliable assessment
system that measures student progress in
terms of learning progression levels.
Key Methodological Problem
Each component of the system must meet two
sets of criteria:
– Statistical criteria based on measurement theory
and practice
– Conceptual criteria based on learning progression
theory and practice, including conceptual
coherence with other components of the
assessment system
Components of assessment system
(NRC 2001)
Components of assessment system
(Wilson 2005)
Component I: Construct Maps:
Learning Progression Framework
(Wilson 2005)
Component I: Construct Maps:
Learning Progression Framework
Learning Progression Hypothesis: Consistency
across processes with respect to principles and
models.
– Students who learn scientific discourse see how
systems and processes are connected, applying
principles and models across processes.
– Lower-level students will not see scientific
connections among processes, but their accounts will
have similarities because they draw on a common
pool of linguistic and conceptual resources.
Component I: Construct Maps:
Learning Progression Framework
Implication of hypothesis:
An assessment system should be able to measure
levels of proficiency in the ability to:
– trace matter and energy through systems
– reason across scales for different processes
that are consistent across processes.
Component 2: Item design
(Wilson 2005)
Component 2: Item design
• We want to measure students’ understanding
of principles and models with minimal effects
from scaffolding and local knowledge.
Example Items
• ENERPLNT (energy practice, plant growth process, MTF+CR)
– Which of the following is(are) energy source(s) for plants?
– Circle yes or no for each of the following: Water, Light, Air, Nutrients
in soil, Plants make their own energy.
– Please explain ALL your answers, including why the things you
circled “No” for are NOT sources of energy for plants.
• THINGTREE (mass/gas practice, plant growth process, MTF+CR)
– A small oak tree was planted in a meadow. After 20 years, it has
grown into a big tree, weighing 500 kg more than when it was
planted. Do you think the tree will need any of the following things
to grow and gain weight?
– Circle yes or now for each of the following: Sunlight, Soil, Water, Air
– If you circled yes, explain how the tree uses it.
• TREEDECAYC (energy practice, decomposer process, CR)
– A tree falls in the forest. After many years, the tree will appear as a
long, soft lump on the forest floor. Is energy involved when the tree
decays?
– Circle one: Yes / No If your answer is yes, please explain how
energy is involved.
Component 3: Outcome space
(Coding Rubrics)
(Wilson 2005)
Component 3: Outcome space
(Coding Rubrics)
Challenges:
– Coding rubrics need to be aligned among items
• among items of a single type of carbon transforming
process (e.g., plant growth item).
• among items of a single type of practice (e.g., tracing
energy).
– Coding rubrics need to be aligned with the learning
progression framework
• Responses that are coded at level 2 are not just partially
incorrect.
• They must be responses that contain indicators of level 2.
(Alonzo et al. 2012)
Component 3: Outcome space:
Developing Coding Rubrics
Methods:
• Development of rubrics that combine the general
Learning Progression framework levels with itemspecific level indicators
• Iterative process of developmental coding and rubric
revising
• Iterative process of full coding, reliability checks,
rubric revising, and recoding
Interviews and written assessments that assess two
dimensions of complex accounts (process and practice)
Component 4: Measurement Model
(Wilson 20005)
How well do codes of the sample
items fit the statistical measurement
model?
Discrimination
ENERPLNT
THINGTREE
TREEDECAYC
0.54
0.50
0.47
Weighted
Mean
Square
1.02
0.88
1.11
Questions about individual items
• Check coding rubrics for conceptual validity
• Check the data on which step thresholds are
based
• Recognize the limitations of individual items
Questions about whole assessment
Criteria for results of validation analyses
– Multi-dimensionality and discrimination:
• a single dimension should account for most of the nonrandom variance
• items should be aligned with that dimension.
– Wright maps and item difficulty:
• Step thresholds in approximately the same region of
vertical space support claim of an underlying level of
proficiency.
– Alignment with student interviews
• There should be correlation between a students written
and interview codes
High correlations among processes
and practices support claim of an
underlying level of proficiency.
Animal
Function
Animal
Function
Animal
Growth
Combustion
Cross
Process
Animal ComCross
Decay
Growth bustion Process
Plant
Growth
0.900
0.789
0.816
0.754
.828
0.862
0.818
0.764
0.837
0.847
0.743
0.847
0.856
0.924
Decay
Plant
Growth
0.880
Process Dimension Correlation Matrix
Correlation between codes of Interviews and IRT
student ability estimates
(r=.526)
Conclusions and Next Steps
• We have multiple lines of evidence that these
written assessments do a good job of
categorizing most of the students that we have,
Level 2 and Level 3 students.
• Not a lot of good evidence we’re able to
discriminate between Level 3 and Level 4
students.
• Investigate conceptual validity of items that are
too hard or too easy