Assessment
What are the questions?
What evidence will we
accept?
Diane Ebert-May
Department of Plant Biology
Michigan State University
[email protected]
http://first2.org
Anonymous
(Change Magazine, 2001):
“I believe we would all agree
that the absolute best teaching
learning-assessment model is
the one-on-one Socratic
apprenticeship model with
unlimited time with the student.
But ever since Socrates took on
two students rather than only
one (to double his income),
teachers have had to make
compromises in teaching.”
Larry Spence, Penn State
“Today’s graduates cannot
meet the demands of
workplace or without
several more years of
learning on the job. They
cannot formulate and solve
messy real-world
problems, work well with
others in high-stress team
situations, write and speak
forcefully and persuasively,
or improve their own
performance.”
Question 1
Please respond on a scale if 0-100
in increments of 10:
How important is it to use
multiple kinds of data to
assess student learning?
How important is it to use multiple forms of
data to assess student learning?
%
Relative Importance
n=127
Question 2
Please respond on a scale of 0 - 100
in increments of 10:
How often do you use data
to make instructional
decisions?
How often do you use data to make
instructional decisions?
%
Frequency
n=127
True or False?
Assessing student
learning in science parallels
what scientists do as
researchers.
Parallel: ask
questions
1. Description:
-What is happening?
2. Cause:
-Does ‘x’ (teaching strategy) affect
‘y’ (understanding)?
3. Process or mechanism:
-Why or how does ‘x’ cause ‘y’?
Parallel: collect data
We collect data to find out what our
students know.
Data helps us understand student
thinking about concepts and
content.
We use data to guide decisions
about course/curriculum/innovative
instruction
Parallel: analyze data
Quantitative data - statistical
analysis
Qualitative data
break into manageable units and
define coding categories
search for patterns, quantify
interpret and synthesize
Valid and repeatable measures
Parallel: peer review
Ideas and results are peer
reviewed - formally and/or
informally.
What is assessment?
Data collection with the purpose of
answering questions about…
students’ understanding
students’ attitudes
students’ skills
instructional design and implementation
curricular reform (at multiple grainsizes)
Why do assessment?
Improve student learning and development.
Provides students and faculty
substantive feedback about student
understanding.
Challenge to use disciplinary research
strategies to assess learning.
Research
Methods
Data collection
approaches
Assessment Gradient
low
Potential for Assessment of Learning
high
Multiple Choice … … Concept Maps … … Essay … … Interview
high
Ease of Assessment
Theoretical Framework
• Ausubel 1968; meaningful learning
• Novak 1998; visual representations
• King and Kitchner 1994; reflective judgment
• National Research Council 1999; theoretical frameworks
for assessment
low
Pre-Posttest Analysis
Does active, inquiry-based instructional
design influence students’
understanding of evolution and natural
selection?
Alternative Conceptions:
Natural Selection
■ Changes in a population occur through a
gradual change in individual members of a
population.
■ New traits in species are developed in
response to need.
■ All members of a population are genetically
equivalent, variation and fitness are not
considered.
■ Traits acquired during an individual’s lifetime
will be inherited by offspring.
Instructional Design
Cooperative groups in class:
Guppy Problem:
sexual vs. natural selection
http://www.first2.org/resources/inqui
ry_activities/guppy_activity.htm
-PBS film
-Simulation
-Analyze data
-Written explanation
Explain the changes that occurred in
the tree and animal. Use your current
understanding of evolution by natural
selection.
(AAAS 1999)
Misconception: individuals evolve new traits
% of Students
n=80; p<.01
Misconception: evolution is driven by need
% of Students
n=80; p<.01
In guppy populations, what are the primary
changes that occur gradually over time?
a. The traits of each individual guppy within a
population gradually change.
b. The proportions of guppies having different traits
within a population change.
c. Successful behaviors learned by certain guppies
are passed on to offspring.
d. Mutations occur to meet the needs of the guppies
as the environment changes.
Anderson et al 2002
Posttest: Student responses to
mc
% of Students
n=171
*
Animal/Tree Posttest:
Gain in student understanding of fitness
% of Students
n=80; p<.01
Design Experiment
Quantitative Data
Qualitative Data
Ebert-May et al. 2003 Bioscience
Question
How do assessment questions help us
determine students’ prior understanding
and progressive thinking about the carbon
cycle.
Instructional Design
•
•
Two class meetings on carbon cycle (160
minutes)
Active, inquiry-based learning
Cooperative groups
Questions, group processing, large lecture
sections, small discussion sections, multi-week
laboratory investigation
Homework problems including web-based
modules
Different faculty for each course
One graduate/8-10 undergraduate TAs per
course
–
–
–
•
–
Experimental Design
Two introductory courses for majors:
Bio 1 - organismal/population biology (faculty
A)
Bio 2 - cell and molecular biology (faculty B)
Three cohorts:
Cohort 1
Bio 1 (n=141)
Cohort 2
Bio1/Bio2 (n=63)
Cohort 3
Other/Bio2 (n=40)
Assessment Design
Multiple iterations/versions of the carbon cycle
problem
Pretest, midterm, final with additional
formative assessments during class
Administered during instruction
Semester 1 - pretest, midterm, final exam
Semester 2 - final exam
Grandma Johnson
Problem
Hypothetical scenario: Grandma Johnson
had very sentimental feelings toward Johnson
Canyon, Utah, where she and her late husband
had honeymooned long ago. Her feelings
toward this spot were such that upon her death
she requested to be buried under a creosote
bush overlooking the canyon. Trace the path of
a carbon atom from Grandma Johnson’s
remains to where it could become part of a
coyote. NOTE: the coyote will not dig up
Grandma Johnson and consume any of her
remains.
Analysis of Responses
Used same scoring rubric (coding scheme) for
all three problems - calibrated by adding
additional criteria when necessary, rescoring:
Examined two major concepts:
Concept 1: Decomposers respire CO2
Concept 2: Plants uptake of CO2
Explanations categorized into two groups:
Organisms (trophic levels)
Processes (metabolic)
Coding Scheme
Code
Organisms
Code
P rocesses and pathways
1
Decomposers
IA
Cellular Respiration
IB
Release CO
IIA
P athway of Carbon
2
P rimary
producers
2
IIA _1: through Air
IIA _2 : through Root
IIA _3 : no mention about pathway
3
4
Herbivore
Carnivore
IIB
Make Glucose
II C
P hotosynthesis
III
Respiration
(glycolysis, Kreb cycle)
IV
Respiration
(glycolysis, Kreb cycle)
Correct Student Responses (%)
Cellular Respiration by Decomposers
Bio1/Bio2
Other/Bio2
Friedmans, p<0.01
Correct Student Responses (%)
Pathway of Carbon in Photosynthesis
Bio1/Bio2
Other/Bio2
Friedmans, p<0.05
IRD Team at MSU
Janet Batzli - Plant Biology [U of Wisconsin]
Doug Luckie - Physiology
Scott Harrison - Microbiology (grad student)
Tammy Long - Plant Biology
Jim Smith - Zoology
Deb Linton - Plant Biology (postdoc)
Heejun Lim - Chemistry Education
Duncan Sibley - Geology
*National Science Foundation
What is the question?
What research and
instructional designs?
What data collection
methods?
How to analyze and
interpret data?
Are findings valid and
generalizable?
What are the next
questions?
WHO?
What evidence
will we accept?