Using automated analysis to
assess student ideas in
ecology
Luanna Prevost
Dept of Integrative Biology
University of South Florida
My lab
• Investigates student learning in
undergraduate ecology
• Develops constructed response questions
and automated analysis models
Research Questions
1.How can we design assessments that
identify students scientific and nonscientific ideas?
2.How can we scale this up for large
classes?
3.What conceptual difficulties student do
face when learning ecology?
Key Concepts and Standards
• Vision and Change Core Concept: Energy and Matter Pathways
and Transformation (AAAS,2011)
• SC.8.L.18.3 Construct a scientific model of the carbon cycle to
show how matter and energy are continuously transferred within
and between organisms and their physical environment.
• SC.7.L.17.1 Explain and illustrate the roles of and relationships
among producers, consumers, and decomposers in the process
of energy transfer in a food web.
• NGSS HS-LS2-5. Develop a model to illustrate the role of
photosynthesis and cellular respiration in the cycling of carbon
among the biosphere, atmosphere, hydrosphere, and geosphere.
• NGSS MS-LS2-3. Develop a model to describe the cycling of
matter and flow of energy among living and nonliving parts of an
ecosystem.
Welcome to the Southwestern Desert
Wile. E. Coyote and Road Runner
https://youtu.be/tQ9ND-D25fM
Who do we have here?
Animals
Rabbit
Road runner
Coyote
Diet (where do they get their carbon)
Plants, seeds
Insects, cactus, snakes
Road runner, rabbit, other small mammals
Carbon storage
Plants
Atmosphere
Soil
Diet (where do they get their carbon)
Carbon dioxide from ?
From respiring plants and animals
From decomposition
Sad news! We mourn the
death of Rabbit
• A rabbit passed away and became buried
under the soil in the desert.
• How might a carbon atom in that rabbit end
up in the tissue of a coyote?
• Note: The coyote does not dig up the rabbit
and eat it.
• In groups of four, use your assembly kit to
show how the carbon atom moves through
this ecosystem
1. How can we design assessments that
identify students scientific and nonscientific ideas?
We can use activities where students model
ecological processes
a. Monitor placement of organisms and abiotic carbon
pools (cards)
b. Evaluate the direction of carbon flow (arrows)
c. Evaluate processes used to transform carbon
(labels on arrows)
Modifications
• Give students processes or allow
students to think about these on their
own
• Discuss the forms carbon takes. In what
compounds is carbon stored at each
stage?
• Have students transfer this to another
ecosystem
• E.g. everglades, forest, Florida scrub
Research Questions
1.How can we design assessments that
identify students scientific and nonscientific ideas?
2.How can we scale this up for large
classes?
3.What are conceptual difficulties student
face when learning ecology?
How can we scale this up for large classes?
Student represents
his/her understanding in
own words
CR assessments give
instructors insight into
student’s
understanding
CR Assessment
X
!
!
Large enrollment
class
Instructo
r
X
Instructor gives student feedback
Ecology
Analyzing CR assessment
in large enrollment courses
Student represents
his/her understanding in
own words
CR assessments
automatically scored
CR Assessment
Computerized
lexical analysis
Student
Ecology
Instructor gives student feedback
Instructo
r
Instructors have
insight into student’s
understanding
Automated assessment
of
written responses
Human coding
Student
responses
Machine Learning
& Text analysis
Statistical
Models that
predict scores
for future
responses
Coding scheme
• Processes – e.g. Decomposition, photosynthesis
• Correct pathways/ Scientific Ideas –
e.g. Carbon travels from atmosphere to the cactus
• Incorrect pathways/ Non-scientific ideas –
e.g. Carbon travels from soil to cactus
How would you code this response?
Response 1
The jackrabbit starts decomposing, and this process releases
carbon atoms in the atmosphere in form of carbon dioxide,
which then are taken by the cactus, which performs
photosynthesis to release oxygen into the atmosphere.
Category
Coding
Photosynthesis
1
Decomposition
1
Carbon from atmosphere to
plant
1
Carbon from soil to plant
0
How would you code this response?
Response 2
The decomposing matter of the dead jackrabbit emit levels of
carbon dioxide. The plant roots of the cactus absorb the carbon
dioxide and transform it into carbon. The carbon in the cactus is
then ingested by the jackrabbit when it eats the cactus.
Category
Coding
Photosynthesis
0
Decomposition
1
Carbon from atmosphere to
plant
0
Carbon from soil to plant
1
Machine learning and text analysis
• Extracts words and phrases
decomposition, cactus, soil, the soil, in the soil, roots
• Uses Rules to associate related words and phrases
• OR[decompose, decomposed, decomposer, decomposers,
decomposes, decomposing, decomposition]
• AND[OR[roots, roots_of, the_roots, roots [GAP] cactus ,
roots [GAP] the , roots [GAP] the cactus , roots of , the
roots , the roots of , through [GAP] roots ], OR[soil, soil_,
soil_and, the_soil, from [GAP] soil , in [GAP] soil , in the
soil , into [GAP] soil , into [GAP] soil . , into the soil , into
the soil . , soil . , soil [GAP] the , soil [GAP] then , soil
and , the soil , the soil . , the soil [GAP] the ]]
2. How can we scale this up for large
classes?
• Machine learning provides models with high
agreement (kappa ≥0.8) with human coding
Category
Kappa
Kappa of 1 = perfect agreement
Photosynthesis
0.956
Decomposition
0.901
Atmosphere to
cactus
0.817
Soil to cactus
0.809
Kappa of 0 = no agreement
Research Questions
1.How can we design assessments that
identify students scientific and nonscientific ideas?
2.How can we scale this up for large
classes?
3.What are conceptual difficulties student
face when learning ecology?
What scientific and non-scientific ideas were
present in student responses?
Percent Responses
100
80
60
40
20
0
Processes
N = 139
Correct Pathways
Category
Incorrect Pathways
93 % of student responses mention decompostion
Correct Process
Decomposition
The carbon in the deceased jackrabbit
returns to the soil via some type of
decomposer. The cactus uptakes nutrients
and that carbon is transferred to the coyote.
There is likely some sort of primary consumer
that may each parts of the cacti or the cacti
flowers and it is then eaten by the coyote.
The carbon atom from an old deceased
jackrabbit is released in the form of CO2
through the breakdown of the dead
organism's remains by decomposers…
What scientific and non-scientific ideas were
present in student responses?
Percent Responses
100
80
60
40
20
0
Correct Pathways
N = 139
Category
20 % of student responses describe movement from the
atmosphere to plants
Contains Correct Pathway
Atmosphere to Cactus
The jackrabbit is consumed by decomposers and
carbon is released into the soil when digested
and discarded. it is released into the atmosphere
and used by a plant to grow. the plant is then
eaten by a rabbit which is then eaten by a
coyote.
The jackrabbit starts decomposing, and this
process releases carbon atoms in the
atmosphere in form of carbon dioxide, which
then are taken by the cactus, which performs
photosynthesis to release oxygen into the
atmosphere.
I would say that the coyote inhales some of the
air that contains carbon dioxide and ends up
within the coyote.
What scientific and non-scientific ideas were
present in student responses?
Percent Responses
100
80
60
40
20
0
N = 139
Processes
Correct Pathways
Category
Incorrect Pathways
21 % of student have the misconception
that plants absorb carbon directly from the soil
Soil to
Cactus
Jackrabbi
t to soil
Example Responses
The nutrients of the dead rabbit can be
absorbed by the cactus, with the help of
detritivores/ decomposers, can be eaten by a
primary consumer that the coyote eats.
The decomposing matter of the dead
jackrabbit emit levels of carbon dioxide. The
plant roots of the cactus absorb the carbon
dioxide and transform it into carbon. The
carbon in the cactus is then ingested by the
jackrabbit when it eats the cactus.
Plant to
herbivore
Herbivore to
coyote
The fungi and bacteria in the soil can help
break down the jackrabbit's remains. These
organic molecules, including carbon, can be
used by the cactus in order to grow and for
nutrients…
3. What conceptual difficulties do
students face when learning
ecology?
• Students have difficulty understanding
the flow of carbon from the soil to the
atmosphere via decomposition
• Students incorrectly think plants take up
carbon through the soil.
How can this inform instruction?
• Gain insight as to why students have
struggled continuously with certain concepts
• Use example responses in each group to
create clicker questions and hold discussions
• Create future in-class and homework
activities to improve student writing skills
Incorporating written assessment in the
classroom
•
Interface: Institution’s online learning management
system or online survey tool
•
Encourage student participation by giving credit for
homework assignments
•
Allow time between each assignment and the next class
for preparing/modifying instructional activities
•
Support for faculty
• Interpret analysis
• Develop/select instructional activities
Acknowledgements
The AACR Research Group
www.msu.edu/~aacr
Kellie Carter
Andrea Espina
Kirsti Martinez
Margaurete
Romero
Funding
This material is based upon work supported by the National Science Foundation under Grants
No. 1347626. Any opinions, findings, and conclusions or recommendations expressed in this
material are those of the author(s) and do not necessarily reflect the views of the National
Science Foundation.