Supporting
Changes in Teacher
Practice with a
Learning
Progression:
Results of the
Elevate Study
Erin Marie Furtak
University of Colorado at Boulder
June 30, 2017
University of South Bohemia
Cesky Budejovice
2000-2002
Lingering Questions
?
Furtak (2012)
Professional
Development
Teacher
Cognition
Activities &
Assessments
Classroom
Practices
Student
Achievement
Learning Progressions
Hierarchical representations of
how student ideas develop in a
content domain
Correct, scientifically accurate
ideas at top
Everyday ideas a.k.a.
misconceptions at the bottom
Usually used for curriculum and
assessment development
Formative Assessment
Professional
Development
Teacher
Cognition
Activities &
Assessments
Classroom
Practices
Student
Achievement
Learning Progression
Formative Assessment
Tools
Practices
Participants
8
Formative Assessment
Open-ended questions about natural
selection investigation (model of Darwin’s Finches)
Tool
Practices
Participants
9
10
Formative Assessment
Open-ended questions about natural
selection investigation (model of Darwin’s Finches)
Tool
How can we capture these?
Practices
High School Biology Teachers
Participants
11
Teacher 31
Highest
Degree
Earned
Undergra
duate
Major/s
Years of
Teaching
Teacher 34
Teacher 33
BS/BA MS/MA BS/BA MS/MA BA/BA MS/MA
Bilingu
Philoso
Educati
Educati
al
Biology
Biology
phy &
on
on
Educati
Biology
on
12
17
4
6
Formative Assessment
Open-ended questions about natural
selection investigation (model of Darwin’s Finches)
Tool
How can we capture these?
Practices
High School Biology Teachers
Participants
13
Notes from Video
Predictions
Teacher 31
Teacher 34
Leave
blank
for now
Teacher 33
As you watch the videos, write down a few
notes and observations about what you notice
about each teacher’s classroom practices.
14
Beans and Beaks Formative Assessment
Question 5
If the environmental conditions select
for the elimination (extinction) of a
certain beak type, and the
environment changes back to its
original conditions, would the
eliminated (extinct) beak type make a
return?
15
Teacher
31
16
T31: So if the environment conditions select for the
elimination of a certain beak type, like we keep talking
about the wooden sampler spoon going away. And then
the environment changes back to its original conditions,
what would happen to the birds? What would happen to
that bird population?
[No student response]
T31: So everybody pretty much thought while some birds
might come back that generally speaking the ones that
went extinct would not return. Now why wouldn’t they
return? Why wouldn’t they just come back?
Student: Their genes are gone.
17
T31: Huh? Their genes are gone? You mean to tell me if
those birds disappeared I couldn’t take some water,
dump it on some bird fossils and they wouldn’t suddenly
pop out of the ground like little plants?
Student: That would not happen.
18
T31: That would not happen. No. No. Once a bird
population has disappeared, or any population, they
wouldn’t come back. The wooly mammoth existed during
the ice age 10,000 years ago. Whether it was because of
the change in environmental conditions, or there are
theories that humans hunted them out of existence,
whatever the case may be, they’re gone. If we all of a
sudden had an ice age would they re-appear?
Student: No.
T31: No. Those genes have left the population. What
would be the only possible way that those genes could
possibly return if conditions returned to favor what they
were doing?
19
Teacher
34
20
T34: If the environmental conditions select for the
elimination of a certain beak type and the environment
changes back to its original conditions would the
eliminated or extinct beak type make a return? Give me
thumbs up if you think yes. <pause> Give me thumbs
down if you think no. <pause> A lot of us said yes. If you
said yes can you explain why?
Lisa: Because that’s how that beak came to be in the first
place. To get the food, each finch after several centuries
grows a beak to help them get the most food possible.
21
T34: Hold on. Do they grow that beak to help them get,
they’re like I want this kind of beak so they grow that to
help them get it? What do you mean by that?
Lisa: Genes kind of change.
Addie: Evolve.
Lisa: Yeah, evolve after several generations.
T34: Can anyone help Lisa? Yeah, Mason.
22
Mason: I said maybe. Because the beak would evolve
back again to that beak size.
T34: Ok. Here’s the answer. The big, big glaring answer
to number 5 is no. So here’s the deal. Thank you for
being a brave underdog and being like “uh I thought no.”
Here’s the deal. Could that beak type return? Well
there’s one way you could say yes but it’s not very likely.
So why is the answer no could someone explain it to
me? Zac, yeah?
23
Teacher
33
24
T33: If the environmental conditions select for the
elimination of a certain beak type and then we go back to
our original world of seeds, what would happen to the
birds? First of all, what happens if you eliminate a beak
type? What word do we use for that?
Casey: Extinct.
25
T33: They go extinct. So if we lose these guys right here.
If they’re gone they totally died out. They’re gone. Then
we get all those seeds back. Will the birds change their
beaks os they can be the same gene type? A mutation in
the gene would allow that beak type to come back?
While some birds would be more successful in the
changed environment the extinct ones would come
back? What do you guys think?
Nora: It would come back or wouldn’t come back?
26
T33: This says they would. Do you guys think that C is
possible?
David: Maybe.
T33: If the wooden samplers are gone can they come
back?
Ross: No
Maddie: There’s a chance that a genetic variation
mutation could happen.
27
T33: Ok so Maddie is saying maybe there’s a chance for a
genetic mutation to kind of happen where you might get
that beak type back. It’s a little complicated I think in that
respect. Overall, if we lose species overall on the planet,
they’re gone. Do we get them back?
Cora: No.
T33: No. Unlikely although there’s some pretty crazy
genetic stuff that might happen. You usually know. we
can have some other mutations of other beak types. This
guy is really not coming back.
28
What differences do you observe between the
formative assessment practices these three
teachers used with their students?
Discuss with a partner for 5 minutes
Share comments/thoughts
31
Teacher 31
34
Teacher 34
33
Teacher 33
29
What do YOU think...
Menti.com and enter in
Go to
the code
57 74 38
30
Hold on to those ideas...we’ll get back to them
later in the slide show.
31
Lingering Questions
?
Furtak (2012)
Formative Assessment Design Cycle
Reflect on
Enactment
Explore
Student
Ideas
Enact
Tools
Practice
Using
Tools
Develop
Tools
Furtak, 2012; Furtak, Morrison & Kroog, 2014;
Furtak & Heredia, 2014
Formative Assessment Design Cycle
Teachers learn about
‘big ideas’ in science
and how student
understanding of these
ideas develops
Reflect on
Enactment
Explore
Student
Ideas
Enact
Tools
Practice
Using
Tools
Develop
Tools
Formative Assessment Design Cycle
Reflect on
Enactment
Explore
Student
Ideas
Enact
Tools
Practice
Using
Tools
Develop
Tools
Teachers develop
formative assessments
designed to elicit
student thinking related
to the ‘big ideas’
Formative Assessment Design Cycle
Reflect on
Enactment
Explore
Student
Ideas
Enact
Tools
Teachers rehearse
enactment of the
activities and anticipate
how they will respond to
certain student ideas
Practice
Using
Tools
Develop
Tools
Formative Assessment Design Cycle
Teachers enact the
activities with their own
students, the enactment
is videotaped, and
copies of student work
are collected
Reflect on
Enactment
Explore
Student
Ideas
Enact
Tools
Practice
Using
Tools
Develop
Tools
Formative Assessment Design Cycle
Teachers reflect on
evidence of enactment
to determine
effectiveness and next
steps
Reflect on
Enactment
Explore
Student
Ideas
Enact
Tools
Practice
Using
Tools
Develop
Tools
Formative Assessment Design Cycle
Reflect on
Enactment
Explore
Student
Ideas
Enact
Tools
Practice
Using
Tools
Develop
Tools
Increasingly sophisticated ideas
Sequencing of ‘correct’ ideas
Educative Learning
Progressions as Tools for
Teacher Development
2010-2014
Operationalizing Formative Assessment
More Aligned with
Formative
Assessment
More Traditional/
Didactic
Designing Tasks
Eliciting Student Ideas
Interpreting Student Ideas
Providing Feedback
Predictions
Teacher 31
Teacher 34
Teacher 33
Where would
you place these
teachers for
each category?
Why do you
think so?
42
What do YOU think...
Menti.com and enter in
Go to
the code
57 74 38
43
Better
Identifying and
Interpreting of
student ideas
Participants
Better
Formative
Assessment
Task Design
Reflect on
Enactment
Explore
Student
Ideas
Enact
Tools
Asking higherquality
questions
Practice
Using
Tools
Tools
Develop
Tools
Giving Higherquality feedback
Increase in
student
achievement
Practices
Research Questions
1. To what extent does participation in the
Formative Assessment Design Cycle support
increases in the quality of teachers’ formative
assessment task design, questions to elicit
student thinking, interpretation of student ideas,
and feedback?
1. To what extent does teachers’ proficiency in
these formative assessment abilities predict
changes in student achievement?
Study Design
Participants & Intervention
● 9 biology teachers at 3 high schools
Students NBaseline= 472; NYear 3= 428
● Monthly, 60-90 minute professional
development sessions
Reflect on
Enactment
Explore
Student
Ideas
Enact
Tools
● Separate meetings at
each partner school
Practice
Using
Tools
Develop
Tools
Element
Sources of Data
Analytic Approach
Designing formative
assessment tasks
Copies of teacher-designed
tasks
Task quality rating
system
Asking questions to elicit Videotaped classroom
student ideas
enactment of tasks
Eliciting question video
coding system
Interpreting student
ideas
Teacher responses to a
sorting task
Direct agreement with
expert rater
Providing feedback
Videotaped classroom
enactment of tasks
Eliciting question video
coding system
Designing Formative Assessment Tasks
Likert-Scale items, 1=Traditional, 5=High-quality formative assessment
1. What type of space is there in the activity for students to share their ideas?
2. What is your impression of the type of instruction that would accompany this
activity, based on how it is written?
3. What type of knowledge is being elicited/promoted/targeted?
4. What type of information about student ideas does this activity seem
designed to provide?
5. What is the potential of this activity to make students’ scientific
understandings explicit?
6. What is your impression of how difficult or easy it might be to interpret these
scientific understandings?
● Six raters (all biology teachers trained in formative
assessment) applied ratings to all tasks collected
● ICC=0.80 to 0.96.
(t(8)=-1.53 p=0.16)
31
34
33
● 89 videotapes across 4 years of study
● Stratified random sample of 20% for reliability;
Cohen’s K: question = 0.90; teacher feedback = 0.85
● Sampled teachers enacting formative assessments in
baseline and final year of study
31
(t(8)=-2.79, p=0.02)
34
33
(t(8)=-2.28, p=0.05)
(t(8)=-2.68, p=0.03)
31
34
33
Pre-Post Assessment (Furtak et al., 2011)
• Multiple-choice pre-post assessment; orderedmultiple-choice design; 17 items
• Baseline pretest α=0.34, posttest α=0.61
• Year 3 pretest α=0.35, posttest α=0.62
Baseline
Year 3
Baseline
Year 3
Baseline
Year 3
Baseline
Year 3
Baseline
Year 3
Baseline
Year 3
Baseline
Year 3
Baseline
Year 3
Baseline
Year 3
Our interpretations
• Prior to the study, feedback quality was a
determining factor in the quality of student
learning
• However, following participation in the
formative assessment design cycle, we see
the important predictors changing to idea
interpretation and eliciting question quality
– Suggests efficacy of professional
development approach in supporting
changes in formative assessment abilities
60
Taking a step back from this
analysis, what do YOU think this
means? Which dots have been
connected, and what would you still
like to know?
Discuss with a partner for 5 minutes
Share comments/thoughts
31
34
33
What do YOU think...
Menti.com and enter in
Go to
the code
57 74 38
62
Thank you!
This material is based upon work supported by the National Science
Foundation under Grant No. 0953375. Any opinions, findings, and
conclusions or recommendations expressed in this material are those
of the authors and do not necessarily reflect the views of the National
Science Foundation.