Observing and Assessing Workshop
How Do They (or You) Know
What They Know?
Holly A. Taylor
Tufts University
Psychology Department
They Don’t Know What Contributes
to Their Learning: Multimedia Advantage
• Learn procedures
for assembling toys
from
– Picture
– Text
– Multimedia
(Pictures + Text)
(Bruny é, Taylor, & Rapp, 2003, 2004, in prepration)
They Don’t Know What Contributes
to Their Learning: Multimedia Advantage
• Assessment of Learning
– Recall Procedures
– Verify Step Order
– Recall Learning Format
(Bruny é, Taylor, & Rapp, 2003, 2004, in prepration)
They Don’t Know What Contributes
to Their Learning: Multimedia Advantage
– Recalled more after
multimedia
• Verify Step Order
– Faster and more
accurate verification
after multimedia
Percent Correct Response
• Recall Procedures
1
0.9
0.8
Full
Attention
0.7
Divided
Attention
0.6
0.5
0.4
Pictures
Text
Multimedia
Learning Format
(Bruny é, Taylor, & Rapp, 2003, 2004, in prepration)
They Don’t Know What Contributes to
Their Learning: Multimedia Dis-Advantage?
– “I only looked at the
pictures”
– “I didn’t need to read
since the pictures
were there.”
0.6
0.5
Average Occurrence
• Recall Learning Format
– Misremembered
multimedia as
picture-only
0.4
0.3
Full
Attention
Verbal
Divided
0.2
0.1
0
TP
TMM
PT
PMM
MMT
MMP
Error Type
(Bruny é, Taylor, & Rapp, 2003, 2004, in prepration)
Take-Home Message
• Students may not be aware of how they
are learning.
• Students may not be aware of where
they are allocating their attention during
learning.
They Don’t Know What Influences
Their Thinking: Framing Effects
• How fast were the
cars going when
they…
-smashed?
-collided?
-bumped?
-contacted?
(Loftus & Palmer, 1974)
They Don’t Know What Influences
Their Thinking: Framing Effects
• stronger verb = faster speed
• faster speed = greater
likelihood of misremembering
broken glass
(Loftus & Palmer, 1974)
They Don’t Know What Influences
Their Thinking: Framing Effects
Program A:
Program A:
Of 600 people, 200 will be
Of 600 people, 400 will
saved.
die.
Program B:
Program B:
1/3 chance that 600
1/3 chance that 0 people
people will be saved.
will die
2/3 chance that 0 people
2/3 chance that 600
will be saved.
people will die.
80
80
60
60
40
40
20
20
0
0
A
B
A
B
(Tversky & Kahneman, 1981)
They Don’t Know What Influences
Their Thinking: Framing Effects
• Eruption!
– Effects of framing
scientific data on
data-driven
decisions.
(Taylor, Renshaw, & Jensen, 1997)
They Don’t Know What Influences
Their Thinking: Framing Effects
Positive Frame
Seismicity (Events/Interval)
< 20
20-50
50-60
60-70
Probability of an Eruption in next 48 Hours
0% Chance
50% Chance
70% Chance
90% Chance
Negative Frame
Seismicity (Events/Interval)
< 20
20-50
50-60
60-70
Probability of No Eruption in next 48 Hours
100% Chance
50% Chance
30% Chance
10% Chance
(Taylor, Renshaw, & Jensen, 1997)
They Don’t Know What Influences
Their Thinking: Framing Effects
– “Well, if there’s an 80%
chance of eruption, there’s
a 20% chance of no
eruption.”
80
70
Money Allocated
• Eruption! on computer
increased awareness of
both frames, thereby
changing overall
decisions.
60
Positive Frame
50
Negative Frame
40
30
20
10
0
College
Control
Group
GSA
Members
Paper
Lab
Computer
Lab
• Students unaware of this
influence.
(Taylor, Renshaw, & Jensen, 1997)
Take-Home Message
• People are unaware of how they use
data in making decisions.
• People are poor judges of how “good”
their decisions are
– More generally, poor judges of how a
learning exercise impacts their decision
making skill.
If They Like it They Will Learn?
• Automating math frees
cognitive resources for
understanding concepts
• Study
– Computer does math or
student does.
– Test within and acrossdomain dimensionless
graph interpretation
(Sinclair, Renshaw, & Taylor, 2004)
If They Like it They Will Learn?
• Automating math does
not free cognitive
resources for
understanding concepts
• Need basic skills
• But, students love when
computer does the
math for them.
1
Study 2
0.8
0.6
0.4
0.2
0
Automated
Computer
Non-Automated
Computer
(Sinclair, Renshaw, & Taylor, 2004)
Take-Home Message
• Engagement in an exercise does not
guarantee learning.
• Automation needs to be population
appropriate
– Automate only when it’s clear they could do
it on their own.
If They Don’t Know, Use Outside
Observer
• What happens when you
tell a parent their son ate
sugar?
– Rate their son as
significantly more
hyperactive.
– Exercise more control over
their sons
• criticize, look at, and talk
to them more.
(Hoover & Milich, 1994)
Take-Home Message
• Beliefs going into a learning situation
affect interpretation
– “But, I’m not good at math.”
– “I just don’t understand maps.”
– “Men are better with maps than women.”
How Do They (or you) Know What They
Know: Conclusions
• Use student insight into their own learning
with caution.
• Consider fact that there are many, often
unforseen, influences on self-assessments of
learning.
• “Unbiased” Observers may not be (unbiased,
that is)