Teaching, Learning, & Transfer of
Experimental Procedures
in
Elementary School Science
David Klahr
Department of Psychology
Pittsburgh Science of Learning Center (PSLC)
Program in Interdisciplinary Education Research (PIER)
Carnegie Mellon University
Society for Research on Educational Effectiveness
First Annual Conference Dec 10 - 12, 2006
Topic: Assessing different methods for
teaching experimental procedures to
middle school children
• In the lab
• In both “easy” & “challenging” classrooms
• To students of widely varying abilities
More specifically:
Teaching “CVS”
What is (CVS)?
• NOT
CVS:This:
Control of Variables Strategy
• A simple procedure for designing
unconfounded experiments:
- Vary one thing at a time (VOTAT).
• The conceptual basis for making valid
inferences from data:
- isolation of causal path.
Why study CVS?
Practical importance
Topic:
Core topic in early science instruction
Assessment: State standards
High stakes assessments
NCLB to start testing science
Best Instructional approach for teaching CVS?
Heated controversy in profession
Legislative battles (e.g., CA and “hands on” science)
Theoretical issues
Surface vs deep mapping during transfer of procedures
and concepts at different transfer “distances”.
Goal: Compare different types of
instruction for teaching CVS.
Chen & Klahr (1999), Child Dev.
•
Participants: 60 2nd - 4th graders
•
Assessment:
– Measure learning & transfer at different “distances”
from initial instruction.
•
Materials: 3 different physcial domains
– Springs
– Ramps
– Sinking objects.
Between
subjects
design
Springs domain
Which attributes determine how far a spring will stretch?
Materials:
8 springs: 2 lengths x 2 widths x 2 wire sizes & 2 pair of weights
Execution:
• Select two springs
• Select two weights
• Hang springs on rack hooks
• Hang weights on springs.
• Compare amount of stretching.
Question: does the length of a spring make a
difference in how far it stretches?
An unconfounded test:
A
Length: short
B
long
Width:
wide
wide
Wire:
thin
thin
Weight: light
light
Two types of instruction
(between subjects)
• Exploratory:
–Hands on: work with physical materials
– Goal provided: “find out if x makes a difference”
• Explicit = Exploratory plus:
– Training: Explicit, good and bad examples
– Training: Reasons why, focus on deep structure
– Probe questions: Can you tell for sure? Why?
Different transfer “distances”
• Near transfer (within domain):
– CVS “tests” in same domain as training, but on a
different dimension.
– Time: minutes after training
– Location, context, etc.: same as training
• Far transfer (between domain):
– CVS tests in different domain from training.
– Time: few days after training
– Location, context, etc., same as training
• Remote transfer (more later)
Study Phases
70%
Day 1
Day 2
60%
50%
40%
30%
20%
10%
0%
Exploration
(Pre-test)
Training Manipulation
Near
Transfer
Far Transfer
Explicit immediately better than Exploration
and remains so
% of unconfounded experiments
(4 experiments per child in each phase)
70%
Explicit
60%
50%
Exploratory
40%
30%
20%
10%
0%
Exploration
Near
(pre-test)
Transfer
Training Manipulation
Far Transfer
(Day 2)
CVS mastery by individual children
(at least 3 out of 4 unconfounded experiments)
% of children becoming
Masters
100
75
50
25
0
Explicit
Exploratory
Extensions
1. Initial transfer measures are very close to
training objectives.
2. Need a more “distant” ( “authentic”?) assessment
of children’s understanding.
3. Will training effects remain with such extended
assessments?
Procedure
Create a more “authentic” assessment:
•Ask children to judge science fair posters.
• Score their comments and suggestions.
CVS Training and Science Fair Assessments
(Klahr & Nigam, 2004)
1. Participants: 112 3rd & 4th graders
2. Train on CVS via Explicit or Exploration method.
3. Assess effectiveness of CVS skill.
4. Present poster evaluation task.
5. Look at how CVS skill, training condition, affect
poster evaluation performance.
Study Design
Day 1
1 week
70%
60%
50%
40%
30%
20%
10%
0%
Exploration
Training Manipulation
Near transfer
Far transfer
Poster
Evaluation
Scoring Rubric for Children’s Poster Critiques
1. Adequacy of research design
2. Theoretical explanation
3. Controlling for confounds in:
Subjects/Materials, Treatment, Experimenter bias, etc.
4. Measurement:
Reliability/Variability, Error, Data Representation
5. Statistical Inferences:
Sample size/population, effect size
6. Completeness of conclusion: Supported by data, Relate to hypothesis
Poster Score =


all valid, non-redundant,
critiques about a poster
Grand Poster Score = (Pingpong Poster) + (Memory Poster)
Possible subtle effects of type of instruction
Do the few kids who master CVS in the
Exploratory condition do better on poster
evaluation than the many who master
CVS in the Explicit Instruction condition?
Possible subtle effects of type of instruction
Do the few kids who master CVS in the
Exploratory condition do better on poster
evaluation than the many who master
CVS in the Explicit Instruction condition?
• More specifically:
– What is the relation between Poster Scores
and Path to CVS mastery?
•Method:
– Secondary analysis based on “learning paths”
Different “paths” to mastery or
non-mastery of CVS
How do these children following
these different paths perform on
poster evaluations?
Note: following based on combining results from
two studies: original K&N plus a replication
Poster Assessment Score (standardized)
n = 59
n = 15
n = 19
Hotshot (Natura l)
Experts
otshot (Exp loratory)
Masters
Exploratory
Explicit
Hotshot (Exp
licit)
Masters
1
.8
.6
.4
.2
0
-.2
-.4
-.6
-.8
n = 25
Explicit
n-Hotsho
t (Expli cit)
non-Masters
n = 66
Exploratory
tsho t (Explo ratory)
non- Masters
n.s.
n.s.
p < .001
o CVS mastery is
associated with high
poster scores
o Non-mastery with low
poster scores
o Path to mastery, or
non-mastery is
irrelevant
Question for cognitive research:
Why does training on CVS (narrow) lead
to better poster evaluations (broad)?
Focused search for causal paths
Decomposition (attention to detail)
Nature of science
Rhetorical stance
Science as argument
Stay tuned ….
Question for applied research:
Can CVS be taught in a normal
classroom setting?
Procedure (in a nutshell):
Translate experiment “script” into
teacher lesson plan.
Teach in “normal” science classes (in
high SES schools).
(Toth, Klahr, & Chen, 2000)
Participants in Classroom Study
• 77 4th graders from 4 classrooms in two
different private schools
• 2 different science teachers
• Neither school had participated in “lab” studies
What to hold and what to fold?
These are issues of “engineering design”.
Keep
Pedagogy:
– Goal – teach CVS
– Type of teaching:
Explicit instruction
Assessment:
– Same as laboratory
– Plus, some new
assessments in
classroom
Change & adjust
Context:
– Lesson plan, not
“script”
–Teacher, not researcher
–Scheduling
– Student/teacher ratio
– Group work
– Record keeping
– Error and multiple trials
Results of Classroom Implementation
% unconfounded
designs
Individual students
classified as
“Experts” (8 of 9 correct)
100
80
Pretest
Posttest
5%
91%
60
40
20
0
Pretest
Post Test
What about more challenging classrooms?
(“Lesson Planning Project”, w/Junlei Li, Stephanie Siler, Mandy Jabbour)
One facet of the Lesson Planning Project:
• Two classrooms (5th and 6th graders) in
urban school
• 90% eligible for free lunch.
• Teacher is researcher (Junlei Li)
Teaching & Assessment of CVS with Urban 5th and 6th Graders
(n = 42)
(Klahr & Li, 2005)
Our CVS Tests
Standardized
Test Items
100%
Dyads
80%
Student Design
Mastery-based
Local (CTBS)
Formative Assessment
% Correct
60%
National
(NAEP)
International
40%
Dyads
(TIMSS)
Focused
Analogical
Mapping
20%
0%
2-Day Classroom Replication of
CVS Training
Domain: Ramps
2-Day CVS
Transfer &
Retraining
Domain: Pendulum
2-Week Delay:
Transfer to “real
world”, “highstakes” items
% correct for various groups on a TIMMS CVS item
Typical TIMMS CVS item
He wants to test this idea: The heavier a
cart is, the greater its speed at the bottom
of a ramp. Which three trials should he
compare?
Significance
Brief, theoretically grounded, focused
instruction:
 Is highly effective for middle class students
 In the sort run & over longer durations
 On “far transfer” assessments
Path independence:
 “What” matters more than “how”.
BIG differences in effectiveness with different
student population. Thus, current approach
requires:
 Adaptation, Modification, & Individualization
Questions to pursue
(Next steps)
NCLB in “the small”:
Goal: No child who can’t understand &
execute CVS
Method: Develop an “intelligent tutor” that
can adapt to wide variability in children’s
learning
Wide variety of individual learning patterns
(From Chen & Klahr, 1999)
4
4
4
4
3
3
3
3
2
2
2
2
1
1
1
1
0
0
0
0
Ex As T1 T2
Ex As T1 T2
TYPE
FAST GAIN
Explicit
Socratic
UP DOWN UP
GRADUAL GAIN
HIGH CONSTANT
10%
7%
7%
0%
0%
14%
0%
4
4
3
3
3
2
2
2
1
1
1
0
0
0
Ex As T1 T2
Explicit
Socratic
Ex As T1 T2
31%
4
TYPE
Ex As T1 T2
UP & DOWN
Ex As T1 T2
STEADY DECLINE
Ex As T1 T2
LOW CONSTANT
7%
0%
37%
18%
7%
60%
Design a Tutor for Experimental Design
w/ Mari Strand Cary, Stephanie Siler, Junlei Li
Thanks to
Funding $ources:
• McDonnell Foundation, NICHD, NSF, IES
Recent & Current collaborators
Zhe Chen, Eva Toth, Junlei Li, Mari
Strand Cary, Stephanie Siler, Milena
Nigam, Amy Masnick, Lara Triona
END
Extras
Remote transfer items
A page from the 15-item test booklet
Does the amount of water
affect plant growth?
Why “remote”?
• Temporal
– Training - test interval: 7
months
• Domain
– Physical - biological, et al
• Format
Good Test
Bad Test
– Physical materials vs. paper
and pencil test booklet
• Context
- One on one with
Experimenter vs whole class
test taking
Remote Transfer Results
Trained
Untrained
Does the amount of water affect plant growth?
Good Test
Bad Test
Mean % correct on 15-item far transfer test
100
75
50
25
0
3rd
4th
Ramps Domain
Question: Does the surface of a Ramp make a
difference in how far a ball rolls?
A completely confounded test
A
B
Surface:
Run:
Steepness:
Ball:
smooth
short
high
golf
Surface:
Run:
Steepness:
Ball:
rough
long
low
rubber