Learning Gains on Newtonian Conceptual Reasoning in an
Iterative, Project-Based Design
Paul J. Camp
Spelman College
Why U-shaped Development Severely Limits the Validity of
Pre/Post Testing for Research
Abstract
I compare the normalized gain on conceptual Newtonian reasoning in a course based on iterative design projects to a linearly organized non-project based course in middle school. In addition to pre/post gains, I also measure
incremental changes over time and detect a degradation of performance at intermediate times. This is interpreted as an example of U-shaped development, which in turn implies a fairly stringent criticism of standalone pre/post
testing as an assessment or research strategy.
A greatly expanded version of this paper is online at http://arxiv.org, document reference arXiv:1007.1647v1
Design for a Study of
Cognitive Development
The Evidence for Non-monotonic (U-Shaped)
Developmental Processes
Because of its iterative nature, LBD (see below) easily affords a study of the development of Newtonian Concepts and
scientific reasoning over time, which we undertook in 2002-2004. This study deployed a multiple measures approach
to determine the developmental trajectory, including:
As can be seen, the LBD class significantly outperformed the comparison (p << 0.01, r=0.83, d=3.0).
What happens between pre and post in the LBD class is considerably more interesting and forms the
core of this paper. Quiz items were divided into clusters by concept. Student performance at each
quiz was placed into one of three categories – N (Newtonian, defined as at most one error), NN (NonNewtonian, defined as at least one correct answer), and T (transitional, defined as anything else).
Due to the nature of the data it was not possible to make every classification at every point on every
concept. Nevertheless, the pattern is quite interesting.
1. Conceptual quizzes pre, post and after each activity cycle. Questions drawn from FMCE, FCI and
some custom based on the developmental stages identified by Thornton.
2. Structured Interviews of a subset of students along with each quiz, based on a simple experiment
and intended to determine the level of prompting necessary to elicit Newtonian responses.
3. Ethnographic observations in class between two and three times per week throughout the
implementation
4. Performance assessments pre and post based on a single day group design and experiment
activity with discussions videoed.
First Law 02-03
120
100
80
60
40
20
0
This paper is primarily concerned with the quiz data but will draw on other sources for support and interpretation.
First Law 03-04
#N
#T
#NN
Pre
CC
BC
RB
100
100
80
80
80
60
60
60
40
40
20
20
0
0
Post
Pre
100
2002-2003
80
60
Pretest (%)
Posttest (%)
Normalized
Gain (%)
Pretest
(%)
Posttest (%)
Post
37 ± 7.3
Comparison
N=84 (02-3), 85
(03-4)
30 ± 8.5
76 ± 8.4
61 ± 12
34 ± 6.6
68 ± 12
11 ± 20
28 ± 8.7
36 ± 7.7
CC
BC
RB
Post
80
80
60
60
40
40
BC
RB
Post
0
Pre
CC
BC
RB
Post
BC
RB
Post
20
20
0
0
BC
RB
Post
Pre
CC
BC
RB
Post
Observation
Most of the data (acceleration excepted) shows the
same general pattern of performance change: There
is an advance to a high state of performance,
followed by regression to a prior state. With
continued experience, good performance eventually
returns.
U-shaped cycles have been observed in language development , artistic creativity, facial perception ,
social cognition, ratio reasoning, musical understanding, intuitive thinking, scientific reasoning, phoneme
discrimination, auditory localization, motor development , development of written representations, object
permanence, vocabulary acquisition, gesture-referent relations, memory and false memory storage and
retrieval, category formation, and transitions between levels of expertise in various knowledge domains
among adults. This vast array of totally unrelated skills showing the same developmental process
suggests that it is a general feature of human learning and cognition.
I have now documented it in physical reasoning. It seems, given the nature of the explanation, that it will
be a pervasive feature of physics pedagogy that has hitherto gone unnoticed, perhaps due to
overreliance on pre/post testing. In any case, results from pre/post testing that do not involve
independent means of determining where in the developmental cycle the post test occurred are
intrinsically impossible to interpret.
11 ± 16
Why Does Non-monotonic (U-Shaped) Development
Happen?
2003-2004
2002-2003
100
100
80
80
60
60
40
40
20
20
0
0
-20
Generalization of Results
40
Siegler2 describes three general categories of developmental explanations for this phenomenon.
Pre
time
Standalone pre/post testing gives a measure of a state of understanding at one moment in time. It
does not tell us if that change is meaningful, or stable, or how it came about, or if there was no
change does that imply no understanding. It may be useful as a rough measure during curriculum
development but it is not a reliable method for documenting the effectiveness of an intervention.
Net Force 03-04
60
CC
Post
Gain
-20
1. Newly acquired reasoning processes may be over generalized, and
applied in situations for which the new approach is inherently flawed.
2. A newly acquired processing approach places greater demands on the
processing system than does one that is of long experience. Therefore,
the greater cognitive load may result in temporary decreases in efficiency.
3. The skill in question is actually composed of multiple subskills which
develop (monotonically) at different rates. One monotonically developing
subskill may therefore temporarily outstrip another, producing multiple
peaks in performance as different subskills come on line.
LBD
Comparison
Pre
Post
This is especially problematic for short time scale
measures as are often used in tutorial validation.
Here is your pretest
Post
80
Pre
20
CC
Without another line of evidence, you cannot tell
where in the developmental cycle the posttest
occurred.
100
Acceleration 03-04
100
CC
RB
40
Pre
0
BC
60
100
Pre
39 ± 9.7
60
80
Acceleration 02-03
52 ± 21
CC
Net Force 02-03
0
RB
20
LBD
N=90 (02-3), 82
(03-4)
Pre
100
20
BC
Pre
Post
80
20
CC
0
120
40
Pre
Normalized Gain
(%)
RB
20
100
40
0
2003-2004
BC
P
e
r
f
o
r
m
a
n
c
e
The short answer is that they are not standalone. The presence of intermediate quizzes,
ethnographic class observations and structured interviews over the same period of time allows us to
document where in the developmental cycle the post test occurred.
40
Third Law 03-04
120
At a suburban middle school in Atlanta, GA, we compared an honors LBD class with a standard honors
class in two successive years. Teachers were carefully matched with regard to background and
experience. The non-LBD classroom was not entirely lecture based, but activities tended to be much
smaller scale, more directive in approach and of significantly shorter duration.
CC
Where is your posttest?
Pre/post testing uses performance as a proxy for
understanding. U-shaped development shows that it
is not. Performance and conceptual understanding
are not tightly coupled.
Why the Pre/Post Results for LBD Are a Valid
Measure
Second Law 03-04
100
Third Law 02-03
Pre/Post Between LBD and Comparison Class
Second Law 02-03
U-shaped Development Cartoon
Gain
Learning by Design™ Course
Structure
Learning by Design (LBD)1 uses engineering design projects as a context within which to learn
qualitative Newtonian concepts along with general scientific reasoning and design skills. It is
fundamentally a cognitive apprenticeship approach to learning with a learning cycle and activities
based on Case-Based Reasoning, a cognitive model of reasoning by analogy that contains within it a
model of how memories are stored that will afford transfer to new situations. The class is organized
around a nested learning cycle in which design-and-test activities motivate and contextualize scientific
investigations which in turn provide information relevant to redesign.
Since it is impossible to over generalize Newton’s laws in this context, and since the prior reasoning
is phenomenological so the second category would not account for the first performance peak, the
most relevant for us appears to be the third type of explanation. There are two obvious broad
categories of subskill involved in the production of physically correct Newtonian performance –
conceptual understanding, and appropriate access to that understanding. In the case-based
reasoning model of cognition [7], access to understanding crucially depends on links (or indices)
between that memory and contextual cues. We would say that when performance is variable, the
relevant memories are poorly and/or sparsely indexed and therefore mostly inaccessible except in
very specific situations. The correct pedagogical approach, predicted by this model, would therefore
be to shift the same content to a series of different contexts, which require drawing on the same
memories in order to be successful, and thereby deliberately triggering a series of representation
failures and therefore dips in performance.
Student A
I: What laws do you recall?
A: I think the first one is any object in motion stays in motion unless it is acted upon by
another object. Third law [long pause] I don’t remember third law. It’s got something to do
with acceleration and velocity because I remember the second and third law quiz was
finding velocity from, finding acceleration from the starting velocity and the final velocity
over time.
I: This semester you guys learned about forces in pairs. What do you know about forces
in pairs?
A: They’re the same size.
I: What does it mean? What does forces in pairs mean?
A: That two forces acting upon each other, like going against each other I guess.
I: OK, give me an example.
A: Like I know we did a lot of forces in pairs with the balloon cars, where the air was
pushing out on the balloon and the balloon was pushing back in on the air which forced
the air out. . .
I: In this experiment, with the car going down the track, are there any places along here
where forces in pairs are demonstrated, where you see forces in pairs?
A: I don’t think so but I’m probably wrong.
I: When the car is in the launcher, do you think there’s forces in pairs in the launcher?
A: [quickly] Yeah, yeah, yeah. Because the car pushes back on the launcher, and the red
block [the launcher] pushes on the car.
I: Can you talk to me about those two forces in terms of their size and direction?
A: Um, well, the car has the same force on the red block as the red block has on the car.
[Indicates opposite directions by pointing fingers when mentioning each force].
Student B
Prediction of the Model
If this explanation is correct, then an immediate prediction is that during dips in performance we
should see students exhibit understanding of Newtonian explanations but having difficulty
accessing them in appropriate situations. This would imply that understanding is continuing to
advance while performance degrades. We can use the structured interviews to check this
prediction.
Structured Interviews
In the main physical science unit, students make at least four passes through this cycle in the course of
designing and testing powered vehicles which must traverse a specified test track. Each time, they vary the
engine, requiring progressively more extensive Newtonian reasoning to understand its functioning.
In contrast to many curriculum designs, in LBD students are wrestling with the full range of relevant force
and motion concepts from the beginning. Learning comes from successive refinement of that range of
concepts through multiple passes in varying contexts. Those passes are labeled according to the engine:
Coaster Car (CC – a ramp), Balloon Car (BC – balloons), Rubber Band Car (RB – a rubber band wrapped
around the axle) and Hybrid Car (HC – best functioning design drawing on all the engines, including a falling
weight that is introduced by demonstration after RB).
1J.
Kolodner, P. Camp, D. Crismond, B. Fasse, J. Gray, J. Holbrooke, S. Puntambekar and M. Ryan, Problem-based learning meets Casebased reasoning in the middle-school science classroom: Putting Learning by Design into practice, J. Learn. Sci., 12, 4, p495 (2003)
The structured interview task involved a toy car launched down a track with a spring loaded
launcher. A rough surface could be added to or taken away from the track, and the spring
launcher could be substituted with a hill that caused the car to travel the same distance. The
following excerpts come from the third law portion of the interviews, for which the quizzes show
the clearest case of U-shaped development. Four students were interviewed from the LBD
class. Two of them had spontaneously used third law logic at an earlier point in the interview
and two had not. I present one example of each.
2R.
S. Siegler, U-Shaped Interest in U-Shaped Development – and What it Means, J. Cog. Dev.5, 1, p1 (2004)
I: Earlier we were talking about laws, Newton’s laws, and stuff, and you said “I don’t
know the numbers.” Uh, and you said something about objects pushing on each other.
C: Yeah, equal but opposite forces.
I: Can you explain what that is, or just give me an example or something?
C: Ms. [Teacher] did a really good one, it was when, like a huge truck and a little car
and they hit each other they would each experience a force and it would be equal but
it would be opposite . . .
I: Are there any instances with this launcher. . .where we’re seeing these equal and
opposite forces being demonstrated to us?
C: Uh, [long pause] well, the only thing I can think of that would really affect it is if it hit
something, like another car on the ramp or something. Um, because I’m not so sure
about the force of friction, um, if that would be an equal, I don’t think that’s an equal
and opposite force.
I: What about the launcher and the car?
C: [quickly] Yeah, the car’s pushing back on the launcher and the launcher is pushing
forward on the car, and the forward force of the launcher
Each of these excerpts shows a student with an understanding of the third law but an inability to
access it as needed. That is in fact the story for all of the interview participants at this time.