Brett van de Sande, Kurt VanLehn, & Tim Nokes
University of Pittsburgh
Cognitive Analysis of Student Learning
Using LearnLab
1
Agenda
I.
LearnLab methodology
II.
Demonstration of Andes,
an intelligent homework tutor
III. Log File Analysis
2
Goal:
To understand physics learning
• Policy level
– e.g., Physics for high school freshman?
• Instructional level
– e.g., How much assistance to give?
– e.g., How much practice per topic?
– e.g., How to handle errors?
Our focus
• Neurocognitive level
– e.g., Can neuroimaging distinguish deep from
shallow studying of a text?
3
Traditional methods
for studying learning
• Design experiment
– Modify text, classroom activities, tests…
– e.g., Project Scale-up
• Lab experiment
– Modify just one factor
– Brief; money instead of grades, …
4
PSLC methods
• Educational data mining
– Logs from instrumented courses
– Some analysis is automated
Next
• In vivo experiments
– Control of variables
– Instrumented courses
5
Instrumented courses
(Called LearnLab courses)
• Existing class + data collection
– Homework done on a tutoring system or
photocopied and analyzed
– Photocopies of quizzes, exams
– FCI given before and after the course
– Demographics, GPAs, Majors…
– Handouts, slides, clicker data,…
• Instructor, student & IRB cooperation
– Anonymity
6
Existing
Physics LearnLab Course(s)
• US Naval Academy
– Course take by all 2nd year students
– LearnLab is in 4 of about 20 sections
– Profs. Wintersgill, McClanahan
• Your course here
7
Basic data mining question
• What features of students’ histories are
statistically associated with learning gains?
• e.g., What are the differences between
histories of Student A and Student F?
Student A: 25% on Semester-long history
pretest
85% on
post-test
Student F: 25% on Semester-long history
pretest
20% on
post-test
8
Knowledge decomposition
hypothesis
• Decompose knowledge to be learned
into a set of knowledge components
– e.g., Newton’s third law
– e.g., Centripetal acceleration
• Assume each knowledge component is
learned independently
– An approximation/idealization
9
Data mining with
knowledge components (KCs)
For each KC, find statistical associations
between histories and gains.
Student
A
A
KC
1
2
Pre-test
35%
15%
History
…
…
Post-test
85%
10%
A
B
B
3
1
2
25%
50%
10%
…
…
…
20%
20%
10%
B
3
25%
…
80%
10
History decomposition hypothesis
• Decompose the student’s history into events
such that each event addresses only one (or
a few) knowledge components.
–
–
–
–
Reading a paragraph about Newton’s 3rd law
Drawing a reaction force vector
Seeing the instructor draw a reaction force
Drawing a centripetal acceleration vector
• Assume that a KC’s learning gain depends
only on that KC’s events
11
Events for 1 student on 1 KC
(e.g., Newton’s 3rd law)
Time
Context
Behavior
8/27/07 9:05
FCI item 3
Incorrect
8/27/07 9:12
FCI item 10
Incorrect
9/13/07 18:06
Textbook, pg. 111
Highlighted
9/13/07 21:11
Problem 5-11,
drawing FBD
Omitted force on the
hand due to block
9/14/07 9:12
Lecture, slide 20
Taking notes
9/15/07 22:05
Problem 5-11,
drawing FBD
Draws force on the
hand due to block
etc.
12
Some events
are not currently available
Time
Context
Behavior
8/27/07 9:05
FCI item 3
Incorrect
8/27/07 9:12
FCI item 10
Incorrect
9/13/07 18:06
Textbook, pg. 111
Highlighted
9/13/07 21:11
Problem 5-11,
drawing FBD
Omitted force on the
hand due to block
9/14/07 9:12
Lecture, slide 20
Taking notes
9/15/07 22:05
Problem 5-11,
drawing FBD
Draws force on the
hand due to block
etc.
13
More feasible data mining
• Predict learning gains of a KC given the
sequence of events relevant to that KC
• On an event that assesses mastery of a
KC, predict the student’s performance
during that event given the sequence of
preceding events relevant to that KC
14
Predicting correctness of
events that assess mastery
Context
Event type
P(Correct)
FCI item 3
Assessment
0.10
FCI item 10
Assessment
0.12
Reading textbook, pg. 111,
paragraph 3
Instruction
Not applicable
Problem 5-11, drawing force
on hand due to block
Assessment
0.30
Lecture, slide 20
Instruction
Not applicable
Problem 5-11, drawing force
on hand … with remedial
feedback if needed
Assessment
then instruction
0.55
15
Learning curves
• Plot assessment events on x-axis
– Ordered chronologically
• Plot measure of mastery on y-axis
– Usually aggregated across subjects
e.g., proportion of 100 subjects who
performed correctly on this event
16
Frequency of correct
An expected learning curve
1.0
0.5
0
1
2
3
4
5
6
Assessment events
7
8
17
Summary of
PSLC educational data mining
• Given knowledge to be learned
– Decompose into knowledge components
• Given students’ histories from an
instrumented course
– Divide into assessment/instruction events
– such that one KC (or a few) per event
• For each KC, find a function on a sequence of
events that predicts the KC’s
– learning gain during the course
– learning curve
18
PSLC methods
• Educational data mining
– Logs from instrumented courses
– Some analysis is automated
• Andes produces logs with KCs
• DataShop draws learning curves, etc.
• Correlation ≠ Causation
• In vivo experiments
Next
– Control of variables
– Instrumented courses
19
Two major types
of in vivo experiments
• Short & fat
– During one lesson or one unit
• Long & skinny
– During whole course
– “invisibly”
20
Example of a short, fat,
in vivo experiment (Hausmann 07)
• During a 2-hour period (usually used for lab work)
• ~25 students in the room, each with a laptop and a
headset mike
• Repeat 3 times:
– Study a video while explaining it into the mike
– Solve a problem
• 4 experimental conditions, varying the content of the video
and the instructions for explaining it
• Random assignment of students to conditions
• Dependent measures include learning curves
• Result: Instructions to self-explain worked best regardless
of content of the video
21
Example of a long, skinny
in vivo experiment (Katz 07)
• During 8 weeks of a 13-week course
• Random assignment to 2 conditions:
– Experimental group: After solving certain
homework problems, the student discussed
the solution with a natural language tutoring
system
– Control group: Extra homework problems
• Result: Experiment > Control on some
conceptual measures
22
Robust Learning
• Immediate learning
– During an immediate post-test
– Similar content to training (near transfer)
• Robust learning
– Far transfer
– Retention
– Acceleration of future learning
• Does manipulation of instruction on topic A affect
rate of learning of a later topic, B?
23
Summary of PSLC methodology
• Data mining
– Instrumented (LearnLab) courses
– Knowledge components
– Instructional and assessment events
– Learning curves
• In vivo experiments
– Short & fat vs. long & skinny
– Robust learning
24
Agenda
I.
LearnLab methodology
II.
Demonstration of Andes,
an intelligent homework tutor
III. Log File Analysis
Next
25
26
Define variables
Draw free body diagram
(3 vectors and body)
Define coordinates
(3 choices for this problem)
Upon request, Andes gives
hints for what to do next
27
Red/green gives immediate
feedback for student actions
Principle-based help for
incorrect entry
28
29
# Log of Andes session begun Tuesday, July 17, 2007 12:12:28 by [User] on
[Computer]
...
05:03 DDE (read-problem-info "S2E" 0 0)
...
02:35 Axes Axes-671 64 335 143 296
02:35 Axes-dlg Axes-671 ||
…
02:38 C dir 40
02:42 BTN-CLICK 1 OK
02:42 DDE (assert-x-axis NIL 40 Axes-671 "x" "y" "z")
02:42 DDE-COMMAND assoc step (DRAW-AXES 40)
02:42 DDE-COMMAND assoc op DRAW-VECTOR-ALIGNED-AXES
02:42 DDE-COMMAND set-score 39
02:42 DDE-RESULT |T|
...
10:02 E 0 F1_y+F2_y=0
10:02 EQ-SUBMIT 0
10:02 DDE (lookup-eqn-string "F1_y+F2_y=0" 0)
10:47 DDE-COMMAND assoc parse (= (+ Yc_Fn_BALL_WALL1_1_40
Yc_Fn_BALL_WALL2_1_40) 0)
10:47 DDE-COMMAND assoc error MISSING-FORCES-IN-Y-AXIS-SUM
10:47 DDE-COMMAND assoc step (EQN (= (+ Yc_Fw_BALL_EARTH_1_40
Yc_Fn_BALL_WALL2_1_40 Yc_Fn_BALL_WALL1_1_40) 0))
10:47 DDE-COMMAND assoc op WRITE-NFL-COMPO
10:47 DDE-RESULT |NIL|
...
10:50 DDE-RESULT |!show-hint There is a force acting on the ball at T0 that you
have not yet drawn.~e|
...
16:38 END-LOG
problem name
student action (draw axes)
interpretation:
compare to model
green
student action (equation)
error analysis:
intended action
red
session time
30
Demonstration by Tim Nokes
# Log of Andes session begun Wednesday, April 18, 2007 21:08:07 by [user] on [computer]
...
0:02
DDE (read-problem-info "FARA9" 0 0)
...
0:13
Help-Hint
0:13
DDE (Get-Proc-Help)
0:13
DDE-COMMAND assoc (NSH NEW-START-AXIS 0)
0:13
DDE-RESULT |!show-hint It is a good idea to begin most problems by drawing an
axis. This helps to ground your work and will be useful later on in the process.~e|
…
0:17
Begin-draw 50001 Axes-1 185 331
...
0:30
New-Variable resistance
...
0:39
DDE (define-variable "R" |NIL| |resistance| |R| |NIL| |NIL| Var-2 "20 ohm")
0:39
DDE-COMMAND assoc step (DEFINE-VAR (RESISTANCE R))
0:39
DDE-COMMAND assoc op DEFINE-RESISTANCE-VAR
0:39
DDE-COMMAND assoc parse (= R_R (DNUM 20 ohm))
0:39
DDE-COMMAND set-score 3
0:39
DDE-RESULT |T|
....
0:50
DDE (lookup-vector "B" Unspecified B-field |s| NIL 0 |NIL| Vector-3)
0:50
DDE-COMMAND assoc entry (VECTOR (FIELD S MAGNETIC UNSPECIFIED
TIME NIL) ZERO)
0:50
DDE-COMMAND assoc error DEFAULT-SHOULD-BE-NON-ZERO
0:50
DDE-COMMAND assoc step (VECTOR (FIELD S MAGNETIC UNSPECIFIED
TIME NIL) OUT-OF)
0:50
DDE-COMMAND assoc op DRAW-FIELD-GIVEN-DIR
0:50
DDE-COMMAND set-score 2
0:50
DDE-RESULT |NIL|
...
9:51
DDE-RESULT |T|
9:55
END-LOG
31
Agenda
I.
LearnLab methodology
II.
Demonstration of Andes,
an intelligent homework tutor
III. Log File Analysis
Next
32
Model Solution Set
Solution 1
Solution 0
Principle A
Principle B
Principle A
Op1
Op3
Op6
Op7
Op2
Op3
Op5
Op8
Op10
Principle C
Op10
Op11
Op12
Op1
Op3
Op6
Op7
Principle D
Assumption: Opi = KC
33
# Log of Andes session begun Friday, July 27, 2007 14:29:38 by bobh on BOBH
…
0:02 DDE (read-problem-info "S2E" 0 0)
problem name
…
11:45 Vector-dlg Vector-673 ||
…
11:48 CLOSE type instantaneous
11:48 SEL type 1 instantaneous
11:51 BTN-CLICK 1 OK
11:51 DDE (lookup-vector "a" instantaneous Acceleration |ball| NIL 0 |T0| Vector-673)
11:51 DDE-COMMAND assoc step (VECTOR (ACCEL BALL :TIME 1) ZERO)
11:51 DDE-COMMAND assoc op ACCEL-AT-REST
11:51 DDE-RESULT |T|
…
student actions
green
match model solution:
assoc step = entry
Assoc op = operator
34
14:03
E 8 Fearth_y = m*g
student actions
14:11
EQ-SUBMIT 8
14:11
DDE (lookup-eqn-string "Fearth_y = m*g" 8)
14:11
DDE-COMMAND assoc parse (= Yc_Fw_BALL_EARTH_1_0 (*
m_BALL g_EARTH))
14:11
DDE-COMMAND assoc error MISSING-NEGATION-ONVECTOR-COMPONENT
14:11
DDE-COMMAND assoc step (EQN (= Fw_BALL_EARTH_1 (*
m_BALL g_EARTH))),(EQN (= Yc_Fw_BALL_EARTH_1_0 (Fw_BALL_EARTH_1)))
14:11
DDE-COMMAND assoc op WT-LAW,COMPO-PARALLEL-AXIS
14:11
DDE-COMMAND set-score 74
14:11
EQ-F 8
14:11
DDE-RESULT |NIL|
error
interpretation
guess
intended
red
35
Review Video
Match steps in video to log file
36
Researchable questions
Timing
Sequencing
(order of steps)
Hint Usage
Problem solving skills
Self-correction of errors
Errors as window to mental state
37
DataShop
https://learnlab.web.cmu.edu/
> Launch DataShop
> New user? Sign up now!
> (Create account)
38