Perpetually enhancing human learning through
collaborative, dynamic, personalized experimentation
Joseph Jay Williams
Harvard (Office of the Vice Provost for Advances in Learning)
 National University of Singapore
[I’m originally from the Caribbean, Trinidad and Tobago]
Digital Resources are Increasingly Used in Education
K12 Online
homework
On-Campus
Courses
Learning from Mathematics Problems
(Stigler et al 2010, Heffernan et al, 2016)
MOOCs
Massive Open
Online Courses
Resources for Learning Need Improvement
• Lack of measurable learning (Steenbergen-Hu & Cooper, 2013; 2014)
Problem:
Can’t predict
what works in real-world
Vision: Systems that perpetually
improve – like real teachers
Explanation
2
x =
You should imagine that
matrix(rnorm(m*n),m,n)
people’s ranking depends on
how well they do relative to how
What is the
everyone
elsestandard
is doing. error of
this random variable?
0.078
Explanation 3
Explanation
Think
of it by analogy to
A z-score is defined as the
comparing
teamsdeviations
in the NBA’s
number of standard
a specific and
point Eastern
is away from the
Western
mean.
conferences.
Novel Opportunities for Experimentation
Researchers’ Lab
Research
Students’ Class
Practice
Approach: Making Experiments Collaborative, Dynamic, Personalized
MOOClet
A
B
50%
+ ...
Continually
add conditions
N
github.com/kunanit/mooclet-engine
Education
50%
Outcome Metric
Cognitive Science
Crowdsourcing &
Human Computation
Dynamic Analysis
A
X%
Cognitive Science 2010
J. of Exp. Psych., 2013
B
100-X%
Online Education
Enhancement
Personalization
A
B
A
B
0%
100%
0%
100%
100%
0%
EDM 2015
IJAIED 2016
CHI 2016,
ACM LAS 2016
Bayesian Statistics
& Machine Learning
NIPS 2008, UAI 2013
ACIC 2016
Overview
A
•
•
B
A
+ ...
B
Dynamic Analysis
A
N
•
•
B
X%
100-X%
Enhancement
Personalization
A
B
0%
100%
100%
0%
•
•
Vision: Perpetually Improving Systems
Approach: Collaborative, Dynamic,
Personalized Experimentation
Motivation & Reflection
Adaptive eXplanation Improvement
System (AXIS)
Discovering how to personalize
Future
–Bridging Teachers, Social-Behavioral
Scientists, & Machine Learning
– Apps for Student Goal Setting & Study
Strategies
Overview
A
B
A
+ ...
B
Dynamic Analysis
A
B
X%
100-X%
Enhancement
Personalization
A
B
0%
100%
100%
0%
N
• Motivation & Reflection
• Adaptive eXplanation
Improvement System (AXIS)
• Discovering how to personalize
• Future
Experiments on Math Problems: Motivation & Reflection
Motivational Messages
Remember, the more you practice the
smarter you become!
x = matrix(rnorm(m*n),m,n)
What is the standard error?
Prompts to Reflect
x = matrix(rnorm(m*n),m,n)
What is the standard error?
Answer:
Explain why this answer is correct.
Answer:
Online educational experiments:
Bridging psychological research & real-world learning
Encouraging Growth Mindset about Intelligence
Growth
Mindset Message
Encouraging
Message
Practice-as-usual
Some of these
hard.the
Do smarter
your
Remember,
the problems
more you are
practice
you
best!
become!
N = 200 000 learners
Dweck, 2007
Effects of Messages?
•
•
Growth Mindset > Practice-as-Usual and Encouraging
Message
1% increase in Number problems attempted (p < 0.05)
Encouraging Messages not better than Practice-as-Usual (p >
0.3)
•
“You can learn anything” campaign: 11 million learners
•
Competition inviting researchers to do similar experiments
Experiments on Problems: Motivation & Reflection
Motivational Messages
Remember, the more you practice the
smarter you become!
x = matrix(rnorm(m*n),m,n)
What is the standard error?
Answer:
Prompts to Reflect
x = matrix(rnorm(m*n),m,n)
What is the standard error?
Answer:
Explain why this answer is correct.
Reflection: Help Students Help Themselves
Prompts to Reflect
x = matrix(rnorm(m*n),m,n)
What is the standard error?
Answer:
Explain why this answer is correct.
Prompts to explain “why?” help people find patterns
Cognitive Science, 2010
JEP: General, 2013
Benefits of Question Prompts to explain “Why?”
•
Psychology, Education, Philosophy, ML & AI
•
Lombrozo, 2012; Aleven & Koedinger, 2002; Chi et al, 1994; McNamara,
2004; Wellman, 2012; Woodward, 2013; DeJong & Mooney, 1986
•
General benefits:
Engagement (e.g. Siegler, 2002)
• Selective effects:
A Subsumptive Constraints account
(Williams & Lombrozo, 2010, Cognitive Science)
Explanation for “why?” interprets fact as instance of pattern
Explaining engages search for underlying generalizations
Explanation & Learning
•
Discovery of patterns
(Williams & Lombrozo, 2010, Cognitive Science)
•
Overgeneralize at expense of specific facts
(Williams et al, 2013, Journal of Experimental Psychology: General)
•
Use of prior knowledge
(Williams & Lombrozo, 2013, Cognitive Psychology)
•
Children’s causal learning
(Walker, Williams, Lombrozo, Gopnik, 2012)
•
Comparison of examples
(Edwards, Williams, Lombrozo, Gentner, 2013)
•
Contradictions & belief revision
(Williams et al 2016, Computer-Human Interaction)
Explaining Anomalies – Contradictions to Prior Beliefs
•
•
•
•
Anomalies often ignored (Chinn & Brewer, 1993)
Prompts to explain:
No effect
Reduce belief revision b/c favors contradictory prior
knowledge (Williams & Lombrozo, 2013; Williams, Lombrozo, Rehder,
2013)
• Promote belief revision
–Unifying generalizations (Williams & Lombrozo, 2010)
15
Relative Rank using standard deviation
•
Learn to rank using z-scores/standard deviation (Schwartz & Martin, 2004;
Belenky & Nokes, 2011)
Tom was ranked higher.
Sarah got 85% in a Sociology class, where the average score
was 79%, the average deviation was 8%, the minimum score
was 67%, and the maximum score was 90%.
Tom got 69% in a Art History class, where the average score
was 65%, the average deviation was 3%, the minimum score
was 42%, and the maximum score was 87%.
Personal Class
Score Average
Class
Maximum
Class
Deviation
Sarah
85%
79%
90%
8%
Tom
69%
65%
87%
3%
Ranking Rule
Use of rule
Higher
ranked
✖
Higher score
85 > 69
Sarah
✖
Greater distance from
average
(85 – 79) >
(69 – 65)
Sarah
Closer to maximum
(90 – 85) <
(87 – 69)
Sarah
More deviations
above the average
(85-79)/8 <
(69-65)/3
Tom
✖
✔
16
Design
Pre-test
Sarah got 85% in a Sociology class, where the average score
was 79%, the average deviation was 3%, the minimum score
was 67%, and the maximum score was 90%.
Tom got 69% in a Art History class, where the average score
was 65%, the average deviation was 8%, the minimum score
was 42%, and the maximum score was 87%.
Study
Items
Posttest
~6 ranked student
pairs
Explain
Write
thoughts
Few Anomalies
Overlapping
Many Anomalies
Distributed
17
Overlapping vs. Distributed
2 out of 6 anomalies condition
Ranking Rule
Overlapping condition
Distributed condition
1
2
3
4
5
6
1
2
3
4
5
6
Higher score
✖
✖
✔
✔
✔
✔
✖
✖
✔
✔
✔
✔
Greater distance from average
✖
✖
✔
✔
✔
✔
✔
✔
✖
✖
✔
✔
Closer to maximum
✖
✖
✔
✔
✔
✔
✔
✔
✔
✔
✖
✖
More deviations above average
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
18
Explaining x Anomalous Information
Write
Thoughts
Explain
Few
Single
Anomalies
Anomaly
Multiple
Anomalies
Overlapping
19
Distributed
Contributions: Increasing Motivation & Reflection
Motivational Messages
Prompts to Reflect
x = matrix(rnorm(m*n),m,n)
What is the standard error?
Remember, the more you practice the
smarter you become!
Answer:
x = matrix(rnorm(m*n),m,n)
What is the standard error?
Explain why this answer is correct.
Answer:
x = matrix(rnorm(m*n),m,n)
What is the standard error?
A
B
+ ...
N
Answer:
Explanation
A z-score is defined as the number of
standard deviations a specific point is
away from the mean.
Analyze &
Dynamically Adapt
A
X%
B
100-X%
Continually add
conditions
Overview
A
B
A
+ ...
B
Dynamic Analysis
A
B
X%
100-X%
Enhancement
Personalization
A
B
0%
100%
100%
0%
N
• Motivation & Reflection
• Adaptive eXplanation
Improvement System (AXIS)
• Discovering how to personalize
• Future
AXIS: Adaptive eXplanation Improvement System
x = matrix(rnorm(m*n),m,n)
What is the standard error?
CHI 2016
Answer:
Explanation
A z-score is defined as the number of
standard deviations a specific point is
away from the mean.
Explain why this answer is correct.
Analyze &
Dynamically Adapt
A
B
+ ...
N
Continually add
conditions
A
X%
B
100-X%
Renkl, 1997
ACM Learning @ Scale 2016
(Nominee for best paper)
Learners Rate & Generate Explanations
Linda is training for a marathon, which is a race that is 26
miles long.
Her average training time for the 26 miles is 208 minutes,
but the day of the marathon she was x minutes faster than
her average time.
Explanation
A
Explanation
B
What was Linda's running speed for the marathon in miles
per minute?
Explanation
C
26/(208 - x)
Explanation
Linda's speed is the distance she ran divided by the time it took. The
distance Linda ran was 26 miles. The time it took her was 208 – x.
Linda's speed was 26/(208 - x)
How helpful was the above information for your learning?
Completely
Perfectly
Unhelpful
Helpful
0
1
2
3
4
5
6
7
8
9
10
To help you learn, explain in your own words why the answer is correct.
Explanation
C
Dynamic Experimentation: Exploration vs Exploitation
• Multi-Armed Bandit (Reinforcement Learning)
• Randomized Probability Matching (Thompson Sampling)
Action a
Reward R
A
Explanation
The probability is 3/7 * 5/8, because the number of
cookies is changing.
Rating
How helpful was the above information for your learning?
0
1
2
3
4
5
6
7
8
9
10
Policy
Parameters
Exp1
Exp 2
Exp 3
15%
65%
20%
(Probability of Explanation being Rated Helpful)
(0 to 10 Rating by Student)
AXIS Deployment
• AXIS deployed with n=150
1
100
1
2
1
2
50
50
20
80
1
2
3
10
60
30
AXIS Policy: Probability distribution over explanations
1
2
18
0
13
0
33
44
5
6
7
8
9
10
11
100
4
50
8
0
18
0
22
0
6
0
3
0
1
0
2
0
Evaluation of AXIS explanations
• Do AXIS explanations help learning?
Problem
Problem
x = matrix(rnorm(m*n),m,n)
What is the standard error?
Answer:
x = matrix(rnorm(m*n),m,n)
What is the standard error?
Answer:
AXIS Explanation
Problem
Problem
x = matrix(rnorm(m*n),m,n)
What is the standard error?
x = matrix(rnorm(m*n),m,n)
What is the standard error?
Answer:
Answer:
Filtered Explanation
Instructor Explanation
Accuracy Increase
Impact of AXIS Explanations on Learning
Instructor reported the
AXIS explanations
comparable to their own
20%
12%
9%
3%
2%
0%
Original Problems AXIS Explanations
(No Explanations)
Filtered
Explanations
Instructor's
Explanations
Contributions
• Students help peers in course of learning
• Dynamic experimentation put data into practice
• Limitations & Future
–
–
Involve teachers
Broader applications to goal-setting and motivation
Overview
A
B
A
+ ...
B
Dynamic Analysis
A
B
X%
100-X%
Enhancement
Personalization
A
B
0%
100%
100%
0%
N
• Motivation & Reflection
• Adaptive eXplanation
Improvement System (AXIS)
• Discovering how to personalize
• Future
Discover how to personalize emails
Emails
Compare introductory message
Question about course participation
Dear Sam,
Brief
Would you please take this short survey, so
we can improve the course for future
students?
Click here to take the survey.
Mention Absence
It has been a while since you logged
into the course, so we are eager to learn
about your experience. Would you please
take this short survey, so we can improve
the course for future students?
Response Rate
Optimization through Personalization
0.3
Brief
Mention Absence
0
Overall
Low Activity
High Activity
• 14.5% more responses
Personalization
A
B
0%
100%
100%
0%
Overview
A
B
A
+ ...
B
Dynamic Analysis
A
B
X%
100-X%
Enhancement
Personalization
A
B
0%
100%
100%
0%
N
• Motivation & Reflection
• Adaptive eXplanation
Improvement System (AXIS)
• Discovering how to personalize
• Future
Collaborative, Dynamic Experimentation
Social-Behavioral
Scientists
Teachers
Enhancement
A
B
C
+ ...
N
A
B
A
B
0%
100%
50%
50%
On-Campus
Courses
Dynamic Analysis
A
X%
B
100-X%
Atlantic Causal Inference Conference, 2016; CSCW, under review
App for Experimentation on Problems in Canvas
• tiny.cc/cdesite is a website for using the app
Co-Design of Explanations, Hints, Learning Tips
Analogical explanations in Public Policy course
Probability
of
Condition
Mean
Student
Rating
Number of
Students
Standard
Deviation of
Rating
Standard
Error of the
Mean
Instructor
Rating
Instructor
Confidence
1. Quantitative
Explanation
0.23
7.26
46.00
1.87
0.28
7/10
2/5
2. Analogical
Explanation
0.77
7.48
56.00
1.59
0.21
5/10
2/5
Version
Student Interaction with Calculus Problem
From the graph of y=f'(x) on its entire domain of [a,h],
at which x-value(s) is f' least?
Feedback
Correct! This is the lowest
point on the graph of f'.
•
•
•
•
x=e
x=b
x=a
x=c
Feedback and Prompt to Review
Correct! This is the lowest point on the
graph of f'.
If you would like to review this sort of
question further, you can look back at your
notes about how we found where the
function g was greatest and least.
Or you can look at the relevant video. Click
this link to open the video in a new window.
Learning Tips in Calculus course
SEM Next
Problem
Accuracy
Instructor
Rating
Instructor
Confidence
Version
Probability
of
Condition
Mean
Student
Rating
Number
of
Students
Standard
Deviation
of Rating
SEM Rating
Next
Problem
Accuracy
1. Feedback
0.48
9.00
18.00
2.18
0.51
0.64
0.07
7/10
4/5
2. Feedback
and Prompt
to Review
0.52
9.10
18.00
1.91
0.45
0.72
0.07
8/10
4/5
Qualitative Instructor Feedback
•
•
•
•
Lowered Barriers: “I’m not aware of any tools that do this sort of
thing… even if I found one, I don't think that I have the technical
expertise to incorporate it”
Reflection on pedagogy: “I never really seriously considered typing
up multiple versions as we are now doing. So even if we don't get
any significant data, that will have been a benefit in my mind”
Making research practical: “a valuable tool. Putting in the hands
of the teacher to understand how their students learn. Not just in
broad terms, but specifically in their course” “you must know
plenty of general things about how students learn, whereas I know
specific things about how they get calculus”
Directly helping students: “improved the experience of many of
the students by giving them answers that are more helpful… the
earlier ones can help improve the experience of the later students.
That’s pretty neat”
Apps for Student Goal Setting
• tiny.cc/keepingengaged
• On-Demand prompts to specify goals, think through
obstacles, receive reminders
Student Study Strategies: Reflective Questions
App providing Situational, On-Demand Prompts
• tiny.cc/rqs Reflective Questions Strategy
Conclusion
A
•
•
B
A
+ ...
B
Dynamic Analysis
A
N
•
•
B
X%
100-X%
Enhancement
Personalization
A
B
0%
100%
100%
0%
•
•
Vision: Perpetually Improving Systems
Approach: Collaborative, Dynamic,
Personalized Experimentation
Motivation & Reflection
Adaptive eXplanation Improvement
System (AXIS)
Discovering how to personalize
Future
–Bridging Teachers & Scientists
–Apps for Goal Setting & Study Strategies
• Postdoc available at NUS
–[email protected]
–nus.ed.sg/alset Institute for Application of
Learning Sciences to Educational Technology
Thank You!
• Juho Kim, Krzysztof Gajos, Anna Rafferty
• Harvard VPAL (Vice Provost for Advances in Learning)
Research
• Tania Lombrozo & Tom Griffiths
• Candace Thille & John Mitchell
• Jascha Sohl-Dickstein, PERTS, Khan Academy
• Sam Maldonado
• Lytics Lab
Conclusion
A
•
•
B
A
+ ...
B
Dynamic Analysis
A
N
•
•
B
X%
100-X%
Enhancement
Personalization
A
B
0%
100%
100%
0%
•
•
Vision: Perpetually Improving Systems
Approach: Collaborative, Dynamic,
Personalized Experimentation
Motivation & Reflection
Adaptive eXplanation Improvement
System (AXIS)
Discovering how to personalize
Future
–Bridging Teachers & Scientists
–Apps for Goal Setting & Study Strategies
• Postdoc available at NUS
–[email protected]
–nus.ed.sg/alset Institute for Application of
Learning Sciences to Educational Technology