Modes and Mechanisms of
Game-like Interventions
in Intelligent Tutoring Systems
Dovan Rai
_
Department of Computer Science
Worcester Polytechnic Institute
Joseph E. Beck
Ivon Arroyo
Charles Rich
Kristen Dicerbo
1
Educational Technology
Games
Intelligent
TutoringSystems
2
Rich experiences !
Games
! Cognitive Overload
! Design Constraints
! Time Constraints
! Resource Constraints
3
Learning gain !
Intelligent
TutoringSystems
Can they hold students attention for prolonged time?
Can they provide adequate platform for
rich learning experiences?
4
Integrate Game and Tutors ?
Games
Intelligent
TutoringSystems
5
Game-like Interventions in Intelligent Tutoring Systems
6
Deconstruction of Games
Game-elements
Game-like-elements
Games
Intelligent
TutoringSystems
Gamifica5on
Mini-games
Game-like Interventions
7
Modes and Mechanisms of
Game-like Interventions
In Intelligent Tutoring Systems
ResearchQues5ons
1.WhataredifferentMODESofgame-likeinterven5ons?
2.WhataretheMECHANISMSoflearningoutcomes
ingame-likeinterven5ons?
8
What are the different ways we can
have game-like interventions in
computer tutors?
games
Tutors
games
games
9
game-like interventions
COGNITIVE
METACOGNITIVE
Monkey s Revenge
Learning Dashboard
game-like math tutor
Gamification
Tutors
AFFECTIVE
Mosaic
Mini-games
10
EXPERIMENTS
Learning
Dashboard
METACOGNITIVE
Monkey
s Revenge
COGNITIVE
Gamification
game-like math tutor
1.  Pilot study (N=58)
1.  Pilot study (N=58)
2.  RCT (N=39)
Modified intervention
2.  Pilot study (N=39)
improved intervention
3.  RCT (N=297)
Modified study design
3.  RCT (N=252)
Modified intervention
Improved intervention
Modified study design
4.  RCT (N=252)
Mosaic
Mini-games
RCT (N=186)
11
game-like interventions
METACOGNITIVE
COGNITIVE
Monkey s Revenge
Progress Page
game-like math tutor
progress report with
Tutor
gamification
AFFECTIVE
Mosaic
Mini-games on math
12
Math problems wrapped in a visual narrative
13
Game Elements
Achievements
Fantasy
Narrative
Levels
Personalization
Collecting badges
Competition
Visual feedback
Resource Management
Speed
Structure building
Quests
Tile laying
14
Game-like Math Tutor
Narrative
Personalization
Visual feedback
Collecting badges
Structure building
15
Experiments with Monkey’s Revenge
1.  Pilot study (N=58)
Modified intervention
2.  Pilot study (N=39)
3.  RCT (N=297)
Modified intervention
Modified study design
4.  RCT (N=252)
16
EXPERIMENT DESIGN
Tutor version
Visual
feedback
Narrative
Other game-like
elements
Monkey s Revenge
✔
✔
✔
without visual feedback
✕
✔
✔
without narrative
✔
✕
✔
Basic tutor
✕
✕
✕
17
Monkey’s
Revenge
Without
narrative
18
Basic Tutor
19
Monkey’s
Revenge
Without
Visual feedback
20
Hypothesis
Cangame-liketutor
enhancestudent’senjoyment?
generatehigherlearninggain?
WITHOUT
crea5ngcogni5veoverload
andtakingtoomuch5meawayfromprac5ce
21
Isitengaging?(Likingandsa5sfac5on)
Isiteffec5ve?(learninggain)
Isitefficient?(5meoverload?cogni5veoverload?)
22
Is it engaging? (Liking and satisfaction)
SurveyResponsesacrosstutors(meansand95%CI)
Tutorversion
Liketutor
Hadfun
Tutorhelped
BeUer
Monkey sRevenge
4.0±0.3
4.1±0.4
3.9±0.3
3.9±0.3
withoutvisual
feedback
3.9±0.4
3.9±0.4
3.6±0.4
3.7±0.4
withoutnarra5ve
3.6±0.5
3.3±0.5
3.2±0.5
3.8±0.5
Basictutor
3.0±0.5
3.0±0.5
3.1±0.5
3.4±0.5
Students liked the system.
Theyshowedmorelikingofthetutorversion
withgame-likeelements.
23
Is it effective? (learning gain)
learninggainacrosstutors(meansand95%CI)
Tutor
Pretestpercentcorrect
Learninggain
Monkey sRevenge
66%
10%±9%
withoutvisualfeedback
69%
5%±7%
withoutnarra5ve
70%
7%±8%
Basictutor
74%
3%±7%
24
Is it efficient? (cognitive overload?)
Studentperformanceacrosstutors(meansand95%CI)
Tutor
Pretestpercent Problemscorrectinthetutor
correct
(max=27)
Monkey sRevenge
68%
20.3±1.1
withoutvisualfeedback
64%
19.8±2
withoutnarra5ve
70%
18.6±1.2
Basictutor
74%
18.5±1.5
We did not have a robust way to measure cognitive load. But at least, cognitive
overload and distraction is not obstructing students in game-like version to
perform well within the tutor.
25
Is it efficient? (time overload?)
Tutor
Total5me
(inminutes)
Non-tutor5me
(inminutes)
Monkey sRevenge
50
10
withoutvisualfeedback
47
13
withoutnarra5ve
42
9
Basictutor
56
5
26
Conclusions
Cangame-liketutor
enhancestudent’senjoyment?YES
generatehigherlearninggain?noconclusion
withoutcrea5ngcogni5veoverload
andtakingtoomuch5meawayfromprac5ce
seemsyes
27
Limita5onsandFutureWork
Noconclusiveresultsonlearninggains!
Possiblereasonsare:
Interven5onwasbrief
Involvedvarietyofskills
Alotofkidsdidnotcompletethepost-test.
Thelargestandarderrorsuggestsstudentswerenottakingthetest
seriously
Extendinterven5onformul5plesessions.
28
game-like interventions
COGNITIVE
METACOGNITIVE
Monkey s Revenge
Learning Dashboard
game-like math tutor
Gamification
Tutors
AFFECTIVE
Mosaic
Mini-games
29
game-like interventions
METACOGNITIVE
COGNITIVE
Learning Dashboard
Progress Page
gamificationprogress report with
Tutor
gamification
AFFECTIVE
Mosaic
Mini-games on math
30
MathSpring:IntelligentTutoringSystem
31
Learning Dashboard
MathTree
StudentProgressPage(SPP)
TopicDetails
32
33
34
Student Progress Page (SPP)
35
Student Progress Page (SPP)
EFFORT
MASTERY
36
Pepperplants:representa5onofeffort
Effort è Pepper plant grows
No Effort è Pepper plant wilts
37
Hypothesis
Providingmeta-cogni5vesupportthroughtheSPPwill
generatecascadingeffects:
àenhancestudents’affec5vestate
àwhichshouldincreasestudentengagementand
produc5vebehaviorssuchasspending5meonhelp,
whichshouldleadtohigherlearning
38
Experiments with Learning Dashboard (SPP)
1.  Pilot study (N=58)
2.  RCT (N=39)
3.  RCT (N=252)
39
ExperimentalGroups
{When students show negative affect}
NOSPP
SPP
SPP
SPP
SPP button
not present
SPP button
PROMPT
SPP
FORCE
SPP
(N=49)
(N=53)
(N=52)
(N=55)
40
ExperimentandResults
Grade7(N=209)
California
3consecu5veclasssessions
PRETEST
survey
Session1
Session2
POSTTEST
Session3
survey
MathSpringpromptedstudentstoself-reporttheiraffect
“Howinterestedareyoufeelingrightnow?”Notatall
interested(1)…somewhatinterested(3)…extremely
interested(5)
41
DoestheSPPImpactStudentAffect?
{on negative affect}
Excitement
Interest
NOSPP
SPP
SPP
SPP
SPP button
not present
SPP button
PROMPT
SPP
FORCE
SPP
2.5
2.6
2.6
2.8
2.7
2.7
2.7
2.5
No Conclusion
42
IsSPPusageassociatedwithPosi5veAffect?
{on negative affect}
NOSPP
SPP
SPP
SPP
SPP button
not present
SPP button
PROMPT
SPP
FORCE
SPP
43
IsSPPusageassociatedwithPosi5veAffect?
{on negative affect}
NOSPP
SPP
SPP
SPP
SPP button
not present
SPP button
PROMPT
SPP
FORCE
SPP
!MathSpringencouragedSPPusageintwoofthecondi5ons(prompt
andforce)whenlowinterestorlowexcitement
wasselfreported.
44
{on negative affect}
NOSPP
SPP
SPP
SPP
SPP button
not present
SPP button
PROMPT
SPP
FORCE
SPP
45
NOSPP
SPP
SPP button
not present
SPP button
Interestwasposi5velyassociatedwithSPPusage
(r=.24,p=.023)
Excitementalsowasposi5velyassociatedwithSPPbutthis
didnotreachsignificance(r=.13,p=.26).
46
NOSPP
SPP
SPP button
not present
SPP button
SPP usage à positive affect ?
or
positive affect à SPP usage ?
Interestwasposi5velyassociatedwithSPPusage
(r=.24,p=.023)
Excitementalsowasposi5velyassociatedwithSPPbutthis
didnotreachsignificance(r=.13,p=.26).
47
NOSPP
SPP
SPP button
not present
SPP button
SPP usage à positive affect ?
or
positive affect à SPP usage ?
controlledforstudents pre-exis5ngaffectasderived
fromthepre-affectsurvey
48
SPP usage à positive affect ?
or
positive affect à SPP usage ?
NOSPP
SPP
SPP button
not present
SPP button
controlledforstudents pre-exis5ngaffectasderived
fromthepre-affectsurvey
Interestwass5llsignificantlyassociatedwithSPPusage
(r=.25,p=.036)
Excitement(r=.14,p=.3)
49
HowdoCondi5onsImpactAffec5veStateTransi5ons?
(e.g., if they got “stuck” in the negative deactivating states)
50
HowdoCondi5onsImpactAffec5veStateTransi5ons?
Models created by Wixon and Kasia
Visual representation of the high-level path models for excitement in the
no-button, prompt and force conditions from left to right, respectively
51
SPPisaffec5velybeneficialforstudents,
Promo5ngExcitement,
&
Decreasingthelikelihoodofpaths
thatleadtoBoredom.
52
Limita5onsandFutureWork
Frompersonalobserva5onofexperimentersand
teachers,studentsseemtoenjoySPP
WeneedbeUermetricstomeasurestudentaffect
andengagement
MakeSPPmoreaccurateandintui5ve.
Longerstudy!
53
game-like interventions
COGNITIVE
METACOGNITIVE
Monkey s Revenge
Learning Dashboard
game-like math tutor
Gamification
Tutors
AFFECTIVE
Mosaic
Mini-games
54
game-like interventions
COGNITIVE
METACOGNITIVE
Mosaic
Progress Page
mini-games progress report with
Tutor
gamification
AFFECTIVE
Mosaic
Mini-games on math
55
LONDON
56
LONDON
57
MANHATTAN
58
MANHATTAN
59
Gameasaffec5verepair
Tutor
Mosaic
mini-game
After playing mini-games, students come
back to tutor with better affect.
60
ExperimentalGroups
No-Mosaic
N=60
Prompt
Mosaic
Force
Mosaic
N=62
N=64
61
ExperimentandResults
Grade7students(N=186)
PRETEST
survey
Session1
POSTTEST
survey
MathSpringpromptedstudentstoself-reporttheiraffect
“Howinterestedareyoufeelingrightnow?”Notatall
interested(1)…somewhatinterested(3)…extremely
interested(5)
62
Hypothesis
Domini-games
enhancestudentaffect?
enhancestudentenjoymentandpercep5onoftutor?
Improvetheirengagementasaresultofrepairedaffect?
63
Self-reportontheirexperienceinMathspring;
Studentsindifferentexperimentalgroups
Group
Performed
Participants
Total
well
with
complete
participants
(max 5)
survey
Learned a
Lot
(max 5)
Enjoy using
Mathspring
(max 5)
No Mosaic
60
34
3.3 (1)
2.3 (1.1)
2.6 (1.2)
Prompt
Mosaic
62
42
3.5 (1.2)
2.5 (1)
2.9 (1.2)
Force Mosaic
64
41
3.4 (1.4)
2.4 (1.2)
2.9 (1.3)
64
Self-reportontheirexperienceinMathspring;
StudentswhousedMosaicandwhodidnotuseMosaic
Participants Performed
Total
well
with complete
participants
survey
(max 5)
Group
Learned a
Lot
(max 5)
Enjoy using
Mathspring
(max 5)
Did not use
Mosaic
88
54
3.11
2.17
2.57
Used Mosaic
98
63
3.59
2.62
3.08
0.04*
0.03*
0.02*
p-value
65
InterestandFrustra5onaveragedoverpar5cipants
whousedanddidnotuseMosaic
Group
Interest
(max 5)
Frustration
(max 5)
Participants who
skipped affect survey
Did not use Mosaic
(N=88)
2.4 (N=55)
2.5 (N=66)
16 (18%)
Used Mosaic (N=98)
2.5 (N=65)
2.4 (N=60)
10 (10%)
p-value
0.4
0.08
66
Limita5onsandFutureWork
Thewholeexperimentwasonlyforasingle
classsession.
Theinterven5onwasbriefandaffectsampling
inadequate.
67
2.WhataretheMECHANISMSof
learningoutcomes
ingame-likeinterven5ons?
68
CausalModelingiscontestedandnotawidelyused
witheduca5onaltechnologycommunity.
Therefore,wemadeacasestudyfirsttoanalyzeand
understandthisapproach.
WeareusingCausalmodelingasan
ExploratorytoolratherthanaConfirmatoryone.
69
Causal Modeling Tool
TETRAD
hUp://www.phil.cmu.edu/tetrad/
Doug Selent’s enhancement
hUps://sites.google.com/site/dougstetrad/tetrad
70
CausalmodelingonMonkey’sRevenge
CausalmodelingonMathspringSPP
CausalmodelingonMosaic
71
CausalmodelingonMathspringSPP
We created variables from survey data
Samples:
MathLike : Do you like your math class?
IntePre: In general, do you feel interested when solving math problems?
AnxiPre: Do you get anxious while solving math problems? DiffConcentration: Do you have difficulty concentrating?
enjoyMathSpring: I enjoyed using the system. 72
Student State variables
Student State
DescripAon
SOF
SolvedonfirstaUemptwithouthelp
ATT
Answeredaper1-2incorrectaUemptsandself-corrected,
withouthelp.
SHINT
Answeredwiththeuseofsomehelp,butnotall,inatmost
2aUempts.
GUESS
AnsweredaperseveralaUempts,morethan2aUempts
NOTR
Notenough5metoread
GIVEUP
Enough5metoread,butmovedonbeforeanswering.
73
INTE:averagevaluefor“HowInterestedareyou?”
EXC:averagevaluefor“Howexcitedareyou?”
SPP:numberof5mesSPPaccessedbystudent
74
Clustersofvariables
•  Positive Learning vs. Negative learning behavior
•  Performance Oriented vs. Enjoyment Oriented
75
Positive learning vs. negative learning behavior
76
Positive learning vs. negative learning behavior
Assignment of these clusters are logical demarcation based on our domain
77
knowledge rather than actual statistical distinction.
Posi5ve
Learningbehavior
Nega5ve
Learningbehavior
Effort
(SOF,SHINT,ATT)
è
NoEffort
(NOTR,GUESS,
SKIP,GIVEUP)
Pepperplant
grows
è
Pepperplant
wilts
78
Enjoyment Oriented vs. Performance Oriented
79
Enjoyment Oriented
students who used more tutor help
(SHINT) reported enjoying the system
more (enjoyedSystem), finding the tutor
more helpful (tutorHelpful) and being more
excited (EXC) and interested (INTE).
Performance Oriented
students who found math difficult
(mathDifficult) solved less problems
correctly in the first attempt (SOF),
reported higher anxiety (AnxiPre) and
higher frustration (frusPre) 80
EFFORT
MASTERY
SOF à Mastery
SHINTà Effort
81
Specialpepperplantsasreward
Monster Pepper
Rainbow Pepper
SOF-SOFsequences
Excep5onalKnowledge
SHINT-SOFsequences
Excep5onalHelpUsage
82
Modes and Mechanisms of
Game-like Interventions
In Intelligent Tutoring Systems
Conclusions
83
Major Limitations
•  LackofUsabilitystudies
•  ShortInterven5ondura5on
•  InadequateStudydesign(objec5verobustmeasures
otherthansurveys)
84
Conclusions
Experimental data and classroom observations
indicate that we are on the right path of creating
optimal game-like interventions.
We need longer intervention duration and more
robust study design to generate evidence that our
interventions can enhance both enjoyment and
learning gains.
85
What’sthenextop5malcombina5onof
tutorsandgames?
Intelligent
Tutoring
Systems
Games
Students enter games with prior
knowledge of the content.
as
assessment
tools
We can focus on creating rich learning experiences
within an educational game while intelligent tutoring
systems take care of providing robust learning
86
Thank You !
87