Students’ enjoyment of a game-based tutoring system
G. Tanner Jackson1, Natalie L. Davis1, Danielle S. McNamara1
University of Memphis
Memphis, Tennessee
{gtjacksn, nldavis, dsmcnamr}@memphis.edu
1
Abstract. Many Intelligent Tutoring Systems (ITSs) have started to incorporate
game-based components in an attempt to improve student engagement during
system interactions. iSTART-ME is a new game-based learning environment
that was developed on top of an existing ITS (iSTART). Initial investigations
focused on individual, isolated components. However, to increase ecological
validity, the current study was designed such that a smaller number of students
(n=9) engaged with the entire system over an extended period of time, spanning
several weeks. The participants indicated that they enjoyed the game-based
aspects of the system significantly more than the non-game aspects. These
results support the use of iSTART-ME as a system that promotes long-term
enjoyment.
Keywords: Serious Games, Intelligent Tutoring Systems, game-based tutoring
1 Serious Games
Two common areas of educational research focus on discovering how students are
motivated to learn and how to enhance achievement in learning. A great deal of
research is based on the notion that placing educational material or educational tasks
in meaningful, interesting, or engaging contexts will aid or increase motivation and
learning. These contexts can be realized with simple additions such as basic forms of
feedback or through multiple enhancements, as with educational games [1], [2], [3],
[4].
It is generally assumed that games are more engaging and potentially ‘could’ lead
to better or more sustained learning [5], [6]. [7]. These gaming environments differ
widely, ranging from incorporating points accrued within a traditional tutoring
environment to full-fledged games with narration, beginning and end states,
interaction, rules, and reward system [6]. Games require the application of skills and
knowledge to either win or explore the constraints of a complex world [8]. However,
for the game to be effective, the learner must want to play the game. Interest in the
actual content of the game is a preferred method of obtaining involvement, but not all
learners share interests. While the content is important for determining interest,
perhaps more critical is how the content is framed. Thus, the game itself can be used
as a springboard to capture the interest of the student.
2 Development of iSTART-ME
The Interactive Strategy Training for Active Reading and Thinking (iSTART) tutor is
a web-based system for young adolescent to college-aged students designed to
improve reading strategies [9]. iSTART training consists of three main modules:
Introduction, Demonstration, and Practice. The Introduction module contains three
animated agents that engage in a vicarious dialogue to introduce the learner to the
concept of self-explanation and each of the reading strategies. The Demonstration
module includes two animated agents who generate and discuss the quality of
example self-explanations and prompt the learner to identify which strategies may
have been used within each example. The Practice module requires learners to
generate their own self-explanations and an animated agent (Merlin) provides
qualitative feedback on how to improve the self-explanation quality. This feedback is
based on a natural assessment algorithm that has demonstrated performance
comparable to humans [10]. An Extended Practice environment continues this
generative practice over a longer time period and allows teachers to assign specific
texts.
Students using iSTART have demonstrated significant improvement in reading
comprehension, comparable to the performance within SERT with average effect
sizes ranging from .68 to 1.12 depending on prior knowledge of the learner [11].
Learners have consistently made significant improvements through interacting with
iSTART. However, skill mastery requires long-term interaction with repeated practice
[12]. One unfortunate side effect of this long-term interaction is that students often
become disengaged and uninterested in using the system [13]. Thus, iSTART-ME
(motivationally enhanced) has been developed on top of the existing ITS and
incorporates serious games and other game-based elements [14], [15].
3 iSTART-ME Selection Menu and Games
The iSTART-ME game-based environment builds upon the existing iSTART system.
The main goal of the iSTART-ME project is to implement several game-based
principles and features that are expected to support effective learning, increase
motivation, and sustain engagement throughout a long-term interaction with an
established ITS. iSTART-ME has been extensively described in previous work [14],
[15], therefore only a brief description will be presented here.
The previous version of iSTART automatically progressed students from one text
to another with no intervening actions. The new version of iSTART-ME is controlled
through a selection menu (see Figure 1 for screenshot of the selection menu).
Researchers claim that motivation and learning can be increased through multiple
elements of a task including feedback, fantasy, personalization, choice, and curiosity
[1], [4]. Therefore these features have been incorporated into the design of the
iSTART-ME selection menu. This selection menu provides students with
opportunities to interact with new texts, earn points, advance through levels, purchase
rewards, personalize a character, and play educational mini-games (designed to use
the same strategies as in practice).
Fig. 1. Screenshot of iSTART-ME selection menu
Several educational mini-games have been incorporated within iSTART-ME. In
general, each of these mini-games has been designed so that a single session should
be playable to completion within 10–20 minutes. The compilation of mini-games
model strategy use and aim to improve: identification of strategies, generation of new
self-explanations, metacomprehension awareness, and/or vocabulary. Each minigame focuses on one or two of these areas of improvement, and situates it within a
game-based environment. After completion of a mini-game, students are directed
back to the main iSTART-ME selection screen (pictured in Figure 1).
There are three methods of generative practice (Coached Practice, Showdown, and
Map Conquest) as well as three isomorphic identification mini-games (Strategy
Match, Bridge Builder, and Balloon Bust). Coached Practice is the updated version of
the original iSTART practice, in which learners are asked to generate their own selfexplanation when presented with a text and specified target sentence. Students are
guided through practice by Merlin, a wizard who provides qualitative feedback for
user-generated self-explanations. Merlin reads sentences aloud to the participant and
then asks the participant to self-explain each target sentence. After a self-explanation
is submitted, points are awarded, Merlin provides verbal feedback on how to improve
the self-explanation, a feedback bar indicates overall quality, and the students may
either revise their self-explanation or move on to the next target sentence.
Showdown and Map Conquest are two game-based methods of practice that use
the same natural language assessment algorithm as Coached Practice. In Showdown,
students compete against a computer player to win rounds by writing better selfexplanations. After the learner submits a self-explanation, it is scored, the quality
assessment is represented as a number of stars (0-3), and an opponent self-explanation
is also presented and scored. The self-explanation scores are compared and the player
with the most stars wins the round. The player with the most rounds at the end of the
game is declared the winner. Map Conquest is the other game-based method of
practice where students generate their own self-explanations. In this game the quality
of a student’s self-explanation determines the number of dice that student earns.
Students place these dice on a map, and use them to conquer neighboring opponent
territories, which are controlled by two virtual opponents.
iSTART-ME also contains three isomorphic versions of identification games. All
three games require students to view an example self-explanation (along with original
text) and identify which iSTART strategy was used. Each game situates this same
task within different combinations of game features. Strategy Match, the most basic
game, consists of a drag and drop interface where the students can earn points and
move up levels. Bridge Builder uses a similar drag and drop interface with points and
levels, but it also includes a virtual scene where the users slowly construct a bridge by
identifying the correct strategies. Balloon Bust adds in a perceptual element by
including points, levels, and a virtual scene where users must follow and click on the
correct balloons in order to identify which strategy was used.
4 Current Study
Previous research with iSTART-ME has focused on single session studies that
investigated individual elements within the system. The current study deviates from
this precedent and includes fewer participants that interacted with the full iSTARTME system across multiple sessions spanning several weeks. This current work was
designed to improve ecological validity and allow for student interactions that mimic
how iSTART-ME could be implemented within a classroom environment. All
participants (n=9) completed the full iSTART-ME training, including Introduction,
Demonstration, Practice, and the Selection Menu. After completing the initial training
and Practice module, students spent the remainder of the sessions freely using all
features within the Selection Menu. After interacting with iSTART-ME for 8
sessions, participants completed a posttest survey, which included questions about
attitudes, enjoyment, and motivation.
4.1 Module Comparisons
During the posttest survey participants responded to a series of questions for each
module within iSTART-ME (Introduction, Demonstration, Practice, Selection Menu).
Table 1 displays the questions along with means and standard deviations. A withinsubjects ANOVA that compared the four iSTART-ME modules for the question “I
had fun using this module” revealed a significant effect, such that ratings for the
introduction module were significantly lower than all other modules, ratings for
demonstration and practice were equivalent, and the selection menu was rated
significantly more fun than all other modules, F(1,8) = 28.89, p = .001. Another
within-subjects ANOVA on the modules was performed for the survey question “This
module was easy to use.” This second ANOVA revealed no significant differences
between modules, with all modules equivalently easy to use (scale of 1-6, with all
means above 4). The equivalence of ease is a particularly encouraging result because
it indicates that the new Selection Menu interface was not too complex and that
students understood how to interact with it to make selections. A final within-subjects
ANOVA compared each of the modules for the question “I would recommend this
module to a friend.” This analysis yielded a significant effect, such that ratings for the
introduction module were significantly lower than all other modules, ratings for
demonstration and practice were equivalent, and the selection menu was rated higher
than all other modules, F(1,8) = 20.17, p = .002.
Table 1. Means, standard deviations, and p value for module questionnaire (scale 1-6, with
higher numbers indicating stronger agreement).
I had fun using this module
This module was easy to use
I would recommend this
module to a friend
Intro
Demo
Practice
Menu
1.22a
(0.44)
5.00a
(1.32)
1.44a
(0.53)
3.00b
(1.58)
4.78a
(1.39)
2.78b
(1.64)
2.78b
(1.20)
4.89a
(1.05)
2.89b
(1.27)
4.33c
(1.73)
4.78a
(1.39)
4.11c
(1.90)
*Subscripts indicate significantly different subgroups within a row, p < .05.
4.2 Game Comparisons
The posttest survey also included questions designed to address participants’ attitudes
towards each of the generation and identification games. Two separate comparisons
were made to investigate differences within the group of generation games and the set
of isomorphic identification games.
Within-subjects ANOVAs on the three generation games yielded significant
differences for several posttest survey questions (means and subgroups are displayed
in Table 2).One of the most interesting results from these comparisons is the
seemingly conflicting ratings for Map Conquest. This game was rated as significantly
more frustrating than the other generation games, F(1,8)=7.84, p=.02; however, it was
also rated as the most enjoyed generation game, F(1,8)=7.20, p=.03. Follow-up
interviews with participants indicated that the map portion of the game was initially
confusing (and therefore frustrating), but was also one of the most game-like and
enjoyable aspects of the environment.
Within-subjects ANOVAs were also conducted on the posttest survey to compare
ratings between the identification games. There was a clear trend for Balloon Bust to
receive the most positive ratings (for almost every question). Balloon Bust was rated
as significantly more enjoyable and participants were more likely to play it again
compared to the other two identification games, F(1,8)=6.67, p=.04, F(1,8)=12.11,
p=.01, respectively. These findings are not particularly surprising, given that Balloon
Bust combines the most number of game features within a single identification game.
These results indicate that the mini-games with more game-like elements (i.e., Map
Conquest and Balloon Bust) received the highest positive ratings. This is similar to
previous research that has suggested a positive linear relation between the number of
game elements and user enjoyment (Cordova & Lepper, 1996; Papastergiou, 2009).
Table 2. Means (SD) for mini-game questionnaires (scale 1-6, with higher numbers indicating
stronger agreement).
Generation Games
Prac Show
Map
I liked the graphics in this game
3.44a 3.44a
3.89a
(1.24) (1.13) (1.45)
I liked the sound effects in this game 2.33a 4.33b
4.22b
(1.23) (1.22) (1.56)
I liked the music in this game
2.78a 4.22b
4.00b
(1.64) (1.09) (1.73)
This game was fun to play
2.56a 3.33a
3.44a
(1.13) (1.41) (1.88)
I would play this game again
2.56a 3.22a
3.22a
(1.42) (1.86) (1.79)
Identification Games
Match Bridge Balloon
3.50x
3.50x
4.12x
(1.20) (1.69) (1.73)
3.13x
3.62x
3.75x
(1.13) (1.60) (1.50)
3.50x
3.75x
3.75x
(1.51) (1.75) (1.49)
3.38x
3.50x
4.62x
(0.92) (0.93) (1.30)
2.50x
3.62y
4.62z
(1.20) (0.74) (1.30)
This game was frustrating
3.13x
(2.03)
2.44a 2.33a
(1.33) (1.22)
4.00b
(2.06)
2.50x
(1.31)
2.62x
(1.06)
I enjoyed playing this game
2.67a 3.44ab
3.67b
3.00x
3.62x
4.38y
(1.32) (1.74) (1.73)
(1.41) (1.06) (1.19)
Prac = Coached Practice, Show = Showdown, Map = Map Conquest, Match = Strategy
Match, Bridge = Bridge Builder, Balloon = Balloon Bust
*Subscripts indicate significantly different subgroups within a row, p < .05.
5 Conclusions
The current study provides an exploration of how the iSTART-ME game-based
environment is viewed by students over a long-term interaction. The results support
the current design of iSTART-ME and indicate that students enjoyed interactions with
the new game-based aspects of the system over an extended period of time.
Specifically the students provided higher ratings for those modules and mini-games
that contained more game-like aspects. One particularly interesting finding was that
Map Conquest received the highest ratings for both frustration as well as enjoyment.
While the interface complexity of Map Conquest may have contributed to the
frustration, the students persisted and ultimately the enjoyment of the game was able
to counteract the negative effects from the initial conflict.
One limitation of the current study is the small sample size. Previous work with
iSTART-ME has adopted the traditional empirical approach and used larger samples
that focused on the immediate effects of game-based learning. However, long-term
studies offer significantly more interactions per participant, and provide insight into
effects that may develop differently over time. Additionally it is often impractical,
and expensive, to conduct long-term laboratory studies that span multiple weeks and
have large samples. Therefore, this study was conducted on a smaller sample and
served as a longer term investigation to explore how students would use and benefit
from the newly developed game-based system. Of course, a smaller sample size limits
generalization to a broader population of users. However, despite this potential
limitation, the data indicate several interesting trends that are also supported by
previous research. The game-based versions of practice were preferred over the nongame practice (supported by [1], [4]. The isomorphic identification games each
possess an added or subtracted game element, and the posttest survey questions seem
to indicate a linear relation between the number of game elements and enjoyment
(also supported by [1], [4]. Therefore, despite its limitations, the current work still
contributes to the growing body of research with serious games.
The results from the current study are encouraging because they indicate that
iSTART-ME can successfully sustain user enjoyment over an extended amount of
time. This finding provides a foundation for future work that more fully investigates
the timelines of effects for specific game elements (e.g., competition, challenge,
variety, control, etc.). The current ability to provide enjoyable prolonged interactions
suggests that iSTART-ME is a significant improvement over previous versions, and it
has the potential to greatly increase skill acquisition through a higher likelihood of
interested, returning users [5], [6], [7].
Acknowledgments. This research was supported in part by the National Science
Foundation (IIS-0735682). Any opinions, findings, and conclusions or
recommendations expressed in this material are those of the authors and do not
necessarily reflect the views of the NSF.
References
1.
2.
3.
4.
5.
6.
7.
8.
Cordova, D.I., Lepper, M.R.: Intrinsic Motivation and the Process of Learning Beneficial
Effects of Contextualization, Personalization and Choice. Journal of Educational
Psychology, 88, 715-730 (1996)
Jackson, G.T., Graesser, A.C.: Content Matters: An Investigation of Feedback Categories
within an ITS. In: R. Luckin, L. Loedinger & J. Greer (eds.), Artificial Intelligence in
Education: Building Technology Rich Learning Contexts that Work, pp. 127-134.
Netherlands: IOS Press (2007)
McNamara, D.S., Jackson, G.T., Graesser, A.C.: Intelligent Tutoring and Games (ITaG).
In: Y.K. Baek (ed.), Gaming for Classroom-based Learning: Digital Role-playing as a
Motivator of Study. IGI Global (2010)
Papastergiou, M.: Digital Game-based Learning in High School Computer Science
Education: Impact on Educational Effectiveness and Student Motivation. Computers and
Education, 52, 1-12 (2009)
Garris, R., Ahlers, R., Driskell, J.E.: Games, Motivation and Learning: A Research and
Practice Model. Simulation and Gaming, 33, 441-467 (2002)
Gee, J.P.: What Video Games Have to Teach us About Learning and Literacy. New York:
Palgrave Macmillian (2003)
Steinkuehler, C.A.: Massively Multiplayer Online Videogaming as Participation in a
Discourse. Mind Culture & Activity, 13, 38-52 (2006)
Gredler, M.E.: Games and Simulations and Their Relationships to Learning. In: D.H.
Jonassen (ed.), Handbook of Research on Educational Communications and Technology,
pp. 571-582. Mahwah, NJ: Lawrence Erlbaum Associates (2004)
9.
McNamara, D.S., Levinstein, I.B., Boonthum, C.: iSTART: Interactive Strategy Trainer
for Active Reading and Thinking. Behavioral Research Methods, Instruments and
Computers, 36, 222-233 (2004)
10. Jackson, G.T., Guess, R.H., McNamara, D.S. Assessing Cognitively Complex Strategy
Use in an Untrained Domain. Topics in Cognitive Science, 2, 127-137 (2010)
11. McNamara, D.S., O'Reilly, T., Rowe, M., Boonthum, C., Levinstein, I.B.
iSTART: A Web-Based Tutor that Teaches Self-Explanation and Metacognitive
Reading Strategies. In D.S. McNamara (Ed.), Reading comprehension strategies:
Theories, Interventions, and Technologies, pp. 397-420. Mahwah, NJ: Erlbaum
(2007)
12. Jackson, G.T., Boonthum, C., McNamara, D.S.: The Efficacy of iSTART Extended
Practice: Low Ability Students Catch Up. In: J. Kay & V. Aleven (eds.), Proceedings of
the 10th International Conference on Intelligent Tutoring Systems, pp. 349-351.
Berling/Heidelberg: Springer (2010)
13. Bell, C., McNamara, D.S.: Integrating iSTART into a High School Curriculum.
Proceedings of the 29th Annual Meeting of the Cognitive Science Society, pp. 809-814.
Austin, TX: Cognitive Science Society (2007)
14. Jackson, G.T., Boonthum, C., McNamara, D.S.: iSTART-ME: Situating Extended
Learning within a Game-based Environment. In: H.C. Lane, A. Ogan, & V. Shute (eds.),
Proceedings of the Workshop on Intelligent Educational Games at the 14th Annual
Conference on Artificial Intelligence in Education, pp. 59-68. Brighton, UK: AIED (2009)
15. Jackson, G.T., Dempsey, K.B., McNamara, D.S. (2010). The evolution of an Automated
Reading Strategy Tutor: From Classroom to a Game-enhanced Automated System. In:
M.S. Khine & I.M. Saleh (eds.), New Science of Learning: Cognition, Computers and
Collaboration in Education, pp. 283-306. New York, NY: Springer (2010)