Exploring Students’ Cognitive Process in Game-based
Learning Environment by Eye Tracking
Meng-Jung Tsai1 and Hung-Ta Pai2
Graduate Institute of Digital Learning and Education, National Taiwan University of Science and
Technology, Taipei, Taiwan
2
Graduate Institute of Communication Engineering, National Taipei University, New Taipei City, Taiwan
1
Abstract - Game-based learning has been proposed and
developed for years; however, its effectiveness is still open
with inconsistent findings. This study attempted to explore
students’ visual attention distributions in game-based
learning environment by using eye-tracking techniques. An
online game, Talking Island, designed to enhance elementary
students’ English ability was used for eye-tracking
experiments. Subjects were twenty 3rd graders in Taiwan.
During the experiment, subjects played the game for 20
minutes individually and their eye movements was tracked and
recorded by a MobileEye eye-tracker. After the experiment, a
vocabulary test was used to evaluate students’ learning.
Percentage of fixation duration and percentage of viewing
time were analyzed by T-tests and effect sizes between genders
and between different performance groups. Heat maps and
fixation sequences were further observed for each subject.
Results showed that overall students can focus on learningrelated elements in this game. Findings will be discussed in
detail in the conference.
Keywords: game-based learning, eye tracking, humancomputer interaction, user studies, cognitive process
1
Background
Researchers have been devoted to develop gamebased learning environments and design related contents for
years; however, the effectiveness of game-based learning is
still open with inconsistent findings. Most of the prior studies
evaluating digital game-based learning use self-reported
questionnaires to survey students’ acceptances of technology
or use log-files to analyze interactions between learners and
computer systems. Self-reported questionnaires are usually
conducted before or after the learning process; and log-files
can only record students’ responding behaviors during the
learning process. Neither survey nor log data can reveal
learners’ cognitive process during this specific learning
context. For example, how students pay attention to the digital
contents or mind-tools designed in game-based learning
environments is still unknown.
The eye tracking technique has been typically adopted
to examine human visual attention based on the eye-mind
assumption [1]. In general, eye fixation location reflects
attention and eye fixation duration reflects processing
difficulty and amount of attention (the longer the information
is fixated, the more complex it is or the deeper it is processed).
Specifically, fixation duration varies on types of information
(e.g. texts or graphics) and types of tasks (e.g. reading or
problem solving). Furthermore, fixation locations and
duration reflect the individuals’ reading strategies and prior
knowledge or experience [2]. Besides, scan path patterns
exhibit individuals' cognitive strategies utilized in goaloriented tasks [3]. The eye-tracking method has been
successfully applied in research fields including reading [4]
and information processing (for a detailed review, see [5]),
arithmetic problem solving [6], human-computer interactions
[7] and emergent literacy [8]. In sum, eye-tracking studies
thus far have provided some insights on how students pay
attention to read texts, view graphics or solve math problems.
However, little study has been conducted to explore how
students learn in a game-based learning environment,
especially the distributions and shifts of students’ attentions.
2
Purpose
With the rapid development of eye-tracking
techniques, researchers can observe more deeply about
learners’ cognitive process in digital learning environments,
such as the visual attention distributions on designed digital
contents. This poster will demonstrate a pilot study
attempting to explore students’ visual attention allocations in
a game-based learning environment by using eye-tracking
techniques. Research questions focus on how students pay
their attention to the interface design in a game-based
learning environment. For example, how many percentages of
time do students spend on viewing learning-related elements
in such a learning environment? Do boys and girls pay
different attentions to different elements designed in gamebased learning interfaces? Do successful learners and
unsuccessful learners show different attention distributions on
screen when they learn via a game-based learning
environment? A mixed method including eye-tracking
techniques is proposed and used in this study to answer the
above research questions.
3
Method
An online game, Talking Island, designed to enhance
elementary students’ English speaking abilities has been used
for conducting an eye-tracking experiment in this pilot study.
Participants of the experiment are twenty 3rd graders in
Taiwan. All of them have at least two-year English learning
experience in elementary schools. In this game, participants
need to talk to other online players or virtual players in
English in order to finish tasks and then gain scores and
energy to explore a virtual New York Island. Students can
freely explore anywhere in any scene of the game by clicking
a mouse. Also, a microphone is provided for students to
respond with particular targeted English vocabularies or
sentences and the system will promptly provide adapted
feedbacks for individuals. For the interface design, the game
provides four tools for learning supports: a map, a chat room,
useful tools and score information, which are statically shown
in four corners on screen. Sometimes, when a specific
vocabulary is popped out for students to practice
pronunciation, another window with the vocabulary will be
shown in the center of the screen. The more practices of
English vocabularies, the higher scores and more energy the
learners can gain for fighting others and successfully
exploring the virtual island.
All twenty subjects passed the eye-tracking calibrations
and participated in an eye-tracking experiment individually.
In the experiment, each participant was asked to learn English
through playing this game for 20 minutes. Wearing with the
ASL MobileEye eye-tracker just like a goggle on face,
participants were free to move their heads during the whole
experiment. With sampling rates of 60Hz for the eye-camera
and 30 Hz for the scene camera, MobileEye tracked and
recorded all gaze points allocated on screen (i.e. visual
attention allocations) by each subject while playing the game.
The process of experiment including all the interactions
between the participants and computers such as mouse paths
and speaking voices were videotaped for further observation.
The vocabularies shown for practicing speaking for
individual participant were monitored and recorded by
researchers. After the experiment, a corresponding
vocabulary test including pronunciation was used to evaluate
students’ learning retention through the game playing.
Students’ English scores of the last semester and prior
English learning experience were also collected before
experiments.
Regarding the eye-tracking data, several areas or windows on
screen were defined Areas of Interests (AOIs) by using
GazeTracker software. Each area was defined as either
related AOI or unrelated AOI representing learning-related
elements (such as vocabulary windows, maps, scores) or
learning-unrelated elements (such as chat rooms, fighting
tools). The two types of AOIs were then served as bases of
statistical analyses for eye-tracking data. In this study, eye-
Figure 1. The subject’s gaze point (the cross point of two red
lines) was allocated on the target sentences (the white
window in the center of screen)
tracking indices including percentage of total fixation
duration and percentage of total viewing time were used to
indicate participants’ visual attention distributions on screen.
Therefore, the two indicators were calculated for and
compared between related AOIs and unrelated AOIs. T-tests
with effect sizes calculations were used to examine possible
differences between boys and girls as well as between high
performance and low performance groups. Furthermore, a
heat map and a scan path output from GazeTracker will be
used to further analyze each participant’s visual attention
distribution and fixation sequence on the game-based
learning interface.
4
Preliminary Result
Currently, the study is still under data collection and
data analyses. The preliminary data shows that most of the
students can gaze on or pay attention to the learning-related
elements designed in the game-based learning environment.
For example, Figure 1 shows that the participant’s gaze point
was allocated on the target sentence for practicing English
speaking. The cross point of the two red lines indicates a gaze
location; and the white window in the center of the screen
shows a popped out English sentence for practicing (i.e., a
learning element designed in the game-based learning
environment). Detailed statistical results and findings will be
demonstrated and discussed in the conference poster
presentation.
5
References
[1] Just, M. A., & Carpenter, P. A., “A theory of reading:
From eye fixations to comprehension,” Psychological Review,
87, 329-354, 1980.
[2] Hyönä, J., Lorch, R. F., Jr., & Kaakinen, J. K.,
“Individual differences in reading to summarize expository
text: Evidence from eye fixation patterns,” Journal of
Educational Psychology, 94(1), 44-55, 2002.
[3] Gandini, D., Lemaire, P., & Dufau, S., “Older and
younger
adults’
strategies
in
approximate
quantification,” Acta Psychologica, 129(1), 175-189, 2008.
[4] Rayner, K., Chace, K. H., Slattery, T. J., & Ashby, J.,
“Eye movements as reflections of comprehension process in
reading,” Scientific Studies of Reading, 10(3), 241-255, 2006.
[5] Rayner, K., “Eye movements and information
processing: 20 years of research,” Psychological Bulletin,
124(3), 372-422, 1998
[6] Hegarty, M., Mayer, R. E., & Green, C.,
“Comprehension of arithmetic word problems: Evidence from
students’ eye fixations,” Journal of Educational Psychology,
84(1), 76-84, 1992
[7] Jacob, R. J., & Karn, S. K., “Eye tracking in humancomputer interaction and usability research: Ready to deliver
the promises,” In J. H. Radach & H. Deubel (Eds.), In the
mind's eye: Cognitive and applied aspects of eye movement
research (pp. 573–605). Amsterdam: Elsevier Science, 2003.
[8] Evans, M. A., & Saint-Aubin, J., “What children are
looking at during shared storybook reading: Evidence from
eye movement monitoring,” Psychological Science, 16(11),
913-920, 2005.