Exploring the Effects of Gaze
Awareness on Multiplayer Gameplay
Joshua Newn
Microsoft Research Centre for
Social Natural User Interfaces
The University of Melbourne
Melbourne, Australia
[email protected]
Frank Vetere
Microsoft Research Centre for
Social Natural User Interfaces
The University of Melbourne
Melbourne, Australia
[email protected]
Eduardo Velloso
Microsoft Research Centre for
Social Natural User Interfaces
The University of Melbourne
Melbourne, Australia
[email protected]
Marcus Carter
Microsoft Research Centre for
Social Natural User Interfaces
The University of Melbourne
Melbourne, Australia
[email protected]
Abstract
During tabletop gameplay, players monitor each other’s
gaze throughout, providing a form of implicit nonverbal communication. A player can infer the intention and potential
strategies by estimating the gaze of another, especially in
co-located gameplay. This early work-in-progress paper details an exploratory study design that explores the effects
of gaze awareness on gameplay, in particular when the
gaze of one or more players is augmented over the game
and revealed to others. We believe that players may very
well change their strategies when gaze becomes ‘common
knowledge’, but can also be used as a form of deception.
We will explore these effects with the same game in two
settings: a screen-based digital version with players in separate rooms and a co-located tabletop board game version
augmented with a projector. This work evaluates the effects
of gaze awareness in both settings, providing new insights
towards the emerging research of EyePlay.
Author Keywords
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the Owner/Author.
Copyright is held by the owner/author(s).
CHI PLAY’16 Extended Abstracts, October 16-19, 2016, Austin, TX, USA
ACM 978-1-4503-4458-6/16/10.
http://dx.doi.org/10.1145/2968120.2987740
Eye tracking; Gaze tracking; Co-located play; Multiplayer;
Tabletop games; Board games; Video games; Cognitive
strategies; Gaze awareness; EyePlay
ACM Classification Keywords
H.5.m [Information interfaces and presentation (e.g., HCI)]:
Miscellaneous; K.8.0 [General]: Games
Introduction
Gaze is a very powerful nonverbal signal. It signals dominance, attraction, shared attention, filiation, intention, among
other social effects. It is also a very informative signal in
gameplay, as players monitor each other’s gaze during
gameplay [7]. This is evidenced by poker players’ common
practice of covering their eyes with sunglasses to hide any
“tells” [14]. It also plays an important part in turn-taking by
indicating when it is another player’s turn [11]. However, in
conventional social interactions, gaze is largely invisible.
This project aims at exploring what happens in the game
when we make this invisible signal visible.
Figure 1: Illustration of deception
when gaze is visible to both players
in a co-located multiplayer gaming
environment.
This paper seeks to understand the effects of gaze awareness on gameplay in two types of multiplayer games i.e.
tabletop board games and multiplayer screen-based digital
games. On tabletops, players are encouraged to interact
face-to-face with one another by engaging in collaboration, competition or cooperation depending on the game.
Speech and gestures are well-known modalities used for
communication during gameplay, typically during one’s turn
in a turn-based game. Gaze happens throughout the game
providing a form of visual evidence towards intention and
insights into potential strategies. Gaze also plays an important part used for communication cues for turn-taking
and can decrease by up to 25% when absent [11]. Tabletop
games in particular demand constant attention, requiring
regular thought processing and frequent changes to short
and long term strategies. Explicit gaze can provide insights
into these processes, changing the dynamics of gameplay.
On multiplayer digital games, the gaze of players is absent,
varying aspects of gameplay especially if the game was
originally designed for co-located gameplay. Players are
still able to guess based on previous actions but without
social cues, it may prove to be somewhat difficult. This dif-
ficulty is exacerbated when gameplay is purely visual (e.g.
poker). With the advancement of current eye tracking technology, the gaze of another player can be made explicit on
their respective screens, potentially bridging this gap. We
look towards EyePlay, an emerging field of research which
refers to the playful experiences that take input from the
eyes [8, 9]. Our work aims to extend this body of literature
by demonstrating that gaze can be used to different ways to
create new experiences in multiplayer environments.
Background
In games that employ social gaze mechanics, the adaptive
AI can change aspects of the game based on a player’s
gaze [12, 13]. If a computer can perform such actions, it is
possible for humans to adapt and make changes to strategies as well once gaze information is made visible. Infants
within the first 18 months of life develop mechanisms of
joint visual attention (looking where someone else is looking), a natural behaviour [2]. Gaze plays a major role in
social aspects of our lives. Researchers in both areas of
psychology and HCI have investigated the role of multiuser
gaze behaviour, patterns and awareness during communication (e.g. [10]). However, there is a lack of investigation
specifically on gaze behaviour in multiplayer games and
none on the effects of gaze explicit during gameplay in both
digital and co-located contexts. Gaze can be visualised
on surfaces in several ways such as by using scanpaths,
heatmaps dominantly used in usability studies that employ
eye tracking or by displaying the real-time gaze point [3].
When the gaze of all players is revealed, gaze becomes
‘common knowledge’, a term used in game theory with regards to logic that affects strategy assuming that all players
are rational [6]. For example, take two players, P1 and P2.
P2 looks at the gaze point of P1 and P1 now knows that P2
is looking at her gaze point. In this situation, multiple ratio-
nal scenarios may occur. One possible scenario is the act
of deception where P1 can shift her gaze to another part of
the surface and potentially confusing P2. This ‘I know that
you know that’ concept is illustrated in Figure 1. Such deception techniques go back to World War 2 spycraft [5], for
example, signalling left but turning right to deceit a shadowing vehicle. This touches on theories of ‘shared awareness’
from behavioural sciences [4].
Study Designs
Following our feasibility study, we designed two studies to
determine the effects of gaze in both traditional and digital
versions of the same board game. Results from a study that
compared both versions of the same game showed that
when the digital version is a direct adaptation of the traditional version, there is little subjective difference between
them [1]. We believe that by making gaze explicit on both
versions, the results will be different, and we will compare
these results accordingly. Therefore, we have chosen a
board game that is identical in both versions and designed
two studies in accordance, testing effects of the play setup
(remote x co-located) and type of gaze visualisation (realtime x averaged over time).
Figure 2: Demonstration of joint
visual attention analysed from our
feasibility study. Top: The subject
glances at the gaze direction of
another player. Middle: The subject
looks at the area that he/she thinks
the other player is looking at.
Bottom: The player’s gaze shifted
further and the subject’s gaze
follows accordingly.
One of the many insights we hope to gain from comparing
both versions is whether gaze awareness has any effect
at all especially in the co-located setup. The eyes of the
opponent are present in this setting, and players can infer
from the each other’s eyes in addition to the exposed gaze
on the interface. In contrast, players have to rely on actions
and the gaze visualisation (if present at all) in the remote
setup. We intend to explore concepts of gaze awareness
further, including drawing comparison between the both.
Board Game
We selected Ticket to Ride1 , a 2-5 player board game that
can be learned in under 15 minutes and have chosen to
use North American edition in which the physical and digital
versions are almost identical. In Ticket to Ride, each player
must build railroads connecting specific cities on the map
that has been randomly drawn by each player. In general,
the longer routes the better but there are a number of ways
to score or keep points. Players do not know which cities
their opponents are meant to connect and must do their
best to hide their goals, as other players might cut their railroad off. Therefore, players must balance building their own
railway, hiding their intentions, and trying to discover their
opponents’ goals to prevent them from winning.
Feasibility Study
To test the technical feasibility of the study, we conducted a
pilot exploratory study. We recorded and analysed the eyetracking data of a single player using the Tobii Pro Glasses
22 eye tracker during a hour long gameplay session with
four players. Our analysis has already identified several interesting behavioural patterns during gameplay. In many
instances, the player was attempting to infer the area of
the board in which another player is looking through joint
visual attention (see Figure 2). With regards to eye tracking on conventional desktops (e.g. PC), we know from the
EyePlay literature that gaze can be tracked effectively using consumer-based low-cost remote eye trackers. Another
finding is that the player’s gaze shifts quickly between the
cards in his hands and the board when he does not have a
strategy before hand or needs to change it during his turn.
The player will determine the best possible move without
taking too long, primarily so that other players are not kept
1
2
http://www.daysofwonder.com/tickettoride/en/
http://www.tobiipro.com/product-listing/tobii-pro-glasses-2/
waiting but it is possible that he/she is attempting not to give
away his potential strategies as other plays are watching.
Study 1: Remote Gameplay
In our first study, we examine the effects of gaze with two
players using desktop computers seated in separate rooms.
Each desktop will be fitted with a Tobii EyeX3 eye tracker
with data streaming to each other. The gaze visualisation
will be augmented onto the game using a custom built application that runs in the background. The following conditions will be tested:
Figure 3: Study 2 Configuration.
1. Gaze is not visible to either player (baseline).
2. Both players can see each other’s gaze, and both
players know that the other player can see it (symmetrical aware).
3. Both players can see each other’s gaze, but both believe that only they can see it (symmetrical unaware).
4. Only one player sees the other player’s gaze, but the
other player does not know that the other player can
see it (asymmetrical unaware).
Condition 1 serves as the baseline, reflecting the conventional gameplay with no gaze visualisation. In this condition
we will only record the gaze data. In Condition 2, players
have both the knowledge of the other player’s gaze and that
the other player also can this player’s gaze. This condition
is analogous to the co-located version, where the game
happens on a shared board. In Condition 3, both players
will see each other’s gaze point, but each will believe that
they are the only ones with this power. In this condition, we
are interested in whether players will be able to discover
that the other player can see their own gaze simply by their
gaze patterns and changes in game strategy. Finally, in
3
http://www.tobii.com/xperience/
Condition 4, we are interested in what happens when the
game is tipped in favour of a single player, who can see the
opponent’s gaze point.
Study 2: Co-located Gameplay
In this study, two players will play the board game version of
Ticket to Ride on a tabletop. We will equip both players with
Pupil Pro4 wearable eye trackers which streams gaze data
to a single computer connected to a top-down projector
(see Figure 3). Gaze information will be processed, and
its visualisations will be projected from above, augmenting
both the board and the game objects. We note that this is
a major technical challenge i.e. registering the play area to
the eye tracker and ensuring the projections corresponds
to the correctly gaze points of both players. As both players
share the same surface, we will only test conditions 1 and
2. By augmenting gaze in this setup, the dynamics would
be very different. For example, it is possible actions such as
deception may arise more evidently as one player is able to
tell whether the opponent is engaged in following their gaze
as both players share the same play area and view.
4
http://pupil-labs.com/pupil/
Procedure
For both studies, we will recruit experienced players from
the local board game club and we will form random pairs.
Both players are required to fill up a consent form and a demographics questionnaire. Once completed, each player
will perform the respective calibration procedure. The pair
will start by playing the game once they are calibrated with
the condition where gaze is not visible to create a baseline.
In the co-located setup study, as only one other condition
can be applied, the pair will be asked to play two rounds
under the second condition. In the remote setup study, we
are able to test all conditions, therefore requiring the pair to
play four rounds in total. The second and third conditions
will be tested sequentially. For the last condition, we will tip
the favour of the player with the lesser wins. We would like
to determine if condition has clear advantageous benefits
to the player. We estimate each game to take a maximum
for 30 minutes, but on average lesser, as the players will
be familiar with the game. After completing all the required
conditions in either setting, we will evaluate the views of the
pair. The type of gaze visualisation will be cycled through
for each pair. We will take observation notes during the sessions and which includes the time stamps for evaluation
purposes. In addition to the screen and eye tracker recordings, we will record the gameplay using a top-down camera
and various angles afforded by our observation lab.
Evaluation & Measures
The selective cued recall research methodology will be
used immediately after each pair has finished playing the
necessary rounds and conditions. They will be asked to report what they were thinking in that moment as we show
them clips from the study. In particular, sections of the
video that we have noted down during our observation. We
will attempt to find out the points in which players changed
their strategy or deceived their opponent through the gaze
information provided by our system. Participants will be
prompted for further discussion, i.e. which gaze visualisation they found useful and their views on the particular
condition given. This data will form the basis of our qualitative end of our evaluation. As for quantitative data, we
will measure the gaze against the actions of the users and
the what the user was looking at. This includes the player’s
own tracks, future tracks, and the tracks that they believe
the other player will build on. Moreover, we will analyse our
data collected in order to quantify the amount of time the
player has spent on determining their own strategy, their
opponent’s strategy and percentage of time that they spent
looking at these two types of places in their turn compared
to other players turn. This will be compared across the conditions employed. In the second study, we are able to measure beyond the board itself such as by determining the
percentage of time the other player spent looking at other
player’s hands, faces or eyes. At this point, we remain unsure which gaze visualisation technique works best to enable gaze awareness between the players. We will take this
into account through the performance of the players.
Conclusion
By making gaze explicit and aware to players in multiplayer
gaming environments, we have opened up numerous questions with regards to its effects upon gameplay. This includes the type of gaze visualisations employed, the awareness of players and the differences between co-located and
remote settings. This work hopes to provide new insights
and contribute towards the emerging research of EyePlay.
References
1. Alessandro Antonietti and Rosa Mellone. 2003. The
Difference Between Playing Games With and Without
the Computer: A Preliminary View. The Journal of
Psychology 137, 2 (2003), 133–144. DOI:
http://dx.doi.org/10.1080/00223980309600604
2. George Butterworth and Nicholas Jarrett. 1991. What
minds have in common is space: Spatial mechanisms
serving joint visual attention in infancy. British Journal
of Developmental Psychology 9, 1 (1991), 55–72.
3. Marcus Carter, Joshua Newn, Eduardo Velloso, and
Frank Vetere. 2015. Remote Gaze and Gesture
Tracking on the Microsoft Kinect: Investigating the Role
of Feedback. In Proceedings of OzCHI ’15. ACM, New
York, NY, USA, 167–176. DOI:
http://dx.doi.org/10.1145/2838739.2838778
4. Morris F. Friedell. 1969. On the structure of shared
awareness. Behavioral Science 14, 1 (1969), 28–39.
DOI:http://dx.doi.org/10.1002/bs.3830140105
5. Malcolm Gladwell. 2010. Pandora’s Briefcase. The New
Yorker (May 10, 2010).
6. Martin J Osborne and Ariel Rubinstein. 1994. A course
in game theory. MIT Press.
7. Edward Tse, Saul Greenberg, Chia Shen, and Clifton
Forlines. 2007. Multimodal Multiplayer Tabletop
Gaming. Comput. Entertain. 5, 2, Article 12 (2007).
DOI:http://dx.doi.org/10.1145/1279540.1279552
8. Jayson Turner, Eduardo Velloso, Hans Gellersen, and
Veronica Sundstedt. 2014. EyePlay: Applications for
Gaze in Games. In Proceedings of the First ACM
SIGCHI Annual Symposium on Computer-human
Interaction in Play (CHI PLAY ’14). ACM, New York, NY,
USA, 465–468. DOI:
http://dx.doi.org/10.1145/2658537.2659016
9. Eduardo Velloso and Marcus Carter. 2016. Emergence
of EyePlay: A Survey of Eye Interaction in Games. In
Proceedings of the 2016 ACM SIGCHI Annual
Symposium on Computer-Human Interaction in Play
(CHI Play’16). ACM, New York, NY, USA. DOI:
http://dx.doi.org/10.1145/2967934.2968084
10. Roel Vertegaal, Robert Slagter, Gerrit van der Veer,
and Anton Nijholt. 2001. Eye Gaze Patterns in
Conversations: There is More to Conversational Agents
Than Meets the Eyes. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems
(CHI ’01). ACM, New York, NY, USA, 301–308. DOI:
http://dx.doi.org/10.1145/365024.365119
11. Roel Vertegaal, Gerrit van der Veer, and Harro Vons.
2000. Effects of Gaze on Multiparty Mediated
Communication. In Proceedings of the Graphics
Interface 2000 Conference, May 15-17, 2000,
Montr’eal, Qu’ebec, Canada. 95–102.
12. Melodie Vidal, Remi Bismuth, Andreas Bulling, and
Hans Gellersen. 2015. The Royal Corgi: Exploring
Social Gaze Interaction for Immersive Gameplay. In
Proceedings of the 33rd Annual ACM Conference on
Human Factors in Computing Systems (CHI ’15). ACM,
New York, NY, USA, 115–124. DOI:
http://dx.doi.org/10.1145/2702123.2702163
13. Stefanie Wetzel, Katharina Spiel, and Sven Bertel.
2014. Dynamically Adapting an AI Game Engine Based
on Players’ Eye Movements and Strategies. In
Proceedings of the 2014 ACM SIGCHI Symposium on
Engineering Interactive Computing Systems (EICS
’14). ACM, New York, NY, USA, 3–12. DOI:
http://dx.doi.org/10.1145/2607023.2607029
14. Michael Wheeler. 2003. True or False? Lie Detection at
the Bargaining Table. Negotiation (2003).