Data Cracker: Developing a Visual Game Analytic Tool for
Analyzing Online Gameplay
Ben Medler
Georgia Institute of Technology
Atlanta, GA
[email protected]
Michael John
Electronic Arts, Inc. (EA)
Redwood City, CA
[email protected]
ABSTRACT
Game analytics is a domain that focuses on the systems and
methods used to analyze game-related data. In this paper we
present how a visual game analytic tool can be developed to
analyze player gameplay behavior. Our tool, Data Cracker,
was built for monitoring gameplay in Dead Space 2, the
latest game in the Dead Space franchise. We use Data
Cracker as a case study to inform a larger discussion of
designing a visual game analytic tool while working with a
game team. Our design approach focuses on increasing the
data literacy of a game team. This means getting an entire
team interested and involved with game analytics. We
found that building our tool during the early game
development cycle, creating multiple early visual
prototypes and branding the tool to the Dead Space team
caused more team members to become interested in our
tool. Increasing interest in analytics is also a means, we
argue, for changing the common occurrence within the
game industry to disband teams after a game is released.
Instead, we promote the creation of “live” teams which stay
attached to a game long after it is release in order to
continue the analysis process. Additionally, we discuss the
barriers one might face when developing game analytic
tools, such as prejudice against analytics or the technical
issues involved when collecting large data sets. All of these
examples are presented as insights we gained while
coupling analytic tool design to game development.
Author Keywords
Game analytics, visual analytics, information visualization,
team communication, game design, player behavior.
ACM Classification Keywords
K8.0. General: Games; H5.m. Information interfaces and
presentation (e.g., HCI): Miscellaneous.
INTRODUCTION
Analytics is currently a hot topic within the game industry,
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise,
or republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee.
CHI 2011, May 7–12, 2011, Vancouver, BC, Canada.
Copyright 2011 ACM 978-1-4503-0267-8/11/05...$5.00.
Jeff Lane
Great Northern Way
Vancouver, BC, Canada
[email protected]
driven in part by the extensive use of web-style analytics by
the social gaming powerhouses [2, 20], but adoption of
analytics in mainstream games tends to be more cocktail
conversation than actual ongoing use. Game analytics is a
domain that focuses on the systems and methods used to
analyze game-related data (also referred to as game
telemetry) and therefore can be a large area to cover.
Analyzing a game’s financial records, development
progression and software metrics can all be considered
game analytics. Instead, we refer to the type of game
analytics which influences how games are designed by
analyzing players, an area of analysis that is very prominent
among industry professionals [17, 21, 23, 42] and academic
researchers, with a good overview in [27]. Monitoring
player behavior means logging in-game events, or metrics,
such as when a player begins a level or performs an action
like jumping while in the midst of gameplay [24]. From a
game design point of view, behavior data is useful for
determining how players are actually playing a game in
their natural environment, versus other evaluation methods
like self-reporting surveys or controlled play tests.
We have developed a visual game analytic tool, Data
Cracker, in order to explore how visual game analytic
practices based on player behavior analysis can augment the
game design process and seek to use the tool as a case study
to instigate a wider discussion of game analytics. Data
Cracker is a prototype system built at Electronic Arts (EA),
one of the world’s largest game publisher/developer, as part
of the company’s initiative to provide its game studios with
analytic tools. In the past many EA studios have built
analytic tools for analyzing player data but have typically
been focused on recording metrics for monitoring financial
prosperity. Only a few visual systems built at EA track
player behavior and with a few notable exceptions, prior to
development of the Data Cracker, these had largely been
side projects initiated by various EA employees who built
the systems outside of their normal duties. As a response,
the Data Cracker initiative is a formal attempt at EA to
innovate and create best practices for building game
analytic tools. However, development of Data Cracker was
more than an experiment to determine how to best develop
a game analytic tool, it was a chance to explore the
difficulties that can arise when inserting analytic tools into
a creative process such as game design.
Data Cracker was built by the authors while employed by
EA for an internal EA game studio Visceral Games and
deployed to monitor player behavior from the game Dead
Space 2 (DS2) [45], the sequel to the critically acclaimed
game Dead Space [44]. Our team worked directly with the
DS2 multiplayer team in order to help monitor online
multiplayer gameplay. This is the first time that a
multiplayer component has been offered in a Dead Space
game, which is one reason way Visceral’s DS2 team was
eager to use a tool for analyzing player gameplay behavior.
Through our interaction with the DS2 team and the
development of Data Cracker we gained many insights
related to working with an external game team in addition
to the best practices we discovered related to developing a
visual game analytic tool. This joint effort format between a
game development team and an analytic team will likely
continue at EA making the insights we gained valuable for
future analytic projects.
In this paper we present the insights gained from working
with the DS2 team and how the design of the visual game
analytic tool Data Cracker shifted as the game, and team,
shifted. We begin with a review of other game analytic
systems and relate them to the field of visual analytics
which revolves around the exploration of data through
visualization. In the past, EA teams would typically analyze
data using spread sheets, or hastily created web-based
visualizations. With Data Cracker we provide the DS2 team
with multiple visualizations that help them explore their
data from a different perspective and makes data analysis
much more accessible. Next, we describe DS2’s
multiplayer gameplay and present the final Data Cracker
architecture. Using our development cycle as a case study
informs the subsequent discussion of important guidelines
for implementing a visual game analytic tool as part of a
game team’s development process. This includes covering
the production methods we found necessary to build Data
Cracker along with the technological and human barriers
we faced when working with a team during their
development cycle. Finally, we critique our game analytic
insights gained from this project in order separate the
idiosyncrasies of our situation from game analytics at large.
This will help other game teams gauge how our insights
may or may not apply to their game or development
process.
DEFINING VISUAL GAME ANALYTICS
Perhaps the most predominant examples of visual game
analytic tools are the services available for games built on
the Flash platform. Mochibot [26], Nonoba [28] and
Playtomic [29] are three examples of visual analytic
services Flash developers can use to monitor player
gameplay behavior. In the past, earlier services like
Mochibot were used to determine the number of users
loading a particular game, which justified ad revenue for
game developers who placed ads in their games. These
analytic services have expanded now to include features
such as player achievements, micro-transactions systems
and player behavior tracking through monitoring gameplay
events. However, these systems are only available for
games built on the Flash platform and act as basic analysis
systems similar to other, offline, visual analytic tools like
Tableau [39] or Spotfire [37].
Shifting to the PC and console platforms, Zoeller [49]
presented an example of a developer-facing visual analytic
tool, SkyNet, used by the game developer Bioware. This
system focuses on the game development process by
monitoring developer behavior rather than players. Bug
tracking, software metrics and social networking features
are combined to create an online-portal where developers
can both stay in touch with one another and have consistent
access to their game’s development status. While many
features in SkyNet use spreadsheets to represent data, visual
callouts such as color-coding important information are
used, in addition to other visual graphs, to display
information such as the number of bugs filed and fixed.
Heat maps are also used to denote where crashes occur in a
game’s space, even giving developers the option to click on
the map in SkyNet and be transported to the location within
the game environment. SkyNet is an example of how visual
game analytics can be used to monitor other game-related
data, such as the developer behavior and software metrics.
Data Cracker, in contrast, is used to analyze the player’s
experience, both during development (e.g. player testing
and beta tests) and after the game is released.
Beyond a single game studio’s tools, user research groups
at larger game publishers also use game analytic systems
for analyzing player behavior. Microsoft’s TRUE system
[17] is one example. TRUE is a game analytic system built
for both user testing and beta testing games. When used for
user testing, the system can record player metrics, capture
their video output and prompt players with surveys to gauge
their attitude. Collecting both video recordings and player
attitudinal data help inform the behavior metrics which
could otherwise be misleading without proper context.
However, video and attitudinal data is only collected within
an onsite lab-based setting. When TRUE is used to capture
data from beta testers it essential performs in the same way
as Data Cracker, only logging player metric data.
Additionally, TRUE has means of visualizing the data that
is collected and has proved useful for game designers when
making decisions on how to alter a game’s mechanics. In
order to expand on systems like TRUE we seek to bolster
the uniqueness and life cycle of game analytic tools on top
of features like video recording and data visualization. As
we discuss in the following sections, branding tools for
specific game teams and promoting data literacy are needed
to further advance game analytics.
Stepping away from game analytics for a moment, two
other larger areas devoted to visualizing and analyze data
include visual analytics [40] and information visualization
[3, 6, 8, 41, 46]. These growing research areas explore the
advantages of visualizing data [46] which include
enhancing a person’s: comprehension of large datasets,
perception of emergent properties, ability to find problems
within datasets, and capability to form hypotheses. “Vision
is not only the fastest and most nuanced sensory portal too
the world, it is also the one most intimately connected with
cognition,” [6] which is why we feel that visual analytics is
an important area to pursue when providing game analytic
tools for developers.
Visual game analytics is also a way to provide greater
universal usability [36] and data literacy for members of a
game team. From interacting with the DS2 team we found
that if analytic tools can be created to provide greater
accessibility to more members of the game team those tools
become significantly more useful in advancing design
discussions amongst the team. Providing visualized
examples of gameplay data is less intimidating than
handing a game developer a spread sheet of numbers. While
there may be specific personal qualities that make someone
adept at becoming data analysts [6] there are certainly
examples of how “casual” information visualization can
increase a person’s literacy of data analysis and allow
further reflection [30] that can be beneficial for game
teams.
armored and only has a short-ranged, melee attack.
Necromorph players thus cannot switch between different
weapons while playing and instead are forced to choose one
of the four classes each time they are killed (before they
respawn and re-enter gameplay). Additionally, players gain
experience for performing actions during play, some only
available for either humans or necros, which lead to further
rewards. Normally, gathering data about these various
distinctions happens through play testing a limited
population, which can be time consuming and difficult to
analyze in a statistically significant way. However by
recording data telemetrically, data from hundreds of
thousands of games can be analyzed.
DS2 GAMEPLAY & DATA CRACKER’S ARCHITECTURE
In this section we describe the type of gameplay found in
Dead Space 2 and the architecture behind Data Cracker.
This information provides context to our discussion of
design insights and procedures for working with the DS2
team in the next section.
Dead Space 2 Gameplay
Dead Space 2 multiplayer gameplay is based in the thirdperson shooter genre and places two asymmetric teams
against each other in battle, the human security forces
verses an alien team known as the necromorphs (necros).
Matches between these two teams consist of two rounds
where one team plays as the humans first, the other team as
the necros, and these teams switch sides in the subsequent
round. Humans and necros are asymmetric teams because
each side has their own unique abilities and different goals
for winning. For example, the human team has a series of
objectives (holding a position, turning on a machine,
gathering components) that they must complete within a
given timespan, while the necro team attempts to stall the
human team until time expires. This makes balancing these
two teams a subtle and very challenging task for the game’s
designers because the diversity of abilities can make one
side more powerful than the other causing an unfair game.
These and other differences between the teams affected
what Data Cracker needed to monitor. The human security
team uses an assortment of projectile weapons (i.e. guns) as
their primary weapons and each team member has the
ability to heal one another with health packs. Humans can
take multiple weapons with them into each match which
makes monitoring which weapons they choose an important
factor. Moreover, players on the necro team must choose
from four different classes with fixed abilities and weapons.
For instance, the class known as the Pack is fast, lightly
Figure 1. The Data Cracker architecture combines both
server-side systems analyzing data from Dead Space 2 and a
client-side web interface for data visualization.
Recording Players
Player events from DS2 multiplayer matches are recorded
to monitor how each team is behaving and allows designers
to balance the human and necromorph teams. Recording
events in DS2 works by adding telemetric “hooks” or
functions into a game’s code which send event data to a
separate server location whenever an event in the game is
triggered. The basic data format that is sent by a hook in
Dead Space 2 includes: an anonymous unique user
identifier; region information; event identifiers such as an
event’s name and type; a timestamp of when the event
occurred both on the local machine and on the server; and,
finally, specific information about the event which are often
represented using key-value pairs (i.e. a variable name and
its value). In DS2 these key-value pairs hold information
related to:









Identifying when unique matches start and end.
When the human team completes objectives.
The position of players as they traverse the play area.
When players equip weapons or classes.
How much weapon damage players inflict on other
players.
When players are killed by or kill another player.
When players respawn after being killed.
Whether a team wins or loses a match.
How much experience each player gains from a playing
in a match.
After events are logged, the server-side portion of Data
Cracker begins organizing the collected player data.
Data Cracker Server-side Architecture
Data Cracker uses a client-server architecture (Figure 1),
processing event logs from DS2 on the server-side and
presenting a visual analytic tool on the client-side. The
server portion organizes and aggregates events together into
data formats required for the client-side analysis tool.
Beginning with raw text logs containing recorded player
events from DS2, each event is stored within a MySQL
table as a single element. The organized events are used to
create additional aggregated tables representing the data
and is similar to an online analytic processing (OLAP)
system [5]. OLAPs aggregate large datasets along a single
measurement, creating a separate list of multiple
dimensions to query the measurement against. In DS2, each
weapon type found in the game is an example of a
measurement and two dimensions that are created include
how long players held a weapon during a match and the
amount of damage a weapon caused during a match. Even
though each individual weapon event is stored in one table,
additional aggregated weapon tables are created as a preprocess to ease the burden of later analysis and improve
responsiveness of the tool.
The processes through which these aggregated tables are
created are called Extraction, Transformation, Loading
(ETL) processes. ETLs “are pieces of software responsible
for the extraction of data from several sources, their
cleansing, customization and insertion” into databases [43].
ETLs in Data Cracker create aggregated tables, often
representing averages and totals, that are separated by day.
We found that aggregating based on single days was
detailed enough for the DS2 team to gain insight from the
data (however this process can be duplicated using shorter
or longer time frames). The aggregated data produced by
the ETLs are stored in separate MySQL tables which are
directly referenced by the client-side website.
Figure 2. Screenshot of the Data Cracker client-side website.
Data Cracker Client-side Website
The client-side website (Figure 1 and 2) is an online visual
analytic tool designed using a user-centered approach. Our
team worked alongside the DS2 multiplayer team as they
developed their gameplay, regularly meeting to design and
test the tool (we go into further detail about the design of
the tool in the next section). Data Cracker’s client-side
website visualizes the aggregated data created by the server
ETLs using Protovis [31] and jQuery [14], both javascript
libraries that run on the client’s machine. Queries are served
by AJAX calls to PHP scripts which grabs data from the
aggregated tables in the MySQL database.
The basic interface and feature layout of the tool includes:
The Summary Graphs at the top of the page show
overview values about the collected player gameplay data.
This includes how many players have been tracked, how
many matches have been played, the winning percentages
of the human and necro teams and the number of matches
played on each map in the game across time. These
summary graphs change whenever the user changes the
time frame they are viewing using the timeline.
The Timeline is an interactive selection table that displays
the range of days which are currently being analyzed.
When users click and drag across a range of days every
graph in Data Cracker alters itself to display data from that
time frame. Once a date range has been selected it can also
be dragged to encompass other dates, giving users the
ability to always view a fixed number of days (such as a
week’s worth of data). Data Cracker also allows user to
mark events on the timeline, such as acknowledging when a
new patch was released for the game, which may affect
how a user should interpret data from that time period.
The Main Graph area displays the graphs built for
analyzing player gameplay data. These graphs display data
related to: unique/total users being monitored, kill/death
ratios, number of rounds played and won, experience points
gained by players, weapon statistics, and objective
completion rates. Following the Shniderman visualization
mantra of “overview first, zoom and filter, then details-ondemand,” [35] each data area, for example number of
rounds played, has an overview graph which display the
total values for the current date range selected. Users can
then navigate to more detailed graphs which allow them to
zoom and filter the data, for instance data can be separated
based on a specific map. Some graphs also offer details-ondemand through the ability to sort values or only displays
values as users interact with the graph. Finally, each graph
displays a title, its date range and, if necessary, its axes/key
values so that screenshots of graphs can be taken and added
to gameplay reports, which are often created by game
development teams.
GUIDELINES FOR DEVELOPING A VISUAL GAME
ANALYTIC TOOL
Working with the Dead Space 2 team brought to light many
important design factors about visual game analytic tool
development. In this section we list a number of insights we
gained from our development process, each one discussing
specific features that should appear within a game analytic
tool or how the development process should be structured.
Insights are broken into three categories: production,
functional and game team integration. Production insights
revolve around the goals and development procedures used
to build Data Cracker. These production insights are the
most novel in comparison to the other game analytic
systems covered. Functional insights highlight the actual
features and interaction capabilities of the tool while game
team integration insights focus on our interactions with the
DS2 game team. These insights are perhaps less novel, and
more apparent to seasoned analytic tool developers, but are
listed in order to reaffirm their necessity while developing a
game analytic system.
Production Insights
Create analytic tools in parallel with game development
Commonly, analytic tools are created after a dataset, or the
procedure for creating that dataset, is finalized. Building a
tool for investigating airline ticket ordering [22], housing
prices [48], document analysis [16], or network structures
[7] can rely on the datasets being produced in a standard
manner (e.g. airline ticket data will not suddenly change
formats). However, in these investigative cases analytics
tool are used to gain insight from data in order to make
decisions which will not affect the data’s content [16].
Someone analyzing documents for security threats will not
alter the contents of those documents. Metric analysis of
player behavior in games, in contrast, can be used as a
powerful extension of the traditional design process of
iteration and refinement, altering the very nature of the data
players can produce. For example, the DS2 team
continually added new telemetric hooks and altered
mechanics like the types of weapons available in the game,
altering the data produced along with those gameplay
changes. The consideration and specification of the
telemetric data format became a positive feedback loop for
the game’s design team during this iterative process.
Having access to gameplay data forced them to quantify
their experiential expectations of each gameplay alteration.
Developers therefore can benefit from analytic tools not
only after a game is released, where the tools may benefit
the creation of patches to make gameplay alterations, but as
soon as a team begins testing their gameplay, which
generally happens very early in the development cycle. For
example, during the DS2 beta test the team was able to use
our tool to find an exploit players were using involving a
particular weapon called the javelin gun. Using Data
Cracker to review the weapons usage percentage and
damage dealt the team was able to tweak the weapons
power to balance gameplay. At the same time players were
also complaining online that the necro team was too weak
in multiplayer matches. The DS2 team was able to verify
these complaints using Data Cracker and subsequently
knew how much to tune the necro class’ health and damage
by analyzing the values visualized by the tool.
Produce early visual prototypes
Just like with game development prototyping [32],
presenting the graphs and analytic features of a tool early is
vital to help communication between the analytic team and
the game team. Each graph produced for Data Cracker was
first prototyped and discussed in our weekly meetings with
our DS2 team contacts. The entire process, from events
being triggered to visualizing the data, was discussed at our
meeting as we prototyped each graph through the
development process. Giving the DS2 team high fidelity
prototypes that functioned and displayed real data helped
the team understand how the tool would be beneficial to
them once completed. We also used prototypes to win
arguments about adding certain features or what data should
be collected. For example, after presenting our graphs for
analyzing player experience points gained during matches
we were able to show that the points needed to be separated
by team type in order to show inequities between the human
and necro teams, even though experience points are only
awarded after matches where a team had played as both
sides. Prototypes brought out these types of situations in our
discussions with the DS2 team: thinking of different ways
of interpreting the data and what might be missing from our
analysis.
Build for a Broad Audience
Building a game analytic tool during a game development
process means the data being collected could affect a large
number of members on a game’s team. Designers and
producers may have to change the game’s mechanics which
affects programmers who have to code the mechanics and
artists who have to create content related to the mechanics.
Giving the power to analyze game data to each of these
groups during the development process can help teams
communicate changes more effectively.
Early game analytic tools should thus be designed with the
expectation that a broad audience will use them, not just a
set of “superusers” who intuitively understand the game’s
underlying structure. This means providing tiers of analysis,
or ways that users of different levels can access the data.
With Data Cracker we achieve this by providing graphs that
visualize the same set of data but offer different levels of
interaction. For example, users can view one graph that
shows the total number of rounds played, a second graph
that breaks down the total rounds played by map and the
win percentage for each team, and a third set of graphs for
each map breaking down the percentage of human teams
that completed that map’s series of objectives. Game
producers may only wish to view the total number of
rounds played while a game designer may want to know
how often teams are completing a certain objective on a
specific map.
Performing usability tests on tools are also invaluable in
proving and iterating the tool’s broad accessibility. We
conducted tests both internally with multiple members of
the DS2 team and with outside EA employees to gauge how
quickly they could understand the interface and features. As
a result we added features such as help icons that explain
graphs to users and help indoctrinate team members who
had never used our tool before. We also focused on using
simple visualizations methods like bar graphs and line
charts which can decrease data interpretation errors [11].
Certainly, as tool development continues, features that dive
deeper into the data can be added but continuing to support
the earlier broader features will keep a team focused on the
benefits of using analytics.
Brand a tool to a particular team
Data Cracker could have been visually designed in a
generic fashion so any game’s data could be connected to
the tool and analyzed. Other data visualization software
takes this approach [37, 39] and offers users a common
interface no matter what data is being analyzed. However,
for Data Cracker we took a different approach and branded
the tool specifically for Dead Space 2.
The main reason for designing the tool in such a way was to
make sure that the DS2 team knew that this was their
personal tool. Bateman et al. [1] found that users find charts
and graphs that are visually embellished (with
“unnecessary” visual elements) to be more enjoyable and
help with retention. While most of the graphs in Data
Cracker are not embellished with extreme graphics, certain
DS2 symbols were added to mark Data Cracker as a DS2
tool. First, the term “Data Cracker” was actually decided
upon because there is a term called ‘Planet Cracking’ that
plays a major role in the lore of the Dead Space franchise.
Second, the title graphic for Data Cracker also uses artwork
from DS2 depicting the main character and other DS2
artwork is used throughout the tool. Third, the color
scheme used in the tool is the same color scheme used in
DS2. For instance, a certain shade of blue is often
associated with ‘safety’ in the game and is used to depict
the human team’s data, while red symbolizes ‘distress’ and
used to show data associated with the necro team. Finally,
we also made use of DS2 artwork when ever creating
material describing or promoting the tool. Each of these
examples works to create a common identity for Data
Cracker that links it directly with the DS2 team.
Encourage “live” teams
As the final insight into producing a game analytic tool, it is
important to encourage the game team to create “live”
teams which will incorporate analytic tools into the
development process. A live team is a portion of a game
development team that supports a game after it is released.
Often times supporting a game after release refers to
customer service representatives or development team that
create patches or extra downloadable content (DLC).
Alternatively, when referring to live teams in relation to
game analytics it means a team that continues to analyze the
game after release and develop further tools. Their duties
should go beyond finding bugs to fix. Live teams should
monitor gameplay for months after the game is released
keeping track of the ebb and flow of player experiences,
altering tools as required. Analysis reports should be
created detailing how players are reacting to the game’s
mechanics and should be shared with other teams creating
DLC or working on games within the same genre. Game
teams are often dissolved after a game is released which is
one leading cause for why game analytics has been slow to
take off, there is no one around after the game is released to
analyze player reactions. By promoting live teams which
stay active after a game comes out it ensures that analytic
tools are properly used and have a greater impact on future
games produced.
Functional Insights
Aggregate data for quick analysis
Query time is often a big problem when dealing with large
datasets where searching through each element can be time
consuming [5]. DS2 could have upwards of hundreds of
thousands of players after the game is released, each of
which is reporting their behavior events back to EA’s
servers. The common practices of OLAP [5] and ETL
processing [43] were implemented in Data Cracker in order
to get around this problem. For example, one set of graphs
shows how many matches were played on the different
multiplayer maps available. The calculations to add the
total matches played on every map, each day would have
been extremely costly to perform multiple times. As a
result, an ETL was produced that added the total matches
played on every map, each day and placed in a separate
table. Data from thousands of events are combined into a
single element making querying and visualization much
faster while still leaving those events intact, within another
table, for other systems to access or data mine.
Use time-based queries
Time emerged as a key dimension for almost all the
gameplay analytics we came across for DS2. In most cases,
time provides the key added dimension to give context to
gameplay data which has been the case for other game
analytic systems [17]. For the Data Cracker tool, the
importance of time as a data dimension is why the timeline
is the highest-order interactive element of the tool (affecting
all graphs in the tool), and why it received the most
attention in terms of usability. Our initial designs included
two text boxes that marked the start and end dates of the
time frame shown on each individual graph. This eventually
changed to a single set of boxes that governed the time
frame for all graphs in the tool because the DS2 team
wanted to compare different graphs within a single time
period without having to set each graphs time frame
individually. After more iterations, the time frame was
shown on an interactive timeline making it easier to select a
date range, move the range to different dates, as well as
mark the timeline with key events that could affect the
interpretation of the data (e.g. when downloadable content,
DLC, is added to the game and may affect player behavior).
Debug the game and data hooks with the tool
The creation of telemetric hooks to track player behavior is
a software process requiring the use of an API and as such
is as prone to error as any other part of the software.
Insuring that event hooks accurately represent player
behavior is a key part of the analytic system development
process [15]. Having game analytic tools functioning early
within the development cycle allows another avenue to
debug both the metric system and the game. At one point
during Data Cracker’s development, for instance, we started
receiving additional weapon related events from players
who had already completed a match, events that should not
have been occurring. When we reported this to the DS2
engineers it was found that the game was creating multiple
copies of each player and each was sending telemetry data.
The engineers were quickly able to rectify the mistake
thanks to our tool. If that data had been allowed to persist
our aggregated calculations would have been skewed,
affecting how the data was to be interpreted.
Create functionality to deal with legal Issues
When game companies collect player data they have to
follow different rules depending on the geographic location
of the player. Compliance with these rules is extremely
important in the current climate where online privacy is
heatedly being debated [4, 19]. EA, for instance, cannot
collect data from players in over 100 different countries. In
North American countries like the U.S., Canada and
Mexico users must agree to a Terms of Service document in
order for the company to track data. Other countries have
an opt-out policy where data is collected by default but can
be turned off by the player. Procedures within an analytic
system must be in place to handle these different
geographic categories.
Another legal issue surrounding recording player data is
keeping the player’s identity safe. An event tracking system
has to know which unique user an event is coming from in
order to maintain an accurate depiction of the events that
took place in a game. However this must be stored in such a
fashion that the identifier (in our case, an integer) does not
identify the player personally. Data Cracker, working in
concert with EA’s centralized online infrastructure,
maintains compliance with both the geographic and identity
related restrictions.
Game Team Integration Insights
Meet with an interdisciplinary team
Interdisciplinary teams are a necessity within the game
industry as much as they are within other design and media
professions [18]. Game analytics too is an interdisciplinary
practice and each game development discipline use
analytics for different purposes [25]. While developing
Data Cracker we regularly met with a team consisting of the
lead programmer, producer and designer on the DS2
multiplayer team. Each was invested in the tool differently.
Our main programmer adding the telemetric hooks into the
game needed to know what data we wanted to collect and
voiced his own opinions on what data was possible to
collect. The lead producer and designer were more
interested in how the tool was going to help them balance
the game’s mechanics. Meeting with an interdisciplinary
team meant we were both able to make relevant design
choices that supported multiple members while also
promoting the transfer of game analytic knowledge between
the members themselves. This helped bypass common
interdisciplinary team communication barriers [9] by
allowing our DS2 team members to understand how the
tool functioned and everyone’s analytic needs.
Prepare for analytic prejudices
Collecting player data is certainly not a new concept and
EA already had an entire infrastructure built for monitoring
player events. One persistent problem, however, was many
game teams incorporated data collection but never analyzed
the data. Understandably, these meant some Dead Space 2
team members were skeptical about our promise to provide
a tool that would make use of player data, because they had
seen how ineffective previous efforts had been. This type of
prejudice against a new procedure or technology is common
in other research. Herbsleb et al., for example, had similar
experience while adding an instant messaging technology to
a distributed team [12]. In our case, this prejudice made
development difficult because we relied on the team to
provide us with the necessary gameplay information and
telemetric hooks that sent a player’s data to our server. In
order to get around this opposition we create example
scenarios of how each data point would be used and how
they would inform the team. For example, showing how
increased fidelity of the experience point data would offer a
better picture of which team, humans or necros, were
achieving certain experience point events more often.
Anticipate the effects of game design changes
Development of Data Cracker was happening at the same
time as the DS2 multiplayer gameplay was being
developed: if the gameplay changed, the analytics had to
change. Weekly meetings with our DS2 contacts were
essential and kept us informed about changes to the
gameplay. Most of these changes were cosmetic (e.g., maps
or weapons being added or taken out) and only affected
how certain data points were labeled. However, at one point
the DS2 team limited human players to only two weapons
and were forced to choose one specific weapon. This
radically altered how often players switched weapons and
the variety of weapons they had to choose from during play.
Analyzing weapon data had to change as a result and while
the graphs didn’t change visually the interpretations of the
graphs changed. Tracking those interpretation changes was
vital for keeping the team updated about our progress and,
in the future, would help anyone attempting to analyze DS2
data with our tool as well.
Update the team regularly
While staying connected to the DS2 multiplayer team
helped us keep track of gameplay changes it was also
important to stay connected with the DS2 team as a whole.
Simple ways of doing this was to talk with other DS2 team
members whenever possible and to participate on internal
email lists, generally acting like a fellow team member. The
other major way we connected with the DS2 team at large
was distributing a weekly data analysis report that covered
the progress of Data Cracker and interesting information we
gleaned from analyzing DS2 player data. Like Data
Cracker, these weekly reports were DS2 branded, often
incorporating artwork from the game, and were designed to
look handmade to distinguish them from standard analysis
reports that game teams receive from marketing or
management departments. Each issue had a certain theme
based on an important feature added to Data Cracker or a
specific anomaly we found in the data that week. One
weekly issue focused on the Fourth of July holiday
weekend, for example, where no data was logged from
DS2, meaning everyone successfully stayed away from the
office. We often received comments that these reports kept
people interested in the tool even if they were not affiliated
with the DS2 multiplayer team or DS2 in general.
DISSCUSSION
Data Cracker has been in service since August 2010. There
are current plans to continue iterating on the design of the
tool and to provide each team within Visceral Games with
their own version. The tool has already proved useful by
helping find bugs in DS2’s multiplayer environment,
allowing analysis of player tests and through our efforts in
promoting the tool has caused other EA teams to begin
designing their own analytic systems. The DS2 single
player team has also used the Data Cracker’s systems and
web services to create complex analytics for their in-house
play tests, over and above our work. While Data Cracker
has been a useful tool for the DS2 team, further critiques
and evaluations are required to understand how game
analytics combines with the game development process.
Our insights reflect the design properties we focused on
while building Data Cracker: accessibility, branding and
longevity. An entire game team should have access to
analytic tools throughout the game development process.
Systems like TRUE seem to be particular to only user
research teams [17]. We instead argue each game team
must have basic data literacy skills and be able to use
analytic tools as the amount of data games collect and use
increases [24]. One way to raise data literacy is through
branding tools for specific teams. The Skynet system [49] is
an example that demonstrates how a tool can create a
community around game analytics. With Data Cracker, we
sought to bring a semblance of that community feature by
branding our tool to the DS2 team which boosted the team’s
interest in the tool. Finally, analytic tools should be built to
work across the entire development process. Analytics
should represent a reliable means of testing a game and live
teams can be created to handle the analysis of data after a
game is released. A regular occurrence in the game industry
is to dissolve a team after completing a game. However, the
new industry emphasis on supporting games after their
release (with DLC, patches and online player dossier
systems [25]) demands live teams as necessary for postrelease content development.
There are also limitations to some of the insights we
presented. First, building a tool for broad audiences and
aggregating data would not be ideal for developing data
mining analytic tools. Data mining player events would
require different methods [47] and would likely be built for
users with special analytic or statistics training [6]. Second,
creating branded systems and establishing “live” teams may
create significant cost overhead that game studios are not
prepared to invest. While we argue that game analytics are
vital, worth the initial investment, others question the overzealous use of metric-driven design [10, 13].
In contrast to the limitations of our insights, there are a
number of avenues where our insights can be expanded
upon in the future. Data Cracker was built to promote data
literacy among development teams and as their literacy
rises developers will be prepared for more robust analytic
systems. Adding in data mining functionality for more
automatic analysis, such as player modeling [33], could
increase the complexity of the available analysis that a tool
provides to developers. Other visual representations can be
added as well. For instance, player events happen within a
hierarchy (organized by map, level, play session, etc.) and
graphs such as sunbursts [38] or treemaps [34] work well
for displaying hierarchical data. Finally, there has been an
increase in recent years in the number of game analytic
systems that are provided to players instead of only game
developers [25]. Future game analytic systems may
combine developer-facing and player-facing systems with
each audience given different analytical functionalities.
CONCLUSION
Data Cracker is a tool for monitoring player behavior by
tracking and organizing in-game player events. The tool
was built using a client-server model where ETLs are used
to aggregate data on the server-side and a client-side game
analytic website visualizes the collected data with a series
of interactive graphs. Similar systems have been built
before, such as the TRUE system [17], but unlike other
game analytic tools our design approach for building Data
Cracker was to increase the Dead Space 2 team’s data
literacy. As digital games increasingly utilize player data
for gameplay and as part of their development process it is
not enough to design for small subsets of “superusers” who
understand the importance of data analysis. Therefore we
set out to build an analytic tool for monitoring player
behavior that is accessible to an entire game team.
The insights we gain from our design and development
process were split into three separate categories:
production, functionality and game team integration.
Through our production process we found that building our
tool in parallel with the game development cycle and
having early visualization prototypes familiarized the DS2
team with Data Cracker. We used tiers of analysis to lay out
visualizations offering different levels of detail so team
members were not inundated by too much data at once.
Branding the tool to Dead Space 2 also helped identify the
tool as part of the DS2 team and created further interest. For
Data Cracker’s functionally we built a system that
aggregated data across different dimensions including time,
in-game maps and gaming platform. This meant the tool
was quick to respond to queries further lowering the barrier
for team members to access their data. The tool had to also
follow a number of legal requirements since a broader
audience would have access to potentially sensitive data.
Finally, we had to maintain communications through our
interactions with the game team. Initially we were met with
prejudice against analytic tools and had to watch out for
unexpected changes made to the game that affected our
tool’s functionality. We were able to minimize these
problems by continually meeting with an interdisciplinary
team working on DS2 to present functional prototypes that
showed the effectiveness of the tool and determine if any
game changes would affect the design.
The insights into designing these tools and working with
game teams which we present in this paper are only the
beginning to what is capable within the domain of game
analytics. As a domain that is still growing game analytics
shows great potential when used to augment the game
design process. It is now possible for game designers to
support their design intuitions with quantitative
measurments, allowing for quicker iterations during the
development cycle. Analytic tools can monitor millions of
players after a game is released giving developers data they
never had the capabilities of accessing. Developers need to
continue to discuss their analytic processes and pursue
creating permanent “live” teams within their company to
work as tool developers and gameplay analysts.
ACKNOWLEDGMENTS
The authors would like to thank Scott Probst, Simon
Cooper and the entire Dead Space 2 multiplayer team for
their support of this work.
REFERENCES
1. Bateman, S., Mandryk, R. L., Gutwin, C., Genest, A.,
McDine, D. and Brooks, C. Useful Junk? The Effects of
Visual Embellishment on Comprehension and
Memorability of Charts. In Proc. CHI 2010. (2010).
2. Caoili, E. Zynga Chooses Tableau For Data
Visualization, Analysis. Gamasutra. (2010)
http://www.gamasutra.com/view/news/28408/.
3. Card, S., Mackinlay, J. and Shneiderman B. Readings in
Information Visualization: Using Visualization to Think.
Morgan Kaufmann (1999).
4. Carvin, A. Debate Continues Around Facebook Privacy
Changes. NPR (2010).
http://www.npr.org/blogs/alltechconsidered/2010/04/29/
126394012/npr-listeners-react-to-facebook-privacychanges
5. Chaudhuri, S. and Dayal, U. An overview of data
warehousing and OLAP technology. ACM SIGMOD
Record, 26, 1 (1997), 65-74.
6. Few, S. Now You See It: Simple Visualization
Techniques for Quantitative Analysis. Analytics Press
(2009).
7. Freire, M., Plaisant, C., Shneiderman, B. and Golbeck,
J. ManyNets: An Interface for Multiple Network
Analysis and Visualization. In Proc. CHI 2010. (2010).
8. Fry, B. Visualizing data. O'Reilly Media (2008).
9. Haythornthwaite, C. Knowledge Flow in
Interdisciplinary Teams. In Proc. System Sciences 2005,
IEEE (2005).
10. Hecker, C. Achievements Considered Harmful?. GDC
2010. (2010).
11. Heer, J. and Bostock, M. Crowdsourcing graphical
perception: using mechanical turk to assess visualization
design. In Proc. CHI 2010. (2010).
12. Herbsleb, J., Atkins, D., Boyer, D., Handel, M., and
Finholt, T. Introducing instant messaging and chat in the
workplace. In Proc. CHI 2002, ACM Press (2002), 171178.
13. Jamison, P. FarmVillains. SF Weekly (2010).
http://www.sfweekly.com/2010-0908/news/farmvillains/.
14. jQuery. http://jquery.com/.
15. Kaner, C. and Bond, W. Software Engineering Metrics:
What Do They Measure and How Do We Know?
Software Metrics Symposium, IEEE (2004).
16. Kang, Y., Gorg, C. and Stasko, J. Evaluating Visual
Analytics Systems for Investigative Analysis: Deriving
Design Principles from a Case Study. In Proc. VAST
2009. (2009).
17. Kim, J., Gunn, D., Schuh, E., Phillips, B., Pagulayan, R.
and Wixon, D. Tracking real-time user experience
(TRUE): a comprehensive instrumentation solution for
complex systems. In Proc. CHI 2008, ACM Press
(2008), 443-452.
18. Kim, S. Interdisciplinary Cooperation. In Laurel B.
(Ed.) The Art of Human-Computer Interface Design (pp.
34-44). Addison-Wesley, Reading, MA, (1990).
19. Kincaid, J. This Is The Second Time A Google Engineer
Has Been Fired For Accessing User Data. Techcrunch
(2010). http://techcrunch.com/2010/09/14/googleengineer-fired-security/.
20. Kontagent. http://www.kontagent.com/.
21. Lazzaro, Nicole (2007). The 4 Most Important Emotions
of Game Design. GDC 2007. (2007).
22. Liu, Z., Stasko, J. and Sullivan, T. SellTrend: InterAttribute Visual Analysis of Temporal Transaction
Data.In Proc.Infovis 2009. (2009).
23. Ludwig, J. Flogging: Data collection on the high seas.
Austin GDC 2007. (2007).
24. Medler, B. Generations of Game Analytics,
Achievements and High Scores. Eludamos – Journal for
Computer Game Culture, 3, 2 (2009), 177-194.
25. Medler, B. and Magerko, B. Analytics of Play: Using
Information Visualization and Gameplay Practices for
Visualizing Video Game Data. Parsons Journal for
Information Mapping, 3, 1 (2011). (in press).
26. Mochibot. http://www.mochibot.com/.
27. Nacke, L., Drachen, A., Kuikkaniemi, K., Niesenhaus,
J., Korhonen, H., Hoogen, W., Poels, K., IJsselsteijn,
W., and Kort, Y. Playability and Player Experience
Research. In Proc. DiGRA 2009. (2009).
28. Nonoba. http://nonoba.com/.
29. Playtomic. http://playtomic.com/.
30. Pousman, Z., Stasko, J., and Mateas, M. Casual
Information Visualization: Depictions of Data in
Everyday Life. IEEE Transactions on Visualization and
Computer Graphics, 13, 6 (2007), 1145-1152.
31. Protovis. http://vis.stanford.edu/protovis/.
32. Rollings, A. and Morris, D. Game Architecture and
Design: A New Edition. New Riders (2004).
33. Sharma, M., Ontañón, S., Mehta, M. and Ram, A.
Drama Management and Player Modeling for Interactive
Fiction Games. Computational Intelligence, 26, 2
(2010), 183-211.
34. Shneiderman, B. Tree visualization with tree-maps: 2-d
space-filling approach. ACM Transactions on Graphics,
11, 1 (1992), 92-99.
35. Shneiderman, B. The eyes have it: A task by data type
taxonomy for information visualizations. IEEE Visual
Languages, (1996), 336– 343.
36. Shneiderman, B. Universal usability. Communications
of the ACM, 43, 5 (2000), 84-91.
37. Spotfire Professional. http://spotfire.tibco.com/
38. Stasko, J. and Zhang, E. Focus plus context display and
navigation techniques for enhancing radial, space-filling
hierarchy visualizations. In IEEE InfoVis, (2000), 5765.
39. Tableau Desktop. http://www.tableausoftware.com/
40. Thomas, J. and Cook, K. Illuminating the Path: The
Research and Development Agenda for Visual Analytics.
IEEE Computer Society (2005).
41. Tufte, E. The Visual Display of Quantitative
Information. Graphics Press (1983).
42. Valve Corporation. Steam & Game Stats.
http://www.steampowered.com/v/index.php?area=stats
43. Vassiliadis, P., Simitsis, A. and Skiadopoulos, S.
Conceptual modeling for ETL processes. In Proc. 5th
ACM Workshop on Data Warehousing and OLAP.
(2002).
44. Visceral Games. Dead Space. [PC], USA: Electronic
Arts. (2008).
45. Visceral Games. Dead Space 2. [Xbox360], USA:
Electronic Arts. (2011).
46. Ware, C. Information Visualization: Perception for
Design. Morgan Kaufmann (2000).
47. Weber, B. and Mateas, M. A Data Mining Approach to
Strategy Prediction. IEEE CIG Symposium, (2009).
48. Williamson, C. and Schneiderman, B. The Dynamic
HomeFinder: Evaluating Dynamic Queries in a Realestate Information Exploration System. In Proc. of R&D
in Information Retrieval. (1992). 338-346.
49. Zoeller, G. Development Telemetry in Video Games
Projects. GDC 2010. (2010).