Wake up and smell the metrics!
Metrics Driven Development in
online games
AGC ‘08
Larry Mellon
Darius Kazemi
Data
mining is
pure
gold!
What
stops
people
from
reaching
it?
Cultural collision
is our Smaug!
Boxed goods
mentality
Online
service reality
Wrong
assumptions
lead to painful
decisions!
Larry Mellon
(Consultant)
Research era
• Alberta Research Council & Jade Simulations
– Distributed computing, 1982+
– Optimistic computing, 1000+ CPU virtual world
– Fault-tolerant cluster computing
• Synthetic Theatre of War: virtual worlds for training
– DARPA: 50,000+ entities in real-time virtual worlds
– ADS, ASTT, HLA & RTI 2.0, interest management
Wife era
EA (Maxis): The Sims Online, The Sims 2.0
•
•
•
•
Scalable server architecture
Automated testing to accelerate production and QA
Player, pipeline & performance metrics
Emergent Game Technologies (CTO)
•
•
Architect for scalable, flexible MMO platform
Darius Kazemi
Orbus Gameworks
[email protected]
Boxed goods
industry is
evolving online
and needs to
adapt to service
industry
requirements!
Why is it hard to get funding for something as
clearly useful as metrics?
Nobody wants to pay for it, because no one has
traditionally paid for it!
$$$$$$$$$$
Features
$$

QA
Metrics, CS, …
Diasporas kill off the home-rolled systems
Business Intelligence has driven the success of
many other industries for years!
Las Vegas Strip
Metrics help manage complexity
“When you can measure what you are
speaking about and can express it in
numbers, you know something about it.
But when you cannot measure it, when
you cannot express it in numbers, your
knowledge is of a meager and
unsatisfactory kind."
- Lord Kelvin Institution of Civil Engineers, 1883
“The general who wins the battle makes many
calculations in his temple before the battle is fought.
The general who loses makes but few calculations
beforehand.”
-- Sun Tzu
The people who led the scientific
revolution depended on metrics!
Pasteur: A science is as mature as its measurement tools.
Maxwell: To measure is to know.
“The three largest factors
that will influence gaming will
be […] and metrics
(measuring what players do
and responding to that)”
-- Will Wright
The Secret of The Sims", PC Magazine, 2002.
http://www.pcmag.com/article2/0,1759,482309,00.asp
– GIGO –
Avoid false
causality by
correlating
data!
Multiple views of data provides a deeper understanding and
fewer analysis errors
Player and game actions
Minute 1
1. AI: open door
2. AI: cook food
Screenshots
AI
data
Minute 2
1. Game: fire breaks out
Screenshots
Time
3 – Final road block:
Metrics are role-specific and hard to move
across boundaries
Player
The “3P's” model Performance
of game metrics
Process
Next Gen Games
 Increased
Complexity
 Increased
Complexity of
Analysis
Art from “Fun Meters for Games”,
17
Nicole Lazzaro and Larry Mellon
18
Next Gen Games
19
Tuning imbalances or exploits can throw your entire
economy out of kilter, but remember to triangulate!
Tuning imbalances or exploits can throw your entire
economy out of kilter, but remember to triangulate!
Know Your Players
Comparing groups is
another no-brainer…
Process metrics
• Find the leaks that are slowing you down or
costing you money!
• Another cultural problem
– Process = evil
– Tools != game feature
• Not ‘fun’ to build
• No ‘status’
– Thus, worst possible programmers are assigned to
most critical systems…
Fixing development leaks is like
adding free staff!
Process: Measurable targets & projected trends give
you actionable progress metrics, early enough to react
Target
Risk!
Any metric
(e.g. # clients)
Time
First sample
Now
Any Time
(e.g. Alpha)
Real data gives you faster focus,
less debate!
• Question for the audience:
• If you have 5 engineers in the room, how
many opinions do you have on what the team
needs to do next?
• Numbers cut the debate from “what is next?”
to “how do we fix the thing we found?”
• You QUICKLY go from finding the key data and
fixing the problem, not searching the entire data
space, just to get started!
Performance: engine metrics
(Dev & Ops)
• Debugging & optimizing large-scale, complex system
– With hackers, non-determinism & constant change
– Tick-tock!! Users (and VPs) are waiting…
• Most engineering metrics tools are for one person,
one machine, one debugger
– Need system-level view to deal with MMOs
• Studio head metric: only team-wide, red-ink email
– GET THIS METRICS SYSTEM IN THE LIVE SERVERS, BY
TOMORROW (tired of ‘black box debugging’ across 20
servers and 200 clients)
Lower dev & ops costs: FREE MONEY!
Profit…
New
Content
~ $10
per
customer
Regression
Customer
Support
Operations
Profit…
~ $10
per
customer
Regression
Customer
Support
Operations
Lower New Content Cost
Profit…
~ $10
per
customer
Lower New Content Cost
Lower Testing Cost
Customer
Support
Operations
Profit…
~ $10
per
customer
Lower New Content Cost
Lower Testing Cost
Happy Customers Don’t Call
Operations
Process (Dev & Ops): Lower recurring costs that are
worth the cost to lower
[makes you more money with the same number of players!]
Profit…
~ $10
per
customer
Lower New Content Cost
Lower Testing Cost
Happy Customers Don’t Call
Operations
Lower bandwidth & CPU
Process: Rapid Iteration is how you find fun!
Project Start
Launch
Implementation options
(both are hard)
1. Grassroots
movement & scale as
required
2. DEFINE metrics as a
fully funded feature
needed for success in
an online service!
1.a How to start your own Mini metrics project
Image by Flickr user RaeA; Creative Commons (BY-NC-SA)
1.b How to scale your own Mini metrics project
Image by Flickr user Dredrk; Creative Commons (BY-NDW)
Culture Shift option:
Treat metrics as a critical feature from day one!
Fund everything that helps both team and
customers, not just game play!
$$$$$$$$$$
Features
$$$$
QA
$$!!!
Metrics
Metrics are your first-pass tool : they quickly find you a
starting point that would take hours/days for you to
find by yourself
Find oddities automatically for you, then you decide
what action to take
Scaling the metrics system as data scales
Tip: make your
system selfmaintaining, and
critical to
programmers
and/or
management
(they’ll fix it the
moment it
breaks  )
Train ‘bots to
look for
patterns of
data when the
datasets get
even larger
Takeaways
• Grassroots or formal feature : either way, get started with metrics
across your entire game service
•
Find something that will help an obviously important question to be answered
with facts rather than guesses
• To get executive and funding support, translate your day-to-day
need for metrics into the metrics the executives understand to
justify the funding
•
•
You will either make more cash with this change!
Or you’ll save more cash with this change!
• Presentation and aggregation are key: use the computer to do the
hard parts
•
People should see high-level data, with easy search and correlate functionality,
and access to the raw data
The migration
online is a
Darwinian
moment for
our industry
• Boxed goods mentalities must shift to online
service realities, or die…
• Complexity: people need metrics to be able to do their jobs
• Player Retention: It’s not just about game features anymore!
Question:
How would you rather live your life?
Measure
Change
Measure
OR
Guess
Change
Hope
Slides are online at http://www.maggotranch.com/biblio.html
www.orbusgameworks.com/blog