Playtesting and Metrics: Getting the
Most out of Your Usability Testing
Jordan Lynn
Player Experience Researcher, Volition Inc.
Overview
●
Intro
●
Playtesting
Datalogging
● Benefits
● Pitfalls
Successes with Mixed Methods
Tips & Tricks
Outro
●
●
●
●
Brief Note on Other Systems
●
Ideal System
+
+
+
Integrated datalogging, player feedback, and
video recording
Custom user interface
Significantly reduced analysis time
Brief Note on Other Systems
Ideal System
- Requires large investment of time and money up front
Clarification of Terms
●
Playtesting
●
●
Bringing in new players to experience your
game and observe player behavior
Datalogging
●
Using back-end software solutions to measure
player and NPC behavior
Overview
●
●
●
●
●
●
Intro
Playtesting
Datalogging
● Benefits
● Pitfalls
Successes with Mixed Methods
Tips & Tricks
Outro
How Our Playtesting Works
Recruit outside players for a week

Participants attempt a natural playthrough

Participants stop to fill out a brief survey
occasionally
How Our Playtesting Works
●
We learn
●
●
●
●
What are players doing
Why players are doing things
How players feel
We identify areas of bad frustration
Playtest Setup
Overview
●
Intro
●
Playtesting
●
Datalogging
●
Benefits
● Pitfalls
Successes with Mixed Methods
Tips & Tricks
Outro
●
●
●
How Our Datalogging Works
Significant Event

Record status variables

Upload information to database
Example: Player Death
How Our Datalogging Works
●
Reliably record
●
●
●
●
Who is performing
What action
When and Where
Designer expectations: met or no?
How Our Datalogging Works
●
Purpose
●
During Production
●Inform
●
game design
Post Release
●Identify
successes/failures
●Inform game design for future titles
●Address glitches/bugs with patches
How Our Datalogging Works
●
Playtesting
●
●
Record 100%
Post Release
●
Record 10%, drop to 1%
Overview
●
Intro
●
Playtesting
Datalogging
●
●
Pitfalls
Successes with Mixed Methods
Tips & Tricks
Outro
●
●
●
●
Benefits
Benefits of Datalogging
●
Identify Combat Imbalance
Combat Balance
Spot the Difficulty Spike!
Combat Balance
Benefits of Datalogging
●
●
Identify Combat Imbalance
Identify Economy Imbalance
Economy Balance
Most “Respect” comes
from weapon purchases?
Economy Balance
Mystery
Solved.
Benefits of Datalogging
●
●
●
Identify Combat Imbalance
Identify Economy Imbalance
Identify areas of excessively high failure
Mystery Death Spike
Benefits of Datalogging
●
●
●
●
Identify
Identify
Identify
Identify
Combat Imbalance
Economy Imbalance
areas of excessively high failure
weapon imbalances
Evolution of
Weapon
Balance
• Test 1
Evolution of
Weapon
Balance
• Test 2
Evolution of
Weapon
Balance
• Test 3
Benefits of Datalogging
●
●
●
●
●
Identify Combat Imbalance
Identify Economy Imbalance
Identify areas of excessively high failure
Identify weapon imbalances
Improve workflow in other departments
SR: TT is Big
2.2 Million
deaths
illustrated
QA and Datalogging
●
Productivity
●
Tracking each team member
QA and Datalogging
●
Productivity
●
●
Tracking each team member
Execution
●
Check AI behaviors
QA and Datalogging
●
Productivity
●
●
Execution
●
●
Tracking each team member
Check AI behaviors
Planning
●
Verify that all features are being tested
Overview
●
Intro
●
Playtesting
Datalogging
● Benefits
●
●
●
●
●
Pitfalls
Successes with Mixed Methods
Tips & Tricks
Outro
Communication with Design
●
Instruction-All designers need to know:
●
●
●
System capabilities
Exactly what the data means
How NOT to use the data
Communication with Design
●
●
Instruction
Give designers direct access
Communication with Design
●
●
●
Instruction
Give designers direct access
Rapid reporting is key
Planning Around Constraints
●
Data analysis is time consuming
Planning Around Constraints
●
●
Data analysis is time consuming
Technical concerns of post-release data
●
●
●
●
Sampling rate
Massive amount of incoming data
Storage capacity
Analysis time
Planning Around Constraints
Data analysis is time consuming
● Technical concerns of post-release data
● Datalogging works best with a single
programmer
●
Planning Around Constraints
Data analysis is time consuming
● Technical concerns of post-release data
● Datalogging works best with a single
programmer
● Verify QA has adequate time to test
●
Relying on Player Feedback
Legend
Shape: Playtester
Color: Cause of death
●
Test 1
Relying on Player Feedback
Legend
Shape: Playtester
Color: Cause of death
●
Test 2
Relying on Player Feedback
Legend
Shape: Playtester
Color: Cause of death
●
Test 3
Relying on Player Feedback
Legend
Shape: Playtester
Color: Cause of death
●
Test 4
Relying on Player Feedback
Legend
Size/Color: # of deaths
●
Post-Release
Overview
●
Intro
●
Playtesting
Datalogging
● Benefits
● Pitfalls
●
●
Successes with Mixed Methods
●
Tips & Tricks
Outro
●
Resolving Ambiguous Feedback
●
Example: “This level is too hard.”
●
This is not actionable feedback.
Resolving Ambiguous Feedback
•Location 1: Bottleneck
•Location 2: Elevated
Rocket turret
Conflicting
Feedback
Real-Time Survey Modification
Real-Time
Survey
Modification
Real-Time
Survey
Modification
Outcome
Overview
●
Intro
●
●
Playtesting
Datalogging
● Benefits
● Pitfalls
Successes with Mixed Methods
●
Tips & Tricks
●
Outro
●
Validate Data Using Feedback
●
Datalogging can’t stand alone
Example: Death does not equal difficulty
in all cases.
●
Data Overload
Best Datalogging ROI Across Titles
●
Combat Balance
●
●
Player Death
NPC Death
Best Datalogging ROI Across Titles
●
Economy Balance
●
●
Resources Collected
Upgrades Purchased
Red Faction: Best Datalogging ROI
Red Faction: Best Datalogging ROI
●
Ammo pickup
●
Out of Ammo
Red Faction: Best Datalogging ROI
●
GPS Request
●
Combine with Player feedback to improve
level layout and lighting.
Saints Row: Best Datalogging ROI
Saints Row: Best Datalogging ROI
●
Mission Attempts
●
Activity Attempts
●
●
Failures
Repeated missions
Reporting
Design Leads
Systems Designers
Level Designers
Data
Producers
Audio
QA
Readable Data
●
Name each participant in database
●
●
Date and seat
Example: Test May 1st, player 1 named
05_01_P1
Readable Data
●
●
Name each participant in database
Translate asset names
●
“Infuego” > car_2dr_exoticsports04
Readable Data
●
●
●
Name each participant in database
Translate asset names
Visualization software
●
We use Tableau
Overview
●
Intro
●
●
Playtesting
Datalogging
● Benefits
● Pitfalls
Successes with Mixed Methods
Tips & Tricks
●
Outro
●
●
Questions, Suggestions, Lessons
[email protected]