Automated Testing in MMP Development

Automated Testing In
MMP Development
Controlling Cost, Risk and Schedule
Larry Mellon
Austin Game Conference
Sept 10-11, 2003
Major Functions of Automation
Repeatable, Synchronized Input
Debugging
Performance Tuning
Data Management
Raw Data
Test Results
Regression
Major Weaknesses of Automation
Visual Effects
Rigid Inputs, Rigid Analysis
Judgment
Semi-Automated Testing
(2)
(1)
Command Steps
Automation
(3)
Manual
Validation Steps
Automation: Architecture
Startup
Control
Repeatable, Sync’ed
Test Inputs
System Under Test
System Under Test
&
Collection
&
Analysis
System Under Test
Test Manager
Scripted Use Cases
Report Managers
Test Selection/Setup
Control N Clients
RT probes
Emulated User Play Sessions
Multi-client synchronization
Raw Data Collection
Aggregation / Summarization
Alarm Triggers
Automated Testing: Topics
Automated Testing: Overview
Business Case
Development risk / operational costs
Customer satisfaction
TSO Applications & Implementation
Wrapup
Questions
Development / Schedule Risk
Complex, Large-Scale System
Large Development Team
How do we know when we are ready to launch?
How do we know how fast we are going?
Accelerated, Measurable Development
Repeatable, Synchronized Input
Reference Cases
Stability
Development
Instant Feedback,
Rapid Triage,
Summarized, Aggregated
History
Results
Data Management
Management
Current Status
History
Trends
Customer Satisfaction
Stability
New Content
Automated testing keeps existing code working
while new code is being added.
Subscription: Profit & Running Costs
Profit…
New
Content
~ $10
per
customer
Regression
Customer
Support
Operations
Automation: Lower Running Cost
Profit…
~ $10
per
customer
Regression
Customer
Support
Operations
Lower New Content Cost
Automation: Lower Running Cost
Profit…
~ $10
per
customer
Lower New Content Cost
Lower Testing Cost
Customer
Support
Operations
Automation: Lower Running Cost
Profit…
~ $10
per
customer
Lower New Content Cost
Lower Testing Cost
Happy Customers Don’t Call
Operations
Business Case: Summary
Tool investment accelerates
development, provides tangible
measures of progress
Tool investment pays out over entire
lifecycle of subscription-based
service
Automated Testing: Topics
Automated Testing: Overview
Business Case
The Sims Online
Applications
Implementation
Wrapup
Questions
TSO: Applications
Operations
QA Regression
Development
Load Testing
Stability
Load Testing: Goals




Expose issues that only occur at scale
Establish hardware requirements
Establish response is playable @ scale
Emulate true user behaviour
Resource
Metrics
Load Testing Team
Debugging
Data
Client
Metrics
Load Control Rig
Test
Test
Test
Client
Client
Client
Test
Test
Test
Client
Client
Client
Test
Test
Test
Client
Client
Client
Test Driver CPU
Test Driver CPU
Test Driver CPU
Game
Traffic
System
Monitors
Server Cluster
Internal
Probes
Stability Via Monkey Tests
Continual Repetition of Unit / Integration Tests
Code Repository
Compilers
Reference Servers
Stability Via Monkey Tests
Moving parts
1. Last “known good”
2. What works, right now
3. Aggregate reporting
Aging
Reference Servers
Monkey Test: EnterLot
Non-Deterministic Failures
Critical Path: Unit Testing
Test Case: Can an Avatar Sit in a Chair?
use_object ()
buy_object ()
enter_house ()
buy_house ()
create_avatar ()
login ()
Failures on the
Critical Path block
access to much of
the game.
Worse, unreliable
failures…
Regression
Impact
On
Others
Smoke
Build
days
Bug
Introduced
Checkin
Time to Fix
Development
Cost of Detection
Pre-Checkin Regression: don’t let
broken code into Mainline.
Regression
Smoke
Checkin
Candidate
Code
Sniff
Pass / Fail,
Diagnostics
Development
Working
Code
Automated Testing: Topics
Automated Testing: Overview
Business Case
The Sims Online
Applications
Implementation
Wrapup
Questions
Single, Data Driven Test Client
Testing feature correctness
Testing system performance
Load
Regression
Reusable
Scripts & Data
Single API
Test Client
Data Driven Reporting
Load
Regression
Reusable
Scripts & Data
Single
API
Test Client
Single
API
Key Game States
Configurable
Logs & Metrics
Pass/Fail
Responsiveness
Test Client Derived From Real Client
Game Client
View
Presentation Layer
Logic
Scripted Player Actions
Test Client
Game Client
Script Engine
Game GUI
State
State
Commands
Presentation Layer
Client-Side Game Logic
What Level To Test At?
Mouse
Clicks
Game Client
View
Presentation Layer
Logic
Regression: Too Brittle (pixel shift)
Load: Too Bulky
What Level To Test At?
Game Client
View
Internal
Events
Presentation Layer
Logic
Regression: Too Brittle
(Churn Rate vs Logic & Data)
Gameplay: Semantic Abstractions
Basic gameplay changes less frequently
than UI or protocol implementations.
View
Logic
NullView Client
Presentation Layer
Chat
Route Avatar
…
Enter Lot
Use Object
Scriptable User Play Sessions





Set of Presentation Layer “primitives”
State probes: peek/poke arbitrary game state
Synchronization: wait_until, remote_command
Recordable…
Test Scripts:
Specific / ordered inputs
 Specific game responses


Single user play session, multi-user session
Toolkit Usability
Data Managers
Pushbutton Use: Developers, Testers,
Test Farms, Everywhere
Continual Process Improvement
Tool UI / feature set
False positives
Automated Testing: Topics
Automated Testing: Overview
Business Case
TSO Apps & Impl
Wrapup
Questions
Wrapup
Very Useful, Very Reusable
Development Acceleration
Lower Operational Costs
Tabula Rasa
Cautionary Tales
Tabula Rasa
PreCheckin SniffTest
Keep Mainline working
Hourly Stability Checkers
Baseline for Developers
Dedicated Tools Group
Executive Support
Load Test: Early & Often
Easy to Use == Used
Radical shifts in Process
Break it before Live
Distribute Test Development &
Ownership Across Full Team
Cautionary Tales
Flexible Design Requires Flexible Tests
Signal To Noise Ratio
Defects In The Testing System
Automated Testing: Topics
Architecture & Applications
Business Case
The Sims Online
Applications
Implementation
Questions

Download Report