The art and science of
quality assurance in the
video game industry
Our journey to transform QA at EA Canada
Mario Vasquez
A tale of 2 QAs
The old QA
 Defect detection
 Blackbox test centric
 Audit quality
The new QA
 Defect prevention
 Highly skilled
 Build to quality
Our north
Achieve and sustain the highest
appropriate quality in the industry
while decreasing overall
expenditure
Prevention over detection
From: http://software.isixsigma.com/library/content/c030611a.asp
QA function
Product
Quality
Software
Quality
Management
Highly skilled quality
assurance Team
QA scope
Preproduction
Production
Finaling
Certification
Launch
New
Old
QAQA
End to end Quality ownership
Live
Software Quality (SQ)
Software
Quality
Product
Quality
Management
QA technology
Software
Quality
Product
Quality
Management
SQ helps Development
Software
Quality
Product
Quality
Management
Automated smoke test
Development
“Autosmoke”
Production team
Build
system
Internal or
external QA team
Feedback
Many others …
Software
Quality
Product
Quality
Management
SQ helps manual testing
Telemetry
Test
Automation
Test tools
• Reports on key metrics (e.g. coverage, completion)
• Supports Decision making
• Tailored to project needs
• Reduces manual testing required
• Must have to complete manual test plans
Software
Quality
Product
Quality
Management
Product Quality (PQ)
Software Quality
Product
Quality
Management
The embedded QA cycle
QA
QA
Development
Production
QA
Development
Production
QA
Development
Production
QA
Development
Production
Development
Production
Software Quality
QA
Development
Production
Product
Quality
Management
Embedded helps
Build relationships and allegiances
 Enable creativity and change while
maintaining control
 Increase organizational awareness
of the QA role and value add
 Improve test strategies and plans

Software Quality
Product
Quality
Management
Management
Software Quality
Product
Quality
Management
Communication
Relevant and appropriate
 Consistent
 Timely

Software Quality
Product
Quality
Management
Outsourcing
Flexibility!
 Reduces EA studio footprint
 Reduces seasonal hiring/training
burden
 Load balance
 Cost savings

Software Quality
Product
Quality
Management
YOY test outsourcing
Blackbox Test - Team Profile
120%
100%
0%
1%
16%
13%
20%
80%
41%
60%
100%
99%
40%
84%
87%
80%
59%
20%
0%
FY05
FY06
FY07
On-Site
FY08
FY09
FY10 (Est)
Off-site (outsource)
Software Quality
Product
Quality
Management
Investing pays off
2 year comparison in Outsourcing results
120,000
108,726
109,057
100,000
80,000
+58%
44,440
60,000
28,059
40,000
+16%
38,982
FY07
FY08
20,632
20,000
0
Total Test Hours
Total Bugs Found
Valid bugs
Software Quality
Product
Quality
Management
Organizational considerations
Tackle common components first
 Offer incremental approach
 Prepare to refactor the plane while
in flight
 Never underestimate culture and
team differences

Software Quality
Product
Quality
Management
Moving in the right direction
Overall results
Reduction in issues and shortened
bug lifecycle
 QA cost reduction
 High customer satisfaction ratings
 Strong team health
 Increased Game Quality!

Key takeaways
Team recipe
Product
Quality
This is a marathon
Software
Quality
Management
Highly skilled quality
assurance team
Communication & trust
Thank You!
Mario Vasquez
[email protected]