Best practices for building a
testing culture
Presented by : Mari Ju and John Oppenheimer
Optimizely User Group - August 2015
COMPANY NAME
www.nicecompany.com
About US
Mari Ju
Product Marketing Manager
[email protected]
linkedin.com/in/mariju
@mari4ju
John oppenheimer
Marketing Operations Analyst
[email protected]
linkedin.com/in/oppenheimerjohn
about Autodesk
autodesk
Mission: Help people imagine,
design, and create a better world
Est. 1982
12+ million users
Autodesk consumer and 3d Printing
Mission: Unlock the creativity in
everyone
Est. 2011
170+ million users
Testing at Autodesk
Fall ‘14
Spring ‘14
Start
small
Fall ‘15
Spring ‘15
Build
trust
GENERATE
Excitement
Advance &
REPEAT
Phase 1: Start Small
Training
•
Onsite demo
•
Team sessions
•
Dedicated champions
Original
Variation
vs.
Pick a starter property
•
Simple first test as proof of
concept
Results: Variation showed 24% more
subscriptions compared to the Original
Phase 2: Build trust
Address potential concerns:
•
Test proposals to larger team
•
Allow for plenty of QA
•
Connect developers with Optimizely
support team
•
Have developers launch test
•
One test at a time, simple tests only
PhaSE 3: Generate excitement
Spread testing throughout the organization:
• Add more products/domains
• Get more team members involved
Improve testing program:
• Develop testing workflow:
Data à User Testing à Hypothesis à A/B Test
• Advanced tests (multi-page, redirects, cross-platform)
that leverage integrations
• Establish testing as essential when updating the site
Share results (both wins and losses):
• Distribute reports via email and internal blog
• Celebrate team wins (Optie finalist)
Phase 4: advance
Continuously advance our testing program:
• Standardized idea backlog and prioritization rubric
• Create an archive of previous tests that’s easy to
share and search
• Work closely with designers and developers to
launch more technical tests
• Start experimenting with personalization
• Launch mobile app A/B testing program
Evangelize benefits of testing:
•
Internally – other products, departments, office
locations, and more
•
Externally – with partners and user groups
Original
Variation
Steps for building a testing culture
1
2
3
1. Start small
3. Generate excitement
Demonstrate potential
impact of testing with a
simple test
Increase complexity and
frequency of tests for greater
impact, and share results
2.Build trust
4. ADVANCE
Bring all stakeholders into
the process to gain trust,
and eventually autonomy
Continue to evaluate and
improve testing program
4
Questions?
Mari Ju – [email protected]
John Oppenheimer – [email protected]
COMPANY NAME
www.nicecompany.com