User Experience Design
Bucharest
#uxdesignro
www.facebook.com/groups/uxbucharest/
User Experience Design Bucharest
AGENDA
19:00
Paul RAWSTON
Experience Design Leader (Digital Marcom CoE). Honeywell UK
19:30
Horia GALATANU
Senior Product Manager for Adobe Primetime entitlement,
Adobe Systems US.
20:15
Emil IONESCU
General Managing Partner at BestMusic, METALHEAD & iabilet.ro.
Festivals organizer.
User Experience Design Bucharest
The role of A/B Testing in optimizing the
conversion rate for websites and emails
Paul RAWSTON
Experience Design Leader (Digital Marcom CoE). Honeywell UK
Paul Rawston BA Hons DipM MCIM UXQB
• Experience Design Leader for Honeywell ACS
since Nov 2015.
• Leading a team within Honeywell’s Digital
Center of Excellence providing UX and content
marketing strategies.
• Worked in digital marcoms for over ten years.
• Passion for digital marcoms, UX,
content/inbound marketing, and graphic
design.
• Live in Derbyshire, UK.
6
What we’ll cover
• What is A/B testing?
• Why Test?
• What to test? Real A/B test examples to evaluate and vote on
- Landing Pages
- Display Ads
- Emails
• Tools
• Summary
7
What is A/B testing?
A/B testing (sometimes called split testing) is
comparing two versions of a web page,
email, landing page, display ad to see which
one performs better.
You compare by designing two variants
(A and B) and testing with your users.
8
Why test?
• Understand your audience and their
behaviours - what do they actually
respond to?
VERSION A
vs
VERSION B
• Test those best practice conventions challenge your perceptions, how do you
know what works without testing?
• Small tests can have surprising results –
you could increase your conversion rate
significantly by making some minor
changes to your landing pages, emails,
display ads, etc.
9
What to test? – landing page
Navigation bar
Keep or remove?
Call-to-Action button
Placement, shape, size,
colour and phrase
Headline
Impact
Clarity
Images
Style, size, placement and
even necessity
Body Copy
Paras or bullet points? Key
messaging highlighted?
Less or more?
Online form
Form fields and placement
10
What to test and real test examples
• Individually vote by raising your hand for which version you think won
• Go with your gut and share your reasoning with us
Which ‘Add Button’ colour increased orders for this industrial products supplier?
Version A
Version B
Green button
Orange button
12
Winning version: Version B
• Version B, featuring the orange button, was the clear
winner
• The orange button lifted orders 22.7% compared to the
green version.
• Button colour wasn’t actually what made the difference
here. Rather, it was probably the colour contrast created
from changing the button to orange.
• Looking at the site’s colour palette, the orange button
stood out from the rest of the page. Because it popped,
the eye was drawn to it.
• In comparison, the original green button blended in with
the other teal headers on the page, so it didn’t jump out
as much.
13
Which landing page got more trial downloads?
14
Version A
Shorter headline, award winner visual
Version B
Longer Headline, layout of copy, social proof
15
Winning version: Version B
• Version B increased trial downloads by
98%
• Does the basics really well
• It’s simple and clear of distractions
• Has good headline copy
• Has good supporting copy detailing the
benefits and features
• More testimonials for those who want
to know more
16
Which online form increased leads?
Version A
Version B
17
Winning version: Version A
• Version A increased leads by 368.5%
• Fewer fields keeping things really tight
• Improved grouping to visually shrink the impact of the form
• Additional visuals used
• Additional check boxes as opposed to open text fields
• Clear CTA and bright red button to get a quote
What to test? – online display ad
Personalization
Headline
Impact
Clarity
Messaging / Offer
Images
Style, size, placement and
purpose
Call-to-Action button
Placement, size, colour and
phrase
18
Which Facebook mobile Newsfeed Ad generated
more Sign Ups?
19
Version A
Version B
20
Winning version: Version B
Version B increased Sign Ups by 59.4%
• Much to RummyCircle’s surprise, the
shorter text made all the difference with
mobile users
• Shorter copy worked best to engage
mobile Facebook users. Too much copy
detracted rather than attracted attention.
• Version A worked better for desktop users
• Based on this finding, it seems, the ideal
mobile Facebook ad copy should be short,
clear, direct, and highly actionable.
What to test? – email
Subject line
Pre-header text
Personalization
Call-to-Action button
Placement, size, colour and
phrase
Layout
Maybe change the
placement of different
elements. Single column or
two column?
Images
Style, size, placement and
purpose
Headline
Does it have impact, is it
clear to the user where
they’ve landed?
Body Copy
Paras or bullet points?
Key messaging
highlighted?
Less or more?
21
Which email increased CTR for Microsoft?
Version A
Version B
White Background, Full Product Image, Purple CTA
Orange Background, Cropped Product Image, Pink CTA
22
23
Winning version: Version A
• Version A increased clickthroughs by 325%
• Made it easier to find where
you were supposed to click.
• The contrast of the purple CTA
against the white background
draws your eye immediately.
Which email subject line increased open and click through rates?
Version A
Improve Deliverability in Two
Simple Steps
Version B
Amanda, Improve
Deliverability in Two Simple
Steps
24
25
Winning version: Version B
• Version B increased opens by
5.13% and increased CTR by
17.36%?
• Personalised subject line
• Can be a gamble as it does
increase the amount of characters
• However if done right,
personalization is proven to boost
response rates
“Amanda, Improve
Deliverability in Two
Simple Steps”
Which Honeywell B2B email had a superior CTR?
Version A
Text heavy, images further down the fold, fewer visuals
Version B
Easier navigation, cleaner layout, text reduced, images
moved up
26
Winning version: Version A
• Version A increased click-throughs
by 83%
• Why? We’re still trying to figure
that out!
• Initial thinking is that our
customers have become familiar
with the old design over a number
of years.
• Is the clean design that important
to this particular user group?
Maybe there are some elements
they prefer in Version A.
• The trouble with multivariate
testing is that it’s hard to know
what’s made the difference.
27
Tools
• Optimizely and Visual
Website Optimizer – leading
website and mobile A/B
testing platforms
• VWO’s free A/B Test
Calculator
• VWO’s free A/B Split Test
Duration Calculator
• Google Analytics Experiments
• Google AdWords
28
A word of caution
• Big wins are sometimes hard to achieve
• Many tests won’t provide succinct results
(multivariate tests for example)
• Takes time so don’t waste it testing
everything and anything. Try to focus on
testing landing pages, emails, ads that are
critical to your goals.
• Is A/B Testing truly what you need at this
point? If you’re looking for more visitors
to your site as opposed to increasing
conversions then maybe not.
29
30
Takeaways
• Testing is extremely important. Unless you test, you can never be absolutely sure.
• What opportunities do we miss out on by not testing enough? These companies
experienced better ROI simply by testing a couple of variations.
• Good things can happen when you get to know your users.
• Fail quickly – if one of your tests is creating poor performance – pull it.
• Have goals – know why you’re testing and what the goal is. Test one variable at a
time.
• Be objective and don’t have any preconceived notions of what version may win.
• Use the data to drive decisions.
Thanks for playing!
For more A/B test examples go to whichtestwon.com
[email protected]
https://uk.linkedin.com/in/paulrawston-85055935
31
User Experience Design Bucharest
The TV of Tomorrow and the role of the
UX Design in a changing media
industry
Horia GALATANU
Senior Product Manager for Adobe Primetime entitlement,
Adobe Systems USA.
User Experience Design Bucharest
The TV of Tomorrow and the role of the
UX Design in a changing media
industry
User Experience Design Bucharest
"Events- As designed" – or how to
design the user experience for music
festivals
Emil IONESCU
General Managing Partner at BestMusic, METALHEAD & iabilet.ro.
Festivals organizer.
EVENTS: AS DESIGNED
User Experience intr-un festival de muzica
Nimeni nu mai vine
la concerte
E NEVOIE SA FACI
DOAR CATIVA PASI.
1. Publicul tinta si
nevoile sale
2. Cum adaptezi nevoia de
apartenenta
si de dragoste la brand.
Iteratii anuale
2. Cum adaptezi nevoia de
apartenenta
si de dragoste la brand.
Iteratii anuale
Trenduri
social media
Ce-si doresc
oamenii in
functie de
varsta
Cum alaturi generatii
si nevoi diferite
Cum faci pe toata
lumea fericita:
Ce trupe chemi?
3. Contextul global & local pe nisa
targetata
3. Contextul global & local pe nisa
targetata
Directia industriei
- Local
- Global
Preturi
Bilete
Alte
Evenimente
Artisti Noi vs Existenti
4. Bugetul
4. Bugetul
Pretul biletelor /
abonamentelor
Colaborarea cu diverse
branduri per eveniment
Legarea nevoii de
apartenenta la
experienta de la
concert
Activari, happening-uri
5. Quality (of life)
5. Quality (of life)
Siguranta,
autorealizare
Micile lucruri
care conteaza/
Marile lucruri
Care conteaza
Scenotehnica
Ecrane
Leduri
Echipamente
de calitate
Trupele Mici
dar la moda
6. Comunicarea
6. Comunicarea
Evenimentul e deja un
succes dinainte sa aiba loc
Apartenenta este baza pe
care se construieste totul.
Evenimentul este
o comunitate
User Experience Design
Bucharest
THANK YOU!
www.facebook.com/groups/uxbucharest/