Joshua G Rice. Improving Conversion Rates at 80 Percent Solutions. A Master’s Paper
for the M.S. in I.S degree. August, 2014. 44 pages. Advisor: Paul Jones
This experiment used A/B testing to attempt to improve three tracked conversion goals-“Engagement”, “Paypal Submission”, and “Purchase Clicks” --on 80 Percent Solution’s
product website www.MacFreedom.com. The participants for this study were web users
from any location or device who visited www.MacFreedom.com during the testing
period.
Three randomized controlled experiments were executed using Visual Website Optimizer
to test for statistical significance of the conversion goals. The results of these experiments
showed that:
•
Changing the site’s tagline demonstrated significant success.
•
Changes to landing page call-to-action buttons exhibited negative statistical
significance.
•
Variations of the purchase page call-to-action buttons demonstrated significant
success.
Headings:
A/B testing
Quantitative marketing research
Information services -- Marketing
Response rates
User interfaces (Computer systems)
IMPROVING CONVERSION RATES AT 80 PERCENT SOLUTIONS
by
Joshua G Rice
A Master’s paper submitted to the faculty
of the School of Information and Library Science
of the University of North Carolina at Chapel Hill
in partial fulfillment of the requirements
for the degree of Master of Science in
Information Science.
Chapel Hill, North Carolina
August 2014
Approved by
_______________________________________
Paul Jones
1
Table of Contents
Introduction ........................................................................................................................ 2 Hypothesis .......................................................................................................................... 3 Review of Literature ........................................................................................................... 3 Method ............................................................................................................................... 9 Experiment 1: Tagline Text Variations ...................................................................................... 11 Experiment 2: CSS Call-­‐To-­‐Action Button & Background Div Variations .................................. 13 Experiment 3: Purchase Page “Buy Now” CTA Button Variations ............................................ 19 Results .............................................................................................................................. 21 Discussion ......................................................................................................................... 24 Limitations ........................................................................................................................ 26 Conclusion ........................................................................................................................ 27 Areas for further research ................................................................................................ 28 References ........................................................................................................................ 29 Appendix ........................................................................................................................... 37 1.1 ............................................................................................................................................. 37 1.2 ............................................................................................................................................. 39 1.3 ............................................................................................................................................. 41 2
Introduction
The use of controlled experiments has allowed Amazon.com to “innovate quickly
and effectively” (Kohavi, Henne, & Sommerfield, 2007). Numerous experiments are
thought to be “running at any given time” on Amazon.com “because of its relatively low
cost,” according to Jennifer Cardello, director at Nielsen Norman Group (2014).
Amazon.com, a multibillion-dollar e-commerce company, uses its website as its main
marketplace to sell products thus tracking user engagement to measure company success.
Not just large companies like Amazon use controlled experiments to enhance their
website, but many small e-commerce companies rely on their sites as their main source of
revenue.
For this project, I worked with 80 Percent Solutions, a small digital software
company located in Chapel Hill, North Carolina. 80 Percent Solutions has one ecommerce website responsible for the majority of the company’s profits. Like many
small web companies, 80 Percent Solutions has limited resources for marketing and
advertising much less running controlled experiments on the primary product website. I
predict that running controlled experiments will be extremely important to the company’s
success, leading directly to increased sales.
3
Hypothesis
By manipulating 80 Percent Solution’s MacFreedom.com website tagline, landing
page call-to-action button, and purchase page “Buy Now” call-to-action buttons, I will
produce positive statistically significant conversion rates in comparison with the controls
for the following conversion goals: “Engagement”, “Paypal Submissions”, and “Purchase
Clicks”.
Review of Literature
Most successful 21st century companies have a presence on the Internet usually in
the form of a company website. Web users who visit a company’s website have the
ability to obtain information about the company and potentially purchase associated
goods. Companies can reach customers across physical barriers previously impossible
before the emergence of the World Wide Web. When businesses “trade products or
services conducted via computer networks,” this is known as e-commerce (E-commerce,
n.d.). E-commerce websites can be extremely important for some companies to sell their
products and services, as the site may exist as the only medium to sell.
If companies sell their products solely through e-commerce websites, this makes
the content and design elements of the website very valuable to the success of the
company. Websites should be carefully planned to provide a very intuitive user
experience enabling customers to purchase goods successfully. A customer’s user
experience on a website is extremely important for companies to optimize profit, because
it enables them to get from point A to point B, converting the user from a visitor to a
4
customer. Any lack of clarity in a website’s landing page or home page can result in a
user immediately navigating away from the site and thus not buying the product.
Companies that use a web presence as the only market to sell products should focus on
the design and content of their website to maximize sales.
After doing a large amount of research on how to optimize e-commerce websites,
call-to-actions showed up repeatedly as the key element for optimization. Call-to-actions
(CTAs) within websites are usually buttons or large graphics that prompt users to
perform specified actions. Whether this action is “to download a PDF, fill out a form, buy
a product, or even just click through to another page,” CTAs are the medium through
which the user interactions take place (Aagaard, 2013). There are a few elements
common across various sources that make up CTAs: “size, message (text), placement and
color” (Adlakha, 2013). The size of the CTAs should be “large” as well as consistent in
the same way that “you want to keep your branding consistent” (Bakkestuen, 2013). The
size and placement elements of the CTA button should combine to create buttons that
“stand out without overwhelming the design” (Chapman, n.d.). Numerous sources use the
term “Above The Fold” when describing the general placement and location of CTA
buttons on landing pages (Fishleder, 2014). Jakob Nielsen, a famous website usability
guru, defines the “Fold” as the space on a web page that is “viewable without further
action” (2010). He also makes any information placed in the fold seem important by
pointing out that “Web users spend 80% of their time looking at information above the
page fold” (Nielsen, 2010).
According to the popular analytics, marketing, and testing blog Kissmetrics, color
is the “strongest and most persuasive” visual cue (Smith, 2014). Despite the existing
5
theory and field of science known as color psychology, the color of CTA buttons “should
be in stark contrast to background colors” in order to capture attention (Chawla, 2013).
The text or message of a CTA button should give the user “a reason to click” as well as
“tell them clearly what they’ll get” without the use of “vague words” (Babbar, 2014).
Paras Chopra, chief executive officer of a popular marketing and analytics software
company, wrote a case study describing the increase of customers from a slight change of
text in a CTA button. The takeaway lesson learned from the case study was that if the text
within a CTA button “pushes too hard at the beginning,” a user “might feel distracted,
uncomfortable and leave the site” (Paras, 2011).
In this paper, I focused on creating variations of CTA buttons along with
changing landing page tagline text with the intent to improve defined conversion goals.
Winning variations will directly increase sales and conversions for 80 Percent Solutions.
80 Percent Solutions is a small digital software company based out of Chapel Hill,
North Carolina that uses an e-commerce website to sell software applications. 80 Percent
Solutions provides productivity tools with the main purpose of reducing digital
distractions. The company was founded in 2011 and is single-handedly run by Fred
Stutzman, Ph.D. The company currently is based in a Chapel Hill technology incubator
called Launch Chapel Hill.
Freedom, the company’s first product developed, has roughly 50,000 paid
customers and around 1,500 unique visitors every day. As of June 2014, the site’s main
product web page MacFreedom.com holds a top position on the first page of Google
results for the search term “Freedom”, which signifies the amount of daily traffic to the
website. The application works by allowing users to restrict how long they would like to
6
be without Internet, cell data, or Bluetooth access. There is also the ability to schedule
blocked periods in advance on specific days and times of the week. Freedom has been
proven successful by its popularity and effectiveness since it was first released to the
public for free in its early stages. News stories and articles about Freedom have appeared
across the web and in print at multiple mainstream media outlets such as The New York
Times, USA Today, and Time Magazine (Stutzman, n.d.).
Because of the company’s size and limited resources, no advertising or marketing
techniques are currently in place to help attract new customers. This means that most new
customers find out about Freedom through 80 Percent Solution’s website. The company
is trying to improve its website and increase sales by optimizing three conversion goals.
The first goal 80 Percent Solutions seeks to improve is user conversions, which is
the number of website visitors who buy Freedom and convert to paid customers. This
goal leads directly to increased sales and is the most important conversion goal to
improve.
Before a visitor of the site is able to buy Freedom, they must click to the purchase
page. The next conversion goal needing improvement is the number of clicks to this
page. This also plays a significant role as a conversion goal, because it can determine
whether or not the users are getting close to buying the product. As one would imagine,
the more clicks to the purchase page is probably related to more clicks to buy the product.
The last targeted conversion goal is related to the user’s activity when he or she
first lands on the website. Users who make a click or complete any action on the site are
tracked differently than users that leave the site immediately after they visit. If a user
visits MacFreedom.com and then immediately leaves the website, this would count as a
7
user who did not interact with the site and would thus contribute to a metric known as
bounce rate. In other words, this is the rate at which users visit and leave without
engaging with the website. The more engagement, the lower the bounce rate.
Fortunately, there are solutions and testing tools available that can optimize
usability. These tools allow website administrators to record and gather information about
their site. This may be as simple as tracking the number of clicks or submissions from an
html form. The tools work by embedding snippets of code in the website that collect data
on visiting users’ behavior. By tracking behavior, trial and error comparison tests can be
run to determine which version of the website produces the best results. These tools can
effectively optimize a website to increase user engagement with the site, ultimately
leading to increased purchase rates on the website.
Visual Website Optimizer is website optimization tool that over 1200 companies
have used to increase their sales and conversions (“What is Visual Website Optimizer
and how is it used?”, 2012). There are approximately 100 case studies written up on the
tool’s website that highlight how the tool was used to increase sales and conversions at
other companies. According to Becky Simanowski, an A/B Testing and Optimization
Program Manager for SAS Institute’s main company website, this tool is popularly used
as an “industry standard” (personal communication, May 1, 2014). Visual Website
Optimizer has a comprehensive number of features, but the main one I will be looking at
is the A/B test.
An A/B test is buzzword in marketing and web analytics that simply means a
randomized controlled experiment. In a basic A/B test, you have a control (A) and a
variation (B) that you are testing against each other to see the difference in whatever goal
8
or goals you are attempting to track. With the amount of traffic that 80 Percent Solutions
gets on their primary product website MacFreedom.com, an A/B/n test or A/B split test is
of better use. In an A/B split test, more than one variation can be tested against the
control because there is sufficient website traffic. This traffic is randomized and split up
between each variation and the control. Although very simple in terms of the type and
complexity of the test, A/B tests can provide a large number of advantages to
organizations (Nielsen, 2005).
There have been a few A/B tests run on the landing page since the website first
launched in 2011. None have been run recently, because the company is in a transitional
growth stage trying to hire employees and gather funding for the development into a
software as a service (SaaS) platform. Although the previous tests were done using
Visual Website Optimizer, they are no longer available for reference by the time of my
tests.
After fully understanding the intention of the product previously as a research
intern with the company, I decided to choose two areas of MacFreedom.com that need
improvement. Both areas have the potential to enhance website conversion rates.
The first area of improvement is the text of the website’s tagline. A tagline is a
short “memorable phrase or sentence that is closely associated with a particular person,
product, movie, etc.” (Tagline, n.d.). Jakob Nielsen emphasizes that a website’s tagline
should “tell us what they actually sell” versus using a more general phrase to describe a
benefit of that particular product or service (2001). Macfreedom.com’s current website
tagline is “Freedom for Productivity”, which does not clearly indicate what the company
is selling to the user.
9
The other area of improvement is the call-to-action buttons. CTAs “form a crucial
element of your landing page and the conversion funnel” (Chawla, 2013). The CTA
buttons on a website should stand out while maintaining consistency with other CTAs
used on the site (Bakkestuen, 2013). Currently, MacFreedom.com’s CTA buttons on the
landing page and purchase page vary in color and design consistency.
In this paper, I focused on creating variations of the landing page tagline and CTA
buttons with the intent to improve the above three conversion goals. Winning variations
will directly increase sales and conversions for the company.
Method
Three experiments were run testing three conversion goals set using the online
software tool Visual Website Optimizer. 80 Percent Solutions had an existing license for
the software. Each experiment was tested for “Engagement” and “Paypal Submissions”
set as goals within Visual Website Optimizer. Experiment 1 and Experiment 2 were also
tested for “Purchase Click” as a conversion goal. Because Experiment 3 involved a
variation change already located on the purchase page (MacFreedom.com/purchase), this
experiment tested for “Google Play Click” as the third conversion goal. This conversion
goal tracks clicks to the Google Play “Buy Now” CTA button
(https://play.google.com/store/apps/details?id=com.eightypct.freedom).
I used the term “Engagement” to mean the same definition as engagement within
the Visual Website Optimizer knowledgebase ("What does “engagement” ”, 2013). This
term measured when a user either filled out an html form or clicked on any of the
following seven html elements:
10
1. Anchor element(<a></a>)
2. Any element which is a descendent of Anchor element(e.g. click on img in
<a><img /></a>)
3. Button Element(<button></button>)
4. Any element which is a descendent of button element(e.g. click on span in
<button><span></span></button>)
5. Input element with type button (<input type=”button” />)
6. Input element with type submit (<input type=”submit” />)
7. Input element with type image (<input type=”image” />) ("What does
"engagement on the page" means?", 2013).
“Purchase Click” was defined as a conversion goal to mean the number of times a
user clicked to purchase Freedom. These clicks came from the landing page CTA buttons
located in the top right and bottom middle part of the page. Any clicks to the purchase
page (located in the navigation bar at the top of the web page) also were counted. These
clicks did not necessarily mean that users purchased Freedom. The next conversion goal
tracked this measure.
“Paypal Submissions” was the number of times a visitor confirmed and submitted
payment for a purchase. Ultimately this was the most important goal to track because it
related directly to increasing total revenue and sales.
Three experiments were run one at a time in numerical order, each starting on a
new Monday of the week. A screenshot of each experiment’s variation(s) and control has
been provided along with what was needed to create the variations. Three experiments
were run until a level of statistical significance was reached.
11
Variations were considered successful if the difference of the specified conversion
rate goal and the control was statistically significant. The conversion rate goal was
“represented by a binomial random variable,” which basically means that it either has a
value of converting or not converting (Chopra, 2010). If a variation of any of the
experiments did very poorly and proved statistically significant, the variation was
disabled so that it did not hurt sales for 80 Percent Solutions.
Experiment 1: Tagline Text Variations
Experiment 1 started on June 9th, 2014 and was an A/B split test. Three variations
of the website’s tagline were compared to the control tagline “Freedom For Productivity”
shown below.
12
•
Variation 1: “Eliminate Digital Distractions”
o
•
Text changed in front page html file
Variation 2: “Block Digital Distractions”
13
o
•
Text changed in front page html file
Variation 3: “Freedom From Digital Distractions”
o
Text
o
Text changed in front page html file & “font-size: 25px” changed
within master.min.css file.
Experiment 2: CSS Call-To-Action Button & Background Div
Variations
This experiment was also a split A/B test comparing three different variations of
the landing page top right CTA button with the control shown below.
14
•
Variation 1: CTA button layered with soft grey background div
o
•
Within master.min.css file, edited #buttonDarkBG
§ Replaced existing background-color value #2b333d to #eff1f3
Variation 2: Green CTA button layered with dark background div
15
o
Within master.min.css, edited a.buttonLink, a.buttonLinkWithImage
§ Removed existing background css:
background:url("https://dlvfa1owjktk.cloudfront.net/freedom/web/
844f58909f6c1a1ce71fc411b0ed70a0/img/buttonBGOut.png") 0 0
repeat-x;
§ Added new background gradient css:
background: background: #49d100;
background:
url(data:image/svg+xml;base64,PD94bWwgdmVyc2lvb
j0iMS4wIiA/Pgo8c3ZnIHhtbG5zPSJod…EiIGhlaWdod
D0iMSIgZmlsbD0idXJsKCNncmFkLXVjZ2ctZ2VuZX
JhdGVkKSIgLz4KPC9zdmc+);
background: -moz-linear-gradient(top, #49d100 0%,
#00ad00 100%);
background: -webkit-gradient(linear, left top, left
bottom, color-stop(0%,#49d100), colorstop(100%,#00ad00));
background: -webkit-linear-gradient(top, #49d100
0%,#00ad00 100%);
background: -o-linear-gradient(top, #49d100
0%,#00ad00 100%);
background: -ms-linear-gradient(top, #49d100
0%,#00ad00 100%);
background: linear-gradient(to bottom, #49d100
0%,#00ad00 100%);
filter: progid:DXImageTransform.Microsoft.gradient(
startColorstr='#49d100',
endColorstr='#00ad00',GradientType=0 );
16
o
•
Within master.min.css file, edited
a.buttonLink:hover,a.buttonLinkWithImage:hover
§ Removed existing button hover css:
background:url("https://dlvfa1owjktk.cloudfront.net/freedom/web/
844f58909f6c1a1ce71fc411b0ed70a0/img/buttonBGOver.png") 0
0 repeat-x
§ Added new background gradient hover css:
background: #00ad00; /* Old browsers */
/* IE9 SVG, needs conditional override of 'filter' to
'none' */
background:
url(data:image/svg+xml;base64,PD94bWwgdmVyc2lvb
j0iMS4wIiA/Pgo8c3ZnIHhtbG5zPSJodHRwOi8vd3d3L
nczLm9yZy8yMDAwL3N2ZyIgd2lkdGg9IjEwMCUiIG
hlaWdodD0iMTAwJSIgdmlld0JveD0iMCAwIDEgMSI
gcHJlc2VydmVBc3BlY3RSYXRpbz0ibm9uZSI+CiAgP
GxpbmVhckdyYWRpZW50IGlkPSJncmFkLXVjZ2ctZ
2VuZXJhdGVkIiBncmFkaWVudFVuaXRzPSJ1c2VyU
3BhY2VPblVzZSIgeDE9IjAlIiB5MT0iMCUiIHgyPSIw
JSIgeTI9IjEwMCUiPgogICAgPHN0b3Agb2Zmc2V0PS
IwJSIgc3RvcC1jb2xvcj0iIzAwYWQwMCIgc3RvcC1vc
GFjaXR5PSIxIi8+CiAgICA8c3RvcCBvZmZzZXQ9IjE
wMCUiIHN0b3AtY29sb3I9IiM0OWQxMDAiIHN0b3A
tb3BhY2l0eT0iMSIvPgogIDwvbGluZWFyR3JhZGllbn
Q+CiAgPHJlY3QgeD0iMCIgeT0iMCIgd2lkdGg9IjEiI
GhlaWdodD0iMSIgZmlsbD0idXJsKCNncmFkLXVjZ2
ctZ2VuZXJhdGVkKSIgLz4KPC9zdmc+);
background: -moz-linear-gradient(top, #00ad00 0%,
#49d100 100%); /* FF3.6+ */
background: -webkit-gradient(linear, left top, left
bottom, color-stop(0%,#00ad00), colorstop(100%,#49d100)); /* Chrome,Safari4+ */
background: -webkit-linear-gradient(top, #00ad00
0%,#49d100 100%); /* Chrome10+,Safari5.1+ */
background: -o-linear-gradient(top, #00ad00
0%,#49d100 100%); /* Opera 11.10+ */
background: -ms-linear-gradient(top, #00ad00
0%,#49d100 100%); /* IE10+ */
background: linear-gradient(to bottom, #00ad00
0%,#49d100 100%); /* W3C */
filter: progid:DXImageTransform.Microsoft.gradient(
startColorstr='#00ad00',
endColorstr='#49d100',GradientType=0 ); /* IE6-8 */
Variation 3: Green CTA button layered with soft grey background div
17
o
Within master.min.css file, edited a.buttonLink,
a.buttonLinkWithImage
§ Removed existing background css:
background:url("https://dlvfa1owjktk.cloudfront.net/freedom/web/
844f58909f6c1a1ce71fc411b0ed70a0/img/buttonBGOut.png") 0 0
repeat-x;
§ Added new background gradient css:
background: background: #49d100;
background:
url(data:image/svg+xml;base64,PD94bWwgdmVyc2lvb
j0iMS4wIiA/Pgo8c3ZnIHhtbG5zPSJod…EiIGhlaWdod
D0iMSIgZmlsbD0idXJsKCNncmFkLXVjZ2ctZ2VuZX
JhdGVkKSIgLz4KPC9zdmc+);
background: -moz-linear-gradient(top, #49d100 0%,
#00ad00 100%);
background: -webkit-gradient(linear, left top, left
bottom, color-stop(0%,#49d100), colorstop(100%,#00ad00));
background: -webkit-linear-gradient(top, #49d100
0%,#00ad00 100%);
background: -o-linear-gradient(top, #49d100
0%,#00ad00 100%);
background: -ms-linear-gradient(top, #49d100
0%,#00ad00 100%);
background: linear-gradient(to bottom, #49d100
0%,#00ad00 100%);
18
filter: progid:DXImageTransform.Microsoft.gradient(
startColorstr='#49d100',
endColorstr='#00ad00',GradientType=0 );
o
Within master.min.css file, edited
a.buttonLink:hover,a.buttonLinkWithImage:hover
§ Removed existing button hover css:
background:url("https://dlvfa1owjktk.cloudfront.net/freedom/web/
844f58909f6c1a1ce71fc411b0ed70a0/img/buttonBGOver.png") 0
0 repeat-x
§ Added new background gradient hover css:
background: #00ad00; /* Old browsers */
/* IE9 SVG, needs conditional override of 'filter' to
'none' */
background:
url(data:image/svg+xml;base64,PD94bWwgdmVyc2lvb
j0iMS4wIiA/Pgo8c3ZnIHhtbG5zPSJodHRwOi8vd3d3L
nczLm9yZy8yMDAwL3N2ZyIgd2lkdGg9IjEwMCUiIG
hlaWdodD0iMTAwJSIgdmlld0JveD0iMCAwIDEgMSI
gcHJlc2VydmVBc3BlY3RSYXRpbz0ibm9uZSI+CiAgP
GxpbmVhckdyYWRpZW50IGlkPSJncmFkLXVjZ2ctZ
2VuZXJhdGVkIiBncmFkaWVudFVuaXRzPSJ1c2VyU
3BhY2VPblVzZSIgeDE9IjAlIiB5MT0iMCUiIHgyPSIw
JSIgeTI9IjEwMCUiPgogICAgPHN0b3Agb2Zmc2V0PS
IwJSIgc3RvcC1jb2xvcj0iIzAwYWQwMCIgc3RvcC1vc
GFjaXR5PSIxIi8+CiAgICA8c3RvcCBvZmZzZXQ9IjE
wMCUiIHN0b3AtY29sb3I9IiM0OWQxMDAiIHN0b3A
tb3BhY2l0eT0iMSIvPgogIDwvbGluZWFyR3JhZGllbn
Q+CiAgPHJlY3QgeD0iMCIgeT0iMCIgd2lkdGg9IjEiI
GhlaWdodD0iMSIgZmlsbD0idXJsKCNncmFkLXVjZ2
ctZ2VuZXJhdGVkKSIgLz4KPC9zdmc+);
background: -moz-linear-gradient(top, #00ad00 0%,
#49d100 100%); /* FF3.6+ */
background: -webkit-gradient(linear, left top, left
bottom, color-stop(0%,#00ad00), colorstop(100%,#49d100)); /* Chrome,Safari4+ */
background: -webkit-linear-gradient(top, #00ad00
0%,#49d100 100%); /* Chrome10+,Safari5.1+ */
background: -o-linear-gradient(top, #00ad00
0%,#49d100 100%); /* Opera 11.10+ */
background: -ms-linear-gradient(top, #00ad00
0%,#49d100 100%); /* IE10+ */
background: linear-gradient(to bottom, #00ad00
0%,#49d100 100%); /* W3C */
19
filter: progid:DXImageTransform.Microsoft.gradient(
startColorstr='#00ad00',
endColorstr='#49d100',GradientType=0 ); /* IE6-8 */
o
Within master.min.css file, edited #buttonDarkBG
§ Replaced existing background-color value #2b333d to be #eff1f3
Experiment 3: Purchase Page “Buy Now” CTA Button Variations
The last experiment run was an A/B test comparing one variation of the “Buy
Now” buttons on the purchase page with the control shown below.
•
Variation: Blue “Buy Now” buttons with adjusted Google Play button size.
20
o
Within master.min.css file, added the following new css:
§ Added new input css for new input id blue CTA button:
input[id='blueCTA'] {
background:url("https://dlvfa1owjktk.cloudfront.net/fr
eedom/web/844f58909f6c1a1ce71fc411b0ed70a0/img/but
tonBGOut.png") 0 0 repeat-x;
font-family:"Droid Sans", "Arial", sans-serif;
font-size:18px;
color:white;
padding:10px 30px;
text-decoration:none;
-moz-border-radius:5px;
-webkit-border-radius:5px;
-khtml-border-radius:5px;
border-radius:5px;
border:none;
position:relative;
top:-5px;
cursor: pointer;
}
input[id='blueCTA']:hover {
21
background:url("https://dlvfa1owjktk.cloudfront.net/fr
eedom/web/844f58909f6c1a1ce71fc411b0ed70a0/img/but
tonBGOver.png") 0 0 repeat-x;
cursor: pointer;
}
o
Replaced existing gplaygr.png file with the following to remove
translucent pixels in order to properly sync with new button size:
§ https://dl.dropboxusercontent.com/u/23344752/gplaygr.png
o
Updated html file on purchase page.
§ For each of the 5 “Buy Now” buttons, the 3rd input tag line within
each form was replaced with the following:
<input id="blueCTA" type="submit" name="submit"
value="Buy Now">
//The original line was replaced to look something like this with a
different source:
<input type="image" src="./Permanent_files/btn_buynow_LG.gif"
border="0" name="submit" alt="PayPal - The safer, easier way to pay
online!" style="padding-bottom:5px;">
Results
Experiment 1 was run from June 9th, 2014 to June 11th, 2014. Within that 3-day
window of time, there were 1,509 visitors and 60 total Paypal submissions. The tagline
variation “Block Digital Distractions” was a winning variation with a +102.71%
percentage improvement over the control tagline “Freedom for Productivity”. Below is an
exported table of the results for each conversion goal. See Appendix 1.1 for additional
figures.
22
Experiment 2 was run from June 17th, 2014 to June 20th, 2014. It started on a
Tuesday instead of a Monday because of technical difficulties importing the variations
into Visual Website Optimizer. Over four days, there were 483 visitors and 7 Paypal
submissions. Variation 1 of this experiment was disabled due to statistical significance of
-60.82% and -37.96% percentage improvement over the control. This experiment ended
up being disabled completely by Fred Stutzman on June 20th, 2014 because of the risk of
losing sales and traffic. Below is an exported table of the results for each conversion
goal. See Appendix 1.2 for additional figures.
23
Experiment 3 was run from June 23rd, 2014 to June 28th, 2014. Within the testing
period, there were 916 visitors and 203 Paypal submissions. The change to blue CTA
buttons reached success with a +26.04% percentage improvement over the control Paypal
CTA buttons. On June 25th, 2014, Visual Website Optimizer launched a new platform for
their software tool. This new platform brought along a new interface, which is why the
results below differ in style. See Appendix 1.3 for additional figures.
24
Discussion
Creating new variations can be a difficult task because the experimenter has to
make blind guesses to understand why the current design elements are not producing high
enough conversion goals. Even the results of previous A/B tests lead the experimenter to
make more educated guesses. I will try to explain and analyze each experiment as best I
can, while remembering that any reasoning behind statistical significance cannot be
proven.
Experiment 1 produced variation 2 as a statistically significant variation with a
+120.71% improvement of the Paypal Submissions conversion goal. Variation 1 and 3
both showed above a +60% increase of Paypal Submissions, although they did not reach
significance. All three variations showed positive results for the other conversion goals:
Purchase Click and Engagement. I believe the success of Experiment 1 was because the
tagline variations better explain what Freedom does more directly than the control
25
tagline, which proclaims somewhat vaguely that Freedom can be used to increase
productivity.
Experiment 2 had minimal positive results for Paypal submissions with variation
2 and 3 having a few conversions each. For reasons other than not reaching statistical
significance, the results are not very useful. First, the control had 0 Paypal submissions,
which makes the variations hard to compare with. Second, the change was made to the
landing page and Paypal submissions occur on the purchase page (after being directed
through the Paypal website). Lastly, with the Engagement conversion goal showing an
-8% or more decrease in improvement for the two enabled variations, the positive results
of the Paypal Submissions does not seem like they were caused by variations. Although
in this experiment Paypal Submissions is the most important conversion goal, the result
of the other two conversion goals decrease the value of the Paypal Submissions success.
This is a perfect example for why multiple conversion goals should be run
simultaneously.
By changing the landing page CTA buttons in Experiment 2, I hoped to maintain
design consistency. New variations were created to be more closely aligned with the
existing color palette used on the website. The color choices and removal of the
background border may have taken user attention and focus away from the CTA button.
Although the variations may look better visually, the prominence of the CTA buttons has
more influence and a higher priority. It would have been interesting to let the experiment
run for a longer period of time to see extended results.
Experiment 3 had only one variation running against the control, as well as a
slightly different conversion goal because the test ran on the purchase page instead of the
26
landing page. The variation reached conclusive results by improving the Paypal
Submissions conversion goal. The winning variation of the CTA button maintained
design consistency of colors and button design style across the website compared to the
control CTA buttons that resembled out-of-date Paypal “Buy Now” buttons. The
consistency of CTA buttons work in this experiment over the previous, because users
have already made the decision to purchase Freedom and therefore, the CTA buttons do
not require as much attention. There was also a 4.85% improvement of Engagement over
the control.
The third conversion goal for Experiment 3, the Google Play Click saw a
-17.3% decrease in improvement. This conversion goal did not carry as much weight as
the other two conversion goals in this experiment, because the Google Play Click only
tracked clicks to one of the six CTA buttons on the purchase page. This decrease in
improvement may be because the blue CTA “Buy Now” buttons pulled attention away
from the “Google Play” button and focused more attention on themselves, thus increased
overall user conversions (Paypal Submissions).
Limitations
There are a few general limitations to standard A/B tests that apply to this
experiment. A/B tests are normally focused on short-term optimization, meaning that a
variation can prove statistically significant over a short period, but over a longer period
the control may prove to be more successful. As Nielsen states, if you are “basing your
decisions on short-term numbers…they can lead you astray” (2005). The winning
variation from Experiment 1 should be tested continually over an extended period of time
27
to prove that it does reach statistical significance over the control. The same logic can be
applied to the disabled variation in Experiment 2, but the sales of the company may be
heavily affected in the process of trying to prove significance over the extended
timeframe.
Another major limitation already touched on is that A/B tests do not provide any
understanding or reasoning behind why variations are effective. If variation A is a winner
over variation B, it may increase the tested conversion goals, but may not be helpful for
transitioning other parts of the website in the future. Nielsen compares the narrowness of
A/B testing with an “open-ended method like user testing, where users often reveal
stumbling blocks you never would have expected” (2005). Additional types of testing
may be needed to better understand the user and meaning behind successful and
unsuccessful variations.
Conclusion
By focusing on specific design elements and measuring the impact of changes by
tracking conversion goals, this study has proved 2 of 3 experiments statistically
successful. Although the experimenter does not venture any strong behavioral insights as
to why the variations were successful, A/B tests have and can continue to effectively
increase sales at 80 Percent Solutions. A/B tests are most useful for optimizing tracked
conversion rates and offer an additional benefit of further educating the experimenter
about the product being tested. Other than improved conversion rates, A/B tests can lead
to better-quality decisions and more successful tests in the future.
28
Areas for further research
Follow-up experiments to this study would be very beneficial, as they can provide
more data to strengthen winning variation significance. Outside of following up with A/B
split testing, other methods of increasing usefulness include using tracking, surveys, and
user personas (Cleanthous, 2014). If possible, a longer testing period could certify that
the negative results from Experiment 2 are repeatable. However, this validation may not
immediately occur after an additional timeframe, because as Chopra states “conversion
rate constantly changes…because your traffic mix changes from day to day” (2012).
Testing new landing page CTA button variations may eventually provide further support
one way or the other, but at the risk of losing sales if the variations produce negative
results.
In Experiment 2, I believe the proper balance of design consistency and user
attention is currently missing within the front page CTA buttons. This level of balance
may be present within Experiment 3 from conclusive results, but follow-up experiments
should still be run to increase significance. Further research should be done looking at the
causal relationship between the findings and established behavioral research relating to
colors, style, and attraction.
29
References
Aagaard, M. (2013, May 23). How To Design Call to Action Buttons That Convert.
Retrieved June 2, 2014, from http://unbounce.com/conversion-rateoptimization/design-call-to-action-buttons/
Adlakha, K. (2013, December 10). Designing Call-to-Action Buttons that Customers
Click. Retrieved June 2, 2014, from
http://www.affinityexpress.com/2013/12/10/designing-call-action-buttonscustomers-click
Babbar, S. (2014, March 25). 9 Insanely Simple Ways to Optimize Your PPC Landing
Page. Retrieved June 25, 2014, from https://vwo.com/blog/ways-to-optimize-ppclanding-page
Bakkestuen, M. (2013, February 12). What Makes a Great Call to Action? Retrieved June
25, 2014, from http://blog.acumium.com/2013/02/12/what-makes-a-great-call-toaction/
Benun, I. (2005, June). How to Create a Winning Tagline. Retrieved June 2, 2014, from
http://www.usabilityprofessionals.org/upa_publications/upa_voice/volumes/2005/
june/winning_tagline.html
Birch, N. (2013, June 6). Clever and Conspicuous Tagline is a First Step to a Good
Acquaintance. Retrieved June 2, 2014, from http://designmodo.com/taglinetypography/
30
Bustos, L. (2011, January 20). Is Your PayPal Button Hurting Conversions? Retrieved
June 5, 2014, from http://www.getelastic.com/pay-pal-fud/
Cardello, J. (2014, April 20). Define Stronger A/B Test Variations Through UX Research
Retrieved June 5, 2014, from http://www.nngroup.com/articles/ab-testing-and-uxresearch/
Chapman, C. (n.d.). “Call To Action” Buttons: Guidelines, Best Practices And Examples.
Retrieved June 5, 2014, from http://www.hongkiat.com/blog/call-to-actionbuttons-guidelines-best-practices-and-examples/
Chartrand, J. (n.d.). How to Create a Rock-Solid Tagline That Truly Works. Retrieved
June 5, 2014, from http://www.copyblogger.com/create-a-tagline/
Chawla, S. (2013, April 18). Call to Action Buttons: The Ultimate Guide on Which Ones
Convert and Why. Retrieved June 25, 2014, from https://vwo.com/blog/call-toaction-buttons-ultimate-guide/
Chopra, P. (2010, January 1). How reliable are your split test results? Retrieved June 25,
2014, from https://vwo.com/blog/how-reliable-are-your-split-test-results/
Chopra, P. (2010, January 26). What you really need to know about mathematics of A/B
split testing. Retrieved June 25, 2014, from https://vwo.com/blog/what-youreally-need-to-know-about-mathematics-of-ab-split-testing/
Chopra, P. (2011, October 18). ‘Try demo’ or ‘Buy now’: A/B testing finds which button
increased clickthroughs by 47%. Retrieved June 25, 2014, from
https://vwo.com/blog/testing-call-to-action-buttons/
Chopra, P. (2012, February 23). 5 most common A/B testing misconceptions. Retrieved
June 5, 2014, from https://vwo.com/blog/ab-testing-misconceptions/
31
Cleanthous, A. (2014, March 31). 10 Guidelines To Improve The Effectiveness Of A/B
Testing. Retrieved June 1, 2014, from http://usabilitygeek.com/10-guidelinesimprove-effectiveness-a-b-testing/
Cookson, G. (2011, July 28). What makes a good call to action button? Best practice tips.
Retrieved June 25, 2014, from http://blog.respondhq.com/call-to-action/whatmakes-a-good-call-to-action-button-best-practice-tips/
DeMers, J. (2013, July 25). The Definitive Guide to Crafting Winning Calls to Action in
Your Content. Retrieved June 25, 2014, from
http://www.huffingtonpost.com/jayson-demers/the-definitive-guide-toc_b_3648475.html
E-Commerce. (n.d.). In Wikipedia. Retrieved June 9, 2014, from
http://en.wikipedia.org/wiki/E-Commerce
Erin, A. (2014, March 11). Rethink The Psychology of Your Calls to Action. Retrieved
June 5, 2014, from http://sazbean.com/category/strategy/usability-strategy/
Fishleder, A. (2014, March 20). The Call To Action: Being at the Right Place at the Right
Time. Retrieved June 5, 2014, from http://www.maximizer.com/blog/the-call-toaction-being-at-the-right-place-at-the-right-time/
Gardner, O. (2009, December 3). The Future of the Marketing Call-To-Action Button:
Big, Shiny & Animated. Retrieved June 5, 2014, from
http://unbounce.com/conversion-rate-optimization/the-future-of-marketing-callto-action-buttons/
Geisheker, P. (2013, July 29). The Truth About the Best and Worst Call to Action Button
Colors for Your Website. Retrieved June 5, 2014, from
32
https://smallbusiness.yahoo.com/advisor/truth-best-worst-call-action-buttoncolors-website-212506540.html
Gelberg, J. (2010, March 15). How to Choose a Tag Line That Delivers Your Company’s
Message Effectively. Retrieved June 5, 2014, from
http://www.bluefountainmedia.com/blog/tag-line-delivers-message/
Gube, J. (2009, October 13). Call to Action Buttons: Examples and Best Practices.
Retrieved June 10, 2014, from
http://www.smashingmagazine.com/2009/10/13/call-to-action-buttons-examplesand-best-practices/
Hamill, D. (2009, March 19). Good Call-To-Action Buttons. Retrieved June 10, 2014,
from http://www.uxbooth.com/articles/good-call-to-action-buttons/
Harrison, K. (n.d.). Ensure your home page tagline tells what you actually do. Retrieved
June 10, 2014, from http://www.cuttingedgepr.com/articles/name-your-homepage-tagline.asp
How many types of conversion goals are present in VWO? (n.d.). Retrieved June 5, 2014,
from http://visualwebsiteoptimizer.com/knowledge/how-many-types-ofconversion-goals-are-present-in-vwo/
Kohavi, R., Henne, R. M., & Sommerfield, D. (2007, August). Practical guide to
controlled experiments on the web: listen to your customers not to the hippo. In
Proceedings of the 13th ACM SIGKDD international conference on Knowledge
discovery and data mining (pp. 959-967). ACM.
33
Krug, S. (2014, April 30). Steve Krug on Why Nothing Beats a Good Tagline!™.
Retrieved June 10, 2014, from
http://www.peachpit.com/articles/article.aspx?p=2209312
Linstroth, D. (n.d.). Test Your Message To Build The Best Landing Page. Retrieved June
5, 2014, from http://blog.launchrock.co/test-your-message-to-build-the-bestlanding-page/
Moth, D. (2013, May 30). Seven tips for designing an effective ecommerce call-to-action
Retrieved June 4, 2014, from https://econsultancy.com/blog/62829-seven-tips-fordesigning-an-effective-ecommerce-call-to-action
Nielsen, J. (2001, July 22). Tagline Blues: What's the Site About? Retrieved June 4,
2014, from http://www.nngroup.com/articles/tagline-blues-whats-the-site-about/
Nielsen, J. (2003, November 10). The Ten Most Violated Homepage Design Guidelines.
Retrieved June 4, 2014, from http://www.nngroup.com/articles/most-violatedhomepage-guidelines/
Nielsen, J. (2005, August 15). Putting A/B Testing in Its Place. Retrieved June 7, 2014,
from http://www.nngroup.com/articles/putting-ab-testing-in-its-place/
Nielsen, J. (2010, March 22). Scrolling and Attention. Retrieved July 7, 2014, from
http://www.nngroup.com/articles/scrolling-and-attention/
Nielsen, J. (2013, November 24). Conversion Rates. Retrieved June 7, 2014, from
http://www.nngroup.com/articles/conversion-rates/
Niggulis, O. (2013, January 16). Which Color Converts The Best? Retrieved June 10,
2014, from http://conversionxl.com/which-color-converts-the-best/#.
34
Olyslager, P. (n.d.). Call To Action Buttons and the Psychology of Color. Retrieved June
10, 2014, from http://www.paulolyslager.com/call-to-action-buttons-psychologycolor/
Overgoor, J. (2014, May 27). Experiments at Airbnb. Retrieved June 10, 2014, from
http://nerds.airbnb.com/experiments-at-airbnb/
Raby, J. (2013, October 22). Call To Actions – What Are They and Why You MUST Use
Them on Your Website. Retrieved June 10, 2014, from
http://www.combustible.ca/en/2013/10/call-to-actions-what-are-they-and-whyyou-must-use-them-on-your-website/
Robinson, N. (n.d.). Crafting the Perfect Buy Button. Retrieved June 5, 2014, from
http://www.trinityinsight.com/resources/crafting-perfect-buy-button/
Ryan, P. (2014, February 14). Web Site Usability and Conversion Rates. Retrieved June
5, 2014, from http://www.ryandesignstudio.com/news/2014/web-site-usabilityconversion-rates/
Sharafeddine, L. (2012, October 4). Call To Action Button Best Practices: Internal
Factors. Retrieved June 5, 2014, from http://blog.market8.net/b2b-web-designand-inbound-marketing-blog/Call-To-Action-Button-Best-Practices-InternalFactors
Simmons, G. (2014, April 1). Ten Timely Tips for Terrific Title Tags. Retrieved June 5,
2014, from http://searchenginewatch.com/article/2348627/Ten-Timely-Tips-forTerrific-Title-Tags
35
Smith, J. (2014, June 6). How to Use the Psychology of Color to Increase Website
Conversions. Retrieved July 7, 2014, from
http://blog.kissmetrics.com/psychology-of-color-and-conversions/
Stutzman, F. (n.d.). Praise for Freedom. Retrieved June 26, 2014, from
https://macfreedom.com/press
Tagline. (n.d.). Retrieved June 26, 2014, from http://www.merriamwebster.com/dictionary/tagline
Thomason, L. (2002, August). Promotion Tip: Explain Your Site With A Tagline.
Retrieved June 5, 2014, from
http://www.netmechanic.com/news/vol5/promo_no15.htm
Tseng, A. (2011, April 25). Why Users Click Right Call to Actions More Than Left
Ones. Retrieved June 5, 2014, from http://uxmovement.com/buttons/why-usersclick-right-call-to-actions-more-than-left-ones/
Vertommen, K. (2012, August 8). How to Reduce Your Website’s Bounce Rate.
Retrieved June 25, 2014, from http://www.instantshift.com/2012/08/08/how-toreduce-your-websites-bounce-rate/
Walker, T. (2010, February 24). 25 Examples of Convincing Call-to-Action Buttons.
Retrieved June 25, 2014, from http://designshack.net/articles/inspiration/25examples-of-convincing-call-to-action-buttons/
Weber, H. (2012, October 10). The new ‘Pay with Stripe’ button makes online payments
feel less like a Darth Vader death grip. Retrieved June 25, 2014, from
http://thenextweb.com/dd/2012/10/10/the-new-pay-with-stripe-button-makesonline-payments-not-feel-like-death/
36
Weil, D. (2002, December 3). Top Tips for Taglines That Engage First-Time Visitors.
Retrieved June 25, 2014, from http://www.marketingprofs.com/2/weil3.asp
What does "engagement on the page" means? (2013, May 16). Retrieved June 25, 2014,
from http://visualwebsiteoptimizer.com/knowledge/what-does-engagement-onthe-page-means/
What is Visual Website Optimizer and how is it used? (2012, June 7). Retrieved June 25,
2014, from http://visualwebsiteoptimizer.com/knowledge/what-is-visual-websiteoptimizer-and-how-is-it-used/
37
Appendix
The figures below show results of the experiments in more detail.
1.1
38
Paypal Submission
Purchase Click
Engagement
39
1.2
40
Paypal Submission
Purchase Click
Engagement
41
1.3
42
Paypal Submission
Google Play Click
Engagement
‐‐‐ Aggregate Data ‐‐‐
Engagement
Engagement
Engagement
Engagement
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Purchase Click
Purchase Click
Purchase Click
Purchase Click
Variation Name
Control
Control
Control
Eliminate Digital Distractions
Eliminate Digital Distractions
Eliminate Digital Distractions
Block Digital Distractions
Block Digital Distractions
Block Digital Distractions
Freedom From Digital Distractions
Freedom From Digital Distractions
Control
Eliminate Digital Distractions
Block Digital Distractions
Freedom From Digital Distractions
Control
Eliminate Digital Distractions
Block Digital Distractions
Freedom From Digital Distractions
Control
Eliminate Digital Distractions
Block Digital Distractions
Freedom From Digital Distractions
Campaign Name: Josh Rice Test 1
Campaign Id: 38
‐‐‐Daywise Data‐‐‐
Goal
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
20.86%
24.32%
23.41%
23.10%
2.41%
4.59%
4.88%
3.94%
11.23%
13.78%
15.12%
12.96%
2.69% ‐
2.86%
2.68%
2.86%
1.01% ‐
1.39%
1.36%
1.32%
2.09% ‐
2.29%
2.26%
2.28%
16.63%
12.27%
10.75%
90.93%
102.71%
63.88%
22.74%
34.66%
15.39%
‐
‐
‐
87%
81%
77%
95%
97%
88%
85%
95%
76%
Date
Visitors Conversions Conversion Rate
9/6/2014
67
13
19.40%
10/6/2014
194
38
19.59%
11/6/2014
113
27
23.89%
9/6/2014
73
13
17.81%
10/6/2014
189
48
25.40%
11/6/2014
108
29
26.85%
9/6/2014
80
25
31.25%
10/6/2014
224
47
20.98%
11/6/2014
106
24
22.64%
9/6/2014
68
15
22.06%
10/6/2014
173
40
23.12%
374
370
410
355
374
370
410
355
374
370
410
355
78
90
96
82
9
17
20
14
42
51
62
46
Engagement
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Purchase Click
Purchase Click
Purchase Click
Purchase Click
Purchase Click
Purchase Click
Purchase Click
Purchase Click
Purchase Click
Purchase Click
Purchase Click
Purchase Click
Freedom From Digital Distractions
Control
Control
Control
Eliminate Digital Distractions
Eliminate Digital Distractions
Eliminate Digital Distractions
Block Digital Distractions
Block Digital Distractions
Block Digital Distractions
Freedom From Digital Distractions
Freedom From Digital Distractions
Freedom From Digital Distractions
Control
Control
Control
Eliminate Digital Distractions
Eliminate Digital Distractions
Eliminate Digital Distractions
Block Digital Distractions
Block Digital Distractions
Block Digital Distractions
Freedom From Digital Distractions
Freedom From Digital Distractions
Freedom From Digital Distractions
11/6/2014
9/6/2014
10/6/2014
11/6/2014
9/6/2014
10/6/2014
11/6/2014
9/6/2014
10/6/2014
11/6/2014
9/6/2014
10/6/2014
11/6/2014
9/6/2014
10/6/2014
11/6/2014
9/6/2014
10/6/2014
11/6/2014
9/6/2014
10/6/2014
11/6/2014
9/6/2014
10/6/2014
11/6/2014
114
67
194
113
73
189
108
80
224
106
68
173
114
67
194
113
73
189
108
80
224
106
68
173
114
27
3
4
2
3
6
8
4
9
7
2
6
6
6
27
9
7
24
20
17
27
18
10
20
16
23.68%
4.48%
2.06%
1.77%
4.11%
3.17%
7.41%
5%
4.02%
6.60%
2.94%
3.47%
5.26%
8.96%
13.92%
7.96%
9.59%
12.70%
18.52%
21.25%
12.05%
16.98%
14.71%
11.56%
14.04%
‐‐‐ Aggregate Data ‐‐‐
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Engagement
Engagement
Engagement
Engagement
Purchase Click
Purchase Click
Purchase Click
Purchase Click
Variation Name
Control
Control
Control
Control
Light Background (1)
Light Background (1)
Green Button (2)
Green Button (2)
Green Button (2)
Green Button (2)
Green Button/White BG (3)
Control
Light Background (1)
Green Button (2)
Green Button/White BG (3)
Control
Light Background (1)
Green Button (2)
Green Button/White BG (3)
Control
Light Background (1)
Green Button (2)
Green Button/White BG (3)
Campaign Name: Josh Rice Test 2
Campaign Id: 45
‐‐‐Daywise Data‐‐‐
Goal
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
0%
0%
3.03%
2.07%
22.39%
13.89%
20.45%
18.62%
14.18%
5.56%
12.88%
13.79%
Date
17/06/14
18/06/14
19/06/14
20/06/14
17/06/14
18/06/14
17/06/14
18/06/14
19/06/14
20/06/14
17/06/14
0%
0%
0%
6%
35%
22%
2%
38%
46%
Conversion Conversion Rate
0
0%
0
0%
0
0%
0
0%
0
0%
0
0%
1
3.03%
2
5.13%
1
2.27%
0
0%
2
6.45%
0% ‐
‐
0%
0%
1.91%
0%
1.51%
0%
4.61% ‐
‐
5.22% ‐37.96%
4.49%
‐8.64%
4.14% ‐16.83%
3.86% ‐
‐
3.46% ‐60.82%
3.73%
‐9.17%
3.67%
‐2.72%
Visitors
33
43
37
21
37
35
33
39
44
16
31
134
72
132
145
134
72
132
145
134
72
132
145
0
0
4
3
30
10
27
27
19
4
17
20
Paypal Submission
Paypal Submission
Paypal Submission
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Purchase Click
Purchase Click
Purchase Click
Purchase Click
Purchase Click
Purchase Click
Purchase Click
Purchase Click
Purchase Click
Purchase Click
Purchase Click
Purchase Click
Purchase Click
Purchase Click
Purchase Click
Green Button/White BG (3)
Green Button/White BG (3)
Green Button/White BG (3)
Control
Control
Control
Control
Light Background (1)
Light Background (1)
Light Background (1)
Green Button (2)
Green Button (2)
Green Button (2)
Green Button (2)
Green Button/White BG (3)
Green Button/White BG (3)
Green Button/White BG (3)
Green Button/White BG (3)
Control
Control
Control
Control
Light Background (1)
Light Background (1)
Light Background (1)
Green Button (2)
Green Button (2)
Green Button (2)
Green Button (2)
Green Button/White BG (3)
Green Button/White BG (3)
Green Button/White BG (3)
Green Button/White BG (3)
18/06/14
19/06/14
20/06/14
17/06/14
18/06/14
19/06/14
20/06/14
17/06/14
18/06/14
19/06/14
17/06/14
18/06/14
19/06/14
20/06/14
17/06/14
18/06/14
19/06/14
20/06/14
17/06/14
18/06/14
19/06/14
20/06/14
17/06/14
18/06/14
19/06/14
17/06/14
18/06/14
19/06/14
20/06/14
17/06/14
18/06/14
19/06/14
20/06/14
41
41
32
33
43
37
21
37
35
0
33
39
44
16
31
41
41
32
33
43
37
21
37
35
0
33
39
44
16
31
41
41
32
0
0
1
9
13
4
4
7
2
1‐
7
7
8
5
8
6
5
8
5
7
3
4
0
3
1‐
4
4
7
2
4
5
4
7
0%
0%
3.13%
27.27%
30.23%
10.81%
19.05%
18.92%
5.71%
21.21%
17.95%
18.18%
31.25%
25.81%
14.63%
12.20%
25%
15.15%
16.28%
8.11%
19.05%
0%
8.57%
12.12%
10.26%
15.91%
12.50%
12.90%
12.20%
9.76%
21.88%
‐‐‐ Aggregate Data ‐‐‐
Engagement
Engagement
Paypal Submission
Paypal Submission
Google Play Click
Google Play Click
Variation Name
Control
Control
Control
Control
Control
Control
Control
Control
Control
Control
Control
Control
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
Control
Blue Buttons
Control
Blue Buttons
Control
Blue Buttons
Campaign Name: Josh Rice Test #3
Campaign Id: 47
‐‐‐Daywise Data‐‐‐
Goal
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
55.06%
59.93%
20.24%
24.04%
5.18%
4.69%
2.18% ‐
2.20%
1.76% ‐
1.92%
0.97% ‐
0.95%
8.84%
18.82%
‐9.48%
‐
‐
‐
98%
97%
32%
Date
Visitors
Conversion Conversion Rate
23/06/14
10
3
30%
24/06/14
111
71
63.96%
25/06/14
109
61
55.96%
26/06/14
90
55
61.11%
27/06/14
85
51
60%
28/06/14
63
27
42.86%
29/06/14
72
32
44.44%
30/06/14
75
41
54.67%
1/7/2014
83
40
48.19%
2/7/2014
81
46
56.79%
3/7/2014
62
37
59.68%
4/7/2014
9
4
44.44%
23/06/14
9
6
66.67%
24/06/14
110
62
56.36%
25/06/14
105
58
55.24%
26/06/14
80
51
63.75%
27/06/14
90
57
63.33%
850
811
850
811
850
811
468
486
172
195
44
38
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Engagement
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Paypal Submission
Google Play Click
Google Play Click
Google Play Click
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
Control
Control
Control
Control
Control
Control
Control
Control
Control
Control
Control
Control
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
Control
Control
Control
28/06/14
29/06/14
30/06/14
1/7/2014
2/7/2014
3/7/2014
4/7/2014
23/06/14
24/06/14
25/06/14
26/06/14
27/06/14
28/06/14
29/06/14
30/06/14
1/7/2014
2/7/2014
3/7/2014
4/7/2014
23/06/14
24/06/14
25/06/14
26/06/14
27/06/14
28/06/14
29/06/14
30/06/14
1/7/2014
2/7/2014
3/7/2014
4/7/2014
23/06/14
24/06/14
25/06/14
54
55
70
75
77
79
7
10
111
109
90
85
63
72
75
83
81
62
9
9
110
105
80
90
54
55
70
75
77
79
7
10
111
109
35
27
45
49
40
49
7
1
31
21
17
16
6
6
23
19
15
16
1
2
28
21
23
24
13
12
21
9
18
20
4
1
8
6
64.81%
49.09%
64.29%
65.33%
51.95%
62.03%
100%
10%
27.93%
19.27%
18.89%
18.82%
9.52%
8.33%
30.67%
22.89%
18.52%
25.81%
11.11%
22.22%
25.45%
20%
28.75%
26.67%
24.07%
21.82%
30%
12%
23.38%
25.32%
57.14%
10%
7.21%
5.50%
Google Play Click
Google Play Click
Google Play Click
Google Play Click
Google Play Click
Google Play Click
Google Play Click
Google Play Click
Google Play Click
Google Play Click
Google Play Click
Google Play Click
Google Play Click
Google Play Click
Google Play Click
Google Play Click
Google Play Click
Google Play Click
Google Play Click
Google Play Click
Google Play Click
Control
Control
Control
Control
Control
Control
Control
Control
Control
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
Blue Buttons
26/06/14
27/06/14
28/06/14
29/06/14
30/06/14
1/7/2014
2/7/2014
3/7/2014
4/7/2014
23/06/14
24/06/14
25/06/14
26/06/14
27/06/14
28/06/14
29/06/14
30/06/14
1/7/2014
2/7/2014
3/7/2014
4/7/2014
90
85
63
72
75
83
81
62
9
9
110
105
80
90
54
55
70
75
77
79
7
4
3
2
6
1
4
6
3
0
0
3
4
5
2
5
1
3
8
2
4
1
4.44%
3.53%
3.17%
8.33%
1.33%
4.82%
7.41%
4.84%
0%
0%
2.73%
3.81%
6.25%
2.22%
9.26%
1.82%
4.29%
10.67%
2.60%
5.06%
14.29%