White Paper
Retail Best Practices
Performance Testing Retail Web
and Mobile Properties
Table of Contents
Executive Summary..................................................................................................... 3
1. eCommerce Characteristics.................................................................................. 4
2. What is an Effective Test Strategy......................................................................... 5
3. Do I Have a Successful eCommerce Site Yet?..................................................... 5
4. Testing for Success................................................................................................ 6
5. Testing Tactics...................................................................................................... 10
6. Retail Examples.................................................................................................... 12
7.Security................................................................................................................. 14
Appendix: CloudTest Performance Data Q&A......................................................... 16
About SOASTA, Inc................................................................................................... 17
SOASTA
RETAIL BEST PRACTICES WHITE PAPER | 2
Retail Best Practices:
Performance Testing
Six of the top 10 US retailers trust SOASTA to execute
load and performance tests on their eCommerce sites.
Retailers competing in the high-pressure world of eCommerce have many challenges: not only do
they need to provide an interesting and compelling shopping experience; they need to make that
experience fast and reliable. Shoppers have proven to be discriminating and can easily move from
one online store to another.
A company’s online presence has become as important as a physical store, and for some the online
store is their only sales channel.
Many of those responsible for the performance of retail web sites have come to realize that external
testing against the production infrastructure is the best way to get a true picture of capacity and
performance in the real world. Testing in production is the only way to ensure that online applications will perform as expected. Of course, this does not obviate the need or eliminate the benefits
of testing in a lab environment as well; and it’s important to have continuity between the two.
So, how can a company build confidence that their online store will provide a great shopping experience and withstand the traffic spikes caused by seasonal upswings, events and promotions? How
should a retailer measure success?
SOASTA
RETAIL BEST PRACTICES WHITE PAPER | 3
Third-party content can add functionality or information to
a site, but it also adds a layer of traffic and complexity that
you may or may not control.
1. eCommerce Characteristics
Most retailers have brick and mortar stores to maintain, staff, and ensure are run profitably, in addition to an online site to complement their physical presence. The look and feel of the online site
typically mirrors the target market of their retail stores: discount, high-end woman’s shop, books
and gifts, big box electronics, etc.
As a result, balancing performance with the rich content often used to capture a specific design
goal can be a struggle. Marketing departments spend millions of dollars promoting events intended to drive users to the site. Web developers build compelling sites to enable a visitor to easily
navigate between ‘departments’ and effectively sift through a range of size, color, style and other
product attributes. While these content-rich sites are visually attractive and more easily understood
(hopefully!), they can also cause performance issues as traffic grows.
Third-party content is quite common on eCommerce sites, including ads, cross-promotions, and
relevant consumer information. This content can add functionality or information to a site, but it also
adds a layer of traffic and complexity that you may or may not control. Understanding the impact of
this content is critical to managing the overall performance of the site. And, of course, the goal of
any eCommerce site is to sell products. Back-end and third-party systems for payment processing,
authorization, inventory checking and more can also add performance challenges.
Retailers also know that site performance directly relates to revenue. Aberdeen Group found that
50% of companies lost revenue opportunities because of poorly performing applications and 32%
experienced damage to their brand reputation. With most users more than willing to share their experience on Facebook or Twitter, the news of a poorly performing site can spread around the globe
in minutes.
Without a doubt, much is riding on the performance of an eCommerce site. A retailer can position
themselves for success by following SOASTA’s proven Load and Performance Methodology, which
is described on page 5.
SOASTA
RETAIL BEST PRACTICES WHITE PAPER | 4
2. What is an Effective eCommerce Test Strategy?
“
Creating a successful strategy is key to a successful performance engagement. It begins with
understanding how your users interact with your eCommerce site. For example, the following
information is core to defining the test engagement strategy:
• Main business processes (flows) followed by site visitors
• Average time a user stays on the site
• Average percentage of users that complete a purchase vs. users who browse only
• Abandonment ratios and where in the process users abandon
• Average and peak number of concurrent users/hour
• Average and peak page views per minute/hour
• Average and peak orders placed per minute/hour
• Traffic pattern differences for the above metrics for specific events such as Black Friday or Cyber
Monday
• Geographic regions of the site traffic’s source
• Percentage of traffic that comes from mobile devices, the types of devices, and the differences in
user flows and the impact on the above metrics
• If a CDN is used, percentage of content that is served from the CDN
This information helps to define a successful performance testing strategy and drives how your test
will be architected and executed.
3. Do I Have a Successful eCommerce Site Yet?
IF A RETAILER WANTS TO BE
IN THE TOP 25 CATCHPOINT
RANK THEY NEED TO HAVE
PAGE LOAD TIMES OF TWO
SECONDS OR LESS.
One of the toughest questions to answer is ‘Is the site ready?’ That question becomes infinitely
easier when you have a solid strategy in place. eCommerce sites have specific key performance
indicators (KPIs) that help a retailer measure their site’s performance:
• Orders per Minute – for an eCommerce site, orders completed in a period of time is the gold
standard KPI. Orders directly translate to revenue. SOASTA’s CloudTest® platform can track
exactly how many orders are successfully completed each minute (or over any period of time) as
well as how many had errors.
• Page Views per Hour or Minute – Customers can only complete orders if they can effectively
use the site. Ensuring that the eCommerce site can serve web content as users search, compare
and interact is critical.
• Sessions per Hour – Sessions are individual users or systems interacting with the site. Ensuring
that those users can establish and maintain a session for the duration of their interaction is very
important.
• Errors – It may seem obvious, but keeping track of both the overall error rate and the type of
errors you are receiving are critical. Not all errors are created equally.
• Average Response Time – Understanding how long, on average, pages and page assets take to
be served is important for uncovering potential bottlenecks. It’s also the basis for many customers measuring the utility of an eCommerce site. SOASTA CloudTest shows the Average Response
Time for an entire test scenario, transactions, individual pages, and (unlike other performance
applications) provides information about each individual asset that makes up a page. It even
provides the ability to split out Average Response Times by domain so it’s easy to see if a thirdparty asset, a CDN or specific data center is impacting performance.
• 90th Percentile Response Time – This provides an even finer level of granularity when it comes
to response time. 90th Percentile removes the slowest 10% response times. This eliminates
timeouts (which can average 120 seconds) while giving a true indication of the response for 90%
of the users.
SOASTA recommends that a retailer striving for good site performance have an Average Response
Time of less than 3 seconds with a 90th Percentile of less than 2.75 seconds. A retailer wanting
to ensure their site is among the leaders in performance needs to know how fast is fast enough.
Catchpoint Systems ranks the site performance of the top 50 US retailers each year. If we consider
their rankings for 2011, we note that if a retailer wants to be in the top 25 they need to have page
load times of 2 seconds or less on Cyber Monday – one of the heaviest traffic days of the year.
SOASTA
RETAIL BEST PRACTICES WHITE PAPER | 5
This indicates that even an additional second of delay can put you at the back of the pack. It is no
accident that six of the top 10 US retailers have worked with SOASTA to ensure they have worldclass performance every single day.
“
It’s worth noting there are different expectations for different types of pages. A user browsing the
site will expect lightning fast response times while the initiation of a video or the final step in a
checkout process is generally expected to take a bit longer. If an eCommerce site can maintain a
target level of performance while testing with 150-200% of expected peak load, a retailer can go
into any marketing event or holiday season with increased confidence.
3.1 What to Watch for When Testing
Understanding what makes an eCommerce site fast or slow helps to focus the testing effort. Many
eCommerce sites are quite complex and made up of several different components and application
tiers. It’s important to understand each of these components and how they interact with each other.
Some of the common areas to focus on while testing are:
Application Issues – There is no such thing as perfect code. Watch for inefficient code, synchronization issues, garbage collection, memory leaks, and application dead locks.
Database Performance – This is the core of performance. Watch for locking and contention, missing indexes, inefficient queries, memory management, connection management and unmanaged
growth of data.
Configuration Settings – The default settings are rarely optimal. Check for differences between
environments and understand the tuning options and best practices for the various devices in the
architecture.
Load Balancing – Use hardware efficiently. Look for algorithms that are not optimized and underutilized features and capabilities.
EVEN AN ADDITIONAL
SECOND OF DELAY CAN
PUT YOU AT THE BACK OF
THE ECOMMERCE PACK
Connectivity – Communication is vital. Ensure that systems can communicate with a minimum of
latency, that the firewall has sufficient capacity, that the system is optimized for mobile networks,
that the DNS routing is correct, and that CDN caching is optimized.
Bandwidth – Can you be reached? Ensure that the bandwidth pipe is big enough for the traffic.
Review the content making up the pages. Rich content can mean big data and bandwidth requirements. Verify that the site can support different connection types/speeds including mobile devices.
Architecture – Match the car to the engine. Watch for unbalanced tiers, mismatched technology
choices or a dead-end scalability path.
I
Third-Party Services – You are only as fast as your slowest link. Ensure that analytics and tracking,
payment systems, aggregated content, social networks or CDNs are not slowing the site down.
4. Testing for Success
It’s clear that to test for success an effective eCommerce test strategy is critical. Having the right
team in place to implement the strategy will then enable the tests to be executed successfully. Testing for success means understanding the value of testing in production and in the performance lab.
SOASTA’s methodology emphasizes various tests in different places to find what is often a needle
in a haystack.
4.1 SOASTA’s CloudTest Methodology
The SOASTA Performance Test Methodology is a software QA performance testing strategy with a
set of lightweight processes that fit into the software development lifecycle and ensure that online
applications are ready for peak usage. The methodology includes testing applications in a lab
environment inside of the firewall and processes for testing the production infrastructure used by
customers.
Cloud Testing leverages the efficiencies of the cloud to test web applications and sites. By using
the infrastructure and services provided by companies like Amazon Web Services and Rackspace,
SOASTA customers can reduce the cost of load and performance testing while increasing the accuracy of representing the actual traffic to the site.
CloudTest is a distributed architecture that can also be deployed completely behind the firewall,
or in a combination of on-premise and cloud-based configurations. Based on years of experience
testing from the Cloud and leveraging existing best practices and proven performance testing
methodologies, the SOASTA methodology extends traditional approaches to address new opportunities and challenges. These include:
SOASTA
RETAIL BEST PRACTICES WHITE PAPER | 6
• Testing both in the lab and with live web-based applications in production
“
• Leveraging the Cloud to test at both typical and peak traffic levels, from hundreds of users to
millions
• Responding to the acceleration of development cycle times by making agile performance testing
a realistic alternative
• Generating geographically dispersed load to most accurately represent real-world traffic
• Generating both internal and external load and against both the lab and production environments
for the most efficient and effective results
• Analyzing performance intelligence in real-time to speed problem resolution
Tests are not executed until the latter half of the methodology. This is because a successful performance test is the result of defining a strategy that is best for an eCommerce site, and then putting
the right people and processes in place.
4.2 Strategy and Planning
SOASTA’s approach to performance engineering calls for an umbrella strategy with associated
individual test plans. Test plans roll up into an overall view that ensures confidence in the ability of
key revenue generating applications to perform as expected. The result is an ongoing performance
engineering strategy throughout a site’s evolution. It includes a number of test plans centered on
individual objectives, such as holiday readiness, a major architectural change, or the release of a
major version of code.
Having a well-defined strategy, with explicit test plans, provides business and engineering leaders
with a high degree of confidence in operational readiness. Using this approach gives greater insight
into an application’s performance.
HAVE
A WELL-DEFINEDHAVE
A WELL-DEFINED
STRATEGY,
WITH
EXPLICITLY
STRATEGY,
WITH
EXPLICDEFINED
TEST PLANS.
ITLY DEFINED
TEST PLANS .
Using an iterative process within test plans to achieve defined goals allows for a stream of continuous improvement in the applications being tested. It starts with the test definition and ends with
obtaining actionable intelligence results.
The process of creating a test plan starts with the Define Phase. During this phase, the flows to be
tested throughout the site are defined, metrics to be monitored are established, and success criteria for the tests are agreed upon.
In the Design Phase, the user scenarios are written and test parameters are set up. Things such as
the mix of users executing different parts of the application, the virtual user targets, and the ramptime are modeled.
The Test Phase is where the execution of tests takes place, and where data is collected for assessment.
Finally, the Assess Phase, parts of which may occur during the test execution, is when the data collected throughout test execution is used to provide actionable intelligence.
4.3 Production Testing
There are a variety of reasons for testing in production, but in large part it’s becoming more common because it’s now affordable and because the positive experience of SOASTA’s customers has
reduced the fear associated with testing a production infrastructure. The only way to truly build
confidence in a site is to test that site. Due to the sheer number of users accessing their site at any
given time, very few companies can replicate a production environment in a staging environment or
in a performance lab. SOASTA has developed a methodology for testing in production that minimizes the impact on real users while getting valuable, actionable information.
Testing in production is the best way to get a true picture of capacity and performance in the real
world and is the only way to ensure that online applications will perform as expected.
Can retail companies really test in production? The answer is a resounding yes!
SOASTA
RETAIL BEST PRACTICES WHITE PAPER | 7
The SOASTA CloudTest
Methodology is a software QA
performance testing strategy
with a set of lightweight processes that ensure that online
applications are ready for peak
usage.
SOASTA’s production testing methodology helps identify the invisible walls that show up in architectures after they move out of the lab. Traditionally, testers have been limited to making extrapolations over time about whether small tests on a few servers in a lab can support exponentially higher
amounts of load in production. Without proper testing, these types of assumptions always result in
unexpected barriers. There are many things that a production testing approach catches that cannot
be found with traditional test methods. These include:
• Batch jobs that are not present in the lab (log rotations, backups, etc.) or the impact of other
online systems affecting performance
• Load balancer performance issues such as misconfigured algorithm settings
• Network configuration problems such as 100MB settings instead of 1GB on switches and routing
problems
• Bandwidth constraints
• Data pipe not configured as ‘burstable’ to handle spikes in traffic
• Latency between systems inside and outside of application bubbles
• Misconfigured application servers or web servers
• CDN not configured to serve up new content
• Radically different performance depending on database size
4.4 Testing in the Performance Lab
Ongoing performance testing in a lab allows application engineering teams to assess performance
over time, and helps catch any show-stopping performance bugs before they reach production. In
addition, the lab provides a place to performance test code and configuration changes for performance regression before releasing changes to production and outside of the normal build cycle.
This could include things like a quick bug fix in a page, or a seemingly minor configuration change
that could have a performance impact. Often, these kinds of changes are deployed with little to no
testing and come back later as a cause of performance issues.
Testing in the lab also gives you the ability to break the system without having to worry about the
impact on users. It is important to understand how the eCommerce site responds when taken to
failure and how it recovers. No one wants to bring a live production site to failure.
SOASTA
RETAIL BEST PRACTICES WHITE PAPER | 8
Building an End-to-End Test
Plan where production testing
is an essential component.
4.5 Types of Tests
In addition to testing for baseline performance metrics, the methodology includes failure, duration,
spike and break point testing. There are a number of different test types that might be included in
a test strategy, which are described in more detail in the Testing Tactics section on page 10.
Executing different types of tests in different environments is a key element in finding and resolving
bottlenecks in architectures that make up an eCommerce site, and are often quite complex. The
graphic below illustrates how different types of tests in the lab and in production can help pinpoint
specific issues as you work toward optimal performance.
4.6 Content Delivery Network (CDN) and Third-Party Content
TESTING IN PRODUCTION
IS THE BEST WAY TO GET A
TRUE PICTURE OF CAPACITY
AND PERFORMANCE IN THE
REAL WORLD AND IS THE
ONLY WAY TO ENSURE THAT
ONLINE APPLICATIONS WILL
PERFORM AS EXPECTED.
When a retail company tests in production, it can also fully test the caching/offloading capabilities of its CDN. This is vital to understanding the true performance of a production environment.
A primary purpose of a CDN is to reduce the amount of times content has to be requested from
the origin servers by delivering certain content from strategically placed servers within the CDN
network.
When preparing to test in production it is imperative to engage with the CDN provider early in
the process to ensure that a production test has the support of the CDN provider. SOASTA has
worked with Akamai to develop a set of best practices for tests including CDN assets.
The vast majority of test labs do not have a CDN as part of the infrastructure. Therefore, it’s
impossible to test how the CDN affects performance without testing in production. One SOASTA
customer, a performance engineer for a multi-billion-dollar retailer with more than 2000 storefronts, considers the inclusion of a CDN a vital part of production testing. He says, “When we test
in production, we generally have Akamai caching enabled. This definitely influences the testing in
terms of number of requests that reach origin (main site) and the response times depending on
from where the page is served - Akamai or origin. In the lab it is most likely that we won’t include
Akamai. This means we have to include the CDN in our production tests or we have no true idea of
our production performance.”
Many eCommerce sites use third-party providers to enhance their overall site content. It’s vital to
involve those third-party providers that might have an impact on performance when the strategy
is being formulated. On the other hand, you would not typically include domains such as Google
Analytics or Omniture metrics as part of your test. They do not want to be surprised by a test or
have it bring down their service or site with fake transactions. Involving third-party providers early
in the process helps to ensure their support. They may even want to be measured in the process.
After all, they want to have a well performing site, too.
SOASTA CloudTest gives retailers the ability to analyze the performance of individual third-party
provider content and provide that information to the third-party provider.
SOASTA
RETAIL BEST PRACTICES WHITE PAPER | 9
4.7 Potential Live Customer Impact
Testing in production can be done with or without live users on the environment. The majority of
customers testing in production with CloudTest are doing so on a live environment. It’s possible,
but not always feasible, to put up a maintenance or turn-away page and wait until all existing user
sessions have finished. In reality, this method is actually rarely utilized because the right tool and
methodology working together can allow testing to take place with active users on the site in all but
the most extreme cases. To use one SOASTA customer’s exact words, “The cost of not testing is
far greater than the potential impact on live customers”. You can make revenue at the same time
you’re testing — with the right approach.
It’s also possible to segment a portion of the live environment during a low-traffic period and allow
for testing on this sub-environment. Or test against one data center while live users are all routed
to another data center. Typically a separate IP address is configured on a load balancer and servers are moved out of the live pool and placed into the test pool. Sometimes configuration changes
need to be made to other shared components. This is a more costly approach due to the need for
extra hardware and the maintenance overhead. It’s also somewhat less reliable because you start
to deviate from the actual production configuration, and you cannot test to true scale. It is, however, a more realistic test than simply testing in a lab.
4.8 Three Requirements for Successful Live Testing
The first requirement for being able to do a large-scale test with live customers is having real-time
analytics in your test tool. With up-to-the-second information about site performance, you can
quickly tell if a site has started to experience poor performance or become completely unresponsive.
The second requirement is a good “kill switch”. Pressing stop or abort in a running CloudTest will
stop the load immediately. Bandwidth, concurrent connections, threads in use, and other typical
pinch-points will all drop back to normal. With real-time analytics and the kill switch, you can stop
a test as soon as you suspect customers may be impacted. If the test tool takes a long time to kill
a test then this poses a serious risk to real customers. CloudTest is designed to accomplish a nearimmediate stop.
EXECUTING DIFFERENT
TYPES OF TESTS ACROSS
DIFFERENT ENVIRONMENTS
IS KEY TO FINDING AND
RESOLVING BOTTLENECKS
IN THE ARCHITECTURES OF
ECOMMERCE SITES.
Finally, having good monitoring practices internally, preferably integrated with CloudTest, can prevent you from ever needing to abort a test because of live user impact. Real-time visibility into metrics like CPU utilization, heap usage, garbage collection, and connection counts on load balancers
or firewalls can help prevent those thresholds from ever being violated by routine testing.
5.0 Testing Tactics
Running the tests is exciting because it is the culmination of many hours of work. This is not the
time to take shortcuts! If a good test strategy has been developed then the time has come to execute it.
SOASTA’s testing methodology, as described above, has proven successful for retailers of all sizes.
The ‘Test and Assess’ phases are where you see the fruits of your labor. This is when you simulate the expected traffic; monitor the results in real-time, and collect the data to help identify any
necessary remedial actions. Many SOASTA customers use the dashboards while the test is running,
greatly reducing the break/fix cycle time.
Testing in production allows the tests to be executed the same way the customer will interact with
the application. It’s worth repeating that the SOASTA methodology is a continuous process. Retailers should test early and test often. Minor changes can have major impact and is another reason
we recommend testing in both the performance lab and in production. It helps find performance
issues earlier in the testing process.
5.1 Roles and Responsibilities
It is vital to have the right people involved in the test execution. The size of your company and staff,
the scale of the site, the complexity of the testing, whether or not you’ve outsourced development
or deployed third-party applications within your site, the use of a managed service provider for
infrastructure and more all influence how many individuals are involved in a test.
In some cases it’s one or two people who do everything. In other cases, such as the complete
reconstruction of a site or testing a popular site for seasonal readiness, there will be multiple people
for many of the responsibilities, from architects to individual contributors responsible for specific
elements of the infrastructure.
SOASTA
RETAIL BEST PRACTICES WHITE PAPER | 10
There are a number of responsibilities to address as part of a testing strategy:
• Coordination – Coordination of testing activities with all key stakeholders. This includes application engineering, operations, vendors and third-party connected services, as well as business
and senior leadership.
• Communication – Communication of test results, issues, issue resolution plans, and progress
against overall strategy to business and engineering leadership.
• Strategy – Definition of an overall strategy that includes establishing business processes for applications, key performance indicators, capacity plans, monitoring coverage, and individual test
plans that roll up into a performance management aspect of the software development life cycle.
• Architecture – Delivering best practices on creating high-performing application and infrastructure architectures for new projects or for improving existing applications.
• Test Creation – Turning business process definitions into executable tests, creating individual
performance test workloads from capacity plans, and maintaining the current library of test cases
on an ongoing basis.
• Test Execution – Executing ongoing performance tests in the performance lab and production
environment with delivery of test results.
WHEN THE RIGHT MIX OF
PEOPLE ARE BROUGHT TOGETHER, TESTS ARE MUCH
MORE PRODUCTIVE.
• Analysis – Reviewing test results from all environments and analyzing the performance with
respect to previous tests. Responsibility for calling out any violations of established success criteria, thresholds that may be violated on key performance indicators, or deviation from previous
baseline tests.
• Diagnosis – Root cause analysis of bottlenecks or performance problems that might arise. Using
domain knowledge and specialized tools such as profilers, memory leak detectors, and monitoring tools to pinpoint problem areas.
• Tuning – Tuning involves applying best practices, tuning recommendations and isolating parts of
the application or infrastructure that could be optimized to give better capacity or performance.
• Measurement – Responsibility for analyzing activities and their progress against the overall strategy, in addition to making process improvement recommendations for optimizing all strategic or
tactical activities.
In small companies one person may take on all of these responsibilities, often assisted by SOASTA
in test strategy, creation, execution and analysis. Some or all of the roles may be outsourced, typically to take advantage of expertise, create focus on areas lacking attention and/or to lower costs.
Normally there is a blend of individuals and roles. For example, the presence of a Project Manager,
performance engineers and/or specialists would either remove the need for a tech lead, or allow
that person to fill other gaps in the process.
Below are common roles/titles and how they might map to the various responsibilities:
• Project Manager – Coordination, communication
• Technical Lead – Coordination, communication, strategy, architecture, analysis, diagnosis, tuning, measurement
• Architect – Infrastructure architecture, app architecture
• Performance Engineer – Develop strategy, architecture, analysis, diagnosis, tuning, measurement
• Test Engineer – Test creation, execution, analysis
• Specialist – Diagnosis, tuning
When the right mix of people are brought together, tests are much more productive. When issues
arise, having the right team available enables those issues to be quickly taken care of and decisions
to be made when actionable intelligence is provided.
5.2 Types of Tests
A question that often comes up is ‘What kind of tests need to be run?’ There are several test options – baseline, stress, spike, endurance and failover. For a retailer, all of them have value.
• Baseline – Retailers need to establish a baseline of what is acceptable performance for their
site when under ‘average load’. We recommend taking the last six months of analytics and the
busiest hour from each day and using that as your average load values for page views/hour and
orders/minute.
SOASTA
RETAIL BEST PRACTICES WHITE PAPER | 11
• Stress – Retailers need to run stress tests to ensure that site performance does not degrade
under heavy load for an extended period of time. It is not unusual for memory leaks or faulty garbage collection to cause performance issues that are not seen until stress tests are conducted.
SOASTA recommends that stress tests be executed at 150-200% of peak expected load.
• Spike – A spike test is when a very large number of users ramp-up very quickly or almost simultaneously. For retailers spike tests are vital. Many retailers have spike events – flash promotions, Black Friday, Cyber Monday, Valentines Day, etc. – and these events can bring a site to its
knees. Retailers have to ensure that during such events orders are not lost or users turned away.
SOASTA recommends that spike tests be run at 200% of peak expected load.
• Endurance – Endurance tests differ from stress tests in magnitude. While a stress test will have
a very high number of page views and orders, an endurance test will simulate slightly less load
but for a longer period of time. These tests have great value as they can uncover additional areas
where performance can degrade over a prolonged period of time. There are batch jobs, database
backups and cache refreshes that can be scheduled and run on a recurring basis. Running an
endurance test over several hours may show that these types of events effect performance.
• Failover – Most retailers have disaster recovery or failover servers that should spring into action
in case of a failure. If a hardware failure occurs during a period of heavy load can the backup
hardware adequately step into the void? There is no way to know unless a failover test is executed.
THE COST OF NOT TESTING
IS FAR GREATER THAN THE
POTENTIAL IMPACT ON LIVE
CUSTOMERS.
In early testing the goals are often simply to understand the performance characteristics of the site.
For those tests the ramp-up rates may be slower and more methodical. Once these baselines are
established, and you have a good sense of what the site can handle, it’s critical that the tests be
executed as realistically as possible. This includes having realistic user levels and ramp-up rates.
Reviewing the analytics from previous years will help you build user ramp-up levels. It is imperative
that you do not cheat the test. Do not alter think times or remove critical components of the test.
Removing or altering them will only skew the results and lessen the value of the tests.
5.3 Monitoring the Test
Monitoring is an integral part of the testing process. CloudTest includes system monitors that capture real time data on the systems under test. It monitors the performance of the entire application
stack and enables all test participants to monitor how the system under test is reacting under load.
Additionally, SOASTA integrates with third-party application monitoring tools like CA Application
Performance Management, AppDynamics, AWS CloudWatch, Correlsense and New Relic. This
allows even greater insight into the application stack during the test. SOASTA also works with the
monitoring systems that retailers already have in place. Most retailers have extensive monitoring covering the breadth of their architecture. When a test is executed that combines SOASTA’s
monitoring with the retailer’s monitoring, unparalleled insight can be gained into the system’s true
performance.
Understanding exactly how the infrastructure reacts under load allows for modifications to be made
during the test. For example, it is not uncommon to identify an application server that is not in rotation or a thread pool that is incorrectly configured. Due to SOASTA’s real-time analytics, changes
can be made ‘on the fly’ during the test and the impact immediately measured.
6. Retail Examples
Are there retail companies that follow this methodology today? Yes! SOASTA works with several
retailers that test from lab to production and the results have been outstanding.
6.1 Multi-billion-dollar Retailer with 2000+ Storefronts
The company conducts load and performance tests in both a test environment and a production
environment. They test on a continual basis to see if there are performance issues that need to
be addressed very early in the development cycle. Prior to the 2011 Holiday Season, the retailer
conducted 21 tests in their production environment. This enabled them to fully test several different
configurations to achieve optimal performance.
Additionally, the company knew their system’s exact capacity and where the first bottleneck could
potentially appear. That is powerful knowledge to have going into a busy retail season. Consider
how much time is lost trying to pinpoint the bottleneck and how to overcome the performance
issue. This allows a contingency plan to be put in place and ready to execute if higher than anticipated load is observed.
SOASTA
RETAIL BEST PRACTICES WHITE PAPER | 12
“We mandate that we test in production. ... This is the only way we can accurately test our load balancers, switches and third-party providers, which are
generally not present in the lab.”
Performance Engineer,, Multi-Billion-Dollar Retailer
AFTER PRODUCTION
TESTING, THE RETAILER
EXPERIENCED THEIR BEST
ONLINE RETAIL EVENT
EVER. OPTIMAL PERFORMANCE WAS MAINTAINED
DURING PEAK PERIODS.
THEY WERE ABLE TO
EASILY HANDLE 3X THEIR
PREVIOUS HIGH VOLUME
DAY.
How could these tests be executed in production? Wasn’t it afraid of affecting actual customers?
Naturally it was concerned about its customers. Hence, when tests were executed in production,
careful monitoring was conducted at several layers by both SOASTA and by its internal monitoring systems. This served a dual purpose. First, any significant degradation in performance could
be immediately detected and the test stopped, if necessary. Second, it was a real-world test of the
systems that needed to be monitored, stopped or rebooted when Black Friday arrived. The team
tasked with ensuring the viability of the entire company infrastructure had great confidence going
into Black Friday as they had already run more than 20 realistic tests.
The company’s performance engineer observed, “Due to the discrepancies between the lab and the
production environment, there is no guarantee that the application that is certified to be performing well in the lab would perform the same way in production. Hence, we mandate that we test in
production, which enables us to test the application with even higher than expected load levels
since there are different hardware constraints. This is the only way we can accurately test our load
balancers, switches and third-party providers, which are generally not present in the lab.”
This testing allowed the retailer to place 6th in Catchpoint System’s rank of the top 50 retailers
when they tested Document Complete times.
6.2 Dillards, a Department Store Chain with 330 Storefronts
Dillard’s was new to the ‘testing in production’ model. Previously, they had conducted all load and
performance testing in their test environment and had experienced performance issues during critical retail events. Not only did this lead to a loss of revenue, but their reputation was also impacted.
Dillard’s, in true proactive fashion, took on the challenge to get to the root cause of the performance
issues. For Dillard’s, this meant looking at everything from the architecture and code to the hardware. Significant changes were made across the board.
As might be expected, Dillard’s wanted to start testing in production gradually. SOASTA conducted
low-volume tests of a few hundred users hitting the site at a rate of a few thousand page-views per
hour and generated a relatively low number of orders. These low-volume tests allowed Dillard’s to
understand the methodology and gain a comfort level with SOASTA, and the confidence to test in
production. Once these low-volume tests proved successful, larger tests could be conducted to
reach the page view per hour and order per hour benchmarks Dillard’s had set as targets. These
increasingly larger tests began to reveal performance bottlenecks. Resolving these issues would
prove critical to having acceptable performance during peak traffic periods.
A few database issues were identified that weren’t found in the QA environment. It was discovered
that a certain database job would kick off at certain times each day. This would not impact performance during average retail periods. However, when SOASTA would run a peak-level load test,
performance would noticeably degrade when this job was active. Without knowing this, it’s likely
that the customer experience would have been adversely affected during these periods. In addition,
some database locks were identified when the site was under high load. Had these locks not been
identified it would have proven catastrophic.
It was also observed during one of the large load tests that the connection pool limit was being
reached on all the application servers. This led to further tuning to optimize the performance of
each of the application servers.
Dillard’s experienced their best online retail event ever. Their environment maintained optimal performance during peak periods. They were able to easily handle 3X their previous high volume day.
SOASTA
RETAIL BEST PRACTICES WHITE PAPER | 13
6.3 One of the world’s largest retailers
One of the world’s largest retailers conducted extensive performance tests in production. Dozens
of tests were conducted against production sites located on three continents, including mobile applications. This company has a corporate mandate of always testing in production. In fact, over 250
personnel, including the CTO, attended the final load test before Thanksgiving, certainly an indication of how important such testing is to the entire organization. They fully understand that the only
way to find large-scale bugs is to test at scale.
During one round of testing a significant database contention issue was discovered. This issue was
only observed at very high scale – thousands of orders per minute with tens of millions of page
views per hour. This would have been impossible to replicate in a lab or test environment. Finding
this issue directly led to a successful Black Friday event where their systems were able to remain
stable, even under extreme load.
Another example of how vital testing in production proved to be was illustrated when just three
weeks before Black Friday, a new shopping cart flow was released. It had passed all internal tests
and was believed ready for large-scale load. However, when SOASTA performed a load test it was
quickly apparent that under heavy load that this new shopping cart flow would not remain stable.
This discovery allowed changes to be made so it could withstand the expected influx of users.
Without this test, it is quite likely that many customers would not have been able to complete their
purchases on Black Friday.
7. Security
Of course, retail companies, like most enterprises, have specific considerations when it comes to
security, data integrity and systems management when testing their infrastructure.
THIS COMPANY HAS A
CORPORATE MANDATE
OF ALWAYS TESTING IN
PRODUCTION. IN FACT,
OVER 250 PERSONNEL,
INCLUDING THE CTO,
ATTENDED THE FINAL
LOAD TEST BEFORE
THANKSGIVING, CERTAINLY AN INDICATION
OF HOW IMPORTANT
SUCH TESTING IS TO
THE ENTIRE ORGANIZATION.
Data security is one of the key concerns when it comes to performance testing. Companies do
not want their corporate data outside corporate firewalls. SOASTA does not ask customers to
export data or their application outside their firewalls. We typically generate synthetic data or use
scrubbed data depending on the customer’s business and security requirements. CloudTest does
not save any response content on the CloudTest servers or in the result database. Only key http/
https metrics and statistical data are captured for reporting, analytic, and diagnostic purposes.
Security is one of those things that no one notices until something goes wrong. Data is just supposed to be secure. You rarely see a story on the news about how a company is doing a great job
of keeping their customer data secure. But you will see a story about a company letting customer
data slip into the wrong hands. With good reason, customers must be cautious about letting a
company have their data unless they are sure it will be safe. To ensure that our customer’s data remains safe, SOASTA has implemented a stringent security policy that meets the needs of our retail
customers.
7.1 Test Development
Data that is used for test development by a Performance Engineer (i.e. HTTP(s) GET/POST requests
and responses) is not typically retained by SOASTA.
At the customer’s request, this data will be deleted immediately after test development.
Test definitions (SOASTA test clips) are saved in the SOASTA repository for future testing. These are
the steps that comprise the scenario and, while they can be deleted upon request, since there is no
confidential data or other company private information they are typically retained for ongoing use.
7.2 Test Metrics
This includes the performance results of the test(s), such as response times, errors, data transfer
rates, etc., as well as infrastructure metrics for any system resources that are monitored as part
of the test. Monitored metrics may come from SOASTA monitoring agents, SSH access to system
information or third-party monitoring systems such as described above. Test result metrics for
on-demand engagements are kept for a minimum of one year, primarily to allow for comparative
reporting over time. Additionally, if SOASTA is delivering the testing as a service, results can be
exported to a CloudTest instance that is located on the customer’s premises.
At the customer’s request, this data (performance metrics and system resource monitors) will be
deleted immediately after the delivery of test reports or if the results are exported and sent to the
customer.
SOASTA
RETAIL BEST PRACTICES WHITE PAPER | 14
AWS
IBM
SOASTA
CloudTest
GoGrid
Rackspace
SUT/AUT
Azure
Cloud Testing with SOASTA
provides completely distributed architecture, automated
deployment and multiple cloud
providers to effectively evaluate
the System Under Test.
Firewall
7.3 Data Encryption
Passwords and usernames for the SOASTA CloudTest application and for system resource monitors are encrypted. Seed data that uses any actual customer data can also be encrypted. Synthetic
seed data that is fabricated for logging into an application or web site to emulate real users is not
encrypted. All information used to setup monitors is retained in the SOASTA repository for future
testing.
At the customer’s request, this data, the usernames and passwords, will be deleted immediately
after testing.
7.4 Test Infrastructure
When delivered as a service, CloudTest customers have their own tenant (with associated credentials) and this tenant is maintained for at least one year from the date of the last test. At customer
request, the tenant will be deleted immediately after all testing and reporting has been deemed
completed.
The actual load generators and results aggregation servers are temporary cloud-based instances
and run only during test execution. This provides an additional level of security as all instances are
discarded after each test, with performance metrics retained only in the results database.
See Appendix A for Performance Data Q & A that covers additional topics related to data security.
SOASTA
RETAIL BEST PRACTICES WHITE PAPER | 15
Sample Master Data
APPENDIX A: CloudTest Performance Data Q&A
What is Performance Data?
There are many different types of performance data that are leveraged as part of performance testing. Performance data can be broken up into three major categories: master data, user-generated
data and external data. Master data typically exists in the customer’s database(s) and is required for
conducting business (usernames, passwords, etc). User-generated data is anything that the user
inputs in editable fields on the application (new email address, new address, etc). External data is
provided upon execution of the application (i.e. confirmation numbers, session ids, etc).
What data is used during testing?
Data requirements are dependent on the application and the business processes under test. For
example, a static site might not require any performance data, just access to the site to make the
requests, while a more complex application might require all three types of performance data.
What happens to the response data received?
All customer-related response data is discarded from CloudTest servers during load testing. Only
performance data related to that response data is retained (response times, errors, etc.). In addition,
as noted above, CloudTest servers are temporary instances discarded after each test, with only
performance metrics retained in the results database.
However, all customer and external data is stored on the CloudTest server during script creation
and debugging. The data used to create scripts can be deleted after script creation is completed to
ensure no customer data is store on SOASTA servers.
How are log files or other metrics that capture information about data handled?
SOASTA does not keep log files or other metrics that capture information about customer data during load tests. But as previously noted, we do store this type of data during script creation.
What data is sent back to the CloudTest servers?
During load testing, only key external data is kept, such as http response codes, cookies, session
IDs, etc. All data is sent back to the CloudTest servers to parse, at which point it is fully discarded
from any SOASTA servers.
SOASTA
RETAIL BEST PRACTICES WHITE PAPER | 16
What is the difference between scrubbed and synthetic data?
Scrubbed data resides in a customer database and has gone through a process so it no longer
includes any real customer data; it is, in essence, data that has been transformed and/or created
from actual customer data. Synthetic data is generated from scratch and is designed to be an accurate representation of the customer’s production database.
How do we create synthetic data?
SOASTA works closely with your application team to ensure a rich spread of representative data is
created. Data is also created to target the business process under test and can expand as testing
expands.
What happens to the data once the testing is over?
When testing is complete, the CloudTest servers only store performance metric data relevant to the
test runs. No corporate data is retained on the CloudTest servers.
How is the test data stored in the cloud?
For tests executed by SOASTA, the test results are stored in the cloud. The results are on Linux
servers that are taken down at the conclusion of the testing event. These servers are only available
during testing sessions or when results are being analyzed. In most cases, the result data is stored
in a relational database in EC2 in an EBS (elastic block store). This data is only available to SOASTA
employees.
Can test result data be removed from the cloud?
Yes, results can be deleted and removed from the cloud at the request of the customer. When
requested, the results are not deleted until after the result report is completed. Once the results are
deleted, they are permanently deleted; there are no backups of deleted results.
Note that deleting results limits the ability to compare results from prior and future tests. The only
way to compare results once a result is deleted is by looking at the result report.
Can test result data be exported from the cloud for offsite storage?
Yes. All results are exportable in XML format. Results can be exported from the cloud and moved
to a customer-defined location for safekeeping. Depending on the length of the test and amount
of performance data related to the test, it can take up to 24-48 hours to export the data from the
cloud and move to a secure location. Optionally, the results can then be deleted from the cloud as
well.
Contact us:
866-344-8766
444 Castro St.
Mountain View, CA 94041
To learn more visit:
soasta.com
or email us at
[email protected]
©2012 SOASTA. All rights reserved. SOASTA,
the SOASTA logo, and SOASTA CloudTest
are trademarks of SOASTA. All other product
or company names may be trademarks and/
or registered trademarks of their respective
owners.
EST-1001-1-0512
Connect with us: