Case Study
Objective
Maintain operational e-commerce
efficiency throughout an extensive
website re-platform project
Approach
Built a test environment that then went
through a vendor selection process to
find suitable test software
John Lewis protects
critical online trading
through rigorous testing
HPE Performance Center on SaaS bombards
site with 15,000 virtual visitors
IT Matters
• Provides a scalable, on-demand test
environment with no need to purchase
new hardware or upgrade networks
• Enables stress and soak tests with
15,000 virtual visitors and
30 test scripts
• Ensures that the new website can
cope with seasonal spikes of up to
1.5 million page views an hour
Business Matters
• Maintains company’s e-commerce
growth rate of over 20 per cent
• Represents a generic reduction
of 30 per cent on Total Cost of
Ownership compared to on-premise
implementations
• Gives John Lewis the confidence
that its systems can support future
business growth
Online sales can account
for 40 per cent of total
trade at UK department
store chain John Lewis.
When it re-platformed its
website, the company had
to protect that income
and maintain growth by
ensuring total operational
efficiency from day one.
The answer was extensive
pre-launch testing with
HPE Performance Center
on Software-as-a-Service.
Challenge
Need to protect sales and growth
The UK’s Centre for Retail Research confirms
that e-commerce is the fastest growing
retail market in Europe with combined sales
revenue across eight countries expected to
reach £111.2 billion (€155.3 billion) in 2014.
Typical of this trend is the UK’s John Lewis
Partnership whose JohnLewis.com website
accounts for between 25 and 40 per cent
of the company’s total trade, depending
on seasonal variations. In 2013/14, online
shopping generated income of £1.2 billion
(€1.48 billion) for the company and it is
growing in excess of 20 per cent per annum.
Case study
John Lewis
Partnership
Industry
Retail
Page 2
“Testing with HPE Performance Center on SaaS allowed us to launch
our new website with confidence. When we need to tackle really high
volumes of traffic we now know that our systems are in good shape and
this supports the on-going growth of the business.”
— Paul J. Smith, performance test manager, John Lewis Partnership
John Lewis Partnership was launched in
1864 with one London store and is now the
UK’s largest department store retailer with
30 stores, ten smaller ‘at home’ shops and
309 Waitrose supermarkets. The company’s
visionary way of doing business means that
all of its 91,000 staff are partners, so are
dedicated to serving the customer which
results in annual gross sales of over £10
billion (€12.3 billion).
Because online shoppers now account for
so much of that revenue, it’s essential that
the e-commerce website is slick and easy to
use. JohnLewis.com launched in 2001 and as
additional features were bolted on over the
years, it was in need of a refresh. There was
particular concern about the time taken
to launch products onto the old site, so a
new platform was needed. The 250-strong
website IT team worked on developing a
new site for two years and during that time
thorough testing was vital to ensure that it
would cope with massive usage spikes that
are typical of the retail calendar.
“We have two clearance sales each
year. One in June and one which
starts on Christmas Eve,” explains Paul
J. Smith, performance test manager for
JohnLewis.com. “These are the two biggest
spikes, but we also have the strong preChristmas trade and new US shopping
traditions that the UK has embraced like
Black Friday and Cyber Monday.”
“At the start of the clearance sale on
Christmas Eve, we see page views going
from 450,000 an hour to 2.3 million an
hour; and because people sit on the site
waiting for bargains to be published, the
rise happens in just 60 to 90 minutes.
This aggressive fivefold increase in a very
short space of time puts the website under
intense pressure. We also expect to see
page views of over 1.5 million an hour for
the first week of the sale; and that happens
against a backdrop of increasing business
activity, with new imagery and clearance
prices being constantly published to the site.
“The new website was going to be launched
into a very busy period, so its performance
had to be at least as good as the previous
site to maintain our growth trend. Extensive
pre-launch testing was crucial.”
Solution
Realistic test environment
To achieve realistic testing, the team built
a test environment that was 60 per cent of
the capacity of the old site and it needed
test software that could cope with high
volumes of traffic. Having conducted a
vendor selection process and looked at
various options, John Lewis started using
HPE Performance Center (PC) internally to
test individual components. Then, when the
infrastructure and code were in place and
high volume testing was required, they used
HPE PC on Software-as-a-Service (SaaS) for
a further 18 months.
Case study
John Lewis
Partnership
Industry
Retail
Page 3
HPE PC on SaaS is a scalable, on-demand
application for performance testing,
designed to validate the performance of
applications under realistic conditions and
align them with business objectives. Hewlett
Packard Enterprise hosts the application
which minimises the need for hardware and
software investments and HPE analysis
shows that this reduces Total Cost of
Ownership by 30 per cent when compared
with on-premise implementations.
The John Lewis team used web analytics
to build detailed models of customer
behaviour, pinpointing areas that were
highly used, complex or critical to
e-commerce functionality. Each area was
given a score, then testing started with
the highest priorities and worked down.
Using HPE PC they created 25 to 30 test
scripts covering different activities such as
searching, navigating, adding goods to the
basket, checkout and payment.
“Having been using HPE PC internally
there were lots of assets available that we
knew would be compatible with SaaS so
we wouldn’t have to completely rewrite
everything,” says Smith. “Also, HPE PC is
pretty much industry standard which meant
that when we brought in contract resources
to help with the scripting and testing it
would not be unfamiliar to them.
Creating 15,000 virtual users, the team ran
stress tests that put the site under intense
short-term pressure and soak tests that ran
for several days at a time. Different scenarios
were also created to simulate the different
load patterns created by seasonal spikes.
“We chose the SaaS model because it
brought an ‘outside looking in’ perspective
which allowed us to hit the public facing
infrastructure and tune our external facing
network devices. SaaS enabled us to
progressively increase the demand that we
put on the systems and we realised that if
we had tried to generate these large test
loads internally it would have meant a big
investment in infrastructure. Also there were
other unseen challenges such as whether
our network capacity might have had to be
increased to enable us to generate the load
internally. Sending large volumes of traffic
out to the Internet for them to come back in
would have been an inefficient process.”
“While the old website was still running in
the background we were building our test
environment, developing and testing the
site and also building and testing against
the live production environment which was
just sat there waiting to be switched on. We
built a suite of tests that replicated different
customer activities then ran the tests,
analysed live traffic, compared it to test
traffic, pinpointed differences and tuned the
tests to close the gap between the live and
test,” explains Smith.
For switchover, the visitor population was
divided into different sections and each of
these was activated in turn. Both the old and
new sites ran in parallel during the transition
but everything was transferred to the new
site within a matter of days.
Case study
John Lewis
Partnership
Industry
Retail
Customer at a glance
Software
• HPE Performance Center on
Software-as-a-Service (SaaS)
“Effective pre-launch testing ensured that growth of over
20 per cent was maintained throughout the launch of the
new website and in subsequent months.”
— Paul J. Smith, performance test manager, John Lewis Partnership
Benefits
Successful launch
The new website had to hit the ground
running from day one otherwise there
was a danger that the annual e-commerce
growth rate of over 20 per cent could have
stagnated or even declined. Thanks to
stringent testing with HPE PC on SaaS,
the project was a complete success and
the new site went live with no problems.
It is delivering the speed and performance
levels required to retain discerning online
shoppers and high conversion rates have
maintained growth in excess of 20 per cent.
Testing isn’t just a single event, John
Lewis runs high volume testing against its
production environment on a frequent basis.
“The Summer clearance sale was the
website’s first major challenge and then
it faced a further stringent test with
Christmas,” reports Smith. “It passed with
flying colours because we used HPE PC
on SaaS to test it for 18 months and had
used HPE PC internally for six months
before that.”
The website is continuously having
new functionality added with different
propositions and different delivery options,
so we constantly have to maintain the
scripts. We continue testing with HPE PC
to make sure that what we have changed
hasn’t degraded performance.
“Throughout the project we worked closely
with Hewlett Packard Enterprise and they
delivered the quality of service we needed.
Thanks to HPE PC on SaaS, confidence in
our testing results has grown to the level
that we are now having input into the
company’s marketing strategy. For example,
if we sent out millions of promotional
emails, what level of response could the
environment cope with? The confidence of
knowing that our systems are in good shape
enables us to contribute to the ongoing
growth of the business.”
Learn more at
hpe.com/go/alm
Sign up for updates
Rate this document
© Copyright 2014-2015 Hewlett Packard Enterprise Development LP. The information contained herein is subject to change
without notice. The only warranties for Hewlett Packard Enterprise products and services are set forth in the express warranty
statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty.
Hewlett Packard Enterprise shall not be liable for technical or editorial errors or omissions contained herein.
4AA5-3589EEW, November 2015, Rev. 1