WHI
TEPAPER
Onl
i
neRet
ai
l
er
s:
Get
speci
ficwhendef
endi
ng
agai
nstpr
i
cescr
api
ng
bot
s
Pr
i
c
es
c
r
api
ngandpr
oduc
tdat
a
s
c
r
api
ngar
er
ampant
www.
di
s
t
i
l
net
wor
ks
.
c
om
s
al
es
@di
s
t
i
l
net
wor
ks
.
c
om
1
.
866.
423.
0606
1. Executive Summary
Two stark realities have emerged for online retailers. Online retail has become incredibly competitive and unsafe.
Online retailers are threatened by the Internet underbelly of nefarious online actors, including big industry competitors. These
threat constituencies are leveraging bad bots in numerous ways that hurt many online retailers. These include bad bots that
scrape prices and product data, perform click fraud, and endanger the overall security of e-commerce websites, your loyal
consumers, and your brand.
Of all these threats, price scraping and product data scraping are by far the most rampant and costly to online retailers. The
easiest means that your competitors and cyber criminals have to do this is by launching bad bots. They are fast, cheap, and
highly effective against sites that do not have bot-specific defenses in place. Certainly, companies have deployed WAF, CDN/
DDoS capabilities and homegrown solutions against a variety of attack types, but these security defenses simply do not address
all bots.
This paper will explain the specifics around bad bots, how they impact online retailers from competitive, revenue and cost
perspectives, and the best way to mitigate bad bots with technology specifically designed to remediate all bots.
2
www.distilnetworks.com
[email protected]
phone: 1.866.423.0606
2. Introduction: An Industry Built to Steal Online
Retailer Pricing & Product Data
Online retailers have spent years and millions of dollars establishing their brand presence online and garnering a loyal following
of customers. These customers represent the lifeblood of the business. Yet, a large and growing pool of online competitors,
nefarious actors and bot creators are working hard each day to steal away these customers and permanently win their business.
These bad actors seek to scrape information from legitimate online retail sites to gain product and pricing intelligence that can
be used to undercut their pricing or position against their offerings.
Whether termed ‘price scrapers’, ‘pricing bots’, or ‘pricing intelligence solutions’, an entire industry has grown around the use of
automated bots dedicated to scraping as much data as possible from online retailers’ websites. Attacks occur in a large variety of
ways and for multiple reasons. However, having catalogued, tracked and blocked over 10 billion bots, we at Distil Networks have
identified the following five types of bot actions that occur most often against online retailers:
Price Scraping. Bots target the pricing section of a site and scrape away all pricing information to share with online competitors.
Product Matching. Bots collect and aggregate hundreds, or thousands, of data points from a retail site in order to make exact
matches against a retailer’s wide variety of products.
Product Variation Tracking. Bots scrape product data to a level that accounts for multiple variants within a product or product
line, such as color, cut and size.
Product Availability Targeting. Bots gather product availability data to enable competitive positioning against an online retailer’s
products based on inventory level and availability.
Continuous Data Refresh. Bots visit the same online retail site on a regular basis so that buyers of the scraped data can react to
changes made by the targeted retail site.
The virtualization and botnet technology that underpin bots makes them attractive to a wide variety of users. They are easy to
create with programming scripts, cheap because hosting costs are low, quick to launch and hard to trace behind millions of IP
addresses. This combination ensures that bots will continue to attack online retail sites in ever-larger volumes for years to come.
Bot Scraping of Online Retailer Sites to Continue Unabated
Despite the publicity garnered by highly effective bot attacks and theft from online retail sites, little has been done to stop
the actors who perform bot attacks. There are several reasons for this. First, the nature of the industry makes identifying and
prosecuting those who launch bots extremely difficult, time-consuming and expensive. Bots can originate from practically any
location in the world and most often originate from well-known hosting providers and networks that organizations trust. Second,
the highly illegal bots frequently originate from international locations, where US laws provide little or no recourse once bot
originators are identified. Third, most bot technologies are not even considered criminal. In fact, vendors that offer price scraping
solutions publicly tout the names of large, legitimate businesses as their customers.
Consider a vendor named Upstream Commerce that markets a price scraping and product datascraping solution. Their homepage
streams the names of retail industry customers, including Staples, Toys-R-Us, Shoebuy, Woodcraft and eBags. In addition, they
make no apologies as they promote their bot-based scraping technologies this way: “Upstream Commerce transforms how
retailers around the world grow sales and boost margins through real-time competitive insights, smarter pricing and optimal
product assortment.” Scraping vendor Kapow Software even more brazenly describes their own business: “Outwit the competition
with dynamic price monitoring across thousands of products in real-time.”
3
www.distilnetworks.com
[email protected]
phone: 1.866.423.0606
As more vendors compete to offer increasingly sophisticated price scraping and product data scraping products, an arms race
of capabilities is taking place. For online retailers, this means a drastic rise in the need for bot defenses that evolve with new and
ever-changing forms of bot attacks.
Advanced Bot Threats in Scraping Price and Product Data
Termed ‘pricing intelligence solutions’, these products are bringing advanced features and higher levels of sophistication to
attacks on retailer websites. Built atop traditional price scraping functionality, they incorporate additional elements to steal and
use victims retail site data in the most scalable and damaging manner possible. Here’s how they work.
Analytics
engine
Matching
engine
SITE
Retail Websites
Site-crawler and
product-data extractor
Client gains actionable
insight from analysis and
reporting
1. Crawling: Ever-changing and custom built bots crawl a retailer’s site. Many hit the site each day, targeting the product pages
specifically and scanning every product.
2. Morphing: Bots take the form of data extractors that
morph in line with changes to a retailer’s site. This
way, bots can attack a site, even if the retailer adds
complexity.
The Business Pain of Bots for Online Retailers
When bad bots scrape pricing and product data, they can
cripple an online retailer’s entire business in the following ways:
3. Matching: Bots collect thousands of pieces of information
to make exact matches against a retailer’s products.
They lift pricing, inventory and availability data from the
shopping cart and other site locations.
• Leading edge retailers innovate, only to get “showroomed”
as customers learn what’s new and cool from their sites
but purchase elsewhere at sites with a reputation for lower
prices.
4. Analytics: Leveraging semantic analysis and data
mining, analytics engines sift through the data scraped
by bots and make matches to prices and products,
even if the scraped data is not an exact match. This, and
other advanced analytics, enables the recipient of the
scraped data to make multiple competitive positioning
moves that steal away competitive advantage.
• Once-loyal customers feel they are being unfairly overcharged for products from the reputable retailer, changing
their affinity for the retailer.
The ramifications of this type of advanced approach
to retail site scraping are dire for many online retailers.
The fact is, online retailers must stop bots from scraping
their data in the first place, before the real harm occurs
downstream. Otherwise, the retailers face a losing battle
once competitors apply advanced analytics against the
data. This final step can take apart an online retailer’s
business — product by product.
• Retailers spend millions of dollars promoting new product
categories and specific products, only to face shrunken
windows of optimal price points. Custom SEO positions
and customer targeting campaigns get copied fast.
• Significant reductions in product margins decrease revenue,
and in the case of small and medium-sized retailers, cause
bankruptcy.
Big retail competitors are doubling down on price scrapers
and content scrapers to not just take today’s customers
from you, but to steal them away forever. No retailer, large
or small, is immune to the impact of bad bots scraping price
and product data.
4
www.distilnetworks.com
[email protected]
phone: 1.866.423.0606
3. AdWord Spying and Click Fraud: Bots Use Ads
Against Retailers
Online retailers have worked through years of iterations and trial-and-error techniques to devise the optimal SEO and PPC
programs. Recently, a new type of bot attack has hurt many online retailers with ‘AdWord spying’ that results in a loss of AdWord
differentiation. In these attacks, bots track where an online retailer has purchased display advertising, the retailer’s variations in
ad language based on audience type and which ads show the best results. Thanks to bots, this type of spy-based intelligence
comes cheap, fast and easy. For example, SpyFu leverages AdWord spying bots to provide a product offering which the company
describes in this manner: “Download your competitors’ keywords for only $79/month.”
Vendors specializing in launching bots to copy successful online retailer SEO and ad campaigns are eroding retailer margins. Yet,
it’s clear they do so with no fear of negative repercussions, such as legal action or industry backlash. In fact, the SpyFu website
homepage blatantly depicts what their service is all about: “SPY ON YOUR COMPETITORS! PROFIT FROM THEIR SUCCESS.”
A closer look at the information gathered by “AdWord spying” bots underscores the need for a bot-specific defense to stop them.
They commonly spy and report in the following ways:
• Analyze which text and image ads perform best for certain keywords and phrases
• View the ads and advertisers that perform best at any individual, specific URL
• Discover sources of traffic that online retailers have discovered work for their campaigns
• Track ad testing patterns of specific online retailers
As with price scrapers, AdWord spying bots can update results daily, quickly rendering ineffective an online retailer’s competitive
adjustments.
Click Fraud
The more commonly known abuse to online retailers’ ads comes from click fraud bots. These are bots that drive up the CTR (clickthrough rate) of ads, costing retailers dearly in ad placement expense with no offsetting revenue benefit. It’s not uncommon for
retailers who fall victim to click fraud bots to report CTRs at 10-20 times the normal rate.
4. Site Security: Bots are Everywhere
Many people associate bots only with price scraping and click fraud, but whether they are called bots or scripts, automated
attacks are confounding online retailers’ efforts to keep their sites secure. Bots manipulate many touch points on a site and can
enable access to a retailer’s most critical data. As such, they have become the top threat to online retailers.
From the early days of bots, one of their main security breach methods has been to pose as human traffic, posting comments and
ads in forums as they attempt to imitate users. Today, it’s the new brute force tactics of bots that have security personnel concerned.
Bots are responsible for hijacking user accounts at many online retailers. Many times faster than a human could, bots rifle through
millions of email and password combinations to enter sites under false pretense. They can perform similarly high-volume forgery
attempts via “credit carding”, a ploy in which the bots test the validity of a mass of stolen credit card numbers, using those that work.
5
www.distilnetworks.com
[email protected]
phone: 1.866.423.0606
Finally, bots hit almost all parts of a site in search of security holes that allow access in a tactic known as vulnerability scanning.
They quickly identify places where the retailer’s site code is not patched, often resulting in database breaches. Once the database
has been breached, the potential for harm to a retailer and its customers can skyrocket, particularly when personal data such as
social security numbers and bank account information become compromised.
As is the case with price scraping, product data scraping, AdWord spying and click fraud, traditional security measures and
solutions cannot differentiate between bots and real users in order to keep an online retailer’s site secure.
The only way online retailers can secure their sites from these attacks and future bot threats is to deploy a bot-specific security stance.
5. Distil Networks: Real-time Bot Detection and
Remediation for Online Retailers
Distil Networks developed the world’s first reverse proxy solution for bot detection that identifies, tracks and mitigates bots.
Delivered via SaaS in a public or private cloud, the solution is platform agnostic for complete system compatibility.
On behalf of online retailers, the Distil solution incorporates multiple proactive
techniques to detect bots in real time. Additionally, Distil leverages machine-learning
algorithms that pinpoint behavioral anomalies specific to each online retailer site’s
unique traffic patterns. This maximizes the effectiveness of the security solution
based on a retailer’s unique protection needs.
All online retailers share a common benefit when using Distil. They gain multiple
layers of proactive defenses designed specifically to eliminate the bots before they
enter a retailer’s site.
legitimate
bad
Bot Detection
40+ criteria
Proactively identifying bad bots
Distil develops a unique identifier for each device software that attempts to connect
with an online retail site. The unique ID is composed of more than 40 bits of information
from device and software. Delving far deeper than user agents and IP addresses,
Distil looks into all connection properties from the first time a request is made to the
site. Our technology then inserts JavaScript into the connection stream to further
validate user agents and detect browser automation. Once a unique identifier is
detected to be a bot, the bot has no way of escaping our detection ever again.
Tracking Bots across the Online World
Distil automatically and proactively tracks bots once our system has identified them
by their unique ID. Even if a bot’s controller makes various attempts at obfuscation,
such as accessing a retailer’s site using proxies or the TOR network, the Distil unique ID is inescapable. Additionally, the unique
ID has so many specific properties that detection of a single bot enables Distil to render an entire botnet harmless to our retail
industry customers.
6
www.distilnetworks.com
[email protected]
phone: 1.866.423.0606
Machine Learning Your Business
Our detection methodology does not stop at the unique ID level. We leverage machine learning algorithms to understand how
the typical users on a retailer’s specific website interact with both that site and one another while there. By understanding the
normal behavior patterns of a retail site’s human users, we quickly recognize any abnormal patterns and can take action against
those bots.
The proactive defense posture that Distil enables is the key element to fending off the most advanced bot attacks. This becomes
clear in the case of advanced pricing intelligence bots. They cause vast amounts of damage to an online retailer once they can
apply analytics to the data scraped by their bots. Using Distil, online retailers never allow the bots to enter their site in the first
place, rendering all phases of the price intelligence solution useless to the nefarious actor(s).
Case Study: Tool King Puts An End To Price Scraping and Content Theft
Established in 1978, Tool King has built an online and offline retail business based on offering shoppers education and
advice around their tool purchases. The company’s focus on customer satisfaction, selection and expertise has translated
strongly to the online world, where ToolKing.com has become a favorite among Internet shoppers. In fact, the site has
ranked in the ‘Top 50’ for Best of the Web from Internet Retailer several years in a row.
ToolKing.com’s great reputation for product variety, price and customer acquisition made the site a high-priority target for
competitors and would-be competitors seeking to scrape prices and valuable expert content. The IT team at ToolKing.com
knew their site was getting scraped, so they signed on with Distil’s SaaS-based bot detection and remediation solution. In
the first month alone, Distil blocked approximately eleven (11) million bad bots seeking to penetrate ToolKing.com. Months 2
and 3 saw Distil block an additional 9 million and 7 million bad bots, respectively. According to Brandon Allhands, Director of
IT at ToolKing.com, “We knew we’d made the right choice in our bot defense with Distil. Our own folks had been spending
hours fending off bots manually, and using Distil, attacks went down almost 40 percent, with 100 percent of bots identified
and blocked.”
ToolKing.com has benefitted in several major ways as a result of implementing bot defenses from Distil. First, the online
retailer is keeping away competitive scrapers who would sell their data to online competitors. Second, Tool King has not
received a single complaint from legitimate shoppers regarding session blocking. Third, the retailer’s hosting costs have
dropped so much that they entirely cover the cost of the added bot defense solution. “We’re pleasing our customers and
chasing away the big threat to our business model, all for a zero-add to our cost structure,” says Mr. Allhands.
Further Distil Advantages for Bot Security
Each site protected by Distil benefits from a knowledge community that has been identifying and tracking bots for years. Bots that
Distil identifies for one customer instantly get shared across the Distil customer community. In this way, brand new customers of
Distil immediately benefit from the industry’s largest knowledge base of malicious bots. From day one, they gain protection equal
to that of a company that has been tracking and fending off bot networks in great detail for years.
Distil also pinpoints the entirety of a threat faster than any other solution. Rather than merely tracking IP addresses, Distil’s unique
ID technology uniquely tracks each bot based on its software and hardware attributes, not its connection. Using Distil’s bot
unique identifiers, customers can halt an entire bot network from a single unique identifier and reject a bot, even if it attempts to
connect from random or changing IP addresses.
7
www.distilnetworks.com
[email protected]
phone: 1.866.423.0606
Distil recognizes that each online retailer may want to respond to bot threats in their own way. Some may choose to immediately
block all bots, while others may choose to vary responses. These can include throttling down bot acceptance gradually, monitoring
a bot or even providing fake pricing data to a bot.
Distil’s complete focus on bot identification and remediation — combined with its flexible SaaS delivery model — provides online
retailers with the most thorough and proactive bot threat mitigation solution on the market today.
6. Conclusion
To date, most online retailers have attempted to fend off price scrapers and other bad bots with a variety of security technologies
that were never designed for this purpose. Standard firewalls, web application firewalls (WAF) and CDN-based protection against
DDoS attacks all fail to keep pace with the fast-changing nature of bots and cannot register the real threat. As a result, bots
bypass these security solutions unnoticed, to the demise of the retailer.
Some retail sites have developed their own homegrown solutions to identify and block bots, but they quickly realize that the
ever-changing nature of bot attacks requires a larger set of resources and dedication to full-time bot detection than anticipated.
In the end, they cannot keep up, and their security stance grows obsolete fast. Given their entire business model relies on being
competitive online, obsolete bot defense can mean the difference between success and failure.
Fortunately, online retailers can now deploy very specific bot defenses, enabling them to continuously fend off competitors and
improve market position. To get specific protection, you need to understand what bad bots are, how they are launched, and what
they do. Only Distil does that.
7. Connect with Distil Today!
Distil’s solutions for bot detection and mitigation are simple to understand. Contact us today by calling 1.866.423.0606 or sending
an email to [email protected] and let us show you exactly how it’s done and how we can specifically defend your online
retail site(s).
8
www.distilnetworks.com
[email protected]
phone: 1.866.423.0606
About Distil Networks
Distil Networks is the global leader in Bot detection and mitigation offering the first software-as-a-service solution focused
on stopping automated attacks to make the web more secure. Distil works like a protective shield and blocks malicious bots,
malware, and competitors that try to scrape or copy your website data without permission. Prevent web scraping, eliminate form
spam and click fraud, reduce infrastructure costs, and regain your competitive advantage with Distil Networks.
Visit us on the web at
http://www.distilnetworks.com
Our Cloud Locations
Seattle, WA
San Jose, CA
Los Angeles, CA
Denver, CO
Dallas, TX
Chicago, IL
New York, NY
Washington, DC
Miami, FL
São Paulo
Dublin
London
Amsterdam
Singapore
Hong Kong
Tokyo
Sydney
9
www.distilnetworks.com
[email protected]
phone: 1.866.423.0606