HIGH-SPEED BRIDGE TO CLOUD STORAGE
HIGH-SPEED BRIDGE TO CLOUD STORAGE
Signiant Flight: Addressing throughput bottlenecks to Amazon S3 and Microsoft Azure
+781.221.4051 | WWW. S IGNIA NT. CO M
1
HIGH-SPEED BRIDGE TO CLOUD STORAGE
INTRODUCTION
The next step is optimizing the way
We are in an era of data inundation. According to a report by IDC, by 2020 there will
data is moved to and from cloud
be as many digital bits in existence on the Internet and in storage devices as there are
storage, especially at scale.
known stars in the universe. Much of that data has a neglected life, created only to be
forgotten moments later. But a subset is highly valued, from government secrets and
personal account information to movie files and life-saving research. Those are the files
that we want to protect, analyze, edit, share, backup, and store.
However, as we move into a future where data will only continue to grow in amount and
importance, we will need more storage for the valuable bits we want to keep as well as
much improved digital mechanisms to move them around. Many businesses want to
utilize cloud object storage services such as Amazon S3 and Microsoft Azure to solve the
storage issue, and to gain access to business specific cloud applications. But moving
large data to and from the cloud, especially at scale, remains an obstacle to effectively
incorporating cloud storage. This eBook discusses the technical reasons behind this
challenge, the pros and cons of options provided by cloud vendors and other common
data ingest methods, and introduces Signiant Flight.
+781.221.4051 | WWW. S IGNIA NT. CO M
2
HIGH-SPEED BRIDGE TO CLOUD STORAGE
3
GROWING STORAGE NEEDS AND THE PROBLEM OF
HIGH BANDWIDTH, HIGH LATENCY ENVIRONMENTS
In theory, data travels at the speed of light. But, in practice, switching and
Cloud object storage fits the needs of many companies, giving them scalability
100ms) to travel around the world, but that’s without taking into account any
and global access to their business-critical files. For some of those companies,
switching overhead. And, for longer distances, more switches are involved.
getting files into and out of the cloud as quickly as possible — usually in order to
The latency when sending data half way around the world and back can be
capitalize on an action serviced by cloud software — is also critical. However, the
over 1/3 of a second. Even when sending data between two locations in the
available mechanisms for doing so could use some vast improvements. Typically,
same building, data can travel the equivalent of across the continent if it’s on
the way that content is moved over the Internet, including in and out of cloud
different service provider’s networks; so it’s the network distance, rather than the
object storage, is through HTTP (hypertext transfer protocol) and its underlying
physical distance that’s important. Network distance is what causes throughput
TCP (transmission control protocol). Both HTTP and TCP were originally designed
bottlenecks in most systems, slowing down content transmission even if you’ve
in the 1970s and 80s and constantly stall out in high latency environments —
got a big bandwidth pipe. These throughput bottlenecks can disrupt a successful
even those with high bandwidth.
business model if workflows depend on moving data into and out of cloud object
What is the difference between bandwidth and latency? Bandwidth is the
other overhead in the network slows it down or creates higher latency. The
speed of light is pretty fast; it takes light approximately 1/10th of a second (or
storage.
theoretical maximum rate data can be transferred over a given network or
network segment (usually measured in bits/second). And latency is the amount
of time it takes for data to travel across the network or network segment (usually
measured in thousandths of a second or milliseconds). If you think of the network
as a garden hose, the bandwidth is the width of the hose and the latency is the
time it takes for water to travel the full length of the hose. What can be confusing
WHEN TIME TO AND FROM CLOUD STORAGE MATTERS
here is that latency is not impacted by bandwidth; no matter how big your hose is
Signiant launched Flight to meet the needs of companies dealing with any
the data still has to travel the distance.
situation requiring accelerated access to cloud storage, and to fill the technology
gap they face between their on-premises storage and the cloud storage provider.
+781.221.4051 | WWW. S IGNIA NT. CO M
HIGH-SPEED BRIDGE TO CLOUD STORAGE
TRADITIONAL INTERNET FILE TRANSFER PROTOCOLS
As mentioned above, typically, the way content is moved into and out of
cloud object storage is through the hypertext transfer protocol or HTTP. For
its initial intended purpose, HTTP was optimized for loading web resources
in a browser. The problem is, when you get very high bandwidth pipes and
some distance involved, HTTP and its underlying TCP (Transmission Control
network. And while this approach works great for content that is consistent — the
same image or the same video being served to thousands of people around the
world — it’s not a practical approach when you’re either uploading content into
the cloud or when you’re downloading relatively unique content out of the cloud.
Plus, streaming a video like on Netflix is very different need than downloading an
entire asset as quickly as possible.
Protocol) start to slow down quite dramatically. Flight greatly improves on these
traditional protocols. But before we get into how, let’s look at other ways HTTP’s
performance problem has been addressed with CDNs and IaaS.
Content Distribution Networks
The way CDNs (content distribution networks) work is by caching content at the
edge of the network, closer to the users that are going to consume it. They are
typically used for serving the same content to thousands of people distributed
around the world or distributed around a particular geography.
For example, with a CDN, when a user goes to load an image in their web
browser or watch a video in their web browser, it’s typically not actually being
served from a central data center; it’s being served from a cached copy at the
edge of the
CONTENT DISTRIBUTION NETWORKS
+781.221.4051 | WWW. S IGNIA NT. CO M
4
HIGH-SPEED BRIDGE TO CLOUD STORAGE
Infrastructure as a Service
Additionally, various IaaS (infrastructure as a service) vendors offer different
solutions to address moving content into and out of the cloud. And one of the
most basic of those approaches is to stick your content on some form of physical
media and ship it to a provider, who then loads it into their cloud object storage.
In some scenarios, this approach works, especially if you have a predictable and
unchanging amount of data being transported every day. But, it doesn’t work
well if you need many points or connections into the cloud. And, because you’re
deploying fixed network infrastructure to address the problem, it is especially
inefficient if you have peaks and valleys in the amount of data you are sending. It’s
not agile or elastic and cannot adapt to changing storage demands.
While shipping hard drives works and you can’t beat the bandwidth of a truck,
when you have to do this over and over again, it starts to have some serious
logistical challenges. And it’s not very easy to automate. So, of course, most of
us prefer an electronic, network-based approach. For IaaS, these basically fall
into two categories. One reduces the latency between the person using cloud
object storage and the cloud, and the other compensates for that latency:
How IaaS reduces latency
• AWS DIRECT CONNECT
• AZURE EXPRESS ROUTE
IPE
DP
FIXE
IaaS options for reducing latency, like dedicated network connections such as
• NOT ELASTIC
Amazon Web Services’ Direct Connect and Azure’s ExpressRoute, basically
• NOT AGILE
provide a way to connect up a fixed telecommunications circuit from your
enterprise directly into cloud infrastructure. In order to take advantage of this, first
you have to go to your friendly neighborhood Telco and order a fixed circuit that
runs from your premise into their premise (assuming you’re not already co-located
in one of their data centers with their application). Then you have to wire that
circuit up to your Internet protocol network with firewalls and routers. Then you
have to connect it into the Azure or Amazon network. Really, their offering is only
that last mile connection between your pipe and their on-premises infrastructure.
+781.221.4051 | WWW. S IGNIA NT. CO M
5
• STILL HAVE LATENCY
HIGH-SPEED BRIDGE TO CLOUD STORAGE
How IaaS compensates for latency
Most IaaS service providers offer a multi-part upload capability that helps
compensate for latency, where you can run multiple HTTP transfers in parallel
using a proprietary API to break objects into chunks and upload each of those
MULTI-PART UPLOAD
chunks independently. Again, this approach has some benefits. But you have
to be relatively technically savvy to use the different API that each of the cloud
vendors offers for doing this multi-part upload.
Also, you get diminishing returns as you add parallel streams. If one parallel
stream is added, you get twice the throughput. But when you add a third parallel
stream, you’re only getting 50% improvement in throughput. So there’s a limiting
and diminishing return on adding these parallel HTTP streams, and you still have
latency to deal with.
+781.221.4051 | WWW. S IGNIA NT. CO M
• DIFFERENT API FOR EACH CLOUD
• DIMINISHING RETURNS
6
HIGH-SPEED BRIDGE TO CLOUD STORAGE
7
HOW FLIGHT ACHIEVES EXTREMELY HIGH
TRANSFER RATES
Through these two pipeline steps — Signiant transport protocol over a long
Flight accelerates file movement into the cloud in two pipelined stages, minimizing
high transfer rates. With speeds up to 200 times faster than standard Internet
latency at both stages of the process. The first stage compensates for latency
transfers, Flight also eliminates interruptions in transmission with a featured called
by using the Signiant Acceleration Protocol. The second stage minimizes latency
checkpoint restart: any interruptions in your file transfers will be automatically
by running the Signiant transfer server close to the storage. We’ve improved
restarted, so files will never get stuck en route. Additionally, Flight automatically
on traditional HTTP and TCP by leveraging the parallel HTTP technology used
encrypts all files for the high security transfers.
distance or metropolitan area IP connection and the parallel multi-part HTTP
pipelined on the other end into the cloud object storage — we achieve extremely
by IaaS providers. If we speak specifically about working with Amazon cloud
storage, Amazon EC2 instances have a very low-latency, high-bandwidth
connection to the S3 storage. Therefore, parallel HTTP in the cloud works and
performs quite well from those instances into object storage. We’ve also added
Signiant’s advanced WAN (wide area network, a network that covers a broad
geographical area) acceleration protocol. Essentially, Signiant’s accelerated
SIGNIANT TRANSPORT
transfer servers are run in the infrastructure as a service provider’s environment,
while using the Signiant transport protocol to get content into the cloud.
• ACCELERATED’TRANSFER
• HIGH PERFORMANCE
• ELASTIC & AGILE TRANSPORT
FLIGHT MINIMIZES LATENCY IN TWO STAGES
+781.221.4051 | WWW. S IGNIA NT. CO M
HIGH-SPEED BRIDGE TO CLOUD STORAGE
MORE THAN JUST SPEED: THE ADDITIONAL
ADVANTAGES OF FLIGHT
MANAGED SAAS
Flight, like any true cloud-born software, is run as a managed SaaS offering.
Rather than having to go out and deal with machine instances — turn them
up, make sure they’re running, manage them, and monitor them, for instance
FLEXIBLE CLIENT INTEGRATION
With Flight, there’s no need to change the way you work. You can install
command line clients for Mac, Windows, or Linux on a local machine and you
are ready to move files. There’s also a JavaScript API that can be integrated with
your own applications. Flight subscriptions include the ability to use any of the
clients from any location however you want, without any limits or requirements to
purchase another client. It’s all part of the service.
— Signiant runs the servers for you. In contrast with accelerated file transport
solutions that are not truly cloud-based, with Signiant you wind up with an elastic
high-performance solution that is easy to get started and doesn’t have ongoing
costs associated with managing your own servers, like applying software updates
and monitoring your servers for availability and scale.
SAAS CONTROL
LAYER
MANAGED CLOUD
SERVERS SOFTWARE
SAAS CONTROL LAYER
Flight’s SaaS control layer gives you automatic updates of your web plug-in client,
but also allows you to abstract the back end object storage that you’re moving
content into or out of. As we build out more capabilities in the SaaS control
layer, you will have visibility into real time and historic activity reports and have
total control over what object storage credentials are being used to upload or
download.
AUTOMATIC SOFTWARE UPDATES
AUTOMATIC SCALABILITY
With auto-scaling, redundancy, and load balancing built-in, Flight scales
automatically for you and can handle your largest bursts of files or a wave of
new projects. That’s all included in your subscription and you never have to
worry about spinning up new servers, instances, or finding the headcount and
resources to move large files fast.
+781.221.4051 | WWW. S IGNIA NT. CO M
8
• EASY DEPLOYMENT &
MANAGEMENT
• RELIABILITY/AVAILABILITY/
SCALABILITY
HIGH-SPEED BRIDGE TO CLOUD STORAGE
GLOBAL ACCESS TO MULTIPLE CLOUD STORAGE
PLATFORMS
Flight is the only solution that makes it easy to move files to multiples cloud
platforms from the same system, including Amazon, Microsoft Azure, and Google
Cloud Platform. So you can use Amazon S3 today, and if tomorrow Microsoft
Azure is a better solution for your business, you can switch storage providers and
keep the same initial Flight Subscription. Flight gives you the freedom to choose
the best platform for you business, so you can avoid cloud vendor lock in or use
multiple cloud platforms.
CUSTOMER’S NETWORK
So the Signiant Flight solution is not only fast, it’s easy to use, easy to get started
with, scales with your requirements, and supports multiple clouds. One of the main
reasons it’s so easy to use is that it only requires three simple steps to get started.
+781.221.4051 | WWW. S IGNIA NT. CO M
9
HIGH-SPEED BRIDGE TO CLOUD STORAGE
10
Increasingly, cloud data storage is becoming a critical aspect of software-defined
companies. And, when time is of the essence, the power to efficiently interact
with cloud storage provides a definite competitive edge. If your business needs
THREE STEPS TO GET STARTED
1. Install the Flight client software on your premise.
a high-speed bridge to cloud data storage, learn more, try Flight for free or
contact us to discuss how Flight and Signiant can help you achieve your cloud
data storage goals.
2. Use the Flight service to connect to your cloud storage tenancy.
3. Start transferring files.
ABOUT SIGNIANT
Within minutes, you are moving content to your cloud object storage in an
Signiant software enables high-speed movement of large, valuable files in their
accelerated, high-performance, and secure manner.
entirety over public and private networks. With acceleration technology to
minimize the impact of latency, enterprise class security, and central control,
our software powers most of the world’s Media & Entertainment companies —
SUMMARY
from small post-production houses to Hollywood studios — and is increasing
being adopted for use with satellite imagery, big data analytics, and genome
Addressing slow performance in high bandwidth, high latency environments for
sequencing. At Signiant, we’ve embraced cloud-formed software development,
large-scale data migration to and from cloud storage has been a challenge for
and offer unique SaaS file transfer and cloud storage upload solutions.
businesses. Flight’s advanced acceleration technology addresses throughput
bottlenecks with a dual pipelined approach, providing transfer rates up to
200 times faster than traditional Internet protocols, and doing so with security
encryptions not available in other offerings. As a true SaaS solution, Flight
includes automatic updates, auto scaling for large data bursts, flexible client
integration, and a control layer into the backend of your cloud-stored files.
+781.221.4051 | WWW. S IGNIA NT. CO M