Solace, Arista and NetEffect
Collaborate to Demonstrate New
Level of Performance in Low Latency
Market Data Distribution
The extremely time-sensitive nature of program trading activities means that every microsecond
represents differentiation and profitability. Firms are making wholesale architectural changes to
their Algorithmic trading systems just to remain competitive. Strained legacy software running over
traditional networks simply can’t keep up with more modern architectures that route, filter and
accelerate automated trading activity in hardware.
Advances in messaging and networking hardware are driving the latency out of market data
delivery and, most importantly, keeping it low even as market data volumes increase. This paper
introduces an architecture that leverages advanced messaging from Solace Systems, Gigabit
Ethernet switches from Arista Networks and TCP offload engines from NetEffect, an Intel
company. The combination of these technologies provides ultra-low, ultra-consistent latency to
algorithmic trading engines, which is a key requirement of predictable and profitable trading.
A New Higher-Performance Architecture
This paper introduces an architecture that includes Solace, Arista and NetEffect equipment for a
hardware data path that delivers lower latency than any software-based architecture.
To validate and document the performance of this architecture, Solace, Arista and NetEffect
collaborated on a series of tests to measure end-to-end market data delivery performance under
two different real world conditions: 500,000 messages a second to simulate peak market data
rates for the entire equity, commodities and currency instruments, and at 1,000,000 messages a
second to simulate peak market data rates for OPRA.
All tests measured performance from publisher API to subscriber API with full message routing
from multiple publishers to subscribers, simulating configurations common within financial firms.
Test results are summarized as follows:
Copyright 2008
Solace Systems, Inc.
http://www.solacesystems.com
http://www.aristanetworks.com
Data Volume
Mean
Latency
Standard
Deviation
99.9th %
Latency
500,000 messages/second
31 µsecs
6 µsecs
47 µsecs
1,000,000 messages/second
35 µsecs
7 µsecs
55 µsecs
Collaboration Demonstrates New Level of Performance
in Low Latency Market Data Distribution
Architectural Overview
In most ultra low latency architectures, high-speed messaging
runs as software on the publisher and subscriber servers,
network protocol stacks are implemented in software and
network connectivity is via store-and-forward layer 2 switches,
all of which contribute to higher and more volatile latency.
The architecture shown here and discussed in this paper
removes the bottlenecks and inconsistencies that exist in realworld, high-volume scenarios.
o Solace’s content router implements the per-client message
routing and filtering in purpose-built FPGA-based hardware.
o Arista’s datacenter switches accelerate Ethernet packet
delivery by streaming packets instead of forwarding them.
o NetEffect’s TCP offload engines migrate the publisher and
subscriber networking stacks into dedicated hardware to
accelerate network interactions.
Since the entire delivery chain is in hardware, this configuration
delivers the lowest possible latencies through industry standard
TCP and Ethernet. The only software components are the
applications themselves.
Performance Highlights
Solace, Arista and NetEffect worked together to test the performance advantage of a solution including all three companies’
hardware. Testing was done in a configuration common to any production customer environment, so firms should expect to
obtain very similar results in their own networks using this configuration. The configuration consisted of a Solace 3260
Content Router, Arista 7124S Ethernet Switch in 1GigE mode, and a NetEffect Gigabit Ethernet adapter.
Note that these tests do not represent the maximum capabilities of each product, just their performance in the stated real
world simulation. For example, in a two blade configuration, a Solace content router can handle over 10 million messages
per second (5 million in, 5 million out) although simulating the highest volume market in the world, the OPRA feed, requires
routing of only of just 2 million messages per second (1 million in, 1 million out).
Using Arista Networks’ low latency switching technology in 1GigE mode, NetEffect Gigabit Ethernet adapters on client
computers and a Solace Systems 3260 Content Router for ultra-low latency messaging, these tests measured end-to-end
market data delivery from the publishing feed infrastructure routed to the appropriate trading applications. The combination of
these three technologies demonstrated that tens to hundreds of microseconds can be removed from high-volume market
data configurations when compared with standard software and Ethernet switching technology.
31 µsecs of Latency at 500k msgs/second
35 µsecs of Latency at 1M msgs/second
Tests used 100 byte messages and 12 byte topics at rates
of 500,000 messages per second to simulate peak US
market data rates for all equities, commodities and FX
instruments.
A second test with 100 byte messages and 12 byte topics at
1,000,000 messages per second was run to simulate rates
greater than peak OPRA market data activity.
Testing resulted in an average latency of 31 microseconds
and a 99.9th percentile latency of 47 microseconds with a
standard deviation of just 6 microseconds.
2
This testing yielded an average latency of 35 microseconds
and a 99.9th percentile latency of 55 microseconds, with a
standard deviation of 7 microseconds.
Collaboration Demonstrates New Level of Performance
in Low Latency Market Data Distribution
Arista High-speed Ethernet Datacenter Switches
The Arista 7100 Series is a family of high performance, very low latency layer 2/3/4 10 Gigabit Ethernet datacenter switches.
They support up to 960 gigabits per second, 720 million packets per second, with latency as low as 600 nanoseconds.
Offered with 24 and 48 ports in a compact 1U chassis with redundant power and cooling, the Arista 7100 Series features
front-to-rear airflow when mounted in either direction for flexible rack-top server aggregation deployments.
All ports accommodate the full range of 10GbE SFP+ or GbE SFP optical or copper physical layer options, allowing for
maximum flexibility and investment protection as customers of all sizes migrate their server connections from Gigabit to 10
Gigabit Ethernet.
Arista EOS™
The Arista 7100 Series runs Arista EOS™, a datacenter-class operating system with a fine-grained modular protected
memory architecture that ensures the highest levels of reliability and availability. Each process is monitored and restarted
automatically in response to failure, while in-service software upgrades (ISSU) allow individual software components to be
updated without disrupting system operation. Arista EOS can be extended to support virtualized environments and is
customizable to meet specific customer functionality requirements.
Solace’s High Volume Low Latency Messaging
Solace makes content routers—carrier-grade, rack-mountable devices that implement the functions of messaging and
middleware in hardware. All the logic is in silicon for a lightning-fast datapath that is much more predictable and resilient than
software-based solutions. Modular blades offer capabilities to meet each customer’s requirements, such as topic-based
routing, content-based routing, content transformation and guaranteed messaging.
The reliable messaging capabilities that support ultra low latency market data distribution are provided by two of these
blades: the Network Acceleration Blade (NAB) processes the TCP network stack in hardware, and the Topic Routing Blade
(TRB) matches subscriptions with the incoming messages. Support for up to eight GigE interfaces offers very high bandwidth
and low serialization delay. Solace’s low latency solution has the following capabilities:
•
Average end-to-end latency as low as 31 microseconds in high-volume scenarios.
•
Throughput of ten million messages per second per NAB.
•
Ten million client subscriptions. This gives clients the ability to subscribe to precisely what they are interested in
without the need to filter unwanted messages that share the same topic space.
The Advantages of Delivering Ultra Low Latency in Hardware using TCP
Solace has architected its 100% hardware messaging solution around broker-based message filtering and unicast delivery to
client applications of only the messages they have subscribed to. Conventional wisdom has historically been that multicast
was required to achieve low latency and that all processing between the sender and receiver needed to be eliminated, but
many of the reasons for this are no longer true. Use of multicast requires that client devices use interrupt coalescing (which
adds many 10’s of microseconds of latency) to deal with the large incoming packet rates, that all these packets be copied
from kernel space to user space and that message filtering of unwanted messages be done in software at each and every
client machine. Solace’s content routers perform message filtration and replication centrally in hardware, rather than on each
client machine, which enables very high rates and very low consistent latency compared to software processing.
3
Collaboration Demonstrates New Level of Performance
in Low Latency Market Data Distribution
The value of granular subscriptions
Each Solace content router can support ten million client subscriptions, so each algo engine can subscribe to exactly the
market data it requires. This eliminates the wasteful fan-out that drives many implementations to turn to multicast. This
delivers the following benefits:
o More efficient client-based subscriptions removes the need to filter unwanted data, resulting in faster application
performance and reduced subscriber CPU burden.
o Better use of the network, since only data with matching subscriptions is forwarded to subscribers by the content router.
o Reduced TCP fan-out because of the ability to define topic subscriptions to a much finer granularity.
As such, with the proper use of subscriptions, TCP-based message delivery is almost always faster and more predictable
than it would be using multicast. Especially under high load, multicast solutions tend to show very high latency and outliers.
Solace’s solution eliminates outliers and ensures that latency is always low and consistent
Superior Monitoring & Troubleshooting
Solace’s content routers gather usage rates in real-time, making it much easier to monitor key performance metrics and
troubleshoot developing situations before they negatively and noticeably impact operations, clients and the bottom line. Wall
Street estimates that one hour of trader downtime costs approximately $35,000, so the ability to quickly identify and address
problems is compelling and has significant value.
4