WHITE PAPER
March 2013
Highly Accurate Time Synchronization with ConnectX®-3 and TimeKeeper®
Highlights and Benefits:
• Sub-microsecond time synchronization, using off-the-shelf hardware
• Support for both PTP and NTP protocols and master clocks
In collaboration with
• TimeKeeper® is easily deployable on any existing ConnectX®-3 network adapter card
FSMLabs
3
Upgrading your trading platforms to reliable and precise time is achievable at a low cost and a rapid
deployment model via the combination of Mellanox’s ConnectX-3 network adapter cards and TimeKeeper
Client software. TimeKeeper can assure sub-microsecond time precision from both the newer IEEE 1588
Precision Time Protocol (PTP) or the standard Network Time Protocol (NTP) over shared (not dedicated)
network links. Flexibility in time sources and automatic adapability to network quality allows for
incremental changes to enterprise systems, and immediate high precision timing in critical components
while less critical components see incremental performance improvement. For high quality links and time
Precision in microseconds/NTP
feeds, applications can see time locked to reference well within 500 nanoseconds of variation.
medium quality link
MicroSeconds
1
0.5
0
-0.5
-1
Samples
Figure 1. Precision in Microseconds/NTP Medium Quality Link
MicroSeconds
High performance comes from the complementary action of two technologies: TimeKeeper’s state-of-theart time synchronization and management and the Mellanox ConnectX-3 hardware based packet timestamping. TimeKeeper uses information that the ConnectX-3 provides to factor out packet delay variation
(PDV) caused by buffering, network stacks, and OS scheduling and automatically improves precision of
locking the local timer to thePrecision
reference time.
the ConnectX-3 can time-stamp both PTP and NTP
in Because
microseconds/PTP
packets, TimeKeeper can greatly improve
performance
of both
protocols.
medium quality
link
1
0.5
0
-0.5
-1
Samples
Figure 2. Precision in Microseconds/PTP Medium Quality Link
©2013 Mellanox Technologies. All rights reserved.
WHITE PAPER: Highly Accurate Time Synchronization with ConnectX®-3 and TimeKeeper®
page 2
TimeKeeper’s Client software will automatically detect and take advantage of the time stamp capabilities
of the ConnectX-3 – the only configuration needed, is to point the software at the right network interface,
IEEE 1588 and with NTP. After which, TimeKeeper can also be used to setup failover, auditable trace
logs, automatic spoof and equipment failure detection, and automatic notification of error conditions to
management consoles (via SMTP) and IT staff (via email). Customers can also use the web interface to
validate network setup and to optimize their time distribution networks.
The graph below, Figure 3, is from the TimeKeeper web management console and shows two time
sources being tracked and validated against the other. Discrepancies can trigger a switch of time sources
or an alert or both- depending on configuration. TimeKeeper can track any number of independent time
sources. This increases system resilience and also produces a trace of system behavior to pinpoint causes
of errors. The free software for PTP and NTP can fail silently – allowing application code to keep trading
on the basis of incorrect time. The “time map”, Figure 4, capability introduced in TimeKeeper 6 discovers
the surrounding time distribution network and helps IT staff find configuration problems and identify
sources of network time fragility. This feature is in use in some of the world’s leading financial trading
organizations and has uncovered numerous previously undetected system vulnerabilities
Source tick rate (per second)
2266689500
Source 0 ideal tickrate
2266689000
Source 1 ideal tickrate
2266688500
2266688000
2266687500
2266687000
2266686500
2266686000
2266685500
Samples
Figure 3. Cross Check Second
132.163.4.103
ACTS
129.6.15.28
64.113.32.5
69.36.224.15:GPS
this_system
69.36.224.15
128.252.19.1:GPS
128.252.19.1
69.25.96.13 128.138.140.44 172.16.1.36
216.171.124.36
132.163.4.101
128.7.1.66
129.250.35.250
204.42.158.152
128.9.57.133
204.123.2.5:GPS
204.123.2.5
216.45.57.38
209.51.161.238
43.77.130.254
199.4.29.166
128.67.113.92 166.255.32
216.218.254.202
216.218.254.202:CDMA
108.71.253.20
129.7.1.66:GPS
209.51.161.238:CDMA
129.250.35.251
192.5.41.40
143.166.82.251
128.9.176.30
209.81.9.7
128.4.1.1
18.26.4.105
108.71.253.20:GPS
192.5.41.40:IRIG
128.4.1.1. PPS
18.26.4.105.PPS
209.81.9.7:GPS 128.9.176.30:GPS
Figure 4. Time Map
Because TimeKeeper and the ConnectX-3 are protocol agnostic, it is not necessary to make huge,
disruptive changes in existing time distribution networks. Various parts of the network can use PTP or
NTP as needed, leveraging the strengths of each and the fault tolerance derived from using sources
carried over both protocols.
©2013 Mellanox Technologies. All rights reserved.
page 3
WHITE PAPER: Highly Accurate Time Synchronization with ConnectX®-3 and TimeKeeper®
Distribution of NTP/PTP in Upgraded TK Network 2
TK enabled
Application Server
Application Server
TK enabled
Application Server
TK enabled
Application Server
Application Server
NTP
PTP
Application Server
Application Stratum
Server
TK GrandMaster
Appliance
Switch
Switch
NTP
PTP
Figure 5. Distribution of NTP/PTP in Upgraded TimeKeeper Network
About Mellanox ConnectX-3 Network Adapters:
Mellanox’s cost-effective, high-performance, ConnectX-3 network adapters support a variety of network
protocols from 1/10/40GbE and FDR 56Gb/s InfiniBand. ConnectX-3 network adapters are available
through OEM server vendors in a variety of form factors, including: LOM, add-on cards, blade mezzanine
cards and PCI expansion cards. With ConnectX-3 network adapters, enterprise server engineers can
utilize a single, cost-effective adapter for all of their data center needs – trading, time synchronization,
database, and virtualization.
For more information on Mellanox ConnectX-3 network adapter card, visit:
http://www.mellanox.com/page/products_overview
For more information on TimeKeeper, contact: [email protected]
TimeKeeper and FSMLabs are registered trademarks of Finite State Machine Labs Inc.
350 Oakmead Parkway, Suite 100, Sunnyvale, CA 94085
Tel: 408-970-3400 • Fax: 408-970-3403
www.mellanox.com
© Copyright 2013. Mellanox Technologies. All rights reserved.
Mellanox, BridgeX, ConnectX, CORE-Direct, InfiniBridge, InfiniHost, InfiniScale, MLNX-OS, PhyX, SwitchX, UFM, Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies, Ltd.
Connect-IB, CoolBox, FabricIT, Mellanox Federal Systems, Mellanox Software Defined Storage, MetroX, Open Ethernet, ScalableHPC and Unbreakable-Link are trademarks of Mellanox Technologies, Ltd.
All other trademarks are property of their respective owners.
15-513WP Rev 1.0