FAST in 45 minutes
Rolf Andersson
CEO Pantor Engineering
Agenda
z
z
z
z
z
z
z
z
z
Characteristics
Protocol Layers
Vocabulary
Transfer Encoding
Data Types
Field Encoding
Templates
An Example
Beyond 1.0
Screen Event Amsterdam September 2006
2
FAST Protocol
Characteristics
z
Basic Feature Set
z
z
z
z
z
z
z
z
Empirically validated
z
z
z
z
Optimized for message streams
Content aware (requires knowledge about message structure)
Byte-oriented binary representation
Variable-length fields
Each message contains one or more fields
A presence map enables efficient use of default values
Several ways of deriving default values
Very fast processing (encoding / decoding)
High compression ratios on tested exchange feeds
Simple implementation
Extensible
Screen Event Amsterdam September 2006
3
FAST Protocol Layers
Application Messages
Template
Management
Session Control
Field Encoding
Transfer Encoding
UDP
TCP
IP
Screen Event Amsterdam September 2006
4
FAST Vocabulary (1)
z
A data stream consists of a sequence of messages.
A message consists of a sequence of one or more fields.
A field consists of a sequence of one or more bytes.
z
Wire format refers to the byte representation that is used to
z
z
transfer data on the wire.
z
z
z
Encoding is the process of translating to wire format.
Decoding is the process of translating from wire format.
A CODEC (enCOder/DECoder) provides support for encoding or
decoding a stream of messages.
Screen Event Amsterdam September 2006
5
FAST Vocabulary (2)
z
z
z
A byte consists of seven data bits and a stop bit (the ‘SBIT’) that
when set indicates the byte is the last byte of a field.
A presence map (‘PMAP’) is a field that is interpreted as a vector of
bits, one bit (‘PBIT’) for each field in a message.
A field operator enables previous values to be reused.
Screen Event Amsterdam September 2006
6
FAST Transfer Encoding – Fields
7 bits
14 bits
21 bits
nn bits
Screen Event Amsterdam September 2006
7
FAST Transfer Encoding – SBIT Details
z
SBIT coding provides space efficient delimitation of fields
z
z
z
More compact than byte delimiter for fields shorter than 7 bytes
Empirical data shows average FAST field sizes of ~2 bytes
~33% more compact than a byte delimiter at 2 bytes
z
Serialization of SBIT format is simple and efficient
z
There are exceptional cases when SBIT coding is suboptimal
z
Byte vector can be used for these cases
Screen Event Amsterdam September 2006
8
FAST Transfer Encoding – Messages
The number of fields in a message depends on the pmap content
The tail of a pmap may be compressed
A message may be as short as one byte (an empty pmap)
Screen Event Amsterdam September 2006
9
FAST Field Data Types
z
z
z
z
z
Unsigned Integer
Signed Integer
Scaled Number
ASCII String
Byte Vector
The FAST transport representation supports unlimited size fields.
Implementations will likely limit the supported size to match the
native data representation in applications.
Screen Event Amsterdam September 2006
10
Unsigned Integer Transfer Representation
Value
(hex)
Seven Data Bits per Byte (bin)
127
0000007F
0000000 0000000 0000000 0000000 1111111
1
16383
00003FFF
0000000 0000000 0000000 1111111 1111111
2
2097151
001FFFFF
0000000 0000000 1111111 1111111 1111111
3
268435451
0FFFFFFF
0000000 1111111 1111111 1111111 1111111
4
4294967295
FFFFFFFF
0001111 1111111 1111111 1111111 1111111
5
Screen Event Amsterdam September 2006
Len
11
Scaled Number Transfer Representation
A scaled number is represented using an exponent and a mantissa
number = mantissa * 10 ^ exponent
The mantissa and exponent are represented as signed integers
A base 10 exponent is used to provide exact decimal representation
Examples
Mantissa
Exponent
Value
-1
0
-1*10^0 = -1
127
-2
127*10^-2 = 1.27
1
4
1*10^4 = 10000
Screen Event Amsterdam September 2006
12
A Transfer Encoding Example
BeginStr SeqNum
SenderID
SendingTime
Price
Symbol
8=FIX.4.4|34=10000|49=CLIENT1|52=20060126-13:06:58.100|44=1200|55=FOO1|
8=FIX.4.4|34=10001|49=CLIENT1|52=20060126-13:06:58.200|44=1210|55=FOO1|
8=FIX.4.4|34=10002|49=CLIENT1|52=20060126-13:06:58.300|44=1190|55=BAR2|
Original size 71 bytes
FIX.4.4|10000|CLIENT1|20060126-13:06:58.100|1200|FOO1|
FIX.4.4|10001|CLIENT1|20060126-13:06:58.200|1210|FOO1|
FIX.4.4|10002|CLIENT1|20060126-13:06:58.300|1190|BAR2|
FIX.4.410000CLIENT120060126-13:06:58.1001200FOO1
FIX.4.410001CLIENT120060126-13:06:58.2001210FOO1
FIX.4.410002CLIENT120060126-13:06:58.3001190BAR2
FIX.4.4nnCLIENT120060126-13:06:58.100nnFOO1
FIX.4.4nnCLIENT120060126-13:06:58.200nnFOO1
FIX.4.4nnCLIENT120060126-13:06:58.300nnBAR2
Screen Event Amsterdam September 2006
Implicit Tagging
54 bytes (-24%)
SBIT Encoding
48 bytes (-33%)
SBIT + Binary Encoding
43 bytes (-39%)
13
Beyond Transfer Encoding
Further compaction requires information about data affinity and
predictability.
Affinity
BeginStr SeqNum
SenderID
SendingTime
Price
Symbol
8=FIX.4.4|34=10000|49=CLIENT1|52=20060126-13:06:58.100|44=1200|55=FOO1
8=FIX.4.4|34=10001|49=CLIENT1|52=20060126-13:06:58.200|44=1210|55=FOO1
8=FIX.4.4|34=10002|49=CLIENT1|52=20060126-13:06:58.300|44=1190|55=BAR2
Predictability
Screen Event Amsterdam September 2006
14
Templates
z
z
A template specifies how to encode an application
message into the transfer encoding
A template conveys
z
z
z
Message Structure
Data Types
Field Operators
Screen Event Amsterdam September 2006
15
Characteristics of FAST Templates
z
Semantics are formally and unambiguously defined
z
z
Enables consistent behavior and interoperability among implementations
Concrete Syntax
z
z
z
Default format for authoring, storing and interchanging templates
Human and machine readable
XML Syntax
z
z
z
z
Supports evolution of the template specification
Extensible – enables inclusion of custom data in templates
Widespread knowledge and good tool support
Alternative concrete representations
z
z
Hard coded in encoder and/or decoder
FAST encoded as specified by the Session Control Protocol (SCP)
Screen Event Amsterdam September 2006
16
Field Operators
z
z
z
z
z
Constant – Always the same value
Increment – Frequently previous value incremented by one
Copy – Frequently the same as the previous value
Delta – Values differ slightly
Default – Frequently a specific value
Screen Event Amsterdam September 2006
17
Template Overview
Field Operators
Structure
<template name="ExampleOrder">
<messageRef name="NewOrderSingle"/>
<string name="BeginStr"> <constant value="FIX.4.4"/> </string>
<u32 name="SeqNum"> <increment/> </u32>
<string name="SenderID"> <copy/> </string>
<string name="SendingTime"> <delta/> </string>
<decimal name="Price"> <delta/> </decimal>
<string name="Symbol"> <copy/> </string>
</template>
Data Types
Screen Event Amsterdam September 2006
18
A Field Encoding Example
BeginStr SeqNum
SenderID
SendingTime
Price
Symbol
8=FIX.4.4|34=10000|49=CLIENT1|52=20060126-13:06:58.100|44=1200|55=FOO1|
8=FIX.4.4|34=10001|49=CLIENT1|52=20060126-13:06:58.200|44=1210|55=FOO1|
8=FIX.4.4|34=10002|49=CLIENT1|52=20060126-13:06:58.300|44=1190|55=BAR2|
Original size 71 bytes
Constant
Increment
Copy
Delta
Delta
Copy
|10000|CLIENT1|20060126-13:06:58.100|1200|FOO1|
|||200|10||
|||300|-20|BAR2|
Screen Event Amsterdam September 2006
19
FAST – Beyond 1.0
z
z
Session Control Protocol
Templates
z
z
z
z
Byte Vector serialization extension
z
z
z
Support for efficient reading of blocks of data
Support for extended character representations (in addition to US ASCII)
Presence map extensions
z
z
Compact notation has limitations
XML-based notation to provide full expressional strength
Additional field operators / operator combinations
Zero of more bits per field (currently one bit per field)
Implementations
z
z
z
Gain experience from real-life use
Continued research led by FPL
Improve ability to analyze / reason about alternatives
Screen Event Amsterdam September 2006
20
FAST – 1.1 specification RSN
z
Session Control Protocol
z
z
Templates
z
z
z
Specified and reviewed
Presence map extensions
z
z
XML-based notation to provide full expressional strength
String delta extension + backward compatibility
Byte Vector
z
z
Spec completed and reviewed
Zero or one bits in 1.1 (multiple bits deferred to a later version)
Implementations
z
z
z
z
ARCA in production (proprietay format based on 1.0)
Fast API implementation to be updated to 1.1
Extensive research being performed by Pantor, Rapid Addition and others
Validated performance targets
Screen Event Amsterdam September 2006
21