Schema On Read In Financial Services
Presented by: Lee Pollington, Solutions Director, Amir Halfon, Financial Services CTO
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Financial Services Use Cases
SLIDE: 2

Trade Operational Data Store

Reference Data Management

Regulatory and Legal Compliance

Customer Insight

Financial Crime Prevention

Information Distribution
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Old Generation Trade Store: Schema on Write
Limited, fragmented analytics
and reporting capabilities
Expensive, error-prone
post-trade processing
Long, costly
development cycles
Derivatives
Rates
FX
Matching
ETL
…
Clearing
Settlement
…etc.
…etc.
Multiple Relational Data Stores for different
instrument types
SLIDE: 3
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
What’s the problem

Huge Volumes of Data – (billions of trades per day, PBs/TBs per day)

10s of Source Systems

10s of Users of Different Roles

Few Source Systems + Single Client

Cross asset view

One schema can’t work

There are two time axes, valid & system
SLIDE: 4
✔
✖
✖
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
The Back Office Landscape
Fraud Detection
10s
Analyst
Derivatives
Securities
Uncleared Margin Reporting
Accountant
FX
Liquidity
Manager
10s
Reference Data
Liquidity Reporting
Reference Data
Reference Data
SLIDE: 5
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
The Mechanics Of The Schema Problem
35667
75335
95577
TRADER_ID
T_ID
TID
James Gold
Ted Cash
Amy Wedge
AN
DT
PB
TRD_ID
SLIDE: 6
TRID
TRADERID
245 | AMF
980 | GHN
234 | QTE
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Transformation, Mastering & Science
SLIDE: 7
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Now Add Multiple Reference Data Systems

Reference data describes traded instruments (stock, bonds) and
the related legal entities (e.g. issuer, counterparty).

Data has a temporal aspect (two time axes, valid and system)



Corporate actions - companies go through mergers,
acquisitions and spinoffs
Local
Reference Data
Tactical
Reference Data
Corrections over time
Provenance is critical (regulatory compliance)
Strategic
Reference Data
SLIDE: 8
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Now Add Reference Data…
Data Acquisition
Content Management
Data Distribution
Trading
Risk
Golden Copy
ETL
ETL
normalize,
arbitrate
…
etc.
SLIDE: 9
Raw Data (staging)
Compliance
Research
etc.
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
What Are The Client’s Needs?

Only the set of data I am interested in

As fresh as possible

In the format I understand

In time chunks I can process

With the ability to do bitemporal operations
SLIDE: 10
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Schema on Read with MarkLogic
Analyst
Derivatives
Securities
Accountant
FX
Liquidity
Manager
Reference Data
Reference Data
Reference Data
SLIDE: 11
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
A Future Approach – Enter Semantics

Provenance captured right along with the data

Attributes were captured in an ontology

Dataset mapping modeled ontology

Semantic-based data integration

SLIDE: 12
Dynamic inference at runtime to avoid “schema on write”
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Data Provenance Using Semantics
<Trade>
<Cashflows>
<PartyIdentifier>
<TradeID>123456 </TradeID>
</PartyIdentifier>
</Cashflows>
<provenance>
<triple> <subject> Cashflows</subject> <predicate> wasDerivedFrom </predicate> <object> CDS_xyz </object> </triple>
<triple> <subject> TradeID </subject> <predicate> wasAttributedTo </predicate> <object> System_123 </object> </triple>
</provenance>
</Trade>
SLIDE: 13
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Reference Data in RDF
Swap123
Leg1
JPMC
counterParty
hasLeg
Cashflow
Payment
Date
SLIDE: 14
Payment
Amount
Notional
Currency
BrokerDealer
Derivatives
LEI
Equities
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Semantic Data Integration and MDM
Source systems ontology
Source systems
MDM
…
SLIDE: 15
Master Data Management
1.
2.
3.
Match on metadata (table name etc.)
Match on content (row count, columns, etc.)
Determine action based on weighting:
a. Auto-match
b. Non-match
c. Human intervention (workflow)
Operations
workflow
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
New Generation Data Management: Schema on Read
Single store for all instrument types
persisted as-is off a message bus
Single source of truth for compliance
Simplified workflow architecture
Minimized trade exceptions
Cost effective platform
Matching
Clearing
Settlement
…etc.
Post Trade
Processing
Executed Trades
Exceptions Management
Surveillance, Risk
& Compliance
Load data “as is”
From multiple refdata vendor feeds
SLIDE: 16
etc.
Deliver data
to each consumer
in the right format
Historical Analysis
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Recap – the Key Ingredients

Document orientation

Flexible Indexing

Dynamic Transformtion

Semantics

Bi-temporality
SLIDE: 17
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Q&A
SLIDE: 18
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.