Complex Event Processing (CEP) is the continuous and incremental
processing of event streams from multiple sources based on
declarative query and pattern specifications with near-zero latency.
Database Applications
Event-driven Applications
Query
Paradigm
Ad-hoc queries or
requests
Continuous standing
queries
Latency
Seconds, hours, days
Milliseconds or less
Data Rate
Hundreds of events/sec
Tens of thousands of
events/sec or more
request
response
Event
input
stream
output
stream
2
Latency
Months
CEP Target Scenarios
Days
Relational Database Applications
hours
Operational Analytics
Applications, e.g., Logistics, etc.
Data Warehousing
Applications
Web Analytics Applications
Minutes
Seconds
100 ms
Monitoring
Applications
Manufacturing Applications
< 1ms
0
10
100
1000
10000
Financial trading
Applications
100000
~1million
Aggregate Data Rate (Events/sec.)
3
Manufacturing:
• Sensor on plant
floor
• React through
device controllers
• Aggregated data
• 10,000 events/sec
Web Analytics:
• Click-stream data
• Online customer
behavior
• Page layout
• 100,000 events /sec
Financial Services:
• Stock & news feeds
• Algorithmic trading
• Patterns over time
• Super-low latency
• 100,000 events /sec
Power, Utilities:
• Energy
consumption
• Outages
• Smart grids
• 100,000 events/sec
Stream Data
Store & Archive
Visual trend-line and KPI monitoring
Batch & product management
Automated anomaly detection
Real-time customer segmentation
Algorithmic trading
Proactive condition-based maintenance
Asset Specs &
Parameters
Data Stream
Data Stream
Asset Instrumentation for Data Acquisition, Subscriptions to Data Feeds
Event Processing
Engine
• Threshold
queries
Lookup
• Event correlation from
multiple sources
• Pattern queries
4
CEP advantage
Industry trends
• Data acquisition
costs are
negligible
• Raw storage costs
are small and
continue to
decrease
• Processing costs
are non-negligible
• Data loading
costs continue to
be significant
Monitor KPIs
Record raw
data (history)
Manage
business via
KPI-triggered
actions
Mine historical data
Devise new KPIs
• Process data
incrementally,
i.e., while it is in
flight
• Avoid loading
while still doing
the processing
you want
• Seamless
querying for
monitoring,
managing and
mining
5
Data Sources, Operations, Assets, Feeds, Sensors, Devices
Input
Data Streams
Monitor
&
Record
Input
Data Streams
Mine
&
Design
CEP Engine
f(x)
g(y)
Operational
Data Store &
Archive
f'(x)
Output
Data Streams
Manage
&
Benefit
h(x,y)
CEP Engine
Results
f(x)
g(y)
f'(x)
h(x,y)
6
CEP Application
Development
IDE
Event sources
Event targets
CEP Application at Runtime
Devices, Sensors
Event
CEP Engine
Pagers & Monitoring devices
Standing Queries
Event
Event
Event
Event
Event
Event
Event
Output Adapters
Input Adapters
Web servers
.NET
C#
LINQ
`
KPI Dashboards,
SharePoint UI
Event
Trading stations
Event stores & Databases
C_ID
C_NAM
E
C_ZI
P
Stock tickers & News feeds
Event stores & Databases
Static reference data
7
>
>
>
>
Payload/ value 
>
a
t1
b
t2
c
t3
Time 
d
t4
e
t5
>
>
>
>
>
Timestamps
/Metadata
Long
pumpID
…
…
String String
Type Location
…
…
Double
flow
…
Double
pressure
…
>
>
>
>
>
>
>
>
>
>
>
10
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
LINQ Example – JOIN, PROJECT, FILTER:
from e1 in MyStream1
join e2 in MyStream2
on e1.ID equals e2.ID
where e1.f2 == “foo”
select new { e1.f1, e2.f4 };
Join
Filter
Project
LINQ Example – GROUP&APPLY, WINDOW:
from e3 in MyStream3
group e3 by e3.i into SubStream
from win in SubStream.HoppingWindow(
FiveMinutes,ThreeSeconds)
select new { i = SubStream.Key,
a = win.Avg(e => e.f) };
Grouping
Window
Project &
Aggregate
demo
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>var
outputStream =
from win in
input.TumblingWindow(TimeSpan.FromSeconds(10))
select new
{
v = win.TimeWeightedAverage(e => e.dVal)
};
public class TimeWeightedAverage :
CepTimeSensitiveAggregate<double, double>
{
public override double GenerateOutput(
IEnumerable<IntervalEvent<double>> events,
WindowDescriptor wd)
{
double avg = 0;
foreach (IntervalEvent<double> e in events)
{
avg += e.Payload *
(e.EndTime - e.StartTime).Ticks;
}
return avg / (wd.EndTime wd.StartTime).Ticks;
}
}
>
>
>
>
>
>
>
>
Analytics & Queries
Data Sources
QT1
CEP Engine
AT1
>
>
>
Standing Queries
Event
Event
Q1
Q1’
Event
Q1’’
Event
Event
Output Adapters
AT3
Input Adapters
AT2
Event
Query
>
>
Output Adapter
>
Type
Type
Query
Template
>
>
Proj.
Join
Type
Type
Type
Type
Input
Adapter
Input
Adapter
Web servers
Data Sources
CEP
Sensors
CEP
Feeds
Devices
CEP
Aggregation &
Correlation
CEP
CEP
Event processing engines are deployed
at multiple places on different scales
• At the edge – close to the data source
• In the mid-tier – consolidate related
data sources,
• In the data center – historical archive,
mining, large scale correlation.
CEP
CEP
CEP
CEP
CEP
Complex Analytics &
Mining
CEP
CEP
CEP for lightweight processing and
filtering
CEP for aggregation and correlation
of in-flight events
CEP for complex analytics including
historical data
21
>
>
Programming/
Deployment
Implicit Server
Explicit Server
IObservable
Abstracts from
CEP server,
adapters and
binding
Full control over
CEP server,
adapters and
binding
Use reactive
framework
interfaces
In-process
Supported
Supported
Supported
Separate
processes
N/A
Supported
Supported
Easy to use
No query
template re-use
Full control
Query template
re-use
Easiest to use
Data sources
need
IObservable
interface
demo
Asset
Class
Ticker
Region
Country
Exchange
SUM
Volume
SUM
Bid
SUM
Ask
Stock
MSFT
NA
US
NASDAQ
100
100
100
Stock
IBM
NA
US
NASDAQ
200
200
200
Deriv.
IBM
EMEA
CH
SWIX
10
10
10
Output Adapters
Input Adapters
StreamInsight
Output Adapters
Input Adapters
Output Adapters
Input Adapters
StreamInsight
StreamInsight
Ticker
Exchange
SUM
Volume
MSFT
NASDAQ
100
Exchange
SUM
Bid
SUM
Ask
NASDAQ
100
100
NASDAQ
200
200
Development experience with .NET,
C#, LINQ and Visual Studio 2008
CEP Application
Development
Event sources
CEP platform from Microsoft to build eventdriven applications
CEP applications
Engine
Event-driven
are
Standing Queries
fundamentally different from
Event
Event
traditional database
Event
Event
applications:
queries
are
Event
continuous, consume and Event
produce streams,
and compute
Event
Event
results incrementally
Event
C_ID
C_NAM
E
C_ZI
P
Static reference data
`
Output Adapters
Input Adapters
Flexible adapter SDK with
high performance to
connect to different event
sources and sinks
Event targets
The CEP platform does the
heavy lifting for you to deal
with temporal characteristics
of event stream data
25
Scenario 1: Custom CEP Application Dev Scenario 2: Embed CEP in Application
.NET, C#, LINQ
ISV Application with CEP Engine
Custom CEP Application
CEP Engine
CEP Engine
Reference data
Reference data
Scenario 3: CEP Enabled Device
Scenario 4: Operational Intelligence w/ CE
KPIs
KPI mining
Device with Embedded CEP Engine
CEP Engine
CEP
CEP
CEP
CEP
CEP Engine
ETL Pipeline with CEP engine
Madison
>
>
>
>
>
TechEd
Announcement
August CTP
TAP program
November CTP
TAP continues
May 2009
Aug 2009
Nov 2009
Release
2010
time
question & answer
© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market
conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT
MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.