Event Processing
Course
Filtering and transformation
(Relates to Chapter 8)
Copyright ©2009 Opher Etzion
Lecture outline
 Filtering
 Transformations
 Derivations
 FFT examples
 Some code examples
Copyright ©2009 Opher Etzion
2
Filter on input terminal: by event type
A filter expression (assertion) takes the form of a predicate that is evaluated against an event.
The event passes the filter if the predicate evaluates to TRUE
and fails the filter if the predicate evaluates to FALSE[ .
Copyright ©2009 Opher Etzion
3
Filter EPA
Filtered In
Filter EPA
Filtered Out
Filtering
Input terminal
filter expression
Principal filter
expression
Non-Filterable
Copyright ©2009 Opher Etzion
4
Two consecutive filters
Gold
Silver
Gold
In
Filter 1
Bronze
Out
Silver
Bronze
Membership
level = Gold
In
NonFilterable
Filter 2
Out
Membership
level = SIlver
Copyright ©2009 Opher Etzion
5
Silver
NonFilterable
Bronze
Various filtering
Filtered In
Filter EPA
Filtered Out
Filtering
Partition
selection
Input terminal
filter expression
Principal filter
expression
Context
expression
Non-Filterable
Copyright ©2009 Opher Etzion
6
Filter features
Feature
Interpretation
Attribute inclusion
Checks whether an attribute with that name exists in the event instance
Attribute has value
Checks that the value of the attribute is not null
Data type assertion
Checks that the attribute's value is valid with respect to its data type, for example
that it is valid as a number, string or timestamp
Binary relation
Tests the value of an attribute against a constant or against the value of another
attribute, using a relation such as <, >, = ,  ,  , or 
Conjunction
Expression is the conjunction ("AND") of two or more expressions
Disjunction
Expression is the disjunction ("OR") of two or more expressions
Negation
Expression is the negation of another expression
Copyright ©2009 Opher Etzion
7
XPATH filters
Expression returns
Success condition
Example
Node-set
Node-set contains at least one node
/occurrenceTime
Boolean
Value is TRUE
/ranking > 5
Number
Value is non-zero
25
String
String has non-zero length
“OK”
Copyright ©2009 Opher Etzion
8
Stateful filters
First m—This passes the first m event instances
in the context partition window.
Last m—This passes the last (most recent) m
event instances in the window.
Random m—This passes a random set of m
instances.
Copyright ©2009 Opher Etzion
9
Transportation types
Translate
Compose
Aggregate
Enrich
Split
Project
Copyright ©2009 Opher Etzion
10
Transportation logic
Output
Derivation
Partition
selection
Input terminal
filter expression
Derivation
expression
Context
expression
Failure
Copyright ©2009 Opher Etzion
11
Project EPA
Input event
Ticker=“xyz”
MinPrice = 46.7
MaxPrice = 49.5
Vwap = 48.3
Output event
Ticker=“xyz”
Vwap = 48.3
Project EPA
Projection List
{Ticker, Vwap}
Copyright ©2009 Opher Etzion
12
Translate EPA
Input event
Output event
Translate EPA
NumItems =10
Price = 17
OrderType =“Low”
Price = 14.78
Total = 170
Derivation Expression:
OrderType = “Low”
Price = Price/1.15
Total = NumItems * Price
Copyright ©2009 Opher Etzion
13
Enrich EPA
Output
Enrich
EPA
Derivation
Partition
selection
event
Input terminal
filter expression
Context
expression
Query
Parameters
Derivation
expression
Query
expression
Failure
global state
Copyright ©2009 Opher Etzion
14
Multiple results policy for ENRICH
The multiple results policy defines the behavior of an enrich
event processing agent when its query returns more than
one result. The possible policy values are: first, last, every,
and combine.
The four values for this policy are defined as follows:
First—Use only the first row that is returned.
Last—Use only the last row that is returned.
Every—A separate output event is generated, one for each
row that is returned.
Combine—A single output event is returned, but the
derivation rules have access to all the rows when preparing
the output
Copyright ©2009 Opher Etzion
15
Split EPA
Output 1
Split EPA
Output 2
Derivation
Partition
selection
Input terminal
filter expression
Output…
Derivation
expressions
Context
expression
Failure
Copyright ©2009 Opher Etzion
16
Aggregate EPA
Output
Aggregate EPA
Partition
selection
State
accumulation
Input terminal
filter expression
Derivation
Derivation
expression
Context
expression
Failure
Copyright ©2009 Opher Etzion
17
Aggregate EPA
RequestId = R429243
RequestId
= R427763
Driver
= “Arthur
Boyd”
RequestId
= R429531
Driver
= “Arthur
Boyd”
RequiredPickUpTime
= 15:30
Driver = “Arthur Boyd”
RequiredPickUpTime
15:30
RequiredDeliveryTime
==16:30
RequiredPickUpTime
15:30
RequiredDeliveryTime
Store
= “Exotic Flowers” ==16:30
RequiredDeliveryTime
=
Store = “Exotic Flowers” 16:30
Store = “Exotic Flowers”
Aggregate EPA
Derivation expression:
Driver = any(Driver);
AssignmentCount = count();
Segmentation context: Driver
Event Interval Context:
Initator = Assignment
Copyright ©2009 Opher Etzion
18
Driver = “Arthur Boyd”
Assignment Count = 3
Aggregation operators
Name
Argument
Returns
first
Attribute Name
Attribute value taken from the first event that arrived in the
context partition
last
Attribute Name
Attribute value taken from the last event that arrived in the
context partition
any
Attribute Name
Attribute value taken from an arbitrary event in the context
partition
min
Attribute Name
Smallest of all the values of the attribute
max
Attribute Name
Largest of all the values of the attribute
sum
Attribute Name
Sum of all the values of the attribute
avg
Attribute Name
Arithmetic mean of all the values of the attribute
stdev
Attribute Name
Standard deviation of all the values of the attribute
distinct
Attribute Name
Number of distinct values of the attribute
concat
Attribute Name
A list that contains all the attribute values
dConcat
Attribute Name
A list that contains all the distinct attribute values
count
-
Number of event instances in the current context partition
partitionCount
-
Number of active partitions for the current context
globalCount
-
Number of event instances in all active partitions of the current
context
Copyright ©2009 Opher Etzion
19
Compose EPA
Compose EPA
Output
Input terminal
filter expression
Left
Right
Partition
selection
Input terminal
filter expression
Accumulation
and matching
Buffer
specification
Derivation
Match
condition
Derivation
expression
Context
expression
Failure
Copyright ©2009 Opher Etzion
20
Compose EPA example
Registration = ABC123
Registration = ABC123
Arrival = 15:14:12
Entry = 15:14:12
Departure = 15:16:23
Exit = 15:16:23
Compose EPA
OccurenceTime = 15:14:12
OccurrenceTime=15:14:12
Registration = ABC123
Registration = ABC123
OccurenceTime = 15:16.23
OccurrenceTime=15:16.23
Registration = ABC123
Registration = ABC123
Derivation expression:
Registration= Left/Registration
Entry = Left/OccurrenceTime
Exit = Right/OccurrenceTime
Buffer Specification
Left =10 minutes
Right = 0
Match expression:
Left/Registration = Right/Registration
Copyright ©2009 Opher Etzion
21
Compose policies
Left buffer specification—This controls how many event instances from the
Left input terminal should be retained. It can be specified either as a count
of instances or as a time interval.
Right buffer specification—This controls how many event instances from
the Right input terminal should be retained. It can be specified either as a
count of instances or as a time interval.
Unmatched Left Policy—This states what should happen when an event is
evicted from the left buffer if that event hasn’t been matched with anything
prior to eviction.
Unmatched Right Policy—This states what should happen when an event is
evicted from the right buffer if that event hasn’t been matched with anything
prior to eviction.
Match condition—This is the condition used to judge whether an event from
the left stream matches one from the right stream. It can be a simple
equality test, such as Left/A = Left/B, or a more complex expression
involving both events, such as the XPath expression count(Left/A) =
count(Right/B) + 7.
Copyright ©2009 Opher Etzion
22
Derivation expression
A derivation expression is an expression that
assigns values to the attributes of the derived
event. A derivation expression can refer to values
of the input event attributes.
Copyright ©2009 Opher Etzion
23
Header derivation
Attribute Name
Default output value
Settable by derivation step?
eventIdentity
New system-generated identity
No
event Type
Specified by output terminal definition
Yes
detectionTime
See earlier discussion
No
occurrenceTime
As detectionTime
Yes1
eventAnnotation
Not present
Yes
eventCertainty
Not present
Yes
eventSource
Identity of the event processing agent
Yes
Copyright ©2009 Opher Etzion
24
FFD – Bid Request Creator EPA
requestId = R429531
addresseeLocation = 5 Main St
requiredPickUpTime = 15:30
requiredDeliveryTime = 16:30
store = Exotic Flowers
minimumRanking = 5
neighborhood = North East
requestId = R429531
addresseeLocation = 5 Main St
requiredPickUpTime = 15:30
requiredDeliveryTime = 16:30
store = Exotic Flowers
minimumRanking = 5
driversList = Arthur Boyd, Mark Smith
Enrich EPA
Multiple results policy:
Combine
Query parameters:
minimumRanking
neighborhood
Query expression:
SELECT Driver FROM DriverStatus
WHERE Ranking > = ?
AND Location = ?
Derivation expression:
Copy attributes from input;
driversList = Driver;
Driver
Status
Table
Driver
Ranking
Location
Arthur
Boyd
7
North East
Joan
Winter
5
South West
Mark
Smith
5
North East
…
…
…
Copyright ©2009 Opher Etzion
25
FFD – Bid Enrichment
requestId = R429531
driver = Arthur Boyd
committedPickUpTime = 15:30
store = Exotic Flowers
auto = No
Bid enrichment
requestId = R429531
driver = Arthur Boyd
committedPickUpTime = 15:30
store = Exotic Flowers
Enrich EPA
Bid routing
In
Filter EPA
Out
Query Parameter =
store
To manual
Auto = Yes
Store
Reference
Query Expression:
SELECT Auto FROM StoreRef
WHERE Store = ?
NonFilterable
Store
MinRank
Location
Region
Auto
Blooms R Us
3
17 East Ave
South East
Yes
Exotic Flowers
5
33 High St
North East
No
…
…
…
…
…
Copyright ©2009 Opher Etzion
26
To auto
FFD - Assignment manager EPA
Left
Compose EPA
Output
Delivery Bid
Delivery Bid
Bid Request
Right
Delivery Bid
Delivery Bid
Derivation expression:
Copy attributes of right event
Failure
No Bidders alert
Segmentation context: requestId
Event Interval Context:
Initiator = Bid Request
Expiration = +2 minutes
Buffer Specification:
Left = 1 event
Right = 0 events
Failure derivation expression:
Copy requestId
Match expression:
Right/committedPickupTime <
Left/requiredPickupTime + 5 minutes
Unmatched left policy: Fail
Copyright ©2009 Opher Etzion
27
Daily Statistics Creator
Aggregate EPA
driver = Arthur Boyd
driver
= Arthur Boyd
day
= 2009-02-14
day
=
2009-02-14
assignmentCount = 3
assignmentCount = 3
Fixed interval context: Daily
day = 2009-02-14
assignmentMean = 5.2
assignmentSTDV= 1.2
Derivation expression:
day = any(day);
assignmentMean=avg(assignmentCount);
assignmentSTDV=stdev(assignmentCount);
Copyright ©2009 Opher Etzion
28
Streambase example
Copyright ©2009 Opher Etzion
29
Rulecore example
<Rule name="CreateAutomaticAssignments" limit="?" evalMode="once"
level="2">
<Description>This is rule CreateAutomaticAssignments</Description>
<Initialize>
<Assert>
<Event>
<base:XPath>sim:EventDef[@eventType="BidRequest"]</base:XPath>
</Event>
<Expression>
<Property name=”Store”>
<InList name=”AutomaticAssignmentStore”/>
</Property>
</Expression>
</Assert>
</Initialize>
<Views>
<ViewRef name="CreateAutomaticAssignments">
<base:XPath>sim:ViewDef[@name=”CreateAutomaticAssignments”]</base:XPath>
</ViewRef>
</Views>
<Situations>
<SituationRef name="CreateAutomaticAssignments">
<base:XPath>sim:SituationDef[@name="CreateAutomaticAssignments"]</base:XPat
h>
</SituationRef>
</Situations>
<Actions>
<SituationDetected situationName="CreateAutomaticAssignments">
<ActionRef name="CreateAutomaticAssignments" eventVisibility="external">
<base:XPath>sim:ActionDef[@name="CreateAutomaticAssignments"]</base:XPath
>
</ActionRef>
</SituationDetected>
</Actions>
</Rule>
Copyright ©2009 Opher Etzion
30
Apama example
Copyright ©2009 Opher Etzion
31
Esper example
insert into BidRequest(requestId, store, location, pickupTime,
deliveryTime,
storeManual)
select d.requestId, d.store, d.location, d.pickupTime, d.deliveryTime,
s.manual
from
DeliveryRequest d unidirectional,
GPSLocationW g
//,sql:DomainDB['select ranking from Driver where driver = ${g.driver} and
ranking
> ${d.minimumRanking}']
,method:Domain.driverRankLookup(g.driver) r
,method:Domain.isStoreManualLookup(d.store) s
where Geo.distanceKM(g.location, d.location) < 10
and r.ranking >= d.minimumRanking;
Copyright ©2009 Opher Etzion
32
Lecture summary
In this lecture we have looked into:
Filtering
Transformation types
Aggregation derivations
Some code examples
Copyright ©2009 Opher Etzion
33