Centinel:
Streaming Data Handler
September 07th, 2015
1
An Overview
2
Centinel: Streaming Data Handler
The Centinel project aims at providing a distributed, reliable framework for efficiently collecting, aggregating and
sinking streaming data across Persistence DB and stream analyzers (example: Graylog, Elastic search, Spark, Hive etc.).
This framework enables SDN applications/services to receive events from multiple streaming sources (example: Syslog,
Thrift, Avro, AMQP, Log4j, HTTP/REST etc) and execute actions like network configuration/batch processing/real-time
analytics.
Core features of Centinel framework are:
 Stream collector - Collecting, aggregating and sinking streaming data
 Log Service - Listen log stream events coming from log analyzer
 Log Service - Enables user to configure rules (example: alerts, diagnostic, health, dashboard etc.)
 Log Service - Performs event processing/analytics
 User Interface - Enable set-rule, search, visualize, alert, diagnostic, dashboard etc.
 Adaptor - Log analyzer plug-in to Graylog and a generic data-model to extend to other stream analyzers (Logstash
etc.)
 REST Services - Northbound APIs for Log Service and Steam collector framework
 Leverages - TSDR persistence service, data query, purging and elastic search
3
Centinel: High Level Architecture
4
Centinel: Functional Overview
Feature
Details
Streams
 Mechanism to route messages into categories in real time while they are processed like stream for
audit logs( install bundle etc.)
 Rule configuration includes message, level, source etc.
 Streams are generated by GrayLog server as per user-defined rules. Event-handler module handles
streams from GrayLog server and persist it into Centinel
Alerts
 Log alerts :
 Alerts get generated based on specific event matching in real-time
 Alert condition types : Message count condition, Field value condition, Field string value condition
 Alerts get cleared if specified condition does not persist. Alert check interval time is configurable,
default is 60 seconds.
 Common alert operations like manual acknowledge , delete, filter , sort etc. will be supported
Search
and
Analyze
 Support for search query language.
 Time range for search can be specified
 Visualization includes histogram
5
Centinel: Functional Overview (continued)
Feature
Details
Dashboard
 Build pre-defined views on data by adding widgets. Domain expert can define search query and save
results on dashboard.
 Search result type : Search result counts, Search result histogram charts, Field value charts,Quick
value results
Diagnostics
 Enables , user to specify/configure group of log messages (events or specific conditions ) as single
rule. Order of messages and condition for each messages can also be configured.
 Notification will be persisted on db if log messages are coming out of order/sequence
 On clicking notification on UI, popup window will open which displays expected order and received
order of log messages.
 User will be able to scroll popup window to see individual received log message.
 Event processor has intelligence to verify sequencing of messages and generate diagnostic reports
 Success notification is displayed when all specified conditions are true.
Health
 User specify list of messages with conditions for feature.
 Feature health will be changed to Critical, Major, Warn on reception of all messages specified
6
Centinel: Project links
Project Main page
Project Proposal
Release plan
Project Email archives
Email
Repository
https://wiki.opendaylight.org/view/Centinel:Main
https://wiki.opendaylight.org/view/Project_Proposals:Centinel
https://wiki.opendaylight.org/view/Centinel:Beryllium_Release_Plan
https://lists.opendaylight.org/pipermail/centinel-dev/
[email protected]
https://git.opendaylight.org/gerrit/centinel
7
Design Approach
8
Incremental updates on design document
Legend
DOCSIS: Data Over Cable Service Interface
Specification
AAA: Authentication, Authorization & Accounting
FRM: Forwarding Rules Manager
ALTO: Application Layer Traffic Optimization
GBP: Group Based Policy
AuthN: Authentication
IoTDM: Internet of Things Data Broker
BGP: Border Gateway Protocol
LACP: Link Aggregation Control Protocol
CAPWAP: Control and Provisioning of Wireless
LISP: Locator/Identifier Separation Protocol
Access Points
MAPLE: Maple Programming
COPS: Common Open Policy Service
DIDM: Device Identification and Driver management NIC: Network Intent Proposal
OVSDB: Open vSwitch DataBase Protocol
DLUX: OpenDaylight User Experience
OPFLEX: Extensible Policy Protocol
DDoS: Distributed Denial Of Service
“LITHIUM”
DLUX
DDoS Protection
OpenStack
Neutron
PCEP: Path Computation Element Protocol
PCMM: Packet Cable MultiMedia
Plugin2OC: Plugin To OpenContrail
SDNI: SDN Interface (Cross-Controller Federation)
SFC: Service Function Chaining
SNBI: Secure Network Bootstrapping Infrastructure
SNMP: Simple Network Management Protocol
SXP: Source-Group Tag eXchange Protocol
TSDR: Time Series Data Repository
TTP: Table Type Patterns
USC: Unified Secure Channel
VTN: Virtual Tenant Network
Network Applications
Orchestrations and Services
VTN Coordinator
SDNI Wrapper
AAA- AuthN Filter
NB APIs
OpenDaylight APIs (REST)
Topology
ALTO
Inventory
FRM
DIDM
GBP
Service
SFC
SDNI
Aggregator
L2
Switch
VTN
Manager
Neutron
Service
NIC
DOCSIS
Abstraction
BGP
PCEP
Plugin20C
TCPMD5
OVSDB
LISP
Service
IoTDM
VPN Service
Reservation
Applications
MAPLE
Topology
Processing
Discovery
NETCONF
OpenFlow Enabled
Devices
SNBI
PCMM/C
OPS
BGP
PCEP
Plugin20C
Open vSwitches
LACP
Centinel
Persistence
MD-SAL / Yangtools
OpenFlow
USC Manager
SNMP
OVSDB
LISP
IoT
USC
CAPWA
P
SXP
TSDR
OPFLEX
Plugin Services
Controller platform
SB interfaces &
protocols plugins
Additional Virtual &
Physical Devices
9
Introduction and Data Models
Centinel framework enables SDN applications/services to
receive events from multiple streaming sources (example:
Syslog, Thrift, Avro, AMQP, Log4j, HTTP/REST etc) and
execute actions like network configuration/batch
processing/real-time analytics.
The entire architecture can be divided into the following
modules:
 User interface
 Centinel NB APIs
 Stream Analyzer Service (LAAS and Plugin)
 Log Service (“Configure Rules” and “Event handler”
Modules)
 Stream Collector Service (Flume and Sqoop based)
 Leverage persistence database service
This design document depicts the subset of functionality
(Alert) and the same is subject to follow-up add-on
updates.
Incremental updates on design document
The stream data handler is a framework for collecting,
aggregating stream data across Persistent DB and stream
analyzer.
To achieve the same, the entire model could be broken up
into two parts:
 Rules – Configurable from Northbound
 Notifications/Events – Output from stream analyzer
10
Rules
The Northbound of Centinel will
allow operators to set Rules for
collecting, aggregating and analysis of
streaming data.
The following diagram explains the
Centinel data model for log services
where in the BaseRule is the basic
model object that is imported by
BaseAlert which is further used by
different types of rules.
BaseRules model object is the basic
data model that will be
shared/imported by all the other
“Log service” features like Alerts,
Streams, Health, Dashboard and any
other feature that Centinel
framework may want to support in
future.
11
Incremental updates on design document
Events
When a user configures a rule
successfully, notifications/Events
corresponding to the rule will be
supplied by the stream analyzer.
The diagram below shows the data
model for events (notifications and
alert) received from stream analyzer.
The base model for
notifications/events is “BaseEvent”
that contains event type and stream
and is further extended by alarm that
provides detailed information about
the event that has occurred for
example the stream which raised the
alarm, alert condition, feature or
module from which alert was raised
etc.
12
Alerts
The diagram below shows the class diagram
for RPCs and listeners. It defines the following
interfaces and java classes for handling the
Alerts:
 LAAS interface and its implementation that
provides log analyzer abstraction layer and
also publish notifications.
 Plugin class that provides alarm call backs
from the analyzer (grey log server).
 AlertService interface and its configuration
Implementation that defines and
implements CRUD on Alert Rules and also
write it in md-sal data-store.
 AlertServiceListener class that listens to
any data change in md-sal database.
 ServiceAggregator and
ServiceEventHandler classes that registers
for notification and receive notification
published by the LAAS as DTO
 ServicePersistData class that stores the
DTO from event handler into the persistent
database
13
Incremental updates on design document
North Bound APIs
Feature
Details
NotifyAlarm()
RPC exposed by LAAS. Plugin use it to notify alarms
SetAlertRule()
Set rules for alerts
RemoveAlertRule()
Remove rules for alerts
GetAlertRule()
Get all alert rules
getAlarms()
Retrieve recent 300 alarm.
14
Incremental updates on design document
Sequence diagram for Rule Handling
Following is the sequence
diagram for the Rule handling
part.
It describes the flow for “How
is a rule for Alert getting
configured”.
15
Incremental updates on design document
Sequence diagram for Event Handling
Below is the sequence diagram for the Rule handling part.
It describes the flow for processing an Event (Alarm in this case) before it get stored in the persistent data store.
16
Incremental updates on design document
Thank You
17