[Demo] - IntroStreamingAnalytics_Allegient

Introduction to
Azure Streaming
Analytics
Warren Sifre
April 4, 2017
Who am I?
Warren Sifre
Data Analytics Solution Architect
Allegient
Email: [email protected]
Twitter: @WAS_SQL
LinkedIn: www.linkedin.com/in/wsifre
About me…
• In the IT Industry since 1998.
• Developed system integration solutions against many different database platforms
for various applications across many industries.
• Passion in Solutions Architecture at both hardware and software levels.
• Interests in SQL Server, MongoDB, Hadoop, Python/C#/PowerShell and Information
Security (Hacking)
• Indy BI PASS User Group Founder and Chapter Leader
• PASS SQL Saturday / Indy PASS User Group presenter
• MSCE: Data Platforms/Business Intelligence, Teradata 14 CTP and many more…
• What is Streaming Analytics?
• How it works?
• Use Cases
• Configuration and Dependencies
• SAQL
• Demo
Typical Environment
Challenges…
• Real-Time Analytics
• Many Steps in between the Source Data and the Visualization/Reporting Output.
• Environment Scalability
• Months/Years of planning is needed to plan out equipment procurement and scale out to
meet increasing demand.
• Resource Cost Management
• Ideal configuration would require the purchasing of enough equipment to handle peak
performance requirements. Although those peak requirements may only be for a few
hours of any given day.
• Disaster Recovery Strategy
• Architecting and maintaining a DR strategy where performance, RTOs, and RPOs are met
can be challenging and leave the organization with a lot of underutilized resources.
What is Streaming Analytics?
WHAT?
A way to evaluate data before it has reached its final repository destination.
Why?
Hours to weeks can be the time it takes for data to be transmitted, received,
processed, aggregated, then visualized in the traditional Data Warehouse
architecture. Business requirements have changed and the desire to glean insights
from this data sooner is now becoming a requirement, not a nice to have.
How it can work…
Streaming Analytics Job
Process and Deliver data to multiple end points
Transmit Data
Event Hub
Queues data for processing
Data Factory
Gather and process data for
Predictive Analytics Process
Azure SQL
Store data
Machine Learning
Predictive Analytics Processing
Power BI
Visualize real-time
data stream
Reimagined Environment
Use Cases
• Transportation
• Reduce the need to pull vehicles from service for routine inspections by using sensors to
determine when actual anomalies are occurring.
• Energy
• Monitor equipment from central locations such as Wind Turbines and Power Generators, thus
reducing time spent on manual/physical inspection or replacement of parts just because of time
instead of actual degraded performance.
• Manufacturing
• Monitor equipment and plant conditions for optimal performance.
• Medical Device
• Through remote monitoring expensive replacement parts can be ordered closer to the end-oflife of an equipment than by a schedule. This can reduce the cost of having an overstock of
parts on-hand.
Configuration Options…
• Add Input(s)
• Data Stream
• Event Hub
• Blob Storage
• IoT Hub
• Reference
Data
• Blob Storage
• Add Query
o Streaming
Analytic Query
Language
(SAQL) - Similar to
T-SQL
• Add Output(s)
o
o
o
o
o
o
SQL Database
Blob Storage
Event Hub
Power BI
Table Storage
Service Bus
Queue
o Service Bus Topic
o DocumentDB
o Data Lake Store
• Settings
o Scale – How much
processing power
desired for SA Job?
o Exception Handling–
What is the definition
of Late Data? What to
do with late or out of
order data?
o Alerts – When do you
want to receive a
notification?
o Functions – AzureML
Integration
SAQL - Elements
DML
SELECT
FROM
WHERE
GROUP BY
HAVING
CASE WHEN THEN ELSE
INNER JOIN
LEFT OUTER JOIN
UNION
CROSS APPLY
OUTER APPLY
CAST
INTO
ORDER BY ASC, DSC
String Functions
Len
ConCat
CharIndex
Substring
PatIndex
Windowing Extensions
TumblingWindow
HoppingWindow
SlidingWindow
Date and Time Functions
DateName
DatePart
Day
Month
Year
DateTimeFromParts
DateDiff
DateAdd
Aggregate Functions
Sum
Count
Avg
Min
Max
StDev
StDevP
Var
VarP
Scaling Extensions
With
Partition By
Over
Temporal Functions
Lag
IsFirst
CollectTop
SAQL - Elements
DML
SELECT
FROM
WHERE
GROUP BY
HAVING
CASE WHEN THEN ELSE
INNER JOIN
LEFT OUTER JOIN
UNION
CROSS APPLY
OUTER APPLY
CAST
INTO
ORDER BY ASC, DSC
String Functions
Len
ConCat
CharIndex
Substring
PatIndex
Windowing Extensions
TumblingWindow
HoppingWindow
SlidingWindow
Date and Time Functions
DateName
DatePart
Day
Month
Year
DateTimeFromParts
DateDiff
DateAdd
Aggregate Functions
Sum
Count
Avg
Min
Max
StDev
StDevP
Var
VarP
Scaling Extensions
With
Partition By
Over
Temporal Functions
Lag
IsFirst
CollectTop
Tumbling Window
Fixed window of time
with no overlap
Hopping Window
Fixed window of time
with a fix time of overlap
Sliding Window
A Fixed window time, but a window
is defined as the moment an event
enters or exits an existing window.
Sliding Window
A Fixed window time, but a window
is defined as the moment an event
enters or exits an existing window.
Sliding Window
A Fixed window time, but a window
is defined as the moment an event
enters or exits an existing window.
Sliding Window
A Fixed window time, but a window
is defined as the moment an event
enters or exits an existing window.
Sliding Window
A Fixed window time, but a window
is defined as the moment an event
enters or exits an existing window.
Sliding Window
A Fixed window time, but a window
is defined as the moment an event
enters or exits an existing window.
Sliding Window
A Fixed window time, but a window
is defined as the moment an event
enters or exits an existing window.
Demonstration
Helpful Links
• More Information on Streaming Analytics
• SAQL
• https://msdn.microsoft.com/en-us/library/azure/dn834998.aspx
• SAQL Query Patterns
• https://azure.microsoft.com/en-us/documentation/articles/streamanalytics-stream-analytics-query-patterns/
• Azure Portal Link
• Azure Portal
Warren Sifre
Email: [email protected]
Twitter: @WAS_SQL
LinkedIn: www.linkedin.com/in/wsifre