Introduction to Azure Streaming Analytics Warren Sifre April 4, 2017 Who am I? Warren Sifre Data Analytics Solution Architect Allegient Email: [email protected] Twitter: @WAS_SQL LinkedIn: www.linkedin.com/in/wsifre About me… • In the IT Industry since 1998. • Developed system integration solutions against many different database platforms for various applications across many industries. • Passion in Solutions Architecture at both hardware and software levels. • Interests in SQL Server, MongoDB, Hadoop, Python/C#/PowerShell and Information Security (Hacking) • Indy BI PASS User Group Founder and Chapter Leader • PASS SQL Saturday / Indy PASS User Group presenter • MSCE: Data Platforms/Business Intelligence, Teradata 14 CTP and many more… • What is Streaming Analytics? • How it works? • Use Cases • Configuration and Dependencies • SAQL • Demo Typical Environment Challenges… • Real-Time Analytics • Many Steps in between the Source Data and the Visualization/Reporting Output. • Environment Scalability • Months/Years of planning is needed to plan out equipment procurement and scale out to meet increasing demand. • Resource Cost Management • Ideal configuration would require the purchasing of enough equipment to handle peak performance requirements. Although those peak requirements may only be for a few hours of any given day. • Disaster Recovery Strategy • Architecting and maintaining a DR strategy where performance, RTOs, and RPOs are met can be challenging and leave the organization with a lot of underutilized resources. What is Streaming Analytics? WHAT? A way to evaluate data before it has reached its final repository destination. Why? Hours to weeks can be the time it takes for data to be transmitted, received, processed, aggregated, then visualized in the traditional Data Warehouse architecture. Business requirements have changed and the desire to glean insights from this data sooner is now becoming a requirement, not a nice to have. How it can work… Streaming Analytics Job Process and Deliver data to multiple end points Transmit Data Event Hub Queues data for processing Data Factory Gather and process data for Predictive Analytics Process Azure SQL Store data Machine Learning Predictive Analytics Processing Power BI Visualize real-time data stream Reimagined Environment Use Cases • Transportation • Reduce the need to pull vehicles from service for routine inspections by using sensors to determine when actual anomalies are occurring. • Energy • Monitor equipment from central locations such as Wind Turbines and Power Generators, thus reducing time spent on manual/physical inspection or replacement of parts just because of time instead of actual degraded performance. • Manufacturing • Monitor equipment and plant conditions for optimal performance. • Medical Device • Through remote monitoring expensive replacement parts can be ordered closer to the end-oflife of an equipment than by a schedule. This can reduce the cost of having an overstock of parts on-hand. Configuration Options… • Add Input(s) • Data Stream • Event Hub • Blob Storage • IoT Hub • Reference Data • Blob Storage • Add Query o Streaming Analytic Query Language (SAQL) - Similar to T-SQL • Add Output(s) o o o o o o SQL Database Blob Storage Event Hub Power BI Table Storage Service Bus Queue o Service Bus Topic o DocumentDB o Data Lake Store • Settings o Scale – How much processing power desired for SA Job? o Exception Handling– What is the definition of Late Data? What to do with late or out of order data? o Alerts – When do you want to receive a notification? o Functions – AzureML Integration SAQL - Elements DML SELECT FROM WHERE GROUP BY HAVING CASE WHEN THEN ELSE INNER JOIN LEFT OUTER JOIN UNION CROSS APPLY OUTER APPLY CAST INTO ORDER BY ASC, DSC String Functions Len ConCat CharIndex Substring PatIndex Windowing Extensions TumblingWindow HoppingWindow SlidingWindow Date and Time Functions DateName DatePart Day Month Year DateTimeFromParts DateDiff DateAdd Aggregate Functions Sum Count Avg Min Max StDev StDevP Var VarP Scaling Extensions With Partition By Over Temporal Functions Lag IsFirst CollectTop SAQL - Elements DML SELECT FROM WHERE GROUP BY HAVING CASE WHEN THEN ELSE INNER JOIN LEFT OUTER JOIN UNION CROSS APPLY OUTER APPLY CAST INTO ORDER BY ASC, DSC String Functions Len ConCat CharIndex Substring PatIndex Windowing Extensions TumblingWindow HoppingWindow SlidingWindow Date and Time Functions DateName DatePart Day Month Year DateTimeFromParts DateDiff DateAdd Aggregate Functions Sum Count Avg Min Max StDev StDevP Var VarP Scaling Extensions With Partition By Over Temporal Functions Lag IsFirst CollectTop Tumbling Window Fixed window of time with no overlap Hopping Window Fixed window of time with a fix time of overlap Sliding Window A Fixed window time, but a window is defined as the moment an event enters or exits an existing window. Sliding Window A Fixed window time, but a window is defined as the moment an event enters or exits an existing window. Sliding Window A Fixed window time, but a window is defined as the moment an event enters or exits an existing window. Sliding Window A Fixed window time, but a window is defined as the moment an event enters or exits an existing window. Sliding Window A Fixed window time, but a window is defined as the moment an event enters or exits an existing window. Sliding Window A Fixed window time, but a window is defined as the moment an event enters or exits an existing window. Sliding Window A Fixed window time, but a window is defined as the moment an event enters or exits an existing window. Demonstration Helpful Links • More Information on Streaming Analytics • SAQL • https://msdn.microsoft.com/en-us/library/azure/dn834998.aspx • SAQL Query Patterns • https://azure.microsoft.com/en-us/documentation/articles/streamanalytics-stream-analytics-query-patterns/ • Azure Portal Link • Azure Portal Warren Sifre Email: [email protected] Twitter: @WAS_SQL LinkedIn: www.linkedin.com/in/wsifre
© Copyright 2026 Paperzz