WPC042 - Data ON
THE ROAD: the IoT
architecture
Erica Barone
Microsoft Technical Evangelist – IoT & Windows
[email protected]
P R E S E N TA
Agenda
• IoT solutions: the importance of data
• A concrete scenario: MyDriving
• IoT Architecture
 Devices
 Field Gateway
 IoT Hub
 Stream Analytics
 PowerBI
www.wpc2016.it – [email protected] - +39 02 365738.11
2
IoT: gain value from Your Things
Things
Connectivity
Data
Analytics
Action
IoT is not just about collecting data,
it’s about how you can use these data to gain VALUE
Money
New Business
Models
www.wpc2016.it – [email protected] - +39 02 365738.11
Happy
Customers
3
Data...?
• Where can I find useful data?
• How can simple data transform my
business?
• How can I manage these data to
transform into concrete value?
Concrete scenario: car insurance
DRIVER AGE
(less than 21 years old,
more than 65 years old, etc)
KM per year
VEHICLE CHARACTERISTICS
(cubic capacity,
kind of vehicle, etc)
COVERAGE PARAMETERS
(insurance ceilings,
weathering, vandalism,
fire and theft, etc)
CITY OF RESIDENCE
(and of main drives)
TRAFFIC COLLISIONS
In the recent period
REAL drivers
• Speed along the road
Data could
help
improve the
insights
• Usage of the car: trip frequency and trip duration
• Time slots of the trips: early morning, late afternoon, night, etc
• Top paths: traffic, kind of roads, etc
• Drive style: avg speed, hard stops, hard accelerations, etc
• Integration with public traffic systems: autovelox, red light
cameras, speed limits on the top paths, etc
Azure IoT Reference Architecture
Data Processing, Analytics and Management
Device Connectivity and Management
IP capable
devices
Business Connectivity
Provisioning API
IoT Client
App Backend
Solution UX
Personal
mobile
devices
Device Registry
Device State Store
Existing IoT
devices
Cloud
Gateway
IoT Client
Stream Processors
Gateway
Low power
devices
IoT Client
Storage
Analytics &
Machine Learning
Business
Integration
Connectors
and
Gateway(s)
Business
systems
Data Path
Optional solution component
Azure IoT solution component
www.wpc2016.it – [email protected] - +39 02 365738.11
8
DEMO
MyDriving
www.wpc2016.it – [email protected] - +39 02 365738.11
9
MyDriving Architecture
On Boarding Diagnostic Device
For Android & Windows
(Bluetooth)
For iOS (Wi-Fi)
Device connectivity and management
End user IoT gateway scenarios
Device Connectivity and Management
IP capable
devices
IoT Client
Existing IoT
devices
IoT Hub
IoT Client
Gateway
Low power
devices
IoT Client
Security and Isolation of Devices
Bridge networks and isolate public Internet
Ensure cloud security and encryption
Integration with any device
Low resource devices
Device agents can be simplified
Connect new and legacy devices
Cloud Offload
Run logic on premises to reduce network and cloud
consumption
Low-latency / critical messages
Segregate message types
Edge processing
Enable time-sensitive decisions
Run edge analytics
Auditing - Central choke point to apply data policies
Batching
Optimize bandwidth and messages
www.wpc2016.it – [email protected] - +39 02 365738.11
12
Field Gateway
WHY A FIELD GATEWAY?
• Most ODB dongles are not internet-capable
• The wireless connection used by ODB devices might not be secure
• The device MUST be able to use one of the secure protocols supported by IoT Hub (HTTPS, AMQPS,
MQTT)
WHY A PHONE AS FIELD GATEWAY?
• Most drivers already have a phone to be used as field gateway
• It’s easy and cheap: you just have to download and install the app
• The phone works as an opaque field gateway
• Additional data (like GPS data) can be collected using the phone, and can
be merged with the ones from ODB device
• The phone caches local data for later visualization purposes
• (if you don’t have a ODB device, the myDriving app simulates ODB data)
Xamarin App
myDriving App – data flow
1
2
3
User Authentication
Device provisioning
DeviceClient creation using connection string
Polling on ODB device every second to retrieve data
Local GPS data every 3 secs / 5 m
Integration: message setup and sendEvent to IoTHub
IoT Field Gateway
•
•
•
•
•
Data Visualization
• PastTrips
• Profile view (aggregate data)
Azure IoT Hub
Hyper scale IoT solution
Cloud-scale messaging
Two-way communication
Per-device authentication
Multi-protocol support
Cloud-scale gateway
www.wpc2016.it – [email protected] - +39 02 365738.11
15
IoT Hub
Security
Operational monitoring
Device SDKs
Provisioning
• Create an entry for the
device in the Device Identity
Registry of our IoTHub. This
creates a unique identity
identity with an associated
key.
• Install the device key on the
device, to enable it to
connect to IoT Hub
Azure IoT Hub
Telemetry
HTTPS,
AMQPS,
MQTT
Device / Gateway
Device Telemetry
(All Devices)
Command and Control
(Per Device)
IoT Cloud Solution
Device (Object)
Service
Properties
Device
App
Device
Properties
Tags
Your Code
Device Management Objects
DM Client
Library
Registry Manager API
Create, Update, Delete Devices
Management
LWM2M over
CoAP/TCP
Application
Your Code
Device Query API
Tags, Device and Service Properties
Device Jobs API
Create, Monitor, Cancel
Device Management APIs
www.wpc2016.it – [email protected] - +39 02 365738.11
17
Device-to-cloud messages
IoT Hub
D2C send
endpoint
Interface
C2D queue
endpoint
AMQP, HTTPS, MQTT device-side endpoint
AMQP service-side endpoint
Device and service SDKs
Device id
C2D send
endpoint
Device …
Compatible with Event Hubs
Device …
Device …
Partitioned receiver, client check-pointing
Integrations with Azure Stream Analytics, Storm, …
100% compatible with Event Hubs receivers
IoT Hub
management
Consumer Group j
Consumer Group 1
IoT Hub services for D2C
Millions of simultaneously connected devices
Per-device authentication
Connection-multiplexing:
C2D and D2C traffic
Across multiple devices for gateway scenarios
D2C receive
endpoint
Partition 1
D2C send
endpoint
Partition 2
Callback partition 1
Callback partition n
Partition n
D2C receive endpoint
www.wpc2016.it – [email protected] - +39 02 365738.11
Worker 1
Worker k
Callback partition z
Callback partition 2
18
IoT Hub - Device Management concepts
www.wpc2016.it – [email protected] - +39 02 365738.11
19
Data Processing
Data Processing, Analytics and Management
Azure Stream Analytics
• Real Time analytics
Provisioning API
App Backend
Solution UX
Device Registry
• No challenges with scale
Device State Store
Cloud
Gateway
Stream Processors
Storage
• Fully managed
Analytics &
Machine Learning
Business
Integration
Connectors
and
Gateway(s)
www.wpc2016.it – [email protected] - +39 02 365738.11
• Rapid development
• Built in temporal semantics
20
Azure Stream Analytics
Components of an Azure Stream Analytics Application
Azure IoT Hub
Azure Blob Storage
Events
Azure SQL DB
Have a defined schema and
are temporal (sequenced in
time)
Azure Blob Storage
Azure Event Hubs
Reference Data
Query runs continuously against incoming stream of events
www.wpc2016.it – [email protected] - +39 02 365738.11
21
Stream Analytics in myDriving
SA Job
Path
Output Name Usage
mydriving-sqlbi
Real time hot-path PowerBISink
SQLSink
Real-time aggregate data for consumption by
Power BI.
Real-time aggregate data for consumption by App
Service APIs (and thus the mobile app).
mydrivinghourlypbi
Real time hot-path PowerBISink
A second hourly aggregation of real-time data for
Power BI.
mydrivingvinlookup
Extensions
EHSink
Runs a simple query to extract the VIN number
from a trip, and sends to Event Hubs for
extensibility
mydriving-archive
Aggregate coldpath
BlobSink
Sends real-time data to blob storage where it
accumulates for longer periods of time, for later
processing by HDInsight and visualization in Power
BI and for retraining Machine Learning
Stream Analytics: mydriving-sqlbi
INPUT
OUTPUT
PowerBI – data visualization
IoTHub data
SQL DB – data consumed by
App Service
QUERY
TripPointRaw
Purpose: shaping the incoming IoT data
TripPointAgg
Purpose: aggregation (max, min, avg)
RoughDrivingStats
Purpose: creating driving statistics
Query to route the specific data to PowerBISink
Query to route the specific data to PowerBISink
Stream Analytics - Queries
Tumbling Window
Tumbling windows:
A 20-second Tumbling Window
1
5
4
6
2
8
6
5
3
6
1
Time
(secs)
1
5
4
6
2
8
•
Repeat
•
Are non-overlapping
An event can belong to only one tumbling window
Query: Count the total number of vehicles entering each
toll booth every interval of 20 seconds.
6
5
3
6
1
SELECT TollId, COUNT(*)
FROM EntryStream TIMESTAMP BY EntryTime
GROUP BY TollId, TumblingWindow(second, 20)
www.wpc2016.it – [email protected] - +39 02 365738.11
25
Business Connectivity
Business Connectivity
Solution UX
Business
Integration
Connectors
and
Gateway(s)
Personal
mobile
devices
Business
systems
PowerBI
•
Live dashboards provide a 360º view of
your business
•
Track your data in real-time with support
for streaming data
•
Drill through to underlying reports to
explore in more detail
•
Pin new visualizations and KPIs
to monitor performance
www.wpc2016.it – [email protected] - +39 02 365738.11
26
PowerBI
• Dataset
• Report
• Dashboard
• Workspaces
PowerBI
Pre-built content packs, consisting of dashboards
and reports, for popular SaaS solutions
Real-time dashboard updates
Secure, live connection to your data sources,
on-premises and in the cloud
Intuitive data exploration by using natural language
query
Integration with familiar Microsoft products, and
commitment for scale and availability in Azure
Fast deployment, hybrid configuration, secure, and
integrated with existing IT systems
www.wpc2016.it – [email protected] - +39 02 365738.11
28
Power BI
Data sources
Power BI service
SaaS solutions
E.g. Marketo, Salesforce, GitHub,
Google analytics
Content packs
Natural language query
Sharing & collaboration
On-premises data
E.g. Analysis Services
Live dashboards
Organizational content packs
Corporate data sources, or external data
services
Visualizations
Azure services
E.g. Azure SQL, Stream Analytics
Reports
Excel files
Workbook data or data models
Power BI Desktop files
Related data from files, databases,
Azure, and other sources
01001
10101
Datasets
Data refresh
www.wpc2016.it – [email protected] - +39 02 365738.11
29
DEMO
MyDriving
www.wpc2016.it – [email protected] - +39 02 365738.11
30
What’s more?
What’s more?
mydriving-vinlookup
Event Hub
Storage - Blob
Xamarin App
(device)
Service Fabric
Machine Learning
mydriving-archive
mydrivingAnalyticsDB
IoT Hub
mydriving-sqlpbi
mydrivingDB
Data Factory
Car (Sensor)
mydriving-hourlypbi
HDInsight
Power BI
Follow the next session to learn more:
WPC047 - Data ON THE ROAD: the
Azure part
Jessica Tibaldi
Q&A
www.wpc2016.it – [email protected] - +39 02 365738.11
33
OverNet Education
Contatti
OverNet
Education
[email protected]
www.overneteducation.it
Tel. 02 365738
@overnete
www.facebook.com/OverNetEducation
www.linkedin.com/company/overnet-solutions
www.wpc2016.it
www.wpc2016.it – [email protected] - +39 02 365738.11
34