Embark on your "Right Time" journey with
Informatica Intelligent Streaming
Amrish Thakkar, Vikram Kunniyur, Neha Raj
Amrish Thakkar
Vikram Kunniyur
Presenter Name
Principal Product
Manager
Principal Software
Engineer
Lead QA Engineer
@ahthakkar
#INFA16
Safe Harbor
The information being provided today is for informational
purposes only. The development, release and timing of any
Informatica product or functionality described today remain at
the sole discretion of Informatica and should not be relied upon
in making a purchasing decision. Statements made today are
based on currently available information, which is subject to
change. Such statements should not be relied upon as a
representation, warranty or commitment to deliver specific
products or functionality in the future.
#INFA16
Key Technology Trends Impacting Enterprises
New Types of
Deployments
Traditional Use Cases
Next-gen Use Cases
Cloud / Hybrid Use Cases
(Data Warehousing, Business Intelligence,
Data Migration, Data Governance, …)
(Big Data, Data Warehouse Offloading,
Data Lake, IoT…)
(Cloud Data Integration, Big Data in the
Cloud, Data Warehousing in the Cloud, …)
Data
Warehouse
Data
Warehouse
New Types
and Speeds
of Data
Real-time / Streaming
Batch
New Types of
Users
Data Steward
Architect
CIO / VP IT
LOB IT
Data Analyst
#INFA16
Poor performance?
Action: Design change
Best weather for flying?
Time to Act
Weeks to
Months
Action: Flight plan
Operational
Insights
Hours to
weeks
Battery level low?
Action: Land.
Seconds to
hours
Sub-second to
seconds
Descending on object
Action: Elevate
Ad-hoc
Insights
Persistent stalling?
Action: Maintenance
Performance
Insights
Real-time
Insights
On Device
Strategic
Insights
Preventive / Predictive
Applications must have the ability to react
immediately AND learn from the past
Value of data to decision-making
Multi-latency in a single platform: the key differentiator
Time-critical
decisions
Actionable
On Stream
Traditional
“batch” BI
Reactive
Historical
In Batch
Insight to action
Adapted from M. Gualtieri, Forrester 2016
SubSecond
Seconds
Minutes
Hour
Day s
Months
Latency
Adapted from M. Franklin, BIRTE 2015
#INFA16
Relational,
Mainframe
Document
s and
Emails
Driven by Business
The Big Data Streaming Analytics Journey
Social
Media, Web
Logs
Machine
Device, Cloud
Driven by IT
First Pilot(s)
“What’s Big
Data & Hadoop
and how does it
work?”
Data
Warehouse
Optimization
“I can’t do all this
in my data
warehouse & it’s
too expensive”
Big Data
Discovery
Analytics
Real-Time
Operational
Intelligence
Improve
Fraud
Detection
Reduce
Security
Risk
“What big data
insights have
the most
business
impact?”
“I need to
operationalize
insights and
respond to
events in realtime”.
Intelligent Data Lake
Lower Infrastructure Cost
Increase
Customer
Loyalty
Added Business Value
Improve
Predictive
Maintenance
Increase
Operational
Efficiency
#INFA16
Big Data Streaming Analytics Architecture
Documents
and Emails
Social Media,
Web Logs
Machine
Device, Cloud
Data Visualization, Advanced Statistics, Machine Learning,
Analytics Modeling & Validation
Data Governance
Data Stewardship, Master Data Management, Data Security &
Retention, Metadata Management
Big Data Streaming Analytic Pipelines
Data Ingestion
(real-time)
Data Transformations
(Stream handling &
event processing)
Data Delivery
(real-time)
Data Preparation
(ETL, data quality, data
matching
Data Delivery
(near real-time,
batch)
Improve model effectiveness
Data Ingestion
(real-time, batch)
Scalable Storage & Compute
Data Lakes, Data Warehouses,
MapReduce, Spark, Blaze, Kafka
Data for discovery
analytics
Relational,
Mainframe
Streaming analytic models
Data Science
Increase
Customer
Loyalty
Improve
Fraud
Detection
Reduce
Security
Risk
Improve
Predictive
Maintenance
Increase
Operational
Efficiency
#INFA16
Informatica Intelligent Data Platform for
Big Data Streaming Analytics
Social Media,
Web Logs
Machine
Device, Cloud
Data Governance
Data Stewardship, Master Data Management, Data Security &
Big Data
Management
Retention,
Metadata
Management
Big Data Streaming Analytic Pipelines
Data Transformations
Data Delivery
(Stream
handlingHandling
&
Stream
&
Processing
(real-time)
event processing)
Data Ingestion
(real-time)
Improve model effectiveness
Data Ingestion
(real-time, batch)
Data Preparation
(ETL, data Big
quality,
data
Data
matching
Data Delivery
(near real-time,
Management
batch)
Scalable Storage & Compute
Data Lakes, Data Warehouses,
MapReduce, Spark, Blaze, Kafka
Business Process Management
Big Data Ingestion
Documents
and Emails
Data Visualization, Advanced Statistics, Machine Learning,
Analytics Modeling & Validation
Data for discovery
analytics
Relational,
Mainframe
Streaming analytic models
Data Science
Increase
Customer
Loyalty
Improve
Fraud
Detection
Reduce
Security
Risk
Improve
Predictive
Maintenance
Increase
Operational
Efficiency
#INFA16
Business Driver: Harness Big Data Velocity
Become Nimbler
•
•
•
Detect and trigger
responses to threats and
opportunities in near
real time
Automate immediate
actions
Gain an up to date
perspective on what is
happening inside and
outside the company
Increase Data ROI
•
Combine the streaming
present with the
analytical past to get
maximum value for
your data
•
Find new sources of
value in data
•
Iterate fast based on
insights
Win in the marketplace
•
Operationalize insights
quickly
•
Continuously optimize
operation of the
company
•
Develop innovative
business models
#INFA16
IT Driver: Time to Value for Customers
Simplify
Data Scientist Data Analyst Data Steward
Data Engineer
Business
Simplify …
• a seamless & collaborative set of user
tools to efficiently meet business
requirements and optimize
infrastructure capacity utilization
Unify
Unify…
• a shared set of resources to enhance
collaboration, and simplify
development increasing team
productivity
Informatica Intelligent Data Platform
Abstract
Abstract…
• the design and run-time frameworks
optimized for performance, scalability,
re-use, maintenance, and available
skills
#INFA16
Real-time Big Data Collection and Streaming
Apache Kafka
#INFA16
Next-gen Streaming Analytics / IoT
• Streaming analytics
• Frictionless integration
conversion/extension of batch
mappings into streaming
context
• Simplified design process for
new logic generation
• Abstracted from runtime
Social
Media, Web
Logs
Machine
Device,
Cloud
Data Centric
Documents
and Emails
MSG Centric
engines underneath the
covers
Logic
Processing
Distribute
& Manage
Relational,
Mainframe
Real-Time
• Unified UI with multiple
Refine &
Enrich
Collection
capability into BDM, ICS
CDC
VDS
Kafka
JMS
http
Stream
Handling and
Event
Processing
RDMBS
HDFS
HBase
Cloud
Kafka
JMS
http
Reduce
Security &
Safety Risk
Improve
Asset
Utilization
Increase
Operational
Efficiency
Batch DI, BPM, LDM, Governance, Security
Informatica Intelligent Data Platform
Controlled
Release
framework
#INFA16
Intelligent Streaming: Architecture
Administrator Console
Mapping Service
Data
Integration
Service
Rule Builder
Model Repository
Translator
Reference
Table
Developer Tool
Executor
#INFA16
Intelligent Streaming: Runtime
#INFA16
#INFA16
User Groups
Informatica User Groups are a great way for
•
you to invest in your professional development
and learn about new Informatica offerings.
LEARN MORE AT IW16 : Go to the
Solutions Expo Informatica Pavilion /
Ecosystem & Innovation Area:
• Local Chapter Leaders manage each IUG
•
•
•
online and via in person meetings
•
•
•
Network and Socialize
•
Discover how colleagues and peers use
Informatica
•
https://network.informatica.com/welcome/
Find and share content, best practices & tips
Learn about the latest technologies and
solutions from Informatica
Talk to regional user group leaders
Learn about meeting plans
Join your regional user group
• When:
•
•
•
Monday 6:00pm – 8:30pm
Tuesday 10:45am – 2:15pm
Wednesday 10:30am – 1:45pm
• Where:
•
Moscone West Hall Level One
#INFA16