Evolution of Business Intelligence
in the Digital Age
Dr. Gautam Shroff
Vice President and Chief Scientist, TCS Research
Copyright © 2013 Tata Consultancy Services Limited
1
Current Business-IT Themes
✔Digital
Re-imagination
Simplification
Governance
Sustainability
2
what is “Digital” ?
data
“raw material”
analytics
=
AI techniques
delivery
“means”
3
The Burden of High Expectations
4
New Digital Business/Service Models
Banking & FS
Payment Hub
to a
Business
Partner
Insurance
& Health
Retail
Manufacturing
Provision
to
Prevention
Blending of
Online &
Physical
3D + Smart
&
Connected
Everything
Life Sciences
Genome-driven Prescriptions
Connected care & prevention
Society
Data-driven
Societal Control Systems
Digitization of Industry and Society
5
‘AI’ Elements of Big Data Analytics
Filtering : Extraction &
Classification
Search
Summaries
Exploration
Decision Making :
Optimization
Collaboration
Listen
Predict
Forecasting, Sequence Analysis
Structure :
Clustering
Rules, Subgroups
Latent Features
Fusion & Reasoning :
Inference – Logical, Probabilistic
& Relational
6
What is ‘BIG’ about Big Data?
data
“raw material”
the big data ‘sales pitch’
analytics
reality today:
delivery
“means”
median 10-100 GB (2012 – 2014)
7
Traditional ‘Business Intelligence’
customer ( x1… xm)
manf. plants ( x1… xm)
days of operation ( x1… xm)
m attributes, each with d cardinality: over o(dm) ‘cubes’
for m=80, d=10 this becomes 1080 > # atoms in the universe
sampling P(X) manually => infinite time / infinite # people!
so what is ‘big’? its not data length, but width
8
TCS Research Programs for Digital
analytics
data
“raw material”
delivery
“means”
?  prescriptions
data “width”
strategic response
ü  predictions
data “length”
real-time response
9
The Offer Economics Challenge
Customer Transactional
History
Offers History
Customer Profile
PIF
Customer Database
4W
Problem
Whom
What
When
Which
•  Should an offer to
be made?
•  Should be in the
offer?
•  Should be an offer
to be made?
•  Offers should be
made?
10
Prescriptive Offer Economics
Prediction
•  Analyze historical customer transactions to unearth customer buying pattern
•  Analyze historical offers detail to find out customer propensity for a type of offer
•  Predict each offer type probability of acceptance to a given type of customer
Optimization
•  For a desired cost – benefit ratio (offer’s cost vs business from customer), find
out unique customer- offer strategy
But what happens in practice? Try, or Simulate
•  Simulate external factors such as competitors offers, market conditions, new
product launches plus internal factors such as customer need, customer buying
behavior (online vs offline) and their impact on prediction
Customer Base = 311,541
Product Categories =838
Product companies = 32,744 companies
Product Brands = 35,691
Offers History = 160,057
Transaction history = 350 million
Precision / recall
68%/14% or 45%/40%
false positives
2% or 18%
(1:3 ground truth) 11
Prescriptive Information Fusion
or ‘Enterprise’ Reinforcement Learning
Y
field data
µ
IEEE FUSION 2014
Pdy ==P(
µ, M | Y
X,Y
P
P(M
) )
Simulation
Model
D
x
Prediction
Model M
Ps = P(M | x, D)
x
Optimization Model
x =x =
argargmin
min E[c(x,
y)]
c(x, y)
Cost Model
c(x, y)
X
recent
decisions
ye =| E[y
P(y
x, M|)x, M ]
µ, M =
argmax δ y Py + δs Ps
x
prescriptions
12
Prescriptive Information Fusion
or ‘Enterprise’ Reinforcement Learning
Y
field data
µ
IEEE FUSION 2014
Pdy ==P(
µ, M | Y
X,Y
P
P(M
) )
Simulation
Model
D Prediction
COMPLEX
SYSTEMS
MODELING
Model
M
x
X
Ps = P(M | x, D)
recent
decisions
yOffer
x,
M|)x, M ]
CAUSAL MODELING – e.g.P(y
Design
e =| E[y
x
µ, M
=
ECONOPHYSICS MODELING - e.g. Energy
Markets
Optimization Model
x =x =
argargmin
min E[c(x,
y)]
c(x, y)
argmax δ y Py + δs Ps
Cost Model
AGENT-BASED HC MODELS - e.g. Campaign Strategy
c(x, y)
x
TEMPORAL MODELING - e.g. Fault Simulation
prescriptions
DISCRETE EVENT MODELS - e.g. Supply Chains
13
Causal Analysis to Derive Prescriptions
movie ticket
gender
books
offer 1
×
region
discount
repeat
buyer
beer
milk
offer 2
P( R | do(discount),
do(meat), do(dinner) )
food
two for one
meat
region
gender
P( gender, region | do(discount),
do(meat), do(dinner) )
offer 3
dinner voucher
14
Prognostic Sensor Fusion for the Industrial Internet
Tele-diagnostics Data
•  Event-detection for understanding operational profiles
ü  Early-warnings of population-level anomalies
ü  Detecting and predicting machine-level anomalies and failures
ü  Contextual supplier intelligence via social listening
ü  Exploratory visual analytics of large-scale multi-sensor data 15
Early-warnings of population-level anomalies
Early-­‐warning of a popula(on-­‐level problem part failure histories
Δ>τ
Failure-based
Reliability Model
Event-augmented
Reliability Model
population-level prognosis / analysis
machine-specific prognosis
real-time sensor data
Early Problem
Detection
Event/Pattern
Diagnosis
16 Ind.CDM 2014 (e.g. ‘poten(al vehicle recall’) AND – why e.g. par(cular plant Early-warnings of population-level anomalies
Early-­‐warning of a popula(on-­‐level problem part failure histories
Δ>τ
Failure-based
Reliability Model
Event-augmented
Reliability Model
population-level prognosis / analysis
machine-specific prognosis
real-time sensor data
Early Problem
Detection
Event/Pattern
Diagnosis
17 Ind.CDM 2014 (e.g. ‘poten(al vehicle recall’) AND – why e.g. par(cular plant ESANN 2015
http://www.cs.ucr.edu/~eamonn/discords
probability
of abnormality
and threshold
{2,2} LSTM network
Office Power Consumption
Detecting Abnormal Behavior using Deep Learning
“Long-short-term memory”
Deep Recurrent Neural Network
*results on {25-20}, {25}
unit networks
18
Detecting specific events from Twitter
Boost
Hash
ICWSM 2012
Fire
Strike
Parse
Classify
+ extract
+ cluster
Flood
Promotion
Discount
Request
What?
Who
Listen
Learn = Focus
Complaint
Why?
Who?
19
Enterprise Contextual Intelligence
Social Profile
Context
Recommendation
Sce
Professional Profile
nari
oM
ode
ling
Enterprise Profile
20
Contextual Intelligence Framework
IEEE WI 2014
P(D | need)
Recommendation
Retrieval &
Relevance Feedback
P(U A ,UC |U B )
P(concept | D)
Content
Models
& Mining
P(need |U A ,U R ,UC )
Context
Models & Mining
(user activity tracking)
Models of
Contextual
Needs
(needs & scenarios)
Information Fusion
21
Operational & Organizational Evolution of BI
Transitioning from a traditional:
Batch ETL
DW
Data Cubes
Statistics (SAS)
business decisions – long cycle, inflexible
to a future analytics life-cycle:
‘in-situ’ Data Sources / Data Streams – minimum ETL into a distributed file system
‘data lake’
Business Data Fusion
22
Visual Data Fusion: “make big data small”
Sensor Data
Bayesian
Network
domain knowledge
(i.e., no structure learning)
interactive queries
without re-visiting large
volumes of sensor data
-  using Bayesian N/W
to summarize data
23
Business Data Fusion Toy Example
§  Inputs:
1.  Census: income distribution by profession by region
2.  Financial (cards/loans/POS): transactions by category, region, time
3.  Location (telecom/apps/social): location by region, time
§  Data fusion platform should processes such data to enable:
queries of the form:
–  which places to advertise a particular product in a given geography
–  which set of product categories to advertise in each location type
ü  which product categories are most likely purchased in a particular
region by a selected set of professions
q  queries are answered probabilistically, without touching raw data
24
Business Data Fusion
IEEE FUSION 2015
Profession by Region
R
Real Estate
Service - high skill
Farming – labor
P
Farming – own land
Trading – small business
R
Manufacturing –
engineers
R’’
R’
R
Manuf. – unskilled
R’’
Retail - employed
R’
Service – low skill
R’
R’’
Government / social
sector
NE
Andaman
MP
J&K
North
South
Metros
East –R
West-R
P
L
F
Bayesian DATA FUSION
Distribution of Financial Transactions
Food
Food
Shelter
Shelter
Personal
Personal
Health
Health
Education
Education
Entertainment
Entertainment
Luxury
Luxury
..
..
..
..
L
F
Location Distribution
Outdoor
Outdoor
Shop
Shop
Personal
Personal
.. ..
……
Restaurant
Restaurant
School/
School/
College
College
.. ..
.. ..
25
Value: Insights and Strategies from Data
1.  Highlight/explain deviations from “known”
Ø  “red vehicles have more accidents {all else being equal}”
2.  Confirm or contradict a hypothesis
Ø  “washing machines run at night/weekend {except in NY}”
3.  Improve an existing business strategy
Ø  “improve claim handling by predicting potenial damages”
4.  Highight possible new business strategies
Ø  “improve cross-sell by explicitly gathering age and income”
Ø  “enter new markets that share similar demographics”
26
Key Messages for Business Intelligence and researchers
(a)  Capability enhancements:
•  advanced analytical techniques
Ø  machine-learning, data fusion
new ‘big data’ technologies
data mining focus - reporting only for
transactional or regulatory reasons
•  statistics is not enough - role of domain models
• 
• 
(b) Cultural changes:
•  share data & results
•  ‘good enough’ is good enough
•  data-driven business experimentation
•  understanding the value of data
27
Thank You
[email protected]