Cost & Complexity Trade-offs in
Prognostics
Dr. George Vachtsevanos
School of Electrical & Computer Engineering
Georgia Institute of Technology
Atlanta, GA 30332-0250
NDIA Conference on Intelligent Vehicles
Traverse City, Michigan
June 9-10, 2003
Condition-Based Maintenance
The Opportunity
Condition Based
Maintenance (CBM)
promises to deliver
improved maintainability
and operational availability
of naval systems while
reducing life-cycle costs
The Challenge
Prognostics is the Achilles heel of CBM systems - predicting the
time to failure of critical machines requires new and innovative
methodologies that will effectively integrate diagnostic results with
maintenance scheduling practices
Prognostics
• Objective
– Determine time window over which maintenance must be
performed without compromising the system’s operational
integrity
The CBM Architecture
PEDS Software System Architecture
Hardware
Hardware
••Chiller
Engine
•Sensors
•Sensors
•DAQ
•DAQ
6. Prognostics
Interface
1.1.DDL
DDL
Virtual
Sensor
(WNN)
failure
dimension
2.
2.Data
Data
Preprocessing
Preprocessing
3.3.Mode
Mode
Estimator
Estimator/ /
Usage
UsagePattern
Pattern
Identification
Identification
Event
EventDispatch
Dispatch
Central DB
DWNN
CPNN
Database
Database
Management
Management
5. Diagnostics
5.5.Feature
Feature
4.
Feature
4. Feature
Extraction
Extraction
Extraction
Extraction
Classifier
Classifier
(Fuzzy)
(Fuzzy)
Classifier
Classifier
(WNN)
(WNN)
The CBM Architecture (continued…)
• Diagnostic Models:
– Fuzzy Logic Based
– Wavelet Neural Network Model
– Rough Set Theory based NN Model
• Prognostic Models:
– Dynamic Wavelet Neural Network Model
– Confidence Prediction Neural Network Model
• Physical Models of Failure Mechanisms
System
(FMTV, PLS,
etc.)
Sensors
DAQ/CPU
Interface
Army Vehicle
Systems/
Components
Preliminary Diagnostics
Object Oriented Hybrid
System Models
Intelligent Selection
Layer
Diagnostic Algorithms
Decision Support
Layer
Prognostic Algorithms
Online
Static and Dynamic
Case Library
Offline
Interface
Layer
Multiagent System
Intelligent Agent
Software Repository
Hardware
Model-Based CBM Architecture
Designer
Statistics
Optimization
Performance
Assessment
Module
Prescription
Maintenance
Plan
Maintainer
Community
of Agents –
Multiagent
System
Centralized Control & KB
Architectures
Sensors
UUT
Events
Preprocessing
Knowledge
Base
Diagnostic
Algorithm
Sensors
UUT
Data-mining
Events
Sensors
UUT
Events
Feature
Extraction
Control
Prognostic
Algorithm
Diagnosis
Prognosis
A Generic Central Control and Knowledge Base Framework
Distributed Control & KB
Architectures
Lo
ca
lK
on
tro
l
Lo
lP
ca
stic
rog
no
ms
rith
Alg
stic
s
th m
ori
B
Knowledge
Fusion
Diagnostic Algorithms Diagnosis
Central
Control
Prognostic Algorithms
Prognosis
Central
Knowledge
Base
Local Control
Events
l KB
Local KB
Loca
Loca
Distributed Control and Knowledge
Base Framework
l Co
ntrol
Loca
UUT
Sens
ors
Loca
ts
UUT
Local Prognostic Algorithms
Sensors
Local Diagnostic Algorithms
Lo
lC
ca
no
Even
s
ent
Ev
s
sor
T
Sen
UU
al
c
Lo
g
Dia
o
Alg
l Dia
l Pro
gnos
tic A
gnos
tic A
lgorit
lgorit
hms
hms
Case-Based Reasoning &
Learning
• CBR - an episodic memory of past experiences
• CBR - initial cases by examples
• CBR Methodology:
Indexing (generate indices for classification and categorization)
Retrieval (retrieve the best past cases from the memory)
Adaptation (modify old solution to conform to new situation)
Testing (did the proposed solution work)
Learning (explain failed & store successful solutions)
Case Library
Failure Mode i
Case #
…
1
2
3
S1
Symptoms
S2 … S m
Tests
Prescription
On-board/Off-board Diagnostics
Operational
Performance
Monitoring
Embedded Diagnostics
Operator
Abnormal
Performance
Detection
Periodic
Health
Check
Maintainer
Pre-Diagnostic
Session
Diagnostic Session
Post-Diagnostic
Session
Case-Based Reasoning Architecture
Diagnostics
Manager
Case Based
Diagnostics Reasoner
Knowledge Fusion
Module
Platform Data
Platform Family
Case Library
Platform Historical
Records Database
Current Diagnostics
Session Database
At Platform Diagnostics Session
Topology (Legacy)
1939
DCA
CAN
Embedded
Diagnostics
Interface
Etc..
J-1708
1553
Support Area
Database
*
Updates (brief case model)
*
Subsystem
*
?
Embedded
Diagnostics
Processor
Pressure Sensor
Fuel
Thermocouple
Starter
Current Shunt
Battery
MSD
Embedded
Diagnostics
Interface
*
Diagnostics
Database
Controls
Etc..
SPORT
Portable
Diagnostics &
Maintenance
Aid
Platform
Sensor
Diagnostics
Manager
Etc..
Interactive
Electronic
Technical Manual
(GUI)
Displays
Legend: (Unless Otherwise
Annotated)
Case Based
Diagnostics Reasoner
Uses
Uses
Feeds
Is
MIMOSA
Platform Family
Case Library
*
Knowledge Fusion
Module
Uses
* 1 or more
? 0 or more
Uses
Diagnostic
Test
Selects
Interfaces
Has
Army Central
Database
Updates
Feeds
Embedded
Diagnostics
Data Collector
Uses
Performance Measures
(How to Compare
and
)
Measures
Precision for Prognosis
a measure of the narrowness of an interval in which the remaining life
falls
Reliability
how the system responds to individual component failures
Extensibility or Scalability
how the system can be extended if new components are added
Robustness
how the system tolerates uncertainty
Reuse or Portability
how easy or hard it is to use this system in another problem domain
Accuracy
how an agent improves true positives and true negatives as a result of
learning, self-organization, and active diagnosis
Entropy
a measure of how the system learns and organizes over time.
Decreasing entropy signifies increasing order in a multi-agent system,
resulting in more accurate and timely diagnoses
Network Activity
how much network related activity results if the framework is
implemented for distributed systems
Implementation Issues
Embedded Distributed Diagnostic Platform (EDDP)
• Hardware:
– Modular I/O (e.g. NI’s FieldPoint System, or MAX-IO).
– Embedded PC (e.g. MPC - Matchbox PC of TIQIT or MAXPC of Strategic-Test).
– Network (e.g. Ethernet, PROFIBUS, CAN).
• Software:
– Windows CE, Linux, QNX, VxWorks, or OsX operating
systems.
– Embedded databases (like Polyhedra).
– RAD tools (like eMbedded Visual Studio of Microsoft).
A Possible Agent Node
An Operator Interface
(LCD Display)
A Small PC
(MPC, MAX-PC)
Network (Ethernet, CAN, Profibus)
Distributed I/O System
(FieldPoint)
Sensors
Sensors
Sensors
CBM Performance Assessment
• Objective:
– To assess the technical and economic feasibility of various
prognostic algorithms
• Technical Measures:
– Accuracy, Speed, Complexity, Scalability
• Overall Performance Measure:
– w1Accuracy + w2Complexity + w3Speed + …
(wi - weighting factors)
PM1
PM2
PM3
Algorithm #1
*
*
*
Algorithm #2
*
*
*
Algorithm #3
*
*
*
Performance
Assessment Matrix:
CBM Performance Assessment
• Target Measure:
PM   yr (n f )  y p (n f )   yr (ns )  y p (ns )
• Behavior Measure:
Output y(n)
Real yr(n)
Predicted yp(n)
nf
PM   w(i) yr (i)  y p (i)
i  ns
tpf
• Mean and Variance Measures:
1
E{t pf } 
N
N
 t pf (i )
i 1
Discrete time n
1
V {t pf } 
N
N
2
[
t
(
i
)

E
{
t
}]
 pf
pf
i 1
Complexity/Cost-benefit Analysis
• Complexity Measure
 t p  td 
 computation time 
complexity  E 
 E

t
 time to failure 
 pf 
• Cost/Benefit Analysis
–
–
–
–
frequency of maintenance
downtime for maintenance
dollar cost
etc.
• Overall Performance
Overall Performance = w1accuracy + w2complexity + w3cost + ….
Cost/Benefit Analysis
• Establish Baseline Condition - estimate cost of
breakdown or time-based preventive maintenance from
maintenance logs
• A good percentage of Breakdown Maintenance costs may
be counted as CBM benefits
• If preventive maintenance is practiced, estimate how
many of these maintenance events may be avoided.
The cost of such avoided maintenance events is counted
as benefit to CBM.
Cost/Benefit Analysis (cont’d)
• Intangible benefits - Assign severity index to impact of
BM on system operations
• Estimate the projected cost of CBM, i.e. $ cost of
instrumentation, computing, etc.
• Aggregate life-cycle costs and benefits from the
information detailed above
CINCLANTFLT Study
Question: “What is the value of prognostics?”
Summary of findings:
(1) Notional Development and Implementation for Predictive CBM Based on
CINFCLANTFLT I&D Maintenance Cost Savings
(2) Assumptions
–
–
–
–
–
CINCLANTFLT Annual $2.6B [FY96$] I&D Maintenance Cost
Fully Integrated CBM yields 30% reduction
Full Realization Occurs in 2017, S&T sunk cost included
Full Implementation Costs 1% of Asset Acquisition Cost
IT 21 or Equivalent in place Prior to CBM Technology
(3) Financial Factors
–
–
–
Inflation rate:
Investment Hurdle Rate:
Technology Maintenance Cost:
(4) Financial Metrics:
–
–
NPV
IRR
4%
10%
10% Installed Cost
15 year
20 year
$337M
22%
$1,306M
30%
Concluding Remarks
• CBM/PHM are relatively new technologies - sufficient
historical data are not available
• CBM benefits currently based on avoided costs
• Cost of on-board embedded diagnostics primarily
associated with computing requirements
• Advances in prognostic technologies (embedded
diagnostics, distributed architectures, etc.) and lower
hardware costs (sensors, computing, interfacing, etc.)
promise to bring CBM system costs within 1-2% of a
typical Army platform cost
The Dynamic Case-Based
Reasoning Architecture
Sensory data
Feature interpretation
(static, dynamic, composite)
Case indexing
AS path
Analytical Models and algorithms
Indexing path selection
Indexing rules
PD path
Phase matching
Case retrieval
Case similarity
evaluator
calculation
Case memory
inactive
active
Propagation evaluator
Case adaptation
Model-based reasoner
New case constructor
Test/evaluation
Remembrance calculation
Failure driven learning
Model base
Structure of Static and Dynamic
Case Library
tfailure
Tests
Case #j
Time to Failure
TF1
TF2
TF3
…
…
Dynamic Case Library
Failure Mode i
Conditions
time
y1 y 2 … y m
t1
t2
t3
0
Case Library
Prescription
Tests
…
Static Case Library
Failure Mode i
Symptoms
Case #
S1 S2 … Sm
1
2
3
Prediction-to-Failure Times
output
Mean:
1
E{t pf } 
N
N
t
i 1
pf
(i )
Standard Deviation:
detect
failure
predict
Times
1
S{t df }  [
N
N
 (t
i 1
(i)  E{t pf }) ]
2
pf
1
2
Accuracy
real
predicted
upper bound
real
lower bound
tp
tf
time
DC
predicted
R
accuracy  DC ( yreal ) exp(  R)
An Example
• A defective bearing with a crack causes the machine to
vibrate abnormally
• Vibrations can be caught with accelerometers which
translate mechanical movement into electrical signals
• Bearing crack faults may be prognosed by examining and
predicting their vibration signals
An Experimental Setup
Bearing Vibration Data
Figure 1 Original signals: normal & defective
Figure 2 Spectra: good & defective
0.4
0.2
0.2
0.15
0
0.1
-0.2
0.05
-0.4
-0.6
0
100
200
300
400
500
600
700
800
900
1000
0
6
10
4
8
2
6
0
4
-2
2
-4
0
0
100 Signals
200 300from
400a good
500 and
600 a 700
800 bearing
900 1000
Vibration
defective
Vibration Signals
0
20
0
20
40
60
80
100
40
60 vibration
80
100
PSDs
of the
signals
120
140
120
140
Power Spectrum Densities
Growth of bearing crack fault
Small variations are added
6
7
6
5
5
4
PSD
PSD
4
3
3
2
2
1
0
1
0
20
40
60
Time Window
80
100
0
0
20
40
60
Time Window
80
100
6
6
5
5
4
4
PSD
PSD
Prediction
3
3
2
2
1
1
0
80
85
90
95
Time Window
Prediction by AR
100
105
0
80
85
90
Time Window
95
Prediction by WNN
100
Performance
Similarity
Error
Output
Error
Total
Error
100.00
Time 
Dynamic
Error
1.0
0.1
1.0
N/A
0.20
0.20
0.20
0.20
0.20
1.0
0.4275
0.5200
0.4074
0.3448
0.3200
2.0197
0.1855
0.5500
0.2684
0.2857
0.3200
1.6096
Performance
Measures
TTF
Error Rate
Dynamic
Error
Scaling
Factor
Weighting
Coefficients
AR
Performance
WNN
Performance
1.0
Table: Performances of the AR predictor and the WNN predictor
Overall Performance Error:
---- 2.0197 for the AR predictor
---- 1.6096 for the WNN predictor
Thus, the WNN outperforms the AR in this case