A Comparison of Machine Learning Algorithms for
Proactive Hard Disk Drive Failure Detection
Teerat Pitakrat1,2 , André van Hoorn2 , Lars Grunske2
1
University of Kaiserslautern
AQUA Group
Kaiserslautern, Germany
2 University of Stuttgart
Institute of Software Technology (ISTE)
Reliable Software Systems (RSS) Group
Stuttgart, Germany
June 19, 2013 @ ISARCS 2013, Vancouver
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
1 / 31
Software Failure
Introduction
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
2 / 31
Software Failure
Introduction
“A service failure, often abbreviated here to failure, is an event that occurs
when the delivered service deviates from correct service.”
— Avizienis et al. [2004]
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
2 / 31
Failure Management
T. Pitakrat (U Kaiserslautern)
System recovered
Failure
Prepare recovery
Failure predicted
Availability
Reactive approach
System recovered
Failure
Failure detected
Start recovery
Availability
Introduction
Proactive approach
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
3 / 31
Goals
Introduction
• Comparison of 21 machine learning algorithms for proactive failure
detection in hard disk drives in terms of:
- Prediction quality
- Training time
- Prediction time
• Recommendation for selecting suitable algorithms based on application
constraints
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
4 / 31
Related Work
Introduction
• Machine learning methods for predicting failures in hard drives: a
multiple-instance application [Murray et al. 2005]
• Improved disk drive failure warnings
[Hughes et al. 2002]
• Bayesian approaches to failure prediction for disk drives
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
[Hamerly and Elkan 2001]
Jun. 19, 2013 @ ISARCS 2013
5 / 31
Agenda
1
Introduction
2
Proactive Failure Detection
3
Machine Learning Algorithms
4
Experiments
5
Conclusions
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
6 / 31
Process
Proactive Failure Detection
Processing
Monitoring
Lead time
...
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Prediction
?
Jun. 19, 2013 @ ISARCS 2013
7 / 31
Framework
Proactive Failure Detection
Offline
data
Model
training
Offline process
System
monitoring
Prediction
models
Runtime process
Failure
Runtime
data
Prediction
Non-failure
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
8 / 31
Framework
Proactive Failure Detection
Machine learning
Offline
data
Model
training
Offline process
System
monitoring
Prediction
models
Runtime process
Failure
Runtime
data
Prediction
Non-failure
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
8 / 31
Agenda
Machine Learning Algorithms
1
Introduction
2
Proactive Failure Detection
3
Machine Learning Algorithms
4
Experiments
5
Conclusions
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
9 / 31
General Idea
Machine Learning Algorithms
Machine learning
Offline
data
Model
training
Offline process
System
monitoring
Prediction
models
Runtime process
Failure
Runtime
data
Prediction
Non-failure
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
10 / 31
Categories
Machine Learning Algorithms
• Decision trees
• Rule-based algorithms
• Hyperplane Separation
• Instance-based Learning
• Function Approximation
• Probabilistic models
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
11 / 31
Decision Trees
Machine Learning Algorithms
• C4.5, REPTree, Random forest
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
12 / 31
Rule-based Algorithms
Machine Learning Algorithms
• If x1 ≥ 23 → C1
• If x1 < 17 and x2 < 5 → C1
• If x2 < 2 → C1
• If x1 > 12 and x2 > 12005 → C2
• If x1 > 8 and x2 > 17115 → C2
• ZeroR, OneR, decision table, RIPPER, PART
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
13 / 31
Hyperplane Separation
Machine Learning Algorithms
x
x
x
x
x
x
x
x
x
• Support vector machine, sequential minimal optimization, stochastic
gradient descent
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
14 / 31
Instance-based Learning
Machine Learning Algorithms
x
x
x
?
x
x
x
x
x
x
x x
x
• Nearest neighbour, K-star, locally weighted learning
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
15 / 31
Function Approximation
Machine Learning Algorithms
• Simple logistic regression, logistic regression, multilayer perceptron,
voted perceptron
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
16 / 31
Probabilistic Models
Machine Learning Algorithms
P (C1 |x1 , x2 ) =?
P (C2 |x1 , x2 ) =?
• Naïve Bayes, Multinomial Naïve Bayes, Bayesian network
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
17 / 31
Agenda
Experiments
1
Introduction
2
Proactive Failure Detection
3
Machine Learning Algorithms
4
Experiments
5
Conclusions
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
18 / 31
S.M.A.R.T. Data
Experiments
Self-Monitoring, Analysis, and Reporting Technology
Serial no.
100001
100001
100001
100001
.
.
.
Temp1
10
12
11
9
.
.
.
FlyHeight1
7962
7972
7949
7955
.
.
.
Servo8
0
0
0
0
.
.
.
ReadError17
0
0
8
1280008
.
.
.
WriteError
57005
57005
57005
57005
.
.
.
···
···
···
···
···
···
Data collected from hard disk drives [Murray et al. 2005]
• 178 good drives
• 191 failed drives
An observation is collected approximately every 2 hours
• 68,411 observations in total
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
19 / 31
S.M.A.R.T. Data
Experiments
Healthy drive
Serial no.
100192
100192
100192
100192
100192
100192
Temp1
29
29
58
57
35
37
FlyHeight1
7958
7957
7971
7916
7962
7969
Serial no.
100001
100001
100001
100001
100001
100001
100001
Temp1
10
12
11
9
8
15
23
FlyHeight1
7962
7972
7949
7955
7955
7952
7972
Servo8
0
0
0
0
0
0
ReadError17
0
0
0
0
0
0
WriteError
6
13
36
36
36
37
···
···
···
···
···
···
···
ReadError17
0
0
8
1280008
1280544
1280548
1280548
WriteError
57005
57005
57005
57005
57075
57098
57227
···
···
···
···
···
···
···
···
Failed drive
T. Pitakrat (U Kaiserslautern)
Servo8
0
0
0
0
0
0
0
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
20 / 31
Evaluation Metrics
Experiments
Predicted as good
Predicted as failing
Good drive
Failing drive
True negative (TN)
False positve (FP)
False negative (FN)
True positive (TP)
• True positive rate (TPR) or recall
• False positive rate (FPR)
• Precision
• F-measure
• Receiver operating characteristic (ROC) curve
• Training time
• Prediction time
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
21 / 31
Evaluation Metrics
Experiments
TP
TP + FN
FP
positive rate =
FP + TN
TP
Precision =
TP + FP
2 · precision · recall
F-measure =
precision + recall
True positive rate, recall
False
T. Pitakrat (U Kaiserslautern)
=
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
22 / 31
ROC Curve
Experiments
True positive rate
1
0
T. Pitakrat (U Kaiserslautern)
False positive rate
A Comparison of ML Algorithms for Proactive HDD Failure Detection
1
Jun. 19, 2013 @ ISARCS 2013
23 / 31
Prediction Quality
Experiments
Category
Algorithm
TPR (Recall)
FPR
Precision
F-Measure
Instance-based learning
Decision tree
Decision tree
Decision tree
Rule-based
Rule-based
Instance-based learning
Rule-based
Probabilistic models
Function approx.
Rule-based
Instance-based learning
Probabilistic models
Function approx.
Probabilistic models
Function approx.
Function approx.
Hyperplane separation
Hyperplane separation
Hyperplane separation
Rule-based
Nearest neighbor classifier
Random forest
C4.5
REPTree
RIPPER
PART
K-Star
Decision table
Bayesian network
Multilayer perceptron
OneR
Locally weighted learning
Multinomial naïve Bayes classifier
Logistic regression
Naïve Bayes classifier
Voted perceptron
Simple logistic regression
Stochastic gradient descent
Sequential minimal optimization
Support vector machine
ZeroR
0.974
0.943
0.942
0.913
0.907
0.89
0.875
0.668
0.735
0.585
0.624
0.652
0.252
0.124
0.118
0.094
0.08
0.022
0.015
0.007
0
0.003
0.004
0.008
0.012
0.013
0.012
0.012
0.028
0.078
0.032
0.06
0.082
0.061
0.012
0.022
0.013
0.008
0.001
0
0
0
0.977
0.971
0.95
0.921
0.915
0.921
0.921
0.785
0.592
0.739
0.616
0.552
0.388
0.618
0.457
0.527
0.598
0.792
0.86
0.984
0
0.976
0.957
0.946
0.917
0.911
0.906
0.898
0.722
0.656
0.653
0.62
0.598
0.305
0.206
0.188
0.16
0.14
0.044
0.029
0.014
0
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
24 / 31
Prediction Quality
Experiments
1
0.9
0.8
True positive rate
0.7
NNC
RF
C4.5
REPTREE
RIPPER
PART
KSTAR
DT
BN
MP
OneR
LWL
MNB
LOG
NBC
VP
SL
SGD
SMO
SVM
ZeroR
0.6
0.5
0.4
0.3
0.2
0.1
0
0
T. Pitakrat (U Kaiserslautern)
0.1
0.2
0.3
0.4
0.5
0.6
False positive rate
0.7
A Comparison of ML Algorithms for Proactive HDD Failure Detection
0.8
0.9
1
Jun. 19, 2013 @ ISARCS 2013
25 / 31
Prediction Quality
Experiments
True positive rate
1
0.9
NNC
RF
C4.5
REPTREE
RIPPER
PART
KSTAR
0.8
0
0.1
False positive rate
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
26 / 31
Training and Prediction Time
Experiments
Category
Algorithm
Instance-based learning
Instance-based learning
Instance-based learning
Rule-based
Probabilistic models
Rule-based
Probabilistic models
Probabilistic models
Decision tree
Function approx.
Function approx.
Rule-based
Decision tree
Decision tree
Function approx.
Rule-based
Rule-based
Hyperplane separation
Function approx.
Hyperplane separation
Hyperplane separation
Locally weighted learning
K-Star
Nearest neighbor classifier
ZeroR
Multinomial naïve Bayes classifier
OneR
Naïve Bayes classifier
Bayesian network
REPTree
Logistic regression
Stochastic gradient descent
Decision table
Random forest
C4.5
Voted perceptron
PART
RIPPER
Sequential minimal optimization
Multilayer perceptron
Simple logistic regression
Support vector machine
T. Pitakrat (U Kaiserslautern)
Training (seconds)
Mean
(95% CI)
0.01
0.01
0.01
0.01
0.02
0.38
0.45
1.48
3.56
3.76
7.68
12.20
13.14
14.28
22.48
43.97
98.72
156.60
197.76
271.65
≈ 30 m
(±<0.01)
(±<0.01)
(±<0.01)
(±<0.01)
(±<0.01)
(±<0.01)
(±0.01)
(±<0.01)
(±0.02)
(±0.01)
(±0.01)
(±0.02)
(±0.01)
(±0.01)
(±0.06)
(±0.16)
(±0.26)
(±2.48)
(±0.31)
(±0.33)
(±44.85)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Prediction (seconds)
Mean
(95% CI)
≈ 14 h
≈3h
428.64
<0.01
0.01
<0.01
0.46
0.19
0.01
0.06
0.04
0.06
0.18
0.02
73.80
0.09
0.03
0.04
0.27
0.30
364.26
(±3907.99)
(±177.08)
(±0.16)
(±<0.01)
(±<0.01)
(±<0.01)
(±<0.01)
(±<0.01)
(±<0.01)
(±<0.01)
(±<0.01)
(±<0.01)
(±0.01)
(±<0.01)
(±0.51)
(±<0.01)
(±<0.01)
(±<0.01)
(±<0.01)
(±<0.01)
(±9.46)
Jun. 19, 2013 @ ISARCS 2013
27 / 31
Training and Prediction Time
Experiments
>52403
>11222
2000
Time (seconds)
1500
1000
500
0
M
SV
SL
P
M
O
SM R
PE
IP
R
RT
PA
5
A Comparison of ML Algorithms for Proactive HDD Failure Detection
VP
F
4.
C
R
T
D
D
SG
G
LO ree
T
EP
R
BN
BC
N
R
ne
O
B
N
M
R
ro
Ze
C
N
N
r
ta
KS
L
LW
T. Pitakrat (U Kaiserslautern)
Jun. 19, 2013 @ ISARCS 2013
28 / 31
Training and Prediction Time
Experiments
>52403
>11222
2000
Training
Prediction
Time (seconds)
1500
10
5
Time (seconds)
15
1000
0
R
C
T
D
D
SG
G
LO ree
T
EP
R
R
4.
C
5
4.
T
D
D
SG
G
LO ree
T
EP
F
R
BN
R
ne
B
N
R
BC
N
O
M
ro
Ze
500
0
M
SV
SL
P
M
O
SM R
PE
IP
R
RT
PA
5
A Comparison of ML Algorithms for Proactive HDD Failure Detection
VP
F
BN
BC
N
R
ne
O
B
N
M
R
ro
Ze
C
N
N
r
ta
KS
L
LW
T. Pitakrat (U Kaiserslautern)
Jun. 19, 2013 @ ISARCS 2013
28 / 31
Summary
Conclusions
Algorithms with best prediction quality
• Nearest neighbor classifier
• Random forest
• C4.5
Algorithms with shortest training time
• Instance-based learning
• Multinomial naïve Bayes
• OneR
Algorithms with shortest prediction time
• OneR
• Multinomial naïve Bayes
• REPTree
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
29 / 31
Summary
Conclusions
Algorithms with low false alarm rate
• Support vector machine
• Sequential minimal optimization
Algorithms for online learning approach
• Bayesian network
• OneR
Dataset, program source code and results are available at
• http://aqua.cs.uni-kl.de/HDDDataAnalysis/
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
30 / 31
Future Work
Conclusions
• Apply the approach to monitoring data collected from software systems
- Logfiles (from Computer Failure Data Repository [Schroeder and Gibson 2006])
- Method response time (using, e.g., Kieker framework [van Hoorn et al. 2012])
• Provide lead time prediction
• Make the approach self-tunable
• Develop a reusable online prediction framework
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
31 / 31
Literature
A. Avizienis, J.-C. Laprie, B. Randell, and C. Landwehr. Basic concepts and taxonomy of dependable and secure computing. IEEE Transactions on
Dependable and Secure Computing, 1(1):11–33, 2004. ISSN 1545-5971. doi: 10.1109/TDSC.2004.2.
G. Hamerly and C. Elkan. Bayesian approaches to failure prediction for disk drives. In Proceedings of the 18th International Conference on Machine
Learning, ICML ’01, pages 202–209, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1-55860-778-1.
G. F. Hughes, J. F. Murray, K. Kreutz-Delgado, and C. Elkan. Improved disk-drive failure warnings. In IEEE Transactions on Reliability, volume 51, pages
350–357, 2002. doi: 10.1109/TR.2002.802886.
J. F. Murray, G. F. Hughes, and D. Schuurmans. Machine learning methods for predicting failures in hard drives: A multiple-instance application. Journal of
Machine Learning research, 6:816, 2005.
B. Schroeder and G. Gibson. The computer failure data repository (cfdr): collecting, sharing and analyzing failure data. In Proceedings of the 2006
ACM/IEEE conference on Supercomputing, SC ’06, New York, NY, USA, 2006. ACM. ISBN 0-7695-2700-0. doi: 10.1145/1188455.1188615.
URL http://doi.acm.org/10.1145/1188455.1188615.
A. van Hoorn, J. Waller, and W. Hasselbring. Kieker: A framework for application performance monitoring and dynamic software analysis. In Proceedings of
the 3rd joint ACM/SPEC International Conference on Performance Engineering (ICPE 2012), pages 247–248. ACM, April 2012.
T. Pitakrat (U Kaiserslautern)
A Comparison of ML Algorithms for Proactive HDD Failure Detection
Jun. 19, 2013 @ ISARCS 2013
32 / 31