Theory and Practice, Do They Match ?
A Case with Spectrum-Based Fault
Localization
Tien-Duy B. Le, Ferdian Thung, and David Lo
School of Information Systems
Singapore Management University
1
Spectrum-Based Fault Localization
• Locating buggy program elements by
– Analyzing two sets of execution traces
• Normal traces and faulty traces
– Assigning suspiciousness scores to program
elements
• Two well-known SBFL formulas
– Tarantula
– Ochiai
2
Spectrum-Based Fault Localization
• Xie et al. “A Theoretical Analysis of the Risk
Evaluation Formulas for Spectrum-based
Fault Localization” (TOSEM, 2013)
– Two families of SBFL formulas
• ER1 and ER5 (5 formulas in total)
• Theoretically proven to outperform Ochiai and
Tarantula
– Under the assumption: test coverage is 100%
3
Our Goal
Theoretically Best
SBFL Formula by Xie
et. al.
vs.
Benchmark
Programs
?
Popular SBFL
Formula
4
Notations
Notion
ns (e)
n f (e)
ns (e )
n f (e )
nf
ns
Description
Number of successful test cases that execute e
Number of failing test cases that execute e
Number of successful test cases that do not execute e
Number of failing test cases that do not execute e
n f  n f (e)  n f (e )
ns  ns (e)  ns (e )
5
Popular SBFL Formulas
• Tarantula
n f ( e)
Tarantula (e) 
nf
ns ( e ) n f ( e )

ns
nf
• Ochiai
Ochiai (e) 
n f (e)
n f (n f (e)  ns (e))
6
Theoretically Best SBFL Formulas
7
Dataset
• 10 programs, 199 faulty versions
– Siemens test suite
– Space, NanoXML, XML-Security
• Evaluation Metric
Rank of the first faulty program element
EXAM score 
Total number of executable program elements
– The lower the EXAM score, the better the
performance
8
Results
Technique
Tarantula
Ochiai
ER1a
ER1b
ER5a
ER5b
ER5c
Average % Inspected
23.37%
21.02%
33.34%
21.09%
43.04%
43.04%
54.95%
Standard Deviation
23.44%
21.96%
35.22%
19.48%
19.63%
19.63%
26.83%
• Ochiai has the lowest EXAM score (21.02%)
9
Results
Technique
Tarantula
Ochiai
Average % Inspected
23.37%
21.02%
Standard Deviation
23.44%
21.96%
ER1a
ER1b
ER5a
33.34%
21.09%
43.04%
35.22%
19.48%
19.63%
ER5b
ER5c
43.04%
54.95%
19.63%
26.83%
• Tarantula‘s EXAM score is lower than 4 out of
the 5 theoretically best SBFL formulas
10
Results
Technique
Tarantula
Ochiai
Average % Inspected
23.37%
21.02%
Standard Deviation
23.44%
21.96%
ER1a
ER1b
ER5a
33.34%
21.09%
43.04%
35.22%
19.48%
19.63%
ER5b
ER5c
43.04%
54.95%
19.63%
26.83%
• Wilcoxon signed rank test (significance level of 0.05)
 Ochiai is statistically better than ER5a, ER5b, ER5c
11
100% Test Coverage Assumption
• For 135 out of the 199 faulty versions
– Test coverage < 100%
• Average test coverage of the 199 versions
– 84.97%
 Theoretically best SBFL formulas cannot
outperform popular SBFL formulas
12
Conclusion
• We conduct an empirical study on 10
programs with 199 versions
– Compare performance of 5 theoretically best
SBFL formulas with Tarantula and Ochiai
• We find that:
– Ochiai outperforms all theoretically best formulas
– Tarantula outperforms 4 out of the 5 formulas
– Assumption of 100% test coverage is not valid in
many cases
13
Future work
• In-depth study how test coverage and
other factors affect effectiveness of SBFL
formulas
• Theoretically analyze performance of
SBFL formulas
– Assumption: test coverage < 100%
14
Thank you!
Questions? Comments? Advice?
{btdle.2012, ferdiant.2013, davidlo}@smu.edu.sg
15