Generating a Test Strategy with
Bayesian Networks, …
and Common Sense
Presentation by:
Jean-Jacques Gras and Rishabh Gupta
Software and Systems Engineering Research Lab
Motorola Labs
TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Contents
•
•
•
•
•
•
•
Introduction
Bayesian Networks
Defect Models
Defect Profile Modelling
Test Strategy Generation
Sample Results
Conclusion and Future Work
TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 2
Introduction
• Problem: Software Testing is Difficult
– Testing is coming late in lifecycle, means under pressure.
– Information about Software weaknesses available too late to
produce effective Test Strategy.
• Solution: Anticipate with Defect Prediction Models
– Need advanced decision support tool to optimise V&V activities.
– Use Predictions to drive V&V Activities
– Use Predictions to generate intelligent Test Strategy by
focussing on High Risk Areas.
– Need to base models on real quality drivers: causal models
– Use Bayesian Networks based Defect Models of each Activity to
predict Quality of software artefacts along lifecycle.
TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 3
Bayesian Nets



A graphical framework to reason about uncertainty
Based on Bayes theorem: P(B|A) = P(B) * P(A|B) / P(A)
Prediction: P(Effect | Cause) but also Inference: P(Cause | Effect)
Nodes =
Uncertain variables
P(C)
A: Training
C: Requirements
A={True, False}
or
A={1, 2,…, n} or
Continuous
P(A)
B: Knowledge
P(B|A)
0 to 1
1 to 2
2 to 3
3 to 4
4 to 5
5 to 6
6 to 7
7 to 8
8 to 9
9 to 10
HOURS
4.00
9.00
15.0
24.0
20.0
12.0
6.00
5.00
3.00
2.00
4.1 ± 2
Arcs =
D:Quality
B=
C=
NPT =
Conditional probability tables
D=
TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Influence relations
P(D|B,C)
High
Low
High
Good Poor
0.9
0.6
0.1
0.4
Slide 4
Low
Good Poor
0.4
0.2
0.6
0.8
End-To-End Defect Prediction
Req Size
B
Requirements
T
A
Des Size
Cod Size
X
B
Requirements
T
A FTR
Latent Defects
-
-
-
X
X
X
-
-
-
X
X
X
-
-
-
X
X
X
B
System
Test
T
A
X
-
B
Design
T A
X
B
X
Design
FTR
T
A
X
-
-
B
Coding
T A
X
B
X
X
Coding
FTR
T
A
X
-
TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
B
Integration
T
ATest
-
-
Slide 5
B
Unit
T Test
A
BN Design Process
• Identify Causal Factors
– Identify Factors that drive Software Quality or are indicators of it
– Each factor is given a precise definition that can be understood
by end-users
• Identify States and Ranges
– Identify Node Type (Discrete or Continuous)
– Identify States (for Discrete) or Ranges (for Continuous)
• Establish Cause-Effect Relations
– Identify relationship between nodes
– Use intermediate nodes where appropriate
• Quantify Cause-Effect Relations
– Establish Conditional Probability Tables
– Must define each relation
TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 6
Additional Model
• Working BN Models may not be available
– Models may not have been yet calibrated
– Subjective data may not be available
• Schedule Pressure
• Process not instituted into organisation
• Need Predictions for Test Strategy.
– Defect prediction for each Component/Sub-System
• Use alternative Model
– to bootstrap the BN deployment
– Multiple defect models provide reinforcements and credibility.
• Defect Profile Model (DPM)
TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 7
DPM Method
•Use Defect Data from past Releases to understand Profile of
Defect Discovery across Lifecycle
Defect Profile (Requirements)
1.00
Des = 86%
UT = 98%
Effectiveness (Accumulated)
0.80
Req = 65%
Cod = 94%
0.60
0.40
Estimate: 40/86% Total Req Faults Inserted ≈ 47
0.20
Actual: Req Faults Found ≈ 40
Prediction: Req Faults Unfound ≈ 7
0.00
INIT
Req
Des
Cod
UT
Development Phase
TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 8
IT
ST
Phase Found
DPM Process
Req
Des
Cod
UT
IT
ST
Sum
Req Des Cod Tot
82
82
0
0
55
31
24
0
72
8
51
13
29
5
16
8
13
1
5
7
7
1
2
4
128
98
32 258
TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 9
Phase Found
DPM Process
Req
Des
Cod
UT
IT
ST
Sum
Total Ratio
Total Found
Total Inserted
Total Unfound
Count
82
31
8
5
1
1
128
TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Ratio
0.64
0.24
0.06
0.04
0.01
0.01
Comp Actual
1.00
61
1.00
21
1.00
11
2
1
1
0.95
93
98.38
5.38
Slide 10
DPM Key Points
• Data availability
– Defect Data is usually collected by mature organisations
• No need for any other metrics
– e.g. size of project, total effort
• Based on Component/Sub-System History
– Generate a Prediction per Component/Sub-System
– One DPM model for each Component/Sub-System
– Calculate average ratio across multiple releases
• Historic Projects must be mature
– Need Defect Data from the Field
– More mature Projects mean better Prediction
TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 11
System Test Strategy Generation
Development
Survey
Services
Service
Service 1
Service 2
Service 3
Service 4
Service 5
Service 6
Service 7
Box A
10%
0%
30%
70%
20%
60%
10%
Box B
55%
17%
20%
50%
50%
50%
20%
Box C
90%
60%
70%
80%
30%
70%
10%
Mapping
components
influence
Component
Component
Models
Component
Models
Models
Latent defects ->
Latent defects ->
Latent defects ->
Test Strategy
Services
Component
latent defects
Service 1
Service 2
Service 3
Service 4
Service 5
Service 6
Service 7
Service 8
# TC
Defects
60
300
240
180
300
120
240
60
1
5
4
3
5
2
4
1
Service
BN Service
BN
Service (FA) Models:
Defect exposure calculation
TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 12
Ranked Services:
Defects exposed +
Suggested # Test Cases
Test Strategy – “Exposure” Matrix
SubSys 2
SubSys 1
SubSys 3
Strategy
Faults left
35
Service 4.1
0%
+
20%
+
80%
*
=
63
Service 4.2
80%
+
60%
+
20%
=
52
Service 4.3
100%
+
80%
+
0%
=
47
*
15
*
75
(Exposed Faults)
…
…
…
…
…
“Exposure” weights from experts
•
•
Method
– Defect “Exposure” weight based on System Design information
– Test Strategy = ranked services (per Test Area)
Benefit
– Identifies weakest services
– Determines number of test cases to write for each Service
TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 13
Applying BTA to Drive Testing
BTA Outputs
Development
Development
Development
Preliminary data
Component
Defects
Preliminary
Test Strategy
Create Test Cases
Updated data
Updated
Test Strategy
TCs
System Test
Pre-test
Test Selection
Suite 1
System Test
New Feature Test
Test Selection
TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 14
Suite 2
End-to-End Defect Prediction Results
Success Criteria: Prediction accuracy of +/- 25% to actuals, 75% of the time
Pilot
Type
Sample Size
Accuracy
A
Development Faults
16 components
+/- 25% to actuals,
88% of the time
B
System Test Effectiveness
15 areas
+/- 25% to actuals,
80% of the time
C
Test Case Selection
8 components
19 services
48% more faults detected than random
selection = 5 yr. expert
D
Third-Party Faults
5 vendors
+/- 25% to actuals,
80% of the time
E
Development Faults
5 component
+/- 16.4%,
100% of the time
7% effort reduction
F
Test Strategy
1 system release
67 test categories
1/3 number of test cases
2X functional areas
TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 15
Test Strategy Impact
Test Cases
2.50
Functional Areas
BTA
Orig
BTA
Orig
Orig
2
# Defects Found
1.00
BTA
2
# Functional Areas
# Test Cases
2.00
1.50
Defects Found
2.5
2.5
1.5
1
1.5
1
0.50
0.5
0.5
0.00
0
0
• Test
• Cases
Functional
• Defects
Areas
Found
– BTA–Test
BTAStrategy
Test
– BTA
Strategy
Test
recommends
Strategy
recommends
would
1/3 the2x
have
number
thefound
number
of all the
of Defects
Test Cases
Functional
asFound
the
Areas
original
by as
thethe
Test
original
original
Strategy
Test
Test
Strategy
Strategy
TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 16
Test Strategy Optimization
700
Original
Optimised
Original unoptimised Test
Strategy Execution
600
Defects Remaining
500
400
Time to remove
all Defects
300
200
Optimised Test Strategy
Execution
Time to remove all
Defects Reduced
100
0
1
6
11
16
21
26
31
36
41
46
51
56
61
66
71
76
81
86
Services Tested
TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 17
91
96 101 106 111 116 121 126 131 136 141
Savings
Future Work & Conclusion
• BN Defect Models
– Better calibration techniques – machine learning
– Simpler “Lightweight” model
• DPM
– A “2D” version to allow earlier predictions with less information
– Integration BN models with DPM approach leverage strength of
each
• Test Strategy
– More detailed exposure information
• Include type or category of Defect
• Exposure per Defect Type, not just Component
– Automatic extraction of Exposure information,
• e.g. through UML / MSC Models
TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 18
Q&A
Thank You
Contacts
[email protected]
[email protected]
TAIC-PART - Cumberland Lodge, UK – 29 Aug 2006
MOTOROLA and the Stylized M Logo are registered in the US Patent & Trademark Office.
All other product or service names are the property of their respective owners. © Motorola, Inc. 2005
Slide 19