Using Logic Criterion Feasibility
to Reduce Test Set Size While
Guaranteeing Fault Detection
Gary Kaminski and Paul Ammann
ICST 2009
March 24 Version
Motivation
Current logic criteria:
– generate large test sets (Combinatorial)
or
– do not guarantee detecting logic faults (RACC)
Goal:
- generate smaller test sets while still
guaranteeing fault detection
Assumption:
- restrict attention to minimal Disjunctive Normal Form
(DNF) Boolean predicates tested in isolation
A Word About Infeasibility
• Infeasible Test Requirements are a hassle!
– They can bloat test sets
– They can thwart subsumption hierarchies
– Example: RACC and CACC
• May be infeasible to satisfy RACC, but feasible to satisfy CACC
• RACC subsumes CACC, yet for a literal in a predicate, CACC
may yield a test case when RACC does not
• Coverage Criteria for Detecting Logic Faults
–
–
–
–
If all test requirements feasible, simple criteria are enough
More complex criteria needed to fill in the gaps
This paper analyzes feasibility at a “low” level
Result: Minimal, fault-detecting test sets
Building Test Sets Guaranteed
to Detect Faults (Current)
Apply
Criterion 1
T1
Apply
Criterion 2
T2
Test Set =
Predicate P:
ab + a!c
T1
Apply
Criterion 3
+ T2 + T3
T3
Apply Each Criterion to P, Component by Component
• If criterion feasible on component, generate test
• If criterion infeasible on component, satisfy as much as possible
Result: Tests from all Criteria on all Components
• Criteria are all necessary; but individual tests may be unnecessary
Analyzing Criterion Feasibility at
Component Level
Extract
Components
Criterion 1
Feasible?
Yes
Apply
Criterion 1
T1’
No
Criterion 2
Feasible?
No
Predicate P:
ab + a!c
Yes
Apply
Criterion 2
Apply
Criterion 3
T2’
Test Set =
T1’ + T2’ + T3’
T3’
Criterion Feasibility Analyzed, Component by Component
• If criterion feasible on component, generate test and FINISH
• If criterion infeasible on component, partially satisfy and go to next criterion
Result: Every resulting test has a reason for being there
Note: Some details glossed over in this figure…
Minimal DNF
• Terms separated by OR, literals by AND
ab + a!c vs. a(b + !c)
• Make each term true and other terms false
ab + ac vs. ab + abc
• Impossible to remove a literal or term without
changing the predicate
ab vs. abc + ab!c
Minimal DNF Logic Faults
Original: ab + bc
•
•
•
•
•
Literal Insertion Fault:
Literal Insertion Fault:
Literal Reference Fault:
Literal Reference Fault:
Literal Omission Fault:
abc + b!c
ab!c + b!c
ac + b!c
a!c + b!c
b + b!c
A test set detecting these faults also detects others
Lau and Yu’s Fault Hierarchy
• A test set that guarantees detection of a source
fault guarantees detection of a destination fault
• Ignores effect of criterion feasibility
LIF
LOF
LRF
TOF
LNF
ORF.
ORF+
TNF
ENF
Unique True Points and
Near False Points
• UTP: An assignment of values such that
only one term evaluates to true.
ab + !ac: 110 and 111 are UTPs for ab
• NFP: An assignment of values such that
the predicate evaluates to false but when
a literal is omitted, it evaluates to true.
ab + !ac: 100 and 101 are NFPs for b
MUTP Criterion
• Find UTP tests for each term such that all
literals not in the term attain 0 and 1.
• Detects LIF and if feasible, detects LRF
• Inexpensive to satisfy
• Feasible for ab + !ac
ab – 110, 111
!ac – 001, 011
• Infeasible for ab + ac
ab – 110
CUTPNFP Criterion
• Find a UTP - NFP pair such that only the literal of
interest changes value.
• Detects LOF and if feasible, detects LRF
• More expensive to satisfy
• Feasible for b in ab + ac
UTP for ab is 110
NFP for b in ab is 100
• Infeasible for b in first term of ab + b!c + !bc
UTP for ab is 111
NFP for b in ab 100 (101 makes !bc true)
MNFP Criterion
• Find NFP tests for each literal such that all
literals not in the term attain 0 and 1.
• Detects LOF and if feasible, detects LRF
• Most expensive to satisfy
• Feasible for a in first term of ab + ac
010, 011
• Infeasible for a in first term of ab + !ac
010 (011 makes !ac true)
MUMCUT Criterion
• Combine CUTPNFP, MNFP, and MUTP
- detects LIF, LRF, and LOF but expensive
- without considering feasibility need all 3
criteria to detect LRF
• Other criteria require less inputs but do not
guarantee fault detection (RACC)
• Can we reduce MUMCUT test set size
while still guaranteeing LRF detection?
MUTP Feasibility and LRF
If MUTP is feasible for a term: Black – Green
- MUTP detects LRF
- CUTPNFP not needed to detect LRF
- MNFP not needed to detect LRF
For
Each
Term
MUTP
feasible?
For
Each
Literal
In Term
CUTPNFP
feasible?
Test Set =
MUTP + NFP
Test Set =
MUTP + CUTPNFP
MNFP
Test Set =
MUTP +
MNFP
CUTPNFP Feasibility and LRF
If MUTP is infeasible for a term but CUTPNFP is feasible
for a literal in the term: Black – Red – Black - Green
- MUTP does not detect LRF
- CUTPNFP detects LRF
- MNFP not needed to detect LRF
For
Each
Term
MUTP
feasible?
For
Each
Literal
In Term
CUTPNFP
feasible?
Test Set =
MUTP + NFP
Test Set =
MUTP + CUTPNFP
MNFP
Test Set =
MUTP +
MNFP
MNFP Feasibility and LRF
If MUTP is infeasible for a term and CUTPNFP is infeasible
for a literal in the term: Black – Red – Black – Red – Black
- MUTP does not detect LRF
- CUTPNFP does not detect LRF
- MNFP will detect LRF
For
Each
Term
MUTP
feasible?
For
Each
Literal
In Term
CUTPNFP
feasible?
Test Set =
MUTP + NFP
Test Set =
MUTP + CUTPNFP
MNFP
Test Set =
MUTP +
MNFP
Minimal-MUMCUT Criterion
• MUTP feasible  MUTP detects LRF
• CUTPNFP feasible  CUTPNFP detects LRF
• Both infeasible  MNFP detects LRF
Minimal-MUMCUT:
• Always need MUTP tests to detect LIF
• CUTPNFP tests only when MUTP infeasible
• MNFP tests only when both are infeasible
“Minimal” means that every test in the test set is needed to
guarantee fault detection – not minimized
New Fault Hierarchy
• Black arrow: relation always holds
• Green arrow: relation holds if MUTP is
feasible
• Red arrow: relation holds if MUTP is
infeasible and CUTPNFP is feasible
LIF
LRF
TOF
LOF
LNF
ORF.
ORF+
TNF
ENF
Case Study
• Analyzed 19 Boolean predicates in an
avionics software system (Weyuker, Chen,
Lau, and Yu)
• Number of unique literals range: 5 to 13
• Determined MUTP feasibility for each term
and CUTPNFP feasibility for each literal
• Examined test set size for MUMCUT vs.
Minimal-MUMCUT
Case Study Results
• Minimal-MUMCUT size is 12% of MUMCUT size
• Savings in test set size comes from
1) CUTPNFP feasible for all 853 literals: no MNFP
2) For 24% of literals, MUTP detects LRF: no
CUTPNFP
3) 16 of 19 predicates had a MUTP feasible term
Test Set Size vs.
Number of Unique Literals
1600
1400
1200
1000
M inim al-M UM CUT
800
M UM CUT
600
400
200
0
5
6
7
8
9
10 11 12 13
Conclusion
• Used criterion feasibility to reduce test set
size without sacrificing fault detection
• Modification of fault detection relations in
Lau and Yu’s hierarchy based on criterion
feasibility
• Introduction of the Minimal-MUMCUT
criterion based on minimal DNF
• Applications for software testing of
programs with large predicates