Recap
Weak Normal EC
Strong Normal EC
Robustness
Equivalence Class Testing
Mohammad Mousavi
Eindhoven University of Technology, The Netherlands
Software Testing, 2013
Mousavi: Equivalence Class Testing
Combined with WCT
Recap
Weak Normal EC
Outline
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Mousavi: Equivalence Class Testing
Strong Normal EC
Robustness
Combined with WCT
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Functional Testing (recap)
I
functional testing:
program is an input from a certain domain to a certain range
I
impossible to check all input/output combinations:
need to choose some
Mousavi: Equivalence Class Testing
Recap
Weak Normal EC
Strong Normal EC
Boundary Value Testing (recap)
I
boundary value testing:
choose extreme values.
I
variants:
I
I
I
I
worst-case
robust
robust worst-case
other (non-standard)
variants:
I
I
special value
random
Mousavi: Equivalence Class Testing
Robustness
Combined with WCT
Recap
Weak Normal EC
Strong Normal EC
Robustness
Boundary Value Testing: Pros and Cons (recap)
+ straightforward test-case
generation
- no sense of covering the
input domain
- awkward for logical vars.
- only independent input
domains
- not using white-box
information
Mousavi: Equivalence Class Testing
Combined with WCT
Recap
Weak Normal EC
Strong Normal EC
Robustness
Boundary Value Testing: Pros and Cons (recap)
+ straightforward test-case
generation
- no sense of covering the
input domain *
- awkward for logical vars. *
- only independent input
domains *
- not using white-box
information
*: Today’s order of business.
Mousavi: Equivalence Class Testing
Combined with WCT
Recap
Weak Normal EC
Outline
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Mousavi: Equivalence Class Testing
Strong Normal EC
Robustness
Combined with WCT
Recap
Weak Normal EC
Strong Normal EC
Weak Normal EC: Idea
I
Define equivalence classes on the
domain (range) of input (output)
for each variable:
(independent input)
I
cover equivalence classes for the
domain of each variable:
single fault assumption
I
how many test-cases are needed?
I
also called: (equivalence, category)
partition method
Mousavi: Equivalence Class Testing
Robustness
Combined with WCT
Recap
Weak Normal EC
Strong Normal EC
Little Puzzle
What is the minimal number of tokens
that are needed to be put in an m × n
grid such that each row and column
contains at leats one token?
Mousavi: Equivalence Class Testing
Robustness
Combined with WCT
Recap
Weak Normal EC
Strong Normal EC
Little Puzzle
What is the minimal number of tokens
that are needed to be put in an m × n
grid such that each row and column
contains at leats one token?
max(m,n):
Put token number i at
(max(i, m), max(i, n)).
Mousavi: Equivalence Class Testing
Robustness
Combined with WCT
Recap
Weak Normal EC
Strong Normal EC
Weak Normal EC: Idea
I
Define equivalence classes on the
domain (range) of input (output)
for each variable:
(independent input)
I
cover equivalence classes for the
domain of each variable:
single fault assumption
I
how many test-cases are needed?
maxx | Sx |.
Mousavi: Equivalence Class Testing
Robustness
Combined with WCT
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Mortgage Example (recap)
Spec. Write a program that takes
three inputs: gender (boolean), age([18-55]), salary ([0-10000])
and output the total mortgage for one person
Mortgage = salary * factor,
where factor is given by the following table.
Category
Young
Middle
Old
Male
(18-35 years) 75
(36-45 years) 55
(46-55 years) 30
Mousavi: Equivalence Class Testing
Female
(18-30 years) 70
(31-40 years) 50
(41-50 years) 35
Recap
Weak Normal EC
Strong Normal EC
Robustness
Weak Normal EC Testing
Category
Young
Middle
Old
Male
(18-35 years) 75
(36-45 years) 55
(46-55 years) 30
Female
(18-30 years) 70
(31-40 years) 50
(41-50 years) 35
I
age: difficult!
I
salary: [0-10000]
I
male: as strange as boundary value!
Mousavi: Equivalence Class Testing
Combined with WCT
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Weak Normal EC Testing
Category
Young
Middle
Old
Male
(18-35 years) 75
(36-45 years) 55
(46-55 years) 30
Female
(18-30 years) 70
(31-40 years) 50
(41-50 years) 35
I
age: difficult! [18-30], [31-35], [36-40], [41,45], [46-50],
[51-55]
I
salary: [0-10000]
I
male: as strange as boundary value! true, false
Mousavi: Equivalence Class Testing
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Weak Normal EC Testing
if (male) then return
((18 ≤ age < 35)?(75 ∗ salary ) : (31 ≤ age < 40)?(55 ∗ salary ) : (30 ∗ salary ))
else return ((18 ≤ age < 30)?(75 ∗ salary ) : (31 ≤ age < 40)?(50 ∗ salary ) : (35 ∗ salary ))
Gender Age Salary Output Correct Out. Pass/Fail
male
20
1000
75*1000 75*1000
P
female
32
1000
50*1000 50*1000
P
male
38
1000
55*1000 50*1000
P
female
42
1000
35*1000 35*1000
P
male
48
1000
30*1000 30*1000
P
female
52
1000
35*5000 too late!
F
Mousavi: Equivalence Class Testing
Recap
Weak Normal EC
Outline
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Mousavi: Equivalence Class Testing
Strong Normal EC
Robustness
Combined with WCT
Recap
Weak Normal EC
Strong Normal EC
Strong Normal EC Testing
I
I
cover the all combinations of
equivalence classes for the domain
of all variables:
multiple fault assumption
Q
number of test-cases? x | Sx |
Mousavi: Equivalence Class Testing
Robustness
Combined with WCT
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Strong Normal EC Testing
Category
Young
Middle
Old
Male
(18-35 years) 75
(36-45 years) 55
(46-55 years) 30
Female
(18-30 years) 70
(31-40 years) 50
(41-50 years) 35
I
age: [18-30], [31-35], [36-40], [41,45], [46-50], [51-55]
I
salary: [0-10000]
I
male: true, false
Mousavi: Equivalence Class Testing
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Strong Normal EC Testing
if (male) then return
((18 ≤ age < 35)?(75 ∗ salary ) : (31 ≤ age < 40)?(55 ∗ salary ) : (30 ∗ salary ))
else return ((18 ≤ age < 30)?(75 ∗ salary ) : (31 ≤ age < 40)?(50 ∗ salary ) : (35 ∗ salary ))
Gender Age Salary Output Correct Out. Pass/Fail
female
20
1000
75*1000 70*1000
F
female
32
1000
50*1000 50*1000
P
female
38
1000
50*1000 50*1000
P
female
42
1000
35*1000 35*1000
P
female
48
1000
35*1000 35*1000
P
female
52
1000
35*5000 too late!
F
Mousavi: Equivalence Class Testing
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Strong Normal EC Testing
if (male) then return
((18 ≤ age < 35)?(75 ∗ salary ) : (31 ≤ age < 40)?(55 ∗ salary ) : (30 ∗ salary ))
else return ((18 ≤ age < 30)?(75 ∗ salary ) : (31 ≤ age < 40)?(50 ∗ salary ) : (35 ∗ salary ))
Gender Age Salary Output Correct Out. Pass/Fail
male
20
1000
75*1000 75*1000
P
male
32
1000
50*1000 75*1000
F
male
38
1000
55*1000 50*1000
P
male
42
1000
30*1000 55*1000
F
male
48
1000
30*1000 30*1000
P
male
52
1000
30*1000 30*1000
P
Mousavi: Equivalence Class Testing
Recap
Weak Normal EC
Outline
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Mousavi: Equivalence Class Testing
Strong Normal EC
Robustness
Combined with WCT
Recap
Weak Normal EC
Strong Normal EC
Weak Robust EC
I
includes weak normal; adds out of
range test-cases for each variable
I
number of test-cases?
(maxx | Sx |) + 2 ∗ n
Mousavi: Equivalence Class Testing
Robustness
Combined with WCT
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Weak Robust EC Testing
if (male) then return
((18 ≤ age < 35)?(75 ∗ salary ) : (31 ≤ age < 40)?(55 ∗ salary ) : (30 ∗ salary ))
else return ((18 ≤ age < 30)?(75 ∗ salary ) : (31 ≤ age < 40)?(50 ∗ salary ) : (35 ∗ salary ))
Gender Age Salary Output
Correct Out. Pass/Fail
male
17
1000
30*1000
too young!
F
female
56
1000
35*1000
too late
F
male
36
-1
55*-1
0
F
female
36
10001 50*10001 50*10000
F
Mousavi: Equivalence Class Testing
Recap
Weak Normal EC
Strong Normal EC
Strong Robust EC
I
Same as strong normal but also
checks for all out of range
combinations
I
number
of test-cases?
Q
x (| Sx | +2)
Mousavi: Equivalence Class Testing
Robustness
Combined with WCT
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Strong Robust EC
if (male) then return
((18 ≤ age < 35)?(75 ∗ salary ) : (31 ≤ age < 40)?(55 ∗ salary ) : (30 ∗ salary ))
else return
((18 ≤ age < 30)?(75 ∗ salary ) : (31 ≤ age < 40)?(50 ∗ salary ) : (35 ∗ salary ))
Mostly similar faults to Weak Robust EC:
Gender Age Salary Output
Correct Out.
male
17
1000
30*1000
too young!
female
56
1000
35*1000
too late
female
17
1000
35*1000
too young!
male
56
1000
30*1000
too late
male
36
-1
55*-1
0
female
36
10001 50*10001 50*10000
...
Mousavi: Equivalence Class Testing
Pass/Fail
F
F
F
F
F
F
Recap
Weak Normal EC
Strong Normal EC
A Brief Comparison
A → B: Test-cases of A
(faults detected by A) is a
subset of those of B.
Mousavi: Equivalence Class Testing
Robustness
Combined with WCT
Recap
Weak Normal EC
Strong Normal EC
Robustness
Mortgage Case: #Test-Cases
test-cases/faults
100
80
60
40
20
technique
rob wc wc
rob bv
Mousavi: Equivalence Class Testing
bv
st rob ec st nor ec wk rob ec
ec
Combined with WCT
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Mortgage Case: Detected Fault
detected faults
10
8
6
4
2
technique
rob wc wc
rob bv
Mousavi: Equivalence Class Testing
bv
st rob ec st nor ec wk rob ec
ec
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Mortgage Case: #Test-Cases/Fault
test-cases/faults
20
16
12
8
4
technique
rob wc wc
rob bv
Mousavi: Equivalence Class Testing
bv
st rob ec st nor ec wk rob ec
ec
Recap
Weak Normal EC
Outline
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Mousavi: Equivalence Class Testing
Strong Normal EC
Robustness
Combined with WCT
Recap
Weak Normal EC
Strong Normal EC
Idea
I
Functional techniques need to be
combined...
I
Example:
Robust WCT + Robust BV
Mousavi: Equivalence Class Testing
Robustness
Combined with WCT
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Strong Robust EC + Robust BV
Gender
male
male
male
male
male
male
female
female
female
female
female
female
Age
17
17
17
56 ‘
56
56
17
17
17
56
56
56
Mousavi: Equivalence Class Testing
Salary
-1
1000
10001
-1
1000
10001
-1
1000
10001
-1
1000
10001
Output
30*-1
30*1000
30*10001
30*-1
30*1000
30*10001
30*-1
30*1000
30*10001
30*-1
30*1000
30*10001
Correct Out.
too young!
too young!
too young!
too late
too late
too late
too young!
too young!
too young!
too late
too late
too late
Pass/Fail
F1
F1
F1
F2
F2
F2
F3
F3
F3
F4
F4
F4
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Strong Robust EC + Robust BV (Cont’d)
Gender
female
female
female
female
female
female
female
female
female
female
female
female
female
Age
18
19
20
29
30
31
32
34
35
36
38
39
40
Mousavi: Equivalence Class Testing
Salary
1000
1000
1000
1000
1000
1000
1000
1000
1000
1000
1000
1000
1000
Output
75*1000
75*1000
75*1000
75*1000
35*1000
50*1000
50*1000
50*1000
50*1000
50*1000
50*1000
50*1000
35*1000
Correct Out.
70*1000
70*1000
70*1000
70*1000
70*1000
50*1000
50*1000
50*1000
50*1000
50*1000
50*1000
50*1000
50*1000
Pass/Fail
F5
F5
F5
F5
F6
P
P
P
P
P
P
P
F7
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Strong Robust EC + Robust BV (Cont’d)
Gender
female
female
female
female
female
female
female
female
female
female
female
female
Age
41
42
44
45
46
49
50
51
52
53
54
55
Mousavi: Equivalence Class Testing
Salary
1000
1000
1000
1000
1000
1000
1000
1000
1000
1000
1000
1000
Output
35*1000
35*1000
35*1000
35*1000
35*1000
35*1000
35*1000
35*1000
35*1000
35*1000
35*1000
35*1000
Correct Out.
35*1000
35*1000
35*1000
35*1000
35*1000
35*1000
35*1000
too late!
too late!
too late!
too late!
too late!
Pass/Fail
P
P
P
P
P
P
P
F7
F7
F7
F7
F7
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Strong Robust EC + Robust BV (Cont’d)
Gender
male
male
male
male
male
male
male
male
male
male
male
male
male
Age
18
19
20
29
30
31
32
34
35
36
38
39
40
Mousavi: Equivalence Class Testing
Salary
1000
1000
1000
1000
1000
1000
1000
1000
1000
1000
1000
1000
1000
Output
75*1000
75*1000
75*1000
75*1000
75*1000
55*1000
55*1000
55*1000
55*1000
55*1000
55*1000
55*1000
55*1000
Correct Out.
75*1000
75*1000
75*1000
75*1000
75*1000
75*1000
75*1000
75*1000
75*1000
55*1000
55*1000
55*1000
20*1000
Pass/Fail
P
P
P
P
P
F8
F8
F8
F9
P
P
P
F 10
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Strong Robust EC + Robust BV (Cont’d)
Gender
male
male
male
male
male
male
male
male
male
male
male
male
Age
41
42
44
45
46
49
50
51
52
53
54
55
Mousavi: Equivalence Class Testing
Salary
1000
1000
1000
1000
1000
1000
1000
1000
1000
1000
1000
1000
Output
30*1000
30*1000
30*1000
30*1000
30*1000
30*1000
30*1000
30*1000
30*1000
30*1000
30*1000
30*1000
Correct Out.
30*1000
30*1000
30*1000
30*1000
30*1000
30*1000
30*1000
30*1000
30*1000
30*1000
30*1000
30*1000
Pass/Fail
P
P
P
P
P
P
P
P
P
P
P
P
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Strong Robust EC + Robust BV (Cont’d)
Gender
female
female
......
Age
17
18
Mousavi: Equivalence Class Testing
Salary
-1
-1
Output
35*-1
75*-1
Correct Out.
0
0
Pass/Fail
F 11
F 11
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Strong Robust EC + Robust BV (Cont’d)
Gender
female
female
...
Age
17
18
Mousavi: Equivalence Class Testing
Salary
10001
10001
Output
35*10001
75*10001
Correct Out.
too young!
75*10000
Pass/Fail
F 11
F 12
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Strong Robust EC + Robust BV (Cont’d)
Gender
male
male
...
Age
17
18
Mousavi: Equivalence Class Testing
Salary
-1
-1
Output
30*-1
70*-1
Correct Out.
0
0
Pass/Fail
F 12
F 12
Recap
Weak Normal EC
Strong Normal EC
Robustness
Combined with WCT
Strong Robust EC + Robust BV (Cont’d)
Gender
male
male
...
Age
17
18
Mousavi: Equivalence Class Testing
Salary
10001
10001
Output
30*10001
70*10001
Correct Out.
too young!
75*10000
Pass/Fail
F 12
F 12
Recap
Weak Normal EC
Strong Normal EC
Problems
I
Example:
Strong EC + Robust BV
number
Q of test-cases:
∼ x 4(| Sx | +1), whopping!
Mousavi: Equivalence Class Testing
Robustness
Combined with WCT
Recap
Weak Normal EC
Strong Normal EC
Problems
I
>100 test-cases for the mortgage
example
I
too many for any real-life program
e.g., 5 vars., each 5 partitions:
∼ 8 million test-cases
1 sec. for each test-case:
3 months testing!
Mousavi: Equivalence Class Testing
Robustness
Combined with WCT
Recap
Weak Normal EC
Strong Normal EC
Problems
I
Problems:
1. No constraints on the equivalence
classes
2. Dependencies among different
variables not taken into account
3. No choice among relevant classes
(e.g., apply worst-case testing on
some and boundary values on
others)
I
Solutions: Attend the coming
lecture!
Mousavi: Equivalence Class Testing
Robustness
Combined with WCT