2014 IEEE International Conference on Software Testing, Verification, and Validation Workshops
Functional Dependence and Equivalence Class
Factors in Combinatorial Test Designs
George B. Sherwood
Testcover.com, LLC
Colts Neck, NJ USA
[email protected]
coverage. Thus, the test designer has a new tool to evaluate the
partitioning and strength of a test design, as it relates to some
or all equivalence classes.
Abstract—Functionally dependent (FD) mappings permeate
software systems and impose constraints on designs for
verification. This paper develops concepts for accommodating
FD relations in combinatorial test designs and derives rules to
determine combinatorial coverage for FD factors. FD
equivalence class factors are introduced to assess coverage of
expected results classes before test case generation. Examples are
given for test designs with FD factors, for applications of the
coverage rules, and for coverage improvement techniques. A
nondeterminant strength classification is formulated.
Keywords—combinatorial
testing;
coverage
equivalence
class;
equivalence
partitioning;
dependence; interaction testing; test design
I.
Six coverage rules are derived for FD test factors and are
illustrated with examples. In particular, the nondeterminant
strength rule is the basis for the equivalence class coverage
formulation. A body mass index (BMI) report application
provides examples of equivalence classes in various test
designs. Combinations of five input factors (Age, Weight,
Height, Sex, Intake) cause the application to generate one or
two reports. These are the Medicare report (depending on
Age), the Child report (depending on Age and Sex), and the
Adult report (depending on Age, Weight, and Height). The
nondeterminant strength rule applied to an FD equivalence
class factor for each report gives the minimum coverage for the
report’s equivalence classes. A strength-3 test design using a
single partition for all classes is shown to have limited
coverage for the Adult report equivalence classes
(underweight, normal, overweight, obese). The limitation is
improved two ways, with
analysis;
functional
INTRODUCTION
Software systems comprise innumerable processes and data
records with deterministic relationships. Functions that accept
input data and return specific results are essential to many
programming languages. The idea that data values can be
determined by other values is central to the design of relational
databases [1]. This paper aims to use properties of these
ubiquitous relations for better treatment in combinatorial test
designs.
x An increase to a strength-4 design, or
x A repartitioned strength-2 design aligning partitions
with selected equivalence classes.
Functionally dependent (FD) relations appear as constraints
in test models. A test factor is functionally dependent when its
value is identified by the values of its determinant factors. For
example, to test an application with a date input, the value
last_day(month,year) may be needed to verify month
boundaries. To test a state model for a shopping cart
application, an event QTY(i,q) may be required to change the
ith item’s quantity to q. The general model that test input and
configuration values map to an expected result is an FD
relation itself.
The repartitioning offers additional freedom to choose values,
e.g. to verify equivalence class boundaries.
Use of the FD coverage rules with equivalence class factors
enables coverage assessment for individual equivalence
classes, which can lead to more design choices and more
effective testing. Moreover, the classification of designs
according to nondeterminant strength may offer some
clarification for the testing challenges of large software
systems [3].
Combinatorial testing, as commonly practiced, involves
generating one partition of test cases for multiple equivalence
classes. A partition includes the allowed combinations of test
factor values for a test case generation instance. An
equivalence class includes the combinations of test factor
values leading to one class of expected results. The prospect of
a partition yielding results in multiple equivalence classes is
clear from the invitation to this workshop [2], which notes “the
difficulty in determining expected results for the generated
tests.” This paper provides a formulation for minimum
coverage of each equivalence class from a single partition. The
formulation is based on the number of determinant test factors
in the FD relation of the equivalence class. An analysis of
generated test cases is not needed to find the minimum
978-0-7695-5194-4/14 $31.00 © 2014 IEEE
DOI 10.1109/ICSTW.2014.12
II.
FUNCTIONALLY DEPENDENT TEST INPUTS
A. Equivalence Classes
Equivalence partitioning is a well-known technique for
separating test factor combinations into classes of similar
expected results. For example, test inputs to a program that
converts Celsius temperatures to Fahrenheit may have two
classes, one for valid input temperatures, like 20 oC, and an
invalid class for inputs below absolute zero. Inputs for the first
class are converted to Fahrenheit temperatures according to the
usual formula. Inputs for the second are required to give an
exception result instead. The partitions divide the inputs
107
108
x freezing(inScale), boiling(inScale); two functions
representing the freezing and boiling temperatures of
water on inScale
according to their different classes of results, and inputs in the
same partition should yield equivalent results.
Organizing test factor combinations into equivalence
classes provides a framework for test design, which can help to
avoid masking [4,5,6] and to identify test cases at the
boundaries between classes. Generally mixing test cases from
different classes can obscure what is or is not tested. Moreover,
if test cases for different classes are combined by one partition,
some classes of results might receive less than the intended test
coverage.
These values are chosen so that any combination of the input
values is in the valid equivalence class 1.
The specification of values in functionally dependent form
[7] can clarify the constraint analysis in a test design by
making the relations among factor values explicit. The
last_day(month,year) function has one meaning although it
represents four values: 28, 29, 30, and 31. Without the
minimum(inScale) designation it is not so clear when –273 and
0 are boundary values. And without the index notation of
CHECK(i), the behavior of the shopping cart is more
challenging to analyze.
Each equivalence class has a set of allowed combinations.
In the valid class of this example, 20 oC is an allowed input,
but –300 oC is disallowed because it is below absolute zero. In
the invalid class –300 oC is allowed, but 20 oC is disallowed.
B. Functionally Dependent Factors
Each partition in the test model has a test array with k
factors, and the jth factor has vj values. The partition’s N-by-k
test array has strength t when every subarray of t factors has
every combination of values (every t-tuple) in at least one test
case. Constraints arise when some combinations are disallowed
in a partition. In the valid class of the example, the minimum
allowed temperature represents a constraint. It sets the
boundary between valid and invalid input temperatures. So for
integer inputs the minimum valid temperature is –273 oC; the
maximum invalid temperature is –274 oC.
C. Functionally Dependent Test Cases
Direct Product Block (DPB) notation [7,8] specifies test
combinations and their constraints using blocks of values. All
combinations from each block are allowed. The combinations
from all the blocks define the partition’s allowed combinations,
independent of the test design strength.
Table I describes the temperature conversion partition in
FD form, using DPB notation. The first line contains a title,
and the next three label the test factors. The line starting with a
number sign (#) marks the beginning of the partition, which
contains a block of three lines. Each test factor takes the values
on its respective line. All combinations of values in the block
are allowed.
The jth factor is functionally dependent when its value is
identified by the values of lj determinant factors. Other
independent factors which are not part of this relation are
nondeterminant (ND). There are three input factors in the
example:
Typically the partition in Table I is converted from FD
form to fixed values form [9] before the test cases are
generated. The individual values of the functions are
represented in separate blocks conforming to the mappings of
the functions. Table II shows the temperature conversion
partition in fixed values form. Now the block from Table I is
split into three blocks according to the values of inScale. In
Table II the lines starting with a plus (+) start the three blocks.
Again, all combinations of values in each block are allowed 2.
x inScale, the scale for the input temperature (Celsius,
Fahrenheit or Kelvin)
x inTemp, a number of degrees on the input scale
x outScale, the scale for the converted temperature
(Celsius, Fahrenheit or Kelvin)
Now the minimum valid inTemp value depends on inScale.
That temperature is –273 oC, –459 oF or 0 oK. Thus,
Table III lists the strength-2 test cases generated from the
partition of Table II. All pairs from each block are included,
but other pairs are not included due to the constraints. E.g.
x inScale is a determinant factor
x inTemp is an FD factor
x outScale is an ND factor
The functional dependence can be expressed
inScale ĺ minimum or as minimum(inScale).
TABLE I.
as
PARTITION WITH FUNCTIONALLY DEPENDENT COMPOSITE
FACTOR
Temperature Conversion - functionally dependent form
inScale
inTemp
outScale
# inTemp functions depend on inScale values
Celsius Fahrenheit Kelvin
minimum(inScale) 20 68 freezing(inScale) boiling(inScale)
Fahrenheit Kelvin Celsius
Typically FD factors are composite. They may include
fixed values and possibly multiple functions. In the date input
example, the day factor may need the first day and an
intermediate day, as well as the last day: 1, 10, and
last_day(month,year). In addition to QTY(i,q), the event factor
for the shopping cart application may need a CHECK(i)
function to mark the ith item for deletion and UPDATE to
complete the change. In the temperature conversion example,
the inTemp factor has the following values.
1
Equivalence classes and factor values in this paper are chosen to illustrate
points briefly. A complete test plan would address all equivalence classes and
include other factor values.
2
The values of each function can be expressed in terms of its domain or
range, e.g. boiling(Fahrenheit) or 212. Using the range has advantages of
possibly fewer blocks, and of avoiding the need for another substitution after
test case generation. In this example, values in terms of the domain are used
so that the FD relations continue to be explicit.
x minimum(inScale); a function to test the equivalence
class boundary
x 20, 68; two fixed values
109
108
TABLE II.
PARTITION WITH FIXED VALUES
TABLE IV.
Temperature Conversion - fixed values form
inScale
inTemp
outScale
# block split by inScale values
+Celsius inScale
Celsius
minimum(Celsius) 20 68 freezing(Celsius) boiling(Celsius)
Fahrenheit Kelvin Celsius
+Fahrenheit inScale
Fahrenheit
minimum(Fahrenheit) 20 68 freezing(Fahrenheit) boiling(Fahrenheit)
Fahrenheit Kelvin Celsius
+Kelvin inScale
Kelvin
minimum(Kelvin) 20 68 freezing(Kelvin) boiling(Kelvin)
Fahrenheit Kelvin Celsius
inScale
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
III.
inTemp
outScale
freezing(Celsius)
minimum(Celsius)
20
freezing(Fahrenheit)
minimum(Fahrenheit)
68
freezing(Kelvin)
minimum(Kelvin)
boiling(Kelvin)
boiling(Celsius)
68
boiling(Fahrenheit)
20
20
68
minimum(Celsius)
freezing(Celsius)
boiling(Celsius)
minimum(Celsius)
freezing(Celsius)
boiling(Celsius)
minimum(Fahrenheit)
freezing(Fahrenheit)
boiling(Fahrenheit)
minimum(Fahrenheit)
freezing(Fahrenheit)
boiling(Fahrenheit)
boiling(Kelvin)
minimum(Kelvin)
freezing(Kelvin)
boiling(Kelvin)
minimum(Kelvin)
freezing(Kelvin)
Fahrenheit
Kelvin
Celsius
Fahrenheit
Kelvin
Celsius
Fahrenheit
Kelvin
Celsius
Kelvin
Kelvin
Kelvin
Fahrenheit
Kelvin
Fahrenheit
Fahrenheit
Celsius
Fahrenheit
Celsius
Kelvin
Celsius
Fahrenheit
Celsius
Fahrenheit
Celsius
Kelvin
Celsius
Kelvin
Fahrenheit
Celsius
Fahrenheit
Celsius
Kelvin
outScale
Fahrenheit
Celsius
Kelvin
Kelvin
Celsius
Kelvin
Fahrenheit
Celsius
Fahrenheit
Kelvin
Celsius
Fahrenheit
Celsius
Fahrenheit
Kelvin
COVERAGE WITH FUNCTIONALLY DEPENDENT FACTORS
A. Individual Factor Rules
The rules in this section describe coverage of test cases
generated in FD form for one FD factor. The factor may be
composite, with fixed values and/or a number of different
functions with the same lj determinant factors and lj < t.
TEST CASES GENERATED WITH FIXED VALUES
Celsius
Celsius
Celsius
Fahrenheit
Fahrenheit
Fahrenheit
Kelvin
Kelvin
Kelvin
Celsius
Celsius
Fahrenheit
Fahrenheit
Kelvin
Kelvin
Celsius
Celsius
Celsius
Celsius
Celsius
Celsius
Fahrenheit
Fahrenheit
Fahrenheit
Fahrenheit
Fahrenheit
Fahrenheit
Kelvin
Kelvin
Kelvin
Kelvin
Kelvin
Kelvin
inTemp
minimum(inScale)
20
68
freezing(inScale)
boiling(inScale)
20
20
68
68
minimum(inScale)
minimum(inScale)
freezing(inScale)
freezing(inScale)
boiling(inScale)
boiling(inScale)
It is also evident that all pairs of determinant and ND
values are present. However the coverage of the FD and ND
values is not so strong. Each outScale value is paired with each
of the inTemp fixed values, but it is paired with each function
only once. For example, the outScale value Fahrenheit is paired
with minimum(Celsius) in test case 1, but it is not paired with
minimum(Fahrenheit) or minimum(Kelvin).
One question that may arise is what happens if test cases
are generated from factor values in FD form? The answer for
this example is given in Table IV, which shows the strength-2
test cases generated from the partition of Table I. In Table IV
the inTemp minimum(inScale) function is paired with all three
inScale values, so the three boundary inputs are present. Each
of the other two functions and the two fixed values for inTemp
also are paired with the three inScale values. Thus, all allowed
inScale
Celsius
Fahrenheit
Kelvin
Fahrenheit
Kelvin
Celsius
Kelvin
Celsius
Fahrenheit
Fahrenheit
Kelvin
Celsius
Kelvin
Celsius
Fahrenheit
value pairs of the determinant and FD factors occur.
inScale Kelvin is not paired with boiling(Fahrenheit) or 212.
There are 11 fixed values for the inTemp factor, so the number
of test cases is the minimum (33 = 11 · 3).
TABLE III.
TEST CASES GENERATED WITH FUNCTIONALLY DEPENDENT
COMPOSITE FACTOR
In a partition’s test array, a subarray consisting of a
functionally dependent factor j and its nj nondeterminant
factors is a nondeterminant subarray J. J has nondeterminant
strength sj when every subarray of sj factors including j has
every combination of values (every sj-tuple) in at least one test
case. The ND strength has an upper bound: sj ” nj + 1.
The definition of ND strength applies when J is a covering
array of strength sj. It also applies when the subarray of ND
factors covers with strength sj – 1 and the FD factor is a singlevalue factor [7]. In this case every (sj–1)-tuple of ND factors is
associated with the FD factor’s value.
C1. Determinant coverage rule: The subarray of the FD
factor and its determinant factors covers all allowed
(lj+1)-tuples.
All allowed lj-tuples of the determinant factors are
covered because lj < t. These combinations are
associated with each of the FD factor’s functions
because lj + 1 ” t. These combinations also determine
the allowed values of the dependent functions. Thus,
all the allowed (lj+1)-tuples are covered.
C1 shows that in FD form the coverage of a
composite FD factor with its determinant factors can
be less than or equal to the generation strength t. In
the temperature conversion example lj = 1, so all
110
109
allowed pairs (2-tuples) of inScale and inTemp
values are covered.
The new partitions still correspond to the valid equivalence
class. The repartitioning is just a tactic to force the generation
of test cases which associate ND and FD values.
C2. Independent coverage rule: Every allowed (t–1)tuple
of
independent
(determinant
and
nondeterminant) factors is associated with at least
one value of each of the FD factor’s functions.
Table VI gives the strength-2 test cases generated from the
partitions of Table V. There are nine test cases from each
partition for a total of 36, three more than the minimum shown
in Table III.
The rule follows from the strength-t test case
generation with each function represented as one
value.
Now, in the test arrays of each of the single-function
partitions, when the outScale value is paired with the inScale
value, it is always paired with the FD value in the same test
case. Thus all pairs of inTemp and outScale values are covered.
The inTemp-outScale subarrays have ND strength 2.
C2 shows that in FD form the ND subarray of a
composite FD factor has ND strength 1. In the
temperature conversion example each value of each
inTemp function is associated with only one outScale
value. Rule C1 specifies stronger coverage for
inScale-inTemp pairs than C2.
C. Single-function Partition Rules
The following rules describe coverage of test cases
generated in FD form for a total of M FD factors in a singlefunction partition. A functionally dependent set is any subset of
m FD factors whose values are identified by l ” t determinant
factors. The FD set has n ND factors with fixed values. Thus
l + n + M = k.
The coverage of the FD factor with its ND factors could be
stronger if the FD factor were not composite, i.e. if it had just
one function and no fixed values. If every lj-tuple of the
determinant factors is associated with that function in every
test case, then the function could be associated with the ND
combinations also. This is the idea behind single-function
partitions in the next section.
C3. Determinant coverage rule: Any subarray consisting
of an FD set of m factors and its l determinant factors
covers all allowed (l+m)-tuples.
B. Single-function Partitions
A partition in FD form is repartitioned into single-function
partitions when each factor j in each partition
TABLE VI.
inScale
x has values of one function identified by the fixed values
of lj determinant factors, or
x has fixed values.
min1
min2
min3
min4
min5
min6
min7
min8
min9
fxd1
fxd2
fxd3
fxd4
fxd5
fxd6
fxd7
fxd8
fxd9
frz1
frz2
frz3
frz4
frz5
frz6
frz7
frz8
frz9
bol1
bol2
bol3
bol4
bol5
bol6
bol7
bol8
bol9
Table I shows the temperature conversion partition for the
valid equivalence class in FD form. The inTemp factor is
composite, containing three functions of inScale and two fixed
values. If this partition is split into four partitions, as in
Table V, each of the inTemp functions specifies the only
inTemp values in its respective single-function partition.
The other partition contains the fixed values of inTemp 3.
TABLE V.
TEST CASES GENERATED WITH SINGLE-FUNCTION
PARTITIONS
PARTITION SPLIT INTO SINGLE-FUNCTION PARTITIONS
Temperature Conversion - single-function partitions
inScale
inTemp
outScale
#min minimum inTemp function
Celsius Fahrenheit Kelvin
minimum(inScale)
Fahrenheit Kelvin Celsius
#fxd fixed inTemp values
Celsius Fahrenheit Kelvin
20 68
Fahrenheit Kelvin Celsius
#frz freezing inTemp function
Celsius Fahrenheit Kelvin
freezing(inScale)
Fahrenheit Kelvin Celsius
#bol boiling inTemp function
Celsius Fahrenheit Kelvin
boiling(inScale)
Fahrenheit Kelvin Celsius
3
When there are multiple FD factors the partition splitting is repeated for
each.
111
110
Kelvin
Celsius
Fahrenheit
Kelvin
Fahrenheit
Celsius
Kelvin
Fahrenheit
Celsius
Kelvin
Celsius
Kelvin
Celsius
Fahrenheit
Fahrenheit
Fahrenheit
Kelvin
Celsius
Kelvin
Celsius
Fahrenheit
Kelvin
Fahrenheit
Celsius
Kelvin
Fahrenheit
Celsius
Kelvin
Celsius
Fahrenheit
Kelvin
Fahrenheit
Celsius
Kelvin
Fahrenheit
Celsius
inTemp
minimum(inScale)
minimum(inScale)
minimum(inScale)
minimum(inScale)
minimum(inScale)
minimum(inScale)
minimum(inScale)
minimum(inScale)
minimum(inScale)
20
20
68
68
68
20
20
20
68
freezing(inScale)
freezing(inScale)
freezing(inScale)
freezing(inScale)
freezing(inScale)
freezing(inScale)
freezing(inScale)
freezing(inScale)
freezing(inScale)
boiling(inScale)
boiling(inScale)
boiling(inScale)
boiling(inScale)
boiling(inScale)
boiling(inScale)
boiling(inScale)
boiling(inScale)
boiling(inScale)
outScale
Celsius
Kelvin
Fahrenheit
Fahrenheit
Celsius
Celsius
Kelvin
Kelvin
Fahrenheit
Celsius
Kelvin
Fahrenheit
Celsius
Kelvin
Fahrenheit
Celsius
Kelvin
Fahrenheit
Celsius
Kelvin
Fahrenheit
Fahrenheit
Celsius
Celsius
Kelvin
Kelvin
Fahrenheit
Celsius
Kelvin
Fahrenheit
Fahrenheit
Celsius
Celsius
Kelvin
Kelvin
Fahrenheit
m FD factors are identified by l ” m determinant
factors because each lj = 1. Thus the t-tuple is
determined by at most l + (t – m) ” t independent,
covered factor values. I.e. all allowed t-tuples are
covered. Each nondeterminant subarray has ND
strength sj = t by C5 because lj = 1.
The m-tuple of the FD set is identified by each l-tuple
of its determinant factors. When the partition
contains all of the allowed l-tuples, all the allowed
(l+m)-tuples are covered.
C3 shows that in a single-function partition in FD
form, the coverage of an FD factor with its
determinant factors may or may not be the same as
the generation strength t. In each single-function
partition of the temperature conversion example,
l = 1 and m = 1. So all allowed pairs of inScale and
inTemp values are covered.
C6 shows the conditions when test cases generated
from a single-function partition in FD form have
equivalent strength as those generated from a
partition in fixed values form. In the single-function
partitions of the temperature conversion example,
lj = 1. All allowed pairs are covered.
C4. Independent coverage rule: Any subarray consisting
of an FD set of m factors, its l determinant factors,
and t – l ND factors covers all allowed (t+m)-tuples.
When t • l + n, all allowed (l+n+m)-tuples for the FD
set are covered; the allowed k-tuples of the single
function partition are covered also.
EQUIVALENCE CLASS FACTORS
IV.
In a deterministic system every mapping of input and
configuration values to a result can be considered a
functionally dependent relation. And if each result maps to one
equivalence class, then that class is determined by the input
and configuration values as well.
When t ” l + n, there are at least t – l ND factors. The
m-tuple of the FD set is identified by each l-tuple of
its determinant factors. Each l-tuple also is associated
with the allowed ND (t–l)-tuples because the
independent factors cover with strength t. When the
partition contains all allowed l-tuples, they are
associated with the allowed FD m-tuples and with the
allowed ND (t–l)-tuples. Thus all allowed (t+m)tuples are covered. When t • l + n, each l-tuple is
associated with all FD m-tuples and with all ND ntuples, so all allowed (l+n+m)-tuples are covered.
Since the partition contains all the allowed
independent (l+n)-tuples, all M FD factors are
determined, and all k-tuples are covered.
input, configuration values
ĺ result
ĺ equivalence class
In the temperature conversion example, an equivalence class
factor can be defined as in Table VII. This factor is
functionally dependent on two input factors, inScale and
inTemp.
The equivalence class factor characterizes expected results,
so it can be useful for analyzing coverage of test designs. The
idea is to use equivalence class factors defined by requirements
as FD factors in single-function partitions. Thus the
functionally dependent coverage rules apply, and coverage of
the equivalence classes can be assessed, even before test cases
are generated.
C4 shows that in a single-function partition in FD
form, the coverage of an FD factor with its
independent factors can be greater than the
generation strength t. In the single-function partitions
of the temperature conversion example, all allowed
3-tuples are covered.
A. Equivalence Classes from Requirements
Requirements for a body mass index (BMI) report
application include the following.
R1. The listed input data will be stored in the patient
database table.
a. Age in years
b. Weight in pounds
c. Height in inches
d. Sex (female, male)
e. Intake in kilocalories per day
R2. The BMI will be calculated (in kilograms per meter
squared) as 703.06957964 Weight / Height2 and
stored in the patient database table.
R3. If Age is 65 years or older, the Medicare report will
be generated.
C5. Nondeterminant strength rule: A nondeterminant
subarray J has nondeterminant strength sj = t – lj + 1
when t ” lj + nj; sj = nj + 1, when t • lj + nj.
C5 follows from C4 when m = 1, and the lj
determinant factors are not included in the subarray.
C5 shows that in a single-function partition in FD
form, the ND strength might be less than the
generation strength t due to constraints of the FD
relation. In all the single-function partitions of the
temperature conversion example, all allowed
inTemp-outScale pairs are covered, and the ND
strength is 2.
TABLE VII.
C6. Equivalent coverage rule: When each factor in a set
of M FD factors has exactly one determinant factor,
all allowed t-tuples are covered, and each
nondeterminant subarray has nondeterminant strength
sj = t.
TEMPERATURE CONVERSION EQUIVALENCE CLASS FACTOR
DEFINITION
Input factors
inScale
Kelvin
Kelvin
Celsius
Celsius
Fahrenheit
Fahrenheit
When a t-tuple contains values of m FD factors, it
contains values of t – m independent factors also. The
112
111
inTemp
” –1
•0
” –274
• –273
” –460
• –459
Equivalence class
factor
invalid
valid
invalid
valid
invalid
valid
TABLE VIII.
R4. If Age is younger than 20 years, the Child report
containing the BMI percentile will be generated for
the corresponding listed classification.
a. Girl, from the female BMI-age table
b. Boy, from the male BMI-age table
R5. If Age is 20 years or older, the Adult report will be
generated for the corresponding listed classification.
a. Underweight, BMI < 18.5
b. Normal, 18.5 ” BMI < 25.0
c. Overweight, 25.0 ” BMI < 30.0
d. Obese, 30.0 ” BMI
Body Mass Index Report Application
Age
Weight
Height
Sex
Intake
Medicare equivalence class
Child equivalence class
Adult equivalence class
#
19 42 67
131 180
64 71
female male
2000 3000
Medicare_report(Age)
Child_report(Age,Sex)
Adult_report(Age,Weight,Height)
The BMI calculation is an expected result of R2. It is an FD
factor of Weight and Height. R3, R4, and R5 suggest three
equivalence class factors which are functionally dependent on
inputs:
Age
Age, Sex
Age, Weight, Height
PARTITION WITH FIXED INPUT VALUES AND FUNCTIONALLY
DEPENDENT EQUIVALENCE CLASS FACTORS
ĺ Medicare classes
ĺ Child classes
ĺ Adult classes
The Child value (girl or boy) indicates the Child report is
an expected result for the indicated class; no means no Child
report is expected. Each equivalence class is associated with
the allowed pairs of its determinant factors, Age and Sex. The
ND subarray of Weight, Height, Intake, and Child factors has
ND strength 2. I.e. all allowed values of the Child ND factors
are associated with each equivalence class.
These FD relations are used to assess coverage for their
equivalence classes.
B. Strength-3 Design
If a strength-3 test design is chosen using the five inputs of
R1 in one partition, the coverage rules imply the following for
the FD equivalence class factors Medicare, Child, and Adult.
The Adult value (underweight, normal, overweight or
obese) indicates the Adult report is an expected result for the
indicated class; no means no Adult report is expected. Each
equivalence class is associated with the allowed 3-tuples of its
determinant factors, Age, Weight, and Height. The ND
subarray of Sex, Intake, and Adult factors has ND strength 1.
I.e. all allowed values of the Adult ND factors might not be
associated with each equivalence class. Examination of the test
cases in Table IX shows that only six of the eight Sex-Adult
pairs are present; female-obese and male-underweight are
missing. Similarly two of the eight Intake-Adult pairs, 3000obese and 2000-underweight, are missing.
C3 and C4 apply to any set of m FD factors for which l ” t.
For example, the two FD factors, Medicare and Child have two
determinant factors, Age and Sex. C3 shows that all their
allowed 4-tuples are covered. C4 indicates that when one ND
factor (Weight, Height or Intake) is included in the subarray,
all allowed 5-tuples are covered.
C5 applies to individual FD factors. It describes the
coverage of each FD factor with its ND factors. For the
Medicare, Child, and Adult factors, the ND strengths are 3, 2,
and 1 respectively, indicating 3-tuples, 2-tuples, and 1-tuples
are covered in the corresponding ND subarray.
C. Strength-4 Design
If one of the test design goals is to associate each
equivalence class with its ND factor values, the test cases of
Table IX do not meet that goal. One straightforward response
is to increase the strength of the design. Table X lists test cases
for the same partition shown in Table VIII. The input factor
values from R1 cover with strength 4 now. The corresponding
fixed values for the equivalence class factors are given. The
expected BMI is shown also.
C6 does not apply to the BMI report example because lj > 1
for the Child and Adult factors.
Table VIII shows the partition to generate test cases in this
example. It has the five input factors from R1 and the three
equivalence class factors from R3, R4, and R5. Three Age
values are chosen to verify R3, R4, and R5. Weight and Height
values are chosen so that all four BMI classifications result.
Table IX lists the test cases for this partition. The input
factor values from R1 cover with strength-3. The
corresponding fixed values for the equivalence class factors are
given. The expected BMI is shown for reference.
Now the ND strengths of the Medicare, Child, and Adult
factors are 4, 3, and 3 respectively. In particular the strength-4
design of Table X achieves ND strength of 3 for the Adult
factor, which exceeds the minimum given by C5.
The Medicare value (yes or no) indicates the Medicare
report is or is not an expected result. Each equivalence class is
associated with the allowed values of its determinant factor,
Age. The ND subarray of Weight, Height, Sex, Intake, and
Medicare factors has ND strength 3. I.e. all allowed pairs of the
Medicare ND factors are associated with each equivalence
class.
Now all allowed (t–l)-tuples (3-tuples) of the Medicare ND
factors are associated with each equivalence class. Similarly all
allowed pairs of the Child ND factors are associated with each
equivalence class. And all allowed pairs of the Adult ND
factors are associated with each equivalence class.
113
112
TABLE IX.
STRENGTH-3 TEST CASES AND CORRESPONDING EQUIVALENCE CLASS FACTOR VALUES
Input factors
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Age
Weight
Height
19
19
19
19
19
19
42
42
42
42
42
42
67
67
67
67
67
67
131
131
131
180
180
180
131
131
131
180
180
180
131
131
131
180
180
180
64
64
71
64
71
71
64
64
71
64
71
71
64
64
71
64
71
71
Equivalence class factors
Sex
female
male
male
female
female
male
female
male
female
male
female
male
female
male
female
male
female
male
Intake
Medicare
2000
3000
2000
3000
2000
3000
2000
3000
3000
2000
2000
3000
2000
3000
3000
2000
2000
3000
no
no
no
no
no
no
no
no
no
no
no
no
yes
yes
yes
yes
yes
yes
no
no
no
no
no
no
normal
normal
underweight
obese
overweight
overweight
normal
normal
underweight
obese
overweight
overweight
An expected result
BMI
22.5
22.5
18.3
30.9
25.1
25.1
22.5
22.5
18.3
30.9
25.1
25.1
22.5
22.5
18.3
30.9
25.1
25.1
In this design, the Medicare equivalence classes do not
have their own partitions. According to C5 for t = 2, the ND
subarray of Weight, Height, Sex, Intake, and Medicare factors
has ND strength 2, so all allowed values of the Medicare ND
factors can be associated with each equivalence class in one
partition. The choice of an Age value of 65 or older in an Adult
partition can insure the coverage.
STRENGTH-4 TEST CASES AND CORRESPONDING EQUIVALENCE CLASS FACTOR VALUES
Input factors
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
girl
boy
boy
girl
girl
boy
no
no
no
no
no
no
no
no
no
no
no
no
Adult
is the lesser of t + 1 or nj + 1. (The ND strength cannot exceed
nj + 1 by definition.) The following strength-2 design illustrates
the repartitioning for the Child and Adult equivalence classes.
D. Strength-2 Design
If one of the test design goals is to associate each
equivalence class with its ND factor values, another
straightforward response is to align test case generation
partitions with their equivalence classes as needed for the
desired coverage. When a partition is designed for results
within one equivalence class, the corresponding FD factor is a
single-value factor of that partition. It is associated with all
allowed t-tuples, including its ND factors. Thus, when a
partition is aligned with one equivalence class, its ND strength
TABLE X.
Child
Age
Weight
Height
19
19
19
19
19
19
19
19
42
42
42
42
42
42
42
42
67
67
67
67
67
67
67
67
131
131
131
131
180
180
180
180
131
131
131
131
180
180
180
180
131
131
131
131
180
180
180
180
64
64
71
71
64
64
71
71
64
64
71
71
64
64
71
71
64
64
71
71
64
64
71
71
Equivalence class factors
Sex
female
male
female
male
female
male
female
male
female
male
female
male
female
male
female
male
female
male
female
male
female
male
female
male
Intake
Medicare
2000
3000
3000
2000
3000
2000
2000
3000
3000
2000
2000
3000
2000
3000
3000
2000
2000
3000
3000
2000
3000
2000
2000
3000
no
no
no
no
no
no
no
no
no
no
no
no
no
no
no
no
yes
yes
yes
yes
yes
yes
yes
yes
114
113
Child
girl
boy
girl
boy
girl
boy
girl
boy
no
no
no
no
no
no
no
no
no
no
no
no
no
no
no
no
Adult
no
no
no
no
no
no
no
no
normal
normal
underweight
underweight
obese
obese
overweight
overweight
normal
normal
underweight
underweight
obese
obese
overweight
overweight
An expected result
BMI
22.5
22.5
18.3
18.3
30.9
30.9
25.1
25.1
22.5
22.5
18.3
18.3
30.9
30.9
25.1
25.1
22.5
22.5
18.3
18.3
30.9
30.9
25.1
25.1
TABLE XI.
For the Child equivalence classes, C5 shows that the ND
subarray of Weight, Height, Intake, and Child factors has ND
strength 1. I.e. the Child ND factor values might not be
associated with each equivalence class in one partition.
Including partitions for the girl and boy equivalence classes
can insure this association. Since each class is a single-value
factor, it is associated with all allowed pairs of ND factors. So
the Child ND subarray has ND strength 3 in each partition.
Similar reasoning suggests separate partitions for the Adult
equivalence classes, underweight, normal, overweight, and
obese.
Body Mass Index Report Application
Age
Weight
Height
Sex
Intake
Medicare equivalence class
Child equivalence class
Adult equivalence class
#gl Child girl
15 19
115 135
63 67
female
2000 3000
Medicare_report(Age)
Child_report(Age,Sex)
Adult_report(Age,Weight,Height)
#by Child boy
15 19
125 168
66 71
male
2000 3000
Medicare_report(Age)
Child_report(Age,Sex)
Adult_report(Age,Weight,Height)
#uw Adult underweight
20 64 65
118 128
70 72
female male
2000 3000
Medicare_report(Age)
Child_report(Age,Sex)
Adult_report(Age,Weight,Height)
#nm Adult normal
20 64 65
129 154
66 70
female male
2000 3000
Medicare_report(Age)
Child_report(Age,Sex)
Adult_report(Age,Weight,Height)
#ow Adult overweight
20 64 65
155 174
64 66
female male
2000 3000
Medicare_report(Age)
Child_report(Age,Sex)
Adult_report(Age,Weight,Height)
#ob Adult obese
20 64 65
175 211
61 64
female male
2000 3000
Medicare_report(Age)
Child_report(Age,Sex)
Adult_report(Age,Weight,Height)
Table XI shows the six partitions to generate test cases in
this example. As before, it has the five input factors from R1
and the three equivalence class factors from R3, R4, and R5.
The six partitions are for the Child girl and boy classes; and for
the Adult underweight, normal, overweight, and obese classes.
Age, Weight, and Height values are chosen so that expected
results conform to their equivalence classes, and boundaries
between classes can be verified 4.
Table XII lists the test cases for the partitions shown in
Table XI. The input factor values cover with strength 2. The
corresponding fixed values for the equivalence class factors are
given. The expected BMI is shown also.
In this design each Medicare equivalence class is expected
in each of the third through sixth partitions. In each of these
test arrays, in accordance with C5, each Medicare ND factor
(Weight, Height, Sex, Intake) has its values associated with
each equivalence class.
In the first partition the Child class factor has a single
value, girl. This value is associated with all allowed pairs of the
independent factors (Age, Weight, Height, Sex, Intake).
Similarly, in the second partition the Child class boy is
associated with all allowed pairs.
In each of the remaining partitions an Adult equivalence
class factor is a single-value factor. Thus it is associated with
all allowed pairs in its respective partition’s test array.
Finally, the use of multiple partitions in this design offers
additional freedom to choose values that verify boundaries
between equivalence classes. Tests for the following
boundaries are included in Table XII.
x
x
x
x
x
x
x
PARTITIONS WITH FIXED INPUT VALUES AND FUNCTIONALLY
DEPENDENT EQUIVALENCE CLASS FACTORS
Medicare: lower Age
Child: upper Age
Adult: lower Age
Adult, underweight: upper BMI
Adult, normal: lower and upper BMI
Adult, overweight: lower and upper BMI
Adult, obese: lower BMI
For example, at Height 70, test case uw2 with Weight 128
is in the underweight class; test case nm1 with Weight 129 is in
the normal class. At Height 66, test case nm2 with Weight 154
is in the normal class; test case ow1 with Weight 155 is in the
overweight class.
4
This example uses integer inputs; BMI applications often use fractions for
Age, Weight and Height.
115
114
TABLE XII.
STRENGTH-2 TEST CASES AND CORRESPONDING EQUIVALENCE CLASS FACTOR VALUES
Input factors
gl1
gl2
gl3
gl4
gl5
by1
by2
by3
by4
by5
uw1
uw2
uw3
uw4
uw5
uw6
nm1
nm2
nm3
nm4
nm5
nm6
ow1
ow2
ow3
ow4
ow5
ow6
ob1
ob2
ob3
ob4
ob5
ob6
Age
Weight
Height
15
19
19
19
15
15
19
19
19
15
65
65
64
20
64
20
65
65
64
20
64
20
65
65
64
20
64
20
65
65
64
20
64
20
115
135
135
115
135
125
168
168
125
168
118
128
128
118
118
128
129
154
154
129
129
154
155
174
174
155
155
174
175
211
211
175
175
211
63
67
63
67
67
66
71
66
71
71
72
70
72
70
70
72
70
66
70
66
66
70
66
64
66
64
64
66
64
61
64
61
61
64
V.
Equivalence class factors
Sex
female
female
female
female
female
male
male
male
male
male
male
female
male
male
female
female
male
female
male
male
female
female
male
female
male
male
female
female
male
female
male
male
female
female
Intake
Medicare
2000
2000
3000
3000
3000
2000
2000
3000
3000
3000
3000
2000
2000
2000
3000
3000
3000
2000
2000
2000
3000
3000
3000
2000
2000
2000
3000
3000
3000
2000
2000
2000
3000
3000
no
no
no
no
no
no
no
no
no
no
yes
yes
no
no
no
no
yes
yes
no
no
no
no
yes
yes
no
no
no
no
yes
yes
no
no
no
no
Child
girl
girl
girl
girl
girl
boy
boy
boy
boy
boy
no
no
no
no
no
no
no
no
no
no
no
no
no
no
no
no
no
no
no
no
no
no
no
no
An expected result
Adult
BMI
no
no
no
no
no
no
no
no
no
no
underweight
underweight
underweight
underweight
underweight
underweight
normal
normal
normal
normal
normal
normal
overweight
overweight
overweight
overweight
overweight
overweight
obese
obese
obese
obese
obese
obese
20.4
21.1
23.9
18.0
21.1
20.2
23.4
27.1
17.4
23.4
16.0
18.4
17.4
16.9
16.9
17.4
18.5
24.9
22.1
20.8
20.8
22.1
25.0
29.9
28.1
26.6
26.6
28.1
30.0
39.9
36.2
33.1
33.1
36.2
coverage to all values with each class. In contrast, all pairs of
ND factors can be expected with each repartitioned class in the
strength-2 design.
DISCUSSION
The FD coverage rules and their application to FD
equivalence class factors formulate how constraints from FD
relations can alter coverage from the nominal generation
strength t. Of particular interest is C5, the nondeterminant
strength rule, because it explains how equivalence class
coverage might be less than t. Specifically each equivalence
class having lj determinant factors is associated with at least
(t–lj)-tuples of ND factors. Two types of responses to this
reduction are suggested.
Table XIII shows the ND strength achieved by the three
BMI example designs. All of the listed values are consistent
with coverage rule C5. The strength-4 design of Table X
achieves ND strength 3 for the Adult factor, which exceeds the
minimum given by C5. The strength-2 design has ND
strength 3 for the Child and Adult factors because their
partitions are aligned with their equivalence classes.
The first is to increase the strength of the test case
generation from t to t’. Its effect is to associate each
equivalence class with ND (t’–lj)-tuples. When lj is large, an
even larger strength is indicated.
The point of applying the coverage rules to FD equivalence
class factors is to enable testers to make informed decisions
about designs. And the fact that these choices can be made
before the generation of test cases can make the design process
more efficient.
The second response is to align test case generation
partitions with their equivalence classes as needed for the
desired coverage. This repartitioning associates the equivalence
class with ND t-tuples.
TABLE XIII.
NONDETERMINANT STRENGTH FOR EXAMPLE DESIGNS
Design
Equivalence class factors
Medicare
The coverage difference between the two partitioning
methods can be significant. According to C5, in the strength-3
design the Adult equivalence classes are not expected to be
associated with all ND values. The test cases of Table IX show
that only 50% of the values are covered in two of the classes.
C5 implies that the strength-4 design raises the minimum ND
Table IX
Table X
Table XII, all partitions
116
115
3
4
2
Child
2
3
3
Adult
1
3
3
REFERENCES
However the choices can be helpful later in the process
also. If the Adult report fails whenever the inputs are for an
underweight male, the strength-3 test cases of Table IX cannot
detect that failure. When analysis shows that a combination of
four inputs, Age, Weight, Height, and Sex induce the problem,
responses can include either of the two described above:
x Increasing the strength to 4 using appropriate values in
a single-partition design (as in Table X)
x Choosing a strength-2 design with appropriate values
and repartitioning (as in Table XII)
[1]
Codd, E. F. (1972). Further normalization of the data base relational
model. In Data Base Systems (pp. 33-64). Upper Saddle River, NJ:
Prentice Hall.
[2]
IBM. (2014). The 3rd International Workshop on Combinatorial Testing
(IWCT 2014). Retrieved February 12, 2014, from Research.ibm.com:
https://www.research.ibm.com/haifa/Workshops/iwct2014.
[3] Kuhn, D. R., Kacker, R. N., & Lei, Y. (2013). Combinatorial Methods in
Testing. In D. R. Kuhn, R. N. Kacker, & Y. Lei, Introduction to
Combinatorial Testing (pp. 1-20). Boca Raton, FL: CRC Press.
Both responses increase the ND strength for the Adult
equivalence class factor to 3, but there are additional choices.
Generation strength 3 could be retained. With appropriate
values and repartitioning for the Adult classes, the ND strength
would increase to 3 (which is nj + 1, the maximum value for
the Adult factor). Moreover, different choices can be made for
different equivalence class factors.
To make use of these choices test case generation tools
need to support the use of multiple partitions having different
factor values and constraints, to align partitions with
equivalence classes as needed. Without this capability the only
recourse appears to be an escalation of strength, yielding test
plans of increasing cost to execute. Of course the test oracle
problem—determination of expected results—contributes
greatly to the challenge [2,3]. However, when information
about equivalence classes and expected results is available, test
design tools and processes should make the best use of it.
Finally the formulation of the equivalence class coverage
rules suggests new combinatorial classifications may be useful.
Empirical research examining the effectiveness of designs
using different ND strengths may clarify the outcomes of test
design choices.
117
116
[4]
Tatsumi, K. (1987). Test case design support system. Proceedings of the
International Conference on Quality Control (ICQC'87), (pp. 615-620).
Tokyo.
[5]
Sherwood, G. (1994). Efficient testing of factor combinations.
Proceedings of the Third International Conference on Software Testing,
Analysis, and Review, (pp. 151-166). Washington, DC.
[6]
Czerwonka, J. (2006). Pairwise testing in the real world: Practical
extensions to test-case scenarios. Proceedings of the Twenty-fourth
Annual Pacific Northwest Software Quality Conference, (pp. 419-430).
Portland, OR.
[7]
Sherwood, G. (2013). Managing System State. In D. R. Kuhn, R. N.
Kacker, & Y. Lei, Introduction to Combinatorial Testing (pp. 113-141).
Boca Raton FL: CRC Press.
[8]
Testcover.com, LLC. (2008). Submitting Test Case Generator Requests.
Retrieved
January
6,
2014,
from
Testcover.com:
http://testcover.com/pub/how2in.php.
[9]
Testcover.com, LLC. (2013). Fixed Values Procedure. Retrieved
January
6,
2014,
from
Testcover.com:
http://www.testcover.com/pub/fvproc.php.