5. Test Generation – Finite State Models
Andrea Polini
Software Engineering II – Software Testing
MSc in Computer Science
University of Camerino
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
1 / 31
Models in the Design Phase
Design Phase
I Between the requirements phase and the implementation phase “The last you
start the first you finish”
I Produce models in order to clarify requirements and to better formalize them
I Models can be the source of test set derivation strategies
Various modeling notations for behavioral specification of a software system have
been proposed. Which to use depends on the system you are developing, and the
aspects you would like to highlight:
Finite State Machines
Petri Nets
Statecharts
Message sequence charts
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
2 / 31
Finite State Machines
FSM
A finite state machine is a six-tuple <X , Y , Q, q0 , δ, O> where:
I
X : finite set of input symbols
I
Y : finite set of output symbols
I
Q: finite set of states
I
q0 ∈ Q: initial state
I
δ: transition function (Q × X → Q)
I
O: output function (Q × X → Y )
Many possible extensions:
Transition and output functions can consider strings
Definiton of the set of accepting states F ⊆ Q
Non determinism
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
3 / 31
Properties of FSM
Useful properties/concepts for test generation
I
Completely specified (input enabled)
∀(qi ∈ Q, a ∈ X ).∃qj ∈ Q.δ(qi , a) = qj
I
Strongly connected
∀(qi , qj ) ∈ Q × Q.∃s ∈ X ∗ .δ ∗ (qi , s) = qj
I
V-equivalence (distinguishable)
Let M1 and M2 two FSMs. Let V denote a set of non-empty string
on the input alphabet X , and qi ∈ Q1 and qj ∈ Q2 . qi and qj are
considered V − equivalent if O1 (qi , s) = O2 (qj , s). If qi and qj are
V − equivalent given any set V ⊆ X + than they are said to be
equivalent (qi ≡ qj ). If states are not equivalent they are said to be
distinguishable.
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
4 / 31
Properties of FSM....cntd
Useful properties/concepts for test generation...cntd
I
Machine equivalence
M1 and M2 are said to be equivalent if ∀qi ∈ Q1 .∃qj ∈ Q2 .qi ≡ qj
and viceversa.
I
k-equivalence
Let M1 and M2 two FSMs and qi ∈ Q1 and qj ∈ Q1 and k ∈ N.
qi and qj are said to be K − equivalent if they are V − equivalent
for V = {s ∈ X + | | s |≤ k }
I
Minimal machine
an FSM is considered minimal if the number of its states is less
than or equal to any other equivalent FSM
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
5 / 31
Conformance Testing
Conformance Testing
Relates to testing of communication protocols. It aims at assessing
that an implementation of a protocol conform to its specification.
Protocols generally specify:
I
Control rules (FSM)
I
Data rules
Developed techniques are equally applicable when the specification is
refined into an FSM
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
6 / 31
The Testing Problem
FSM and Testing
I
Reset inputs (X = X ∪ {Re}, and Y = Y ∪ {null})
I
Testing based on requirements checks if the implementation
conforms to the machine on a given requirement.
I
The testing problem is reconducted to an equivalence
(nevertheless finite experiments). Is the SUT (IUT) equivalent to
the machine defined during design?
Fault model for FSM – given a fault model the challenge is to
generate a test set T from a design Md where any fault in Mi of
the type in the fault model is guaranteed to be revealed when
tested against T
I
Operation error (refers to issues with O)
Transfer error (refers to issues with δ)
Extra-state error (refers to issues with Q and δ)
Missing-state error (refers to issues with Q and δ)
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
7 / 31
Mutation of FSMs
Mutant
A mutant of an FMS Md is an FSM obtained by introducing one one or
more errors one or more times.
I
Equivalent mutants: mutants that could not be distinguishable
from the originating machine
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
8 / 31
The Testing Problem
Fault coverage
Techniques to measure the goodness of a test set in relation to the
number of errors that it reveals in a given implementation Mi .
I
Nt : total number of first order mutants of the machine M used for
generating tests.
I
Ne : Number of mutants that are equivalent to M
I
Nf : Number of mutants that are distinguished by test set T
generated using some test generation method.
I
Nl : Number of mutants that are not distinguished by T
The fault coverage of a test suite T with respect to a design M is
denoted by FC(T , M) and computed as follows:
FC(T , M) = Number of mutants not distinguished by T /
Number of mutants that are not equivalent to M
= (Nt − Ne − Nf )/(Nt − Ne )
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
9 / 31
Characterization Set
Let M =<X , Y , Q, q1 , δ, O> an FSM that is minimal and complete. A
characterization set for M, denoted as W , is a finite set of input
sequences that distinguish the behaviour of any pair of states in M.
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
10 / 31
K-equivalence partitions
The notion of K − equivalence leads to the notion of
K − equivalence partitions.
Given an FSM a K − equivalence partition of Q, denoted by Pk , is a
collection of n finite sets of states denoted as Σk1 , Σk2 , ..., Σkn such
that:
I
∪i=1...n ΣK1 = Q
I
States in Σkj , for 1 ≤ j ≤ n are K − equivalent
I
if ql ∈ Σki and qm ∈ Σkj , for i 6= j, then ql and qm must be
K − distinguishable
K − equivalence partitions can be derived using an iterative approach
for increasing number of K
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
11 / 31
K-equivalence partitions
The notion of K − equivalence leads to the notion of
K − equivalence partitions.
Given an FSM a K − equivalence partition of Q, denoted by Pk , is a
collection of n finite sets of states denoted as Σk1 , Σk2 , ..., Σkn such
that:
I
∪i=1...n ΣK1 = Q
I
States in Σkj , for 1 ≤ j ≤ n are K − equivalent
I
if ql ∈ Σki and qm ∈ Σkj , for i 6= j, then ql and qm must be
K − distinguishable
K − equivalence partitions can be derived using an iterative approach
for increasing number of K
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
11 / 31
How to derive W from K-equivalence partitions
1
2
Let M an FSM for which P = {P1 , P2 , ..., Pn } is the set of k-equivalence partition.
W =∅
Repeat the steps (a) through (d) given below for each pair of states (qi , qj ), i 6= j,
in M
(a) Find r (1 ≤ r < n such that the states in pair (qi , qj ) belong to the same
group in Pr but not in Pr +1 . If such an r is found then move to step (b)
otherwise we find an η ∈ X such that O(qi , η) 6= O(qj , η), set
W = W ∪ {η} and continue with the next available pair of states. The
length of the minimal distinguishing sequence for (qi , qj ) is r + 1.
(b) Initialize z = . Let p1 = qi and p2 = qj be the current pair of states.
Execute steps (i) through (iii) given below for m = r , r − 1, ..., 1
(i) Find an input symbol η in Pm such that G (p1 , η) 6= G (p2 , η). In case
there is more than one symbol that satisfy the condition in this step,
then select one arbitrarily.
(ii) set z = zη
(iii) set p1 = δ(p1 , η) and p2 = δ(p2 , η)
(c) Find an η ∈ X such that O(p1 , η) 6= O(p2 , η). Set z = zη
(d) The distinguishing sequence for the pair (qi , qj ) is the sequence z. Set
W = W ∪ {z}
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
12 / 31
Example
Termination of the W − procedure guarantees the generation of
distinguishing sequence for each pair.
Si
1
1
1
1
2
2
2
3
3
4
Si
2
3
4
5
3
4
5
4
5
5
(Software Engineering II – Software Testing)
x
baaa
aa
a
a
aa
a
a
a
a
aaa
O(Si , x)
1
0
0
0
0
0
0
0
0
1
O(Sj , x)
0
1
1
1
1
1
1
1
1
0
5. Test Generation – Finite State Models
CS@UNICAM
13 / 31
Example
Termination of the W − procedure guarantees the generation of
distinguishing sequence for each pair.
Si
1
1
1
1
2
2
2
3
3
4
Si
2
3
4
5
3
4
5
4
5
5
(Software Engineering II – Software Testing)
x
baaa
aa
a
a
aa
a
a
a
a
aaa
O(Si , x)
1
0
0
0
0
0
0
0
0
1
O(Sj , x)
0
1
1
1
1
1
1
1
1
0
5. Test Generation – Finite State Models
CS@UNICAM
13 / 31
The W-Method
The W-Method aims at deriving a test set to check the implementation
(Implementation Under Test - IUT) of an FSM model
Assumptions
I
M is completely specified, minimal, connected, and deterministic
I
M starts in a fixed initial states
I
M can be reset to the initial state. A null output is generated by
the reset
I
M and IUT have the same input alphabet
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
14 / 31
The W-Method
The W-Method aims at deriving a test set to check the implementation
(Implementation Under Test - IUT) of an FSM model
Assumptions
I
M is completely specified, minimal, connected, and deterministic
I
M starts in a fixed initial states
I
M can be reset to the initial state. A null output is generated by
the reset
I
M and IUT have the same input alphabet
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
14 / 31
W-Method steps
Given an FSM M =< X , Y , Q, q0 , δ, O > the W-method consists of
the following steps:
1
Estimate the maximum number of states in the correct design
2
Construct the characterization set W for the given machine M
3
Construct the testing tree for M and determine the transition
cover set P
4
Construct set Z
5
P · Z is the desired test set
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
15 / 31
Computation of the transition cover set
P - transition cover set
Let qi and qj , i 6= j be two states of M . P consists of sequences s · x
s.t. δ(q0 , s) = qi ∧ δ(qi , x) = qj for s ∈ X ∗ ∧ x ∈ X . The set can be
constructed using the testing tree for M .
Testing tree
The testing tree for an FSM M can be constructed as follows:
1
2
State q0 is the root of the tree
Suppose that the testing tree has been constructed till level k . The
(k + 1)th level is built as follows:
Select a node n at level k . If n appears at any level from 1 to k − 1
then n is a leaf node. Otherwise expand it by adding branch from
node n to a new node m if δ(n, x) = m for x ∈ X . This branch is
labeled as x.
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
16 / 31
Computation of the transition cover set
P - transition cover set
Let qi and qj , i 6= j be two states of M . P consists of sequences s · x
s.t. δ(q0 , s) = qi ∧ δ(qi , x) = qj for s ∈ X ∗ ∧ x ∈ X . The set can be
constructed using the testing tree for M .
Testing tree
The testing tree for an FSM M can be constructed as follows:
1
2
State q0 is the root of the tree
Suppose that the testing tree has been constructed till level k . The
(k + 1)th level is built as follows:
Select a node n at level k . If n appears at any level from 1 to k − 1
then n is a leaf node. Otherwise expand it by adding branch from
node n to a new node m if δ(n, x) = m for x ∈ X . This branch is
labeled as x.
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
16 / 31
Constructing Z
The set Z
Suppose number of states estimates to be m for the IUT, and n in the
specification m > n. We compute Z as:
Z = (X 0 · W ) ∪ (X · W ) ∪ (X 1 · W ) · · · ∪ (X m−1−n · W ) ∪ (X m−n · W )
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
17 / 31
Deriving a test set – P · Z
Try sequences:
I
baaaaaa
I
baaba
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
18 / 31
Deriving a test set – P · Z
Try sequences:
I
baaaaaa
I
baaba
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
18 / 31
W -method fault detection rationale
I
A test case generated by the W − method is of the form r · s
where r ∈ P and s ∈ W
Why can we detect operation errors?
Why can we detect transfer errors?
P = {, a, b, bb, ba, bab, baa, baab, baaa, baaab, baaaa}
W = {a, aa, aaa, baaa}
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
19 / 31
W -method fault detection rationale
I
A test case generated by the W − method is of the form r · s
where r ∈ P and s ∈ W
Why can we detect operation errors?
Why can we detect transfer errors?
P = {, a, b, bb, ba, bab, baa, baab, baaa, baaab, baaaa}
W = {a, aa, aaa, baaa}
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
19 / 31
W -method fault detection rationale
I
A test case generated by the W − method is of the form r · s
where r ∈ P and s ∈ W
Why can we detect operation errors?
Why can we detect transfer errors?
P = {, a, b, bb, ba, bab, baa, baab, baaa, baaab, baaaa}
W = {a, aa, aaa, baaa}
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
19 / 31
The partial W − method (aka Wp − method)
Wp − method
Main characteristics:
I
It considers minimal, complete and connected FSM
I
is inspired by the W − method it generates smaller test sets
I
uses a derivation phase split in two phases that make use of state
identification sets Wi instead of characterization set W
I
uses the state cover set (S ) to derive the test set.
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
20 / 31
Identification Set and State Cover Set
Identification Set
The Identification Set is associated to each state q ∈ Q of an FSM.
An Identification set for state qi ∈ Q, where |Q| = n, is denoted by Wi
and has the following properties:
1
Wi ⊆ W per 1 < i ≤ n
2
∃j, s.1 ≤ j ≤ n ∧ s ∈ Wi ∧ O(qi , s) 6= O(qj , s)
3
No subset of Wi satisfies property 2.
State Cover Set
The state cover set is a nonempty set of sequences (S ⊆ X ∗ s.t.:
I
∀qi ∈ Q ∃r ∈ S s.t.δ(q0 , r ) = qi
From the definition it is evident that S ⊆ P
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
21 / 31
Identification Set and State Cover Set
Identification Set
The Identification Set is associated to each state q ∈ Q of an FSM.
An Identification set for state qi ∈ Q, where |Q| = n, is denoted by Wi
and has the following properties:
1
Wi ⊆ W per 1 < i ≤ n
2
∃j, s.1 ≤ j ≤ n ∧ s ∈ Wi ∧ O(qi , s) 6= O(qj , s)
3
No subset of Wi satisfies property 2.
State Cover Set
The state cover set is a nonempty set of sequences (S ⊆ X ∗ s.t.:
I
∀qi ∈ Q ∃r ∈ S s.t.δ(q0 , r ) = qi
From the definition it is evident that S ⊆ P
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
21 / 31
The W p procedure (assuming m = n)
The test set derived using the W p − method is given by the union to
two test sets T1 , T2 calculated according to the following procedure:
1
Compute sets P, S , W , and Wi
2
T1 = S · W
3
Let W = {W1 , W2 , . . . , Wn }
4
5
Let R = {r1 , r2 , . . . , rk } where R = P − S and rj ∈ R is s.t.
δ(q0 , rj ) = qi
T2 = R ⊗ W = ∪Kj=1 ({rj } · Wi ) where Wi ∈ W is the state
identification set for state qi
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
22 / 31
W p − method rationale
Phase 1: test are of the form uv where u ∈ S and v ∈ W . Reach
each state than check if it is distinguishable from another one
Phase 2: test covers all the missing transitions and then check if
the reached state is different from the one specified in the model
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
23 / 31
W p − method in practice
W = {a, aa, aaa, baaa}
P = {, a, b, bb, ba, bab, baa, baab, baaa, baaab, baaaa}
S = {, b, ba, baa, baaa}
W1 = {baaa, aa, a}, W2 = {baaa, aa, a}, W3 = {aa, a}
W4 = {aaa, a}, W5 = {aaa, a}
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
24 / 31
W p − method in practice
W = {a, aa, aaa, baaa}
P = {, a, b, bb, ba, bab, baa, baab, baaa, baaab, baaaa}
S = {, b, ba, baa, baaa}
W1 = {baaa, aa, a}, W2 = {baaa, aa, a}, W3 = {aa, a}
W4 = {aaa, a}, W5 = {aaa, a}
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
24 / 31
W p − method in practice
W = {a, aa, aaa, baaa}
P = {, a, b, bb, ba, bab, baa, baab, baaa, baaab, baaaa}
S = {, b, ba, baa, baaa}
W1 = {baaa, aa, a}, W2 = {baaa, aa, a}, W3 = {aa, a}
W4 = {aaa, a}, W5 = {aaa, a}
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
24 / 31
Is it phase 2 needed?
Let’s consider the following FSM:
Now introduce an operation error or a transfer error on a “c” transition
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
25 / 31
The W p procedure (assuming m > n)
Modify the derivation of the two sets as follows:
I
I
T1 = S · Z where Z = X [m − n] · W
T2 = (R · X [m − n]) ⊗ W
Let S = R · X [m − n] = {s|s = r · u s.t. r ∈ R ∧ u ∈ X [m − n]}
then T2 = S ⊗ W = ∪s∈S (s · Wl ) where δ(q0 , s) = δ(δ(q0 , r ), u) = ql
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
26 / 31
UIO-Sequence Method
UIO-Sequence
A UIO sequence is a sequence of input and output pairs that
distinguish a state of an FSM from the remaining states.
UIO(s) = i1 /o1 , i2 /o2 , . . . , in /on s.t.
∀t ∈ Q∧s 6= t.∃j ∈ [1 . . . n].O(δ(s, i1 i2 . . . ij−1 ), ij ) 6= O(δ(t, i1 i2 . . . ij−1 ), ij )
Assumptions
I
M is completely specified, minimal, connected, and deterministic
I
M starts in a fixed initial states
I
M can be reset to the initial state. A null output is generated by
the reset
I
M and IUT have the same input alphabet
I
M and IUT have the same number of states
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
27 / 31
Distinguishing Signatures
Distinguishing Signature or Sequence (DS)
Sequence of input/output labels that is unique to a state s
Minimal transfer sequence
A minimal transfer sequence is a sequence of input/output that brings
the machine from state j to state i along the shortest path Pi (j)
Given a state i a DS can be built using the identification set and
minimal transfer sequences for each state j with j 6= i . In particular for
an FSM M with k states a DS is given by the following concatenation:
DS(qi ) = W (qi , q1 ) · Pi (t1 ) · W (qi , q2 ) · · · Pi (tk −1 )W (qi , qk )
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
28 / 31
Distinguishing Signatures
Distinguishing Signature or Sequence (DS)
Sequence of input/output labels that is unique to a state s
Minimal transfer sequence
A minimal transfer sequence is a sequence of input/output that brings
the machine from state j to state i along the shortest path Pi (j)
Given a state i a DS can be built using the identification set and
minimal transfer sequences for each state j with j 6= i . In particular for
an FSM M with k states a DS is given by the following concatenation:
DS(qi ) = W (qi , q1 ) · Pi (t1 ) · W (qi , q2 ) · · · Pi (tk −1 )W (qi , qk )
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
28 / 31
Test generation
Let M = hQ, X , Y , q1 , δ, Oi an FSM and E = {hqi , x, y , qj i|qi , qj ∈
Q ∧ x ∈ X ∧ y ∈ Y ∧ δ(qi , x) = qj ∧ O(qi , x) = y } the set of edges of
M
1
Find the UIO for each state in M
2
Find the shortest path from the initial state to each of the
remaining states.
3
4
For each edge e = hqi , x, y , qj i ∈ E , build
T E(e) = Phead(e) (1) · label(e) · UIO(tail(e))
where head(e) = qi , tail(e) = qj , label(e) = x/y
Optionally a unique sequence can be derived using reset actions.
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
29 / 31
Assessment of automata theoretic strategies
Control Flow based techniques are typically assessed according to
different criteria:
State coverage
A test set T is considered adequate with respect to the state cover
criterion for an FSM M if the execution of M agianst each element of T
causes eash state in M to be visited at least once
Transition coverage
A test set T is considered adequate with respect to the branch, or
transition, cover criterion for an FSM M if the execution of M against
each element of T causes each transition in M to be taken at least
once
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
30 / 31
Assessment of automata theoretic strategies
Switch coverage (n-switch coverage)
A test set T is considered adequate with respect to the 1-switch cover
criterion for an FSM M if the execution of M against each element of T
causes each pair of transition (tr1 , tr2 ) in M to be taken at least once,
where for some input substring ab ∈ X ∗ , tr1 : qi = δ(qj , a) ∧
tr2 : qk = δ(qi , b) and qi , qj , qk are states of M
Boundary-interior coverage
A test set T is considered adequate with respect to the
boundary-interior cover criterion for an FSM M if the execution of M
against each element of T causes each loop body to be traversed zero
times and at least once. Exiting the loop upon arrival covers the
“boundary” condition and entering it and traversing the body at least
once covers the “interior” condition.
(Software Engineering II – Software Testing)
5. Test Generation – Finite State Models
CS@UNICAM
31 / 31