Optimisation of Inference algorithm to find State
Indentification Sequences on black-box FSM without
reset.
Jean Bouvattier, under tutoring of Roland Groz and Catherine Oriat
Université Grenoble Alpes, France
{Roland.Groz, Catherine.Oriat}@imag.fr
[email protected]­inp.fr
Abstract. Identifying the (finite state) control structure of a black box system
from the traces observed in finite interaction is of great interest for many
model-based activities, such as model-based testing or model-driven
engineering. However, some of them can't be reset, this lead to the creation of
an inferring method based on the knowledge of a w-set : a set of input which
distinguish every state from another by its outputs. However, inferring means
we do not have all information wished and that me do not necessarly have a wset, leading to another problem: the computation of characterization set. This
paper present ways to compute such set from a black-box system by analysing
outputs to great number of input.
1
Introduction
For model-driven software engineering, model-based testing including, it is important
to rely on models of the software artefacts. It is also essential that the models are upto-date. In many contexts, however, such models are not available.
Various algorithms have been proposed for inference of finite state machines by
testing them. Typically, such components could be accessed over a network, so that
we do not even assume that the executable can be scrutinized: the system can only be
observed at its interfaces. This corresponds to a typical black box testing scenario,
where the tester would send inputs to a system and observe its outputs. This task is
often made easier by the existence of a reset procedure that always lead the automate
in the same state, however reset can be sometime too costly or impossible. To solve
this problem, an algorithm that can infer a black-box implementation without
resetting it was created and presented [see references]. Contrary to classical testing
methods which start from a known specification machine and check equivalence and
use transfert sequences known to lead to a state already specified, this algorithm
cannot return to a known state with such method.
Moreover, the equivalent FSM does not have necessary a distinguishing sequence,
this problem has been investigated in and a characterization set is used. Generation of
a checking sequence (without reset) from a known specification FSM with a
characterization set is too expensive to be used.
This algorithm assumes that a set W of input sequences that characterize the
different states of the black box is provided: each state produces a different set of
output responses to these input sequences. Such a set could be derived from a
previous version of the software or from domain-specific knowledge. But such a set is
not always given.
No algorithm exist that can infere a Black-Box System without a characterisation
set as this algorithm is the first of its kind that does not refers neither to an oracle nor
can reset.
This paper propose an analysis of the size of the characterization set computed by
naïve method on glass-box system, then to determine what would be a good method
to compute a characterization set from a black-box system. Once this characterization
set is calculated, by applying the initial algorithm we would obtain an infering method
that does not rely on reset, oracle or any pre-given characterization set.
The main algorithm use a localizer, because the method cannot rely on any
procedure leading into an unique state, we use a specific method that move to a state
where we know what would be answers to all inputs of the characterization set. A
particularity of this method is that it can be used with any set and not only a
characterization set, this features can then be used with incomplete characterization
set to distinguish states and build a reliable characterization set.
A quotient machine built from an initial machine and a specific set is the state
machine where we only consider sequence from the set as possible input. We then
build a smaller machine.
By then adding sequence to the set, we slowly approach of a much reliable
automata and can except to find a characterization set that can distinguish all states.
2
Definitions
2.1
Basic definitions
A Finite State Machine is a complete deterministic Mealy machine. Formally, a
Finite State Machine (FSM) M is a 6-tuple (S, s0, I, O, , ), where




S is a finite set of states with the initial state s0,
I is a finite set of inputs, and O is a finite set of outputs,
 : S  I  S is a transition function, and
 : S  I  O is an output function.
 and are actually mappings, i.e., dom() = dom() = S  I since we only
consider complete machines. As M is deterministic, a tuple (s, x)  S  I uniquely
determines a transition of M. For simplicity we use (s, x) to denote the transition, thus
omitting its output and final state. We extend the transition and output functions from
input symbols to input sequences, including the empty sequence , as usual: (s, ) =
s and (s, ) =  for s  S; for x  I+, (s, x) = ((s, ), x) and (s, x) = (s,
)((s, ), x). An FSM M is said to be strongly connected, if for each pair of states
s, s'  S, there exists an input sequence   I*, such that (s, ) = s';  is called a
transfer sequence.
Two states s, s  S are distinguishable, if there exists   I*, such that (s, ) 
(s, ). We say that  distinguishes s and s. Given a set H  I*, states s and s are Hequivalent, if (s, ) = (s, ) for all   H. Otherwise, i.e., if there exists   H such
that (s, ) ≠ (s, ), the states are H-distinguishable. We define H-distinguishability
and H-equivalence of machines as a corresponding relation between their initial
states. An FSM M is minimal, if all states are pairwise distinguishable. In this paper,
the machines are assumed to be minimal and strongly connected. A set of inputs W is
a characterization set for an FSM M if each pair of states is W-distinguishable.
A sequence of input/output pairs   (IO)* is a trace. Given traces , , and ,
such that  = , we write  \ , when we need to refer to the result of deleting the
suffix  from the trace , resulting in trace . The notation    is used to refer to a
prefix  of the trace ; A, where A  I  O, denotes the projection of  to A,
obtained by removing symbols that are not in A from the trace . Formally, A = ;
if x  A, then (x)A = (A)x; if x  A, then (x)A = (A).
Given a machine M = (S, s0, I, O, , ) with a state si and an input sequence   I*,
we use tri() to denote the trace from state si such that tri()I =  and tri()O =
( si, ). Given a characterization set W, rather than naming or numbering states (such
as “si”), we may refer to a state by its state characterization Tri(W) = {tri(w) | w  W}.
A state characterization Tri can be seen as a mapping qi from W to (IO)* such that
qi(w) = tri(w). The set of all mappings Q = {q1, q2, …, qn} corresponds to the set of
states of the machine. Namely, for q  Q and s  S, we write q  s if w  W,
q(w)O = (s, w). Inferring an unknown FSM with the same characterization set,
each mapping q  Q can then be considered as its state. In the sequel, we use B
(instead of the general M) to refer to a black-box machine, P (instead of S) to refer to
its states, whereas the inferred model (also called “conjecture”, in line with L *) will
have states denoted by Q.
3
Inference procedure
Main procedure:
A localizer procedure L(, W) is used to ensure that we continue learning
transitions from states visited before. The localizer procedure, given a current trace ,
produces an updated trace ', a set of traces Tr(W), and the last appended trace tr(wp)
to label the state which has produced it and is thus identified with Tr(W). Recall that a
state is defined by its answers to sequences from W, i.e., by Tr(W). At the end of the
localizer, we know the BB is in state (Tr(W), wp), and the state reached after \tr(wp)
can be labelled as Tr(W).
Once in state (Tr(W), wp), we then procede to identify this state by sending input
and observing output. It can be done once and then, the localizer is used again to
continue learning. If, however, this state is completely characterized, we progress
furthermore in the machine until we reach a state not completely characterized.
Localizer procedure:
In the inference procedure, the main use of the localizer procedure is to ensure that
when the BB is in an unknown state q, we can restart a test from a state q' of the BB
for which we can be certain that we know all its traces for all sequences from the Wset, and thus we can identify state q' of the BB. Actually, it is a generalized procedure
that can work with any set of sequences Z, where Z may not be a fully characterizing
set, i.e., there could be non-equivalent states in the BB that are Z-equivalent. The
localizer procedure will be defined recursively with increasing subsets Z of W.
The key idea is that since the number of states is finite and bounded by n, by
repeatedly applying a given input sequence  we can observe at most n different
output sequences. This implies that in the state reached applying n times  the BB
must have reached a cycle, coming to one of the states visited after some i. But we
do not know which one because different states could still have the same output
response to . The length of the cycle is itself bounded by n. It is enough to repeat 
another n - 1 times (so 2n - 1 in total) in the worst case to identify the cycle and be
able to know what would be the response from the BB to the 2n-th application and all
subsequent applications of .
This means that after 2n - 1 applications of , we are in a state p and we can apply
another input sequence , and be sure that we know the response of p to both  and .
If {, } is a W-set, we can continue a test from a state p that has been fully
characterized, or at least from (p, ). Jumping to more than two sequences is a bit
more elaborate, but it is ensured by the recursive definition of the localizer.
This algorithm was originally designed for inference but is used into the
calculation of a characterization set to move into states with different possible
responses.
5
Complexity
This inferencing procedure has been shown to have an expensive time cost. With n
the number of state and p the size of the characterization set, we have a cost in O(p(f
+ p) 2pnp+2), polynomial in n and exponential in p. Having a small characterization set
is very important and is the main problem of this work.
6
Statistic analysis of common characterization set
Algorithm already existed to find characterization set from known Finite State
Machine. In particular a naive one was already implemented on the framework and
was used to fix goals for characterization set size. We generated 1000 automates of
size n for each n between 2 and 18 and analyzed the size of the characterization set
computed with this naive method. This method try a random sequence and search for
undistinguished states, he then take another sequence that discriminate both states. It
repeats until all states are distinguishable. Even if it is really naive, it has however
access to the system representation and gives good results:
Fig : percentage of characterization set of
different size compared with size of automates
On all those automates, none gave a characterization set of size 5 or more, many
had a size of 1 which is the best case but those case became very rare as the number
of state increased. Instead, the methods seems to find W-set of size 2 and 3 more
often. The case of characterization set of size 2 and 3 also seems to become stable and
much more reliable in most automate of great size.
In order to obtain a good W-set from a black-box system, we aim to be able to
create w-set of size 4 maximum in most case.
7
Finding distinguishing sequences
The first step of our procedure to find small w-set on black-box machine is
navigate in the system while observing outputs. Different sequences of input/output of
maximum size are then memorized in tables. These tables link each observed
sequences of input to the possible outputs. Our method applied random input but best
strategy could have been used, like using some sequence that looked much more
discriminant to learn more about their potential, it was however decided to use
random input as it is a better comparator between the discrimination potential of
different sequences: Assumption on the automate imply we can never get stuck on
some parts, by applying random input, we assure that in most case, by sending an
infinite number of input, we would go everywhere and then treat every transition.
Example:
With input alphabet {a,b} and output {0,} we decide to store all sequence of 4
inputs or less, begin to apply random input to a black-box system and observe the
results:
1. we apply a and observe a0 : we put a trace a0 in our tables.
2. We apply b again and observe b0, with a0b0 we now have (a0,b0,a0b0) in
out tables, all subsequence are stored.
3. We apply a again and observe a1, with a0b0a1, the table is now (a0, a1, b0,
a0b0, b0a1, a0b0a1). The sequence “a” is now the most distinguishing
sequence in our table, it could distinguish two states.
In this example, any subsequence beginning with a would distinguish at worst as
much state as a, however, the inference algorithm also rely on sequences size so we
decide to keep sequence the smaller possible. This is also why, subsequences stored
are limited by size.
Test:
By generating 200 automate of size n for each n between 2 and 20 with each time
1000 inputs given, we obtained large tables linking outputs to inputs. We then sorted
them by the size of the most distinguishing sequence. What we got as results is:
 the percentage of automate with a single sequence distinguishing all states is
decreasing but slower than the previous naive algorithm on transparent
system. This lead us to think that this method could provide a truly reliable
way to compute w-set.
 Even automate of size 19 always have good distinguishing sequence. In half
the case, their best distinguishing sequence had a size of 16 or more.
Moreover, best discriminant sequence are sometime of size 6 or more,
Fig : percentage of automate
distinguishing sequence found
.
8
with
Creating characterization set from previous data.
System with a distinguishing sequence are considered as solved because a
singleton is known to be a characterization set. However, in many case such a
sequence is not discovered or does not exist. Such a set should then be computed from
the same black-box system. The goal is now to use different distinguishing sequences
to form characterization set.
The basic strategy is to add word to an originally empty set to distinguish a certain
number of state, while this number of distinguished state is not equal to the total
number of state, we continue to add words to the set.
The challenge is still to know how to distinguish two state with the same output to
a word but different outputs to another, a solution proposed was to adapt the inference
algorithm localizer to smaller set, the localizer propose a way to move one step ahead
of a state to which we know all answers to a set of input. It can so demonstrate the
existence of states with different outputs
Example:
Lets suppose that after applying a large number of input to a machine we decide to
use its most distinguishing sequence w1 as the first word of our characterization set Z,
it can distinguish a certain number of state but some still have the same outputs. The
algorithm decides to chose a second word w2 and to use the localizer, it return the
automate into a state (p, ), where p answers to w1 and w2, (p, w1) and (p, w2) are
known. If later the localizer is used again and return the automate into a state (q, w2)
we can then match answers from p or q. if (p, w1) = (q, w1) and (p, w2) ≠ (q, w2)
then w2 distinguish p from q although they are not distinguished by w1. We discovered
a new state.
Characterization set calculator algorithm:
% W is the characterization set
% Q is the list of state referenced by their answers to W sequences
W= Q=
While W does not distinguish n states. % n is the known system number of state.
Choose w;
W := W  {w};
for I in 0 .. m do
localize into (q, ) with set W;
update(q,Q);
% update list of state with q description.
done
end while
the update function see if q answers to W sequence match known states. If it does
not match any state, it is added to Q:
match procedure (q,Q) :
if q Q:
return;
else if pQ s.t   ((p,  ) = (q,  ) or (q,  ) = 
% in this case, q already match a state p of Q, this state is updated.
p:= q;
else
% p then describe a new state
Q := Q  {p};
end if
Such algorithm was implemented but revealed not to always work: even if the
localizer always lead to a state where outputs are known, applying many time the
same input can block the automate in some states, even if random sequence are
applied, some states can never be obtained as result from localizing procedure.
However, some good results were obtained, in many cases it discovered set of size
2 that distinguished all states.
9
Discussion and amelioration
Even if the algorithm isn't completed, it showed promising results. A simple way to
evolve it into a reliable algorithm could be to use the same principe as in the initial
inference procedure : by infering the system into a Finite State Machine by
considering a smaller set of sequence than a characterization set, we can find a
Quotient machine of the solution machine, an automate where input alphabet is the
used set of sequence. We could then move to wanted states and try to distinguish
states by their output from other words than those of the chosen set.
Moreover, many point of the algorithm could be enhanced : the choosing function
that select a word to add to the characterization set actually picks the most
discriminant sequences but adding two alike sequences is not the best choice as they
are less likely to distinguish states.
Acknowledgments
I acknowledge Roland Groz and Catherine Oriat for tutoring during my period in
the laboratory LIG. The initial algorithm was implemented by Nicolas Brémond, an
intern from Enseirb-Matmeca building the root of all my work done there.
References
1. Roland Groz, Adenilso Simao, Alexandre Petrenko, Catherine Oriat. Inferring finite state
machines without reset using state identification sequences. International Conference on
Testing Software and Systems, (ICTSS 2015), Dubai, Nov 2015