IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 17, NO. 2, MARCH 2006
397
Basins of Attraction in Fully Asynchronous
Discrete-Time Discrete-State
Dynamic Networks
Jacques M. Bahi, Member, IEEE, and Sylvain Contassot-Vivier, Member, IEEE
Abstract—This paper gives a formulation of the basins of fixed
point states of fully asynchronous discrete-time discrete-state
dynamic networks. That formulation provides two advantages.
The first one is to point out the different behaviors between synchronous and asynchronous modes and the second one is to allow
us to easily deduce an algorithm which determines the behavior of
a network for a given initialization. In the context of this study, we
consider networks of a large number of neurons (or units, processors, etc.), whose dynamic is fully asynchronous with overlapping
updates . We suppose that the neurons take a finite number of
discrete states and that the updating scheme is discrete in time.
We make no hypothesis on the activation functions of the nodes, so
that the dynamic of the network may have multiple cycles and/or
basins. Our results are illustrated on a simple example of a fully
asynchronous Hopfield neural network.
Index Terms—Asynchronism, Hopfield networks, networks dynamic.
I. INTRODUCTION
T
HE role of parallel iterative algorithms is essential in the
domain of scientific computation. An important class of
such algorithms is the discrete-time discrete-state networks
which are useful in numerous applications. Such networks are
usually described as a collection of neurons such that each
neuron takes a finite number of discrete values. If the value
, the global state of the
of neuron is noted ,
and the set of
system is then described by
global states is
where is the finite set of values which can be taken by neuron
. The dynamic of the network is given by the activation function
such that
where each is the activation function of neuron . Those
are supposed to be general and are not restricted to threshold
networks. The global state of the network at the discrete time
(also called iteration ) is denoted by
Manuscript received May 12, 2003; revised July 11, 2005.
The authors are with Laboratoire d’Informatique de L’Université de
Franche-Comté, IUT Belfort-Montbéliard, 90016 Belfort, France (e-mail:
[email protected]; [email protected]).
Digital Object Identifier 10.1109/TNN.2005.863413
Furthermore, the most general execution mode of those networks is the fully asynchronous one. In the literature, different
models of asynchronism are used depending on the way the
communications are managed and the updates are performed
[1]–[3]. In this paper, we consider the fully asynchronous mode
with overlapping updates in the sense defined by Herz and
Marcus in [3].
• The neurons of the network may be updated in a random
order and, moreover, it is possible that some neurons may
not be updated at some times.
• At each time , each neuron updates its own state using
the last received information from the elements (neurons)
it depends on, rather than waiting for their states at time
.
This model corresponds to the most general one which incorporates the sequential, parallel and block-sequential cases. For
a more detailed description of those cases, see [4] and the references therein.
In fully asynchronous networks, since only some of the neurons may be updated at each time , there is a need to define what
, which corresponds to the
is called the strategy, denoted by
set of neurons updated at time . Moreover, since the updating
of each neuron may use values of other neurons computed at
the state of neuron availdifferent times, we denote by
able for neuron at time :
, where
denotes the delay of neuron with respect to neuron . Finally,
some classical conditions are assumed on the
in order to
ensure that the process actually iterates and so evolves.
Definition 1.1: Let us consider an -neuron network and the
, a sequence of subsets of the neurons.
strategy
For
, let
be a sequence of
, such that
c1)
with
,
being the
delay of neuron according to neuron at the discrete
time ;
,
, i.e., although
c2)
the delays associated with neuron are unbounded,
they follow the evolution of the system;
c3) No neuron is neglected by the updating rule. This condition is called fair sampling condition and is equiva,
.
lent to
Then, the fully asynchronous dynamic of the -neuron network associated to the activation function and to the strategy
, and with initial configuration
is described
by Algorithm 1.
1045-9227/$20.00 © 2006 IEEE
398
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 17, NO. 2, MARCH 2006
Algorithm 1 Asynchronous iteration
Given an initial state x = (x ; . . . ; x )
for each time step t = 0; 1; . . . do
for each component
if i 2 J (t) then
x
=
f
=
x
x
;.
i
= 1; . . . ; n do
..;x
else
x
end if
end for
end for
Such asynchronism is useful and even sometimes inherent to
natural and physical systems. Moreover, it enables parallel iterative algorithms to be efficiently implemented on a large scale,
using several machines scattered on different sites which can be
far from each other. Such a context is commonly named grid
computing in the literature (see, for example, [5]).
Unfortunately, the dynamic of such asynchronous networks
is far more difficult to estimate and, thus, to control than those
of synchronous ones. Hence, additional studies are necessary
to define the conditions which ensure their convergence. Numerous studies have been done in this domain such as [2],
[6]–[10]. In the same way, a lot of studies have focused on
the global dynamic [7], [11]–[13] or on the conditions to have
cycle-free dynamics in such networks. Nevertheless, to the best
of our knowledge, there is no study which directly deals with
the description of the complete basins of attraction of the fixed
points (see Definition 2.1) of discrete asynchronous networks,
neither in the asynchronism nor in the neural network literature.
This is however an important issue in order to well understand
the behavior of asynchronous networks.
In [4], our goal was to find as many initial conditions as possible which make a network converge toward a given fixed point.
We have expressed sufficient conditions to extend the estimated
basin around a fixed point by using the discrete derivative. We
have then proposed an algorithm which, for a fixed point of the
considered network and an initial input, tells us if this input
makes the network converge to this fixed point in the fully asynchronous mode.
In this paper, we propose a new result which completely describes the attraction basin of a fixed point of a given network in
fully asynchronous mode, whose activation function is known.
An additional result is an algorithm which determines the resulting state of a given network in fully asynchronous mode for
a given input.
According to this scope, the closest works are those of Robert
[1], [14] who studied local attractions in compact neighborhoods of a fixed point (subpart of the basin), and Pellegrin [15]
who proposed some verification algorithms of those attraction
properties. Most of the other studies related to the basins of attraction took place in synchronous networks.
In the particular context of neural networks, some authors
have tried to study asynchronism in this kind of networks [3],
[16]–[19]. Nevertheless, they used a rather quite different approach since most of them focused on conditions on the network
to ensure convergence or cycle absence.
In the context of our paper, we consider general discrete networks which may have several cycles and several fixed points.
The final goal of this paper is to get as much information as
possible about the dynamic of fully asynchronous networks in
order to be able to modify their construction and/or configuration to exactly obtain the desired behavior. The direct application to neural networks is to provide an efficient and accurate
tool to study their dynamic. Hence, we expect that these results
will be helpful in future works dealing with the enhancement
of the design of recurrent neural networks used, for example, as
associative memories.
In Section II, all the notions and definitions required for the
description of the attraction basin of a fixed point are presented
together with the underlying mechanisms and justifications.
Then, the definition of the attraction basin is given in Section III
together with a deduced formulation of the global dynamic of
an asynchronous network. The proof of this result is placed
in Appendix I for a faster reading. The mechanisms of the
construction of the basin are pointed out in some examples in
Section IV. Then, a brief discussion about the generalization of
our result to block-decomposed networks is given in Section V
and a determination algorithm based on our formulation is described in Section VI. An example of application to a Hopfield
network is finally given in Section VII.
II. PRELIMINARY DEFINITIONS
A. Generalities
Definition 2.1: A fixed point state
verifies
which implies in the context of Algorithm 1 that
,
Once such a state has been reached, the network remains in
this state forever although its components are updated. In other
words, the updating of each component leads to the same value.
In this case, we say that the network has converged to this fixed
point state. That kind of state must be distinguished from those
of stable state. The latter is an extension of the first one in which
an additional constraint is set on the neighborhood of the state.
Hence, a stable state is a fixed point state whose neighboring
states (at least all the ones which have one different component
from it) also lead to it in a finite number of iterations. It is important to notice that this is not necessary the case for a fixed
point state. In the remaining of the paper, we use the expression
fixed point as a shortened form for fixed point state.
is
Definition 2.2: The basin of attraction of a fixed point
in a
defined as the set of all the states which surely lead to
finite number of iterations.
Hence, all vectors belonging to a cycle in the iteration graph
or which can lead to different fixed points from one execution to
another cannot be included in such a basin. To study the basins
of fixed points in such asynchronous systems, we use the discrete distance between two states and of the network given
by
(1)
BAHI AND CONTASSOT-VIVIER: BASINS OF ATTRACTION IN DISCRETE-TIME DISCRETE-STATE DYNAMIC NETWORKS
where
if
if
.
This so-called vectorial distance was introduced by Robert [1],
[14] in the context of component-wise chaotic discrete boolean
iterations, which is a particular case of block-asynchronous iterations on finite sets, and extended to finite sets in [20].
In the following, we suppose that we know the activation
of the network, and we want
function and a fixed point
to find all the vectors which surely make this network converge
in fully asynchronous mode.
to
399
In the first case, all the components will have the same value
at the next iteration independently of the updates. In the second
case, there is only one component which may change. In this
case, we are still confronted to indeterminism since there are
two possibilities (the current state or its image by ). Nevertheless, the hypothesis made on the delays in fully asynchronous
mode ensures us that updates occur in a finite time. Hence, the
evolution of the system from may be itself during a finite
. This means that those two
time and then will surely be
cases can be assimilated to deterministic evolutions.
In the following part, we give some useful tools required for
the formulation of the basin of a fixed point.
C. Additional Tools
B. Fundamental Difference Between Synchronous and
Asynchronous Iterations
All the difference between the synchronous and asynchronous modes lies in the number of possible successors of a
given state of the system.
In the synchronous case, the process is completely deterministic and, to each state of the system, there corresponds only one
successor.
This is not the case in the asynchronous mode. To each state
of the system, there may correspond several possibilities of evolution for the next iteration depending on which components are
updated.
and
Let us consider the example of a 3-node system in
. Then, in syna transition function such that
chronous mode, the state following 011 is surely 000 whereas
there are four possible outcomes in the asynchronous case depending on which components are updated. In our example, the
first component stays the same whether it is updated or not.
Thus, the possible evolutions only depend on the updates of the
last two components. In the following list of the four possible
cases, we do not specify the behavior of the first component
since it leads to the same value:
• 000: The last two components are updated ( sync case).
• 001: The third component is not updated.
• 010: The second component is not updated.
• 011: The last two components are not updated.
Hence, we clearly see here all the difference between those
two kinds of iterations. Synchronous iterations are deterministic
whereas asynchronism induces a nondeterministic evolution of
the system.
It is important to notice that there may be some components
for which the updating is equivalent to the standby. These components are the ones whose values do not change between and
. In our example, it is the case of the first component whose
evolution at the following iteration is the same for updating and
standby. This allows us to give the local conditions under which
the asynchronous mode of execution is equivalent to a deterministic case.
For a given state of the system, we can deduce two cases
where asynchronous iterations are equivalent to synchronous
ones:
• the state is a fixed point;
.
• there is only one different component between and
For any couple of vectors
, we define the set of vec, which contains all the possible mixings of
tors
the component values of and . This construction directly depends on the nonzero elements in the vectorial distance
.
Definition 2.3:
where
if
if
.
We also define the asynchronous successor relation between
), which means that
two vectors and (denoted by
and can be two consecutive states in the asynchronous evolution of the system.
,
, if and only if
Definition 2.4:
and we extend that notion to the asynchronous iteration path
relation between vectors and (denoted by
), which
means that state can lead to state through a sequence of
asynchronous iterations.
,
if and only if there exists a
Definition 2.5:
,
, possibly empty,
sequence
of distinct states of such that
To describe the basin of a fixed point, we also need to define
what we consider to be a cycle in a fully asynchronous network.
D. Characterization of Cycles and Pseudocycles
The difficulty to characterize the cycles in the fully asynchronous mode comes from the fact that, contrary to the
synchronous mode, there are two possible kinds of cycles
in the asynchronous case: The cycles which are attractors of
the system and the ones which are not. Obviously, only the
attractor cycles are relevant in the study of the dynamic of
such networks. Those cycles are described in this subsection
together with complementary notions.
In the asynchronous mode, the general evolution of the
system is directly linked to what in the literature are commonly
called pseudoperiods [21].
400
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 17, NO. 2, MARCH 2006
Fig. 1. Pseudoperiod.
Definition 2.6: A sequence of states
,
is a pseudoperiod of a fully asynchronous network associated to
the set of states and the activation function , if and only if
Fig. 2. Pseudocycle does not contain a complete pseudoperiod.
and
Fig. 3.
where denotes the component-wise logical
operator.
and folThis definition means that starting from the state
, the state
is the first state
lowing the sequence
at which all the components of the system have been updated at
least once since . The first condition ensures that there is no
component which is not updated in the sequence. If the compois updated at the following iteration, then
nent of
we have
and
. Thus, to obtain a null vector, all the components must be updated at least
is
once in the sequence. The second condition ensures that
necessary to have a null vector and, thus, is the first state in the
sequence at which all the components have been updated at least
once.
The necessity of definition 2.6 comes from the fact that, on
the one hand, Algorithm 1 implies that, at each iteration, some
components of the system may not be updated (inducing indeterminism in the system). On the other hand, according to hypotheses c3) in Definition 1.1, it is not possible to have a component which is never updated. Thus, for any time during the
such that all the
execution, there exists a finite integer value
components of the vector have been updated at least once be. Fig. 1 gives an example of a pseutween iterations and
doperiod in a system of four components beginning in A and
finishing in B. The arrows represent the iterations and the arrays
indicate which components of the system are updated at each iteration (white cells not updated, black cells updated). It can
be seen in this figure that the superimposition of all the arrays
gives an array full of black cells. Moreover, some components
are updated several times in this sequence.
Thus, concerning the cycles in such networks, there are two
possibilities based on the fact that either the cycle does not contain any complete pseudoperiod or contains at least one.
The first possibility corresponds to what we call pseudocycles. This naming comes from the fact that, according to the
asynchronous model, it is not possible to stay infinitely in such
a cycle. Let us consider the example given in Fig. 2 and suppose that the system enters the cycle through the state . It can
be seen that when the system goes through the cycle and comes
back to , the third component has not been updated. This and
the fact that there is no complete pseudoperiod in the sequence
imply that the updating of that third component for any state
in the cycle will lead to another state outside that cycle. This
Cycle contains at least one complete pseudoperiod.
leaving of the cycle will happen in a finite time. Thus, this cycle
cannot be gone through infinitely and then it does not represent
the attractor of the system.
By symmetry, the second possibility corresponds to the attractor cycles which are simply called cycles. Since they contain
at least one complete pseudoperiod, as can be seen in Fig. 3,
there is no condition which forces the system to leave such a
cycle which can thus be gone through infinitely. Thus, they can
be considered as attractors of the network.
Finally, the cycles of an asynchronous network can be defined
as follows.
Definition 2.7: Let us consider a fully asynchronous network
associated to the set of states and the activation function .
,
of states of forms
The sequence
a cycle of length in the asynchronous iteration graph of this
network, if and only if
and the sequence
contains at least one pseudoperiod.
It can be noticed that this formulation also includes the fixed
points of the system since they form a cycle of length one in the
.
particular case where
the set of states
In the same way, we denote by Cycle
belonging to at least one cycle in the asynchronous dynamic of
the network associated to the set of states and to the activation
function . Similarly, the set of states in pseudocycles is given
in the following definition.
Definition 2.8: The set of states outside any cycle and involved in at least one pseudocycle containing a given vector
Cycle
is defined by
Cycle
such that
Cycle
This description specifies that there exists an iteration path
from state to itself which goes through state . However,
since all the elements of the sequence
are taken outside
BAHI AND CONTASSOT-VIVIER: BASINS OF ATTRACTION IN DISCRETE-TIME DISCRETE-STATE DYNAMIC NETWORKS
Cycle
, this path can only be a pseudocycle. It can be
Cycle
,
since,
noticed that
.
by definition, we always have
Finally, those previous definitions allow us to describe the
asynchronous convergence toward a fixed point (denoted by
).
be a fixed point of the network assoDefinition 2.9: Let
ciated to the set of states and the activation function .
,
, if and only if
and
such that
and
Cycle
Using all those definitions, we can now describe the basin of
a fixed point state in a fully asynchronous network.
III. BASIN OF A FIXED POINT
(denoted by
Definition 3.1: The basin of a fixed point
) of a fully asynchronous network associated to the set
of states and the activation function can be described by the
,
, recursively defined as
union of the sets
follows:
Cycle
and
let
401
mutual dependence for the inclusion. We can do that because
all the elements of a pseudocycle are equivalent in terms of evolution (this can be easily verified). Thus, not taking into account
the other elements of a pseudocycle does not make us miss any
vector of the basin.
is
Finally, the last important remark is that
. Effectively, although the
always smaller than
vector
is in
, it cannot be in
.
The contrary would imply the presence of a pseudoperiod in an
to
through and, thus, a cycle,
iteration path from
leading to a contradiction with Definition 2.8. Hence, the set
is always nonempty. This definition leads to the following
description of the dynamic of fully asynchronous networks.
Theorem 3.2: Let be a fixed point of a network associated
to the set of states and the activation function . If we consider
, then we have the following.
, all the asynchronous executions of the
•
system starting from lead to the state .
, asynchronous executions starting from
•
may either:
, meaning
— always lead to the same fixed point
all the executions of the network will lead to the same
fixed point different from ;
— lead to different fixed points (eventually to
),
meaning different executions of the network will lead
to different fixed points;
— lead to a cycle, meaning executions of the network may
not lead to a fixed point and so may not terminate.
For the complete proof of this theorem, see Appendix I.
IV. SOME EXAMPLES
In order to exhibit the construction mechanism of the
sets, several examples corresponding to different cases are presented.
The goal of our first example is to clearly show the increset, and to point out the
mental way of construction of the
particular order of inclusion of the elements in its subsets. It
consists of a particular case of global convergence toward the
. The graphical representation of its tranunique fixed point
sition function is given in Fig. 4.
From , we obtain the following sets:
Cycle
This description takes into account all the indeterminism induced by the full asynchronism. It is based on the fact that one
in the fully asynchronous mode if
state will always lead to
and only if all the possible evolutions starting from lead to
without going through any cycle. Hence, the basin is recursively
built by using a reformulation of this convergence property: A
state is in the basin of in the fully asynchronous mode if and
only if all its possible successors at the next iteration are also in
that basin. This is why the description in Definition 3.1 is based
, which allows us to describe the whole set of
on
possible asynchronous successors of at the next iteration.
Moreover, cycles and pseudocycles are also taken into account. All the elements in cycles are systematically discarded
and elements in pseudocycles are not directly involved in the
inclusion of a given vector since they would clearly lead to a
Then, the basin of 000 is fully described by the following
subsets:
402
Fig. 4. Graphical representation of transition function g .
In this example, it can clearly be seen that each element is
subset only if all its possible asynincluded in an
chronous evolutions at the following iteration, except those
subin pseudocycles, are already in the previous
which is
sets. This is particularly striking for the vector
the last one included in the set whereas it is closer to
than some other vectors and its transition is directly
.
In fact, it can be included only when all the elements in
are
already included. The second element of this set being only in,
cannot be included before the
cluded in
subset.
There is no cycle in this example nor in the following one. Yet,
such a case is given in the last example. Moreover, in the folset as the union
lowing examples, we directly write each
subsets in their building order without naming
of their
them.
Another example is the possibility for the network to converge toward different fixed points in a series of executions when
initialized with a same vector. This case is assimilated to a divergence and can be obtained by a slight modification of the
and
transition function into a function so that
for the other vectors. Its graphical representation
is given in Fig. 5.
Then, the two basins are
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 17, NO. 2, MARCH 2006
Fig. 5. Graphical representation of function h.
Fig. 6. Graphical representation of function w .
included in a unique basin. Here again, our construction given in
Definition 3.1 does not include such a point in any of the basins.
The last example which may arise is the presence of a cycle in
the asynchronous iteration graph. If we use the transition function , a graphical representation of which is given in Fig. 6,
and
form a cycle in the
it can be verified that vectors
asynchronous mode.
Nevertheless, in order to exhibit the mechanisms of Definition
.
2.7, we detail the deduction of the cycle
implies
As seen in previous examples,
Thus,
. Moreover, since
and the following chain can be built:
, we have
and
and
and
is not included in any of the
sets since it may lead
or to
from an execution to another. This comes
either to
, since
from the fact that, when starting from
, as already detailed in Section II-B with another example,
there are four possible evolutions (
and
) in
fully asynchronous mode, depending on the eventual updating
of each of the last two components.
, as
The first and third cases lead to convergence toward
already seen. The second case can only evolve toward
since
and
.
there is only one difference between
. The fourth case is the initial vector itself and,
It leads to
according to the hypothesis on the delays, it is sure there will be
a time at which at least one of the last two components will be
updated, leading to one of the other cases.
may
Thus, it can be seen that executions starting from
or to
implying that this vector cannot be
lead either to
This means that
quence (
is actually a cycle since the se) contains a pseudoperiod, and we have
Cycle
Another interesting aspect of this example is the pseudocycle
and
. Effectively, if we consider their
formed by vectors
respective
sets, we have
and
BAHI AND CONTASSOT-VIVIER: BASINS OF ATTRACTION IN DISCRETE-TIME DISCRETE-STATE DYNAMIC NETWORKS
We can build the chain
which denotes a potential cycle. Nonetheless, the evaluation of
its corresponding vector gives
403
where each ,
, is the collection of the activation
functions of the neurons in block .
Since each
is a collection of finite-state components, its
set of values, , is also a finite set of discrete values given by
and the set of global states can then be reformulated as
which reveals that this chain does not contain a pseudoperiod.
) is a pseudocycle, and we have
Thus, the sequence (
Hence,
and
are included in
since all their possible evolutions except those in their pseudocycles are either in
or in
.
The results for this example are then
and
Finally, we can see that Definition 3.1 allows us to build the
basins of all the fixed points of a fully asynchronous network
whose transition function is given. In Section V, we explain how
this result can also be applied to block-decomposed networks.
V. DISCUSSION ABOUT BLOCK-DECOMPOSED NETWORKS
To be clearer, the results given in this paper have been presented in the context of a simple discrete-state discrete-time network, in which each component evolves asynchronously according to the other components. However, those results also
hold in the context of block-decomposed networks. This comes
from the fact that a block-decomposed network can be reformulated as a nondecomposed discrete-state discrete-time network
as shown in this section.
Starting from the formulation of an -neuron network used
in the previous sections, we have
where each
is the finite set of the values which can be taken
by neuron .
blocks, each one
Let us partition this network into
neurons. Thus, the blocks
,
containing
are distinct nonempty subsets of the set of components
. The global value of the network is described
. In this context, it can be noticed that
by
is a particular case of
where
and each block contains exactly one neuron.
Concerning the dynamic of the block-decomposed network,
the activation function is partitioned in a compatible way into
a function such that
Thus, for a network with components ,
, associated to the set of states and to the activation function , a
block-decomposition of this network into blocks can be seen
,
as a nondecomposed network with components ,
and to the activation function
associated to the set of states
. In this way, the results presented in this paper can also be
applied to block-decomposed networks.
In Section VI, we propose an algorithm, based on our formulation of the basins, which determines, for a given initial vector,
the result of the fully asynchronous evolution of a given network.
VI. DETERMINATION ALGORITHM
In the first part of this section, we propose an algorithm which
determines, for an initial state , the result of the fully asynchronous iterations of a network whose activation function is
given. The second part is dedicated to the evaluation of the complexity of that algorithm.
A. Algorithm
As explained in [4], since the synchronous mode is a particular case of the asynchronous one, a necessary condition to have
starting from
in fully
convergence toward the fixed point
asynchronous mode is that leads to in synchronous mode.
Hence, our testing algorithm is decomposed into two main steps
reached when starting from
•
find the fixed point
in synchronous mode;
•
if it exists, test if
is in the basin of
in asynchronous mode.
In the first step, cycles may be detected in the synchronous iteration graph. If this is the case, the second step is not necessary
the system may not
since we can directly deduce that from
converge in a finite time in fully asynchronous mode. In fact, the
nondeterministic behavior of asynchronism may open the cycles
in the iteration graph. This means that if the network is in a state
belonging to a cycle, there may be an evolution which will lead
to a state outside the cycle. If so, there is a possibility to reach a
fixed point. Nevertheless, this cannot be seen as a convergence
since the cycle in the iteration graph still exists and the leaving
of the cycle is not ensured in a finite time.
Algorithm 2 details the first step of the process. It consists
in parsing the iteration graph of the synchronous mode starting
from in order to find either a cycle or a fixed point. To manage
cycle detection, a history of the transient states in the traversal of
404
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 17, NO. 2, MARCH 2006
the iteration graph is stored in the stack stateList. When a fixed
point is found, Algorithm 3 is executed.
= x is already in stateList, we have to test if
the chain of vectors from the top of stateList down
to the first occurrence of x is a true cycle =
;
x
x
6
while x = f (x) and x
add x to stateList
f
)
y
62 stateList do
P
y
x
;
else
in sync mode
!
=
This last algorithm is formulated as a recursive function
which parses the iteration graph of the asynchronous mode
starting from . Its parameters are the current state in the
recursion, the fixed point of the eventual convergence, a stack
stateList containing all the states taken by the system to arrive
at the current state from the initial state , and the activation
function of the network . As in Algorithm 2, this stack is
necessary to detect cycles in the iteration graph and is empty at
the first call of the function.
At each recursive call, we make several tests on the current
state . According to Definition 3.1, we have to make the distinction between a state which is in the basin of , a state which
may lead to other fixed points and a state which belongs to a
cycle. All these cases are addressed in Algorithm 3.
Algorithm 3 function testAsync(x, x , stateList,
if x = x then ==x leads to x in async mode
return True
f
)
else
if x = f (x) then = x can lead to another fixed point
than x in async mode =
return False
True
==
==
test of all evolutions
init of the parsing result
add x to stateList = push x in the stack for the
recursive parsing of the iteration sub-graph =
for all z C P (x; f (x)) x do
res AND testAsync(z , x , stateList, f )
res
2
AND
==
d(f (z ); y )
get the previous state in the cycle
(0; . . . ; 0) then
==
a true cycle has been
returning True here, allows us to continue the for
all loop above to test the other branches (at least
one) =
return True
x
62 stateList then
x
found
return False
else = there is only a pseudo-cycle in this
branch, we cannot state the final result yet, but
=
is already in stateList
cycle detection
does not lead to convergence
end if
= x
P
z
end for
=
if P
end while
if x = f (x) then
= x
leads to the fixed point
testAsync(x , x, , f )
else
if x
from x
res
x==
do
f (x )
x
(1; . . . ; 1)
== init of the boolean product
init with the last state of the cycle
for each vector z from top of stateList downto
P
Algorithm 2 function testSync(x ,
stateList
nf g
if res = False then no need to continue
Stop the for loop
end if
end for
remove x from stateList = pop x from the stack
to come back at the current parsing level =
return res == result for the sub-graph from x
else
end if
end if
end if
end if
The first condition corresponds to the terminal case of a
branch of the iteration graph which leads to . All the recursive calls of the algorithm will arrive in this case for a
state which actually belongs to the basin of . According to
subset of
Definition 3.1, this case corresponds to the
.
The second condition corresponds to the divergences since
whereas
. In this case, convergence toward a
unique fixed point is not assumed, and then cannot belong to
the basin of .
The third case is the general recursion on the iteration graph
. If the current state is not
to determine if belongs to
in stateList, it means that it has not been traversed yet and all
asynchronous iterations starting from must be tested. This is
equivalent to evaluate the subpart of the asynchronous iteration
graph whose root is . This is recursively performed for every
possible evolution of and the results of all the branches are
aggregated to get the final evaluation. It is enough for one branch
not to lead to in order to detect that is not in the basin of .
The recursion in this determination algorithm proceeds in the
set described
opposite way of the construction of the
set is built from inner
in Definition 3.1. Indeed, although the
to outer
subsets, the determination algorithm can be seen as
a parsing of the subsets from outer to inner ones. Hence, for a
given vector , if all the elements in
converge to , it means that there exists a finite such that those
elements belong to
. Then, we can deduce from
and, thus, to
.
Definition 3.1 that belongs to
Finally, the last case corresponds to the possibility that the
current state is in a cycle. If the current state is already in the
stateList stack, it means that it has already been traversed and we
have to test if the chain of states from the top of stateList down
to the first occurrence of is a true cycle or just a pseudocycle.
When a cycle is detected, the convergence cannot be guaranteed
BAHI AND CONTASSOT-VIVIER: BASINS OF ATTRACTION IN DISCRETE-TIME DISCRETE-STATE DYNAMIC NETWORKS
in a finite time and the determination process is stopped. When
a pseudocycle is detected, this branch of the iteration graph does
not pose any problem for the convergence. In fact, the final result
of the determination algorithm will be obtained by the parsing
of the other branches of the iteration graph outside the pseudocycle. Previous discussions about pseudocycles have pointed
out that such other branches exist.
That description of our determination algorithm clearly
shows its direct correspondence to the description of the
set given in Definition 3.1.
B. Complexity
The complexity of our determination algorithm is directly
linked to the complexity of the traversals of the iteration graphs
of the synchronous and asynchronous modes. It can be quite
large in the worst case since each traversal is proportional to
the number of states of the system and, thus, it is exponential
in function of the number of nodes. Nonetheless, the complete
traversal of the graph is a rare case in practice since it can only
occur in contractions, where all the states lead to a unique fixed
point. In most cases, there will be several fixed points and, thus,
several basins, each of them smaller than the whole set of states.
As our algorithms are designed to traverse a minimal number of
states necessary to perform the determination and since, in most
cases, this number is far less than the total number of states, we
. Moreover,
obtain an average complexity far less than
this also holds for the nonconverging cases since the algorithms
are designed to stop as soon as they find a branch of the iteration
graph leading to a cycle or to a divergence.
In the following, we give the complexity of the determination
algorithm for a given network whose set of states , activation
are given. Since the behavior of
function and initial state
the algorithm is not the same for the three possible kinds of results (convergence, cycle, and divergence), we give the complexity for each case.
1)
Convergence: In case of convergence toward a fixed
point , the complexity is proportional to the number
of states in the basin containing
Complexity
and more precisely, if we know that
then
Complexity
2)
Divergence: In this case, there exist at least two fixed
points
and
which can be reached in different
asynchronous executions initialized by the same . In
this context, it is sure that one of those fixed points, say
, is the result of the synchronous executions starting
from . Thus, we have
Complexity
where
denotes the length of the synto
, and
chronous iteration path going from
3)
405
is the length of an asynchronous iteration
to
.
path from
Cycle: In this last case, there are two possibilities of
detection depending on whether the cycle is in the
synchronous or in the asynchronous iteration graph. In
the first case, the detection is performed by Algorithm
2 whereas in the second case, it is performed by Algorithm 3. This implies two different complexities which
are given below.
If the cycle is in the synchronous iteration graph,
we have
Complexity
where is the first state of reached from , and
denotes the length of cycle .
If is in the asynchronous iteration graph, we obtain
Complexity
Concerning the space complexity, it is directly related to the
length of the stack stateList which contains the current path in
the iteration graph. Hence, it depends on the number of traversed
states and, thus, has a similar expression as the time complexity
for each of the previous cases. The only difference is an additional factor which is the memory needed to store one state ,
which is itself directly related to the dimension of (i.e., the
number of nodes in the system).
VII. APPLICATION
In this section, we take the same example of application as in
[4] in order to be able to compare the results of our algorithm
presented in the previous section to the one presented in that
previous paper. Our previous algorithm was based upon local
features and especially the discrete derivative of the activation
function. This previous work was based on the contraction property of the local derivative while the technical framework of the
present work is based on nested sets.
This example takes place in the context of Hopfield networks
and the reader should refer to [4] to have a detailed state of the
art on Hopfield networks and their connections to discrete-time
discrete-state fully asynchronous networks.
In the following, a Hopfield network whose value of each
, is built from a given set of vectors to
neuron is in
be memorized (see Table I) using the Hebb’s rule. Then, with
the initial vectors given in Table II (noisy patterns), the convergence of the asynchronous network is tested using the algorithm of [4] and the version given in Section VI. The last
column corresponds to the actual result of the network obtained
, whose value is
by a simulation algorithm. The fixed point
, corresponds to a spurious state
induced by the building method of the Hopfield network.
As expected, our algorithm presented in Section VI gives correct results to all tested vectors, which was not the case for our
previous version presented in [4].
In terms of complexity, the number of traversed states is given
for each tested vector in Table III.
It can be seen that the actual number of traversed states is far
. This
less than the total number of states which is
406
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 17, NO. 2, MARCH 2006
SET
OF
TABLE I
MEMORIES (DESIRED FIXED POINTS)
1 AND
1)
(
0
TABLE II
TESTS OF CONVERGENCE FOR EIGHT VECTORS (
OF THE
NETWORK
1 AND 01)
The applicative field has not been neglected since an algorithm has been deduced from this formulation, which gives the
asynchronous behavior of a given network for a given input
state. When the network converges, the algorithm identifies the
resulting fixed point. Otherwise, the algorithm determinates if
the nonconvergence of the network is due to a divergence or to
a cycle.
Finally, this determination algorithm allows us to better study
the behavior of fully asynchronous networks. It can then be
useful to study the dynamic of asynchronous neural networks
and the possible enhancements to their building. Our next goal
is to find rules which should be used to obtain more efficient and
accurate neural networks.
APPENDIX
PROOF OF THEOREM 3.2
TABLE III
NUMBER OF TRAVERSED STATES DURING THE DETERMINATION
EIGHT TESTED VECTORS
OF THE
has a direct impact on the performances of the algorithm and, in
fact, all the determinations presented here have been performed
in real time on a classical PC machine.
Finally, by allowing us to know the complete behavior of
this kind of asynchronous network, our determination algorithm
provides a very accurate tool to study the possible generic optimizations for the construction of neural networks. This objective is of essential interest since the methods used to build neural
networks are often problem dependent. We can reasonably expect that our determination algorithm will help to design a new
kind of generic construction algorithms of neural networks.
The different cases listed in the second part of Theorem 3.2
are not directly detailed in the following proof since they are, in
fact, direct deductions of all the other possible results of such
. They are
a network for any initial vector outside of
listed in Theorem 3.2 more for completeness and accuracy than
as an important part of the theoretical result itself.
Hence, the proof of Theorem 3.2 is made in two steps
,
.
A)
B)
,
and its
Case A) In this case, we consider a fixed point
, and show that this set:
corresponding set
1)
does not contain any other fixed point than ;
2)
does not contain any element of a cycle other than ;
3)
. Obviously, the
Case A.1) Let us consider
implies that
definition of Cycle
Cycle
. This prevents to be in any of
subsets for
according to Definithe
tion 3.1. Hence, the only possibility for to
is in the
subset, which
be in
implies that it is
itself; so,
is the only
.
fixed point in
Case A.2) This case is a direct deduction from Definition
3.1 and Case A.1).
. This implies that
Case A.3) Let
such that
VIII. CONCLUSION
A theoretical formulation of the basins of fixed points of
fully asynchronous discrete-time discrete-state networks has
been given. This formulation directly implies the possibility to
describe all the basins of such networks and then to completely
know their behavior. A formulation of the dynamic of such
networks has then been deduced and its correctness has been
proved. This work has also induced the characterizations of
cycles and pseudocycles in such asynchronous networks which
also represent interesting theoretical results. Moreover, a justification of the validity of those results on block-decomposed
networks has also been given.
If
,
which directly leads to
.
For
, Definition 3.1 implies that
and
As discussed in Section III, the set in Definition 3.1 cannot be empty, which ensures
BAHI AND CONTASSOT-VIVIER: BASINS OF ATTRACTION IN DISCRETE-TIME DISCRETE-STATE DYNAMIC NETWORKS
the existence of at least one vector . Moresubsets, it is
over, by construction of the
desure that there is at least one of the
scribed above which is in
. In the
opposite case, would belong to one of the
,
, instead of
. To
subsets
be simpler, let us only consider the element
, which implies, in turn, that
and
In addition, by recursion on the
we obtain
subsets,
implying, for all the vectors
,
The last line, according to Definition 3.1, leads to
, which is in contradiction with
.
Thus, we have
.
ACKNOWLEDGMENT
and
which ends with
for
obtain a sequence of vectors
, such that
. Thus, we
,
which implies
.
and
Case B) In this case, we consider a vector
we show that it is not possible to have
.
such that
Let us suppose that
and
. According to the transition
function , we have two possibilities
In the first case,
together with the hypothimply that
, which is a contraesis
diction with
. In the second case, from
, we can directly deduce that
the definition of
and then
leading to the recursion
which ends for some
407
, when
The authors would like to thank Mrs. A. Borel for her linguistic contribution.
REFERENCES
[1] F. Robert, Discrete Iterations, A Metric Study. Berlin, Germany:
Springer-Verlag Series in Computational Mathematics, 1986, vol. 6.
[2] D. Bertsekas and J. Tsitsiklis, Parallel and Distributed Computation. Englewood Cliffs, NJ: Prentice Hall, 1999.
[3] A. Herz and C. Marcus, “Distributed dynamics in neural networks,”
Phys. Rev. E, vol. 47, no. 3, pp. 2155–2161, 1993.
[4] J. Bahi and S. Contassot-Vivier, “Stability of fully asynchronous discrete-time discrete-state dynamic networks,” IEEE Trans. Neural Netw.,
vol. 13, no. 6, pp. 1353–1363, Nov. 2002.
[5] Grid Computing Info Centre [Online]. Available: http://www.gridcomputing.com/
[6] J. Bahi and C. Michel, “Convergence of discrete asynchronous iterations,” Int. J. Comput. Math., vol. 74, pp. 113–125, 2000.
[7] M. E. Tarazi, “Some convergence results for asynchronous algorithms,”
Numer. Math., vol. 39, pp. 325–340, 1982.
[8] P. Koiran, “Dynamics of discrete-time, continuous-state Hopfield networks,” Neural Comput., vol. 6, pp. 459–468, 1994.
[9] L. Wang, “On the dynamics of discrete-time, continuous-state Hopfield
neural networks,” IEEE Trans. Circuits Syst. II, Analog Digit. Signal
Process., vol. 45, no. 6, pp. 747–749, Jun. 1998.
[10] X. Wang, A. Jagota, F. Botelho, and M. Garzon, “Absence of cycles in
symmetric neural networks,” Neural Comput., vol. 10, pp. 1235–1249,
1998.
[11] D. Chazan and W. Miranker, “Chaotic relaxation,” Linear Algebra Appl.,
vol. 2, pp. 199–222, 1969.
[12] G. Baudet, “Asynchronous iterative methods for multiprocessors,” J.
Assoc. Comput. Mach., vol. 25, pp. 226–244, 1978.
[13] V. Kozyakin, A. Bhaya, and E. Kaszkurewicz, “A global asymptotic stability result for a class of totally asynchronous discrete nonlinear systems,” Math. Control, Signals Syst., vol. 12, no. 2, pp. 143–166, 1999.
[14] F. Robert, Les Systèmes Dynamiques Discrets. Berlin, Germany:
Springer-Verlag, 1995, vol. 19.
[15] D. Pellegrin, “Algorithmique Discrète et Réseaux D’automates,” Ph.D.
dissertation, Grenoble, France, 1986.
[16] E. Golès, F. Fogelman-Soulie, and D. Pellegrin, “Decreasing energy
functions as a tool for studying threshold networks,” Disc. Appl. Math.,
vol. 12, pp. 261–277, 1985.
[17] C. Marcus and R. Westervelt, “Dynamics of iterated-map neural networks,” Phys. Rev. A, vol. 40, no. 1, pp. 501–504, 1989.
[18] A. Bhaya, E. Kaszkurewicz, and V. Kozyakin, “Existence and stability of
a unique equilibrium in continuous-valued discrete-time asynchronous
Hopfield neural networks,” IEEE Trans. Neural Netw., vol. 7, no. 3, pp.
620–628, May 1996.
[19] C. Marcus and R. Westervelt, “Dynamics of analog neural networks with
time delay,” in Advances in Neural Information Processing Systems I, D.
Touretzky, Ed. San Mateo, CA: Morgan Kauffman, 1989.
[20] J. Bahi, “Boolean totally asynchronous iterations,” Int. J. Math. Algorithms, vol. 1, pp. 331–346, 2000.
408
IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 17, NO. 2, MARCH 2006
[21] J.-C. Miellou, “Algorithmes de relaxation chaotique à retard,” Revue
Française d’Automatique, Informatique, Recherche Operationnelle,
Série Analyse Numérique (RAIRO, R-1), pp. 52–82, 1975.
Jacques M. Bahi (M’03) received the Ph.D. degree
and the HDR degree in applied mathematics at the
University of Franche-Comté, Belfort, France, in
1998.
He is a Full Professor of Computer Science at the
University of Franche-Comté. His research interests
include asynchronism, distributed computing, load
balancing, and massive parallelism.
Dr. Bahi is a member of the IEEE Computer Society.
Sylvain Contassot-Vivier (M’03) received the
Ph.D. degree from École Normale Supérieure, Lyon,
France, in 1998.
He obtained a postdoctoral position at the Facultés
Universitaires Notre-Dame de la Paix (FUNDP),
Namur, Belgium, in 1998. Since 1999, he has been
an Assistant Professor at the Computer Science Laboratory (LIFC) at the University of Franche-Comté,
Belfort, France. His main research interests include
massively parallel systems, asynchronism, grid-computing, image processing, and vision.
Dr. Contassot-Vivier is a member of the IEEE Computer Society.