Information processing in synchronous neural networks
J.F. Fontanari, R. Köberle
To cite this version:
J.F. Fontanari, R. Köberle. Information processing in synchronous neural networks. Journal
de Physique, 1988, 49 (1), pp.13-23. <10.1051/jphys:0198800490101300>. <jpa-00210669>
HAL Id: jpa-00210669
https://hal.archives-ouvertes.fr/jpa-00210669
Submitted on 1 Jan 1988
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
J.
Phys.
France 49
(1988)
Classification
Physics Abstracts
87.30G
64.60C
-
-
Information
13-23
75.10H
-
JANVIER
1988,
13
89.70
processing
in
synchronous
neural networks
J. F. Fontanari and R. Köberle
Instituto de Fisica
SP, Brasil
(Requ
le 16
e
Química de São Carlos, Universidade de São Paulo, Caixa Postal 369, 13560 São Carlos,
juin 1987, accepté le
16
septembre 1987)
Résumé.
Nous obtenons le diagramme de phase du modèle de Little quand le nombre p d’échantillons
mémorisés croit comme 03C1
03B1N, où N est le nombre de neurones. Nous dédoublons l’espace de phase de
façon à accommoder des cycles de longueur deux dans le cadre de la mécanique statistique. Utilisant la
méthode des répliques, nous déterminons le diagramme de phase incluant un paramètre J0 pour contrôler
l’apparition des cycles. La transition de phase entre les phases para- et ferromagnétiques passe du second ordre
au premier ordre au point tricritique. La région de recouvrement de l’information est un peu plus grande que
dans le modèle de Hopfield. Nous trouvons également une phase paramagnétique à basse température qui a
des propriétés physiquement inacceptables.
2014
=
The phase diagram of Little’s model is determined when the number of stored patterns p grows as
03B1N, where N is the number of neurons. We duplicate phase space in order to accomodate cycles of length
within the framework of equilibrium statistical mechanics. Using the replica symmetry scheme we
Abstract.
03C1
=
two
2014
determine the phase diagram including a parameter J0 able to control the occurrence of cycles. The second
order transition between the paramagnetic and ferromagnetic phase becomes first order at a tricritical point.
The retrieval region is some what larger than in Hopfield’s model. We also find a low temperature
paramagnetic phase with unphysical properties.
Recently methods developed in the study of equilibrium statistical mechanics of spin glasses [1, 2, 3, 4],
have been applied to investigate information processing and retrieval in neural networks. Although
there is a long way to go, if reasonably realistic
biological systems are to be described, the models
we are able to control, do exhibit a number of
interesting features, which makes their study a
worthwhile endeavour. Properties such as fault tolerance to errors, information storage and retrieval due
to implementation of auto-associative memories,
etc. have been shown to arise as a consequence of
the existence of an infinite number of ground states
in a spin glass interacting via long ranged forces.
study Little’s model [5 based on
synchronous update in the limit, when an infinite
In this paper
symmetric couplings has limit cycles oft length
points.
The question arises then as to how can equilibrium
statistical mechanics describe a system with cycles
[2]. As we will see, since in our case the cycle’s
length is two, we have to duplicate the phase space
[7]. The concomitant proliferation of order parameters allows the adequate description of this more
complicated behaviour.
with
1. Introduction.
we
number of patterns is to be stored
In contrast to
assynchronous dynamics (such as Monte Carlo update in Hopfield’s model), when all states change
simultaneously in parallel, the system may exhibit
cycles. It is indeed easy to show that Little’s model
[6].
two in addition to fixed
In section 2 we calculate the free energy and
derive the equations for the order parameters, which
are discussed and solved for special cases in section 3. A discussion is presented in section 4.
2. Free energy and order parameters.
Little’s model consists of a network of N neurons
N. The transidescribed by spins S, = ± 1, i
1,
tion probability W (J/I ) from state I
{5,} at time
t to state J at time t + 1 is given by
Article published online by EDP Sciences and available at http://dx.doi.org/10.1051/jphys:0198800490101300
=
...,
=
14
where
For
symmetric couplings Jij Jji, this leads
stationary distribution of states given by e- H,
=
to a
with
Here f3 is a measure of the noise level in the
system and the couplings are
where aj = ± 1, j
N are a set of duplicate
1,
variables.
Ising
Following Amit et al. [3] we now compute the
quenched free energy under the assumption that a
=
finite number s of the
...,
overlaps n
"
remains finite
quenched
free energy per
spin
is
as
N --+
oo.
The
given by the replica
method :
The p patterns (prototypes) {ç r ; i = 1,
N}
=1, ..., p are quenched, independent random
variables taking the values ± 1. Due to the coupling
Jij these patterns are fixed points of the dynamics at
T=0, poo.
...,
J.L
In order to evaluate the trace Tr e- OR
W
we use
where
the
Introducing
identity
..., n we
the
replicated
write (Z")
variables
sf, (T f,
p
=
1,
as
where
with Z°
Let
If
=
us
we
Z v for cp * s and Z°
Z w for cp &#x3E; s.
first evaluate Z’. Performing the average
=
expand
Inserting
this
the
log cosh, keeping
expression
over
terms of order
into Z’ and
performing
the
tiN
the
g¡,
we
integral
we
obtain the factor
get
over
m:, n: and f p 0
we
obtain
finally
15
where
P and R
are
symmetric n
where p pu and r pa
are
defined
Q,
equation (2.12).
Now we proceed to
thermodynamic limit as
x n
only
matrices
given by
for p =1= (T and
evaluate Z v3. First
we
--AL-
have omitted trivial
multiplicative
rescale m, n and f in order to obtain
a
constants in
well defined
Thus
Now
to
use
self-averaging
in the
identity
replace Ni
1 E byand
(2.12)
for Z’
we
obtain
use
the result in
equation (2.15).
With this
expression
for Z v and
equation
finally
where
with the notation
m.
In the limit N - oo the integrand is dominated by
its saddle point, furnishing the following free energy
per spin
with
In this paper we mainly discuss the replica symmetric theory, in which we use the following parametrization
16
With this ansatz the n - 0 limit of the terms in
equation (2.19) may be explicitly performed. We get
[8]
Now we linearize the quadratic terms in the spin
variables by Gaussian transformations, to obtain
after insertion into equation (2.23)
where
where
Finally
Using
we
have to compute
the ansatz
(2.21),
we
rearrange
HI;
into
stands for the average over the 6,’s and
three Gaussian variables Zi, i
1, 2, 3 with
mean zero and unit variance.
Putting everything together we get for the free
energy the final expression
and
over
=
Minimizing f with respect to the parameters f, m, n, p, p, r, r, ql, ql, qo, 40 yields their interpretation in
original variables Si, (T and );’ together with the order parameter equations. We thus obtain
within the replica symmetric theory :
i) the macroscopic overlaps between equilibrium states and prototypes :
terms of the
ii) the Edwards-Anderson
iii)
the total
mean
order parameters
square random
overlaps
with p-s patterns
17
where &#x3E;
The 8
s.
+
3s
coupled equations
for the order parameters
where
are :
Since qo =1
dynamics,
as
are at a fixed point of the
opposed to a limit cycle.
we
In order to evaluate the
following identities
3. Solution of order parameter
P -+
oo
limit
we use
the
equations and phases.
In general it is impossible to solve explicitly the
coupled equations for the order parameters, but in
special limiting cases this can be done. In the
following we will do this in order to obtain relevant
sections of the phase diagram and discuss the nature
of possible solutions.
T = 0.
The sign of Q plays an
important role, if we want to take the limit
J3 --+ oo. Accordingly we distinguish three cases.
3.1 PHASE
for
AT
-
It is
now
straightforward
to
get
equations (2.28) :
These are the same equations as obtained by Amit
et al. [3], for the T = 0 limit of Hopfield’s model.
This is to be expected, since at a fixed point
equation (2.2) may be rewritten at T 0 as
=
where
and
The free energy at T
=
0 becomes
18
Which differs from Hopfield’s model only
Jo and Amit et al.’s analysis for this case
taken over with a slight change in notation.
term
Q
3.1.2
0.
The
-
equations
become
by
the
be
p
=
0, implying p
=
0,
c --+ oo
with energy
can
now :
which is always less than the spin-glass energy (3.9).
Whether fixed point or cycles are realized depends
now on the value of Jo, selecting the lower of two
energies equation (3.5)
versus
(3.10).
case Q = 0 gives identical results to
The resulting phase diagram is shown in
1, where all transitions are first order.
3.1.3 The
Q &#x3E; 0.
figure
-
where
and
Since qo = - 1, we are at a complete cycle, all spins
changing sign at each update. Evaluating the limits
in equations (3.6) we get
The
only
solution of
equation (3.7)
for m ° is
m’ = 0, yielding
Phase diagram in the a - Jo plane. abounds
the region where retrieval states appear, which become
stable inside the F region. a g separates the p and SG
phases. At Jo 0, we get a, - 0.138 and a F == 0.051 as in
Hopfield’s model.
Fig.
with the energy
being given by
Since c is positive semidefinite, this spin glass
exists
solution
only for
a &#x3E; 2/ 1T and
1T
the
last
7() (a /2 )1/1,
inequality resulting from
Q0.
If
c
we
oo, we
relax the implicit assumption (3.8) of
also encounter a paramagnetic phase with
with the order parameter
from which
we
easily
equations
obtain
1.
-
=
3.2 PHASES
FOR a
equation (2.26) yields
=
0.
-
The a --+ 0 limit of
19
Since the lefthand side is &#x3E; 0, whereas the right hand
side is
0 we see that the only solution is
In the following we will
the Mattis type only :
n now
study retrieval solutions of
satisfies
Expanding this equation in powers of n,
for the n # 0 solution
we
obtain
vv
Fig. 2. - a = 0 plot of phase diagram. TCP indicates the
tricritical end-point at T 2/3, Jo = - 1/3 log 2, marking
change from a first order (continuous line) to a second
order transition.
where
=
n2 vanishes
by
as
long
at the critical
temperature Tc defined
3.3 SYMMETRIC AND ANTISYMMETRIC SOLUTIONS.
In the cases studied up to now we only found the
following types of solutions :
i) symmetric
as
This 2nd order transition between a ferromagnetic
and a paramagnetic phase changes to first order,
when condition (3.19) ceases to be verified. This
happens at the tricritical point
where a new solution of
free energy appears.
equation (3.16)
with lower
solutions :
ii) antisymmetric
solutions :
We believe these to be the only ones, because
solutions with n v =1= e v or p =1= r would correspond to
breaking spontaneously the symmetry S - a and
since we have failed to encounter this at T 0, we
conclude that it should not occurr at all.
=
The first order phase boundary between the P and
F phases may be obtained numerically by equating
the free energies of the two phases. The result is
shown in figure 2, where the point T 0, Jo
0.5
at which the first order line touches the T = 0 axis
can easily be obtained analytically. We also show the
curve labeled T, limiting metastable F solutions.
=
In order to obtain information about
compute the parameter qo
=
-
cycles
As for the antisymmetric solutions, they only exist
0. This can easily be seen using the relation
= r
ql/2 in the expression for E 1,
at T
p
=
=
-
equations (2.28)
we
Which would yield complex order parameters.
This doesn’t happen at T 0, because the offending
terms vanish in this limit. Thus for T&#x3E; 0 only
symmetric solutions exist and the order parameter
equations for this case follow from equations (2.28).
Restricting ourselves to retrieval states they are
=
In the P
phase
yielding cycles
this
for
gives
Jo
0.
20
b ) Transition spin-glass
to
paramagnetic.
0, all other parameters are finite.
equation for the 2nd order transition surface
Tg (a, Jo) between P and SG phase is obtained
expanding equations (3.25b), (3.25c) in powers of
Only
n
=
The
qu
which
go the
yields together with the equations for qo
following behaviour for Tg ( ex , J 0) for a
and Jo = 0:
Since
and
1
know that at T 0 this transition is first
that there should exist a tricritical
conclude
order,
line (TCL) on the surface (3.31) separating these
two different critical behaviours. The TCL may be
obtained numerically imposing the existence of two
different solutions of equations (3.25a), (3.25c),
0.
which coincide at the TCL, both having ql
3
of
shows
the
the
TCL
onto
the
Figure
projections
planes T 0 and Jo 0. In agreement with section 3B, the line reaches the a
0 plane at the point
T
2/3 and Jo = - 1/3 log 2. We furthermore observe that it exists only in the half-space Jo
0.
The first-order transition SG-P is obtained equating the free energies of these two phases.
we
=
we
where
=
=
=
=
=
and
From these equations
and transitions :
we
obtain the
following
phases
a) Paramagnetic phase.
Here all order parameters
which satisfy
vanish, except qo and
ilo,
For
high temperatures
this
gives
Whereas for low temperatures
equation (3.27a ) that qf -+ 1 as emay write
we
/ T,
from
that we
see
so
Projections of the tricritical line (TCL) onto the
0 (continuous line).
T
0 (broken line) and Jo
The broken line exists only in the region Jo 0.
Fig. 3.
planes
c)
implying
=
=
Retrieval
phase.
we obtain the
surface T R ( a , 10) below which retrieval states become metastable. The results is shown in figure 4.
For Jo
0 we obtain a phase diagram very similar
the one of Hopfield’s model, except that for
a &#x3E; 3.74 we find a paramagnetic phase with qo
0,
Solving numerically equations (3.25)
this last result agreeing with our discussion about the
0 in the symmetric
occurrence of cycles at T
=
phase.
-
=
21
therefore associate with the existence of
in
model. In the traditional highLittle’s
cycles
P
temperature phase we have qo &#x3E; 0.
which
we
Proceeding
as
in reference
[3],
we
obtain
The behaviour of TF (a, 0) near T = 1 is obtained
equating the free energies of the F and SG phases,
yielding
This
together
with
equation (3.34) gives
where we have included the results for Hopfield’s
model in parentheses for comparison. Thus the F
and retrieval regions are slightly larger in Little’s
model.
e)
The limit
In this limit
4.
Fig.
-
Jo = 0 plot of phase diagram. Tf is the
transition
temperature between F and SG phases. The continuous
lines are first order transitions, while broken ones are
second order. The inset shows the appearence of the P
at T
0 and large values of a.
phase
Jo we
00.
obtain from
equation (3.25)
with
=
The equations reduce to the corresponding ones
of Hopfield’s model provided we rescale T Little
2 7"Hopfic)d’ This result is reasonable considering that
large values of 10 suppress cycles and tend to align
0’i with Si. We also see that the synchronous
dynamics is much more stable to noise than the
asynchronous one for the same retrieval capacity,
provided that neurons have a sufficiently large selfinteraction 1().
=
d)
The
phase-diagram around
T
=
1
for Jo
=
0.
The aim of this section is to obtain the behaviour of
the curves TF ( a ), limiting the stable and metastable
retrieval regions, in the vicinity of the point
T
1 and a
0, where we have discontinuous first
order transitions but with small order parameters, so
that the equation (3.25) may be expanded in powers
of n and t -1 - T. We obtain
=
=
4.
Replica symmetry breaking.
It is well known, that replica symmetry breaking
(RSB) is necessary to stabilize the spin-glass phase at
low temperatures. The effects of RSB in the F and
SG phase are similar to the ones in Hopfield’s model
and thus small. For example, the entropy at
T = 0, J" = 0 in the FM phase is S = -1.4 x
10- , at a = a, 0.138 and vanishing exponentially
=
.
From these
with
we
get
In the SG
phase
at
The analog of the Almeida-Thouless line can be
computed as in reference [3]. The sign of the
replicon mode changes when
22
where
how be discarded. This point is being reserved for a
future study.
Although this P phase is unimportant for retrieval
purposes, for which the replica scheme seems to
provide results in agreement with simulations, we do
not see how these diseases may be cured within the
replica scheme. We have studied this phase by
simulations at T 0. In figure 5 we show the result
of measuring the fraction q
n/N, where n is the
number of spins belonging to cycles, against N. We
interpret these results saying that qo (= 1 - 2 q )
never reaches the value qo = -1, as predicted by
equation (3.6), but rather that qn - + 1 with only a
finite number of spins oscillating. This would exclude
=
=
where
we
tested for instabilities
only
in directions
q a¡3’ ij a¡3 with a =1= f3 .
The entropy of the low temperature P phase is
also negative, going to - oo as T -+ 0 exactly as in
the Sherrington-Kirkpatrick (SK) model [9]. Since
RSB has nothing to say about this unphysical
situation, we leave this matter for a discussion in the
next section.
T
=
0, Jo ± 0 solutions with
Q
thus
stigmatize this P phase as
replica symmetric calculation.
0 of section 3 and
an
artefact of the
5. Discussion.
In this paper we have analised the phase diagram
and storage properties of a synchronous model for
associative memory. Our results are qualitatively
similar to the ones of
model, albeit with
the following differences :
Hopfield’s
a)
there is
a new
parameter Jo, which may be used
to control the occurrence of
cycles
b) a tricritical line appears at the surface separating the P and F phases
c) for or 0 we find a P phase even at T 0,
which is not connected to the usual high temperature
P phase. This phase has negative entropy, whose
=
value goes to -
oo
at T
=
0
as
in the SK model.
RSB is expected to cure the unphysical aspects of
the SG and F phases, but cannot exorcise problems
in the low temperature P phase, since only qo and
go are nonzero here. Breaking the symmetry along
the diagonal in q a¡3 and ij a¡3 is apparently useless,
since we found this phase to be stable in the
directions qa a and 4,,aLet us note that this P phase is our only option in
the region Jo -1, where a solution with qo 0 is
to be expected. The SG phase with qo 0 doesn’t
exist for
a
2
or
as can
be
seen
from
equation (3.8)
and comments there after. RSB may extend the size
of the SG solution, so that the P phase may some
Fig. 5.
Histograms of the fraction q of spins belonging
to cycles for N
100, 200 and 300. The parameters
«
0.5 and Jo
0 are kept fixed and we average over
-
=
=
=
initial states and realizations of the
( );’) .
Acknowledgments.
The research of R.K. is partially supported by CNPq
and J.F.F. holds a FAPESP fellowship. Simulations
were carried out on a VAX 780/11 computer,
partially maintained by funds from CNPq.
23
References
J. J., Proc. Natl. Acad. Sci. USA 79 (1982)
2554 ; ibid 81 (1984) 3088.
[2] PERETTO, P., Biol. Cybern. 50 (1984) 51.
[3] AMIT, D. J., GUTFREUND, H. and SOMPOLINSKY, H.,
Ann. Phvs. 173 (1987).
[1] HOPFIELD,
[4]
[5]
[6]
Parallel Models of Associative Memory, Eds G. E.
Hinton and J. A. Anderson (Lawrence Erebaum
Ass.) 1984.
LITTLE, W. A., Math. Biosci. 19 (1974) 101.
FONTANARI, J. F. and KÖBERLE, R., Phys. Rev. A 36
(1987) 2475.
[7]
This
which arises very naturally, when
solves this model, has also been used by the
authors of reference [3] and J. L. van Hemmen,
Phys. Rev. A 34 (1986) 3435. If the cycle length
is l, we have to introduce l copies of phase
duplication,
one
space.
[8]
See
appendix
A of reference
[3]
for
a
similar cal-
culation.
[9] KIRKPATRICK, S.
B 17 (1978)
and SHERRINGTON, D.,
4384.
Phys.
Rev.