Proceedings o f I993 International Joint Conference on Neural Networks
Stable Dynamic Backpropagation Using Constrained
Learning Rate Algorithm
Liang Jin, Madan M. Gupta, Peter N. Nikiforuk
Intelligent Systems Research Laboratory
College of Engineering, University of Saskatchewan
Saskatoon, Saskatchewan, Canada S7N OW0
ABSTRACT
Equilibrium point learning problem in discrete-time dynamic neural network is studied in this paper
using stable dynamic propagation with constrained learning rate algorithm. The new learning scheme
provides an adaptive updating process of the synaptic weights of the network, so that the target pattern
is stored at a stable equilibrium point. The applicability of the approach presented is illustrated through
a binary pattern storage example.
1. INTRODUCTION
Dynamic back propagation (DBP) concept was first proposed by Pineda [l,21 for the supervised
weight learning problem of a class of recurrent neural networks. For equilibrium point learning problems, however, D B P algorithm provides an effective weight learning process such that the target pattern
is directly stored at an equilibrium point of the network, the stability of the network in the sense of
Lyapunov is not guaranteed at each iterative instant, and the computational requirments increase due
t o the instability. In this paper, a constrained learning rate algorithm for DBP is introduced to avoid
the instability of the network in the dynamic weight learning phase.
2. DYNAMIC NEURAL NETWORKS
2.1 Network Models
Consider a general form of dynamic recurrent neural network described by a discrete-time nonlinear
system
E@
1) = f [ x ( k ) w,
, SI
(1)
Y(k)
= h[x(k)l
where E = [q,. ,E"] E R" is the state vector of the dynamic neural network, z; represents the
internal state of the i - t h neuron, W = [ ~ ; , j ] n xisn the real-valued matrix of the synaptic connection
weights, s E R" is an input vector, y = [ y l , . . .,ymIT is an observation vector or output vector, f :
R" x Rnxnx R"
R" is a continuous and differentiable vector valued neural activation function,
f;(.) and af;/as are respectively bounded and uniformly bounded, and h(5): R" 4Rm is a known
continuous and differentiable vector valued function.
The recurrent neural network consists of both feedforward and feedback connections between the
layers and neurons forming complicated dynamics. In fact, the weight wij represents a synaptic
connection parameter between the i - t h neuron and the j - t h neuron. Hence, the nonlinear vector
valued function on the right side of system (1)may be represented as
..
{
+
-
where
W=
7
2654
i = 1, ...,n
(3)
Eq.(2) indicates that the dynamics of the i - th neuron in the network are associated with all states
of the network, the synaptic weights w ; , ~ , .. .,Wi,n and the input s;.
2.2 Gersgorin's Theorem Based Stability Conditions
The equilibrium points of the system (1) are defined by the following nonlinear algebraic equation
zf = f ( z f , W , s )
(4)
Without a loss of generality, one can assume that there exist at least one solution of the equation (3);
that is, the system (1) has at least one equilibrium point zf.
Gersgorin's theorem were often used to derive the stability conditions. For a known real or complex matrix, Gersgorin's theorem provides an effective approach for determining the position (of the
eigenvalues of the matrix. In order to analyse the stability of the equilibrium points of system (l), let
d f /dx be the Jacobian of the function f (x,W, s) with respect to z. Using Lyapunov's first method, if
then, zf is a local state equilibrium point. Furthermore, if the elements of the Jacobian df/& are
uniformly bounded, then there exist functions g;,j(wi) gi,j(w;, s;) such that
In this case, if
The equilibrium point xf is a unique global or absolute stable equilibrium point of the system (1).
3. GENERALIZED DBP LEARNING ALGORITHM
In this section, a general formulation of dynamic equilibrium point learning algorithm is developed
for the target pattern storage purpose. Let t = [tl,. ..,t,IT be an output target pattern which is
desired to be implemented by an equilibrium point zf of the neural system (1); that is, t = h(zf).
The purpose of learning procedure is to adjust the synaptic weights w;,, such that ti is be realized by
the nonlinear function h;(zf). Hence, an error function is defined by
n
C[tl- h1(zf)I2= -2l C(Jf)'
1,
2 1=1
E =-
1=1
where Jf = tl - h l ( z f ) . Next, we are going to discuss the learning formulations of the synaptic weights.
After performing a gradient descent in E, the incremental change term of the weight w;,j is given as
where q is a learning rate associated with the synaptic weights. Through a complicated derivation,
the synaptic learning formulations are obtained as
wi,j(k
+ 1)
=
wi,j(k)
af;
+ A w i , j ( k ) = wi,j(k) + qz{(k)=(k)
2655
(10)
where zf and z{ are the equilibrium points of the following dynamic system
Xi(k
+ 1)
= f;(z(k), w;,
Si)
(11)
where J ; ( k ) = t; - x ; ( h ) . Eq.(12) is said to be the adjoint equation associated with the Eq.(ll). The
updating rule (10) is not able to guarantee the stability for both systems (11) and (12), thus, when
Eqs.(ll) and (12) do not converge to the stable equilibrium points xf and z f , the iterative process
needs to be repeated throughly adjusting the learning rate q , until the solutions of Eqs,(ll) and (12)
converge to the stable equilibrium points as time k becomes large.
4. CONSTRAINED LEARNING RATE METHOD
The learning algorithm provided in last section performs a storage procedure in which the target
pattern is stored at an equilibrium point of the neural system (1). In fact, the target pattern is desired
to be stored at an stable equilibrium point of the neural system (1); that is, after finishing the learning
procedure, the synaptic weights would satisfy the stability conditions (5) or (6) so that the stored
pattern is an at least stable equilibrium point. For this goal, one may require that the systems (11)
and (12) are global stable at each learning instant, or after finishing the dynamic learning procedure,
the neural system with the learned synaptic parameters is stable in the sense of Lyapunov.
Let the system at the iterative time k satisfy the row sum stability condition; that is
n
Igi,j(wi(k))l
< 1, i = 1,2,. ..,n
(13)
j=1
We consider the stability of the system at the iterative time k t 1. Let
2
lgi,j(wi(k) t Aw;(k
t 1))l < 1,
i = 1 , 2 , . ..,n
(14)
j=1
Expending the terms on the left side of the above inequality t o second order terms of the incremental
term Aw;(k) produces
It is necessary to note that the iterative time k is omitted in above and following formulations.
Moreover, one may denote that
Then, the learning rate may be requested t o satisfy the second-order algebraic equation for a stable
learning purpose ; that is, a2,;q2t a1,;q t ao,; < 0, i = 1,2,. ..,n. Since ao,; < 0, and Q,;, all; > 0
imply U:,; - 4a2,juo,i > 0, thus, for the positive learning rate q > 0, the solution of the inequality is
obtained as
If one sets
qmin
= min{
-q;+ JU:,; - 4 a o , i ~ , i
2a2,;
Then, the stable learning rate q is obtained as 0 < q
2656
< qmin.
1
5. EXPERIMENTAL RESULTS
In this section, a binary image pattern storage process will be discussed to illustrate the applicability
of the algorithm, where the target pattern is 10 x 10 binary image as shown in Figure la. The equations
of the dynamic network have the following simple form
where the neural activation function a(.)was chosen as a(.)
were easily derived as
n
= tanh(u). The global stability conditions
n
All the iterative initial values of the weights were set t o zeros. Figure 1 shows the equilibrium point
patterns of the network in the iterative process. After 400 steps of the learning, the binary pattern as
shown in Figure l a was stored perfectly at the stable equilibrium point of the network as depicted in
Figure lj.
(f
Figure 1. The binary patterns correspond t o the equilibrium point of the network duriing the
learning process. (a). The target pattern; (b). k=O; (c). k=50; (d). k=100; (e). k=150; (f). k;=200;
(g). k=250; (h). k=300; (i). k=350; (j). k=400.
6. CONCLUSIONS
We have developed a new stable dynamic backpropagation learning algorithm which provides an
associative memory model for real valued vectors using discrete-time dynamic neural model. I n such
a dynamic learning phase, the stability of the network is ensured by a suitable learning rate and,
therefore, the standard DBP routines can be used.
References
[l] F. J, Pineda,“Generalisation of back-propagation to recurrent neural networks,” Pbycical Review Letters, ‘Vol. 59,
No. 19, pp. 2229-2232, 1987.
[2] F.J. Pineda, “Dynamics and architecture for neural computation,” J. of Complexity, Vol. 4, pp. 216-245, 1!)88.
[3] Y. Chauvin, “Dynamic behavior of constrained back-propagation networks,” in Advances in Neural Information
Processing System, D.A. Touretzky, Ed., San Mateo, CA: Morgan-Kaufmann, Vol. 2, pp. 519-526, 1990.
2657