Neural Processing Letters (2005) 21: 1–20
© Springer 2005
A Learning Rule to Model the Development of
Orientation Selectivity in Visual Cortex
J. M. JEREZ1 , M. ATENCIA2 and F. J. VICO1
1
Escuela Técnica Superior de Ingenierı́a en Informática, Departamento de Lenguajes y
Ciencias de la Computación, Universidad de Málaga
2
Escuela Técnica Superior de Ingenierı́a en Informática, Departamento de Matemática
Aplicada, Universidad de Málaga
Abstract. This paper presents a learning rule, CBA, to develop oriented receptive fields similar to those founded in cat striate cortex. The inherent complexity of the development of
selectivity in visual cortex has led most authors to test their models by using a restricted
input environment. Only recently, some learning rules (the PCA and the BCM rules) have
been studied in a realistic visual environment. For these rules, which are based upon Hebbian learning, single neuron models have been proposed in order to get a better understanding of their properties and dynamics. These models suffered from unbounded growing of
synaptic strength, which is remedied by a normalization process. However, normalization
seems biologically implausible, given the non-local nature of this process. A detailed stability analysis of the proposed rule proves that the CBA attains a stable state without any need
for normalization. Also, a comparison among the results achieved in different types of visual
environments by the PCA, the BCM and the CBA rules is provided. The final results show
that the CBA rule is appropriate for studying the biological process of receptive field formation and its application in image processing and artificial vision tasks.
Key words. Hebbian learning, orientation selectivity, receptive fields, unsupervised learning,
visual cortex modelling
1.
Introduction
Among the different approaches to imitate the perceptual capabilities of biological
systems, neural-based models have been proposed in the last decades [6, 13, 15, 19,
23], and some have been tested in natural scenarios [11, 20]. Stimulating a single neuron model with natural images, PCA [18] and BCM [6] learning rules were
shown to develop receptive fields (RFs) similar to those found in visual cortex in
the early experiments of Hubel and Wiesel [8, 9]. Approaches based on the ICA
rule [5] were also successfully used with natural scene inputs to produce oriented
receptive fields, although most of these methods for performing ICA [2, 4, 10, 24]
are not single neuron models but multi-neuron algorithms, and therefore do not
fit well into the framework of this study.
Exposing a single neuron model (using PCA and BCM learning rules) to
some stimulation trials transformed a random receptive field in one selective to
2
JEREZ ET AL.
orientation. Each trial included the presentation of a patch of size 13 × 13 pixels,
obtained from a set of 24 grey-scale 256 × 256 pixels images, which were processed
with a difference of gaussian (DOG) filter. Preferred orientations of the resulting
RFs spread out widely and concentrated slightly in the range from 80◦ to 120◦
since the images contained vegetal forms, which aligned more in the vertical orientation.
The resulting RFs contained excitatory and inhibitory regions arranged in a preferred orientation. The emergence of these regions have to do with the potentiating (LTP) and depressing (LTD) characters of the learning rule. According to the
Hebbian postulate, both rules include LTP terms, but differ in the way they implement LTD. While PCA incorporates heterosynaptic competition, BCM produces a
similar effect through homosynaptic competition. These two forms of LTD reinforce inhibition by means of spatial competition among the afferents of a neuron
in the case of PCA, or temporal competition in the case of BCM. The fact that
each of these learning rules relies on a single mechanism for LTD strongly influences the final shape of the RFs, and, consequently, the type of processing performed by the neuron. The RFs resulting from a PCA training are sensitive to low
spatial frequencies (only two regions are differentiated), while those obtained with
BCM show selectivity to high frequencies (three or more bands). Both homosynaptic and heterosynaptic competitions have been described in the nervous system
[1, 3, 14], and its combined effect might yield the wide range of spatial frequencies
that are captured by the RFs of the striate cortex cells [16]. Although, in principle, the BCM learning rule seems to be more suitable to achieve sensitivity to both
low and high frequencies with a proper parameter set, the temporal competition
that implements its LTD mechanism makes hard the fitting process. This problem
arises when the BCM theory is tested using the images from a camera mounted
on a freely moving robot [17].
Taking into account these functional limitations and biological constraints, we
propose here a new learning rule (CBA rule) that incorporates homosynaptic and
heterosynaptic competition. This rule is derived from the one proposed in Ref.
[21] for neural assemblies formation, with the only difference that it incorporates
a decay term.
The rest of this paper is organized as follows. In Section 2, a thorough mathematical analysis of the proposed rule is undertaken. Based upon standard results
of dynamical systems, we conclude that the model possesses a non-zero stable fixed
point. Besides, the basin of attraction of this fixed point is exactly determined. The
interpretation of these results suggests that the CBA rule models the learning process for a wide range of initial weights and parameters. In Section 3, first, the
rule is simulated within a restricted visual environment. Then, realistic images are
presented for the model. Both experiments suggest that receptive fields are formed,
which mix properties of the BCM and the PCA rules. Finally, Section 4 summarizes the main conclusions, and some lines for future research are provided.
DEVELOPMENT OF ORIENTATION SELECTIVITY IN VISUAL CORTEX
2.
3
Dynamical Analysis of the Proposed Model
In this section, the CBA rule is formulated by means of a system of ordinary
differential equations (ODEs) whose variables are the synaptic strengths. Then,
with the aid of standard tools in the qualitative theory of ODEs, the fixed points
of this equation are calculated. Besides, a linearization analysis proves that there
exists a non-trivial fixed point that is asymptotically stable. Furthermore, the basin
of attraction of the stable fixed points is calculated by constructing a Lyapunov
function. Therefore, this construction shows that the CBA rule cannot exhibit chaotic wandering, cycles or any other misbehaviour that can be interpreted as lack of
learning.
First, the activation equation is formulated. Although a strict description of the
neural dynamics requires considering sequences of spikes, a great simplification
is achieved by choosing as variables the averages of the incoming sequence over
appropriate time scales. Thus, the neuron model consists of a vector x of inputs,
representing an averaged presynaptic activity originated from another cell, a vector
w of synaptic weights, and a scalar output y, given by y = w · x, which represents an
averaged postsynaptic activity. The weight vector w can take negative values, since
they can be considered as effective synapses made up of multiple excitatory and
inhibitory connections.
2.1. synaptic modification equation
Once the activation equation is defined, we face the problem of modelling the
learning process. In a previous work [21] we proposed a new correlational learning rule (BA, for bounded activity) that formed stable neural attractors in a recurrent network. The CBA learning rule is essentially a modification of the BA rule
in which an extra term to implement the heterosynaptic LTD has been incorporated. Thus, the CBA synaptic modification equation is a Hebbian-type learning
rule with a specific form of stabilization, defined as
dwi
= αxi y(y − τ )(λ − y) − βywi
dt
(1)
where α is the learning rate; the term λ avoids the unbounded weight growing in a
biologically plausible way, and can be interpreted as a neuronal parameter representing the maximum level of the neuron activity; the parameter τ defines a threshold that determines whether depression or potentiation occurs depending on both
the pre- and postsynaptic neuron’ activities; and, finally, the parameter β controls
the heterosynaptic competition effect. The β value should be lower than α to preserve the dominant character of LTP and homosynaptic LTD over heterosynaptic LTD. With this formulation, the stability is achieved by decreasing LTP when
postsynaptic activity approaches extreme values, while a negligible LTP occurs for
low postsynaptic activity. The effect of this adaptation mechanism in the receptive
fields formation process is that the gradual response after stimulus presentation
4
JEREZ ET AL.
leads the neural activity either to high or resting levels. In the next section, these
intended properties are confirmed by rigorous analysis.
2.2.
fixed points and stability conditions
In this section, an analytical study of fixed points and stability conditions for the
CBA learning rule is presented, which will provide a better understanding of its
fundamental properties. The convergence of the dynamical system to a fixed point,
where additional training does not change the weight vector, represents the success
of the learning process. In this section, a constant input vector (x) is considered to
stimulate the neuron during the whole training process. This simplification is necessary to do the analytical study, and x would represent a single input vector, or
a prototype of the input data that the neuron is selective to. This simplified input
structure shows the main qualitative features of the learning process. In Section 3,
it will become apparent that the behavior of the rule for a general visual environment is qualitatively similar to the constant input case.
Since the model is intended to accurately represent the learning process, it must
possess some desirable properties, which have their mathematical counterparts in
characteristics of the corresponding dynamical system:
• The learning process must terminate, so at least one fixed point of the dynamical system must exist.
• The learning process must evolve towards a fixed point. This corresponds to
the asymptotic stability of the fixed point.
• The set of initial weights converging to a fixed point, i.e., its region of asymptotic stability, should be as large as possible.
• For every initial weight, the learning process should approach a fixed point,
so there should not exist periodic or chaotic trajectories.
• Zero output is biologically identified as lack of selectivity, so the weight vector that results from learning must produce a non-null output y = 0, otherwise
it is considered a trivial fixed point.
We will prove that our model behaves as expected, provided that some relations
among the parameters of the model hold. The fixed points will be calculated, their
stability will be elucidated and the regions of asymptotic stability of each stable
fixed point will be determined.
Consider a single neuron model with n afferent presynaptic connections. This
general model would be modelled by the following system of differential equations:
dwi
= −βywi + αyxi (λ − y)(y − τ )=fi (w),
dt
n
y=
w i xi = w · x
i=1
i = 1, . . . , n
(2)
DEVELOPMENT OF ORIENTATION SELECTIVITY IN VISUAL CORTEX
5
where the input x = (x1 , . . . , xn ) and the weight w = (w1 , . . . , wn ) are both ndimensional vectors.
The learning process ends when the weight vector does not change further. Thus,
the fixed points of the system will be calculated by solving the system of equations fi (w) = 0 for i = 1, . . . , n. In this way, any vector w0 such that y = w0 · x = 0
is a fixed point, constituting in fact an invariant subspace: the hyperplane which is
orthogonal to the input.
If y = 0, two non-trivial fixed points are obtained, resulting in
βwi = αxi (λ − y)(y − τ )
(3)
which shows that the weight vector w is proportional to the input x, with some
constant of proportionality, which we call k. Then, wi = kxi and a fixed point w
satisfies
(−βk + α(λ − y)(y − τ ))x = 0
(4)
so that k can be obtained by solving the second-degree equation
−βk + α(λ − y)(y − τ ) = 0
(5)
which, taking into account y = w · x = kx2 , yields:
−αx4 k 2 + −β + α(λ + τ )x2 k − αλτ = 0
(6)
Now define the constants γ , ρ and R, in order to simplify notation:
√
γ = −β + x2 α(λ + τ ); ρ = 2x2 α λτ ; R = γ 2 − ρ 2
(7)
and the constant k can be expressed as
k=
γ ±R
2αx4
(8)
Consequently, the two fixed points are parallel to the input:
w1 =
1 γ +R
x;
x4 2α
w2 =
1 γ −R
x
x4 2α
(9)
The fixed points w1 and w2 only exist if the root R is real, which reduces to
γ 2 − ρ 2 0. This condition holds if either
√
√
√ 2
λ− τ
(10)
γ ρ > 0 ⇐⇒ β x2 α(λ + τ ) − 2x2 α λτ = x2 α
or
γ −ρ <0
⇐⇒
√
√
√ 2
λ+ τ
β x2 α(λ + τ ) + 2x2 α λτ = x2 α
(11)
6
JEREZ ET AL.
In the sequel, we will restrict ourselves to the region given by condition (10).
Condition (11) violates the assumption done about the relation in magnitude
between α and β(β α).
To elucidate the stability of a fixed point, we will calculate the jacobian of the
system, evaluate it in the fixed point, and observe the sign of its eigenvalues. As
the system, obviously, fulfills the conditions for this linearization to be valid, a
fixed point will be asymptotically stable if every eigenvalue of the jacobian is negative, and it will be unstable if there exists at least one positive eigenvalue. The
expansion of the functions fi (w) yields
fi (w) = −αxi y 3 − βywi + αxi (λ + τ )y 2 − αxi λτy
(12)
and their partial derivative, taking into account ∂y/∂wj = xj , results in
∂fi
= −3αxi xj y 2 − βxj wi − δij βy + 2αxi xj (λ + τ )y − αxi xj λτ
∂wj
(13)
where δij is Kronecker’s delta:
δij =
1
0
if i = j
if i =
j
(14)
A considerable simplification may be obtained with matricial notation:
J(w) = −3αy 2 + 2α(λ + τ )y − αλτ xx − βwx − βyI
(15)
Now the jacobian is evaluated for each fixed point:
• Trivial fixed points. For any vector w0 that is orthogonal to the input x, the
output vanishes (y = 0); thus, the jacobian is
J(w0 ) = −αλτ xx − βw0 x
(16)
This matrix has two eigenvalues (see Lemma A.1): t1 = 0 and t2 = −αλτ x2 .
Although all eigenvalues are non-negative, the existence of one null eigenvalue
does not allow to confirm the asymptotic stability of the nonlinear system.
Next, this stability result, which is intuitively suggested by simulations, will be
rigourously proved by means of the Lyapunov function technique.
• First fixed point. Consider now the first fixed point w1 = x(γ + R)/(2αx4 ).
The corresponding output is
y = x · w1 = x w1 =
γ +R
2αx2
(17)
DEVELOPMENT OF ORIENTATION SELECTIVITY IN VISUAL CORTEX
and the substitution of these values into the jacobian (15) yields
3
λ+τ
J(w1 ) = −
(γ + R)2 +
(γ + R) − αλτ
4αx4
x2
β
β
−
(γ + R) xx −
(γ + R)I
2αx4
2αx2
7
(18)
This matrix has two eigenvalues (see Lemma A.2):
t1 = −
β
(γ + R) < 0
2αx2
(19)
and
t2 = −
3
λ+τ
(γ + R)2 +
(γ + R) − αλτ
4
4αx
x2
β
β
−
(γ + R) x2 −
(γ + R)
2αx4
2αx2
3
−β + x2 α(λ + τ )
2
=−
(γ
+
R)
+
(γ + R) − αλτ x2
4αx2
αx2
3
1
=−
(γ + R)2 +
γ (γ + R) − αλτ x2
(20)
2
4αx
αx2
√
where the previously defined parameter ρ = 2x2 α λτ was used. Taking into
account the definition of R, we have R 2 = γ 2 − ρ 2 or, equivalently, ρ 2 = γ 2 −
R 2 = (γ − R)(γ + R). Thus,
t2 = −
γ +R R
<0
αx2 2
(21)
which proves that the first fixed point w1 is stable, because both eigenvalues
are negative.
• Second fixed point. The substitution of the second fixed point w2 = x(γ −
R)/(2αx4 ) and the corresponding output
y=
γ −R
2αx2
into the jacobian (15) yields
3
λ+τ
(γ − R)2 +
(γ − R) − αλτ
J(w2 ) = −
4
4αx
x2
β
β
−
(γ − R) xx −
(γ − R)I
2αx4
2αx2
(22)
(23)
This matrix has two eigenvalues (see Lemma A.2):
t1 = −
β
(γ − R) < 0
2αx2
(24)
8
JEREZ ET AL.
and
t2 = −
3
λ+τ
(γ − R)2 +
(γ − R) − αλτ
4
4αx
x2
β
β
−
(γ − R) x2 −
(γ − R)
4
2αx
2αx2
3
1
=−
(γ − R)2 +
γ (γ − R) − αλτ x2
4αx2
αx2
(25)
since R 2 = γ 2 − ρ 2 implies R > γ , hence γ − R > 0. Thus,
t2 =
γ −R R
>0
αx2 2
(26)
which proves that the second fixed point w2 is unstable inside the region given
by condition (10). In fact, it is a saddle point, since there is one negative
eigenvalue and one positive eigenvalue.
To summarize, we have proved that w1 = (γ + R)/2x3 α is the only non-trivial
asymptotically√stable fixed point, provided that γ > ρ, where γ = −β + x2 α(λ +
√
τ ), ρ = 2x|2 α λτ and R = (γ − ρ)(γ + ρ).
2.2.1. The Lyapunov function
In the previous sections, the convergence of the learning process was proved, as the
system has a non-trivial, stable fixed point. However, this result has practical significance only if the region of asymptotical stability of this point is large enough.
Otherwise, starting at an initial weight that does not converge to the non-trivial
fixed point would be very probable, so the model would not accurately represent
learning. In this section, we will determine the region of asymptotical stability of
the stable fixed points of the system by means of the Lyapunov function technique
(see, e.g. [22] and references therein).
The Lyapunov function is a non-negative function of the state of the system
V (w) 0, which is zero at the fixed point (or invariant set) whose stability we are
trying to determine: V (w∗ ) = 0. If its derivative with respect to time is negative
dV (w)/dt < 0, at least for the values w in a neighbourhood of w∗ , then this point
is asymptotically stable. Moreover, the trajectories that start at the points w converge to w∗ . Thus, the Lyapunov function is useful both to prove the stability of a
point and to determine its region of asymptotical stability. The main drawback of
this technique is the absence of general methods to find an appropriate Lyapunov
function.
Consider now the trivial invariant set W0 = {w0 |x · w0 = 0}, which is an hyperplane orthogonal to the input x. The simulations performed in our model show
that the trajectories that converge to W0 never increase the distance to this hyperplane or, what is the same, the absolute value of the projection of the weight vector
DEVELOPMENT OF ORIENTATION SELECTIVITY IN VISUAL CORTEX
9
onto the input |x · w| always decreases. This fact, together with the convenience of
V being differentiable, suggests the choice of the following function:
1
y2
V0 (w) = (x · w)2 =
2
2
(27)
Now, the derivative with respect to time of our candidate to Lyapunov function
is calculated:
n
n
i=1
i=1
n
dV0 ∂V0 dwi yxi fi (w) = −βy 2
xi wi
=
=
dt
∂wi dt
+αy 2 (λ − y)(y − τ )
n
i=1
i=1
xi2 = −βy 3 + x2 αy 2 (λ − y)(y − τ )
= −x αy + y −β + x2 α(λ + τ ) − x2 αy 2 λτ
= −y 2 x2 αy 2 − γ y + x2 αλτ
γ +R
γ −R
2
= −y y −
y−
= −y 2 (y − x · w1 )(y − x · w2 )
2αx2
2αx2
2
4
3
(28)
It is remarkable, and it supports the choice for V, that dV0 /dt = 0 at the fixed
points of the system. Let us examine the sign of dV0 /dt. As x · w1 and x · w2 are
positive, expression (28) is negative for any w such that x · w < 0 and, also, when
x · w is positive but it keeps near from the hyperplane y = 0. Thus, dV0 /dt < 0 in a
neighbourhood around any point w0 ∈ W0 , except for other points of the invariant
set. Hence, the result that was inconclusive when using linearization is obtained by
means of the Lyapunov function: the set W0 is an attractor and its points are stable. Moreover, the ordering x · w1 > x · w2 implies dV0 /dt < 0 for any point w with
0 < x · w < x · w2 , so the region of convergence to the trivial set W0 is limited by
the hyperplane that passes through the unstable weight w2 and is orthogonal to
the input: W2 = {w0 |x · w2 = 0}.
The region of asymptotical stability of the stable fixed point w1 is now obtained.
The same reasoning as in the above paragraph, together with the simulated trajectories, suggests that our Lyapunov function could simply be the translation of V0
to w1 , i.e. the following function:
1
(y − x · w1 )2
V1 (w) = (x · w − x · w1 )2 =
2
2
(29)
and its derivative is now easily calculated with the help of Equation (28):
n
n
dV1 ∂V1 dwi y − x · w1 ∂V0 dwi y − x · w1 dV0
=
=
=
dt
∂wi dt
y
∂wi dt
y
dt
i=1
i=1
= −y(y − x · w1 )2 (y − x · w2 )
(30)
10
JEREZ ET AL.
which shows that dV1 /dt < 0 for any w such that x · w > x · w2 . Hence, the complete
weight space is divided by the hyperplane W2 = {w0 |x · w2 = 0} into two regions of
convergence. Any trajectory that starts at the region that contains the origin converges to the trivial invariant set, while trajectories that start ‘distant’ from the origin converge to the non-trivial stable fixed point, but notice that this distance is
measured on the parallel direction to the input.
3.
Simulation Results
In this section, some simulations with the general n-dimensional model (in this
context, dimension means number of afferent synaptic connections) are performed
on a realistic visual environment. Previously, simulations performed on a simpler
input environment (one and two dimensions) will provide a qualitative insight on
the properties of the CBA learning rule.
3.1.
one- and two-dimensional model
In the one-dimensional case, we have only one differential equation, where both
the input x and the weight w are scalars. The fixed points would be the weight
values w that satisfy the condition dw/dt = 0:
w0 = 0,
w1 =
γ +R
,
2x 3 α
w2 =
γ −R
2x 3 α
(31)
where the terms γ , ρ and R have been defined as
γ = −β + x 2 α(λ + τ ),
√
ρ = 2x 2 α λτ ,
R = γ 2 − ρ2
(32)
The derivative f (w) = dw/dt has been drawn in Figure 1. The geometrical intuition suggests that if f (a) > 0 (e.g. if a is largely negative) and the initial state of
the system is w = a, w will increase. On the other hand, starting from w = a, w will
decrease if f (a) < 0, e.g. if a is largely positive. The increasing or decreasing evolution will continue until a fixed point is reached, but the system ‘corrects’ itself so
that its state does not blow up to ±∞.
Further insight is obtained from simulations performed varying some of the
learning rule parameters. In Figure 2, examples of the effect of these parameters
on the weight evolution are shown.
Although the one-dimensional model gives us an idea about the system dynamics, one cannot obtain selectivity with this restricted environment. Thus, the twodimensional case is quite instructive, since input and weight vectors can be plotted
easily in a coordinates axis. Studying the convergence in this reduced environment
gives us an intuitive idea about the structures extracted from the input environment by the learning rule. According to Equation (2), in this two-dimensional
11
DEVELOPMENT OF ORIENTATION SELECTIVITY IN VISUAL CORTEX
f(w)
w 1 – attractor
w – attractor
0
w
w – saddle
2
Figure 1. Graph of the function f(w), which shows the behaviour of the system dw/dt = f (w).
Tau = 0.5
1.5
1
1
Beta = 0.001
Beta = 0.001
Tau = 0.25
1.5
0.5
0
0
1000
2000
3000
4000
0.5
0
0
5000
1000
2000
3000
4000
5000
4000
5000
Tau = 0.5
Tau = 0.25
1.4
1
1.2
0.8
Beta = 0.005
Beta = 0.005
1
0.6
0.4
0.8
0.6
0.4
0.2
0.2
0
0
1000
2000
3000
# Iterations
4000
5000
0
0
1000
2000
3000
# Iterations
Figure 2. Simulation results in the one-dimensional model. The value of the final weights for different initializations as functions of time for different values of the heterosynaptic competition term (β)
and the learning threshold (τ ). The value of the learning constant α is 0.05, whereas the input x was
restricted to the constant value of 0.7.
12
JEREZ ET AL.
1.5
w1
1
0.5
w2
0
w0
–0.5
–1
–1.5
–1.5
–1
–0.5
0
0.5
1
1.5
Figure 3. Phase indicating the weights dynamics from different initial values (represented as circles).
These results were achieved setting the learning constant (α) to the value 0.05, the maximum activity
level (λ) was set to the value 1.0, the threshold (τ ) was 0.25, and the heterosynaptic competition term
(β) was set to 0.0025.
input environment, the output y would be merely the projection of a given vector, x, on the weight vector w. Learning then becomes the search for a particular
direction in the data space which satisfy the criteria of the CBA equations.
Figure 3 illustrates the trajectories followed by different weight initializations
(drawn as circles) in a two-dimensional states space. In this figure, we can observe
one attractor fixed point, w1 , one saddle point, labelled as w2 , and a set of stable points located on a line W0 that is perpendicular to the stable vector w1 and
crosses the origin. Since w1 and the input x are parallel (see Equation (9)), every
vector w0 ∈ W0 produces null output y = w0 · x = 0. If a second line is traced, parallel to the former and crossing the unstable point w2 , then the whole plane is
divided into three regions. The trajectories that are initialized in the two regions
that lie at both sides next to W0 are attracted by W0 , thus producing zero output,
which is interpreted as a lack of selectivity. The size of this no selectivity region
depends on the model parameters. These observations have strongly inspired the
search for a Lyapunov function, and its relation to the size of basins of attraction,
as shown in Section 2.
3.2.
two input patterns environment
Next, in order to validate the analytic results in a more complex input environment, we simulate again the two-dimensional model, but the restriction of constant
13
DEVELOPMENT OF ORIENTATION SELECTIVITY IN VISUAL CORTEX
ws
y
y
x1
y
x1
x1
ws
ws
w0
w0
w0
x2
x2
x2
x
BCM rule
x
x
PCA rule
CBA rule
Figure 4. Fixed points of the two-dimensional system, with two equiprobable inputs, starting from the
initial vector w0 . At the fixed points for the BCM, the weights vector is orthogonal to all but one of the
inputs. The PCA seeks the first principal component. The CBA learning rule behaviour is similar to that
of the PCA, but it presents a stronger bias towards the input with maximum modulus.
input is eliminated. Two input patterns, x1 and x2 are randomly presented to the
neuron with equal probabilities. This frame of analyzing the learning rule dynamics was proposed by Blais [7] to compare the behaviours of BCM and PCA rules.
Blais demonstrated that the only two stable fixed points for the BCM rule are the
ones where the weight vector is orthogonal to all but one of the inputs whereas
the PCA learning rule drives the neuron to seek the first principal component (see
Figure 4).
In order to help elucidate the differences between the dynamics of PCA and
BCM with a two-dimensional input environment, Blais [7] studied the neuron
response to each input pattern when the fixed point has been reached. In Figure 5,
the responses for PCA, BCM and CBA learning rules are compared. The results
show that the PCA rule is trying to have most of its responses strong, the BCM
rule tries to have a small subset of its responses strong and the others weak, and
the CBA gives the maximum response to an input pattern, which is the most
‘important’ due to either its presentation frequency or its modulus. These results
can help understand the relation of the receptive fields achieved by the CBA rule
to that produced by the BCM and PCA rules, when a more realistic visual environment is used in the next section.
3.3.
realistic visual environment
In the n-dimensional model, real two-dimensional natural images processed using
simple retinal models are used as visual inputs. Circular regions from the retina
(monocular rearing) are used to generate activity in the cortical cell.
The visual environment used is similar to that described in Law et al. [11], and
it is made of 24 natural images scanned into 256 × 256 pixel images (Figure 6,
top), where man-made objects have been avoided, since they would make it easier
to achieve receptive fields, given their sharp edges and straight lines characteristics. The retina is composed of square arrays of receptors which have antagonistic
center-surround receptive fields that approximate a DOG filter. The ratio of the
14
JEREZ ET AL.
1.1
Neuron response to input pattern X1
1
CBA
0.9
PCA
0.8
0.7
0.6
0.5
BCM
0.4
0.3
–0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
Neuron response to input pattern X2
Figure 5. Output distributions for BCM, PCA and CBA learning rules. BCM seeks orthogonality to
one of the input vectors, PCA tries to maximize responses to the set of input vectors, while CBA maximizes response to the most significant input.
Figure 6. Input environment. Shown are three of the original 24 natural images (top) and the retinality
processed images with a difference of Gaussian (DOG) filter, used as the real inputs to the neuron (bottom) (Courtesy of the University of Brown).
DEVELOPMENT OF ORIENTATION SELECTIVITY IN VISUAL CORTEX
15
surround to the center of the Gaussian distribution is approximately 3:1, which has
been biologically observed in Ref. 12. Figure 6 (bottom) shows the center surround
effect of the retinal preprocessing modelled by convolving the original images with
this DOG filter.
The model neuron was trained with 13 × 13 pixels patches randomly taken from
the images. For every simulation step, the activity of the input cells in the retina
is determined by randomly picking one of the 24 images and randomly shifting
the receptive field mask. The activity of each input in the model is determined by
the intensity of a pixel in the image. In order to do simulations with this realistic visual environment, a variation of a non-linear neuron with a non-symmetric
sigmoidal transfer function is used, and the cell activity is then given simply as
y = σ (w · x). The exact time course of these simulations depends on the parameter set chosen, so we have examined these over a large range. Table 1 shows the
range of parameters used to obtained the results presented in Figure 7.
At the beginning of the simulation, synaptic weights are initialized to small random values. At this stage, the cortical neuron is relatively unresponsive to visual
stimuli, although the initialization of the weights adds a bias that influences the
final selectivity of the neuron. Figure 7 shows the weights resulting from these
simulations starting from different initial conditions. With this realistic input environment, the CBA neuron develops receptive fields with distinct excitatory and
inhibitory regions. Notice, also, that the variety of oriented receptive fields structures obtained is significant enough. These results are also consistent with the
Table 1. Setting of learning rule parameters for
simulations in a realistic visual environment.
Learning constant (α)
Maximum level of activity (γ )
Threshold level (τ )
Heterosynaptic competition term (β)
Input
Weights initialization
Number of iterations
0.005
1.0
0.15
0.00025
[−10.10, 0.10]
[−10.15, 0.15]
250 000
Figure 7. Different types of cortical receptive fields arising from the CBA learning rule. The individual
plots show the weights vector for a two-dimensional receptive fields with white denoting positive values
and black negative values (synaptic efficacies).
16
JEREZ ET AL.
Figure 8. Example receptive fields achieved by BCM (top) and PCA (bottom) trained in a realistic
visual environment composed by natural images.
receptive fields achieved by other learning rules proposed in the literature [13, 15,
20]. Figure 8 also shows examples of receptive fields for BCM and PCA rules
trained in the same visual environment (same input patterns sequence during the
training process) as CBA rule.
The receptive fields achieved by these three learning rules can be also characterized and compared in terms of their spatial frequency, orientation selectivity and
spatial phase. We have performed this task considering the response of a receptive
field to a stimuli consisting in a circular two-dimensional moving sinusoidal grating with 64 grey levels. The spatial frequency tuning of the response was obtained
by varying its value from 0.45 to 3.0 rad/deg with increments of 0.05 rad/deg.
The spatial phase of the grating was advanced from 0◦ to 360◦ across the receptive field in 45◦ increments. The orientation tuning was determined at the optimal
spatial frequency by changing the angle of the drifting grating from 0◦ to 180◦
step by 7.5◦ . Moreover, the number of subregions within the receptive field is typically expressed in terms of the bandwidth of the cell. The bandwidth is defined
as b = log2 (K+ /K− ) where K+ > k and K− < k are the spatial frequencies of gratings that produce one-half the responses amplitude of a grating with K = k. High
bandwidths indicate that the receptive field has few subregions and poor spatial
frequency selectivity.
Results indicate that BCM receptive fields are clearly selective to bars of lights
at different orientations, with spatial frequency values ranging from 1.5 to 2.5,
whereas PCA develops receptive fields always divided into two antagonist regions,
one of them with synaptic potentiation and the other one with synaptic depression
(spatial frequency of 0.5 approximately). Establishing a comparison to the receptive fields structures in Figure 7, we can assess that the CBA learning rule can
develop receptive fields with properties similar to those achieved by both PCA and
BCM rules. Figure 7 shows examples receptive fields with the same structure as
PCA receptive fields, and others becoming selective to bars of lights at different
positions, but with a spatial frequency less than the receptive fields achieved by
the BCM rule (between 0.5 and 1.05 rad/deg). Preferred orientations of the resulting RFs (for BCM, PCA and CBA rules) spread out widely, and concentrated
slightly in the range from 80◦ to 120◦ , since the images contained vegetal forms
that aligned more in vertical orientation. With regard to the bandwidth, small
DEVELOPMENT OF ORIENTATION SELECTIVITY IN VISUAL CORTEX
17
values for BCM rule (0.85–1.35 octaves) indicate fine spatial frequency selectivity,
whereas higher values for CBA (1.75–2.25 octaves) and PCA (2.5 octaves) rules
imply less spatial frequency selectivity. These results are also consistent with those
values experimentally observed in the work of Hubel and Wiesel [8, 9].
4.
Conclusions and Future Work
This paper has shown that the CBA learning rule is appropriate to develop cells
with oriented receptive fields in visual cortex. This learning rule contributes with a
term for saturating the synaptic growth, such that any additional weight limit and
normalization constraint is avoided. With regard to the type of synaptic competition implemented, CBA integrates both heterosynaptic and homosynaptic methods through different parameters in the synaptic modification equation. The results
have shown that, in a realistic visual environment, the CBA rule develops oriented
receptive fields similar to those achieved by both BCM and PCA learning rules.
In addition, the simulation results presented robustness and a high level of stability on a wide range of parameter values.
Moreover, an exhaustive mathematical analysis of the CBA rule has been done
in the n-dimensional environment, determining the stability conditions as well as
the basins of attraction for the system fixed points. This analysis has provided
a better understanding of the CBA fundamental properties, and a mathematical
relation among the parameters of the learning rule identifying a region where the
system works properly. It is important to outline that the results obtained using
simplified input environments (one and two dimensions) have shown the correspondence between both the theoretically and simulated attractors of the system
dynamics. From these results we can infer that the CBA rule might be appropriate
for studying the biologically process of receptive field formation in visual cortex.
Two immediate steps arise from this research as future works. On the one hand,
it is preceptive to study the properties of CBA modification dynamics and the
influence of the learning rule parameters in normal and deprived environments
through experiments of visual deprivation. Also, the process of direction selective
receptive fields formation in visual complex cells can be studied in terms of the
CBA rule. On the other hand, the receptive fields achieved by this learning rule
might be considered as filters susceptible of being applied as the first stage in
the features extraction process carried out in image processing and artificial vision
tasks.
Acknowledgments
This article would hardly have been finished without the help of Dr Brian Blais
(Bryant College). The approach has also benefited from fruitful discussions with
Dr Leonardo Franco (University of Oxford) and revision by Dr Francisco R.
Villatoro (University of Málaga). Also, we would like to thank the referees for
18
JEREZ ET AL.
their valuable comments and suggestions. This work has been funded by the project CICYT TIC2002-04242-C03-02.
Appendix
A.1 Eigenvalue Calculations
In this section, some calculations will be performed to determine the eigenvalues of
some matrices with a special form. The eigenvalues of a matrix A are the roots of
the characteristic polynomial but, here, an indirect technique will be used to avoid
the difficulties of solving the equation |A − λI| = 0. Instead, we will prove that some
polynomial p(t) annihilates the matrix A and none of its factors does the same,
in other words, that p(t) is the minimum polynomial of A. Then, we will apply
a well-known result from linear algebra (the roots of the characteristic polynomial
are the same, although perhaps with different multiplicity, than the roots of the
minimum polynomial) to finally conclude that the eigenvalues of A are the roots
of p(t).
Along this section, x is assumed to be any n-dimensional vector.
LEMMA A.1. Let w0 be an orthogonal to x n-dimensional vector and let A be the
matrix A = axx − bw0 x . Then, the eigenvalues of A are t1 = 0 and t2 = ax2 .
Proof. Let us see that the polynomial p(t) = t (t − ax2 ) is the minimum polynomial of A. It is obviously minimal, as neither t nor t − ax2 annihilates A. But
p(t) does annihilate A:
p(A) = axx − bw0 x axx − bw0 x − ax2 I
= a 2 x2 xx − a 2 x2 xx − abx2 w0 x + abx2 w0 x = 0
(33)
LEMMA A.2. Let A be the matrix A = axx − bI. Then, the eigenvalues of A are
t1 = b and t2 = ax2 + b.
Proof. Let us see that the polynomial p(t) = (t − b)(t − ax2 − b) is the minimum
polynomial of A. It is obviously minimal, as neither t − b nor t − ax2 annihilates
A. But p(t) does annihilate A:
p(A) = axx + bI − bI axx + bI − ax2 I − bI
= a 2 x2 xx + abxx − a 2 x2 xx − abxx = 0
(34)
DEVELOPMENT OF ORIENTATION SELECTIVITY IN VISUAL CORTEX
19
References
1. Abraham, W. and Goddard, G.: Asymmetric relationships between homosynaptic longterm potentiation and heterosynaptic long-term depression, Nature 305 (1983), 717–719.
2. Amari, S., Cichocki, A. and Yang, H.: A new learning algorithm for blind signal separation, In Advances in Neural Information Processing Systems, Vol. 8. MIT Press, pp. 757–
763.
3. Artola, A., Bröcher, S. and Singer, W.: Different voltage-dependent threshold for inducing
long-term depression and long-term potentiation in slices of rat visual cortex, Nature 347
(1990), 69–72.
4. Bell, A. and Sejnowski, T.: An information-maximization approach to blind separation
and blind deconvolution, Neural Comput. 7(6) (1995), 1129–1159.
5. Bell, A. and Sejnowski, T.: The independent components of natural images are edge filters, Vision Res. 37(23) (1997), 3327–3338.
6. Bienenstock, E., Cooper, L. and Munro, P.: Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex, J. Neurosci. 2(1)
(1982), 32–48.
7. Blais, B.: The role of the environment in synaptic plasticity: towards an understanding of
learning and memory, PhD thesis, Brown University.
8. Hubel, D. and Wiesel, T.: Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex, J. Physiol. 160 (1962), 106–154.
9. Hubel, D. and Wiesel, T.: Receptive fields and functional architecture of monkey striate
cortex, J. Physiol. 195 (1968), 215–243.
10. Hyvärinen, A. and Hoyer, P.: Topographic independent component analysis as a model
of V1 organization and receptive fields, Neurocomputing 38–40 (2001), 1307–1315.
11. Law, C. and Cooper, L.: Formation of receptive fields according to the BCM theory of
synaptic plasiticity in realistic visual environment, Proc. Natl Acad. Sci. 91 (1994), 7797–
7801.
12. Linsenmeier, R., Frishman, L., Jakiela, H. and Enroth-Cugell, C.: Receptive field properties of X and Y cells in the cat retina derived from contrast sensitivity measurements,
Vision Res. 22 (1982), 1173–1183.
13. Linsker, R.: From basic network principles to neural architecture: emergence of orientation-selective cells, Proc. Natl Acad. Sci. 83 (1986), 8390–8394.
14. Lynch, G., Dunwiddie, T. and Gribkoff, V.: Heterosynaptic depression: a postsynaptic
correlate of long-term potentiation, Nature 266 (1977), 737–739.
15. Miller, K.: A model for the development of simple cell receptive fields and the ordered
arrangement of orientation columns through activity-dependent competition between onand off-center inputs, J. Neurosci. 14 (1994), 409–441.
16. Miller, K.: Synaptic economics: competition and cooperation in correlation-based synaptic plasticity, Neuron 17 (1996), 371–374.
17. Neskovic, P. and Verschure, P.: Evaluation of a biologically realistic learning rule, BCM,
in an active vision system, In Telluride Workshop on Neuromorphic Engineering.
18. Oja, E.: A simplified neuron model as a principal component analyzer, Math. Biol. 15
(1982), 267–273.
19. Sejnowsky, T.: Storing covariance with nonlinearly interacting neurons, J. Math. Biol. 4
(1977), 303–321.
20. Shouval, H. and Liu, Y.: Principal component neurons in a realistic visual environment,
Network 7(3) (1996), 501–515.
21. Vico, F. and Jerez, J.: Stable neural attractors formation: learning rules and network
dynamics, Neural Process. Lett. 18 (2003), 1–16.
20
JEREZ ET AL.
22. Vidyasagar, X. and Mathukumalli, X.: Nonlinear Systems Analysis. Prentice Hall, Englewood Cliffs, NJ, 1993.
23. Von der Malsburg, C.: Self-organization of orientation sensivity cells in striate cortex,
Kybernetik 14 (1973), 85–100.
24. Zhang, L. and Mei, J.: Shaping up simple cell’s receptive field of animal vision by ICA
and its application in navigation system, Neural Networks 16 (2003), 609–615.