1
1
Appendix
2
Analytical derivation of the conditions for the evolution of cross-talk in the simple model
3
The activities of the input neurons (x) and the output neurons (s) are represented by the two-
4
dimensional vectors:
5
x 
s 
x   1 , s   1
 x2 
s2 
6
When the network is presented with an input sample, the neuronal activities change in time
7
according to the following first order dynamics:
8

9
where  is a time constant. After the recurrent network reaches steady-state, the activities are given
ds1
ds
  s1  g w1 x1  k12 s2  ,  2   s2  g w2 x2  k 21s1  ,
dt
dt
10
by:
11
s1  g w1x1  k12s2   g h1  , s2  g w2 x2  k21s1   g h2 
12
The function g is a non-linear monotonic (invertible) function.
13
The recurrent interactions form a 2x2 matrix, K. Assuming no self-coupling, the main diagonal is
14
set to zero:
15
 0 k12 
K 

k 21 0 
16
The information maximization learning rule for the recurrent interactions in this case is
17
(Methods; 27):
18
k    T   T as T
19
where  is the learning rate. The matrix  is given by:   I  GK 1 G  G 1  K  ,
20
where G is a diagonal matrix with elements Gij  g (hi )ij  giij .
21
The components of the vector a are given by:
1
2
g k
g 
g 
1
1
 I  GK  k g k  k 3  I  GK  k  k 2
3
g k 
g k 
g k 
22
ak  kk 
23
The triangular brackets denote averaging over the input samples. Applying this to our simple
24
network model, we obtain the following equations for the off-diagonal elements of the
25
interaction matrix:
26
 g  g g  
k12   q  g1g 2 k21  q 2   1  1 2 k21  s2   f1
g 2
 g1

27
 g  g  g  
k21   q  g1g 2 k12  q 2   2  2 1 k12  s1   f 2
g1
 g 2

28
where q 
29
The resulting learning rules form a set of two coupled dynamical equations of the form:
30
 k12 
 f1 k12 , k 21 
k     f k , k 
 21 
 2 12 21 
31
We assume that under normal conditions, when there are no correlations between the inputs to
32
the two modalities, a state of no cross-talk should be a fixed-point of the learning dynamics.
33
The conditions for this state to be a fixed-point are:
34
 k12 
0
k 
 21  K 0
35
Non-zero cross-talk connections will evolve when this fixed point becomes unstable.
36
In order to analyze the behavior of the learning dynamics near a state of no cross-talk, we
37
linearize the learning rules around the fixed point of K=0:
38
 f1
 k
 k12 
 12




 f 2
 k21 
 k
 12
39
Where:
1
.
1  g1 g 2 k12 k 21
f1 
k21   k12 
 k 
 
 J  12 

f 2   k21 
 k21 

k21 
3
40
41
f 1
k12
f 2
k 21

K 0
K 0
s 22
g1

g12  , f1



g

 1

g1  k 21

2

s12 
g 2   f 2




g2 
,
g 2 
g 2  k12
 g1  g 2  s1 g 2
K 0

K 0
f1
k 21
g1
g 
 s 2 g1 2
g1
g 2
K 0
42
The discrete-time dynamics are given by:
43
k12 
k12 
k12 
k12 
k12 
k   k   J k   I  J k   Ak 
 21  n1  21  n
 21  n
 21  n
 21  n
44
and the formal solution is:
45
k
k12 
n  12 
k   A k 
 21  n
 21  0
46
The condition for stability is convergence of An when n   . Therefore, the eigenvalues of
47
A must satisfy:
48
i 1
49
The eigenvalues of A can be expressed using those of J:
50
 i  1    i
51
The eigenvalues of the matrix J are:
52
1, 2 
53
Where:
54
Tr  J  
1 
2
 Tr J   Tr J   4 Det J 

2 
g 2  s 2 
g  2 
f1
f 2
s2 

 2  g1 1   1  g 2  2 
k12
k 21
g1 
g1  g 2 
g 2 
Det  J  
55
56
f 1
f 2
f 2
f 1




k 12
k 21
k12
k 21
2
2


s 22 
g 1 
s12 
g 2  
g 1
g 

 s 2 g 1 2
 g 1
 
 g 2 
  g 1  g 2  s1 g 2
g 1 
g 1 
g 2 
g 2 
g 1
g 2
To be concrete, we choose the logistic function:
2
4
57
g x  
58
This function satisfies the following relations:
59
g x   g x   1  g x  , g x   g x   1  2  g x  , g x   g x   1  2  g x   2  g x 
60
We obtain:
1
1  ex
2
f1
k12
K 0
s2
 2
g1
2
2




g1 
2 g1  g1 






g


s

 1

2
  
g1 
 g1  g1  


 s22  1  2  g1  x   2 g1  x   1  g1  x   1  2  g1 x 
61

2
2


s1  g1  x 

 2 s1  s12 s22  2s1s22  2s12 s22
f1
k 21
 g1  g 2  s1 g 2
K 0
g1
g 
 s2 g1 2
g1
g 2
62
 s1 1  s1 s2 1  s2   s1s2 1  s2 1  2s1   s1s2 1  s1 1  2s2 
63
Using symmetry considerations, we also obtain:
64
f 2
k21
 s1s2  s1s22  s2 s12  s12 s22  s1s2 2  3s1  3s2  4s1s2   3s1s2  4s12 s2  4s1s22  5s12 s22
 2s2 s12  2s12 s22 ,
K 0
f 2
k12

K 0
f1
k 21
K 0
65
The trace and determinant are thus given by:
66
  Tr J   4 s12 s22  2 s1s22  2 s2 s12
67
  Det J   4 s12 s22  s1s22  s12 s22  s2 s 21  3 s1s2  4 s1s22  4 s2 s12  5 s12 s22
68
We assume that s1 and s2 are statistically independent. Therefore:
69


 

2
s1n s 2m  s1n s 2m
70
The requirement for K=0 to be a fixed point of the learning dynamics gives the following
71
condition:
5






g1 x 
s2
g1  x 
g 2  x 
s1
g 2  x 


    s 2 1  2 s1      s 2 1  2 s1    0





 s1 1  2 s 2  
 s1 1  2 s 2  


72
 k12 
 k 
 21  K 0
73
The only relevant solution is:
74
s1  s 2 
1
2
75
Denoting si2   i2 we obtain
76
  412 22  12   22
77

78
The eigenvalues of the matrix A should satisfy:
79
 1, 2  1       2  4  1
80
Thus, the following inequalities must be satisfied:
81
  0 ,   0 (else we get  1  1 ).
82
There are two options that we need to consider – real eigenvalues and complex. If the
83
84
eigenvalues are complex, their magnitudes are:
9
29
 3 12  3 22  4 14  4 24   12 22  18 12 24  18 14 22  21 14 24
16
2
1
2




2
85
 1, 2
86
The critical value of  is when the magnitude equals 1:
87
88
2
1
 1  1

 1       2  4  1       4   2   1     2 
2
 2  2

1     2   1    2   0   


6
89
If the eigenvalues are real, we have to check that 1 (the larger eigenvalue) is no greater than 1
90
and that  2 (the smaller eigenvalue) is no smaller than -1:
91
1
1
1   critical     2  4  1   c     2  4  0     2  4  0
2
2
92
Thus,  1 does not depend on  .
93
1
1
  2  4
1  critical    2  4  1  c    2  4  2  c  
2
2

94
The value of  c can be written in general (for both complex and real eigenvalues) as:
95
c  
96
Numeric calculations show that under the conditions of   0 ,   0 the discriminant is
97
positive. Therefore the relevant solution is  c  
98
this expression diverges to infinity.
99
As shown above, the value of 1 does not depend on  , but does depend on  12 ,  22 . It can be
100
noticed that the only solution for    2  4  0 is when   0 . In this case, 1 reaches the
101
critical value of 1, and the dynamics become unstable. The condition for this instability is:








  2  4 

 Re 







   2  4

. Note that when 1 ,  2   ,
9
29
 3 12  3 22  4 14  4 24   12 22  18 12 24  18 14 22  21 14 24  0
16
2
102
 
2
1



36 24  29 22  6   6 22 2 22  1
3
84 24  72 22  16
103
This relation defines the curves in Figures 3A, 3B and 4.
104
The expression in the root cannot be negative, therefore 0   22 
105
considerations we also obtain 0   12 
106
The variance of the output neuron's activity is:
1
.
2
1
. From symmetry
2
7
107
i2 
s 
108
Thus:
1
1
1
  i2  or 0  i2 
4
2
4
109
i
si

2
 si2  2si si  si
2
 si2  2 si
2
 si
2
 si2  si
2
 i2 
1 1 1 1
  
4 2 4 4