Theoretical and Computational Aspects of
Quaternionic Multivalued Hopfield Neural Networks
Marcos Eduardo Valle* and Fidelis Zanetti de Castro
Department of Applied Mathematics
Institute of Mathematics, Statistics, and Scientific Computing
University of Campinas - Brazil
28 July 2016
Marcos Eduardo Valle (Brazil)
Multivalued Quaternionic HNNs
28 July 2016
1 / 18
Historical Survey
Hopfield Neural Network (HNN)
Hopfield, 1982.
Marcos Eduardo Valle (Brazil)
Multivalued Quaternionic HNNs
28 July 2016
2 / 18
Historical Survey
Hopfield Neural Network (HNN)
Hopfield, 1982.
Complex-Valued Multistate Hopfield Neural Network
Jankowski, Lozowski, and Zurada, 1996.
Marcos Eduardo Valle (Brazil)
Multivalued Quaternionic HNNs
28 July 2016
2 / 18
Historical Survey
Hopfield Neural Network (HNN)
Hopfield, 1982.
Complex-Valued Multistate Hopfield Neural Network
Jankowski, Lozowski, and Zurada, 1996.
Quaternionic Multistate Hopfield Neural Network
or Multivalued Quaternionic HNN (MV-QHNN)
Isokawa, Nishimura, Saitoh, Kamiura, and Matsui, 2008.
Minemoto, Isokawa, Nishimura, and Matsui, 2015.
Marcos Eduardo Valle (Brazil)
Multivalued Quaternionic HNNs
28 July 2016
2 / 18
Outline of the Talk
1
Basic Concepts and Notation
2
MV-QHNN of Isokawa et al.
3
MV-QHNN of Minemoto et al.
4
Concluding Remarks
Marcos Eduardo Valle (Brazil)
Multivalued Quaternionic HNNs
28 July 2016
3 / 18
Quaternions
A quaternion is an hypercomplex number
q = q0 + q1 i + q2 j + q3 k,
where i2 = j2 = k2 = ijk = −1 are the hyper-imaginary units.
Phase-angle representation:
q = |q|eiφ ekψ ejθ ,
where φ ∈ [−π, π), θ ∈ − π2 , π2 , and ψ ∈ − π4 , π4 .
Unique phase-angle representation:
n
πo
A = q = |q|eiφ ekψ ejθ : q 6= 0 and |ψ| =
6
.
4
Marcos Eduardo Valle (Brazil)
Multivalued Quaternionic HNNs
28 July 2016
4 / 18
MV-QHNN models
Like the complex-valued model of Jankowski et al., in a MV-QHNN the
phase-angle intervals are divided in sectors.
Resolution factors and phase quanta
Given positive integers K1 , K2 , and K3 , called resolution factors, the
phase quanta are defined by
∆φ =
2π
,
K1
∆θ =
π
,
K2
and ∆ψ =
π
.
2K3
Network dynamic
Given a quaternionic synaptic weight matrix W , neurons are updated
asynchronously if
n
X
vi (t) =
wij xj (t),
j=1
has an unique phase-angle representation, i.e., vi (t) ∈ A.
Marcos Eduardo Valle (Brazil)
Multivalued Quaternionic HNNs
28 July 2016
5 / 18
MV-QHNN of Isokawa et al.
Let the state and activation of the ith neuron at time t be
xi (t) = eiφi (t) ekψi (t) ejθi (t)
and vi (t) = |vi (t)|eiφ ekψ ejθ .
The neuron is updated asynchronously according to

φi (t)i eψI k eθI j ,

e
xi (t + 1) = or

 φI i ψI k θi (t)j
e e e
where
π
π/4 + ψ
π
π/2 + θ
π+φ
, ψI = − + ∆ψ
, θI = − + ∆θ
.
φI = −π + ∆φ
∆φ
4
∆ψ
2
∆θ
Note that the phase-angles of a quaternionic neuron are not updated
simultaneously!
Marcos Eduardo Valle (Brazil)
Multivalued Quaternionic HNNs
28 July 2016
6 / 18
Numerical Instability
The angles φI , ψI and θI are the clockwise edges of the arcs that
contain φ, ψ and θ.
A small perturbation can yield a significant change in φI , ψI or θI .
Example
The resolution factors K1 = K2 = K3 = 4 yield the following circular
sectors whose the boundaries are marked by black diamonds:
Marcos Eduardo Valle (Brazil)
Multivalued Quaternionic HNNs
28 July 2016
7 / 18
The conference paper contains examples in which:
Rounding errors yield an absolute error far greater than the
machine precision.
The MV-QHNN converges to an unexpected stationary state.
Isokawa et al. claimed that the energy
n
n
1 XX H
x W x,
E(x) = −
2
i=1 j=1
never increases if
wij = w̄ji
and wii ≥ 0,
∀i, j ∈ {1, . . . , n}.
In the following example, the energy oscillates!
Marcos Eduardo Valle (Brazil)
Multivalued Quaternionic HNNs
28 July 2016
8 / 18
Example (Stability problem:)
Let K1 = K2 = K 3 = 4,
0
1 − 2j − k
W =
1 + 2j + k
0
4
− π j
e 4
and x(0) =
.
1
MV-QHNN of Isokawa et al.
Updating all phase-angles
3
Energy
2
1
0
-1
-2
-3
0
Marcos Eduardo Valle (Brazil)
5
10
15
t
20
Multivalued Quaternionic HNNs
25
30
28 July 2016
9 / 18
Example (Stability problem:)
The energy at x(0) is
√
1 H
3 2
E(x(0)) = − x (0)W x(0) = −
= −2.1213.
2
2
The energy at x(1) is
π 1
E(x(1)) = − xH (1)W x(1) = − cos
= −0.92388.
2
8
The difference between the energies is
∆E = E(x(1)) − E(x(0)) = 1.1974.
Therefore, ∆E is positive even if we could work in exact arithmetic!
We address this issue in “Comments on the Stability Analysis of
Quaternionic Multistate Hopfield Neural Networks”.
Marcos Eduardo Valle (Brazil)
Multivalued Quaternionic HNNs
28 July 2016
10 / 18
MV-QHNN of Minemoto et al.
Minemoto et al. presented an MV-QHNN which have been
effectively used for the storage and recall of color images.
The novel model is obtained by shifting the phase-angles φI , ψI ,
and θI by half of the phase quanta.
Example
The resolution factors K1 = K2 = K3 = 4 yield the following circular
sectors whose midpoints are marked by blue diamonds:
Marcos Eduardo Valle (Brazil)
Multivalued Quaternionic HNNs
28 July 2016
11 / 18
MV-QHNN of Minemoto et al.
Let the state and activation of the ith neuron at time t be
xi (t) = eiφi (t) ekψi (t) ejθi (t)
and vi (t) = |vi (t)|eiφ ekψ ejθ .
The neuron is updated asynchronously according to

φi (t)i eψi (t)k eθM j ,

e




or
xi (t + 1) = eφi (t)i eψM k eθi (t)j ,



or



 φM i ψi (t)k θi (t)j
e e
e
where
1
φM =
2
π
π+φ
1
π
4 +ψ
−2π + ∆φ 2
+1
, ψM =
− + ∆ψ 2
+1
∆φ
2
2
∆ψ
and
θM =
1
2
Marcos Eduardo Valle (Brazil)
π
+θ
−π + ∆θ 2 2
+1
.
∆θ
Multivalued Quaternionic HNNs
28 July 2016
12 / 18
MV-QHNN of Minemoto et al.
Numerical Stability:
The phase-angles φM , ψM , and θM are insensitive to small
perturbations in φ, ψ, and θ.
Thus, the MV-QHNN of Minemoto et al. is numerically stable.
Stability Analysis:
Minemoto et al. claimed that the energy
n
E(x) = −
n
1 XX H
x W x,
2
i=1 j=1
never increases if
wij = w̄ji
Marcos Eduardo Valle (Brazil)
and wii ≥ 0,
∀i, j ∈ {1, . . . , n}.
Multivalued Quaternionic HNNs
28 July 2016
13 / 18
Example (Stability problem:)
Let K1 = K2 = K 3 = 4,
0
1 − 2j − k
W =
1 + 2j + k
0
-0.5
π
π π
e 4 i e 16 k e− 8 j
π
π
π
and x(0) =
.
e 4 i e 16 k e 8 j
MV-QHNN of Minemoto et al.
Updating all phase-angles
Energy
-1
-1.5
-2
-2.5
0
Marcos Eduardo Valle (Brazil)
1
2
3
4
5
t
Multivalued Quaternionic HNNs
6
7
8
28 July 2016
14 / 18
Example (Stability problem:)
The energy of the network at x(0) and x(1) are:
1
E(x(0)) = − xH (0)W x(0) = −1.3604,
2
1
E(x(1)) = − xH (1)W x(1) = −0.92388.
2
The difference between the energies at t = 1 and t = 0 is
∆E = E(x(1)) − E(x(0)) = 0.43651.
Despite ∆E positive, the MV-QHNN of Minemoto et al. reaches a
stationary state after 6 steps.
In contrast, the energy of the Modified MV-QHNN does not increase!
Marcos Eduardo Valle (Brazil)
Multivalued Quaternionic HNNs
28 July 2016
15 / 18
Modified MV-QHNN of Minemoto et al.
Let the activation of the ith neuron at time t be
vi (t) = |vi (t)|eiφ ekψ ejθ .
The neuron is updated asynchronously according to
xi (t + 1) = eφM i eψM k eθM j ,
where
1
φM =
2
π
π+φ
1
π
4 +ψ
−2π + ∆φ 2
+1
, ψM =
− + ∆ψ 2
+1
∆φ
2
2
∆ψ
and
θM =
1
2
π
+θ
−π + ∆θ 2 2
+1
.
∆θ
Note that all phase-angles are updated simultaneously!
Marcos Eduardo Valle (Brazil)
Multivalued Quaternionic HNNs
28 July 2016
16 / 18
Conjecture
We conjecture that, if
wij = w̄ji
and wii ≥ 0,
∀i, j ∈ {1, . . . , n},
then the energy
n
E(x) = −
n
1 XX H
x W x,
2
i=1 j=1
decreases if a neuron changes its state.
Thus, the modified MV-QHNN always reaches a stationary state.
Marcos Eduardo Valle (Brazil)
Multivalued Quaternionic HNNs
28 July 2016
17 / 18
Concluding Remarks
In this paper, we first revised the MV-QHNN of Isokawa et al.:
It is numerically unstable.
We cannot assure its convergence to a stationary state.
We also investigated the MV-QHNN of Minemoto et al.:
It is numerically stable.
Analogously, we cannot assure its convergence.
We proposed an MV-QHNN in which all phase-angles are updated.
We conjecture the novel model always reaches a stationary state.
Keep in mind...
we should only consider the improved model from now on!
Thank you!
Marcos Eduardo Valle (Brazil)
Multivalued Quaternionic HNNs
28 July 2016
18 / 18