CHAPTER 3
An
Illustrative
Example
Ming-Feng Yeh
1
Objectives
Use three different neural network
architectures to solve a simple pattern
recognition problem.
Feedforward network: perceptron
Competitive network: Hamming network
Recurrent associative memory network:
Hopfield network
Ming-Feng Yeh
2
Problem Statement
shape
texture
weight
Neural
Network
Sensors
Sorter
Apples
Ming-Feng Yeh
Oranges
3
Problem Statement
Shape sensor: 1 -- round, –1 -- elliptical.
Texture sensor: 1 -- smooth, –1 -- rough.
Weight sensor: 1 -- > 1 pound, –1 -- < 1 pound.
1
 shape 
1
p  texture   p(apple)   1 , p(orange)   1
 


 
 1
 weight
 1
Ming-Feng Yeh
4
Single-layer Perceptron
p
SR
W
R1
1
R
Ming-Feng Yeh
b
S1
n = Wp + b
+
n
a
S1
a = hardlims(Wp + b)
S1
S
Symmetrical Hard Limit
• a = -1, if n  0
• a = +1, if n  1
• MATLAB function: hardlims
5
Two-Input / Single-Neuron
Perceptron
p1
w11
p2
w12

b
1
Inner product
n
a
 p1 
p   , W  w11 w12 
 p2 
 n  Wp  b
a  hardlims (n)
Single-neuron perceptrons can classify input
vectors into two categories.
If Wp  b, then a = +1; otherwise, a = -1.
Ming-Feng Yeh
6
Two-Input / Single-Neuron
Perceptron p
W  w11 w12    1 1, b  1
 n  Wp  b   p1  p2  1
 a  hardlims (n)
n0
W
2
n=0
1
n0
1
1
p1
The decision boundary between the categories is
determined by n = Wp + b = 0.
Because the boundary must be linear, the singlelayer perceptron can only be used to recognize
patterns that are linear separable (can be separated
by a linear boundary).
Ming-Feng Yeh
7
Pattern Recognition
Example
 shape 
p  texture  three - dimensiona l input ( R  3)
weight 
n  Wp  b, a  hardlims (n)
Choose the bias b and the elements of the weight
matrix W so that the perceptron will can distinguish
between apples and oranges.
Ming-Feng Yeh
8
Pattern Recognition
Example
1
1
p(apple )   1  p(orange)   1  W  0 1 0, b  0
 
 1
 1

1 




Orange : a  hardlims  0 1 0 1  0   1





1
  


1 




Apple : a  hardlims  0 1 0 1   0   1





1
  

Ming-Feng Yeh
9
Pattern Recognition
Example
What happens if we put a not-so-perfect orange
into the classifier? That is to say, an orange with an
elliptical shape is pass through the sensor.
 shape 
p  texture 
 weight 
 1   1
p(orange)   1   1
 1  1

 1 




a  hardlims  0 1 0 1  0   1  orange





1
  

Ming-Feng Yeh
10
Hamming Network
Hamming network was designed explicitly to
solve binary (1 or –1) pattern recognition
problem.
It uses both feedforward and recurrent
(feedback) layers.
The objective is to decide which prototype
vector is closest to the input vector.
When the recurrent layer converges, there
will be only one neuron with nonzero
output.
Ming-Feng Yeh
11
Hamming Network
Feedforward Layer
p
R1
1
R
Recurrent Layer
SR
1
W
+
b
1
n1
a1
S1
S1
S1
n1 = W1p + b1
a1 = purelin(n1)
Ming-Feng Yeh
a2(t+1)
SS
2
W
S1
S1
n2(t+1)
S
D
S1
a2(t)
S
n2(t+1) = W2a2(t)
a2(t+1) = poslin[n2(t+1)]
a2(0) = a1
12
Feedforward Layer
p
R1
1
SR
1
W
+
b
1
n1
a1
S1
S1
n1 = W1p + b1
a1 = purelin(n1)
S
R
S1
The feedforward layer performs a correlation, or inner
product, between each of the prototype patterns and the
input pattern.
The connection matrix W1 are set to the prototype patterns.
Each element of the bias vector b1 is set to be R, such that
the outputs a1 can never be negative.
Ming-Feng Yeh
13
Feedforward Layer
T

 1  1  1 1 3
p
1
1
W  T
,b   

3
p 2  1 1  1
T
T


3


p
p


1
1
1
1
1p3
a  W p  b   T p      T

3
p
p
p

3
 2    2

The outputs are equal to the inner products
of each prototype pattern (p1 and p2) with the
input (p), plus R (3).
Ming-Feng Yeh
14
Feedforward Layer
 1
 1
1  1  1   3 4


1
1
1
p   1  a  W p  b  
 1      


1 1  1
3 2


 1
 1
The Hamming distance between two vectors is
equal to the number of elements that are different.
It is defined only for binary vectors.
The neuron with the largest output will correspond
to the prototype pattern that is closest in Hamming
distance to the input pattern.
Output = 2  (R – Hamming distance)
a11 = 2  (3 – 1) = 4, a12 = 2  (3 – 2) = 2
Ming-Feng Yeh
15
Recurrent Layer
a2(t+1)
SS
2
W
S1
n2(t+1)
S1
S
D
S1
a2(t)
n2(t+1) = W2a2(t)
a2(t+1) = poslin[n2(t+1)]
a2(0) = a1
a1
The recurrent layer of the Hamming network is what
is known as a “competitive” layer.
The neurons compete with each other to determine a
winner. After the competition, only one neuron will
have a nonzero output.
Ming-Feng Yeh
16
Recurrent Layer
 2
a12 (t )
1
,
a
(
t
)

,
and


 2 
1 
S 1
a2 (t )
2
2



a
(
t
)


a
2
2 2
1
2 (t ) 

a (t  1)  poslin W a (t )   poslin  2
 a (t )  a 2 (t ) 
1

 2
1
2
W 
 
Each element is reduced by the same fraction of the
other.  The difference between large and small will
be increased.
The effect of the recurrent layer is to zero out all
neuron outputs, except the one with the largest
initial value (which corresponds to the prototype
pattern that is closest in Hamming distance to the input).
Ming-Feng Yeh
17
Recurrent Layer
 1
 0.5
 4
 1


2
1
2
If p   1  a (0)  a   , let   0.5  W  

2

0
.
5
1
 


 1
1


1

  3    3

2   4
2
  poslin       
a (1)  poslin   1



 0  0


1
2
  
    
 2
1


1

  3    3

2   3
2



 
a (2)  poslin   1

poslin




  1.5  0


1
0


  
2





Since the outputs of successive iterations produce the
same result, the network has converged.
Prototype pattern number one, the orange, is chosen
as the correct match.
Ming-Feng Yeh
18
Hopfield Network
SS
p
W
S1
+
1
S
a(t+1)
S1
n(t+1)
S1
D
a(t)
S1
b
S1
S
n(t+1) = Wa(t) + b, a(t+1) = satlins[n(t+1)], a(0) = p
In the Hamming network, the nonzero neuron
indicates which prototype pattern is chosen.
The Hopfield network actually produces the
selected prototype pattern as its output.
Ming-Feng Yeh
19
Hopfield Network
0
0.2 0
 0.9 
a(t  1)  satlins Wa (t )  b , W   0 1.2 0 , b   0 
 0
 0.9
0 0.2
 0.2 0
0
 0.9  





a(t  1)  satlins   0 1.2 0 a(t )   0  
 0




0
0
.
2

0
.
9




 1
0.7
1
1
a(0)   1  a(1)    1   a(2)   1  a(3)   1
 1
  1 
 1
 1
Ming-Feng Yeh
20
Conclusions
The perceptron had a single output, which
could take on values of –1 (orange) or 1 (apple).
In the Hamming network, the single nonzero
neuron indicated which prototype had the
closest match (neuron 1 indicated orange and
neuron 2 indicated apple).
In the Hopfield network, the prototype
pattern itself appears at the output of the
network.
Ming-Feng Yeh
21