Matakuliah
Tahun
Versi
: H0434/Jaringan Syaraf Tiruan
: 2005
:1
Pertemuan 13
BACK PROPAGATION
1
Learning Outcomes
Pada akhir pertemuan ini, diharapkan mahasiswa
akan mampu :
• Menjelaskan konsep Back Propagation.
2
Outline Materi
• Algoritma Back Propagation
3
Multilayer Perceptron
R – S1 – S2 – S3 Network
4
Example
5
Elementary Decision
Boundaries
First Boundary:
1
a 1 = hardlim – 1 0 p + 0.5 
Second Boundary:
1
a2 = hardlim 0 – 1 p + 0.75
First Subnetwork
6
Elementary Decision
Boundaries
Third Boundary:
1
a3 = hardlim 1 0 p – 1.5 
Fourth Boundary:
1
a 4 = hardlim 0 1 p – 0.25 
Second Subnetwork
7
Total Network
–1
1
W = 0
1
0
0
–1
0
1
2
W = 1 100
0 011
3
W = 11
0.5
b 1 = 0.75
–1.5
– 0.25
2
b = –1.5
–1.5
3
b = –0.5
8
Function Approximation
Example
1
1
f  n  = ----------------–n
1 +e
2
f  n = n
Nominal Parameter Values
1
w1 1 = 10
2
1
w 2 1 = 10
w1  1 = 1
2
1
1
b1 = – 10
w1  2 = 1
b 2 = 10
2
b = 0
9
Nominal Response
3
2
1
0
-1
-2
-1
0
1
2
10
Parameter Variations
3
2
3
0
1
b2
2
 20
2
1
1
0
0
-1
-2
-1
0
1
2
3
2
-1
-2
-1
3
–1
2
 w1  2
1
0
0
0
1
2
1
2
2
2
1
-1
0
–1  b  1
1
-1
-2
– 1  w 1 1  1
1
2
-1
-2
-1
0
11
Multilayer Network
m+1
a
= f
m+ 1
m+1 m
W
a +b
m+ 1

m = 0 2   M – 1
0
a = p
M
a= a
12
Performance Index
Training Set
{ p1, t 1}  {p2 , t2}    { pQ, t Q}
Mean Square Error
2
2
F x = E e  = E t – a 
Vector Case
T
T
F x = Ee e  = E t – a   t – a 
Approximate Mean Square Error (Single Sample)
T
T
F̂ x  = t  k – a  k   t k  – a k   = e  k e  k
Approximate Steepest Descent
m
m
 F̂
w i j k + 1  = wi j k  – -----------m
w i j
m
m
F̂
b i k + 1  = b i k  –  -------m-b i
13
Chain Rule
d f  n w  
d f  n  dn  w 
----------------------- = --------------  --------------dw
dn
dw
Example
f  n = cos n
n = e
2w
2w
f  n w  = cos e

d f n  w 
d f n  d n w 
2w
2w
2w
----------------------- = --------------  --------------- =  – sin n   2e  =  – sin e   2e 
dn
dw
dw
Application to Gradient Calculation
m
n i
 F̂
F̂
---------m--- = --------m-  ----------m
 n i wi j
w i j
m
F̂
F̂ n
--------- = ---------  -------i-m
m
m
b i
 n i b i
14
Gradient Calculation
m
ni =
S
m –1

m
m– 1
wi j a j
m
+ bi
j=1
m
m
ni
m –1
------------ = a j
m
 wi j
n i
--------- = 1
m
b i
Sensitivity
 F̂
m
s i  --------mni
Gradient
 F̂
m m–1
---------m--- = s i a j
w i j
F̂
m
-------m-- = s i
b i
15
Steepest Descent
m
m
m m–1
m
wi j k + 1  = wi j k  – s i a j
m
m
m
W  k + 1 = W  k  – s a
m
m
b i k + 1  = bi k  – s i
m–1 T
m
m
m
b k + 1 = b k  – s

F̂
--------m
n 1
F̂
--------mn 2

F̂
- =
s  --------m
n
m
F̂
---------m
n m
S
Next Step: Compute the Sensitivities (Backpropagation)
16
Jacobian Matrix
m+1
m+1

m+1
n 1
---------------m
n
m
S
m+1
n 2
---------------m
m+1
m+1



n 2
n 2
m+1
------------------------------
n
m
m
---------------
n 1
n 2
n m
m
S
n
m+1
n m + 1 n m + 1
n m + 1
S
S
S
---------------------------------------------
m
m
m
n 1
n 2
n m
S
m

m+1 m
m + 1
  wi l a l + b i

m
m+1
 ni
l = 1


a
m+1
---------------- = ----------------------------------------------------------- = wi j -------j-m
m
m
n j
n j
n j
m+ 1
m
m
m m
Ý
f  n1 
m
m
FÝ  n  =
0

m +1
m
 f n 
m
fÝ n j  = ------------------j--m
nj
m
S
m
n
m+1 Ý
m
----------------- = W
F
n 
m
n
m
ni
m + 1 f  n j 
m+1 m m
---------------- = wi j --------------------- = wi j fÝ n j 
m
m
n j
nj
0

0
m m
fÝ  n 2  
0
0
0

m+1
n 1
---------------m
n 2

m+1
n 1
---------------m
n 1
m m
Ý
 f  n m 17
S
Backpropagation
(Sensitivities)
m
s
T
  nm + 1 
m m
 F̂
F̂
m + 1 T F̂
Ý
= ---------- =  ------------m-----  --------m-------- = F  n  W
 ---------------+1
m+1
m
 n   n
n
n
s
m
m m
m +1 T m +1
= FÝ (n ) W
 s
The sensitivities are computed by starting at the last layer, and
then propagating backwards through the network to the first layer.
M
s s
M –1
2
1
s s
18
Initialization (Last Layer)
S
M
si
M
2
   tj – a j 
T
ai
F̂
 t – a  t – a
j
=
1
= ---------- = -------------------------------------- = ----------------------------------- = – 2 t i – a i ---------M
M
M
M
n i
ni
n i
ni
M
M
M
a
a i
 f n i 
M M
---------i- = --------- = ---------------------- = fÝ  n i 
M
M
M
n i
n i
n i
M M
M
s i = – 2  t i – a i  fÝ  n i 
s
M
M
M
Ý (n ) t – a
= –2F
19
Summary
Forward Propagation
0
a = p
m+1
a
= f
m+ 1
m+1 m
W
a +b
m+ 1
m = 0 2   M – 1

M
a= a
Backpropagation
s
m
s
M
M
M
Ý (n ) t – a
= –2F
m
T
Ý (nm) W m + 1  s m + 1
= F
m = M – 1   2 1
Weight Update
m
m
m
W  k + 1 = W  k  – s a
m–1 T

m
m
m
b k + 1 = b k  – s
20