Neural networks
Feedforward neural network - activation function
September
Abstract6, 2012
[email protected]
September
6, 2012 6
iversité de Sherbrooke
September
6,
2012
Septe
ARTIFICIAL
NEURON
Math for my slides “Feedforward
neural network”.
Abst
[email protected]
September
6, 2012
Abstract
2
Abstract
P
Math for my slides
“Feedforward
neu
Topics: connection weights, bias,>activation function
Abstract
x) = bMath
+ for
w
x
=
b
+
w
x
my
slides
“Feedforward
neural
network”.
i
i
Abstract Abstrac
i
Math
for
my
slides
“Feedforward
neural
network”.
• Neuron
September
6,
2012
P
pre-activation (or input activation):
>
Math
for my
slides
“Feedforward
neural
network”.
P
Abstract
P
Math
for
my
slides
“Feedforward
neural
network”.
•
a(x)
=
b
+
w
x
=
b
+
w
x
i
i
P
i “Feedforward neural n
>
Math
for
my
slides
(x)
=
g(a(x))
=
g(b
+
w
x
)
>
•• a(x)
=
b+
+ iwwi ixxi i==b bi++wiwix x P
Math for my slides “Feedforwar
a(x)
=
b
P
i
P
Math for my slides
“Feedforward neural network”.
>>
•P
a(x)
= b=
+ b•+h(x)
=
b
+
w
x
P
P• a(x)
wi=
x
=
b
+
w
x
g(a(x))
=
g(b + > i wi xi )
i wiix
i i
P
P
>
h(x)
=
g(a(x))
=g(b
g(b++ i>w
w•xiix)a(x)
x
b
w
w
=
b
+
w
x
=
b
+
w
x
i)
h(x)
=
g(a(x))
=
1 ••
d
1
d
i
i
P
i
i
•
a(x)
=
b
+
w
x
=
b
+
w
P
i
i
i
•
a(x)
=
b
+
w
x
=
b
+
w
x
i
Abstract
i i
• Neuron (output)i activation
• h(x)
= g(a(x))
=
g(b
+
w
x
)
• h(x)
= g(a(x))
=
g(b
+
w
x
i
i
i
i )P
i
i
•
x
x
b
w
w
1
d
1
d
P
P
••• xxh(x)
b w
w11 =wwdg(b
x=
11 x
d g(a(x))
d +
•
h(x)
=
g(a(x))
=
g(b
+
w
x
)
•
h(x)
=
g(a(x))
=
g(b
+
w
x
)
i
i
i
i
1
i
dforward neural network”. i
i wi
• x1• xx1d xbd wb 1 ww
1 d wd
• w
w
••• w
x1 xd b w 1 w d
•
x
x
•
x
x
1
d
1
d
•
w
>
•
w
+• w arex the connection weights
•
{
••• {{w
•
w
•
{
•
w
•
P
• {
·) b is the neuron bias
•
g(·)
b
w
x
)
i
i
{ isbbcalled the activation function
g(·)
••••ig(·)
•
g(·)
b
• {
• g(·) •b {
(x) = g(a(x))
...
3
ACTIVATION FUNCTION
Math for my slides “Feedforward neural n
Topics: linear activation function
• Performs
squashing
• Not
no input
very interesting...
Abstract
• a(x) = b +
P
>
w i xi = b + w x
P
• h(x) = g(a(x)) = g(b + i wi xi )
i
• x1 xd b w 1 w d
• w
• {
• g(a) = a
1
Abstract
ACTIVATION
Math for myFUNCTION
slides “Feedforward neural network”.
P
>
Topics: sigmoid activation•function
a(x) = b + i wi xi = b + w x
• Squashes the neuron’s
P
pre-activation between • h(x) = g(a(x)) = g(b + i wi xi )
0 and 1
• Always
positive
• Bounded
• Strictly
increasing
• x1 xd b w 1 w d
• w
• {
• g(a) = a
• g(a) = sigm(a) =
1
1+exp( a)
exp(a) exp( a)
exp(2a) 1
4
Math for my slides “Feedforward neural network”.
P
ACTIVATION
FUNCTION
• a(x) = b +
w x =b+w x
i
>
i i
P function
Topics: hyperbolic
tangent
(‘‘tanh’’)
activation
• h(x) = g(a(x)) = g(b +
wx)
• Squashes
the neuron’s
• xbetween
1 xd b w 1 w d
pre-activation
-1 and 1
• w
• Can be positive or
negative • {
i
i i
• Bounded
• g(a) = a
• Strictly increasing
• g(a) = sigm(a) =
• g(a) = tanh(a) =
1
1+exp( a)
exp(a) exp( a)
exp(a)+exp( a)
=
exp(2a) 1
exp(2a)+1
5
6
ACTIVATION FUNCTION
• x1 xd b w 1 w d
• w
Topics: rectified linear activation function
below by 0 • {
(always non-negative)
• Bounded
• Not
upper bounded
• Strictly
• Tends
increasing
• g(a) = a
• g(a) = sigm(a) =
to give neurons
with sparse activities • g(a) = tanh(a) =
1
1+exp( a)
exp(a) exp( a)
exp(a)+exp( a)
• g(a) = max(0, a)
• g(a) = reclin(a) = max(0, a)
=
exp(2a) 1
exp(2a)+1