Artificial Neural Networks
Lab demonstration (2)
Python Modules
• A module is a file containing Python definitions and
statements intended for use in other Python programs.
• There are many Python modules that come with Python as
part of the standard library.
• Once we import the module, we can use things that are
defined inside.
To use elements of a module
• Import the module
• Use the dot to refer to the element of the module
Example: The turtle module
Source: http://interactivepython.org/runestone/static/thinkcspy/PythonModules/modules.html
What modules are available in Python?
• A list of modules that are part of the standard library is available in
Python documentation at: https://docs.python.org/3/pymodindex.html
In your file “network.py”
The random module
• Example applications in which we need to generate random numbers:
• To play a game of chance where the computer needs to throw some dice, pick
a number, or flip a coin,
• To shuffle a deck of playing cards randomly,
• To randomly allow a new enemy spaceship to appear and shoot at you,
• For encrypting your banking session on the Internet.
The random module
The numpy module
• Used to create multidimensional arrays
• In numpy, dimensions of an array are called axes
• The number of axes is called the rank of the array
Example: What are the rank and axes of the following numpy array?
The numpy module
• Used to create multidimensional arrays
• In numpy, dimensions of an array are called axes
• The number of axes is called the rank of the array
Example: What are the rank and axes of the following numpy array?
Rank = 1
1 axis of length 3
How to create a numpy array?
Create a multidimensional numpy array
Initializing the content of an array
Lab Exercise
• Create a 1-dimensional array (1 axis) containing five ones
• Create a 2-dimensional array (2 axes) containing 4 x 5 zeros
• Create a 3-dimensional array (3 axes) containing 4x3x2 ones
Lab Exercise: solution
• Create a 1-dimensional array (1 axis) containing five ones
• Create a 2-dimensional array (2 axes) containing 4 x 5 zeros
• Create a 3-dimensional array (3 axes) containing 4x3x2 ones
Initializing a random array from normal
distribution
Initializing multiple arrays from a normal
distribution
4 arrays: each one is 3x2
Exercise: Generate 3 arrays of random
numbers
• The first array is 3 x 1
• The second array is 5 x 1
• The third array is 2 x 1
Solution
Exercise:
Given a list of layers for a neural network,
generate random bias vectors for each layer
Example for this figure (from mid-term
exam), the bias vectors can be:
𝑏11
0.16548184
=
0.72878268
𝑏21
𝑏12 = 0.7278303
Solution
Specifying a neural network
• Input: a vector of number of neurons in each layer
• The first number in the input vector contains the number of input
variables.
• Ex: [3, 2, 1] ============ >
Initializing biases
Initializing weights
• Sizes = [3,2,1]
• The first weight array is 2x3
1
1
1
𝑤11
𝑤12
𝑤13
• 1
1
1
𝑤21 𝑤22
𝑤23
• The second weight array is 1x2
2
2
• 𝑤11
𝑤12
Initializing weight arrays
Exercise:
• Create a Neural Network Class
• Create an __init__ function for the network class
• Initialize self.biases
• Initialize self.weights
Getting code and data
git clone https://github.com/mnielsen/neural-networks-and-deep-learning.git
For Python 3.4  we need to make some
changes to the mnist_loader
• Open the file mnist_loader.py
• Change cPickle to picke on lines 13 and 43
• On line 43: change the call to
training_data, validation_data, test_data = pickle.load(f,
encoding='latin1')
For Python 3.4  we need to make some
changes to the mnist_loader
• Open the file mnist_loader.py
• Wrap the zip() calls with list() calls
In file network.py
//Add this line
In file network.py
• Find all xrange and replace it with range
The MNIST Data set
• A large number of scanned images of handwritten digits
• Each image is 28 x 28 = 784 pixels
• We need to create a neural network that accepts 784 inputs values
[X1, …. X784]
Load data and create network
Compatibility with Python 3.4
Output
…..
What is this output?
• Recall the algorithm of Least Mean Square:
Calculates error based on 1 input pattern x(n)
Updates weights based on 1 input pattern x(n)
Backpropagation Algorithm
Start with randomly chosen weights [𝒘𝒍𝒋𝒌 ]
While error is unsatisfactory:
Updates weights based on 1 input pattern x(n)
for each input pattern x:
feedforward: for each l = 1, 2,…, L compute 𝒛𝒍𝒌 𝒂𝒏𝒅 𝒂𝒍𝒌
Compute the error at output layer: 𝛿𝑘𝐿 = 𝑑𝑘𝐿 − 𝑎𝑘𝐿 𝜎′(𝑧𝑘𝐿 )
Backpropagate the error: for l = L-1, L-2, … 2
𝑙+1
compute 𝛿𝑘𝑙 = 𝛿𝑘𝑙+1 𝑤𝑘𝑗
𝜎′(𝑧𝑗𝑙 )
Calculate the gradients:
end for
𝜕𝐸
𝑙
𝜕𝑤𝑘𝑗
= 𝜹𝒍𝒋 𝑎𝑘𝑙−1 and
𝜕𝐸
𝜕𝑏𝑗𝑙
= 𝜹𝒍𝒋
end while
© Mai Elshehaly
37
Three strategies to update the weights:
• Update after the network sees every single input pattern
• Update after the network sees a mini_batch of input patterns
• Update after the network sees the entire batch of input patterns
The difference between the three strategies will be discussed in the lecture.
The mini_batch strategy:
Ex.: mini_batch_size = 5
The mini_batch strategy:
1. Input one batch to the network:
2. Adjust weights
3. Move to the next batch
4. Repeat until no more batches in
the training data set
This is one epoch
To increase the accuracy:
‫التكرار يعلم الشطار‬
• Repeat the previous process for a number of epochs
• Don’t input the mini batches in the same order (random.shuffle)
• With each new epoch, you can see that the accuracy increases
Correctly
classified
samples
Total
number of
test samples
To see the effect of parameters on accuracy
• Try passing different values for epochs, mini_batch_size, and
eta
How to implement this shuffling and batching
strategy?
• Example:
•
•
•
•
Say you have a deck of 30 cards with labels 1… 30
You want to take 10 cards in each draw
You want to keep drawing until no more cards
You want to shuffle the cards then repeat 8 times
Shuffling cards
Shuffle for 8 epochs
Explore by 10 cards in each epoch:
Exercise:
• Write a function sum_mini_batches(training_data,
epochs, mini_batch_size) that does the following for
epochs times:
• Shuffles the cards in training_data
• Creates a number of mini batches each of which is of size
mini_batch_size
• Prints the sum of the numbers in each mini batch
Lab Demo: Second Round
Review items
• Numpy’s dot() function
• The weights and biases of ANN
• Zip() function
• Negative indices in Python
• Matrix shape
• The Backpropagation pseudocode
dot() function
zip() function
Example: initializing weights and biases
zip() function
Example: to iterate over layers of weights and biases
Exercise
• Reuse the mini_batches code that we wrote earlier to generate
inputs.
• Iterate over layers of weights and biases to calculate the z values of
different layers. Assume that actual output = net input (a=z) for
simplicity.
• Print z at each iteration.
Negative indices in Python:
Try the following
Solution