KTH ROYAL INSTITUTE
OF TECHNOLOGY
Machine learning IV
Artificial Neural Networks
On the Shoulder of Giants
Much of the material in this slide set is based upon:
- ”Automated Learning techniques in Power Systems”
by L. Wehenkel, Université Liege
- ”Machine Learning” course by Andrew Ng, Associate
Professor, Stanford University
2
Contents
Repeating from last time
Artificial Neural Networks
3
Power Systems Analysis: An automated learning approach
P(X)
input
output
Source: Automatic Learning techniques in Power Systems, L. Wehenkel
4
Contents
Repeating from last time
Artificial Neural Networks
5
Artificial Neural Networks - Introduction
Inspired by the Human Nerve system
A close resemblance? (perceptron)
What should be hypothesis ℎθ 𝑥 ?
Source: Automatic Learning techniques in Power Systems, L. Wehenkel
6
Regression VS classification
Regression (linear and polynomial): relationship between the
independent variable and the dependent variable (e.g., predict
continuous price based on size)
Classification: identifying to which of a set of categories a new
observation belongs
Source: ”Machine learning ” course, Andrew Ng
7
Linear regression: bias unit and normalization
-
Hypothesis: ℎθ 𝑥 = θ0 𝑥0 + θ1 𝑥1 + θ2 𝑥2 + θ3 𝑥3 = θ𝑇 𝑥
Linear Perceptron provides output as sum of weighted inputs
-
Normalizing input data 𝑥
Source: ”Machine learning ” course, Andrew Ng
8
Linear regression: Over/under fitting
- In over fitting, if we have too many features, the learned
hypothesis may fit the training set very well, but fail to generalize
to new examples (predict prices on new examples).
Source: ”Machine learning ” course, Andrew Ng
9
Classification (logistic regression)
-
Hypothesis: ℎθ 𝑥 = 𝑔 θ𝑇 𝑥 ,
𝑧
-
𝑔 𝑧 =
1
1+𝑒 −𝑧
(Sigmoidal function)
Non-linear perceptron normally uses a threhold function for the
output, to limit the extreme values.
Source: ”Machine learning ” course, Andrew Ng
10
Over/under fitting in classification:
- In the next lecture, we will discuss how to cope with the over fitting
problem (e.g., by using Regularization technique).
Source: ”Machine learning ” course, Andrew Ng
11
Example (one-layer perceptron): AND function
Hint: in the next lecture we learn how to tune the weights optimally
Source: ”Machine learning ” course, Andrew Ng
12
Example (one-layer network): OR function
Hint: in the next lecture we learn how to tune the weights optimally
Source: ”Machine learning ” course, Andrew Ng
13
Example (multi-layer perceptron): XNOR function
-
Hypothesis: ℎθ 𝑥 = 𝑔 θ𝑇 𝑥 ,
𝑧
𝑔 𝑧 =
1
1+𝑒 −𝑧
(Sigmoidal function)
Source: ”Machine learning ” course, Andrew Ng
14
Multi Layer Perceptrons (MLP)
-
A network of interconnected Perceptrons in several layers
First layer recives input, forwards to second layer etc.
Normally one hidden layer is sufficient to create good mappings
Source: Automatic Learning techniques in Power Systems, L. Wehenkel
15
Multi-class (more than two) classification
Source: ”Machine learning ” course, Andrew Ng
16
Example: One Machine Infinite Bus (OMIB) system
-
We randomly sample values for Pu and Qu creating a database with 5000 samples (objects) and
for each object we have a set of attributes (Pu,
Qu, V1, P1, Vinf, Xinf, CCT).
Source: Automatic Learning techniques in Power Systems, L. Wehenkel
17
Design ANN for the OMIB problem
-
Perceptrons use linear combination of
inputs and tanh function
-
We want to calculate the clearing time
(CCT), i.e. This is a Regression
problem
Source: Automatic Learning techniques in Power Systems, L. Wehenkel
18
ANN structure and tuned weights
-
After tuning the weights
Source: Automatic Learning techniques in Power Systems, L. Wehenkel
19
Summury
-
Perceptrons used for prediction and classification
-
Hypothesis ℎθ 𝑥 : Linear function and Sigmoidal function are common
choices used for prediction and classification, respectively
-
Multi layer perceptrons (MLP) uses hidden layers to
solve more complicated problems
- In the next lecture:
-
How to tune the weights? (e.g., using back propagation)
How to cope with over/under fitting? (e.g., usingregularization)
20
Assignment
ANN exercise session
In this exercise session, you will read an external file with Iris flowers and create an internal database in Java as it was
done in previous exercise session. A new file contains list of 150 observations of iris flowers from 3 different species –
iris-setosa, iris-versicolor and iris-virginica. There are 4 measurements of given flowers: sepal length, sepal width, petal
length and petal width, all in the same unit of centimetres. In this session you need to implement an Artificial Neural
Network (for which the network weights are given) to define specie of iris flower based on dimensional measurements.
Exercise instructions:
1) Use the code developed in Java exercise session VI to create an internal database FlowerList (ArrayList of Flower
objects) from csv file.
2) Assign a three number array to each flower to denote their type. For example, the flower Set ose can be represented
by (1,0,0).
3) Normalize the data. (𝑥𝑛𝑜𝑟𝑚 = (𝑥𝑖 − min)/(𝑚𝑎𝑥 − 𝑚𝑖𝑛))
4) Develop ANN algorithm to define a specie of an iris flower based on dimensional parameters:
- Create classes Layers, Node and Neurons
- Create an ANN with three layers. Input, hidden and Output.
- Create 4 nodes and 12 neurons in the input layer.
- Create 3 nodes and 9 neurons in the hidden layer.
- Create 3 nodes in the output layer.
- Write feedforward algorithms based on the given weights for the ANN
5) Test the developed ANN algorithm on all the 150 observations and find the accuracy of the algorithm in classifying the
flowers.
21