Machine Learning
Assignment 4
Due date: 1393/10/13
1- Explain why the perceptron cost function is a continuous piecewise linear function.
2-
A) Consider a 2D space or x1–x2 plane. What is the VC dimension of each of the
shapes of the list below where points inside are labeled as 1’s and those outside as
0’s? Draw an example scenario with minimal number of points where these shapes
would fail to shatter the space.
-
Circle
-
Square
-
Rectangle
-
Triangle
B) What is the VC dimension of an ID3 tree?
3- In ensemble learning with bagging, why is it important to do bootstrap resampling
(sampling with replacement) instead of just sub-sampling (sampling without
replacement).
4- Given a set of data points {𝒙𝒙𝒏𝒏 }, we can define the convex hull to be the set of
all points x given by
𝑿𝑿 = � 𝜶𝜶𝒏𝒏 𝒙𝒙𝒏𝒏
𝒏𝒏
where 𝜶𝜶𝒏𝒏 ≥ 𝟎𝟎 and ∑𝒏𝒏 𝜶𝜶𝒏𝒏 = 𝟏𝟏. Consider a second set of points {𝒚𝒚𝑛𝑛 } together with their
corresponding convex hull. By definition, the two sets of points will be linearly separable if
there exists a vector 𝒘𝒘 and a scalar 𝑤𝑤0 such that 𝒘𝒘𝑇𝑇 𝑿𝑿𝑛𝑛 + 𝑤𝑤0 > 0 for all 𝑋𝑋𝑛𝑛 , and 𝒘𝒘𝑇𝑇 𝒚𝒚𝑛𝑛 + 𝑤𝑤0 <
0 for all 𝒚𝒚𝑛𝑛 .
A) Show that if their convex hulls intersect, the two sets of points cannot be linearly
separable, and conversely that if they are linearly separable, their convex hulls do not
intersect.
B) Show that for the case of two linearly separable classes the hyperplane obtained as the
SVM solution is the same as that bisecting the segment joining two closest points
between the convex hulls of the classes.
Programming assignments
5- Logistic Regression
A) Load the “iris” dataset. It has 3 classes and 4 features. Remove the 1st and 2nd column
of the dataset, and then remove all samples of 1st class. So we have a dataset with 2
features and two classes.
B) Plot the dataset in a 2D figure.
C) Use Logistic Regression in order to classify the dataset.
D) Plot the Logistic Regression Line on the above plot.
E) Remove 1st and 2nd column again. Keep all the samples this time. Use the logistic
regression on the dataset to classify 3 classes. Use the “One-vs.-One” and “One-vs.-All”
methods and compare the results.
F) Keep all of the columns and rows of the data this time and use logistic regression for the
classification.
G) Compare the results and explain them.
6- SVM
Use the SVM classifier with linear and RBF kernels for classifying the iris dataset. You
can use libsvm 1(too easy to use). Determine the approximation of the appropriate value
of parameter C in the soft margin SVM(using both RBF and linear kernels). Use “One-
vs.-All” method for the classifying and use the test data for validation of the parameter
C.
In the case you have any questions, please do not hesitate to contact [email protected].
Please send your answers to this email address.
Good luck
1
http://www.csie.ntu.edu.tw/~cjlin/libsvm/