1. Assignments and Questions
2. (a) Let there be two classes with probability density functions p1 and
p 2 respectively, where
p1 ( x) = exp(− x)
x>0
= 0 otherwise
and
p2 ( x) = 2 exp(−2 x)
x>0
= 0 otherwise
. Let the
prior probabilities be 0.8 and 0.2 respectively for the two classes. Find the Bayes
decision rule for separating the two classes, and find its probability of
misclassification.
(b) Suppose you are given the following decision rule for the above setup.
Put x in class 1 if x > 0.5. Otherwise, put it in class2.
Find the probability of misclassification of the above decision rule. Check if it
is greater than the probability of misclassification of Bayes decision rule.
3. Let µ1' = (0.0, 0.0), µ 2' = (1.5, 0.0), and
− 0.5
 1
1 0
 1 0.5
, ∑2 = 
, and ∑ 3 = 
. Let us
µ 3' = (0.0, 1.5). Let ∑1 = 


1 
− 0.5
0 1 
0.5 1 
consider the following eight cases.
(a) Two classes, pdfs are N ( µ1 , ∑1 ), N ( µ 2 , ∑1 ) and the prior probabilities are
0.5 and 0.5 respectively for the two classes.
(b) Two classes, pdfs are N ( µ1 , ∑1 ), N ( µ 2 , ∑1 ) and the prior probabilities are
0.6 and 0.4 respectively for the two classes.
(c) Two classes, pdfs are N ( µ1 , ∑1 ), N ( µ 2 , ∑ 2 ) and the prior probabilities are
0.5 and 0.5 respectively for the two classes.
(d) Two classes, pdfs are N ( µ1 , ∑1 ), N ( µ 2 , ∑ 2 ) and the prior probabilities are
0.6 and 0.4 respectively for the two classes.
(e) Three classes, pdfs are N ( µ1 , ∑1 ), N ( µ 2 , ∑ 2 ), N ( µ 3 , ∑ 3 ) and the prior
probabilities are (1/3) for each class.
(f) Three classes, pdfs are N ( µ1 , ∑1 ), N ( µ 2 , ∑ 2 ), N ( µ 3 , ∑ 3 ) and the prior
probabilities are 0.5, 0.3 and 0.2 respectively for the three classes.
(g) Three classes, pdfs are N ( µ1 , ∑1 ), N ( µ 2 , ∑ 2 ), N ( µ 3 , ∑ 3 ) and the prior
probabilities are 0.2, 0.3 and 0.5 respectively for the three classes.
(h) Three classes, pdfs are N ( µ1 , ∑1 ), N ( µ 2 , ∑ 2 ), N ( µ 3 , ∑ 3 ) and the prior
probabilities are 0.3, 0.5 and 0.2 respectively for the three classes.
Obtain the Bayes decision rule for each of the above eight cases.
The different values for the number of points n is 100, 200, 500, 1000, 2000,
5000, 10000 and 15000.
For each of the above eight cases, and for each n , the following steps are to be
executed.
(i) Generate n points from the mixture density function randomly. Note that
you will get the class label too for each point.
(ii) Take the first (n / 2) points as the training set, and the rest as the test set.
(iii) Estimate mean, covariance matrix and prior probability for each class using
the training set. Assume normal distribution for each class, and obtain the Bayes decision rule
with the estimated parameters.
(iv) Create a 512x512 image for each classifier (note that you have one classifier
using actual parameters for each case, and several estimated classifiers) in the following way.
The number of different gray values in the image corresponds the number of classes [For
example, if two classes are present, then the gray values are 0 for class 1, and 255 for class2.
For three classes, the gray values are 0 for class 1, 127 for class 2 and 255 for class 3. The pixel
j − 255 i − 255 t
) , where t denotes transpose. The gray value
,
(i, j ) corresponds to the point (
51
51
for the pixel (i, j ) is the class label with respect to the classifier.
(v) As the number of points increase, check if the classification obtained by the
estimated parameters is going towards the actual classification using the images, visually.
(vi) Obtain the misclassification rate for each test set and for each classifier. Check if
the misclassification rates for decision rule using estimated parameters become close to the
misclassification rate of the Bayes decision rule with the actual parameters.
3. Let there be two classes with prior probabilities 0.4 and 0.6 for the two classes
respectively. Let the pdf of the i-th class be pi where
p1 ( x) = x; 0 < x < 1
p 2 ( x) = (4 x − 4) / 3 ; 1 < x < 1.75
= 2 − x; 1 ≤ x ≤ 2 , and
= 4(3 − x) / 5 ; 1.75 ≤ x ≤ 3 .
= 0 otherwise
= 0 otherwise
(a) Find the Bayes decision rule for the above classification problem. Find also its probability of
misclassification.
(b) Suppose you are given the following decision rule.
Put x in class 1 if x ≤ 1.3. Otherwise, put it in class 2.
Find the probability of misclassification for this decision rule.
2. Assignments and Questions
1. Calculate Mahalanobis distance between two populations when the common dispersion
 1 0.5
matrix ∑ = 
and the means are µ1 = (0.0,0.0) t and µ 2 = (1.0,1.0) t where t

0.5 2 
denotes transpose.
2. Calculate the probability of misclassification for the Bayes decision rule for the two class
classification problem where the probability distributions are N ( µ1 , ∑) and N ( µ 2 , ∑)
where µ1 , µ 2 , ∑ are as stated in the above problem.
3. Use the artificial dataset generated for the previous assignment. Classify every point in
every test set by using the points in the training set with the help of k-NN rule where the
different values of k are 4,5,6,7,8,9, and 10. Assume the distance function to be used as
Euclidean distance. Obtain misclassification rates for every test set and for each k.
Compare these rates with the rates obtained by the Bayes classifiers using estimated
parameters.
4. Use the artificial dataset generated for the previous assignment. Classify every point in
every test set by using the points in the training set with the help minimum distance
classifier. Assume the distance function to be used as Euclidean distance. Use the
estimated means as the means of the classes. Obtain misclassification rates for each test
set. Compare these rates with the rates obtained by the Bayes classifiers using
estimated parameters.
5. Use the datasets in UCI Archive. Take those datasets where the training and test sets are
given. Apply k-NN rule by taking different values of k and find misclassification rates for
each k.
3. Assignments and Questions
1. Write a c-program for k-means algorithm.
2. Give an example of a dataset where the results of k-means algorithm are different from
the clusters satisfying the minimum within cluster distance criterion.
3. Write a c-program for DBSCAN method.
4. Let the covariance matrix for a 4-dimensional random vector ( X 1 , X 2 , X 3 , X 4 ) ' be
2 1 0 0 
1 2 0 0 

 . Find three principal components of the random vector.
0 0 2 1 


0 0 1 1.5
5. Let a1' = (0.0,0.0), a 2' = (0.0,1.0), a3' = (1.0,0.0), a 4' = (1.0,1.0). Let there be two classes,
and let a1 , a 2 belong to class 1, and a3 , a 4 belong to class 2. Let λ = 0.05. Let the initial
straight line under consideration for the perceptron method be y = 0.95 − 0.9 x Apply
perceptron method for the classification of data.
Note: One can find PR problems in many books such as (a) Fukunaga’s book on stat. PR, (b)
Duda and Hart’s book on Pattern Recognition, (c) Duda, Hart and Stork’s book on Pattern
Analysis, (d) Devijver and Kittler’s book on Stat. PR. and many others. Some of the problems
you see here may be found in these books.