Nanjing University of Science & Technology
Pattern Recognition:
Statistical and Neural
Lonnie C. Ludeman
Lecture 28
Nov 9, 2005
1
Lecture 28 Topics
1. Review Clustering Methods
2. Agglomerative Hierarchical Clustering
Example
3. Introduction to Fuzzy Sets
4. Fuzzy Partitions
5. Define Hard and soft clusters
2
K-Means Clustering Algorithm: Basic Procedure
Randomly Select K cluster centers from
Pattern Space
Distribute set of patterns to the cluster
center using minimum distance
Compute new Cluster centers for each
cluster
Continue this process until the cluster
centers do not change or a maximum
number of iterations is reached.
Review
3
Flow Diagram for K-Means Algorithm
4
Review
Iterative Self Organizing Data
Analysis Technique A
ISODATA Algorithm
Performs Clustering of unclassified
quantitative data with an unknown
number of clusters
Similar to K-Means but with ability to
merge and split clusters thus giving
flexibility in number of clusters
Review
5
Hierarchical Clustering Dendrogram
Review
6
Example - Hierarchical Clustering
Given the following data
(a) Perform a
Hierarchical Clustering
of the data
(b) Give the results for
3 clusters.
7
Solution:
Plot of data vectors
8
Calculate distances between each pair of
original data vectors
(10)
S5 U S7(10)
= S5(9)
Data sample x5 and x7 are the closest together thus
we combine them to give the following for 9 clusters
9
S5(9)
Combine Closest Clusters
S5
(10)
U S7
(10)
= S5
(9)
10
Compute distances between new clusters
(9)
(9)
Clusters S8 and S9 are the closest together thus
we combine them to give the following for 8 clusters
11
S8(8)
(8)
S5
Combine Closest Clusters
S8
(9)
U S9
(9)
= S8
(8)
12
Compute distances between new clusters
(8)
(8)
Clusters S5 and S8 are the closest together thus
we combine them to give the following for 7 clusters
13
(8)
S8
(8)
S5
(7)
S5
Combine Closest Clusters
S5(8) U S8(8) = S5(7)
14
Compute distances between new clusters
(7)
Clusters S1 and
(7)
S4 are the closest together thus
we combine them to give the following for 6 clusters
15
(7)
S5
(6)
S1
Combine Closest Clusters
S1
(7)
U S4
(7)
= S1
(6)16
Continuing this process we see the
following combinations of clusters
at the given levels
17
Level 5
18
Level 4
19
Level 3
20
Level 2
21
Level 1
22
Dendogram for Given Example
23
(b) Using the dendrogram determine the
results for just three clusters
From the dendrogram at level 3 we see
the following clusters
(3)
S5 = { 5, 7, 8, 9, 10 )
(3)
(3)
S2 = { 2, 6 }
S1
= { 1,4,3 }
Answer
Cl1 = { x5, x7, x8, x9, x10)
Cl2 = {x2, x6}
Cl3 = { x1, x4, x3}
24
Introduction to Fuzzy Clustering
K-means, Hierarchical, and ISODATA clustering
algorithms are what we call “Hard Clustering”.
The assignment of clusters is a partitioning of the
data into mutually disjoint and exhaustive non
empty sets.
Fuzzy clustering is a relaxation of this property
and provides another way of solving the clustering
problem.
Before we present the Fuzzy Clustering algorithm
we first lay a background by defining Fuzzy sets.
25
Given a set S composed of pattern
vectors as follows
S = { x1, x2, ... , xN}
A proper subset of S is any nonempty
collection of pattern vectors. Examples
follow
A = { x1, x2}
B = { x2, x4, xN}
C = { x4}
D = { x1, x3 , x5, xN-1}
26
Given the following Set
S = { x1, x2, ... , xk, ... , xN}
We can also specify subsets by using the
characteristic function which is defined
on the set S as follows for the subset A
µA(xk) = 1
=0
if xk is in the subset A
if xk is not in the subset A
Characteristic Function for subset A
µA(.): [ µA(x1), µA(x2), ... , µA(xk), ... , (xN ) ]
27
Examples
A = { x1, x2}
µA(xk): [ 1, 1, 0, 0, 0, ... , 0 ]
28
Examples
A = { x1, x2}
µA(xk): [ 1, 1, 0, 0, 0, ... , 0 ]
B = { x2, x4, xN}
µB(xk): [ 0, 1, 0, 1, 0, ... , 1 ]
29
Examples
A = { x1, x2}
µA(xk): [ 1, 1, 0, 0, 0, ... , 0 ]
B = { x2, x4, xN}
µB(xk): [ 0, 1, 0, 1, 0, ... , 1 ]
C = { x4}
µC(xk): [ 0, 0, 0, 1, 0, ... , 0 ]
30
Examples
A = { x1, x2}
µA(xk): [ 1, 1, 0, 0, 0, ... , 0 ]
B = { x2, x4, xN}
µB(xk): [ 0, 1, 0, 1, 0, ... , 1 ]
C = { x4}
µC(xk): [ 0, 0, 0, 1, 0, ... , 0 ]
D = { x1, x3 , x5, xN-1}
µD(xk): [ 1, 0, 1, 0, 1,...,1 , 0 ]
31
Partitions of a set S
Given a set S of NS n-dimensional pattern
vectors:
S = { xj ; j =1, 2, ... , NS }
A partition of S is a set of M subsets of S,
Sk , k=1, 2, ... , M, that satisfy the following
conditions.
Note: Thus Clusters can be specified as a
partition of the pattern space S.
32
Properties of subsets of a Partition
1.
Sk ≠ Φ
Not empty
2.
Sk ∩ Sj ≠ Φ
Pairwise disjoint
K
∩
3.
Sk = S
k=1
Exhaustive
where Φ is the Null Set
33
Partition in terms of Characteristic Functions
µS1: [ µS (x1), µS1(x2), ... , µS1(xk), ... , µS (xN ) ]
1
1
µS : [ µS (x1), µS (x2), ... , µS (xk), ... , µS 2(xN ) ]
2
2
2
...
...
...
2
M
M
µSM: [ µS (x1), µS (x2), ... , µS (xk), ... , µS (xN ) ]
M
Sum = 1
for each
column
M
µS j(xk) = 0 or 1
for all k and j
34
Cl2
Cl3
Cl1
x1 x 2 x3 x4 x5 x6 x 7
Cl1: [ 1 0 1 0 0 0 0 ]
Hard
Cl2: [ 0 1 0 0 0 0 0 ]
Partition
Cl3: [ 0 0 0 1 1 1 1 ]
35
A Fuzzy Set can be defined by extending
the concept of the characteristic function to
allow positive values between and
including 0 and 1 as follows
Given a set S
= { x1, x2, ... , xN}
A Fuzzy Subset F, of a set S, is defined
by its membership function
F: [ µF(x1), µF(x2), ... , µF(xk), ... , µF(xN ) ]
µ
(x
)
<
1
where xk is from S and 0 <
F
k
=
=
36
S
Function defined on S
37
Example:
Define a Fuzzy set A by the following
membership function
Or equivalently
38
A Fuzzy Partition F, of a set S, is
defined by its membership functions
for the fuzzy sets Fk : k =1, 2, ... , K
)]
Fuzzy
Partition
39
where
Each value bounded by 0 and 1
Sum of each columns values =1
Sum of each row less than n
40
“Hard” or Crisp Clusters
P1
x5
x1
x3
x2
x4
P2
x6
Set Descriptions
P1 = {x1 , x2 , x3 } P2 = {x4 , x5 , x6 }
41
“Soft” or Fuzzy Clusters
x5
x1
x3
x2
Pattern Vectors
x4
x6
Membership Functions
42
Hard or Crisp Partition
Soft or Fuzzy Partition
43
Membership Function for F1
Domain
Pattern
Vectors
Membership Function for F2
Domain
Pattern
Vectors
44
Membership Functions for Fuzzy Clusters
Note: Sum of columns = 1
45
“Fuzzy” Kitten
46
Lecture 28 Topics
1. Reviewed Clustering Methods
2. Gave an Example using Agglomerative
Hierarchical Clustering
3. Introduced Fuzzy Sets
4. Described Crisp and Fuzzy Partitions
5. Defined Hard and soft clusters
47
End of Lecture 28
48