Crisp Clustering
Data in RN
C1
C2
Ci  Ø
Ci  Cj = Ø
CK
Fuzzy Clustering
Data in RN
C1
C2
Ci  Ø
Ci  Cj  Ø
CK
Spatial Clustering
Spatial Clustering
Fuzzy C-means

The clustering criterion in FCM is




Jm(U,V) = S S (uik)m Dik2(vi,xk)
U is a fuzzy partition matrix
m is the weighting exponent
V = [v1,v2,…,vc] is the set of cluster
centers (prototype parameters),
Dik : distance of xk from center vi
Crisp vs. Fuzzy Membership

Membership matrix: Uc×n


uik is the grade of membership of sample k with
respect to prototype i
Crisp membership:
uik  1, if || xk  vi ||2  min || xk  v j ||2
j
uik  0, otherwise

Fuzzy membership:
c
u
i 1
ik
 1, k  1,, n
Fuzzy c-means (FCM)

The objective function of FCM is
c
n
c
n
m
2
u
d

u
||
x

v
||
 ik  ik k i
i 1 k 1
m
ik
2
i 1 k 1
FCM (Cnt’d)

Introducing the Lagrange multiplier λ with
respect to the constraint
c
u
j 1
jk
 1,
we rewrite the objective function as:
c


 
m 2
J   uik d ik      u jk   1
 j 1

i 1




c
FCM (Cnt’d)

Setting the partial derivatives to zero, we
obtain

J  c

   u jk   1  0
  j 1 
J
 m  uikm1  d ik2    0
uik
FCM (Cnt’d)

From the 2nd equation, we obtain
   m11

uik  
2 
 m  dik 

From this fact and the 1st equation, we obtain
   m11

1   u jk   
2


j 1
j 1  m  d jk 
C
C

 
m
1
m 1 C
 1 
 2 



j 1  d jk 
1
m 1
FCM (Cnt’d)

Therefore,   
 
m
and
1
m 1
1

 1 
 2 



j 1  d jk 
c
  

uik  
2 
 m  dik 
1
m 1
1
m 1
FCM (Cnt’d)

Together with the 2nd equation, we obtain the
updating rule for uik
uik 
1
 1 
 2 



j 1  d jk 
1
c

 d ik2 
 2 



j 1  d jk 
c
1
m 1
1
m 1
 1  m11
  2 
 d ik 
FCM (Cnt’d)

On the other hand, setting the derivative of k
with respect to vi to zero, we obtain
J

0

vi vi
c
n
m
2
u
||
x

v
||
 ik k i
i 1 k 1

 u
|| xk  vi ||2
vi
k 1
n
m
ik
n
  uikm
k 1
n

(xk  vi )T ( xk  vi )
vi
 2 uikm ( xk  vi )
k 1
FCM (Cnt’d)

It follows that
n
J
  uikm ( xk  vi )  0
vi k 1

Finally, we can obtain the update rule of ci:
n
vi 
m
u
 ik xk
k 1
n
m
u
 ik
k 1
FCM (Cnt’d)

To summarize:
n
uik 
1
 d ik 





j 1  d jk 
c
2 ( m 1)
vi 
m
u
 ik xk
k 1
n
m
u
 ik
k 1
Fuzzy C-Means Clustering
Pros and Cons

Advantages



Unsupervised
Always converges
Disadvantages



Long computational time
Sensitivity to the initial guess (speed, local minima)
Sensitivity to noise
 One expects low (or even no) membership
degree for outliers (noisy points)
Fuzzy C-Means Clustering
Optimal Number of Clusters

Performance Index
c
n

min  P(c)   uikm ( x k  vi
(c)
i 1 k 1


 vi  x ) 

n
1
xk
Average of all feature vectors x 

n k 1
c
n
 u
i 1 k 1
m
ik
( xk  vi )
2
Sum of the
within fuzzy cluster fluctuations
(small value for optimal c)

c
2
2
n
 u
i 1 k 1
m
ik
( vi  x )
2
Sum of the
between fuzzy cluster fluctuations
(big value for optimal c)
Fuzzy-Possibililstic C-Means
Idea



uik is a function of x k and all c centroids
tik is a function of x k and v i alone
Both are important
 To classify a data point, cluster centroid has to
be closest to the data point  Membership
 For Estimating the centroids  Typicality
for alleviating the undesirable effect of outliers
Fuzzy-Possibililstic C-Means
(FPCM), OF and Constraints


Objective function
c
n

m

2 
min  J m, (U, T, V )   (uik  tik ) Dik 
( U ,T, V )
i 1 k 1


Constraints


Membership
Typicality

c
u
i 1
ik
 1 ,k
n
t
k 1
ik
 1 ,i
Because of this constraint, typicality of a data point to a
cluster, will be normalized with respect to the distance of
all n data points from that cluster  next slide
Fuzzy-Possibililstic C-Means
Minimizing OF

Membership values


Typicality values


Same as FCM, but uik
resulted values may
be different
Depends on all data
Cluster centers

m
m
vi    (uik  tik )x k
 k 1
n
tik
 c
 Dik

  

j 1  D jk






 n
 Dik 

  



j 1  Dij 


2
m 1
2
 1

(u  t )  , i

k 1

n
m
ik
m
ik





1





1
, i, k
, i, k
Typical 
in the interval
[3,5]