FUZZY COLOR SIGNATURES
Andres Dorado
Ebroul Izquierdo
Javeriana University
Computer Science Department
[email protected]
Queen Mary, University of London
Electronic Engineering Department
[email protected]
ABSTRACT
With the large and increasing amount of visual information
available in digital libraries and the Web, efficient and robust systems for image retrieval are urgently needed. In
this paper a compact color descriptor scheme and an efficient metric to compare and retrieve images is presented.
An image adaptive color clustering method, called fuzzy
color signature, is proposed. The original image colors are
mapped into a small number of representative colors using a
peaks detection function derived from the color distribution.
Fuzzy color signatures are then used as image descriptors.
To compare image descriptors the Earth Mover’s Distance is
used. The cost associated with the estimation of this metric
was modified by applying fuzzy logic. Several experiments
have been conducted to assess the performance of the proposed technique.
1. INTRODUCTION
With the large and increasing amount of visual data available, systems that depend on image and video retrieval are
facing new challenges in dealing with information overload.
The search technologies employed need to be efficient and
exploit features that make this environment special. Finding an image on the web is extremely difficult. If images
have been manually labelled with an identifying string, e.g.
”city centre”, then the problem may appear to have been finessed. However, the adequacy of such a solution depends
on human interaction, which is expensive and time consuming and therefore infeasible for many applications. Furthermore, such semantic based search is completely subjective
and depends on semantic accuracy in describing the image.
While a human operator could label an image as ”city centre” a second one would prefer the term ”traffic jam”. Much
work on image indexing and retrieval has focused on the
definition of suitable descriptors and the generation of metrics in the descriptor space. Although system efficiency in
terms of speed and computational complexity has been also
subject of research many related problems are still unsolved.
Fast, scalable and accurate image indexing and cataloguing
are fundamental requirements of any user friendly and effective multimedia portals.
The use of low-level visual features to search and retrieve
relevant information from image and video databases has
drawn much attention in the recent years. As a result, several systems have been developed to search through image
databases using color, texture, and shape attributes. Texture and color play an important role in the Human Vision
System (HVS). The first commercial content-based image
search engine using color histograms was QBIC [1], Photobook added texture and shape features. In the VisualSeek,
Netra, and Virage Systems spatial relationships were Included [2]. In the context of distributed databases some image search engines have been also developed. Among others PicToSeek and ImageRover [3] can be stressed. Color
histograms have been improved eliminating luminance’s Effects [4], incorporating semantic cues [5], including spatial
correlation [6], and defining regions of interest [7].
Although color histograms are probably the most important
and widely employed image descriptors, their use requires
some sort of abrupt quantization leading to information loss.
Furthermore, metrics to compare images using histogram
descriptors are based on exact matching without considering
smooth color transitions in natural images. Consequently,
the use of fuzzy logic (FL) to improve efficiency and robustness of search engines appears to be very promising. In
this context the Fuzzy C-Means Algorithm [8] can be cited
as one pioneering work to cluster image features and overcome exact matching and abrupt quantization problems.
In this work a FL based approach is proposed. First, a
”fuzzy” color quantization method is used to map the original color distribution into a small number of relevant image
colors using a suitable membership function. This method
is called Fuzzy Color Signature (FCS). It relies on the detection of peaks in the color distribution function to perform
color quantization. The resulting color signature [9] is a
very compact, efficient and reliable image descriptor highly
suitable for indexing and retrieval in large and distributed
databases. In section 2 a detailed description of this image
descriptor is presented.
To compare images using FCS Earth Mover’s Distance [10]
is used. The cost associated with the estimation of this metric was also modified by applying FL techniques. This is
described in Section 3. In section 4 selected experimental
results are reported and the paper concludes with a summary
in section 5.
2. IMAGE DESCRIPTOR
In [11] the potential of using color as image descriptors
and color histograms for color image indexing was demonstrated. Indeed, color histogram is the most widely used
color descriptor in content based retrieval.
Since color histograms capture the global color distribution
in an image, it is easy to compute. Unfortunately, it results
in large feature vectors and consequently in high computational cost when measuring image similarity. Color clustering is an appropriate method to reduce the complexity of
the feature vectors used as descriptors and consequently the
computational cost of the retrieval process.
Since the HLS color model resembles the human perception, this model is used in this work. The color distribution
C is defined as
C = {{xi , fi }, i = 1, . . . , n}
(1)
where n is the total number of color clusters in the image, xi
is a position in the quantized vector and fP
i is its frequency.
When the distribution is normalized, the i fi = 1 and fi
is the percentage of pixels satisfying the condition of the
feature represented by xi .
The number of clusters n is image dependant, the smaller
n the lower the dimensionality and complexity of the color
distribution. Colors in the original image can be assigned to
its closest cluster using the C-Means, Fuzzy C-Means or a
Peaks Detection Function.
A signature SI is a compact color distribution from an image I, denoted by
SI = {(xi , wi )}ni=1
(2)
where the weight wi gives the strength of color xi in the
image I.
In this work a ”fuzzy” clustering method called Fuzzy Color
Signature is proposed as image descriptor. It is based on the
observation that a small number of colors is usually enough
to characterize the color information in an image.
The color signature consists of the representative colors and
their relative distribution. To identify these colors the following four functions are used.
2.1. Cumulative distribution function
Each image is transformed from RGB to HSL color space.
The cumulative distribution function (cdf ) is defined for the
hue channel only. The S and L channels are quantized uniformly.
The initial hue vector has 360 values, i.e., 1 degree quantization. For each hue value its corresponding frequency fi
is calculated. The cdf function is defined as:


if i ∈ (−∞, 0),
0P
k
cdf (i) =
(3)
f
if
i ∈ [0, . . . , 359],
j=1 j

Pn
if i ∈ [360, ∞).
j=1 fj
where i is a bin in the quantization vector of the hue channel
and fj the frequency of this hue value.
2.2. Average cumulative distribution function
After cdf calculation, the average value acdf of the cdf over
a window of size W is estimated. The width W of the
smoothing window determines the degree of smoothing and
the subsequent sensitivity of peak detection function. The
acdf is calculated as follows:
acdf (i) =
i+W
X
j=i−W
cdf (j)
2W + 1
(4)
where W is a small integer number.
2.3. Peaks detection function
The peaks detection function pdf is the core of the cluster method. It allows to estimate modes and peaks in the
distribution function. Through zero-crossing values identification, the pdf function detect peak’s ranges, i.e., those
positions where peaks start and end. This procedure is close
related to the generation of linear scale-spaces for multiresolution analysis. The pdf is defined as:
pdf (i) = cdf (i) − acdf (i), i ∈ {1, . . . , n}
(5)
2.4. Fuzzy mapping function
Once most relevant hue values have been selected using the
pdf function, the last step consists of mapping each hue
value into the peak’s range to the position of local maximum. To do this, the following a membership criterion is
applied:
X
w(i) =
µPi (j)fj , j ∈ Pi
(6)
where Pi is the range of the peak i and µPi (j) gives the
contribution of value j to the peak’s energy. If position j
coincides with the peak’s mode then j has a total membership and µPi (j) = 1. If j is outside the peak’s range, then
µPi (j) = 0 or total non-membership. The remainder positions within the peak’s range have a partial membership
between 0 and 1. The membership function is defined by
µPi (j) = e
−m
ki−jk
∆Pi
Image
decomposition
(RGB)
Image
Database
Image
Transformation
(HSL)
Fuzzy
Color
Signature
Image
Quantization
Feature Extraction
Similarity
Judging
(EMD)
Similarity Measurement
Fig. 1. Indexing and Retrieval using FCS
, m≥1
(7)
where ∆Pi is the peaks range and m is a measure of fuzzyness.
Applying this clustering method a compact and efficient
color signature or image descriptor is obtained.
database using the proposed technique along with their similarity value are displayed in ascendant order.
1) 0.00
#10871
2) 11.38
#13838
3) 31.68
#16137
5) 44.97
#13837
7) 66.67
#13850
6) 71.98
#10874
4) 78.78
#10877
8) 109.0
#16136
3. COLOR SIMILARITY
Once image descriptors have been defined the crucial task
is to find an appropriate metric in the descriptor space. We
propose to use the Earth Mover’s Distance (EMD) introduced by Rubner et al [10] as metric to measure the distance between color cells or bins. This allows the user to
have different cell sizes so that precision can be kept where
required and a coarser approximation can be used when it is
not necessary. Observe that for a given smoothing window
W the size of image descriptors change according to the
image content. The EMD reflects the minimal amount of
work that must be performed to transform one feature vector into the other by moving the distribution mass around.
Intuitively, given two n–dimensional vectors one of them
can be seen as the mass of earth spread in space and the
other as a collection of wholes in the same space. The proposed metric measures the least amount of work needed to
fill the wholes with earth. The EMD is obtained as follows:
Pn Pm
Cij kpi − qj k
i=1
Pn j=1Pm
EM D(p, q) =
(8)
i=1
j=1 Cij
4. EXPERIMENTAL RESULTS
To assess the performance of the presented approach several
experiments have been conducted. A database containing
several thousands of images has been used. Fig. 1 outlines
the different modules and processes involved in the indexing and retrieval system.
In Fig. 2 the query is shown at the upper-left corner (image #10871). The most similar images extracted from the
Fig. 2. Retrieval results
Fig. 3 gives the results of applying the cdf , acdf and, pdf
functions to image #10871. In this Fig. 3 the vector quantization shows that the highest frequencies are between 250–
359, Table 1 lists the color distribution for few positions
including the highest frequency: h259, 111417i.
xi
wi
12
26209
150
54
165
23
259
111417
Table 1. color signature for the 10871 image
5. CONCLUSION AND FUTURE WORK
In this work, a compact color-based image descriptor is proposed. The descriptor is based on the distribution of representative colors in the image. It determines the introduced
fuzzy color signature. This signature is robust in describ-
6. REFERENCES
Functions for 10871( 1440 colours )
Window size: W=64
[1] M. Flickner, “Query image and video content: Qbic
system,” IEEE Computer, vol. 28, no. 9, 1995.
120000
Frequency
cdf
Acdf
pdf
cdf
[2] I. J. Cox, M. L. Miller, T. P. Minka, T. V. Papathomas,
and P. N. Yianilos, “The bayesian image retrieval
system, pichunter: Theory, implementation, and psychophysical experiment,” IEEE Transactions On Image Processing, vol. 9, no. 1, pp. 20–37, Jan 2000.
100000
Pixels
80000
acdf
[3] T. Gevers and A. W. M. Smeulders, “Pictoseek: Combining color and shape invariant features for image retrieval,” IEEE Transactions On Image Processing, vol.
9, no. 1, pp. 102–119, Jan 2000.
60000
pdf
40000
20000
[4] D. A. Adjeroh and M. C. Lee, “On ratio-based color
indexing,” IEEE Transaction On Image Processing,
vol. 10, no. 1, pp. 36–48, Jan 2001.
f
[5] A. Vailaya, M. A. T. Figueiredo, A. K. Jain, and
H. Zhang, “Image classification for content-based indexing,” IEEE Transactions On Image Processing,
vol. 10, no. 1, pp. 117–130, Jan 2001.
0
-20000
240
260
280
300
320
340
360
bins
Fig. 3. cdf, acdf and pdf functions for the 10871 image
ing color image primitives and can be used for efficient image retrieval in large fixed and distributed databases. It is
estimated using four basic functions: cumulative distribution function (cdf ), average cumulative distribution function (acdf ), peaks detection function (pdf ) and fuzzy mapping function (f mf ). Their calculation is straightforward
and computationally inexpensive.
The conventional Earth Mover’s Distance is modified and
used as similarity measure. The cost calculation of the improved metric is based on a ”fuzzy” membership function
providing better retrieval results without overloading the system.
Regarding the implementation of a complete image Portal, several schemes and user preferences can be considered
using the presented approach. For instance, the user can:
query images with at least 30% blue and 10% red, outline
the query by painting it in a simple icon- or cartoon-like
form or, give an query example and search for similar images (the usual query-by-example paradigm.)
One limitation of the presented system is that it does not
handle spatial relationships of colors in the image. Since the
spatial distribution of colors is also an important source of
information, it will be considered in further developments.
[6] J. Huang, S. R. Kumar, M. Mitra, W. Zhu, and
R. Zabih, “Image indexing using color correlograms,”
Proc. IEEE Conf. Computer Vision Pattern Recognition, pp. 762–768, 1997.
[7] Y. Deng, B. S. Manjunath, C. Kenney, M. S. Moore,
and H. Shin, “An efficient color representation for image retrieval,” IEEE Transactions On Image Processing, vol. 10, no. 1, pp. 140–147, Jan 2001.
[8] N. B. Karayiannis and J. C. Bezdek, “An integrated approach to fuzzy learning vector quantization and fuzzy
c-means clustering,” IEEE Transaction On Fuzzy Systems, vol. 5, no. 4, Nov 1997.
[9] S. Cohen and L. Guibas, “The earth movers distance
under transformation sets,” Proceedings of the 7th
IEEE International Conference on Computer Vision,
Sep 1999.
[10] Y. Rubner, C. Tomasi, and L. J. Guibas, “The earth
mover’s distance as a metric for image retrieval,”
Technical Report STAN-CS-TN-98-86, Stanford University, Computer Science Department, Sep 1998.
[11] M. J. Swain and D. H. Ballard, “Color indexing,” International Journal of Computer Vision, vol. 7, no. 1,
pp. 11–32, 1991.