Image Indexing using Weighted Color Histogram
Du-Sik Parkyx , Jong-Seung Parkz , TaeYong Kimx and Joon Hee Hanx
y
Signal Processing Lab., Samsung A.I.T., Yongin, Korea
Human Computing Research Dept., ETRI, Taejon, Korea
x
Department of Computer Science and Engineering, POSTECH
San 31, Hyo-Ja-Dong, Pohang, 790-784, Korea
fpds@, parkjs@falcon., kimty@hawk., [email protected]
z
Abstract
Image indexing is the process of image retrieval from
databases of images or videos based on their contents.
Specifically, histogram-based algorithms are considered
to be effective for color image indexing. We suggest a
new method of color space quantization in the CIELUV
color space, named weighted LUV quantization. With this
method, each bin in the LUV space has a different weighting
factor, which is applied to the histogram intersection.
The weighted LUV histogram intersection provides the
advantage of perceptual uniformity of the CIELUV color
space. An additional advantage is the consideration of
perceptual sensitivity to more saturated colors by the use
of a weighting factor.
with each axis. Each bin in the LUV space has a different
weighting factor according to the volume of the CIELUV
color space included within it. We denote this quantization
scheme as weighted LUV quantization. The weighting
factor for each bin is applied to the calculation of the
histogram intersection to measure the similarity between a
query image and database images. This weighted histogram
comparison algorithm is similar to those of Swain and
Ballard [7], Stricker and Swain [6], and Funt and Finlayson
[3] except the concept of weight. Using this weighting
scheme, we remove a number of bins that are almost useless
for discriminating images in the view of perceptivity. We
show the plausible performance of our algorithm for image
indexing through experiments.
2. Image indexing
1. Introduction
The image indexing process must satisfy the automated
extraction of features, efficient indexing and the effective
retrieval of images within a database. Color features have
proven to be efficient in discriminating between relevant
and non-relevant images. Moreover, histogram-based
techniques have been widely studied and are considered
to be effective for color image indexing [9]. The key
issue of histogram-based techniques is the selection of an
appropriate color space and the quantization of the selected
color space.
In this paper, we suggest a quantization and histogram
matching algorithm in the CIELUV color space. In the
proposed quantization, the LUV space, surrounding all of
the CIELUV color space defined by conversion of RGB
color space, is subdivided into uniform sized bins along
This work was supported in part by the Automation Research Center
(ARC) designated by KOSEF.
In this section, we review the histogram comparisons
and the quantization of the selected color space for image
indexing.
2.1. Histogram-based indexing
The advantage of using color histograms is their
robustness with respect to geometric changes of projected
objects. Histograms are invariant to translation and rotation
around the viewing axis and vary slowly with changes of
view angle, scale, and occlusion. The colors in an image
are mapped into a discrete color space containing n colors.
A histogram of image I is an n-dimensional vector, where
each element represents the number of pixels of color j in
image I . Each element of a histogram is normalized so that
the histogram represents the image without regard to the
image size. The element of the normalized histogram H (I )
3. Color spaces
is defined as
Hj (I ) = H~ j (I )=
X~
n
j =1
Hj (I ) ;
(1)
~ j (I ) is the number of pixels of color j in image
where H
I . We denote d(H (I ); H (I 0 )) as the distance between two
histograms.
Swain and Ballard [7] introduced a histogram matching
method called histogram intersection. Given a pair of
histograms, H (I ) and H (I 0 ), of image I and image I 0 ,
respectively, each containing n bins, they defined the
histogram intersection as follows:
Pn min(H~ (I ); H~ (I 0))
j
0
~
~
:
H (I ) \ H (I ) = j Pn j~
H (I )
=1
j =1
(2)
j
The denominator normalizes the histogram intersection and
makes the value of the histogram intersection between 0 and
~ (I ) \ H~ (I 0 ) is analogous to the L1 -norm,
1. The measure H
dL1 (H~ (I ); H~ (I 0 )) =
n
X
j =1
jHj (I ) ? Hj (I 0 )j ;
(3)
as defined by Swain and Ballard [7], Stricker and Swain
[6], and Funt and Finlayson [3]. For a given distance T ,
two histograms are said to be similar if their distance is less
than or equal to T , and an image in a database is retrieved
in response to the given query image.
The word “color” may be interpreted in several ways: a
certain kind of light, its effect on the human eye, or most
important of all, the result of this effect in the mind of
the viewer [2]. Color is the perceptual result of visible
light, which lies within the range of approximately 380nm
- 750nm of the spectrum wavelength, incident upon the
retina.
3.1. CIE XYZ color space
The assumption that there are three types of cone
receptors in the retina is widely accepted, so three
components are necessary and sufficient to describe a
color. Accordingly, the set of all perceivable colors can be
represented within a three-dimensional space.
The axes of a color space, called primary colors, can
be chosen arbitrarily. A convenient set, universally used
for color measurement, is the CIE 1931 (X,Y,Z)-system
0
adopted by the Commission Internationale de l Éclairage
(CIE) [10]. Each distinct point in the CIE XYZ space
corresponds to a unique color perception. In this space,
the pure color component in the absence of brightness,
such as hue and chroma, can be represented with x and y
chromaticity coordinates defined by
x = X + XY + Z ; y = X + YY + Z :
(4)
2.2. Quantization
3.2. RGB color space
The histogram dimension (the number of histogram bins)
is determined by a color representation scheme and
quantization level. Most color spaces represent a color as
a 3D vector with real values (e.g. RGB, CIE XYZ, HSV,
CIELUV). We quantize the color space of three axes into k
bins for the first axis, l bins for the second axis and m bins
for the third axis. The histogram can be represented as an
n-dimensional vector where n = k l m.
In general high resolution schemes, the histogram of
RGB color space with [255; 255; 255] range of three axes is
represented as a 224 -dimensional vector, HSV color space
with [360; 100; 100] range of three axes as a 3600000dimensional vector, CIELUV space with [100; 354; 262]
range of three axes as a 9274800-dimensional vector. These
high resolution representations, however, are unnecessarily
large for image indexing. Because the retrieval performance
is saturated when the number of bins is increased beyond
some value, normalized color histogram difference was
a satisfactory measure of frame dissimilarity, even when
colors were quantized into only 128 bins (8 green by 8 red
by 4 blue) [4, 9].
RGB color space is represented with red (R), green (G),
and blue (B) primaries and is an additive system. An
additive RGB system is specified by the chromaticities of its
primaries and its white point. This system’s extent (gamut)
is given in the (x,y) chromaticity diagram [10] by a triangle
whose vertices are the chromaticities of the primaries.
RGB values in a particular set of primaries can be
transformed to and from CIE XYZ by three-by-three matrix
transform. To transform from RGB into CIE XYZ the
following transform is used [5]:
n
2 X 3 2 0:607 0:174 0:2000 32 R 3
4 Y 5 = 4 0:299 0:587 0:114 54 G 5 :
Z
0:000 0:066
1:116
B
(5)
Because white is normalized to unity, the middle row sums
to unity. To recover the white point, we need to transform
RGB = [1; 1; 1] to CIE XYZ before computing x and y.
The RGB color space provides a simple and fast
computation. However, it is neither the perceptually
uniform space nor the intuitive space.
3.3. HSV color space
The hue (H), saturation (S), and value (V) system is
based on a warped version of the RGB space and is directly
related to intuitive color notions of hue, saturation and
brightness.
The conversion from the RGB color space to the HSV
color space is done through the following equation [1]:
8
>
<
h0 = >
:
?b) if r = max
?
if g = max
4+(r ?g )
if
b = max
(6)
The intuitiveness of the HSV color space is very useful
because we can quantize each axis independently. Wan and
Kuo [9] reported that a color quantization scheme based on
HSV color space performed much better than one based
on RGB color space. Because HSV color space involves
different computations around 60 degree segments of the
hue circle, the visible discontinuities occur in the color
space. HSV color space linearly converted from RGB color
space is also not a perceptually uniform space.
3.4. CIELUV color space
Consider the distance from color C1 = (X1 ; Y1 ; Z1 )
to color C1 + C , and the distance from color
C2 = (X2 ; Y2 ; Z2 ) to color C2 + C , where C =
(X; Y; Z ). Both distances are equal to C , yet
in general they will not be perceived as being equal
[1]. Perceptual uniformity means that the same C at
two different points in the color space makes the equal
perceivable color difference. CIE XYZ, RGB, and HSV
color spaces do not exhibit perceptual uniformity. There are
two perceptually uniform color spaces that are the agreed
standards in CIE. One of the two is CIELUV color space.
The three variables L , u , and v are defined by [10]:
(
= [(L )2 + (u )2 + (v )2 ] 21 ;
Euv
(11)
where L , u and v are the difference in L , u and
v , between two colors, respectively.
(g
2+(b r )
v = max
? min
s = maxmax
(7)
h = h0 60;
where max = MAX (r; g; b), min = MIN (r; g; b),
= max ? min, and (h; s; v) is the corresponding point of
(r; g; b) in the HSV color space. For r; g; b 2 [0 1], the
conversion gives h 2 [0 360] and s; v 2 [0 1].
L =
The tristimulus values Xn ; Yn and Zn are those of the
illuminant, with Yn equal to 1. The total color difference
between two colors in CIELUV color space is
Euv
calculated from:
116( YYn ) 13 ? 16 if
903:3 YYn
Y
Yn
> 0:008856
otherwise
u = 13L(u0 ? u0n ) ;
v = 13L(v0 ? vn0 )
where u0 ; v 0 and u0n ; vn0 are calculated from,
u0 = X +154XY +3Z ; v0 = X +159YY +3Z :
0
un = Xn +154XYnn+3Zn ; vn0 = Xn +159YYnn +3Zn
(8)
(9)
4. Weighted quantization in CIELUV space
In uniform quantization, each axis of the color space
is uniformly subdivided into a prespecified number of
bins and each bin is the same size. The advantage
of uniform splitting method is that it is straightforward
and simple [9]. However, the disadvantage of uniform
quantization for perceptually non-uniform color spaces,
such as RGB and HSV spaces, is that they don’t take into
account the perceptual similarity between different bins. To
overcome this perceptual dissimilarity problem, we select
the CIELUV color space, which is the same color space that
Taycher [8] chose for uniform quantization.
Figure 1 shows RGB space and CIELUV color spaces
surrounded by LUV space. LUV space is represented by
the uniformly quantized lattices as shown in Figure 1(b)
and (c). Each bin in the LUV space includes a different
volume of the CIELUV color space defined by a two step
transformation of the RGB space using Eqs.(5), (8) and (9).
Here, we introduce the weighted quantization method
in LUV space. In this method, each bin in LUV space
has a different weighting factor according to the volume
of CIELUV color space included within it. We denote
this method as weighted LUV quantization. The weighting
factor for each bin is applied to the calculation of the
histogram intersection in the form of
h
i
H~ (I ) \ H~ (I 0 )
Pn W min(H~ (I ); H~ (I 0))
j
j
= j Pj n
;
w
W H~ (I )
=1
j =1
j
j
(12)
where Wj is the weight for the bin j in the LUV space.
We denote this histogram intersection, with subscript w,
H~ (I ) \ H~ (I 0 ) w , as the weighted histogram intersection.
The weight for each bin is determined by the following
procedures:
h
i
1) Transform R, G, and B values at the lattice points
in a uniformly subdivided RGB color space with a
prespecified number into L ; u ; and v values of the
CIELUV color space.
2) Compute the probability as an estimated volume
included in each bin:
(10)
Pi = PnNi N ; i = 1; 2; :::; n;
j =1
j
(13)
(b) Side view (-u’ axis) of the CIELUV color space surrounded by the LUV space.
(a) Uniformly quantized RGB space represented by lattice points.
(c) Top view (+z axis) of the CIELUV color space surrouned by the LUV space.
Figure 1. RGB space and a perspective view of the CIELUV color space surrounded by LUV space.
where Ni is the pixel number in bin i, n is the total
number of bins in the space, and Pi is the probability
that a pixel will fall in bin i.
3) Compute the weight,
8
< 0
Wi = : PnP1i
1
j=1 Pj
if
Pi = 0;
otherwise
(14)
where Wi is the weight for bin i. The weight of a bin, which
includes the smaller volume within it, will be the larger.
That is, the bin including highly saturated colors will get
the larger weight.
Weighted LUV quantization is designed to provide
the advantage of perceptual uniformity in the histogram
similarity comparison. Using this weighting scheme,
bins that do not reflect the perceptual discrimination are
eliminated, and the sensitivity of a highly saturated color
is considered to be important by the weight.
5. Experimental results
Experiments were carried out to present the color
indexing performance of weighted LUV quantization. To
compare the performance, the histogram intersection is
calculated in the HSV space and the LUV space with the
typical uniform quantization and weighted quantization.
In computing the weighting factor, we have examined
sufficient samples for the probability Pi in order to be
proportional to the volume of the CIELUV color space
included within a bin.
Captured frames from “Star TV” and the movie “Deep
Impact” were used as database images. We denote the Star
TV images as the first database set and the Deep Impact
images as the second database set. The first database set
has 47 images and the second database set has 63 images.
A query image was selected from a database set and the
histogram of the image was compared with the histograms
of the rest of the images in the database set.
Figure 2 shows the result when color indexing for the
first database set is applied to the quantization scheme for
the each space. The first two columns in Figure 2 show the
sequence of the original images and the next three columns
display the sequence ranked as the top ten similar images
using three quantization schemes: uniform quantization of
HSV space, typical uniform quantization of LUV space
(LUV), and weighted LUV quantization (LUVw). The
lowermost image of each quantization scheme is the query
image followed by nine matched images in the order of
similarity. The numbers on the bottom of each image
are the sequence number in the original scene and the
value of normalized similarity to the query image (italicized
number).
Table 1 shows the variance (VAR) of the three relevant
images (including query image and the two top ranked
images) and the percentile of the difference (AD(%))
between the average histogram similarity of the three
relevant images and the average histogram similarity of the
seven non-relevant images. Q. Step means the quantization
step used. The adaptive quantization step has the 15 9 9
step for the HSV space and the 7 13 13 step for the LUV
space, which produces the desirable result as described by
Wan and Kuo [9].
Figure 3 shows the result of color indexing for the second
database set. The quantization step used is the Adaptive for
HSV and LUV and the 5 5 5 step for LUVw.
Figure 2. Comparison of the uniform
quantization in HSV space and LUV space for
the first database set.
Figure 3. Comparison of uniform quantization
in the HSV space and the LUV space for the
second database set.
Table 1. Comparison of the variance and the
percentile of the difference of the average
similarity for the first database set.
Q. Step
333
555
Adaptive
Measure
VAR
AD(%)
VAR
AD(%)
VAR
AD(%)
HSV
0.0006
7.60
0.0016
10.27
0.0024
14.64
LUVw
0.0087
39.23
0.0012
60.02
0.0032
57.51
LUV
0.00013
6.29
0.00052
5.20
0.00068
11.01
Table 2. Comparison of variance and the
percentile of the difference of the average
similarity for the second database set.
Q. Step
555
Adaptive
Measure
VAR
AD(%)
VAR
AD(%)
HSV
0.00012
11.08
0.00036
17.98
LUVw
0.0007
33.80
0.0017
34.69
LUV
0.00004
10.84
0.00006
11.12
Table 2 shows the variance (VAR) of the four relevant
images and the percentile of the difference (AD(%)) of
the average histogram similarity between the four relevant
images and the six non-relevant images for the second
database set.
In all quantization steps, the uniform LUV quantization
scheme has the least variance for the relevant images
and the weighted LUV quantization scheme has the
maximum difference of the average similarity between
the relevant images and the non-relevant images. For
image indexing, a large difference of average similarity
and a small variance are useful characteristics for
discriminating between relevant images and non-relevant
images. From the viewpoint of this discussion, the weighted
LUV quantization scheme shows plausible discrimination
performance. Even in the low resolution quantization
step, 5 5 5, this proposed method presents the
best performance in discriminating characteristics, with an
appropriately small variance.
6. Conclusion
To improve color indexing performance, the weighted
LUV quantization scheme has been proposed. For given
database sets, experimental results show that the weighted
LUV quantization method, even for low resolution
quantization, gives better performance than others.
The proposed weighted LUV quantization has a
complete three dimensional subdivision scheme. To speed
up the processing, we can apply a pseudo three dimensional
subdivision scheme in which three axes are quantized
independently of each other. Another interesting issue is
to find the method assigning the weighting factor to special
bins for indexing images having special color, such as flesh
tone. These remain as further investigation issues.
References
[1] J. D. Foley, A. van Dam, S. K. Feiner, and J. F. Hughes.
Computer Graphics: Principles and Practice. AddisonWesley Publishing, 1996.
[2] J. Fred W. Billmeyer and M. Saltzman. Principles of Color
Technology. John Wiley & Sons, 1981.
[3] B. V. Funt and G. D. Finlayson. Color constant color
indexing. PAMI, 17(5):522–529, May 1995.
[4] J. R. Kender and B.-L. Teo. Video scene segmentation via
continuous video colors. In CVPR, pages 367–373, 1998.
[5] W. N. Sproson. Colour science in television and display
systems. Adam Hilger, 1983.
[6] M. Stricker and M. Swain. The capacity of color histogram
indexing. In CVPR94, pages 704–708, 1994.
[7] M. J. Swain and D. H. Ballard. Color indexing. International
Journal of Computer Vision, 7(1):11–32, 1991.
[8] L. Taycher. Image Feature Extraction Subsystem of the
ImageRover WWW Image Search System. Mater thesis,
Boston University, 1997.
[9] X. Wan and C.-C. J. Kuo. Color distribution analysis and
quantization for image retrieval. In SPIE proceedings, vol.
2670, February, 1996.
[10] G. Wyszecki and W. S. Stiles. Color Science, concept and
Methods, Quantitative Data and Formulae. John Wiley &
Sons, 1982.