Fuzzy Classification of Human Skin Color in Color Images
I. A. G. Boaventura, V. M. Volpe, I. N. da Silva, A. Gonzaga
Abstract— In this paper a fuzzy approach for the classification
of skin color tones in color images is presented. The paper is
divided into two stages. The first stage consists of the selection
of images that contained human faces of different skin color
tones. A subset made up of these images was submitted to the
opinion of a group of people with the aim of classifying them
into their respective skin color tones: Black, Brown, and White.
In the second stage of the paper, the information obtained from
the research carried out, jointly with the study of the colors and
their defining tones in relation to the RGB color system, were
used for the definition of the fuzzy sets as well as the inference
rules implemented into the system. In this manner, the
developed system is able to classify a determined color into a
possible skin color.
I. INTRODUCTION
P
attern recognition is a classic problem in computer vision
with various practical applications associated to it. Color
images increase the complexity of pattern recognition, once
external factors, such as light, shade, etc, have harmed the
standard recognition in an image.
The automatic localization of facial regions is an important
initial process for human face detection or recognition
systems. The detection of human faces in uncontrolled
environments is a complex problem, as it involves many
variables that influence recognition such as, glasses, beards,
shade and shadows, occlusion, etc. This problem has been a
research topic within various areas of image processing and
visual computation.
A reliable skin segmentation system could minimize the
complexity and increase the performance as well as the
usability of facial image recognition systems. [3].
The objective of the classification of a pixel into "skin
color" is to determine if the color of a pixel is or is not a skin
color. A good "skin color" classifier should take into
consideration all different skin types (White, Brown, black,
yellow, etc) as well as the environmental factors that influence
in the color of an image, such as illumination [4].
Manuscript received March, 15, 2006.
I. A. G. Boaventura is with the Department of Computer Science and
Statistic, University of State of São Paulo, São José do Rio Preto, SP
15054000 Brazil (corresponding author to provide phone:
55-17-3221-2205;
fax:
55-17-3221-2203;
e-mail:
ines@
ibilce.unesp.br).
V. M. Volpe is with UNIRP – Centro Universitário de Rio Preto, São
José do Rio Preto, SP, Brazil (e-mail: [email protected]).
I. N. da Silva is with the Electrical Engineering Department,
University of
São Paulo, São Carlos, SP (e-mail:
[email protected]).
A. Gonzaga is with the Electrical Engineering Department,
University of São Paulo, São Carlos, SP (e-mail: [email protected])
.
The detection of skin is a complex process, with a high
degree of uncertainty attached to it, as well as being subject to
external factors. The definition of skin color, when dealing with
a digital representation, becomes complex as besides
environmental influences connected to the place where the
photo was taken, there also exist different skin characteristics
in relation to a peoples’ geographical location, as well as skin
tone variations depending on the individual’s race. The term
"skin color" is also in itself a subjective idea, especially when
based on the point of view of human interpretation. In this
sense, the idea of "skin color" takes on an imprecise and vague
definition, thus making the use of fuzzy logic an appropriate
modeling tool. This paper has as objective to classify pixel
colors into "skin color" tones. The images should be separated
into regions where color is the separating characteristic. These
regions will be classified where possible as "skin color" and, if
there is positive confirmation a skin color tone is applied to
that region.
II. PROBLEM DEFINITION
The problem when mapping a skin color consists of
identifying a continuous irregular distribution (mathematically
complex). A skin color captured by the human eye or by some
type of photosensitive equipment, depends strongly on the
quantity of illumination.
Therefore, the problem being dealt with is to carry out this
mapping through the use of a fuzzy system to verify if
determined pixel regions meet on the color strip that represents
skin colors and, besides, this supply a classification of skin
tones for these regions
In working with colors in a general sense, there exist various
questions that need to be taken into consideration. In a natural
scene, the colors of the objects and the illumination are neither
restricted nor controlled. Texture deformations such as shade
and shadows, occlusions, light variations along with other
problems, make the segmentation of an image difficult. This
difficulty can be better understood by representing the colors
in the image through the use of an adequate representative
space such as RGB, HSV or any other such space. Due to the
nature of the scene’s characteristics, the colors are seen within
these spaces as a "cloud" formation of diverse configurations,
some being lightly scattered while others denser, and some
presenting a large variation in the value of the perceived color.
This is also something that occurs with "skin color".
In relation to the representation of the "skin color" class
within the color spaces, a very incisive recent result made a
statement as to the best choice of color space for the detection
of skin [5]. As a general conclusion, the authors suggest that
the best spaces are YCbCr and RGB when dealing with
separability, besides this, the RGB space is ranked in first or
second place in the lists out of five of the eight performance
measures, or be it, in most of the color space transformations.
Based on this result, this paper establishes the RGB color
space for the mapping of "skin color". The next section shows
in details the steps taken in reaching the adopted solution to
the problem.
III. PROBLEM SOLUTION
In defining fuzzy sets for the classification of "skin color"
an initial database was created, which contained images of
human faces, with a wide variety of skin tones. Images from the
AR database were used [1], together with images collected
from the Internet. Figure 1 illustrates some of the images
contained in the database, which present different "skin color"
tones.
A research was carried out in the form of a questionnaire,
which presented a set of images of human faces to the
participants, who then were asked to classify them in
accordance to their skin color. The participants were
specifically asked not to classify the images in accordance to
race, but only to skin tone. All the images were standardized by
the removal of the hair, so that this information would not
influence the participant’s decision. The initial skin tones
considered were: White, Yellow, Light Brown, Dark Brown and
Black. The answers given in this questionnaire were analyzed
and for each image used in the questionnaire a point system
was elaborated to measure the relevance of each image to the
"skin tone" classes being considered.
Through the use of these images from the questionnaire, as
well as those from the constructed database, a detailed study
was carried out in relation to each of the RGB coordinates of
the pixels that represent skin colors, the coordinate R (Red),
the coordinate G (Green) and the coordinate B (Blue). To
complete this study, samples that contained exclusively skin
were extracted from diverse regions of the face.
In this way, a lot of care was taken, to avoid regions with a
great deal of light, shade from beards, etc on masculine faces.
In this sense one can evaluate with greater precision the value
of the interval that corresponds to each RGB coordinate value
for the diverse skin tones collected. This study involved the
analysis of the maximum, average and minimum of pixels, in
each of the images, in a set of 120 samples of different skin
tones.
With the results obtained from the performed studies, the
fuzzy sets and the inference rules were defined for the fuzzy
system.
With the obtained results in hand, one comes to the
conclusion that the ideal would be to use only the White,
Brown and Black tones, once that the yellow tone has had its
minimum, average and maximum values evaluated towards the
White tone. The Light Brown and Dark Brown tones were
classified under one set of Brown as their minimum, average
and maximum values were all very similar. Therefore, the skin
tone set in this paper can be summarized as White, Brown and
Black. It is important to note that the White, Brown and Black
classification is extremely subjective.
By considering the White, Brown and Black tones, the fuzzy
sets were made from the red, green and blue coordinates of the
images along with the skin tone sets.
For each of the images classified as White a pixel average
was generated for each of the RGB coordinates. With the
average values in hand, the minimum, average and maximum
values were calculated. The same procedure was carried out
for the brown and black tones. These values were then used to
put together the fuzzy sets from each of the RGB coordinates
for each of the classified tones.
Fig. 1. Examples of "skin color" tones.
One can consider that the universes of discourse for this
system are the values that belong to the interval [0, 255], and
which represent the pixel values in RGB, and the quantity of
points generated is 255.
To generate the membership functions associated with the
fuzzy sets, the trapezoidal function was considered for the
Black and White tones, once the minimum pixel values for the
Black images reached close to 30, or be it, near to the least
values of the universe of discourse. For the images classified
as White, the maximum pixel values reached above 230, close
to the highest values of the universe of discourse.
The images classified as Brown, in being a subjective
classification, did not have a set of images that could be
classified as a totally defined set. In all of the images, this tone
set had a variation in the classification between the White,
Brown and Black sets. For this reason, the membership
function used in the fuzzy set was the triangular function.
Therefore, the minimum, average and maximum points of this
set are the values used for the construction of the membership
function. Being that, the minimum and maximum have a
membership degree of zero and the average membership
degree is one.
The fourth set which represents the variable "skin color",
was obtained through the average color of the images. For the
reasons previously mentioned the trapezoid membership
function was used for the Black and White colors and the
triangle membership function for the Brown color.
In figure 2 the membership functions for each of the R, G
and B sets are represented in skin tone as well as the
membership functions for the White, Brown and Black skin
color fuzzy sets.
To generate the inference rules, all the combinations of all
the terms for the three fuzzy sets were made, indicating the
coordinates R, G and B. In this way, 27 inference rules were
implemented. Table I gives examples of some inference rules.
TABLE I
RULES OF INFERENCE
Red
Green
Blue
Skin color
BL
BL
BL
BL
WH
WH
WH
WH
BR
BR
BR
BR
BR
BL
BL
BL
BR
WH
WH
WH
WH
BL
WH
-
BL
WH
BL
-
For understanding of Table I, BL refers to Black, WH to
White and BR to Brown. Some of the combinations do not
produce inference rules, once that, after running the analysis, a
skin color is not observed when one has for example, very high
values towards the Red and Blue, representing White skin and
very low values towards Green representing Black skin. When
this occurs, the hyphen represents the non-existence of an
inference rule.
With the intention that the inference procedure manipulates
the fuzzy output actions, the Mandani implication operator was
applied as well as the respective inference rules.
To determine the fuzzy region in relation to all activated
rules, an aggregated operator Max-Min was used, which was
thus applied to the three-defuzzification methods: the Central
Area Method (CDA), the average of maximums Method
(MDM) and the First Maximum Method (MPM). Three
defuzzification methods were applied to see which presented
the best results to the problem in question.
Figure 2: Fuzzy sets
IV. COMPUTATIONAL RESULTS
The developed system was executed using a base
containing 120 "skin color" sample images, with 30 samples
being drawn from images classified as White, 60 samples
drawn from images classed as Brown, and 30 samples from
those classified as Black.
Table II brings together the obtained results, from the three
defuzzification methods.
TABLE II
HIT RATE
Hit Percentage
CDA
MPM
MDM
66,6
70,8
76,6
The highest incidence of errors was found in the Brown
"skin color" set.
Table III shows the obtained results for the application of
each of the three-defuzzification methods, for each of the skin
tones.
TABLE III
ERROR RATE
Error Percentage
Skin color
CDA
MPM
MDM
WH
20,0
13,3
13,3
BR
30,0
46,6
28,3
BL
16,6
0,0
13,3
Table IV shows the quantity of images classified incorrectly
after applying each of the three-defuzzification methods.
For the 30 White tone sample images, 6 of the images were
classified incorrectly. For the 60 Brown samples, 34 were
classified incorrectly and for the 30 Black samples, 5 were
classified incorrectly. It must be emphasized that the numbers
shown represent the total errors of the set for the "skin color"
classification, where there exists an overlapping of errors
between the methods.
TABLE IV
ERRORS IN ABSOLUTE NUMBERS
Image Quantity
Skin color
CDA
MPM
MDM
WH
6
4
4
BR
18
28
17
BL
5
0
4
V. RESULT A NALYSIS
Through the analysis of Table II one observes that the
MDM method showed itself as being the most efficient for the
classification of "skin color". One also notices through Table
III, that this method produces good results for the three tone
sets. The errors found in this method for "skin color" are not
greater than those found in the other two methods.
The CDA method showed itself to be the least efficient for
the classification of "skin color". But for the classification of
the Brown skin tone this method produced results close to
those of the MDM method. For the classification of White and
Black, the CDA method classified incorrectly all the images
classified as wrong, as seen in Table IV. Once this method
calculates the average area of the resulting set from the
aggregation of the active sets, the method tends to
concentrate its results on areas with a greater degree of
incidence in the Brown tone.
The MPM method is less efficient for the Brown "skin color"
classification and more efficient for the Black "skin color"
classification. This occurs as the method uses the value of the
first set maximum which results from the aggregation of the
active sets, which corresponds to the least intensity of the
RGB color on the pixel, increasing thus the probability of a hit
in the fuzzy set for the Black tone.
VI. CONCLUSIONS
When one considers that the classification of "skin color"
tones is extremely subjective from the human interpretation
point of view and that some of the images used came from
uncontrolled environments, with excesses of light and shade,
the system showed itself to be extremely efficient, with a hit
rate above 70%.
The method was not tested for images which did not
represent skin tones, and which should be done to truly check
its efficiency.
In the future, this work should be extended in such a way as
to the development of an algorithm that is able to localize
human skin in images from uncontrolled environments. This
localization would be carried out by the use of an algorithm
developed for the purpose of sweeping images, while looking
for regions that fit the conditions of the fuzzy sets described. If
human skin is localized on the images submitted to the fuzzy
system, this region would be sent to a locally connected neural
network MLP. This image would then be classified as face or
non-face material by the neural network MLP [2].
REFERENCES
[1] A. M. Martinez and R. Benavente, “The AR Face
Database”, CVC Technical Report, no. 24, June 1998.
[2] I. A.G Boaventura, V. M. Volpe, A. L. V. Sanches, A.
Gonzaga, “A face detector Using neural networks and
discrete wavelet transforms”, Anais do SIBGRAPI, 2005.
[3] J. Lu, G. Gu, K. N. Plataniots, J. A. Wang, “Comparative
study of skin color models”, Proceedings of
International Conference on Image Analysis and
Recognition (ICIAR), Toronto, September 28-30, 2005.
[4] S. P. Pung, A. Bouzerdoum, D. Chai, “Skin segmentation
using color pixel classification: analysis and
comparison”, IEEE Transactions on Pattern Analysis
and Machine Intelligence, vol. 27, no. 1, pp. 148-154,
2005.
[5] M. C. Shin,; K. I. Chang, L. V. TSAP, “Does color space
transformation make any difference on skin detection?”,
Proceedings of the IEEE Workshop on Applications of
Computer, USA, 2002.