Pattern Recognition 34 (2001) 2459}2466
Automatic extraction of eye and mouth "elds from a face image
using eigenfeatures and multilayer perceptrons
Yeon-Sik Ryu, Se-Young Oh*
Department of Electrical Engineering, Intelligent Systems Laboratory, Pohang University of Science and Technology (POSTECH),
Pohang, KyungBuk 790-784, South Korea
Received 22 September 1999; received in revised form 18 September 2000; accepted 6 October 2000
Abstract
This paper presents a novel algorithm for the extraction of the eye and mouth (facial features) "elds from 2-D gray-level
face images. The fundamental philosophy is that eigenfeatures, derived from the eigenvalues and eigenvectors of the
binary edge data set constructed from the eye and mouth "elds, are very good features to locate these "elds e$ciently.
The eigenfeatures extracted from the positive and negative training samples of the facial features are used to train
a multilayer perceptron whose output indicates the degree to which a particular image window contains an eye or
a mouth. It turns out that only a small number of frontal faces are su$cient to train the networks. Furthermore, they lend
themselves to good generalization to non-frontal pose and even other people's faces. It has been experimentally veri"ed
that the proposed algorithm is robust against facial size and slight variations of pose. 2001 Pattern Recognition
Society. Published by Elsevier Science Ltd. All rights reserved.
Keywords: Facial feature; Eye and mouth "elds; Eigenfeature; Multilayer perceptron; Positive (Negative) sample
1. Introduction
It is known that viewing a person's eyes and mouth is
essential in understanding the information and feeling
they convey. Because of this, automatic extraction of eyes
and mouths from a person's face can be very useful in
many applications. Many researchers have proposed
methods to "nd the eye and mouth regions [1}3] or to
locate the face region [4}8] in an image. These methods
can be classi"ed by their use of three types of information: template matching, intensity and geometrical
features. In general, template matching requires many
templates to accommodate varying pose whereas the
intensity method requires good lighting conditions.
In his pioneering work, Kanade [5] used a Laplacian
operator to "nd an edge image. Since the Laplacian
operator does not give information on edges, Brunelli [7]
* Corresponding author. Tel.: #82-54-279-2214; fax: #8254-279-2903.
E-mail address: [email protected] (S.-Y. Oh).
used horizontal gradients to detect the head top, eyes,
nose base and mouth. He found the location of eyes using
template matching. However, it requires many templates
to correctly locate the eyes while accommodating varying
pose. Beymer [6] used a "ve-level hierarchical system to
locate the eyes and nose lobes. He used many facial
feature templates covering di!erent poses and di!erent
people. Juell et al. [8] proposed a neural network structure to detect human faces in an image. They used three
child-level neural networks to locate the eyes, nose and
mouth. But they used enhanced gray images as input to
the neural networks, thus requiring too many input neurons and training samples to work well with various
poses.
In this paper, we present a novel algorithm for the
extraction of the eye and mouth "elds from 2-D graylevel face images. The fundamental philosophy is that
eigenfeatures, derived from the eigenvalues and eigenvectors of the binary edge data set constructed from the eye
and mouth "elds, are very good features to locate these
"elds. The eigenfeatures are extracted from the positive
and negative training samples of the facial features and
0031-3203/01/$20.00 2001 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved.
PII: S 0 0 3 1 - 3 2 0 3 ( 0 0 ) 0 0 1 7 3 - 4
2460
Y.-S. Ryu, S.-Y. Oh / Pattern Recognition 34 (2001) 2459}2466
are used to train a multilayer perceptron (MLP) whose
output indicates the degree to which a particular image
window contains the complete portion of eyes or the
mouth within itself. The window template within a test
image is then shifted in image space and the one with the
maximum output for the MLP represents the eye or
mouth "eld. It turns out that only a small number of
frontal faces are su$cient to train the networks since they
lend themselves to good generalization to non-frontal
pose and even other people's faces.
Section 2 describes coarse extraction of the eye and
mouth "elds using geometrical constraints. Section 3
presents "ne extraction of the facial features using
neural networks. Experimental results are reported in
Section 4.
2. Coarse extraction of the eye and mouth 5elds from
geometrical constraints
The facial images used in this research consist of
92;112 pixels and have been obtained from the Olivetti
Research Lab (ORL) database. They consist of frontal
views as well as side views containing both eyes and the
mouth. The basic assumption in the database is that the
eyes and mouth reside around the center of the image
against a uniform background. Fig. 1 shows the overall
system for extraction of the facial features. Fig. 2 shows
some of the original images from which to extract the
three facial features of interest. The "rst "ve images of
Fig. 2(a) were used to extract the eye and mouth regions
while the last was used only to extract the mouth. Note
Fig. 2. The face images used for training and testing the proposed algorithm: (a) the images used to extract the training set
for the MLP detector and (b) the images used to extract the test
set.
that only frontal images were used to train the networks
since otherwise, there would be too many pose variations
to consider. Fig. 2(b) shows a portion of the test set used
to estimate the generalization performance.
2.1. Extraction of the face region
Fig. 1. The system block diagram.
It was decided not to go through a priori normalization and edge enhancement process to make the eye
location and facial size "xed since the locations of the
eyes and mouth all di!er according to the facial size and
pose. Instead, a binary edge dominance map [7] was
used "rst to roughly identify the face region excluding the
hair and ears within which the coarse locations of the
eyes and mouth are to be found. This limits the size of the
coarse regions containing these three features thus easing
the computational burden.
The horizontal and vertical projection information
obtained from the edge map will be used to estimate the
approximate locations of the eyes and mouth [7]. From
the gray image I(i, j), the vertical and horizontal edge
maps I and I are obtained "rst. These are the binary
4#
&#
images which have been computed from the original
image through the application of the vertical (horizontal)
gradients followed by proper thresholding. Then, the
vertical and horizontal projections of these maps, shown
Y.-S. Ryu, S.-Y. Oh / Pattern Recognition 34 (2001) 2459}2466
2461
Eq. (4) has been set to allow for enough margin around
y to enclose the entire mouth, since otherwise, y could
*
*
pass through the upper, middle or lower part of the lip
according to di!erent poses of the face.
2.2. Extraction of the eyes and mouth
The following heuristic assumptions are made in order
to simplify the process (see Fig. 4(a)):
E The eyes remain within the upper half of the face
region. Their exact positions, however, can vary according to the height of the forehead and the facial
pose.
E The mouth remains within the lower half of the face
region.
Fig. 3. Determination of a face boundary: (a) vertical edge
dominance map: I (i, j); (b) horizontal edge dominance map:
4#
I (i, j); (c) vertical integral projection of (a): H (i) and (d) hori&#
T
zontal integral projection of (b): H ( j).
F
in Fig. 3, are calculated as
,0
H (i)" I (i, j), 1)i)N ,
(1)
T
4#
!
H
,!
(2)
H ( j)" I (i, j), 1)j)N ,
F
&#
0
G
where N "112 and N "92. From these histograms,
0
!
the face region R(x , y , x , y ) is established, where
* * 0 &
N
(3)
y "arg max H ( j), 1)j) 0 ,
&
F
3
H
N
0 )j)N ,
y "arg max H ( j)#y,
(4)
*
F
0
2
H
N
(5)
x "arg max H (i), 1)i) ! ,
*
T
2
G
N
! )i)N
x "arg max H (i),
(6)
0
T
!
2
G
and R(x , y , x , y ) is the rectangle with (x , y ), (x , y ),
(x , y ) and (x , y ) as its four vertices.
The search boundary y in Eq. (3) is determined from
&
the geometrical observation that the hair boundary line
exists in the upper region of the guessed face region. y in
In order to locate the eyes, it is necessary to separate
the eyebrow from the eye using the information contained in the vertical edge map I . This is facilitated by
4#
observing that although the horizontal edge components
dominate in both the eye and the eyebrow, the eye
also contains more vertical edge components than the
eyebrow. Therefore, the horizontal integration of I is
4#
performed and the row index E resulting in the maxA
imum value is found as
2
,!
H ( j)" I (i, j), 1)j) (y !y ),
4#
&
FT
3 *
G
E "arg max H ( j).
A
FT
H
(7)
(8)
Since E represents the row with the strongest vertical
A
edge component within the region containing the two
eyes, the eye region is centered around that row.
However, even this statement may not be correct when
the eyes are not horizontal due to the changes in pose
or the lighting conditions. Therefore, the eye region is
tentatively set as the one centered around E with
A
$15 pixels safety margin to prevent eye or mouth from
being cut.
The "nal eye region is then found by assuming that the
eyes are more likely to reside in the upper or lower eye
Fig. 4. The 3-stage process of determining the left eye, right eye
and the mouth regions.
2462
Y.-S. Ryu, S.-Y. Oh / Pattern Recognition 34 (2001) 2459}2466
regions with respect to the center line E . Under this
A
assumption, the Eye "eld is determined from the horizontal edge components H ( j) de"ned in Eq. (2) as
F
R(x , E !5, x , E #10) if A*B,
* A
0 A
Eye" R(x , E !10, x , E #5) if A(()B,
* A
0 A
?
R(x , E !10, x , E #10) otherwise,
* A
0 A
where
(9)
#A
#A >
A" H ( j), B" H ( j)
F
F
H#A \
H#A
and is set to 1.5. The size of the eye "eld thus found
becomes (x !x );(15 or 20) and the left and right eye
0
*
"elds, Left}Eye and Right}Eye, are obtained by vertically
splitting the Eye region in half (Fig. 4(c)).
Finally, the mouth region is obtained by searching for
strong horizontal edge components. Similar to the eyes,
row M with the strongest sum of the horizontal edge
A
components H ( j) is searched within the coarse mouth
F
region. Then the mouth "eld is established as follows:
M "arg max H ( j), (y !y ))j)y ,
A
F
*
&
*
H
(10)
Mouth"R(x , M !15, x , M #15),
* A
0 A
(11)
with the size of the mouth "eld being (x !x );30
0
*
(Fig. 4(c)).
3. Fine tuning of the eye and mouth 5elds using neural
networks
In general, two approaches exist to search for objects:
(1) matching against the stored database and (2) eliminating non-matching regions [9]. One way to combine these
approaches is the use of positive and negative samples
for training. The positive samples contain the complete
objects whereas the negative samples do not. In this
research, the positive samples are the eye and mouth
regions extracted from the frontal faces while the
negative samples are obtained from the surrounding
areas of the positive samples. Fig. 5 shows these examples. The eye window is 20;10 and the mouth window is
40;20.
The facial feature extractor consists of 3 MLPs for the
left and right eyes along with the mouth. The MLPs are
then trained with a resilient backpropagation algorithm
[10] to produce a linear output whose value represents
the degree of similarity to the positive samples. That is,
the output is #1 for positive and !1 for negative
samples. Physically, the output will be #1 if the candidate window contains the full set of the relevant facial
features.
3.1. Eigenfeature extraction
The input and output representation has a major in#uence on the neural network performance [11]. Hence, it
would be more e$cient to present relevant features
rather than the raw image in order to reduce dimensionality as well as to facilitate training of the network.
Fig. 6 shows the binary edge map representing a 2-D
elliptic data distribution in (u, v) space. The edge pixels in
a search window may be considered as a 2-D pattern in
(u, v). The distribution of these pixels for the eye and
mouth regions is distinctive in nature compared to other
regions of the face. Herein, the Canny edge extractor was
used to precisely locate the eyes while the Sobel horizontal operator was used for the mouth.
In human face, the eyes and mouth have distinct
shapes compared to other features. That is, the general
shape of the eyes and mouth resides within a certain
range of the space spanned by the eigenvectors and the
eigenvalues describing their shapes. These important features lead to a facial feature detector that not only uses
a relatively small number of training samples, but also
lends themselves to robustness against pose variations.
The features used for the neural network training are
then obtained as follows.
The correlation matrix M is obtained "rst from the
N edge point samples:
N
P"(p ,2, p ,2, p N ),
G
,
(12)
Fig. 5. Examples of the training set for the MLPs: (a) the positive eye samples; (b) some negative eye samples; (c) the positive mouth
samples and (d) some negative mouth samples.
Y.-S. Ryu, S.-Y. Oh / Pattern Recognition 34 (2001) 2459}2466
Fig. 6. The binary edge map and the corresponding eigenvectors for each region.
2463
Fig. 7. (a) The MLP output pro"le map and (b) the resulting
regions of interest found.
where
u
p" G ,
G
v
G
M"P ) P2,
(13)
where (u , v )2 is the coordinate value of the ith sample
G G
point. Let the eigenvalues and the eigenvectors of M be
, , e , e , where ' . The eigenvectors represent
the mutually orthogonal principal directions of the data
distribution while the eigenvalues represent the strength
of variation along the direction of the eigenvectors [12].
From these, the following nine eigenfeatures are derived:
x " , x " , x " , x "e ,
}
}
S
x "e , x "e , x "e ,
T
S
T
(14)
1 ,N
1 ,N
u /N , x "
v /N ,
x "
G
S
G
T
N
N
N G
N G
where N and N are the horizontal and vertical sample
S
T
sizes for the eye or the mouth respectively and }
and
}
are the maximum of and over the entire
training set. All these features are signi"cant in some
sense to represent the shape of the facial features. The
"rst and second eigenfeatures are the normalized eigenvalues while the third one is their ratio. These three
represent the overall shape of the facial features. The
fourth to seventh components represent the orientations
of the facial features. Finally, the eighth and ninth components are the two coordinates of the centroid of the
candidate region containing the facial features. Section
4 will examine the in#uence of using di!erent sets of
eigenfeatures on the extraction performance.
3.2. Fine localization of the facial features from the neural
networks based on eigenfeatures
A small window of either 20;10 for the eyes or 40;20
for the mouth is shifted across and down within the
Fig. 8. The centering process for the region detected by the
MLP: (a) before the centering and (b) after the centering.
coarse Left}Eye, Right}Eye, and Mouth regions found in
Section 2. While being shifted, the eigenfeatures of these
small windows are formed and input to the MLP. As
stated earlier, the output of the MLP denotes the similarity between the eigenfeatures of the current window
and those of the typical facial features. Then for each
facial feature, the index of the sliding window resulting in
maximum similarity is determined and identi"ed as the
"ner eye or mouth "eld.
Fig. 7(a) shows the output distribution of the
M¸P
versus the amount of window shift, while
+MSRF
Fig. 7(b) shows the results of extraction of the three facial
features. Zero outputs occur because the test windows
that have a number of edge pixels below a threshold
(20 for the eye and 40 for the mouth) have been skipped
since in that case the possibility of "nding the facial
features of interest is quite low. Finally, these "ner regions are re-centered as shown in Fig. 8, thereby reducing
the load of learning many di!erent shifts of the same
feature images.
4. Experimental results and discussion
The structure of the neural nets used is (9, 50, 30, 10, 1):
"ve-layer neural nets having input layer of 9 nodes,
output layer of 1 node for showing the degree of
2464
Y.-S. Ryu, S.-Y. Oh / Pattern Recognition 34 (2001) 2459}2466
Fig. 9. Experimental results of extracting the regions of interest. The adjacent pairs of images represent the result of coarse "nding before
matching (on the left) with neural nets and those of "ner "nding after matching (on the right).
similarity between learned features and input feature, and
three intermediate layers with 50, 30 and 10 nodes respectively, with a learning rate of 0.01. As for the facial
database, 61 images taken from 14 persons in total were
used. The database contains the training set as well as the
test set containing faces with di!erent poses (looking left,
right or down) and/or from persons other than those in
the training set. The subjects did not wear glasses nor had
beards (Fig. 2(b)). The positive samples were taken from
the eyes and mouths in Fig. 2(a). The negative samples
(70 for the eyes and 29 for the mouth) partially shown in
Fig. 5(b) and (d) were taken from around the eyes and the
Y.-S. Ryu, S.-Y. Oh / Pattern Recognition 34 (2001) 2459}2466
Table 1
The detection performance (%) according to the number of
eigenfeatures used
MLP
Case 1
Case 2
Case 3
Case 4
M¸P }
*CDR #WC
77.1
(39.3)
83.6
(24.6)
95.1
(24.6)
96.8
(16.4)
M¸P
0GEFR}#WC
72.1
(37.7)
70.5
(36.0)
72.2
(34.4)
96.8
(19.6)
M¸P
+MSRF
88.6
(55.7)
88.6
(40.9)
96.8
(22.9)
100
(11.4)
Six eigenfeatures (x, x, x}x) used without re-centering.
G G G G
Seven eigenfeatures (x}x) used without re-centering.
G G
Nine eigenfeatures (x}x) used without re-centering.
G G
Nine eigenfeatures (x}x) used with re-centering.
G G
mouth plus the ones that resulted in misrecognition by
the neural network.
To save time, only those test windows with more than
a certain number of edge pixels were processed and when
the features were not found in the coarse feature window,
this threshold was lowered and retried. Sometimes, the
eyes could not be found when the coarse eye region was
erroneously set, when the region was too dark, or when
the eyes were too small. Fig. 9, due to lack of space, shows
only some of the results of "nding the three facial features
of test face using the full set of 9 eigenfeatures.
Table 1 compares the performance of the four cases of
di!erent combination of input features selected for the
MLP. Successful localization of the facial features is
de"ned as the case when at least 4/5 of these features falls
within the region, namely when the features are reasonably centered. In Table 1, the numbers within ( ) show the
ratios of the non-centered cases over all cases. The di!erences of 14.7% for the left eye, 1.7% for the right eye and
14.8% for the mouth between Case 1 and 2, for example,
signify that the eigenvalue ratio is important for accurate
localization of the eyes and the mouth. In addition, the
use of the normalized centroid of the data in the image
space is important to locate the mouth 18% better.
Finally, the "nal centering process also improves the
localization performance by 8.2, 14.8 and 11.5% for the
left eye, right eye and mouth, respectively. The experimental data suggest that the use of eigenfeatures and the
neural network leads to accurate "nding of the three
facial features, and further, each eigenfeature plays a
certain role in the process.
5. Summary and conclusions
A facial feature extraction algorithm from black and
white facial images has been presented based on eigenfea-
2465
tures and neural networks. The eigenfeatures were obtained from an expanded set of the eigenvectors and
eigenvalues calculated from the binary edge data set for
each of the facial features. The three MLPs for the facial
features were trained with positive and negative samples
so that the MLP can make a decision as to whether the
current test window contains the facial feature of interest.
The particular window resulting in maximum output for
the MLP is designated as the "nal facial feature location
after image centering. As a preprocessing step, the vertical and horizontal projections of the binary edge map
from the face image have been used to narrow down the
search. The salient features of the proposed methodology
are as follows:
E No a priori image normalization is necessary with
respect to intensity distribution, size, and position.
E The coarse eye and mouth regions are determined
within the facial image. While existing research "nds
the approximate eye position using the horizontal edge
map, the proposed research utilizes the horizontal
projection of the vertical edge map to capture the eye
region within smaller window.
E The eigenfeatures extracted from the 2-D data distribution of the eye and mouth edge features have been
introduced as important and signi"cant features to
locate the eye and mouth.
E The need for a large training set in template matching
has been avoided by using eigenfeatures and sliding
windows within the coarse feature "elds.
E The MLPs have a linear output neuron whose output
represents the similarity to any of the positive samples.
E The feature localization experiments have been performed on 61 facial images of 14 people with varying
facial size and pose and the recognition rate is 96.8%
for the eyes and 100% for the mouth.
E Good generalization towards the same or di!erent
person with largely varying size and pose was achieved
despite using a small training data set taken from only
"ve or six persons.
Facial feature extraction has a wide spectrum of
applications from user veri"cation and recognition to
emotion recognition in the framework of man}machine
interfaces. Future work includes the development of
a robust pose-and lighting-invariant face recognition
system as an extension of the current research.
Acknowledgements
The authors wish to acknowledge the "nancial support
of the Korea Research Foundation made in the program
year of 1998 and also in part by the Ministry of Education of Korea toward the Electrical and Computer
Engineering Division at POSTECH through its BK21
program.
2466
Y.-S. Ryu, S.-Y. Oh / Pattern Recognition 34 (2001) 2459}2466
References
[1] Yankang Wang, H. Kuroda, M. Fujumura, A. Nakamura,
Automatic extraction of eye and mouth "elds from monochrome face image using fuzzy technique, Proceedings of
the 1995 Fourth IEEE International Conference on Universal Personal Communications Record, Tokyo, Japan,
1995, pp. 778}782.
[2] R. Pinto-Elias, J.H. Sossa-Azuela, Automatic facial feature
detection and location, Proceedings of the 14th International Conference on Pattern Recognition, Vol. 2, 1998,
pp. 1360}1364.
[3] Weimin Huang, Q. Sun, C.P. Lam, J.K. Wu, A robust
approach to face and eyes detection from images with
cluttered background, Proceedings of the 14th International Conference on Pattern Recognition, Vol. 1, 1998,
pp. 110}113.
[4] Kin Choong Yow, R. Cipolla, Feature-based human
face detection, Image Vision Comput. 15 (9) (1997)
713}735.
[5] Takeo Kanade, Picture processing by computer complex
and recognition of human faces, Technical Report, Department of Information Science, Kyoto University, 1973.
[6] D.J. Beymer, Face recognition under varying pose, A.I
Memo No. 1461, 1993.
[7] R. Brunelli, T. Poggio, Face recognition: features versus
templates, IEEE Trans. PAMI 15 (10) (1993) 1042}1052.
[8] P. Juell, R. Marsh, A hierarchical neural network for human
face detection, Pattern Recognition 29 (5) (1996) 781}787.
[9] D.F. McCoy, V. Devarajan, Arti"cial immune systems and
aerial image segmentation, IEEE Int. Conf. Systems Man
Cybernet. 1 (1997) 867}872.
[10] M. Riedmiller, H. Braun, A direct adaptive method for
faster backpropagation learning: the RPROP algorithm,
IEEE Int. Conf. Neural Networks 1 (1993) 586}591.
[11] E. Fiesler, R. Beale, Handbook of Neural Computation,
Oxford University Press, Oxford, 1997.
[12] S. Romdhani, Face recognition using principal components analysis, MS Thesis, http://www.elec.gla.ac.uk/
%romdhani/pca.htm, 1996.
About the Author*YEON-SIK RYU received the BS degree in Electrical Engineering from HanYang University, Seoul, Korea in 1988,
and the MS degree from Pohang University of Science and Technology (POSTECH), Pohang, Korea in 1990. He is currently a research
engineer of LG Electronics Inc. and is working toward a Ph.D. in Electrical Engineering. His research interests are in the area of neural
networks, immune system, and evolutionary computation and its application to automation systems.
About the Author*SE-YOUNG OH received the BS degree in Electronics Engineering from Seoul National University, Seoul, Korea in
1974, and the MS and Ph.D. degree in Electrical Engineering, from Case Western Reserve University, Cleveland, OH, USA, in 1978 and
1981 respectively. From 1981 to 1984, he was an Assistant Professor in the Department of Electrical Engineering and Computer Science,
University of Illinois at Chicago, Chicago, IL. From 1984 to 1988, he was an Assistant Professor in the Department of Electrical
Engineering at the University of Florida, Gainesville, FL. In 1988, he joined the Department of Electrical Engineering, Pohang
University of Science and Technology, Pohang, Korea, where he is currently a Professor. His research interests include soft computing
technology including neural networks, fuzzy logic, and evolutionary computation and its applications to robotics, control, and
intelligent vehicles.