MVA2002 IAPR Workshop on Machine Vision Applications, Dec. 11- 13,2002, Nara- ken New Public Hall, Nara, Japan
13-21
Detection of Partially Occluded Face using Support Vector Machines
Sang Min Yoon, Seok Cheol Kee
HCI-LAB, Samsung Advanced Institute of Technology
(smyoon, sckee}@sait.samsung.co.kr
Abstract
Partially occluded face detection is need, because although the technology of the Automated Teller Machines
and face detection is increased, we cannot control the people who wear sunglasses or mask for the crime. To reject
the occluded face, we first trained the features of the normal faces and the occluded faces that wear sunglasses or
mask using Principal Component Analysis and Support
Vector Machines to reduce the dimension and classify eficiently. Then we decide that the detected face is normal
or partially occluded face using the scheme that integrates
the Principal Component Analysis and Support Vector
Machines. In the experiments, we trained the 3200 normal face images that have the variations of illumination
and expression and each 2900 and 4500 partially occluded
face images that wear the sunglasses or mask with 60*25,
and 60*35 resolution. We get the 95.2% and 98.8% partially occluded face detection ratio after face detection and
2.5% and 0% false alarm ratio from the experiments based
on the Purdue University Face DB. The proposed algorithm which is incorporated the face detection system can
help the security using the Digital Video Recorder System
and face recognition
1
Linear subspace analysis has been used as representative
methods, linear Principal Component Analysis (PCA) or
Fisher Linear Discriminant Analysis (FLDA), and eigenface because of its simplicity and efficiency. Support
Vector Machines (SVM) is a very powefil supervised
learning algorithm that is rooted in statistical learning theory[l]. SVM that is used very popularly in the computer
vision area plays an important role in this system. And
this system helps to prevent the crime at the ATM and
waste of the hardware better than the previous DVR system.
In section 2, we show the definition of the partially occluded face. We then present in section 3 the training and
test method to detect the partially occluded face using PCA
and SVM. At that section, we will present the roles of the
PCA and SVM. We will describe the experiment results
at section 4. Our conclusion remark is presented at section 5. Figure 1 depicts the total flow diagram of the
envisioned system. This system is divided of 3 parts: One
is face detection and size normalization, the second is feature extraction by PCA and classification by SVM, and the
third is the identification and warning of the partially occluded face.
Introduction
Recently, the use of Automated Teller Machines (ATMs)
has been increased because of popularization and variety of
services at the ATMs and staff reduction of the back.
With increasing of the popularity of the ATMs, the researchers are developing the security system to protect the
depositors. Most ATMs use the credit card and secret
number for guarantee person's identification. As the
technologies related to ATMs have improved very fast, the
crimes of credit card's loss or exposure of the secret number to other people are growing nowadays.
For
preventing the crime, the Digital Video Recorder (DVR)
system and face detection1 tracking algorithms are supplemented at ATMs to identify the user's particulars. But the
most people who commit the crime wear the sunglasses or
mask to harbor their identity.
In the last ten years, Face detection has become an important area of computer vision such as video surveillance,
human-computer interface, and identity authentication.
But much more research still remains to be done in the
occluded face detection. Previous DVR system records
the difference images to detect the human motion. But
this system stores the unnecessary images that are not existent human face and cannot know the identity of partially
occluded faces that wear the sunglasses, or mask. To
control the access of the occluded face to commit the crime,
we proposed the intelligent algorithm to classify the normal face and occluded face. And this will help to the
security systems that are related to access control of unidentified person.
Figure 1. Total flow diagram of system
2
Definition of the partially occluded face
Generally, people have the same features of eyes, eyebrows, nose, and mouth of the face. The shape and
location of each component is very similar to each person.
But the verification of these features has an important role
to recognize and identify one's face. If one component of
face's feature points is covered with an obstacle, we cannot
know one's identification. To know one's identification
from the image, we must see all features of the face. Eyes
and eyebrows are gathered at the upper part of the face, and
mouth and nose are gathered at the lower part of the facial
image. We define that the partially occluded face is not
able to classify one's identification because of being covered with an obstacle such as hat, sunglasses, mask, or
mufflers. And we define that the face that can identify is
normal face. It is necessary for the person who wears
these obstacles to prevent the use of ATM. As control the
access of ATM, we can reduce the crime and increase the
detection ratio of the normal face.
3
shows the upper part of partially occluded face and figure
3-(b) shows the lower part of that.
Detection of the partially occluded face
To detect the partially occluded face, we train the normal
face and partially occluded face. Normal faces are composed of the faces that have change of illumination,
expression, and rotation. Partially occluded faces are
composed with the faces that wear the sunglasses, mask,
and muffler. Changes of illumination and expression are
included to the partially occluded face DB too.
3.1 Training of the partially occluded faces and normal faces
The performance of this system is influenced by the
methods of feature extraction and classification. So selection of the training images is very important in this part.
The environments in which ATMs are established are very
diverse and the people are not limited to the age and sex
distinction. The training images are reflected to the
variation of the illumination's direction and strength, face
expression, beard, and rotation. Especially, beard and
expression can be mistaken as a partially occluded face.
Occluded faces are trained by varying the shape and intensity of sunglasses, masks, and mufflers.
We normalize the size of training images with 60*60.
Size normalization is based on the eye position that is detected with Gabor wavelet and SVM. We search the
candidate region of the face using Gabor wavelet and extract the face by hierarchical classification varying the size
of the image. ~h~ examples of the trained normal face
and occluded face images are shown in figure 2.
(a) Occluded face that wears the sunglasses
(b) Occluded face that wears the mask or muffler
Figure 3. Training images of occluded face
To extract the feature from the normalized image, we use
PCA to the training images and test image. Linear subspace analysis such as PCA has been used more popular
than raw image data in the computer vision area because of
its simplicity and efficiency. Facial images are not randomly distributed in higher dimensional image space and
thus can be described by a relatively low dimensional subspace.
We show the feature extraction process of the normal
face and partially occluded face that wears the sunglasses
using PCA[2].
r r'
, are represented as vectors of normalized images
of the normal face and occluded face with the size 1 X J .
Total number of the training set of normal face is N and
occluded face that wears the sunglasses is M . Averages
of each training set is defined as the equation (3), (4).
(a) Examples of the normal face images
(b) Examples of the occluded hce images
Figure 2. Size normalized training images
At next 'step, we separate the face image component by
component: the one is the forehead and eyes and the other
is nose and mouth. By separating the image into component, we can reduce the image dimension with 60*25 of the
forehead and eyes, 60*35 of the nose and mouth.
Through this process, we can find where occlusion is
happened easily and reduce the noise' effect such as hair,
background. At the training process, partially occluded
face that wears the sunglasses is mis-detected as eyes from
the face detection algorithm using Gabor and wavelet. So
the position of mis-detected sunglasses is not exact eye
position and this can be changed at the continuous input
image. So we trained the component of the face that wear
the sunglasses with varying the position, scale, and rotation.
As varying the position of the eye's position, lower part of
size normalized image also have changes. Figure 3 shows
the example images used in the training. Figure 3-(a)
As shown the equation (1)-(4), we can calculate the eigenvectors u, ,u, . Figure 4 shows the examples of the
extracted eigenvectors of the normal face of upper and
lower part and occluded face of face's upper face part.
(a) eigenvectors of the normal face'!: upper part
(b) eigenvectors of the occluded fac-s upper part
(c)eigenvectors of the normal face's lower part
Figure 4. eigenvectors of the training image set
We detect the face region using Gabor and SVM first.
From segmented face region, we separate the face region
into upper part and lower part. Figure 6 shows the process of separating component of the face.
The weights of the normal face and partially occluded
face, w,, wf are calculated as shown in equation (5), (6).
By including the eigenvectors of the normal and partially
occluded face, weight value of each feature is influenced
from each training image. As considering the effect of
features of each class, it helps us to have good features to
classify the normal and partially occluded face by SVM.
After obtaining PCA features, we build the SVM training
set {c,,d, I:=, where d, = {I,- 1) is the class type of PCA
feature c, . I is the size of the training set. SVM implicitly maps the data into a dot product space via a
nonlinear mapping function. Thus the SVM learn a hyperplane which separates the data by a large margin.
Generally, SVM has the following form:
Figure 6. The process of separating the face
from the input image
By applying PCA and SVM schemes to extract the feature points and classify that the feature points are near the
normal or partially occluded face. We can reduce the
processing time and noise effect because of PCA and SVM.
If input image is classified to the partially occluded face,
ATMs send the warning message that the users must remove the obstacles of their faces to detect the normal face.
When the users ignore that warning message, we control
the access and discontinue the service of ATMs.
4
where y, andx, are a class label and a training feature
vector respectively, A and b are constants which are
is a polynormial kernel. In clasdecided by learning,
sification using the non-linear SVM, run-time complexity
is very high because it is proportional to the number of
support vectors[3] [5].
In this paper, we use the kernel as polynormial kernel.
Polynomials K ( x , ~ ' =) ( x . x ' ) ~can be shown to map
into a feature space spanned by all orderd products of
input features. This means that support vectors act as
templates for normal face and partially occluded face, thus
relating non-linear support vector's to vector quantization.
Support vector's value is obtained as shown in figure 5.
Figure 5-(a) shows the support vector of the upper part of
normal face and partially occluded face. Figure 5-(b)
shows the support vectors of lower part of the normal and
partially occluded face. Blue points are the support vectors of normal face's upper and lower part and red points
are the support vectors of partially occluded face's upper
and lower part.
k
Experiment results
To detect the partially occluded face using PCA and
SVM, we used the 3200 images of normal face, 2900 images that wear the sunglasses, and 4500 images that wear
the mask, muffler, and non-face. From training images,
we extracted the 100 eigenvectors from each trained class.
The number of the support vectors and error is shown in
the table 1.
Lower part
Number of
support vectors
Number of errors
266
1
Table 1. Number of support vectors and errors
To test this system' reliability and efficiency, we use
Purdue Univ. face DB as test image[6]. Pwdue Univ. face
DB are composed with the normal faces and partially occluded faces images that are included of change of
illumination and expression. Figure 7 show the examples
of the test images of partially occluded faces.
Figure 5. Distributions of support vectors of normal
and partially occluded face
3.2 Test of the partially occluded face
Detection of the partially occluded face of the input image has the same process of the training part as shown in
section 3.1.
--
(a) Test images that wear sunglasses
-
This system help to prevent the crime related with ATM.
At the training process, we include the images that have the
change of the direction and strength of illumination and
expression. By separating the image with upper and lower
component of the face, we reduce the dimension of the
image and effect of the noise such as hair and background.
And we can obtain more reliable and fast detection of the
partially occluded face using PCA and SVM. The proposed system can be used to not only control the access of
the people who use the ATM but also guarantee one's
identification at the electronic commercial transaction.
References
(b) Test images that wear muffler
Figure 7. Test images of partially occluded faces
The detection ratio of test images and FAR (False accept
Ratio) are reported at the table 2. We test 370 images that
wear sunglasses and 390 images that wear mufflers. Each
detection ratio and FAR of the partially occluded faces are
obtained when the detection ration of the normal face is
98% and FAR of the normal face is 0%.
I
I
Detection ratio
I
FAR
Upper part
95.2%
2.4%
Lower part
98.8%
0%
Table 2. Detection ratio and FAR
of the partially occluded face
We had an experiment of on-line test for detection of the
partially occluded face with USB camera that had 320*240
resolution and programmed it using visual C/C++ at
Pentium 4 computer. Result images of real-time partially
occluded face detection is shown in figure 8. Eye positions and its candidate are painted with green circles. And
normal face is painted with blue and partially occluded
face is painted green and red square respectively.
(a) Normal face detection
(b) Occluded face detection
Figure 8. Detection results of on-line test
5
Conclusion
In this paper, we have presented a new algorithm that
integrates PCA and SVM to detect the occluded face.
I
[I] V. Vapnik. The Nature of Statistical Learning Theory.
Springer, New York, 1995.
[2] M.Turk, and A.Pentland, "Eigenface for reconstruction, " Journal of Cognitive Neuroscience, Vo1.12,
No.1, pp71-86, 1991.
[3] Tea-Kyun Kim, Donggeon Kong, and Sang-Ryong Kim,
"Learning a decision boundary for face detection,
"ICIP 2002, Vol. I, pp 920-923,2002.
[4] Sami Romdhani, Philip Ton; Grnhard Scholkopf, Andrew Blake, "Computationally Efficient Face
Detection, " ICCV 2001, Vol. 2, pp 695-700,2001.
[5] Edgar Osuna, Robert Freund, and Federico Girosi,
"Training Support Vector Machines: an application to
Face Detection," WCCC-ICSP 2000, Vol. 3, pp 1427 1431,2000.
[61 Martinez, A.M, "Recognition of partially occh~ded
andlor imprecisely localized faces using a probabilistic
approach, " CVPR 2000, Vol. 1, pp 712-7 17,2000.