Real Time Eye Tracking and Blink Detection Using Low Resolution Web Cam
Yassir Nawaz1 and Shiladitya Sircar1,2
Faculty of Engineering and Applied Science1/ Remote Sensing Group C-CORE2
Memorial University
St. John’s, NF, CANADA A1B 3X5
E-mail: {yassir, sircar}@engr.mun.ca
Abstract: In this paper we present a computationally efficient and cost effective solution for blink detection and
tracking of eyes in real time. We also present a framework that can be used to investigate various strategies for
human computer interaction. We commence with the extraction of human face and its various features from a lowresolution video stream. These extracted features, in particular eyes, are tracked and monitored in real time to
detect any movements and variations. These variations then form the basis of various actions or responses. Lastly
we demonstrate the effectiveness of our technique through experimental results and provide recommendations for
future development and improvement.
1. Introduction
In recent years, scientists are aiming to develop non-intrusive man-machine-interface with vision systems in a
constrained domain. This has led to the emergence of the field of automatic face processing and identification of
visual facial behaviors (such as blinking, smiling, frowning etc.), which are instinctively inherent to every human
being and thus bringing cognitive sciences closer to computer sciences. This paper presents an efficient technique
for eye tracking and blink detection for low resolution web cameras.
2. Feature Point Initialization
In order to track a facial feature it must be first identified in an image stream. The initialization process starts with
the location of face in the first image. Human face is extracted from the image using skin segmentation techniques.
Skin pixels can have very different RGB values for different people depending on their skin color, illumination etc,
however their chromatic red and chromatic blue values defined as chR = R/(R+G+B) and chB = B/(R+G+B) are
clustered in a limited region of (chR, chB) space and are largely independent of skin color or lighting conditions.
Based on their chromatic red and chromatic blue values, the pixels are classified into skin or non skin pixels. After
face segmentation the image is subject to blob analysis to locate the exact position of the eyes. The technique makes
use of the fact that eyes when open have color different from skin and therefore appear as holes in the face, in the
skin segmented image. These holes give the approximate location of the eyes. These locations are then used to track
the eyes in the remaining video stream. Blink detection is used to further improve these location estimates during
tracking.
3. Eye Tracking using Optical Flow
Optical flow is a very powerful and popular technique used in motion estimation. The optical flow algorithm (Lucas
& Kanade) used in this technique tracks a feature point by trying to predict the apparent motion of image brightness
around that point. Lucas & Kanade technique examine the brightness of the pixel neighborhood and then tries to
locate the position of the pixel in the subsequent images assuming that all neighborhood moves with the same
velocity. The selection of the feature points to be used in Lucas & Kanade tracking algorithm is crucial to the
performance of tracking. A good feature point should have a distinct neighborhood so that it can be identified easily
in the subsequent images. In this work it has been found that inner corner of the eye serves as a very god feature
point to track. The tracking algorithm returns the new position of the eyes in each image of video stream. These
positions are then examined to monitor head orientation or loss of a feature point. In case one or more feature points
are lost due to a rapid movement or turning of the head feature points are initialized once again as described in
section 2. Figure 1 shows the tracking of eye using Lucas & Kanade optical flow technique.
4. Blink Detection
The eye tracking reduces the blink detection search space to two eye blobs. The location of these blobs or eyes is
passed on to the blink detection algorithm by the eye tracker in each image. When a person blinks the intensity of
the pixels that represent the eye also changes. This forms the basis of blink detection. A variance map of the change
in intensity values in the corresponding pixels of the two consecutive frames, in an image sequence is created on a
Figure 1 Eye tracking using optical flow
pixel by pixel basis. Two maps, one representing the mean μ and the other variance σ2are initialized. The arrival of a
new image updates these maps recursively according to the following relations:
 j 1 ( x, y) 
j j ( x, y)  I j 1 ( x, y)
j 1

1


 2j 1 ( x, y )  1   2j ( x, y )  ( j  1)(  j 1   j ) 2
j
(1)
(2)
The variance map is thresholded after each update. If the ratio of the number of thresholded variance pixels to the
total number of pixels in the eye blob exceeds a threshold ε a blink is detected. Figure 2 shows two images taken
from a blink sequence and the corresponding variance map generated.
Figure 2 Blink detection using variance maps
5. Eyeball Detection
Eyeball detection algorithm receives the coordinates of two eye boxes and finds the location of eyeballs within these
boxes. Several techniques were considered for the purpose of eyeball detection. Geometrical pattern search or ellipse
fitting techniques provide very good results however they are computationally intensive and were rejected in favor
of circular Hough transform. Circular Hough transform itself is computationally intensive however it can be
improved considerably by pre processing the eye image. This pre processing reduces the number of pixels which can
vote in the Hough transform making it much faster.
5.1 Pre Processing of Eye Image
The purpose of pre processing is to minimize the number of on-pixels in the binary image on which Hough
transform is computed. An ideal image for Hough transform will only contain a circle representing the outline of the
eyeball, however it is difficult to obtain and attempt is made to remove unwanted pixels without removing the eyball
pixels. The pre processing steps are:
5.1.1 Color filtering: Three channel RGB image is converted to a single channel image for edge detection. It has
been found experimentally that green channel image provides best results for edge detection even though difference
with other channels is not huge.
5.1.2 Histogram equalization: The single channel image is equalized to compensate for different lighting
conditions. Since eye image contains two regions of comparable areas and distinct intensities, one representing the
eyeball and the other white area around it, equalization tends to highlight the edges of the eyeballs.
5.1.3 Edge detection: The green channel image is then subjected to edge detection to get an outline of the eyeball.
Since the eyeball pixels have different intensity than the white pixels around it an edge is readily obtained however
it is occluded. A 3 by 3 vertical Prewitt operator is used for this purpose. Horizontal Prewitt operator was also tried
but since the outline of the eyes form horizontal edges, extra edges appear in the edge image. Vertical operator
returns the vertical edges which are sufficient for Hough transform as all the pixels are not required to obtain the
best fit circle.
The steps explained above are shown in figure 3.
Figure 3 Pre processing of eye image for eyeball detection
5.2 Circular Hough Transform
Hough transform can be used to extract various features from an image. The transform takes the geometric equation
of the feature to be found and then inverts it so that x and y become constants in the equation, thus resulting in an
inverse function space. The intersections in this space reveals the constants associated with the equation of the
features discovered in the original image. Circular Hough transform is used to find the coordinates of circles in an
image. In order to determine the location of the eyeball in the processed image a range for the radius of the desired
circle is specified. Each thresholded pixel in the binary image is examined to determine whether it fits the desired
circle. The best circle gives the outline of the eyeball. The technique is very powerful since it can fit circles even
when a partial outline of a circle is available which is common in an eye image due to the occluded eyeball. Figure 4
shows the extraction of eye ball from eye image using circular Hough transform. The image has been equalized and
vertical Prewitt operator has been used to get the approximate outline of the eyeball as explained in section 5.1.
Figure 4 Circular Hough transform to locate eyeball
6. Eyeball Tracking
Hough transform can be used to locate eye ball in each eye image thus determining the location of the eye ball. This
technique however is computationally intensive and slows down the throughput of the algorithm. The advantage of
computing Hough transform on the other hand is that we do not have to worry about the loss of tracking. If the
Hough transform fails to find a circle in one image, this will not affect the transform computation in the subsequent
images. Another technique is to use Lucas & Kanade feature tracker explained in section 3 to track the eyeball once
its center and radius has been obtained through circular Hough transform. This vastly improves the speed of the
algorithm however the tracking will be lost upon the loss of a feature point representing the eyeball. This feature
point could be lost due to a blink or a rapid movement of the eyeball or eye itself. We address this problem by
constantly monitoring the tracked feature points to determine if the tracking has been lost, in which case Hough
transform is used to re initialize the eyeball location. The best feature points for eyeball tracking are the center of the
eyeball and its two outer edges. The relative distance among these three points should remain constant as the eyeball
moves within the eye. A significant change in the above distance will indicate a loss of a feature point.
Figure 5 Eyeball tracking
7. Results and Future Work
Figure 5 shows the results of our technique. Three images taken from the eye tracking video demonstrate the
tracking of the eyeball along with between-the-eyes feature point during the movement of the head in various
directions. The results for blink detection are satisfactory for this project. The eye tracker can tolerate up to 45
degrees head turn in either direction. This can be further improved by using ‘between-the-eyes’ feature point. This
point can tolerate up to 90 degrees rotation of the head and can be used to reinitialize the lost feature points by using
the distance measurements from the other ‘tracked’ eye. The other alternative is to reinitialize all the feature points
using blink detection. Hough transform gives satisfactory results for this project where a very low-resolution web
cam has been used. The images shown above have a resolution of 320×240. If more accuracy is required i.e. very
subtle eyeball movements need to be detected a high-resolution web camera can be used. This will however increase
the number of pixels processed by the Hough transform thus reducing the overall speed. Additional hardware to
increase the computing power may be used to offset the additional calculations. The eye tracker and blink detector
implemented in this project can be used to built various intelligent applications some of which are suggested below
 Tracker and blink detector can be interfaced with a PC mouse so that cursor can be moved on the screen using
eye movements. Voluntary blink detections can be used to simulate mouse clicks thus enabling a hands free
browsing. This application can provide severely disabled individuals, who are unable to operate ordinary
mouse, a way to use personal computers.
 Various human monitoring applications measure the blink rates of human to determine several biological
parameters. This software can be used to monitor and detect the blink rate of a person automatically.
 This software can be used to investigate techniques used in advanced interactive robots that have the ability to
converse with humans. These robots must have the ability to locate a human face and determine where they are
looking to determine their response.
References
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
Bruce D. Lucas and Takeo Kanade. An Iterative Image Registration Technique with an Application to Stereo Vision. International
Joint Conference on Artificial Intelligence, pages 674-679, 1981.
Carlo Tomasi and Takeo Kanade. Detection and Tracking of Point Features. Carnegie Mellon University Technical Report CMU-CS91-132, April 1991.
D. H. Ballard, "Generalizing the Hough transform to detect arbitrary shapes." Pattern Recognition, v. 13, n. 2, pp. 111-122, 1981.
C. Kimme, D. H. Ballard, and J. Sklansky, "Finding circles by an array of accumulators." Communications of the Association for
Computing Machinery, v. 18, pp. 120-122, 1975.
N. Oliver, A.P. Pentland, and F. Berard. Lafter: Lips and face real time tracker. In CVPR, 1997.
Y. Raja, S.J. McKenna, and S. Gong. Tracking and segmenting people in varying lighting conditions using colour.In AFG, 1998.
J. Yang, L. Weier, and A. Waibel. Skin-color modeling and adaptation. TR CMU-CS-97-146, 1997.
Canny, 1986 A computational approach to edge detection. IEEE Trans. PAMI, 8:769 - 698.