International Conference on Software: Theory and Practice (ICS2000)
Quick Button Selection with Eye Gazing
for General GUI Environment
Masatake Yamato1
Akito Monden1
Ken-ichi Matsumoto1
Katsuro Inoue1,2
Koji Torii1
1
Graduate School of Information Science, Nara Institute of Science and Technology
8916-5 Takayama, Ikoma, Nara 630-0101, Japan
2
Graduate School of Engineering and Science, Osaka University
1-3 Machikaneyama-cho, Toyonaka, Osaka 560-8531, Japan
E-mail: [email protected] Fax: +81 743 72 5319
Abstract
This paper proposes an efficient technique for eye
gaze interface suitable for the general GUI
environments such as Microsoft Windows. Our
technique uses an eye and a hand together: the eye for
moving cursors onto the GUI button (move
operation), and the hand for pushing the GUI button
(push operation). In order to make our technique
more effective for the general GUI environments, we
also propose the following two techniques to assist
the move operation: (1) Automatic adjustment and (2)
Manual adjustment. In the automatic adjustment, the
cursor automatically moves to the closest GUI button
when we push a mouse button. In the manual
adjustment, we can move the cursor roughly by an
eye, then move it a little more by the mouse onto the
GUI button. In the experiment to evaluate our
method, GUI button selection by manual adjustment
showed better performance than the selection by a
mouse even in the situation that has many small GUI
buttons placed very closely each other on the GUI.
1. INTRODUCTION
In recent years, user interfaces using humans’ eye
movement as an input have become a popular topic
among HCI researchers [9][11][13][16]. Such
interfaces — called eye gaze interfaces — enable us to
operate GUIs without using hands. The user of such
interfaces only needs to look at the GUI for selecting
buttons and/or menu items, zooming in/out a window,
scrolling a window and so on [2][4][8][14]. Such an
interface is useful not only for the person who has
physically handicaps in his/her hand, but also for the
person who do not have enough working space for
operating mouse and keyboard to use computers.
The increase of researches in eye gaze interface is due
to the improvement of eye tracking devices for eye
measurement [1][6][16]. Formerly, users of an eye
tracking device had to fix their head and/or to wear a
heavy helmet for measurement. Such a device was not
considered as a daily-use input device because of its
poor accuracy, difficulties in wearing for a long time,
and restrictions on humans' actions. However, today'
eye tracking device is getting to be more accurate,
convenient and inexpensive. It is enough compact and
does not restrict the users’ actions. Users do not need
to wear a heavy helmet anymore; they only need to
wear glasses instead [7][8] (see Figure 1). We believe
that eye tracking device will become one of the
popular input devices in general computer
environments in the near future.
However, conventional eye gaze interfaces are not
supposed to use in general GUI environments such as
Microsoft Windows, Mac OS, and X-Window, etc.
They assume specialized GUI environments that users
do not use their hands. In this environment, the eye
tracking device is regarded as a substitute for a
keyboard and a mouse. If we apply such interfaces to
the general environments, in other word, assuming the
situation that we can use our hands, we will have
following two problems[15]:
(p1) Selection error problem:
This problem is known as “Midas Touch Problem,”
which comes from the difficulty in judging whether
the user is gazing on a GUI button to select it or just
looking it [5]. If the interface simply assumes that a
button at which the user looks is to be selected, every
GUI button he/she looks at will be selected. In this
case, we cannot look anywhere without issuing a
button selection. Although such an interface is still
very useful in the situation that we cannot use hands,
most people who use general GUIs do not find any
worth in using such an errorful interface. In addition,
we do have some studies that solve this problem; so
far, most of them use specialized GUIs that are not
easily applicable to the present GUI situations [3][10].
(p2) Selection performance problem:
We have no evidence that the conventional eye gaze
Move operation with
eye tracking device
Where is the
GUI button
Push operation
with mouse
I have found it.
Clicking
Figure 2. Combination technique
Figure 1. Eye tracking device
interface is more efficient than the mouse interface
when we apply it to general GUI environments.
Generally, they were tested only in the specialized
GUI situations that have a few (and also huge) GUI
buttons [6] [16]. Actually, there are many tiny icons
and buttons scattered on the present GUIs. We need an
efficient eye gaze interface suitable to general GUI
environments.
The goal of this research is to realize an errorless and
efficient eye gaze interface suitable to general GUI
environments. As a first step in this goal, this paper
proposes efficient techniques for selecting GUI
buttons on general GUIs. In order to overcome above
problem p1 and p2, we propose following two
techniques:
(1) Combination technique
In the proposed interface, we allow users to use their
eye and hand together in doing operations in the
general GUI environments. An eye is mainly used for
moving a cursor; and, a hand is mainly used for
identifying a GUI button to be operated. Hence, in
order to select a GUI button, for example an icon, a
user has simply to look it and push the mouse button.
The user can move cursor very quickly and does not
be annoyed by Midas Touch Problem, which points
out that no one can decide perfectly whether he/she
wants to select the GUI button which he/she is gazing
on.
(2) Adjustment technique
To improve accuracy and efficiency of cursor
movement by eyes, the proposed interface provides us
with two types of cursor location adjustment:
automatic adjustment and manual adjustment.
Automatic adjustment – the cursor automatically
moves to the closest GUI button when a user pushes a
mouse button.
Manual adjustment – a user can move the cursor
roughly by an eye, and then move it onto the GUI
button precisely by a mouse. A similar technique is
also proposed by Zhai et al [16]. However, their
technique does not consider the situation that has a lot
of buttons placed on general GUIs.
Distance
Size
Highlighted button
Figure 3. Window for experiment
(size: 5cm2 , distance: 5cm)
The rest of this paper first describes characteristics of
eye as an input to computers and our basic technique
to combine an eye tracking device and a mouse
(Section 2). Next, This paper describes a pilot
experiment that probes and clarifies the actual
problems, which may occur when we use our
technique in the general GUI environments (Section
3). Then we propose a cursor adjustment technique
based on the results of the pilot experiment (Section 4).
Afterward, we describe an experiment to evaluate the
adjustment technique in various GUI environments
(Section 5); and in the end, conclusions and future
topics will be shown (Section 6).
2.
INTERFACE
USING
EYE
TRACKING
DEVICE AND MOUSE TOGETHER
In this section, before proposing our eye gaze
interface, we discuss the characteristics of an eye as
an input to computers, on both strong points and weak
points. Based on the discussion, we propose an eye
gaze interface in the next.
2.1 Characteristics of an eye as an input to
computers
Characteristics an of eye as an input to computers are
67.29
Combination
Mouse
Mouse
83.71
Combination
Combination
Mouse
Mouse
Figure 5. Task completion time (pilot experiment)
Figure 4. The number of misses (pilot experiment)
as follows:
(c1) High speed movements
Moving cursors by an eye is much faster than by a
mouse because the eye can move very quickly in
comparison to the hand. Since the range of the eye
movement in the computer display is narrow, the
speed of eye movements is about from 350 degree/sec.
to 500 degree/sec [8]. It takes only about 150 ms to
move the gazing point from a corner to the opposite
corner of 21-inch size display.
(c2) Midas Touch Problem
If we consider of an eye gaze interface that can select
a GUI button simply by looking it, every GUI button a
user looks at will be selected. In this case, the user
cannot look anywhere without issuing a button
selection. Ideally, the interface should act on the user’s
eye input only when he/she wants it to and let him/her
just look around when that’s he/she wants, but two
cases are impossible to distinguish in general. Eye
gaze interfaces need to avoid this problem [5][12].
(c3) Jittery motions during a fixation
Even if a user thinks that he/she is staring at a certain
point on the display, actually it is not true because
his/her eye makes jittery motions. This makes it
difficult for users to point exactly at the very small
GUI button by eye.
(c4) Measurement accuracy
The measurement accuracy of the present eye tracking
device is not high enough to point a very small GUI
button. In general, the accuracy is about from 0.5
degree to 1 degree as the angle of view. In case that
the user sits 50 centimeter away from a display, the
error on the display is about 0.9 centimeter.
Considering that the size of GUI button used in
general GUI like MS-Windows is about one
centimeter square, the accuracy of the eye tracking
device is not high enough.
2.2 Combination technique
We allow users to use their eye and hand together in
doing operations in the general GUI environment.
We split the operation of selecting a GUI button into
two operations: move operation and push operation.
We apply an eye tracking device to the move
operation, and a mouse to the push operation (see
Figure 2). Under this policy, in order to select a GUI
button, for example an icon, a user has simply to look
it and click the mouse button.
On account of characteristic c1 described in 2.1, eye
tracking device is suitable to the move operation. On
the other hand, on account of characteristic c2, eye
tracking device is not suitable to the push operation;
and, the mouse is more efficient for this purpose.
3. PILOT EXPERIMENT
3.1 Objective, materials and task
In this experiment, we probe and clarify the actual
problems that may occur when we use our
combination technique in the general GUI
environments.
Objective
Small button
Large button
Cannot move
the cursor to
Push operation
2
2
Error
The gazing point jitters in
the button area.
The closest button
is selected.
The gazing point jitters and
moves out of the button area.
Figure 8. Automatic adjustment
Figure 6. Influence of the jitter motions
Rough movement by
look
Small button
Large button
Delicate
adjustment by
mouse
2
2
error
error
The error is existing but the
cursor is in the button area.
The error is existing and the
cursor is out of the button area.
Figure 9. Manual adjustment
Figure 7. Influence of the measurement error
The major objectives of this pilot experiment are as
follows:
We need to know how serious the characteristic c3
(jittery motions during a fixation) will be, when we
use the combination technique in actual GUI
environments.
We also need to clarify the influence of characteristic
c4 (measurement accuracy of an eye tracking device).
Subject
The subject is one graduate student at Nara Institute
Science and Technology. The subject uses Microsoft
Windows usually and is familiar with operations by
mouse.
Task
We prepared many sizes of GUI buttons; and, we
placed them in many layout patterns. As a single task
of this experiment, the subject is asked to select one
highlighted GUI button out of nine buttons on a
window for 10 times (Figure 3). The highlighted point
will change randomly after selecting a GUI button.
Ten kinds of the GUI button layout patterns are
prepared: 5 different sizes of GUI buttons (1 square
centimeter, 2 square centimeters, 3 square centimeters,
4 square centimeters and, 5 square centimeters), and
two different distances between GUI buttons (5
centimeters and 1 centimeters).
The subject is asked to execute the task twice: once by
using the combination technique and once by using a
mouse. Before executing a task, the subject is asked to
train each technique well. Therefore, the result of the
experiment is supposed to have little influence of
experiences in each technique.
Collected data
The time spent for selecting GUI buttons 10 times is
collected in each layout pattern. We also counted the
number of selecting misses. Here, we regard it as a
miss when the subject did the push operation on a
button that is not highlighted or he/she did it in
outside of the GUI button areas. If the subject made a
mistake, the window beeps as an alert. In this case, the
subject is asked to try to select the highlighted button
till he/she selects a correct GUI button.
3.2 Results of pilot experiment
The number of misses while executing the task is
shown in figure 4. The task completion time is shown
in figure 5. We describe each of them next.
Misses
As shown in figure 4, we find that only a few misses
are found if the GUI buttons are larger than 3 square
centimeters. In such cases, with the combination
technique, the subject can select a GUI button without
making a mistake. However, it is difficult to select a
GUI button smaller than 3 square centimeters using
the combination technique. If a GUI button gets
smaller than 3 square centimeters, the number of
misses increases more. (The measurement error of eye
tracking device is about one centimeter.)
Task Completion Time
Figure 5 shows that the combination technique is
more efficient than the mouse technique when the
GUI button is larger than 3 square centimeters. This is
observed in both 5cm distance case and 1cm distance
case. However, in case the button is smaller than 3
square centimeters, the combination technique is less
effective than the mouse technique. Especially, in case
the button size is one square centimeter, the
combination technique is extremely ineffective in both
5cm distance case and 1cm distance case.
Considering both the number of misses and the task
completion time, we find it is almost impossible to
select the small button by the combination technique.
Since a small GUI button of one square centimeter is
often used in general GUIs, we need to improve our
technique.
3.3 Problems in combination technique
As a result of the pilot experiment, we find it is
difficult to select small GUI buttons by the
combination technique. We consider reasons of the
difficulty are:
In case the GUI button is small, the user cannot keep
the cursor pointing on a small GUI button because of
the characteristic c3 (the jittery motions during a
fixation). On the other hand, in case the GUI button is
large, the path of jittering gazing point is inside the
area of the GUI button (see Figure 6).
In case the GUI button is small, because of the
characteristic c4 (measurement accuracy is not high
enough), the eye tracking device often cannot detect
the correct GUI button on which the user is actually
gazing. On the other hand, if the GUI button is large,
the GUI button holds the cursor in the correct GUI
button area even if the measurement error does exist
(see Figure 7).
4. ADJUSTMENT TECHNIQUES
In order to reduce the misses and to make our
technique more efficient, we propose two adjustment
techniques that assist the move operation: automatic
adjustment and manual adjustment. In the automatic
adjustment, the interface adjusts the cursor position
automatically on to the nearest GUI button. In the
manual adjustment, the user can move the cursor
roughly by an eye, and then move it onto the GUI
button precisely by a mouse.
4.1 Automatic adjustment technique
If a user uses the automatic adjustment technique, the
closest GUI button from the place where the user is
looking (gazing point) is selected automatically when
the user did the push operation. If the gazing point is
already in the area of GUI button, the behavior of the
cursor in the automatic adjustment technique is same
as the combination technique with no adjustment.
Even if the gazing point is out of an area of GUI
button because of the measurement error and/or jittery
motions, the closest button from the user’s gazing
point will be selected. This technique is useful
especially in case a GUI button is very small (see
Figure 8).
Furthermore, the combination technique with
automatic adjustment might be more effective than the
combination technique without adjustment technique
because if we use the automatic adjustment, we do not
need to look at the GUI button precisely. We can look
at the GUI button roughly to select it.
4.2 Manual adjustment technique
If a user uses the manual adjustment technique, the
input device for the move operation is switched from
the eye tracking device to a mouse when the user
moves the mouse by his/her hand. Position of the
cursor is kept when we switch the input device. The
user of the manual adjustment can use the eye tracking
device for rough cursor movement and can use the
mouse for the delicate adjustment. (see Figure 9). The
behavior of the cursor in the manual adjustment
technique is same as the combination technique
without adjustment if the user did not move the mouse.
If the user found that it is difficult to move the cursor
onto an area of a certain GUI button because of the
measurement error and/or jittery motions, the user can
switch the input device into the mouse. If the user
switches the input device properly, the misses may be
reduced.
5. EXPERIMENT
5.1. Design of the experiment
We have conducted an experiment to evaluate the two
adjustment techniques.
The design of the experiment (subject, task and
collected data) is the same as the pilot experiment,
except the techniques we used. The subject is asked to
execute the task twice: once by the combination
technique with automatic adjustment and once by the
combination technique with manual adjustment.
5.2 Result of the experiment
The numbers of misses observed in tasks are shown in
figure 10. Task completion times are shown in figure
11. In order to make it easy to compare two
adjustment techniques with the mouse technique, the
results of the pilot experiments are also shown in
figure 10 and 11.
Combination
14
Combination
Automatic
Manual
Mouse
Manual
67.29
Mouse
Automatic
Combination
83.71
16
Combination
Automatic
Automatic
Mouse
Manual
Manual
Mouse
Figure 11. Task completion time
Figure 10. The number of misses
Misses
Unlike the result of pilot experiment, the distance
between GUI buttons has an influence on the number
of misses and the task completion time.
From figure 10, in the 5cm distance layout, it is clear
that the number of misses is very few in both
automatic adjustment case and manual adjustment
case. If we use the adjustment techniques, we can
decrease the number of misses as compared with the
case where the combination technique without any
adjustment technique is used. However, in the 1cm
distance layout, many misses were observed when the
user used the automatic adjustment technique. As the
size of the GUI button gets smaller, the number of
misses becomes larger. On the other hand, when the
user uses the manual adjustment, almost no miss was
observed. This means we can use the manual
adjustment technique without any stress.
square centimeters. If the size of a GUI button is 1
square centimeter, it is almost impossible to select
with using automatic adjustment. On the other hand,
manual adjustment technique showed good
performance even if the size of the button is 1 square
centimeter.
Comparisons among the techniques
So far, the manual adjustment showed best
performance among four techniques (combination,
automatic, manual, and mouse) in all the situations.
Table 1 summarizes the result of the experiment. In
the columns of task completion time, comparisons
between proposed techniques and the mouse
technique are shown. In the right column, sums of
misses are shown. From table 1, we see that the
manual adjustment showed very few misses; and also,
it showed better performance than the mouse, in all
the situations. Therefore, we believe that the manual
adjustment is useful in general GUI environments that
have many tiny GUI buttons placed closely each other.
Task Completion Time
6. CONCLUSIONS AND FUTURE TOPICS
In the 5cm distance layout, by using either automatic
adjustment technique or manual adjustment technique,
the user can select the GUI button faster than by using
the mouse technique. Furthermore, automatic
adjustment showed better performance than manual
adjustment. However, in the 1cm distance layout, the
automatic adjustment is less efficient than the mouse
technique when the GUI button is smaller than 4
We have proposed a new approach of the eye gaze
interface in which computer users use both their eye
and hand in doing operations in general GUI
environments. The basic idea of our approach is to use
the eye for moving a cursor and to use the hand for
identifying a GUI button to be selected. To improve
the accuracy and efficiency of cursor movement by an
Table1. Comparison between adjustment techniques
Task completion time
Far
Near
Adjustment
Large
Small
Large
No Adjustment
Automatic
Manual
Faster
Faster
Faster
Slower
Faster
Faster
Faster
Faster
Faster
Sum
of
Small misses
Slower
49
Slower
26
Faster
2
Far and Near stands for the distance between GUI buttons.
Large and Small stands for the size of GUI buttons.
eye, the proposed interface provides two types of
cursor adjustment technique: an automatic adjustment
technique and a manual adjustment technique.
The proposed interface has been applied to an
experiment in which various GUI environments were
employed. The results showed that the proposed
interface augmented by manual adjustment is superior
to the mouse interface in terms of accuracy and
efficiency of GUI button selection even if many small
GUI buttons have been placed with closely. This result
indicates that the manual adjustment technique is
useful in general GUI environments that have many
tiny GUI buttons placed closely each other.
In the future, we are to extend our technique for the
drag-and-drop operation; and also, we are planning to
use a keyboard and an eye together to operate GUI
objects concerning to the text input. We believe that
incorporating an eye with the present GUI
environments will lead us to the more useful,
comfortable and efficient interfaces in the near future.
5.
6.
7.
8.
9.
10.
Reference
1.
2.
3.
4.
Ebisawa, Y., Ohtani, M. and Sugioka, A.
Proposal of a zoom and focus control method
using an ultrasonic distance-meter for
video-based eye-gaze detection under free-head
conditions, Proc. the IEEE Engineering in
Medicine and Biology Society (1996), CD ROM
Edition.
Findlay, J.M., Walker, R. and Kentridge, R.W.
EYE MOVEMENT RESEARCH Mechanisms,
Processes and Applications, Vol. 6 of Studies in
visual
information
processing
(1995).
ELSEVIER.
Hansen, J.P. et al. Eye-Gaze Control of
Multimedia Systems, Symbosis of Human and
Artifact, Anzai, Y., Ogawa, K. and Mori, H.
(Eds.), vol. 20A (1995), 37-42, Elsevier Science.
Howarth, P. Keeping an eye on your interface:
The potential for eye-based control of graphical
11.
12.
13.
user interfaces (G.U.I’s), Proc. HCI’94 (1994).
Jacob, R.J.K. What You Look At is What You
Get:
Eye
Movement-Based
Interaction
Techniques, Proc. CHI’90 (1990), 11-18.
Kuno, Y., Yagi, T., Fuji, H., Koga, K. and
Uchikawa, Y. Development of Eye-gaze Input
Interface using EOG, Transactions of Information
Processing Society of Japan, vol. 39, no. 5
(1998), 1455-1462. (in Japanese)
Ohno, T. Features of Eye Gaze Interface for
Selection Tasks, 3rd Asia Pacific Computer
Human Interaction (1998), IEEE Computer
Society, 176-181.
Ohno, T. Quick Menu Selection Task with Eye
Mark, Transactions of Information Processing
Society of Japan, vol. 40, no. 2 (1999), 602-612.
(in Japanese)
Salvucci, D.D. Inferring Intent in Eye-Based
Interfaces: Tracing Eye Movements with Process
Models, Proc. CHI’99 (1999), 254-261.
Shaw, R., Crisman, E., Loomis, A. and
Laszewski, Z. The Eye Wink Control Interface:
Using the Computer to Provide the Severely
Disabled with Increased Flexibility and Comfort,
3rd
Annual
IEEE
Symposium
on
Computer-Based Medical Systems (1990),
105-111.
Velichkovsky, B.M. and Hansen, J.P. New
Technological Windows into Mind: There is
More in Eyes and Brains for Human-Computer
Interaction, Proc. CHI’96 (1996), ACM Press,
494-503.
Velichkovsky, B., Sprenger, A. and Unema, P.
Towards gaze-mediated interaction: Collecting
solutions of the “Midas touch problems”.
<http://www.phy.tu-dresden.de/psycho/sydney.ht
ml>.
Vertegaal, R. The GAZE Groupware System:
Mediating Joint Attention in Multiparty
Communication and Collaboration, Proc. CHI’99
(1999), 254-261.
14. Yamato, M., Monden, A., Takada, Y., Matsumoto,
K. and Torii, K. Scrolling the Text Windows by
Looking, Transactions of Information Processing
Society of Japan, vol. 40, no. 2 (1999), 613-622.
(in Japanese)
15. Yamato, M., Monden, A., Matsumoto, K., Inoue,
K., Torii, K. Button Selection for Common GUIs
Using Eye and Hand Together, Proc. AVI’2000
(to be appeared).
16. Zhai, S., Morimoto, C. and Ihde, S. Manual And
Gaze Input Cascaded (MAGIC) Pointing, Proc.
CHI’99 (1999), 246-253.