特 集
特集 Drowsiness Detection Using Spectrum Analysis of
Eye Movements and Effective Stimuli to Keep
Drivers Awake＊
中川
剛
Tsuyoshi NAKAGAWA
佐々木
健
Ken SASAKI
河内泰司
有光知理
神野雅行
Taiji KAWACHI
Satori ARIMITSU
Masayuki KANNO
保坂
寛
Hiroshi HOSAKA
This paper presents a driving safety system based on a driving simulator. The system evaluates the driver’s level
of drowsiness by applying LF/HF analysis to the driver’s eye movements. Drowsiness was detected by an increase
of power in the low frequency band. When drowsiness was detected, stimuli were presented to raise the driver’s
level of awareness. Evaluation of stimuli has shown that stimuli involving active bodily motion, such as singing,
chewing, and stretching muscles, are effective means of keeping drivers awake. Passive stimuli, such as listening
to music, alarms, and lights produce a waking effect of shorter duration.
Key words: Drowsiness detection, Driving simulator, Eye movements, Lane deviation, Awakening stimulus,
Awakening temporal duration, Physiological signals, Subjective drowsiness levels
１．INTRODUCTION
２．DRIVING SIMULATOR
Driving with drowsiness is one of the main causes of car
A driving simulator was constructed with a complete set
accidents. In order to prevent drivers from drowsy driving,
of a driving cockpit taken from a real car, including
there is a strong demand for safety driving systems. In this
dashboard, steering wheel, seat, and pedals as shown in
research, a safety driving system that detects driver ’
s
Fig. 2. A screen was placed in front of the driver’
s seat to
drowsiness before it leads to dangerous driving has been
show a computer-generated motion picture of a
developed. This system consists of three parts; sensing,
monotonous freeway driving (Fig. 3) that induces
evaluation and stimulation (Fig. 1).
drowsiness to the driver.
The sensing part monitors driverﾕs physiological data
and surroundings. The evaluation part determines whether
３．SENSING AND EVALUATION
the driver is drowsy or not. When the driver becomes
3.1
drowsy, the stimulator is activated to awaken the driver.
Methods for detecting driver ’
s drowsiness can be
This system was combined with a driving simulator to
classified into two categories; a) analysis of drivers body
evaluate drowsiness detection algorithm and effectiveness
motion such as head motion, and b) analysis of
Drowsiness detection
of the stimuli to keep the driver awake.
Steering wheel
Driver
Sensing
Screen
Stimulation
Dashboard
Seat
Evaluation
Pedals
Fig. 1 Structure of the safety driving system
Fig. 2 Driving simulator
＊Reprinted with permission from“Proceedings of Smart Systems 2006 & ICMA 2006.”
−113−
デンソーテクニカルレビュー Vol.12
No.１ 2007
electrical field related to the orientation of the eyes in the
surrounding tissues of the eyes. Horizontal eye movement
was measured by placing a disposable silver chloride
electrode on the outer corner of each eye, and the third
electrode at the center of the forehead for reference.
Electric potential difference between the two electrodes at
the corners was amplified by an instrumentation differential
amplifier. The analog output was digitized at sampling
Fig. 3 Monotonous freeway picture on screen
frequency of 100Hz. Power spectrum of the signal was
obtained by 1024-point Fast Fourier Transform. LF/HF
physiological data such as eye-blink rate and heart rate.
analysis is best known for its application on ECG. It is
When a driver becomes drowsy, the driver’
s head begins to
known that the power in low frequency spectrum increases
sway or tilt and the car may drift off the course. These
when a human becomes drowsy (Fig. 5).
1)
physical symptoms, however, become apparent only after
the driver starts to doze off. On the other hand,
3.2
Evaluation index
physiological signals start to change in earlier stages of
The next step was to find the optimum low frequency
drowsiness. Therefore, physiological data is more suitable
band for drowsiness detection using LF/HF analysis on
for drowsiness detection for prevention of dangerous
EOG. Analysis of data obtained from 10 subjects has
driving.
shown that the ratio between the power in 0 – 0.3 Hz band
Assuming that the lane deviation was a direct indication
and the power in (0 – 0.3) + (3 – 10) Hz band had the
of drowsiness, we first measured correlations between lane
highest correlation with the lane deviation (correlation
deviation and the following physiological signals: pulse
coefficient: 0.81). Using this result, we have defined an
rate, ratio of low-frequency (LF) to high-frequency (HF)
alertness level indication KE by the following equation (1).
power from spectral analysis of R-R intervals of ECG, eye-
The value of KE increases when the driver becomes drowsy.
movement measured by electro-oculography (EOG), body(dB)
subject of this experiment drove the driving simulator for
104
one hour.
We have chosen EOG for drowsiness detection because
Power
surface temperature, skin resistance, and breath rate. The
102
100
it showed the highest correlation with the lane deviation
10-2
when LF/HF analysis was applied on EOG. The method of
10-4
0.1
1
10
50 (Hz)
EOG is shown in Fig. 4. The permanent electric potential
Frequency
difference between the cornea and the retina generates
(a) When driver is awake
104
Power
Cornea
(dB)
Silver chloride electrode
++ +
−− −
++ +
−− −
Retina
102
100
10-2
10-4
+−
0.1
1
10
50 (Hz)
Frequency
(b) When driver is drowsy
Fig. 5 Power spectrum of eye movements
Fig. 4 The method of electro-oculography (EOG)
−114−
KE
=
Power (0 – 0.3 Hz)
Power (0 – 0.3 Hz) + Power (0 – 10 Hz)
(1)
The evaluation module in the safety system uses this KE
for drowsiness detection. If the drivers become drowsy, the
Correlation cofficient
特 集
value of K E becomes near 1 in our current evaluation
system.
1.0
0.8
0.6
0.4
0.2
0.0
-0.2
-0.4
-0.6
-0.8
-1.0
Body-surface
temperature
Breath rate
KE
Pulse rate LF/HF
(ECG)
Physiological signals
The correlation between lane deviation and the
physiological signals shows in Fig. 6.
Fig. 6 Average value of correlation coefficient
between physiological signals and lane
deviation
4. EFFECTIVE STIMULI
4.1
Skin resistance
Classification of stimuli
Stimuli were classified into active stimuli and passive
intensity of stimulation were selected according to
stimuli, and the passive stimuli were further classified into
commercial products for primary investigation. Effects of
chemical stimuli and external physical stimuli.
these stimuli were compared to find stimuli that kept the
We have evaluated the following stimuli for awakening
the driver. Chew a piece of gum (peppermint and/or
driver awake for 15 minutes, which is an average time to
reach the next rest stop on freeways.
caffeine), attach a caffeine sheet, take 50 ml of water, spray
When the subject became drowsy, one of the stimuli was
aroma oil (A: relaxer or B: refresher), turn on a 40W light
given to the subject, and duration of the awakening effect
bulb placed on the center of the dashboard, set off a alarm
was measured.
sound (approximately 70 dB), play driver’
s favorite songs,
turn on an electric fan, give advice to do light exercises
4.2
Method
such as stretching shoulder and neck, play last and first
We asked two subjects in their twenty’
s, one male and
word game with the examiner, sing driver’
s favorite songs
one female, to make an oral report on their subjective
to music (karaoke) (Fig. 7). Stimuli and their quantities and
drowsiness level every minute while driving in the
Singing
Utterance
Active stimuli
Stimuli to make
driver awake
Chemical stimuli
Passive stimuli
External physical
stimuli
Movement
Stretching
Chewing
Gum
Swallowing
Drinking water
Taste
Mint
Nerve
Caffeine
Smell
Aroma oil
Sight
Light
Hearing
Music
Alarm sound
Touch
Fig. 7 Classification of stimuli to make driver awakeStimuli written
in bold letters were experimented.
−115−
Last and first
word game
Wind
デンソーテクニカルレビュー Vol.12
simulator. A notice was given by a sound every minute to
No.１ 2007
Table 1 Driver’s subjective drowsiness levels
report the drowsiness level. Drowsiness level was classified
into five levels as given in Table 1.
4.3
Comparison of effective durations
One of the stimuli was activated when the driver ’
s
drowsiness level became 3, and the experiment was
terminated when the drowsiness level reached 5. We
compared the effectiveness of the stimuli by measuring the
Level 1
Level 2
Awake
Slightly drowsy
Level 3
Drowsy
(micro sleep occurs once a minute)
Level 4
Very drowsy
(micro sleep occurs few times in a minute)
Level 5
Unable to drive
(Micro sleep is a brief sleep lasting up to a few seconds.)
time it took from drowsiness level 3 to 5.
Without any stimuli, the drowsiness level reached 5 in
chewing a caffeine gum (Fig. 9(b)) could keep drivers
awake for more than 30 minutes, which was more than two
times longer of the targeted duration. Caffeine in the
Drowsiness level
only 6 minutes as shown in Fig. 8. Karaoke (Fig. 9(a)) and
5
4
3
2
1
0
chewing gum has direct effect on the nervous system. The
Duration
10
20
effectiveness of karaoke is due to the forced physical
activity of singing. A similar stimulus such as speaking and
playing last and first word game (Fig. 9(c)), which also
30
40
50
60
Time (min)
Fig. 8 Drowsiness level transition patterns
without stimuli
involve forced active response, could only keep the subject
awake for less than 20 minutes. Subjects have reported that
playing a word game while driving became monotonous
same music as karaoke without singing (Fig. 9(d)) could
and that they quickly lost interest in it. Listening to the
keep drivers awake less than 15 minutes.
5
4
Drowsiness level
Drowsiness level
5
Duration
3
2
1
0
10
20
30
40
50
4
3
2
1
0
60
Duration
10
20
Time (min)
(a) Singing driver’s favorite songs to music (karaoke)
50
60
50
60
5
Drowsiness level
Drowsiness level
40
(c) Playing last and first word game
5
Duration
4
3
2
1
0
30
Time (min)
10
20
30
40
50
4
2
1
0
60
Duration
3
10
20
30
40
Time (min)
Time (min)
(d) Listening to the same music as karaoke
without singing
(b) Chewing caffeine gum
Fig. 9 Drowsiness level transition patterns of different stimuli
−116−
特 集
A sound alarm could keep the drivers awake for mere 10
muscles around mouth. On the other hand, chewing a piece
minutes. These results suggest that effects of passive
of gum (Fig. 12(c)) did not affect EOG and also KE. One of
stimuli are temporary, while stimuli that force the driver to
the reasons might be that chewing caused constant
actively do something have longer awakening effect (Fig.
vibration; therefore the frequency of chewing was cut off.
10). Although active stimuli are more effective, they are
In addition, with long period chewing, the gum would be
not effective unless the driver recognize the warning and
tasteless. Therefore, the subjects might stop chewing and
follow the suggestions. Passive stimuli, although less
that would cause the high value of correlation coefficient.
effective, are more practical and reliable.
While playing last and first word game (Fig. 12(d)),
subjects frequently made humming sound with thinking.
This sound might be a noise for sensing eye movement with
4.4 Influence of stimuli on measurement of eye
EOG. Listening to the same music as karaoke without
movement
Those stimuli, which cause muscle movements and
singing did not highly correlate with lane deviation.
vibration, have influences for EOG and KE. If the data from
Listening to music might have a possibility to make
EOG are inaccurate, it is hard to evaluate driver ’
s
humming noise unconsciously or cause the contraction of
drowsiness level with K E . Therefore, the correlation
muscles near vocal cords (Fig. 12(e)). One of the results of
between K E and the lane deviation during drivers ’
light is indicated in Fig. 12(f), but the other one ’
s
drowsiness level from 3 to 5 was compared. Figure 11
correlation coefficient was 0.10. According to the subject’
s
indicates average values of correlation coefficient. Figure
face image during experiment taken by CCD camera, this
12 shows some specific examples. Singing songs made the
subject squinted eyes because of bright light. This action
worst noise for KE (Fig. 12 (b)). It was possible to be one
might induce muscles tension.
of the reasons that the vibration of subjects’
vocal cords and
Nothing
Karaoke
Last and first
Stimuli to make drivers awake
Stretching
Active stimuli
Water
Caffein gum
Mint gum
Caffein
Aroma A
Aroma B
Passive stimuli
Light
Music
Alarm
Wind
0
10
20
30
Effective duration (min)
Fig. 10 Effective duration of stimuli
−117−
40
50
60
デンソーテクニカルレビュー Vol.12
No.１ 2007
Average value of correlation coefficient
1.0
0.8
0.6
0.4
Wind
Alarm
Music
Light
Aroma B
Aroma A
Caffein
Mint gum
Caffein gum
Water
Stretching
Karaoke
Nothing
0.0
Last and first
0.2
Stimuli to make drivers awake
Fig. 11 Average values of correlation coefficient between lane deviation and stimuli
Correlation coefficient = 0.84
0.3
Lane deviation
0.2
0.2
0.1
KE
0
10
20
30
Time (min)
40
50
0.0
0.6
0.3
0.4
0.2
KE
0.2
0.0
0
0.3
Lane deviation
0.2
0.2
0.1
KE
KE
10
20
30
Time (min)
40
50
Lane deviation (m)
0.4
KE
Lane deviation (m)
Drowsiness level 3
0
0.8
0.4
0.2
Lane deviation
0.2
0.2
KE
KE
0.1
50
Lane deviation (m)
0.4
KE
Lane deviation (m)
0.3
40
0.1
KE
0
10
20
30
Time (min)
40
50
1
0.4
Drowsiness level 3
20
30
Time (min)
0.2
Lane deviation
0.0
Correlation coefficient = 0.46
0.8
10
0.3
(e) Listening to the same music as karaoke
without singing
0.5
0
0.0
0.4
Drowsiness level 3
0.6
0.0
0.0
Correlation coefficient = 0.88
1
0.0
50
0.5
(b) Singing driver’s favorite songs to music (karaoke)
0.6
40
1
0.5
0.6
0.0
20
30
Time (min)
Correlation coefficient = 0.45
Correlation coefficient = 0.16
1
0.4
10
0.1
(d) Playing last and first word game
(a) Without stimuli
0.8
0.4
Lane deviation
KE
0.4
Drowsiness
level 3
0.8
KE
0.4
0.5
0.8
0.6
0.2
0.2
0.1
KE
0
10
20
30
Time (min)
Fig. 12 Correlation coefficient between lane deviation and KE
−118−
0.3
Lane deviation
(f) 40 W light bulb
(c) Chewing mint gum
0.4
0.4
0.0
0.0
0.5
Drowsiness level 3
40
50
0.0
KE
0.6
1
Lane deviation (m)
Drowsiness level 3
0.8
0.0
Correlation coefficient = 0.58
0.5
KE
Lane deviation (m)
1
特 集
５．CONCLUSION
will provide us with safer and more comfortable driving
An evaluation of safety driving system consisting of
cockpit.
drowsiness detection and awakening stimulation has shown
that the system could keep the driver at lower drowsiness
REFERENCES
level for 10 to 30 minutes. Drowsiness was detected by
1) N. Wright and A. McGown,“ Vigilance on the civil
increase of power in the lower frequency spectrum of
flight deck: incidence of sleepiness and sleep during
driver’
s eye movements. Stimuli involving driver’
s active
long-haul flights and associated changes in physiological
motion were more effective than passive stimuli. Our future
parameters”, ERGONOMICS, Vol. 44, No. 1 (2001), pp.
works include development of eye movement sensor based
82-106.
on image processing, refinement of LF/HF analysis of eye
movement, and evaluation of other stimuli for awakening
the driver.
In the near future, the system consisting of non-contact
drowsiness sensor and automatic awakening stimulation
666666666666666666666666666666666666
＜著 者＞
河内 泰司
中川 剛
（かわち たいじ）
（なかがわ つよし）
基礎研究所
情報安全システム開発部
生体情報センシング及び状態推定
ドライバモニタ（体調モニタ）の
技術の研究に従事
開発に従事
有光 知理
神野 雅行
（ありみつ さとり）
（かんの まさゆき）
2007年東京大学大学院新領域創成
2006年東京大学大学院新領域創成
科学研究科博士課程終了
科学研究科修士課程終了
現在はデンソー基礎研究所に勤務
現在は日興シティグループ証券に
勤務
保坂 寛
佐々木 健
（ほさか ひろし）
（ささき けん）
東京大学大学院新領域創成
東京大学大学院新領域創成
科学研究科教授
科学研究科教授
環境情報マイクロシステム分野に
環境情報マイクロシステム分野に
おける研究に従事
おける研究に従事
−119−