BIOSENSORS FOR MICROSLEEPS DETECTION DURING
DRIVE SIMULATIONS
PhD. Student C. Zocchi IEEE member, PhD Student A. Giusti, Prof. A.Rovetta
Robotics Laboratory , Mechanical Department,Politecnico of Milan,Piazza Leonardo da Vinci 32, Milan, Italy
[email protected], [email protected], [email protected]
Dr. F. Fanfulla
Sleep Centre, Rehabilitation Institute of Montescano, IRCCS, S. Maugeri Foundation, Pavia, Italy
[email protected]
Keywords:
Biomedical Signal Processing, Drive Simulations, Quantitative EEG Signals, Microsleep, Statistics
Biorobotics, Biomechanical Devices
Abstract:
This paper deals with the methodology of using biorobotics principles for creating a new intelligent system,
to improve safety in an automobile transportation. Many projects in European Union programs are devoted
to the increase of safety in automotive, in order to reduce deaths and accidents down to 50% in the next few
years. The project here presented, named PSYCAR (feasibility analysis on a car control system by psychicphysical parameters), aims at a new driving system, able to monitor the psychophysical state of the driver,
in order to avoid accidents due to sleep attacks. During 30min drive simulations, galvanic skin resistance,
perhiperal temperature, hearth rate variability beat to beat, beats per minute, electrocardiogram (EKG) and
electroencephalogram (EEG) were measured: pieces of signal related to microsleep apparences (deteced
from EEG analysis) were extracted and then statistically analized. For each parameter, ranges of
significativness were identified and validated, in order to apply them in the development of a future car
controller. Among all parameters, the EKG spectrum is the most important one.
1
INTRODUCTION
The continuous monitoring of the driver’s
attention/vigilance level is the only way to detect an
elevated sleep-attack risk. PSYCAR project, funded
by European Union (EU) in a Regional plan, starting
from Lombardy Italian Region and Austrian Region,
is one of the projects aiming to the determination of
the correct psycho-physical parameters to be
monitored in the driver and car system to increase
safety.
Driving under the influences of drowsiness will
cause (Stoohs 2006, Moller 2006, Philip 1999):
1)longer reaction time, which may lead to higher
risk of crash, particularly at high speeds; 2)vigilance
reduction including non-responses or delaying
responding where performance on attentiondemanding tasks declines with drowsiness; 3)deficits
in information processing, which may reduce the
accuracy and correctness in decision-making. Many
factors can cause drowsiness or fatigue in driving
including lack of sleep, long driving hours, use of
sedating medications, consumption of alcohol and
some driving patterns such as driving at midnight,
early morning, mid afternoon hours, and especially
in a monotonous driving environment.
Two major categories of methods have been
proposed to detect drowsiness in the past few years:
one focuses on detecting physical changes during
drowsiness by image processing techniques. They
use optical sensors or video cameras to detect eyeactivity changes in drowsiness and can achieve a
satisfactory recognition rate (i.e. E.U. project named
“DRIVESAFE”). The other methods, followed in
the PSYCAR project, are about the measure of
driver’s physiological changes, such as Heart-Rate
Variability beat to beat (HRVb-b) and beats per
minute, electrocardiogram (EKG), galvanic skin
response (GSR), pehiperal temperature (THE) and
electroencephalogram (EEG), as a means of
detecting the human cognitive states (Vuckovic
2002, Roberts 2000, Khalifa 2000, Wilson 2000).
2
MATERIALS AND METHODS
2.1 Subjects
B
A
11 subjects (10 men and 1 woman; age range, 24 to
51; mean age 29,7±9 years), with not particular
pathologies, were studied during one or more 30min
driving simulation. These healthy subjects had no
clinical evidence of snoring or sleep apnoeas and no
complaint of Excessive Daytime Sleepiness (EDS).
They were recruited between students of the
Robotics Laboratory (Politecnico of Milan) and the
Linz University (Austria). Informed consent was
obtained from all subjects
2.2 Simulator System
A simulator system has been developed and consists
of a computer games steering wheel (Logitech
Momo) with pedals and a simulated path projected
in front of the driver, giving him the impression of
actually driving (Fig. 1A). GSR (Galvanic Skin
Response), HRV (Heart Rate Variability) and THE
(peripheral body temperature) sensors are placed on
the right hand of the driver instead of the steering
wheel, in oder to reduce movement artifacts (Fig.
1B). A mechanical platform was also developed at
the Keplero University of Linz, for simulating the
car’s behaviour on the road.
According to previous study (Rada 1995), GSR
was measured using 30mm² non polarizable
Ag/AgCl electrodes placed on the median and the
ring finger of the right hand with adhesive tape.
Typical GSR values fell in the range [150;300]103Ω.
For the HRV measure a photoplethysmographic
sensor was placed on the forefinger (fig. 2) and for
THE a thermal inertia thermistor was used. From the
HRV measured, the R-R form and the beats per
minutes (BTS) are calculated.
2.3 Experimental Procedure
All subjects executed one or more 30min driving
simulation. The car at the start of every simulation
session is always positioned at the same point of the
virtual circuit. Each subject, before driving on the
simulation for the first time is also trained to use the
simulator and to always follow the same pre-defined
route for 5min. At the same time 5min basal EEG
was acquired and used to normalize EEG data.
Figure 1. A) Simulator System and a driver during test
seated on the platform chair and B) GSR and HRV
placement on the right hand: THE sensor was fixed on the
middle of the palm.
2.4 EEG Recordings and Analysis
EEG recording (Embla S7000 and Somnologica
Software) was performed using standard procedures
and scored manually in a 30-seconds epoch
according to Rechtschaffen and Kales’ criteria
(Rechtschaffen 1968). The equipment provided
simultaneous measurements of EEGs (C3-A2;C4A1;O1-A2;O2-A1), EOGs, submental EMG, EKG,
airflow by nasal cannula, thoracic and abdominal
respirogram by means of a plethysmographic
method (X-trace, Embla), SaO2 by means of a pulseoximeter. All the recordings were separately
analyzed by two expert technicians individually and
independently and finally reviewed by the physician.
Microsleeps were scored as follows (Priest,
2001): a period of at least 3s with a ϑ(4–7Hz)
rhythm replacing a α rhythm or appearing on a
background of desynchronized EEG on all four EEG
channels, and without eye-blinking artefacts. Slow
eye movements were accepted. The presence of α
rhythm was considered as wakefulness and thus
excluded microsleep. No maximal length was
defined for microsleep, so that it could evolve into
established sleep if it lasted long enough.
3
STATISICS
From the EEG, microsleeps (MS) were identified for
all subjects and then they were extracted, adding
1min basal EEG before (W1) and after (W2) (fig. 2).
W1
MS
W2
Figure 2: MS intervals.
3.1 Biosignals analysis
For all subjects, Hearth Rate Variability beat to beat
(HRb-b), beats per minute (BTS), Galvanic Skin
Response (GSR) and temperature (THE) ranges,
corresponding to the extract EEG ones, were
identified, fitted, plotted and statistically analysed,
using MATLAB. In order to reduce data noisy, after
the normalization, smoothing spline was used for the
fitting phase: it is constructed for the specified
smoothing parameter p and the specified weights wi.
The smoothing spline minimizes (1)
(1)
If the weights are not specified, they are assumed
to be 1 for all data points. p is defined between 0 and
1: p = 0 produces a least squares straight line fit to
the data, while p = 1 produces a cubic spline
interpolant. After the fit, W1 and MS data are
evaluated (interpolation), differentiated (I, II) and
plotted (fig. 3).
activity, while power in the high frequency range
(HF) (0.15;0.50Hz) is due to parasympathetic
activity. The frequency range around the 0.1Hz
region is called the middle frequency (MF) band
and.is more complex, as it can reflect a mixture of
sympathetic and parasympathetic activity. After the
normalisation, the fast Fourier transform (fft) was
calculated on 10s epochs and then ratios were also
calculated and plotted together with their mean
values
in
order
to
evaluate
the
Sympathetic/Parasympathetic activities, as shown in
fig. 5.
Ratio1=LF band/HFband
Ratio1=(LF band+Mfband)/HFband
(2)
(3)
Figure 5: Plot of the ratios 1(blue) and 2(red) : W1 (from 1
to 6), MS (from 6 to 7) and W2 (from 7 till 12) ranges are
plotted.
samplings
Figure 3: Plot of the W1 (from 0 to 60) and MS ranges:
the fit, the I and II derivative.
For the statistics analysis, all W1 were divided in
10s epochs. For each temporal window, minimum,
maximum, amplitude (A=max-min, compared with
the mean value), mean and standard deviation were
calculated.
3.2 Analysis of the EKG Spectrum
The Autonomic Nervous System is divided
anatomically
into
three
components:
the
parasympathetic, the sympathetic and the enteric
nervous system. The best characterization is that the
sympathetic is a quick response mobilizing system
and the parasympathetic is a more slowly activated
dampening system. The power spectrum is divided
into three main frequency ranges: the low frequency
one (LF) (0;0.08Hz) is an index of sympathetic
3.3 Analysis of the EEG Spectrum
After the normalisation, the fast Fourier transform
(fft) was calculated on 10s epochs of the C3 EEG
and four frequency bands were defined as δ(0,753,75Hz), θ(3,75-7,75Hz), α(7,75-12,75Hz), β(12,7520,25Hz). The absolute power in each of these bands
was then calculated (in µV2). Following
methodologies found in literature, different types of
analyses have been carried out:
•
α+β;δ+θ and α+θ;β (Horne, 2004)
•
relative power (Eoh 2005) (4-5-6)
•
ratios (7-8-9-10) for amplifing the
differences within the waves (Priest, 2001)
•
burst index (Eoh, 2005): index of the
increased θ or α power density in response to
sleepiness. The mean value of the first 5min
EEG was set as the threshold value and the
number of bursts above 100-150% of the
threshold value were calculated. For each
epoch the bursts number was averaged.
RPθ=θ/(θ+α+β)
RPβ=β/(θ+α+β)
RPα=α/(θ+α+β)
Ratio1=(θ+δ)/(α+β)
Ratio2=(θ+α)/β
Ratio3=β/α
Ratio4=θ/α
(4)
(5)
(6)
(7)
(8)
(9)
(10)
The plots of all the applied methods are omitted,
because of not good resolution.
4
RESULTS
Table 3: THE ranges.
RANGES
Intensity
A [°C]
StdDev
I
Low
<0,050
<0,01
II
Middle
[0,050;0,100)
[0,01;0,06)
III
High
≥0,100
≥0,06
4.4 EKG Spectrum
The EKG spectrum ranges correspond to HF band
increases, indicating relaxed and calm status due to
the parasympathetic nervous system activation
(Healey, 2005). So lower ratios and high negative
mean square error are considered (table 4):
Table 4: EKG spectrum ranges.
From the observation of all plots, ranges of
significant variations and cut-off values were
identified for each parameter, useful for the future
implementation of a control system.
A stands for Amplitude, i.e. the difference between
the maximum and minimum values in each epoch.
RANGES
Intensity
LF/HF and (LF+MF)/HF
MSE1 and MSE2
I
Low
>30
>0
II
middle
[25;30]
(-1,5;0]
III
high
<25
≥-1,5
II
Low
III
Middle
IV
High
V
Very
High
(5;10]
(2;3,5]
(10;15]
(3,5;5,25]
(15;26)
(5,25;9,3)
≥26
≥9,3
In table 5 the matches of each method in finding
the high intensity ranges are reported: MSE2 is
better, because there are 14 matches out off 22 and
was the only one considered in successive analysis
also because it’s more restrictive than MSE1.
Considering only the MSE2, in W1 intervals there
are perturbations and then the spectrum returns to
the basal trend. In fact 30s before MS there were 10
variations out off 14 (about 71%) respectively: the
driver is in a state of relaxation and gradual
vigilance decrease, due to parasympathetic activity.
Significant events have about 70% of probability
to appear from 60s to 20s before microsleep.
Table 5: Matches of the methods considered for the ranges
definition and their goodness.
4.2 Galvanic Skin Response (GSR)
METHOD
M1 LF/HF
M2 (LF+MF)/HF
MSE1
MSE2
4.1 HR Beat to Beat and Beats Ranges
Table 1: HRbeat to beat and Beats ranges.
RANGES
Intensity
I
Very
low
A
StdDev
≤5
≤2
Table 2: GSR ranges.
RANGES
Intensity
I
Low
II
Middle
III
High
A [kΩ]
StdDev
≤5
≤1650
(5;10]
(1650;3300]
(10;20)
(3300;5425)
IV
Very
High
≥20
≥5425
Significant events have about 81% of probability
to appear from 60s to 40s before the microsleep.
4.3 Pheriperal Temperature (THE)
Significant events have about 74% of probability
to appear from 60s to 10s before the microsleep,
because 14 matches out of 23 were identified.
Match
15/22
11/22
16/22
17/22
%
68
50
73
77
4.5 EEG Spectrum
Some interesting are identified (tab. 6) and are
related to α+β, δ+θ, α+θ, β/α, (α+θ)/β, RPα waves,
indicating drowsiness increase. The other methods
applied to the EEG spectrum were not considered in
this section (table 6).
In table 7 the goodness of each method (the
number of matches in determining a high intensity
range divided by all the high intensity ranges) is
presented: as stated in (Eoh, 2005), the best methods
for evaluate the transition from vigilance to sleep are
the α+β and α+θ with 12 and 10 matches out off 13
respectively.
Tab. 9: Comparison of all parameters: X stands for high or
very high intensity range. ID is the interval code; EEG and
EKG stand for spectra.
Table 6: EEG spectrum ranges: i.e. M1.1 stands for
“Method N° 1.1”.
RANGES
Intensity
M1.1 α+β
M1.2 δ+θ
M2.1 α+θ
M4 (α+θ)/β
M6 α/(α+β+θ)
M8 β/α
I
Low
≤150
≤700
≤200
≤3
≤0,2
>1,2
II
middle
(150;180)
(700;1000)
(200;300)
(3;3,5)
(0,2;0,3)
[1,2;1]
III
high
≥180
≥1000
≥300
≥3,5
≥0,3
<1
Table 7: Matches of the methods considered for the ranges
definition and their goodness.
METHOD
M1.1 α+β
M1.2 δ+θ
M2.1 α+θ
M4 (α+θ)/β
M6 α/(α+β+θ)
M8 β/α
Match
12/13
8/13
10/13
5/13
8/13
9/13
%
92
62
77
38
62
69
Considering only the best methods (M1.1 and
M2.1), in W1 intervals there are perturbations and
then the spectrum returns to the basal trend. In fact
30s before MS for M1.1 and M2.1 there were 10
(about 83%) and 7 (about 70%) variations out off 12
respectively.
The Burst index was calculated in each 10s
epoch as the number of α and θ peaks, better than
the 100% and 150% cut-offs. In table 8 the burst
indexes of the first 4 epochs (40s before MS) are
reported: θ bursts are better identifiable with about
64% of accuracy for 100% cut off.
Table 8: Burst indexes for the 100% and 150% cut-offs.
CUTOFF
α
Θ
5
100%
54%
64%
150%
51%
61%
VALIDATION
Summarizing all the previous analysis and high/very
high intensity ranges, it was possible to validate the
model proposed. In table 9 all parameters are
compared, but only 3 examples are here reported: for
the EEG spectrum methods 1.1 and 2.1 are taken
into account while for the EKG the mean square
error related to ratio2.
ID
BTS
1
X
HRb-b
2
EEG
X
X
EKG
GSR
THE
X
X
X
X
X
…
…
…
…
….
....
….
TOT
5
13
18
22
14
12
In order to validate the proposed model, initially
confusion matrixes for each parameter (beats,
HRbeat–beat, EKG spectrum, GSR and THE)
compared with the EEG spectrum were built.
Table 10: Confusion matrixes for beats and HRbb.
BTS
MS
n
ng’
MS’
4
1
5
n’
14
6
20
ng
18
7
25
HRbb
MS
N
ng’
MS’
10
3
13
n’
8
4
12
ng
18
7
25
Rows represent the real classes (i.e. MS stands
for Microsleep determinated by Dr. Fanfulla, while n
stands for no microsleep) while in the columns there
are the intervals assigned to a class after the analysis
and the ranges of significance determination.
Numbers out the diagonal indicate intervals wrong
assigned from the model to a class different from the
right one. In table 11 the results of the validation are
reported for each parameter (Baldi, 1996).
6
CONCLUSIONS
Significant variations compared with the basal trend
appear for all parameters: this demonstrates
correlations among them as also reported in (Zocchi,
2006). So measuring EKG spectrum or HRbeat-beat
is possible to indirectly and in a non invasive way
determine a drowsiness status: when the HRbeatbeat and EKG spectrum parameters are higher than
the cut-offs (i.e. very high/high intensity ranges) the
overall system can activate itself and warn the
driver. EKG spectrum referred to the EEG one, has
about 83% of goodness, because 15 intervals out off
18 were identified with high intensity. The minor
errors are related to HRbb, EKG spectrum and THE,
while the sensibility, the capacity of MS class to
good represent its objects, is the highest for the EKG
spectrum. The not MS class is good represented
from the beats.
Table 11: Analysis validation parameters.
Parameter
Error rate
Sensibility for
MS class
Sensibility for n
class
Specificity for
MS class
Specificity for n
class
Initial EntropyH0
Relative entropy
for MS class
Relative entropy
for n class
Residual Entropy
= Erel-MS+Erel-n
H Reduction =
H0 - HTot residua
Initial Gini’s
index
Gini’s index for
MS class
Gini’s index for n
class
Residual G =
G.I.rel-MS+G.I.rel-n
G Reduction =
G.I0 – G.ITot
BTS
60%
HRbb
44%
EKG
40%
GSR
52%
THE
43,5%
22,2%
55,6%
83,3%
56,3%
53,3%
85,7%
57%
0%
28,6%
62,5%
80%
76,9%
68,1%
64,3%
72,7%
30%
0%
14,44%
33,3%
85,6%
40,5%
79,4%
22,2%
88,65%
57,2%
41,7%
93,2%
40,4%
70,5%
44,1%
0
29,9%
51,1%
84,9%
84,6%
79,4%
87,1%
91,5%
0,7%
1%
6,2%
1,55%
1,71%
3,2%
20,2%
9,2%
19,1%
21,2%
13,98%
22,7%
9,5%
16,8%
10,7%
0
6,76%
12,7%
20%
19,9%
19,1%
20,74%
22,2%
0,2%
0,3%
1,1%
0,46%
0,5%
The specificity indicates the purity degree of
class, because divides objects of one class from
objects of another: in this case HRbeat-beat, THE
and beats have the best discriminatory capacity. The
Residual total Entropy, calculated as the sum of the
relative entropies of both classes, indicates that the
applied model still has some unknown information.
The best Entropy reduction is given by the EKG
spectrum.Considering the Initial Gini’s Index before
the classification, the model decreases the 1,1% of
the initial impurity quantity (for the EKG spectrum).
In conclusion for the development of a future
controller, the spectrum EKG and the beats are the
best parameter to be analysed.
REFERENCES
Stoohs R. A., Guilleminault C., Itoi A.,. Dement W. C,
Traffic Accidents in Commercial Long-Haul Truck
Drivers: The Influence of Sleep-Disordered Breathing
and Obesity, in Sleep, vol. 17(7), pp. 619-623, 2006
Moller H. J., Simulator Performance, Microsleep
Episodes, and Subjective Sleepiness: Normative Data
using Convergent Methodologies to Assess Driver
Drowsiness in Journal of psychosomatic research. vol
61(3), 2006 pp 335.
Philip P. Simple reaction time, duration of driving and
sleep deprivation in young versus old automobile
drivers in Journal of sleep research, 1999, pp8- 9.
Vuckovic A., Radivojevic V., Chen A. C. N., Popovic D.,
Automatic recognition of alertness and drowsiness
from EEG by an artificial neural network, Med. Eng.
Phys., vol. 24, pp. 349–360, Jun. 2002.
Roberts S., Rezek I., Everson R., Stone H., Wilson S.,
Alford C., Automated assessment of vigilance using
committees of radial basis function analyzers, Inst.
Elect. Eng. Science Measure. Technol., vol. 147, pp.
333–338, nov. 2000.
Khalifa K. B., Bedoui M. H., Raytchev R., Dogui M., A
portable device for alertness detection, in Proc. 1st
Ann. Int. IEEE-EMBS Special Topic Conf.
Microtechnologies in Medicine and Biology, Oct.
2000,pp. 584–586.
Wilson B. J., Bracewell T. D., Alertness monitor using
neural networks for EEG analysis, in Proc. IEEE
Signal Processing Society Workshop on Neural
Networks for Signal Processing X, vol. 2, Dec. 2000,
pp. 814–820.
Rechtschaffen A., Kales A.. A manual of standardized
terminology, techniques and scoring system for sleep
stages in human adults. U.S. Government Printing
Office, Washington. DC, 1968.
Priest B., Brichard C., Aubert G., Liistro G.,. Rodenstein
D. O Microsleep during a Simplified Maintenance of
Wakefulness Test. A Validation Study of the OSLER
Test. Am J Respir Crit Care Med Vol 163., 2001 pp
1619–1625.
Horne J. A.,. Baulk S. D , Awareness of sleepiness when
driving Psychophysiology, 41 (2004), 161–165
Eoh H. J., Chung, M. K. Kim S. H.,
Electroencephalographic study of drowsiness in
simulated driving with sleep deprivation, International
Journal of Industrial Ergonomics 35 (2005) 307–320
Healey, Detecting stress during real-world driving tasks
using physiological sensors, IEEE Transactions on
Intelligent Transportation Systems, vol. 6, pp. 156,
2005.
Schier M. A., Changes in EEG alpha power during
simulated driving: a demonstration International
Journal of Psychophysiology, vol. 37, pp. 155, 2000.
Reyner, Falling asleep whilst driving: are drivers aware
of prior sleepiness? International Journal of Legal
Medicine, vol. 111, pp. 120, 1998.
Reyner, Efficacy of a 'functional energy drink' in
counteracting driver sleepiness, Physiology Behavior,
vol. 75, pp. 331, 2002.
Rada, Dittmar A., Delhomme G. and al. Bioelectric and
microcirculation coutaneus sensors for the study of
vigilance and emotional response during tasks and
tests in Biosensors and Bioelectronics, vol. 10, 1995,
pp. 7-15
Zocchi C., Rovetta A., Fanfulla F., Physiological
Parameters Variation during Driving Simulations,
International Journal of Mechanics and Control
Vol.07, No02, 2006
Baldi P., Appunti di Metodi Matematici e Statistici, Clueb
1996