Camera-based sleepiness detection Final report of the project SleepEYE Authors

Camera-based sleepiness detection Final report of the project SleepEYE Authors
ViP publication 2011-6
Camera-based sleepiness detection
Final report of the project SleepEYE
Authors
Carina Fors, VTI
Christer Ahlström, VTI
Per Sörner, Smart Eye
Jordanka Kovaceva, Volvo Cars
Emanuel Hasselberg, Smart Eye
Martin Krantz, Smart Eye
John-Fredrik Grönvall, Volvo Cars
Katja Kircher, VTI
Anna Anund, VTI
www.vipsimulation.se
Preface
SleepEYE is a collaborative project between Smart Eye, Volvo Cars and VTI (the
Swedish National Road and Transport Research Institute) within the competence centre
Virtual Prototyping and Assessment by Simulation (ViP). The main objectives of the
project were:


Development and evaluation of a low cost eye tracker unit.
Determination of indicator thresholds for sleepiness detection.
The project started in the end of 2009 and ended in 2011. It was financed by the competence centre Virtual Prototyping and Assessment by Simulation (ViP).
Participants from Smart Eye were Martin Krantz, Per Sörner and Emanuel Hasselberg.
Participants from Volvo Cars were John-Fredrik Grönvall, Jordanka Kovaceva and
Håkan Gustafsson.
Participants from VTI were Anna Anund, Carina Fors, Christer Ahlström, Katja
Kircher, Beatrice Söderström, Håkan Wilhelmsson, Mikael Bladlund, Fredrik
Gustafsson, Anders Andersson and Sara Nygårdhs.
Kenneth Holmqvist (Lund University), Torbjörn Åkerstedt (Stockholm University) and
Lena Nilsson (ViP Director) have reviewed the report and provided valuable feedback.
Thanks to all of you who have contributed to this project.
Linköping, September 2011
Carina Fors
ViP publication 2011-6
Cover: Katja Kircher, VTI
Quality review
External peer review was performed on 26 June 2011 by Kenneth Holmqvist, Lund
University and on 9 July 2011 by Torbjörn Åkerstedt, Stockholm University. Carina
Fors has made alterations to the final manuscript of the report. The ViP Director Lena
Nilsson examined and approved the report for publication on 16 November 2011.
ViP publication 2011-6
Table of contents
Executive summary ............................................................................................ 5
1 1.1 1.2 Introduction .............................................................................................. 7 Aims and limitations ................................................................................. 8 Related publications ................................................................................ 8 2 2.1 2.2 2.3 Driver sleepiness ..................................................................................... 9 Indicators ............................................................................................... 10 Algorithms and fusion of indicators ........................................................ 11 Conclusions and recommendations for SleepEYE ................................ 12 3 3.1 3.2 Development of the Smart Eye embedded camera system ................... 14 Cost optimisation ................................................................................... 14 Implementation of driver impairment indicators ..................................... 16 4 4.1 4.2 4.3 4.4 Experiments ........................................................................................... 17 Design and procedure ........................................................................... 17 Field experiment .................................................................................... 19 Simulator experiment ............................................................................. 21 Database ............................................................................................... 22 5 5.1 5.2 Evaluation of the Smart Eye embedded camera system ....................... 24 Method ................................................................................................... 24 Results ................................................................................................... 25 6 6.1 6.2 Determination of sleepiness thresholds ................................................. 37 Method ................................................................................................... 37 Results and analysis .............................................................................. 45 7 7.1 7.2 7.3 7.4 Discussion and conclusions ................................................................... 56 The Smart Eye embedded camera system ............................................ 56 Sleepiness thresholds and classifier ...................................................... 57 Contribution to ViP ................................................................................. 58 Conclusions ........................................................................................... 59 References ....................................................................................................... 60 Appendices
Appendix A
Eye movement based driver distraction detection algorithms
Appendix B
Karolinska sleepiness scale
Appendix C
Algorithms
ViP publication 6-2011
ViP publication 2011-6
Camera-based sleepiness detection
by Carina Fors1, Christer Ahlström1, Per Sörner2, Jordanka Kovaceva3, Emanuel
Hasselberg2, Martin Krantz2, John-Fredrik Grönvall3, Katja Kircher1 and Anna Anund1
1
Swedish National Road and Transport Research Institute, VTI
Smart Eye
3
Volvo Cars
2
Executive summary
The aims of the study were, in brief: 1) to develop and evaluate a low cost 1-camera unit
for detection of driver impairment and 2) to identify indicators of driver sleepiness and
to create a sleepiness classifier for driving simulators.
Two literature reviews were conducted in order to identify indicators of driver sleepiness and distraction. Three sleepiness indicators – blink duration, blink frequency and
PERCLOS – were implemented in the camera system.
The project included two experiments. The first was a field test where 18 participants
conducted one alert and one sleepy driving session on a motorway. 16 of the 18 participants also participated in the second experiment which was a simulator study similar
to the field test.
The field test data was used for evaluation of the 1-camera system, with respect to the
sleepiness indicators. Blink parameters from the 1-camera system was compared to
blink parameters obtained from a reference 3-camera system and from the EOG. It was
found that the 1-camera system missed many blinks and that the blink duration was not
in agreement with the blink duration obtained from the EOG and from the reference 3camera system. However, the results also indicated that it should be possible to improve
the blink detection algorithm since the raw data looked well in many cases where the
algorithm failed to identify blinks.
The sleepiness classifier was created using data from the simulator experiment. In the
first step, the indicators identified in the literature review were implemented and
evaluated. The indicators also included driving and context related parameters in
addition to the blink related ones. The most promising indicators were then used as
inputs to the classifier.
The final set of indicators were an estimated KSS value that was based on the value the
driver reported before the driving session (KSSestSR), standard deviation of lateral
position (SDLP) and fraction of blinks > 0.15 s (fracBlinks, for EOG based and 1camera-based). An optimal threshold for discriminating between KSS above and below
8 was determined for each indicator. The performances were in the range of 0.68–0.76.
Two decision trees based on the selected indicators were created: one using the
fracBlinksEOG and one using fracBlinks1CAM. The performances of the two trees were
0.82 and 0.83 respectively (on the training dataset), i.e., the overall performance of the
EOG based and the 1-camera-based classifier were similar, although individual
differences could be seen. The performance decreased to 0.66 when using a validation
dataset from another study, which illustrates the difficulties in creating a generalized
sleepiness classifier.
ViP publication 2011-6
5
6
ViP publication 6-2011
1
Introduction
Driver impairment caused by for example sleepiness, stress, visual inattention,
workload etc. needs to be predicted or detected in order to avoid critical situations and
crashes. From a scientific point of view there is a need to find suitable indicators for
detection or prediction of these driver states. Such indicators are also needed for
evaluation of the effectiveness and usefulness of warning strategies and/or interfaces
when such driver states have been identified.
In the long run and from an applied point of view, driver state indicators should be
measured with the help of unobtrusive sensors. A camera-based system is an example of
an unobtrusive sensor that can provide relevant information and also is suitable for
driver applications. From an advanced camera-based system it is possible to obtain
information on for example head and gaze direction, eye lid opening and facial
expressions. In order for a vehicle mounted camera system to fulfil its purpose, it must
meet automotive requirements in terms of reliability and crashworthiness and it must
also be able to adapt to various light conditions and to a wide range of facial features
and anthropometric measures. Advanced camera systems for head and gaze tracking are
quite expensive and therefore not possible to use for consumer applications. Nor are
they suitable to use in low-budget studies or non-controlled experiments where no test
leader is present (e.g. naturalistic driving studies) since they often require manual
adjustments and monitoring.
The increasing interest in measuring driver state raises the need for a cost efficient
camera-based system that can be mass-produced and used in safety systems or for
research purposes such as large-scale field operational tests or in low-end simulators.
A low cost camera-based system can be installed and used in most kinds of driver
behaviour studies. This will not only increase the knowledge on driver behaviour but
also facilitate comparisons of results from e.g. different driving simulators.
In order to use a camera-based system for detection or prediction of driver impairment,
indicators of different driver states must be defined and implemented. Measuring driver
state is a complex task since indicators often are influenced by many factors. For
example, partly closed eyes may be related to sleepiness, but it could also be caused by
heavy sunlight or a blowing fan directed to the face. Identifying robust indicators is thus
essential. A way of increasing the reliability is to fuse information from several
indicators into a detector or classifier. Such a classifier can be used in simulator studies
that e.g. aim to evaluate the effectiveness of different warning interfaces or to study
driving behaviour under the influence of some impairment. By using a classifier, driver
state can be determined in an objective and identical way. Furthermore, the use of a
well-defined classifier allows for comparisons of results from different studies. A
reliable classifier is thus a valuable research tool, particularly for simulator studies,
since a driving simulator provides a safe environment for studying driver impairment.
In the present study, the main focus was on driver sleepiness. Driver distraction was
considered to some extent.
ViP publication 2011-6
7
1.1
Aims and limitations
The main aims of the project were to:





Develop a low cost data acquisition unit for head and eye tracking based data.
Identify feasible indicators (camera-based and other) of driver sleepiness and
distraction.
Implement and validate camera-based indicators of driver sleepiness in a field
setting using an instrumented car.
Investigate the performance of the indicators as sleepiness detectors.
Develop a sleepiness classifier for driving simulators (that can be used in future
studies that aim to evaluate e.g. sleepiness warning interfaces).
A secondary aim was to create a database with driver impairment data (sleepiness and
distraction) from both field and simulator tests. The database is intended to be used for
further analyses in future projects.
The project included a literature review on indicators both for driver sleepiness and
distraction. The subsequent work was then limited to include implementation and
evaluation of sleepiness indicators only, since the project would be too extensive
otherwise. In order to evaluate the identified and implemented sleepiness indicators, two
experiments were conducted: one on a real road using an instrumented car and one in a
driving simulator. Both experiments followed the same procedure. They were
principally designed as sleepiness experiments with one alert condition (daytime) and
one sleepy condition (at night), but in the end of the driving session a distraction event
occurred, which can be used in future analyses on driver distraction. The experimental
design was also well suited for simulator validation, since the same subjects participated
in both experiments. Although simulator validation was not a part of the present study,
some events and a questionnaire were added to the experiment in order to provide a
useful dataset for future simulator validation.
1.2
Related publications
In addition to the present report the project has resulted in the following publications:


Ahlstrom C, Kircher K: Review of real-time visual driver distraction detection
algorithms, 7th International Conference on Measuring Behaviour, Eindhoven,
Netherlands, 2010.
Ahlstrom C, Kircher K, Sörner P: A field test of eye tracking systems with one
and three cameras, 2nd International Conference on Driver Distraction and
Inattention, Gothenburg, Sweden, 2011.
The first publication is based on the literature review on distraction indicators that was
done within the project. The complete review can be found in Appendix A. In the
second publication, the 1-camera system is compared with the 3-camera system with
respect to gaze parameters. Analysing gaze parameters was beyond the scope of the
project but nevertheless very interesting and relevant when it comes to identification of
driver impairment.
8
ViP publication 6-2011
2
Driver sleepiness
During the last decade there has been an increased focus on driver fatigue and the risk
of driving under fatigue or sleepiness.
Driver fatigue and sleepiness (or drowsiness) are often used interchangeably (Dinges,
1995). Fatigue refers to an inability or disinclination to continue an activity, generally
because the activity has, in some way, been going on for “too long” (Brown, 1994).
Fatigue is often considered to be a generic term, and sleepiness is one of the major subcomponents. Most often they should be considered separate, even though they are
related to the same concept.
Fatigue is defined as a global reduction in physical or mental arousal that results in a
performance deficit and a reduced capacity of performing a task (Williamson, Feyer et
al., 1996). Sleepiness on the other hand is defined as the physiological drive to sleep
(Dement and Carskadon, 1982). The reason for sleepiness is more or less related to
time being awake, time of the day and hours slept last 24 hours. The reasons for fatigue
could be several, from physical, perceptual, boredom to apathy (Desmond, Matthews et
al. 1997). A person can be fatigued without being sleepy, but a person cannot be sleepy
without being fatigued. The countermeasure for sleepiness is only sleep. The countermeasure for fatigue could be other. This is why they need to be separated.
There are lots of studies focusing on selection of the most promising indicators and
algorithms to detect or predict driver sleepiness. Most of them are based on data from
driving simulators, but recently also data from driving under real conditions are used.
There are also several approaches used in order to classify indicators or fuse indicators
in order to have a high degree of both sensitivity and specificity.
However, there are still problems to solve. Among the most critical ones are the
individual differences between drivers, both in alert and especially when driving in a
sleep deprived condition. Another critical issue is the lack of a reference method
(ground truth) that is possible to use also in an environment with a high risk for
artefacts. The car industry has an interest in designing driver support systems addressed
to sleepy drivers and from their perspective the indicators used need to be captured with
help of unobtrusive, reliable and cost efficient sensors. In this area the development is
going fast, but still the sensors are not mature enough.
A literature review on sleepiness indicators and classifiers was carried out in order to
obtain a basis for the selection of indicators and to get ideas on how to combine the
indicators into a classifier. The literature search was done by VTI Library and
Information Centre. Search words were:



sleepiness/fatigue/drowsiness
driver/driving
indicators/algorithms/measurements/blinking/eye-based/EOG/driving
behaviour/lateral position/steering wheel reversal rate/thresholds.
Five databases were searched: PubMed, TRAX, ITRD, TRIS and Scopus. Only
references from 2005 or later were included in the literature search.
ViP publication 2011-6
9
2.1
Indicators
Several different measures and indicators of sleepiness are described in the literature.
Physiological measures such as EEG and EOG are often used in research, but they are
not feasible for commercial use because they are too obtrusive or impractical. Camerabased systems can provide several measures of sleepiness, for example blink behaviour
and nodding, and they are frequently reported in literature. Camera-based detection
systems are suitable for driving and there are a number of commercially available
devices (Wright, Stone et al., 2007).
Another type of indicators is driving behaviour measures, including e.g. lateral position,
steering wheel movements and speed. Context information, such as time of the day and
trip duration, have also been suggested as indicators of sleepiness (Boverie, Giralt et al.,
2008). Table 1 summarizes indicators used for sleepiness detection.
Table 1 Indicators used for sleepiness detection.
Type of indicator
Measures
Eye activity (camera)
Blinking frequency, blink duration, PERCLOS, fixation duration,
eyelid distance, saccadic peak velocity
Head/face activity
Nodding frequency, face position, yawn frequency, facial actions
Physiology
EEG-based measures, EOG (blink frequency, blink duration,
PERCLOS, closing duration, opening duration, fixation duration,
blink amplitude, delay of lid re-opening, lid closure speed, blink
amplitude/peak closing velocity, saccadic peak velocity), heart rate
based measures, head motion, force applied to steering wheel
Driving behaviour
Steering wheel angle (reversal rate, std, energy in 0-0.4 Hz band),
lateral position (mean, std), speed variability, time to line crossing,
lanex
Contextual information
Trip duration, time of the day, hours of sleep/sleep deprivation
What indicators to use depend on the application. Physiological indicators and driving
behaviour measures are feasible in controlled studies, while camera-based indicators
might be the best choice for commercial systems. In a review on vehicle measures for
prediction of sleepiness, standard deviation of lane position and steering wheel movements are stated as the most important (vehicle) measures (Liu, Hosking et al., 2009). A
limitation of these measures is that they are also related to vehicle type, driver experience, geometric characteristics, condition of the road etc. (Bergasa, Nuevo et al., 2006).
Sandberg has investigated the use of several different driving behaviour signals (variability indicators based on lateral position, yaw, steering wheel angle and derived
measures such as standard deviation of lateral position, time to line crossing etc.) and
concluded that these indicators basically contain the same information (Sandberg,
2008). Schleicher et al. have investigated several oculomotor/EOG indicators of
sleepiness and concluded that blink duration, delay of lid re-opening, blink interval
(frequency), and lid closure speed are the best indicators (Schleicher, Galley et al.,
2008). It was also found that the pattern of fixations changed with increased sleepiness,
so that the proportions of very short (< 150 ms) and overlong (> 900 ms) fixations
increased. Bergasa et al. have investigated camera-based indicators and found that the
10
ViP publication 6-2011
most important measures are fixed gaze, PERCLOS and blink duration (Bergasa, Nuevo
et al., 2006). Åkerstedt and colleagues have studied the responsiveness of several
sleepiness indicators to sleep loss, time of day and time on task in a simulator study
(Åkerstedt, Ingre et al., 2009). Clear main effects of time of day and time on task were
found. The most sensitive indicators were subjective sleepiness, standard deviation of
lateral position and EOG measures of eye closure (duration, speed and amplitude). EEG
measures and line crossings were less responsive. The authors have also studied
individual differences and found that they exceed the fixed effects of the physiological
indicators, but not those of the standard deviation of lateral position and subjective
sleepiness. Individual variations in eye activity are discussed by Yang et al., who
elucidate the fact that some drivers might look awake, although their driving performance is severely deteriorated (Yang, Mao et al., 2009). Eye activity may thus be less
sensitive as a sleepiness indicator. Individual variations imply the need of combining
several indicators.
2.2
Algorithms and fusion of indicators
A majority of the recent literature on sleepiness monitoring and detection focus on how
to combine and fuse several sleepiness related indicators into one single output that
corresponds to driver state. Table 2 summarizes the classifiers that have been used in
different studies.
Table 2 Classifiers used for sleepiness detection.
Classifier
References
Neural network
(Eskandarian and Mortazavi, 2007; Jin, Park et al., 2007; Yang,
Sheng et al., 2007; Boyraz, Acar et al., 2008; Sandberg, 2008;
Zhang, Zhu et al., 2008)
Fuzzy inference
systems
(Bergasa, Nuevo et al., 2006; Boverie, Giralt et al., 2008;
Boyraz, Acar et al., 2008; Wu and Chen, 2008)
Support vector
machines
(Golz, Sommer et al., 2007; Sommer, Golz et al., 2008;
Sommer, Golz et al., 2009)
Bayesian networks
(Yang, Mao et al., 2009)
Ridge regression
(Vural, Cetin et al., 2007)
Boosting
(Vural, Cetin et al., 2007)
Decision trees
(Kim and Hahn, 2008)
K-means clustering
(Ohsuga, Kamakura et al., 2007)
There is no evidence that a certain classifier performs better than another in sleepiness
applications. Boyraz and colleagues (Boyraz, Acar et al., 2008) have compared a neural
network approach with a fuzzy inference system and it was concluded that there was no
significant difference between the two classifiers.
Most authors report that their classifiers correctly identify more than 80% of the occurrences of sleepiness. The rate of correct identifications depends heavily on the amount
ViP publication 2011-6
11
of data, the selection of training and evaluation data sets and the reference method used.
The reported performance should thus be interpreted cautiously.
Several different kinds of reference data for training and validation of a classifier are
reported in literature: Karolinska sleepiness scale (KSS) (Berglund, 2007; Shuyan and
Gangtie, 2009; Sommer, Golz et al., 2009), subjective/video rating (Eskandarian and
Mortazavi, 2007; Ohsuga, Kamakura et al., 2007; Boyraz, Acar et al., 2008; Schleicher,
Galley et al., 2008; Sommer, Golz et al., 2008; Zhang, Zhu et al., 2008), physiological
data (EEG and/or EOG) (Boverie, Giralt et al., 2008; Shuyan and Gangtie, 2009),
simulated drowsiness behaviour (Bergasa, Nuevo et al., 2006) and estimated sleepiness
level based on trip duration and falling asleep events (Vural, Cetin et al., 2007).
Interesting and detailed papers on classifiers for sleepiness detection are for example
Bergasa et al.’s article on a fuzzy inference system using camera-based measures such
as PERCLOS and blink duration (Bergasa, Nuevo et al., 2006) and Boyraz et al.’s
article where camera-based measures are combined with driving behaviour measures
and fed into two different classifiers (Boyraz, Acar et al., 2008). Vadeby and colleagues
have used Cox proportional hazard models in order to study the relationship between
different indicators of sleepiness and lane departure events in a driving simulator
(Vadeby, Forsman et al., 2010). A combination of the ratio of blink amplitude and peak
closing velocity of the eyelid, standard deviation of lateral position and lateral
acceleration relative to the road was found to be the most sensitive predictor. Worth
mentioning, although not camera-related, is a study by Sandberg, where model based
information (time of day etc.) was used in combination with an optimized variability
indicator of driving behaviour (based on yaw angle) (Sandberg, 2008). It was found that
the accuracy of sleepiness detection was greatly improved when the two indicators were
combined using a neural network, compared to using only one of the indicators.
2.3
Conclusions and recommendations for SleepEYE
Almost all recent literature on sleepiness detection emphasizes the fact that a combination of several different indicators are needed in order to be able to classify the level of
sleepiness, because of the complex nature of signs of sleepiness and the great variation
among individuals.
Several kinds of sleepiness indicators, of which some seem more promising than other,
have been suggested in the literature. Given the conditions and requirements in
SleepEYE, where data from the 1-camera system and from the vehicle will be available,
the following parameters were suggested to be looked at:
 Blink duration
 PERCLOS
 Blink frequency
 Lateral position variation
 Trip duration
 Time of day.
Moreover, other interesting indicators are lid closure speed and fixation duration.
However, given the technical limitations of the camera system, these indicators are
probably not possible to obtain.
12
ViP publication 6-2011
In order to combine the indicators, a classifier based on if-then rules or similar (e.g. a
decision tree or fuzzy logic) is suggested. Such a classifier is transparent, in contrast to
the black box approach of many other classifiers, i.e. the logic statements describing the
classification are traceable and comprehensive. This will allow for a fairly simple
description of the algorithm as well as for the re-use and re-implementation of the
algorithm in other ViP projects, which is one of the goals in this project.
ViP publication 2011-6
13
3
Development of the Smart Eye embedded camera system
3.1
Cost optimisation
In order to minimize cost per unit, a self-contained, easy-to-install and automotive
compliant embedded 1-camera system was designed that can run either as a stand-alone
tracker or function as a camera to a PC-based system.
The development of the 1-camera system began before the present project started.
Therefore, some parts of the design and implementation described in Section 3.1 have
been done within other projects.
3.1.1
Requirements
In addition to being cost efficient the camera system also had to fulfil certain criteria in
order to function in a driving situation. The following list of requirements was drawn up
for the camera system1:



















Track head, eye and eyelid in real time.
Day and night time operation, sunlight suppression.
Capable of eyeglass reflex reduction.
Fully automatic stand-alone operation.
Deliver sampled-down real time video of face and eye clips.
Mounting and cabling designed for quick and simple installation
Unobtrusive to driver.
Accommodate both car and truck geometry, distance to face max 1m.
CAN interface for low bandwidth control and tracking results.
Ethernet interface for high bandwidth control and tracking results.
Use power directly from an unfiltered 12 or 24 VDC vehicle system.
Max power consumption of 6 W.
Automotive EMC compliant.
IR safety compliant.
Withstand vehicle vibrations and mechanical shocks.
No dangerous fault modes.
Automotive temperature range.
No out-gassing materials.
Rational production.
All requirements were fulfilled. The hardware and software are further described below.
3.1.2
Hardware
The embedded system was built around an existing prototype camera and signal
processing board supplied by Visteon (Visteon Inc., the US). This board had previously
been used by Smart Eye in earlier projects, and was proven to be a feasible platform. In
order to meet the project deadlines and reduce risk, it was decided that no changes
should be made to this component although the system as a whole would be larger than
1
The requirements were set by Smart Eye AB and Volvo Cars Corporation in a previous project in 2009.
14
ViP publication 6-2011
necessary and include some redundant circuitry. The mechanical outline of the board
thus determined much of the final package size.
Two companion circuit boards were designed: one interface / power supply board and
one dual IR illuminator board. These were sandwiched with the camera and processing
board. A minimal-envelope matte black moulded plastic box consisting of two symmetrical halves and one rectangular IR-transparent window was designed. The box
mounts to the vehicle with two bolts mounted through the top of the package.
In order to adapt to the different nominal distance between camera and driver’s face in
cars and trucks, two different focal length lenses were used. When the lenses were
installed, they were focused to the respective nominal face distance and then fixed
permanently in place. The corresponding lens and camera parameters are configured via
software. IR filtering was integrated in the optical path.
Power, CAN and Ethernet connections are made via a single RJ45 connector on the
back side of the package. This in turn connects via a standard shielded TP-cable to a
remote splitter box, where separate power, CAN and Ethernet connectors are available,
Figure 1.
Testing was done at Volvo’s EMC lab to verify that electromagnetic emission levels
were in compliance with the relevant norms.
Figure 1 The Smart Eye embedded camera system. The camera unit is approximately
10 cm wide.
3.1.3
Software
Apart from porting and optimizing Smart Eye’s gaze tracking software to the embedded
platform, several other software tools also had to be developed, for example support for
remote firmware update, logging and diagnostics, full resolution video logging and offline tracking as well as production tools such as fixed pattern noise calibration.
Since the experiments were designed to capture full resolution video and tracking was
done later off-line, frame loss due to real-time processing bottlenecks had to be
modelled artificially.
The sampling frequency of the camera system is 60 Hz. The software version that was
used in SleepEYE was maud_20110303_0_01_v1021.
ViP publication 2011-6
15
3.2
Implementation of driver impairment indicators
The eye blink based sleepiness indicators identified in the literature review and listed in
Section 2.3 were implemented for the embedded 1-camera system. The focus was
primarily on blink duration, but PERCLOS and blink frequency output signals were also
implemented.
3.2.1
Definition of blink duration
The absolute eyelid opening distance is dependent on a number of factors, such as e.g.
head pose, vertical gaze direction, light conditions and interior climate. A complication
is also that some individuals do not close their eyes completely when blinking. It is thus
not easy to define a generally valid eyelid distance threshold.
The target definition that was settled on for the 1-camera system was to define the blink
duration as the time during which the upper eyelid covers the centre of the pupil. This
takes both eyelid and eye structures into account. For blink duration the priority was to
properly detect the length of the eye blinks at the cost of potentially dropping hard-todetect blink events.
3.2.2
Implementation
Initially, an effort was spent on improving the signals from the underlying low-level
detectors. This involved improving head tracking and eye clip positioning as well as
optimizing the low-level detectors for basic structures such as eyelids, iris and corneal
reflections.
Subsequently, the per-frame low-level signals were fed into an adaptive state machine
for detection of blink events in the time domain.
16
ViP publication 6-2011
4
Experiments
Two experiments that aimed at collecting data for the camera evaluation and for the
development of a sleepiness classifier were conducted: one on a real road and one in a
driving simulator.
4.1
Design and procedure
4.1.1
Experimental design
The experimental design is shown in Table 3. The participants conducted four driving
sessions: real road alert, real road sleep deprived, simulator alert and simulator sleep
deprived. The order of the driving sessions was the same for all participants, for
practical reasons. The alert conditions were carried out in the late afternoon, while the
sleep deprived conditions took place after midnight on the same (or actually the next)
day. In the real road experiment, it was always daylight during the alert session and
always dark during the sleep deprived session. In the simulator it was daylight in both
sessions, since there was no darkness scenario implemented at the time of the
experiment.
Table 3 Experimental design.
Driving
session
Condition
Time of day
Light condition
1
Real road, alert
3:30–7:30 p.m.
Daylight
2
Real road, sleep deprived
0:30–4:30 a.m.
Dark
3
Simulator, alert
3:30–7:30 p.m.
Daylight
4
Simulator, sleep deprived
0:30–4:30 a.m.
Daylight
The study was ethically approved by the regional ethical committee in Linköping,
registration number 2010/153-31. Permission to conduct driving sessions with sleep
deprived drivers on public roads between midnight and 5:00 a.m. was given by the
government, registration number N2007/5326/TR.
4.1.2
Participants
Twenty participants were recruited to the study. Half of them were women. The
participants were recruited from the Swedish register of vehicle owners. The main
inclusion criteria were:
 Between 30 and 60 years old
 No glasses
 Healthy
 Normal weight
 No shift workers
 No professional drivers.
ViP publication 2011-6
17
The reason for excluding drivers with glasses was that a homogeneous group with
regard to the eye tracking systems was desired.
Unfortunately, some subjects cancelled their participation at short notice and it was not
possible to replace all of them by new participants. In total, eighteen subjects participated in the field study and sixteen of them participated in the simulator study. Eight of
the participants were women. One participant was younger than 30 years and one had
glasses. The subjects were compensated 3,000 SEK for their participation.
4.1.3
Procedure
One to two weeks before the experiment the participants were mailed information and
sleepiness and wakefulness forms to be filled in the three nights and two days
immediately prior to the experimental day. The participants were instructed to sleep at
least seven hours per night the three nights prior to the test.
Two subjects participated each experimental day. The first participant arrived at 2 p.m.
and the second at 4 p.m. When the participants arrived they were given written and oral
information about the test and were then asked to fill in an informed consent form and a
responsibility form. They also had to show their driving license and to do a breath
alcohol test. The test leader then applied electrodes for physiological measurements.
Each participant accomplished two driving sessions on each test occasion: the first was
the alert condition and the second was the sleep deprived condition, Table 4.
Table 4 Start and end times for the driving sessions.
Driving session
Start
End
Participant A alert
3:30 p.m.
5:15 p.m.
Participant B alert
5:45 p.m.
7:30 p.m.
Participant A sleep deprived
0:15 a.m.
2:00 a.m.
Participant B sleep deprived
2:45 a.m.
4:30 a.m.
Each driving session lasted for about 90 min. The time between the sessions was spent
at VTI, where the participants could e.g. read or watch TV. The participants were served
dinner after the first driving session and fruits and sandwiches during the night. They
were not allowed to drink any caffeine containing beverages from 1 p.m. on the
experiment day.
The participants were instructed to drive as they would do in “real life”. While driving
they were not allowed to speak, listen to the radio or do anything else that would
counteract their sleepiness. During each driving session the participants rated their
sleepiness level on the 9-grade Karolinska Sleepiness Scale (KSS) every five minutes
(Åkerstedt and Gillberg, 1990), see Appendix B.
After the sleep deprived session, the electrodes were removed and the participants were
sent home by taxi.
18
ViP publication 6-2011
4.1.4
Tasks while driving
The participants were instructed to do three tasks during each driving session:



Drink water from a bottle.
Drive close to the right line marking.
Drive at a constant speed of 100 km/h without looking at the speedometer.
Before the first driving session, the participants were given a written instruction on how
to do the tasks. The tasks were done in the end of each driving session, both in the field
test and in the simulator. In the field test, the test leader told the participant when to do
the tasks. In the simulator, a text was shown on the simulator screen when it was time to
do the tasks.
The aim of the water drinking task was to distract the driver and to get “disturbed”
camera data. The aim of the two other tasks was to collect data for simulator validation
(not to be analysed in this project).
4.1.5
Data acquisition
Two Smart Eye camera systems were installed in the car and the simulator: the 1-camera embedded system that was to be evaluated and a 3-camera Smart Eye Pro (sampling
frequency 60 Hz) which was used as a reference system providing video-based ground
truth. The 1-camera system was mounted as low as possible behind the steering wheel,
in the car on top of the cover of the (adjustable) steering column (Figure 2, lower left),
and in the simulator at the lower edge of the main instrument. The 3-camera system was
mounted on top of the dashboard along with two IR-flashes (Figure 2, lower photos)
and synchronized to the 1-camera exposure pulse so that by slightly shifting its own
time of illumination and exposure, the respective illumination subsystems would not
interfere with each other. The positions of the cameras relative to the car were calibreted. Both camera systems were set up to simultaneously record time-stamped raw video
to hard disk drives, using lossless compression. The recordings from the test drives were
collected and subsequently processed off-line.
Vehicle data, such as speed and lateral position, was logged with 10 Hz from the car as
well as from the simulator. In the field experiment, video films of the vehicle frontal
and rear views, the driver's face and feet were recorded.
Physiological data – EEG, EOG and ECG – was recorded by a Vitaport 3 (TEMEC
Instrument B.V., The Netherlands) with 256 (EEG and ECG) or 512 Hz (EOG).
All data acquisition systems were connected to each other in order to facilitate
synchronization of data.
4.2
Field experiment
The car used in the experiment was a Volvo XC70 with an automatic gearbox, Figure 2.
During the tests, there was a sign on the rear of the car with the text “Mätning”.
The test route in the field test was the E4 motorway from Linköping (exit 111) to
Gammelsta (exit 128) and back, Figure 3. The length of test route was approximately
2 x 79 km and it took about 90 min to drive. The posted speed limit was 110 km/h
ViP publication 2011-6
19
during the whole route, except for a road section of 750 m in Norrköping, where the
posted speed limit was 90 km/h.
Figure 2 Upper left: The instrumented car. Upper right: The car had a sign “Mätning”
on the rear. Lower left: Driver’s seat with the three cameras on the dashboard, the embedded camera on the steering column and the KSS scale on the steering wheel. Lower
right: The screen that shows the driver’s face to the test leader. Photos: Katja Kircher.
Start and end at
exit 111
(Linköping västra)
Turning at exit
128 (Gammelsta)
Figure 3 The test was conducted at the motorway E4. The test route started at exit 111
in Linköping and the turning point was at exit 128 in Gammelsta.
20
ViP publication 6-2011
A test leader was sitting in the front passenger seat. The car had dual command and
there was a small screen in front of the test leader showing the driver’s face, so that the
test leader could see if the participant closed his/her eyes. The test leader was
responsible for the safety and was prepared to take control of the vehicle if the driver
became too sleepy. However, the intention was not to let the driver fall asleep, but to
stop the driving session before the driver fell asleep. The test leader was also supposed
to stop the driving session if the driver drove in an unsafe way, either because of
sleepiness or of other reasons (e.g. exceeded posted speed limit). The participants were
explicitly told to not exceed speed limits for safety reasons. They were also told that
they were allowed to stop for a break if they felt it was necessary for their safety. If the
driver chose to take a break, it was prescribed that the test leader would stop the driver
from continuing to drive.
The KSS ratings that were reported by the driver every five minutes were written down
by the test leader in a paper form.
4.3
Simulator experiment
The simulator that was used in this study is VTI driving simulator III, Figure 4. The
simulator is a moving base simulator with a 120 degrees forward field of view (VTI,
2011). The cabin is a Saab 9-3. In this experiment, the simulator had an automatic
gearbox.
Figure 4 VTI driving simulator III.
In the simulator, a motorway similar to that between Linköping and Norrköping was
used as a test route. Each driving session consisted of four laps on the same motorway
section. The total length of the simulator test route was about 150 km and it took about
75–80 min to drive.
In the simulator scenario there were a few events in the beginning and in the end of the
session. In the beginning there were three overtaking events where the test driver was
supposed to overtake slow vehicles. There were two similar events in the end of the
driving session. These events were intended to be used for (later) simulator validation.
The main part of the driving session was intended to be used for analysis of sleepiness
indicators and thus, interaction with other traffic was kept to a minimum. On average,
every 7 minutes during the whole driving session, a car overtook the test driver. Halfway of the test route, the posted speed limit was changed to 90 km/h for 1 km, since
there was a similar change of speed in the field test (there were actually two such
ViP publication 2011-6
21
changes in the field test but only one in the simulator scenario in order to minimize
influence from alertness-enhancing factors).
A test leader sat outside the simulator and monitored the driver via video and
loudspeakers/microphone. The test leader did not stop the driving session even if the
driver fell asleep. The participants were informed that they could stop driving at any
time if they, for example, felt sick.
4.4
Database
A database was created in order to provide an easily accessible dataset for the present
project, but also to facilitate further analyses in future projects on e.g. simulator
validation or driver impairments.
Driving data and physiological data were synchronized and merged into Matlab struct
files. Some additional parameters were inserted in the dataset: KSS, flags indicating
start, stop, lowering of speed limit and turning (field test), and blink duration calculated
from the EOG (see also Chapters 5 and 6).
Smart Eye data were post processed and blink parameters, i.e. blink duration, blink
frequency and PERCLOS, were computed according to Section 3.2. Smart Eye data
from all simulator sessions and from the night sessions in the field were processed in
bright pupil mode, while the field data from the daylight sessions were processed in
dark pupil mode. Smart Eye data was stored in separate Matlab files, because of their
large size. A common time stamp signal allowed for synchronization with the struct
files containing driving and physiological data.
Unfortunately, there were some technical problems with the driving simulator resulting
in sudden stops or absence of sound in nine driving sessions. A quality parameter was
added to all simulator data files after the experiment was finished, in order to allow for
quick removal of segments with simulator problems.
The parameters relevant for the present project are listed in Table 5. In total, the database contains approximately 30 parameters from the experimental vehicle, 20 parameters from the simulator, 50 parameters from the two camera systems and 10 physiological parameters.
22
ViP publication 6-2011
Table 5 The database parameters relevant for the present project
Parameter
Description
Experimental vehicle
Odometer
Distance driven
Driving simulator
Lateral position
Distance to the centre of the road
Data quality
Flag indicating data quality
1-camera system
Blink duration
Blink duration
Blink frequency
Blink frequency
PERCLOS
PERCLOS
Availability
Percentage of time when the system provides
eye tracking data.
3-camera system
Blink duration
Blink duration
Blink frequency
Blink frequency
Availability
Percentage of time when the system provides
eye tracking data.
Physiological
Blink duration
Blink duration obtained from the EOG
Ratings
KSS
ViP publication 2011-6
KSS ratings
23
5
Evaluation of the Smart Eye embedded camera system
The evaluation of the 1-camera system was done using data from the field test only,
since the system is intended to be used commercially in ordinary vehicles. A real
driving situation is expected to generate much more disturbances (varying light
conditions, head movements etc.) than a driving simulator, and therefore simulator data
is inconvenient to use for this purpose.
5.1
Method
Evaluation of blink parameters is time consuming and difficult. Time consuming
because the only ground truth is video recordings of the eye and difficult because the
onset and end point of a blink is hard to define in an objective manner. In this study,
two complementary approaches were chosen in order to gain some insight in the
performance of the two camera systems:
1. Manual comparison of a one-minute video segment with data from the two
camera systems and with data from EOG. This step mainly investigates the
systems’ ability to detect blinks.
2. Comparison of blink parameters obtained via EOG and via the two camera
systems. This step investigates how the blink frequency and the blink durations
differ between the systems.
The data used in the manual comparison was picked out about 20 km from the starting
point of the sleepiness test route. The one-minute segment with the highest eyelid
quality in the 1-camera system in a five-minute time window was selected for analysis.
The reason for selecting only a single minute per session for analysis is that video
analysis is very time consuming. For each one-minute segment a video of the driver’s
face was annotated manually and all blinks were noted. These blink occurrences were
then compared with the blink detections from the two camera systems and the EOG
(results in Section 5.2.2).
Blink frequency and blink durations from the entire sleepiness experiment were also
compared (results in Sections 5.2.3 and 5.2.4). In this case, no ground truth was
available why the EOG and the two camera systems could only be compared with each
other. In these analyses, the data were sliced in three different ways:
1. Distance: The blink parameters were calculated as aggregated values based on
data from two-kilometre segments. This means that it is possible to investigate
how the different parameters evolve with the distance driven.
2. KSS: The blink parameters were calculated as aggregated values based on KSS.
This means that it is possible to investigate how the different parameters evolve
with sleepiness.
3. Alert vs. sleep deprived: The blink parameters were calculated as aggregated
values based on if the driver is sleep deprived. This means that it is possible to
investigate how the different parameters change with sleep deprivation but it
also reflects for example lighting conditions.
For the 1-camera system, blinks and blink duration were identified according to the
definitions in Section 3.2. The blink complex detector in the 3-camera system defines
blink duration as the time between 50% closing amplitude and 50% opening amplitude.
24
ViP publication 6-2011
The observation window for the whole blink event is constrained in time, so very long
eye closures may be missed.
Blink detection and blink durations extracted from the EOG were estimated with the
LAAS algorithm (Jammes, Sharabaty et al., 2008). The algorithm low pass filters the
EOG data, calculates the derivative and searches for occurrences where the derived
signal exceeds a threshold and falls below another threshold within a short time period.
If the amplitude of the (original, low-pass filtered) EOG signal in such a sequence
exceeds a subject specific threshold, the sequence is assumed to be a blink. Blink
duration is calculated at half the amplitude of the upswing and the downswing of each
blink and defined as the time elapsed between the two.
Data from seventeen out of the eighteen participants were analysed. One participant was
excluded because she wore glasses.
5.2
Results
A prerequisite for sleepiness estimation based on blink parameters is accurate detection
of the blink complex. Results on blink detection performance are presented in section
5.2.2. The most promising sleepiness indicator is the blink duration and such results are
reported in section 5.2.4.
5.2.1
System availability
The availabilities, i.e. the percentage of time when the system provides eye tracking
data, of the 1-camera system and the 3-camera system are illustrated for daytime and
night-time in Figure 5 and Figure 6, respectively. In general, the 3-camera system has
higher availability than the 1-camera system. It is also apparent that head tracking has
higher availability compared to gaze tracking and eyelid tracking.
Gaze tracking availability is somewhat higher during night time whereas head and
eyelid tracking deteriorates a little. Interestingly, availability increases after the
turnaround during night time, which coincides with some participants aborting the test.
This may indicate that the systems have more difficulties tracking sleepy drivers.
Figure 5 Quartiles across participants of the availability of tracking data for gaze
(left), head (middle) and eyelid (right) during daytime in the field test. The grey box
indicates the turnaround, blue indicates the 1-camera system and red indicates the
3-camera system. When the areas overlap, the colours are blended.
ViP publication 2011-6
25
Figure 6 Quartiles across participants of the availability of tracking data for gaze
(left), head (middle) and eyelid (right) during night-time in the field test. The grey box
indicates the turnaround, blue indicates the 1-camera system and red indicates the
3-camera system.
5.2.2
Blink detection
One-minute segments were extracted from each trip and the blink detections from the
EOG, the 1-camera system and the 3-camera system were compared with manually
annotated video. The results are summarized in Table 6 and Table 7 for daytime and
night-time, respectively. During daytime, the percentages of correctly detected blinks
were 99.2% for the EOG, 42.5% for the 1-camera system and 62.0% for the 3-camera
system. During night-time, the corresponding percentages were 97.2%, 29.2% and
71.7%, respectively. Note that none of the systems delivered a high number of false
detections. Also note that since data from the 3-camera system is missing for some
participants, the sum of correct and missed detections is not identical across systems.
26
ViP publication 6-2011
Table 6 Number of correct, false and missed blink detections during daytime session of
the field test based on EOG, 1-camera and 3-camera, respectively. (Reference: video
recordings.)
Correct
False detection
Missed detection
Correct
False detection
Missed detection
1
37
0
0
32
2
5
13
0
24
2
52
1
0
9
0
43
48
0
4
3
12
3
0
0
0
12
11
0
1
5
18
0
0
16
0
2
9
0
9
7
37
0
0
37
0
0
36
0
1
8
25
1
0
12
0
13
23
0
2
9
18
3
0
14
0
4
16
0
2
10
58
0
0
31
2
27
4
0
54
11
26
1
1
10
1
17
18
0
9
12
8
1
0
0
0
8
3
0
5
13
48
0
0
6
0
42
35
0
13
14
9
0
0
8
0
1
0
0
9
16
13
2
0
1
0
12
12
0
1
17
36
0
0
11
0
25
18
22
0
3
1
0
24
4
0
21
19
29
5
0
2
0
27
21
0
8
20
37
0
0
18
0
19
28
0
9
Day
485
17
4
208
5
281
281
0
172
Participant
Missed detection
3-camera
False detection
1-camera
Correct
EOG
ViP publication 2011-6
27
Table 7 Number of correct, false and missed blink detections during night-time session
of field test based on EOG, 1-camera and 3-camera, respectively. (Reference: video
recordings.)
Missed detection
False detection
Correct
False detection
Correct
3-camera
Missed detection
1-camera
Missed detection
False detection
Correct
Participant
EOG
1
23
0
0
4
0
19
14
0
9
2
42
0
0
27
1
15
38
0
4
3
35
7
1
5
0
31
16
0
20
5
5
2
0
1
0
4
2
0
3
7
25
1
0
20
1
5
8
33
0
0
16
0
17
33
0
0
9
29
2
0
4
0
25
24
0
5
10
29
0
1
17
0
13
20
0
10
11
29
0
0
1
0
28
29
0
0
12
8
3
0
0
0
8
4
0
4
13
40
0
2
9
0
33
27
1
15
14
14
1
8
2
0
20
1
0
21
16
21
1
0
9
0
12
10
0
11
17
34
1
0
5
0
29
27
0
7
18
9
0
1
0
0
10
19
35
2
0
0
0
35
31
0
4
20
35
0
0
14
0
21
28
0
7
Night
446
20
13
134
2
325
304
2
120
Three excerpts from these one-minute segments are shown in Figure 7 to Figure 9Figure
9. Figure 7 illustrates the occurrence of long duration blinks. In this case, the EOG
detects both long blinks while the 3-camera system detects the first long blink while the
1-camera system detects the second long blink. In Figure 8 all blinks are detected by the
1-camera system while no blinks are detected by the 3-camera system, but then again, in
Figure 9 the 3-camera system performs better.
28
ViP publication 6-2011
7
EOG / Eye lid opening (au)
6
5
4
3
2
1
0
31995
32095
32195
32295
32395
32495
Frame number
32595
32695
32795
Figure 7 Example of a segment from one participant comparing blink detections from
the EOG (red), 1-camera system (green) and 3-camera system (blue). The black lines
represent the occurrence of true blinks based on video annotations. The grey signals in
the background are the vertical EOG lead and the eyelid opening, respectively.
5
EOG / Eye lid opening (au)
4.5
4
3.5
3
2.5
2
1.5
1
0.5
0
36750
36850
36950
37050
37150
37250
37350
Frame number
37450
37550
37650
37750
Figure 8 Example of a segment from one participant comparing blink detections from
the EOG (red), 1-camera system (green) and 3-camera system (blue). The black lines
represent the occurrence of true blinks based on video annotations. The grey signals in
the background are the vertical EOG lead and the eyelid opening, respectively.
ViP publication 2011-6
29
7
EOG / Eye lid opening (au)
6
5
4
3
2
1
0
45298
45398
45498
45598
45698
45798
45898
Frame number
45998
46098
46198
46298
Figure 9 Example of a segment from one participant comparing blink detections from
the EOG (red), 1-camera system (green) and 3-camera system (blue). The black lines
represent the occurrence of true blinks based on video annotations. The grey signals in
the background are the vertical EOG lead and the eyelid opening, respectively.
5.2.3
Blink frequency
Blink frequency is not only related to sleepiness but also to the allocation of attention
resources, the transition points in information processing and possibly processing mode
and is consequently not a very useful measure of sleepiness (Stern, Walrath et al.,
1984). However, since it reflects the influence of poor blink detections performance it is
an interesting performance indicator of the different systems. Figure 10 shows how the
median value and the quartiles of the estimated blink frequency vary over time. It can be
seen that for all systems, the blink frequency is rather constant throughout the entire trip
during daytime while it tends to increase with distance driven during night-time. There
is, however, a large difference between the systems. The average blink frequency across
all data and all participants are 32 blinks per minute for the EOG, 13 blinks per minute
for the 1-camera system and 20 blinks per minute for the 3-camera system, Figure 11.
Plotting the difference in average blink frequency, between blinks associated with KSS
< 7 versus KSS ≥ 7 for each participant, shows that for about half of the participants the
blink frequency increases when they are sleepy whereas the other half shows a decrease
in blink frequency, see Figure 12. The participants have been sorted in ascending order
according to the EOG. Preferably there should have been a similar increasing trend in
all three systems but there is no relation between the blink detections of the systems.
30
ViP publication 6-2011
Figure 10 Median values (solid line) and quartiles (shaded region) across participants
of the blink frequency (measured in blinks per minute, bpm) as a function of distance
driven in the field test. The dark grey box indicates the turnaround, blue indicates alert
and red indicates sleep deprived.
Figure 11 Mean values of blink frequency across participants (measured in blinks per
minute, bpm) as a function of distance driven in the field test. The grey box indicates the
turnaround. Note that the shaded areas do not have an intrinsic value and are only used
to group the two conditions alert and sleep deprived within the same system.
ViP publication 2011-6
31
35
EOG
1-camera
3-camera
30
Blink frequency difference (bpm)
25
20
15
10
5
0
-5
-10
-15
-20
0
2
4
6
8
10
Participant #
12
14
16
Figure 12 The mean difference between blink frequencies (measured in blinks per
minute, bpm) with KSS<7 and blink frequencies with KSS≥7 for each participant in the
field test. The participants have been sorted in ascending blink frequency difference
order according to the EOG.
5.2.4
Blink duration
The histogram of blink durations as determined by the three systems is shown in
Figure 13. As noted in the previous section, the EOG detects the largest amount of
correct blinks. For the EOG and the 3-camera system, there is a slight shift towards
longer blink durations when the participants are sleep deprived while for the 1-camera
system the blink durations decrease for sleep deprived drivers. Note that the blink
durations are somewhat longer for the 3-camera system as compared to the EOG.
EOG
1-camera
Alert
SDP
12000
4000
2000
10000
Number of blinks
Number of blinks
Number of blinks
6000
8000
6000
4000
2000
0
200
400
Blink Duration (ms)
600
0
Alert
SDP
12000
10000
8000
0
Alert
SDP
12000
10000
3-camera
8000
6000
4000
2000
0
200
400
Blink Duration (ms)
600
0
0
200
400
Blink Duration (ms)
Figure 13 Histograms of blink duration measured with the three systems in the field
test. Blue indicates alert and red indicates sleep deprived.
Scatter plots showing the mean blink duration across all participants are illustrated in
Figure 14. The correspondence between the systems is poor with correlation coefficients
of 0.28 for EOG vs. 1-camera, 0.16 for EOG vs. 3-camera and 0.39 for 1-camera vs.
32
ViP publication 6-2011
600
3-camera in the alert condition. Corresponding coefficients in the sleep deprived
condition are 0.38, 0.33 and 0.37, respectively. All correlations are significant at the
95% level.
Figure 14 Scatter plots of blink durations from the three systems plotted against each
other. Left: EOG (x-axis) and 1-cam (y-axis), middle: EOG (x-axis) and 3-cam (y-axis),
right: 1-cam (x-axis) and 3-cam (y-axis). Blue indicates alert and red indicates sleep
deprived.
Figure 15 shows how the median value and the quartiles across participants of the blink
duration vary over distance driven for alert and sleep deprived drivers. For the EOG and
the 3-camera system there is a small gap between the two conditions but such a difference is not visible for the 1-camera system. The same plot, but with really long blinks
represented by the 95th percentile of the distribution instead, is shown in Figure 16. In
this case the only difference between the groups can be seen in the 3-camera system.
Figure 15 Median values (solid line) and quartiles (shaded region) across participants
of the mean blink duration as a function of distance driven in the field test. The dark
grey box indicates the turnaround, blue indicates alert and red indicates sleep deprived.
Top: EOG, middle: 1-camera system, bottom: 3-camera system.
ViP publication 2011-6
33
Figure 16 Median values (solid line) and quartiles (shaded region) across participants
of the 95th percentile of blink duration as a function of distance driven in the field test.
The dark grey box indicates the turnaround, blue indicates alert and red indicates sleep
deprived. Top: EOG, middle: 1-camera system, bottom: 3-camera system.
Plots where the blink duration data is represented as a function of KSS instead of a
function of distance are shown in Figure 17 and Figure 18. For the EOG and the
3-camera system there is an increase in average blink duration when the driver is very
sleepy (KSS = 9). It can also be seen that the blink duration is longer when the drivers
are sleep deprived even though they report the same KSS values as during the alert
condition. For the 1-camera system the situation is different. Even though the blink
duration increases with increasing KSS values, the blink durations are shorter for sleep
deprived drivers as compared to alert drivers. Similar findings can be found in the 95th
percentile of the blink durations, see Figure 18.
34
ViP publication 6-2011
EOG
1-camera
300
300
Alert
SDP
Alert
SDP
200
150
100
50
250
Blink Duration (ms)
250
Blink Duration (ms)
Blink Duration (ms)
250
0
3-camera
300
Alert
SDP
200
150
100
50
2
3
4
5
6
7
8
0
9
200
150
100
50
2
3
4
5
6
7
8
0
9
2
3
4
5
6
7
8
9
Figure 17 Boxplot showing the average blink duration across participants as a function
of KSS. On each box, the central mark is the median, the edges of the box are the 25th
and 75th percentiles and the whiskers extend to the most extreme data-points not
considered to be outliers. Left: EOG, middle: 1-camera system, right: 3-camera system.
Blue indicates alert and red indicates sleep deprived.
EOG
1-camera
700
400
300
200
500
400
300
200
100
100
0
0
3
4
5
6
7
8
9
Alert
SDP
600
95 Blink Duration (ms)
500
2
700
Alert
SDP
600
95 Blink Duration (ms)
600
95 Blink Duration (ms)
3-camera
700
Alert
SDP
500
400
300
200
100
2
3
4
5
6
7
8
9
0
2
3
4
5
6
7
8
Figure 18 Boxplot showing the 95th percentile of blink duration across participants as
a function of KSS. On each box, the central mark is the median, the edges of the box are
the 25th and 75th percentiles and the whiskers extend to the most extreme data-points
not considered to be outliers. Left: EOG, middle: 1-camera system, right: 3-camera
system. Blue indicates alert and red indicates sleep deprived.
Investigating the blink duration per participant, it can be seen that the EOG consistently
provides longer blink durations for blinks associated with KSS ≥ 7 as compared to KSS
< 7, see Figure 19 and Figure 20. However, the difference between alert and sleepy is
lower than 20ms in all but three drivers. For the 3-camera system blink durations are
longer for sleepy drivers in all cases but one, but also here the difference is less than
20 ms in a majority of the drivers. For the 1-camera system the results are ambiguous
and more difficult to interpret. Comparing the three systems with each other, there are
no systematic similarities across different drivers.
ViP publication 2011-6
35
9
EOG
1-camera
240
KSS >= 7
KSS < 7
3-camera
240
KSS >= 7
KSS < 7
220
220
200
200
200
180
160
140
120
Blink duration (ms)
220
Blink duration (ms)
Blink duration (ms)
240
180
160
140
120
180
160
140
120
100
100
100
80
80
80
60
60
60
40
0
5
10
Participant #
40
15
0
5
10
Participant #
KSS >= 7
KSS < 7
40
15
0
5
10
Participant #
15
Figure 19 Blink duration per participant when alert (KSS < 7) and sleepy (KSS ≥ 7).
Left: EOG, middle: 1-camera system, right: 3-camera system. The participants have
been sorted in ascending order based on the sleepy drivers as measured with EOG.
100
Blink duration difference (ms)
EOG
1-camera
3-camera
50
0
-50
0
2
4
6
8
10
Participant #
12
14
16
Figure 20 The mean difference between blink durations with KSS≥7 and blink
durations with KSS<7 for each participant. The participants have been sorted in
ascending blink duration difference order according to the EOG.
36
ViP publication 6-2011
6
Determination of sleepiness thresholds
The aim of determining thresholds for sleepiness indicators was to develop a sleepiness
detector for simulator applications. The determination of sleepiness thresholds was
therefore done using data from the simulator experiment only.
As a first step, the most promising – from the literature review and by studying the data
– sleepiness indicators were selected and the optimal threshold for each single indicator
was determined. The indicators were then combined into a feasible detector/classifier.
In the last step, the performance of the classifier was evaluated on unseen data from
another project.
In an ideal world, a sleepiness threshold/detector should be able to discriminate across a
range of sleepiness levels, so that a warning system could be triggered on an arbitrary
level. In reality, this is very hard to achieve because of the complex nature of sleepiness
and sleepiness indicators.
In this project, it was decided that the sleepiness level the thresholds/detector should be
able to identify was KSS=8 or higher. This corresponds to “sleepy, some effort to stay
awake”. On this level, there is no doubt the driver is sleepy and triggering a warning on
this level is seen as well motivated and will hopefully also reach a high acceptance by
the driver.
6.1
Method
6.1.1
Preparation of dataset
A dataset that included the indicators suggested in Section 2.3 was prepared. Since the
indicators can be represented in different ways, for example as a mean or a median, a
number of variants of each indicator was computed (see below). For each variant, the
optimal threshold for discrimination between alert and sleepy drivers was determined,
by finding the indicator value that resulted in the highest performance. Performance is
defined as the average of sensitivity and specificity. In this application, sensitivity is the
proportion of correctly classified sleepy drivers while the specificity is the proportion of
correctly classified alert drivers.
2
nsleepyCorrect denotes the number of sleepy drivers correctly classified as sleepy, while
nsleepyIncorrect denotes the number of sleepy drivers incorrectly classified as alert (and vice
versa for nalertCorrect and nalertIncorrect).
ViP publication 2011-6
37
The optimal threshold was determined simply by stepping through the range of
indicator values and in each step calculating sensitivity, specificity and performance.
The procedure is illustrated in Figure 21. Data points where (the average) KSS was >7.5
were regarded as “sleepy” while the rest were regarded as “alert”.
1
Sensitivity
Specificity
Performance
0.9
0.8
Sens, spec, perf
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
1
2
3
4
5
Indicator (a.u.)
6
7
8
Figure 21 Determination of optimal threshold for a single indicator. The performance
is at a maximum when the indicator has a value of approximately 3.5.
All indicators were calculated in a sliding window of 5 min with 1 min resolution. It has
been shown that time windows of at least 60 s in general give better results than shorter
intervals when calculating sleepiness indicators, and that windows longer than 60 s are
beneficial for blink duration (Sandberg, 2011). A window length of 5 min was therefore
considered to be a reasonable choice. The time resolution of 1 min was chosen partly in
order to make the best use of the acquired data, but also with the future application in
mind where the sleepiness detector should be able to warn a driver before he/she
actually falls asleep and drives off the road but where a smaller time resolution probably
won’t be of any use.
The overtaking events in the beginning and the end of the driving session, and the three
tasks in the end of the driving session (see Section 0) were excluded from the dataset.
Sections with simulator failure were also excluded (see Section 4.4). Smart Eye data
was missing for participants 1-alert, 9-alert and 16-alert, either because of technical
problems with the system itself or because of synchronization problems with the
simulator data.
6.1.2
Selection of sleepiness indicators
From the literature review (Chapter 2) it was concluded that blink duration, PERCLOS,
blink frequency, lateral position variation, trip duration and time of day were the most
feasible indicators. The selection of indicators was somewhat modified when the results
from the camera evaluation were available.
38
ViP publication 6-2011
Blink based indicators
In the evaluation of the 1-camera system, it was found that blink frequency was not a
reliable measure since the camera system misses a lot of blinks. Furthermore, it was
demonstrated that the blink frequency obtained from the EOG – which is a fairly
reliable measure – was not a feasible sleepiness indicator, since the blink frequency
increased with sleepiness for some participants and decreased for some other (Section
5.2.2). This could perhaps be explained by the fact that blink frequency also is related to
e.g. attention and cognitive functions (Stern, Walrath et al., 1984). Therefore, blink
frequency was excluded from the threshold analysis.
PERCLOS was also excluded from the analysis, since it can be assumed that this
measure is not very reliable when a lot of blinks are missing.
The only blink based indicator included was thus blink duration. In the evaluation of the
camera system it was concluded that there was a substantial difference in the blink
duration identified from the EOG and from the camera systems, but it is still a bit
unclear which system gives the best results since there are no true answers available.
The camera-based systems have a limitation in their time resolution, while the EOG
algorithm sometimes has problem to identify the beginning and the end of a blink. The
pros and cons of the three systems are listed in Table 8. The camera-based systems have
a great advantage in terms of their unobtrusiveness and they also work in real time.
EOG based blink detection, on the other hand, is much more robust than the camerabased counterparts and it is not dependent on any commercial hardware or software.
The original plan was to use camera-based blink measures from the 1-camera system
only (in combination with other kinds of indicators) when developing the sleepiness
detector. However, since the blink duration obtained from the 1-camera system was not
in agreement neither with the EOG based blink duration nor with 3-camera blink
duration, it was decided that two separate sleepiness detectors should be developed and
compared: one using blink durations from the EOG (calculated with the LAAS
algorithm, see Section 5.1) and one using blink durations from the 1-camera system.
ViP publication 2011-6
39
Table 8 Pros and cons of the blink detectors.
Sensor
Pros
Cons
EOG (LAAS)
Proven to be good for blink
detection
Not possible to use in real time
with existing equipment
Easy to synchronize with other
measures (in SleepEYE)
Most truly not so good to identify
long blinks
Independent
Obtrusive
Work in real time
Not independent
Unobtrusive
Synchronization problems with
simulator data (in SleepEYE)
Camera-based
systems (in general)
Potential to identify long blinks
1-camera system
Installed in all EuroFOT vehicles
and will be possible to install in
future vehicles (low cost system)
Lots of misses in the blink
detection
3-camera system
Better blink identification than 1camera system
The algorithm for blink duration is
not validated yet
The blink duration varies from blink to blink. It is therefore customary to aggregate the
blink duration in short time windows in order to increase the robustness. This
aggregation can be achieved in several ways. The most frequently used blink duration
measure is probably mean blink duration. Alternative measures are for example median
(less sensitive to outliers) or max (focusing on long blinks). Here four different blink
duration based measures were investigated: mean blink duration, median blink duration,
mean of 25% longest blinks and fraction of blinks longer than 0.15 s. The optimal
threshold for discriminating between KSS≥8 and KSS<8 and the corresponding
performance for each measure are given in Table 9.
The performances of the blink measures alone are rather moderate. It seems like the
blink measures from the 1-camera system have slightly better performances than the
measures from the EOG on average. However, it should be noted that there are
differences in the underlying datasets, since camera data is completely missing from
three driving sessions (see also Section 6.1.4).
For the 1-camera system all four measures have similar performance, while for the
EOG, the two measures related to long blinks have the best performance. The fraction
of blinks > 0.15 s (fracBlinks) was selected as an input to the classifier, since it had the
best performance for the camera system and the second best for the EOG.
40
ViP publication 6-2011
Table 9 Thresholds and performances for single blink duration measures.
Measure
Threshold (s)
Performance
EOG mean blink duration
0.17
0.65
EOG median blink duration
0.13
0.64
EOG mean of 25% longest blinks
0.23
0.71
EOG fraction of blinks > 0.15 s
0.26
0.68
1-camera mean blink duration
0.11
0.71
1-camera median blink duration
0.10
0.70
1-camera mean of 25% longest blinks
0.27
0.69
1-camera fraction of blinks > 0.15 s
0.10
0.72
Lateral position variation
Standard deviation of lateral position (SDLP) is one of the most used measures of
driving performance. A drawback with SDLP is that it is influenced by the road
curvature, road width and overtakings, and it might thus not be possible to apply a
threshold for SDLP obtained in a certain study onto a dataset from another study. With
that kept in mind, SDLP is still an attractive measure because of its simplicity and
because it has been found to be associated with driver sleepiness in several, mainly
simulator, studies (see Chapter 2).
Other possible lateral measures are for example number of line crossings, maximum
line exceeding and line crossing area (i.e. the integral of the line exceeding over time or
distance, which reflects both for how long and how much the vehicle has exceeded the
line marking). These measures may reflect a more severe deterioration in driving
performance than SDLP and in the worst case, a detector based on any of these
measures might not send a warning until the driver has driven off the road. The optimal
thresholds and the performances in discriminating between alert and sleepy drivers for
the four described lateral position measures are shown in Table 10.
Table 10 Thresholds and performances for lateral position measures.
Measure
Threshold
Performance
0.28 m
0.73
0.90
0.69
Max line exceeding
0.06 m
0.67
Line crossing area
0.53 m×s
0.62
Std of lateral position (SDLP)
Number of line crossings
SDLP was found to be the best indicator and was therefore used as an input to the
classifier.
ViP publication 2011-6
41
Context information
From context information, such as time of day and trip duration, it is possible to
estimate the driver’s sleepiness level. The Sleep/Wake Predictor (SWP) is an instrument
for estimating a KSS value, given time of day, hours slept and time awake (Åkerstedt,
Folkard et al., 2004). Sandberg et al. have found that in a simulator experiment, the
SWP alone reaches a performance score of 0.78 in separating drivers with KSS≤6 from
drivers with KSS≥8 (Sandberg, Åkerstedt et al., 2011).
In the present study, the SWP was used in combination with a simple model of how
KSS changes with trip duration (time on task). Figure 22 shows the mean KSS for all
participants and sessions in the simulator experiment.
9
Mean KSS
Estimated KSS
8.5
8
KSS
7.5
7
6.5
6
5.5
5
0
10
20
30
40
Trip duration (min)
50
60
70
Figure 22 Mean KSS for all participants and sessions in the simulator experiment
(blue) and the linear function that was fitted to the KSS curve (red).
A linear function was fitted to the mean KSS curve, using a least square method:
_
0.044
5.52
The interpretation of the function is that 5.52 corresponds to the first KSS value the
drivers report in the driving sessions, averaged over all drivers and conditions. The KSS
then increases with 0.044 units per minute on average.
It could be argued that the KSS probably does not change with trip duration in the same
manner when the driver is rested as when the driver is sleep deprived, but in order to
keep the model simple and also to avoid overfitting to the two time-of-day conditions
(late afternoon and after midnight) used in this study, a common model for all
conditions was used.
42
ViP publication 6-2011
The slope from the linearly fitted KSS estimate was combined with the KSS obtained
from the SWP at the start of the driving session, resulting in an SWP based estimate:
0.044
where t is time in minutes since the start of the driving session.
An alternative to KSSSWP is to use the KSS value that the participants report just before
the driving session starts. In this way, individual differences are taken into account to
some extent, which may be beneficial. This self-report based estimate can be written
0.044
where KSSSR is the self-reported KSS value before the beginning of the session. In order
to avoid invalid KSS values, the following condition was added to both KSS estimates:
9,
9
9
Table 11 shows the threshold and performance for the two KSS estimates.
Table 11 Thresholds and performances for the KSS estimates.
Measure
Threshold (-)
Performance
KSSestSWP
7.95
0.82
KSSestSR
7.42
0.76
Interestingly, the SWP estimated KSS performs better than the self-reported
counterpart. The former is therefore the preferred choice as an input to the classifier.
A relevant question here is whether it is appropriate to use an estimated KSS value as an
input to the classifier when the aim is to separate high KSS values from low. It should
be noted that none of the two KSS estimates depend directly on the KSS values that the
drivers report during the driving sessions, although there might be some correlation
between KSSSR and the values reported during the drive. The reason for using a KSS
estimate – or actually context information – is that it carries a lot of information and has
been found to improve sleepiness detection (Sandberg, 2008).
6.1.3
Selection of classifier
A classifier attempts to assign a set of input data to one of a predefined set of classes.
Each input variable is denoted a feature. The classifier is generated by a learning
procedure, where a training data set with known class labels is used. The classifier is
then usually validated with a validation data set that is not used in the learning phase.
The performance of a classifier can e.g. be expressed as the percentage of misclassified
cases or as the average of sensitivity and specificity.
In Section 2.3, it was suggested to develop a transparent classifier for this project, since
it would be relatively easy to interpret and implement. The classification method
selected was a decision tree based on the C5.0 algorithm, which is an extension of the
ViP publication 2011-6
43
earlier ID3 and C4.5 algorithms (Quinlan, 1986; Quinlan, 1993). The C5.0 algorithm is
based on the concept of information entropy, which is a measure of the uncertainty
associated with a random variable, or in other words, the average information content
that is missing if the value of a random variable is not known (Marsland, 2009).
Information gain is defined as the difference in entropy for the whole dataset and the
entropy when a particular feature (i.e. input variable) is chosen. The C5.0 algorithm
computes the information gain for each feature and selects the one that gives the highest
value. A node that splits on the selected feature is created and the procedure is repeated
for the branches. C5.0 generated classifiers can be expressed either as a decision tree or
as set of if-then rules.
The algorithm can be run with one or more options in order to improve the performance. One of the options is to restrict the number of levels in the tree (or actually set a
lower limit on the number of training cases that follows at least two of the branches per
node), in order to avoid overtraining.
The C5.0 code is available free under the GNU GPL license and the source code
(written in C for Linux) can be downloaded from www.rulequest.com.
6.1.4
Final dataset
The final dataset consisted of the following features:




Fraction of blinks > 0.15 s from EOG (fracBlinksEOG).
Fraction of blinks > 0.15 s from Smart Eye 1-camera (fracBlinks1CAM).
Standard deviation of lateral position (SDLP).
SWP estimated KSS (KSSestSWP).
The selected features are those which gave the best performance as single indicators.
This is not a guarantee that they will be the optimal features in a classifier, but since one
aim of the project was to identify thresholds for single indicators it was most consistent
to use the selected indicators also as features.
All features were calculated in a sliding window of 5 min with 1 min resolution, as
described in Section 6.1.1. Each 5 min period of data is denoted a case.
The reported KSS values were used as class labels, where cases with an average KSS of
more than 7.5 was classified as “sleepy” while the rest of the data was classified as
“alert”. Some statistics of the dataset are shown in Table 12.
Table 12 Statistics for the training dataset. *=Smart Eye data available (where
different from the rest of the dataset)
Statistics
Alert
Sleepy
Number of cases
623 (506*)
726 (723*)
Percentage of total number of cases
46.2 (41.2*)
53.8 (58.8*)
Number of participants
15 (14*)
16
Mean number of cases per participant
42 (36*)
45
Minimum cases per participant
0
16
Maximum cases per participant
71
84 (81*)
44
ViP publication 6-2011
6.1.5
Validation method
A validation dataset was created from another simulator study that aimed to evaluate
different warning modalities for a sleepiness warning system. Twelve sleep deprived
participants (age 18–23) conducted three driving sessions on the same motorway that
was used in SleepEYE. The participants had slept three hours between 3 and 6 a.m. the
night prior to the test and conducted the driving sessions after midnight (i.e. they had
been awake for approximately 18 hours).
Sleepiness warnings were triggered based on KSS and a line crossing criteria. Only data
from before the first warning in each session was included in the validation dataset. The
validation dataset consisted of the same indicators/features as the training dataset,
except fracBlinks from the 1-camera system, since such a system was not available in
the study.
Table 13 shows some statistics for the validation dataset. From one of the participants,
no data was available at all, resulting in eleven different participants in the dataset.
Table 13 Statistics for the validation dataset.
Statistics
Alert
Number of cases
448
Percentage of cases
Number of participants
59.3
Sleepy
307
40.6
9
11
50
28
Minimum cases per participant
0
1
Maximum cases per participant
102
79
Mean number of cases per participant
The validation dataset differs from the training dataset in several ways: the drivers were
younger, less experienced and they were severely sleep deprived. However, this is a
typical study in which the classifier could be used and thus, using data from this study
as a validation dataset will give an indication of the classifier performance when it is
used as intended.
6.2
Results and analysis
This chapter starts with a presentation of the classifier performance and the results for
the training and validation datasets. The results are then analysed and discussed in order
to give a better understanding of the difficulties in detecting driver sleepiness.
6.2.1
Classifier performance
As a first step, a classifier that could be evaluated using the validation dataset was
implemented. Since there was no camera data available in the validation dataset, only
the EOG based fracBlinks could be used. In the last step, the fracBlinksEOG was replaced
by fracBlinks1CAM and a camera-based classifier was implemented.
ViP publication 2011-6
45
Table 14 shows the results for different restrictions on the tree size (a minimum number
of training cases that follows at least two of the branches per node), when SDLP,
fracBlinksEOG and KSSestSWP are used as features.
Performance is defined as the average of sensitivity and specificity, as above.
Table 14 Performance for training and validation data, using different restrictions.
Features: SDLP, fracBlinksEOG, KSSestSWP
Restriction (minimum number
of cases per branch)
Performance
Training data
Validation data
2 (default)
0.92
0.56
10
0.90
0.55
20
0.89
0.55
30
0.88
0.58
40
0.88
0.58
50
0.87
0.58
The generated tree performs very poorly on the validation dataset, regardless of what
restriction is put on the algorithm. The bad performance can be explained by the fact
that KSSestSWP, which has the highest information gain in the training dataset, is 8 or 9
for almost all cases in the validation dataset (since these drivers are severely sleep
deprived) while the rated KSS is below 7.5 in almost 60% of the cases.
Therefore, a new tree using KSSestSR instead of KSSestSWP was generated, Table 15. The
performance of the training data set decreases somewhat, while the performance of the
validation data set increases.
Table 15 Performance for training and validation data, using different restrictions.
Features: SDLP, fracBlinksEOG, KSSestSR
Restriction (minimum number
of cases per branch)
Performance
Training data
Validation data
2 (default)
0.90
0.69
10
0.86
0.69
20
0.84
0.73
30
0.83
0.66
40
0.83
0.66
50
0.82
0.72
Using the same features as in Table 15, but substituting fracBlinksEOG to that from the
1-camera, resulted in a tree that had similar performance or somewhat better than the
EOG-based counterpart for the training dataset, Table 16.
46
ViP publication 6-2011
Table 16 Performance for training data for the EOG based and the 1-camera-based
classifiers, using different restrictions.
Features: SDLP, fracBlinks (both EOG and 1CAM), KSSestSR
Restriction (minimum number
of cases per branch)
Performance
Training data, EOG
Training data, 1-cam
2 (default)
0.90
0.89
10
0.86
0.87
20
0.84
0.86
30
0.83
0.82
40
0.83
0.86
50
0.82
0.86
6.2.2
Final classifier
Since the two classifiers in Table 16 had similar performance, both are presented below.
The minimum number of cases per branch was set to 40, which was considered to be a
reasonable trade-off between overfitting versus simplifying the tree too much.
Figure 23 shows the decision tree that is based on the features SDLP, fracBlinksEOG and
KSSestSR. The performance of this tree is 0.83 (sensitivity 0.92 and specificity 0.73) for
the training and 0.66 (sensitivity 0.83 and specificity 0.50) for the validation dataset,
respectively.
Figure 24 shows the decision tree that is based on the features SDLP, fracBlinks1CAM
and KSSestSR. The performance of this tree is 0.86 (sensitivity 0.90 and specificity 0.81)
for the training dataset (no validation dataset available).
A decision tree should be interpreted as follows: first the uppermost node is evaluated,
i.e. here KSSestSR is compared to the value 7.966. If it is higher, then the driver is
classified as sleepy. If it is lower, the algorithm proceeds to the next node where SDLP
is compared to the value 0.285. If it is higher, the driver is classified as sleepy. If not,
the algorithm proceeds to the next node etc.
The classification procedure is then repeated for every new case, for example once a
minute. The features should be calculated over five minutes, since the tree was
generated based on that time interval.
The trees can be rewritten and implemented as a number of nested if statements. The
algorithms can be found in Appendix C.
It can be seen that the two trees are very similar. The two uppermost nodes are equal
and the rest of the trees have a similar structure.
ViP publication 2011-6
47
Figure 23 The decision tree with the features SDLP, fracBlinksEOG and KSSestSR.
Figure 24 The decision tree with the features SDLP, fracBlinks1CAM and KSSestSR.
48
ViP publication 6-2011
If any of these trees is to be used in a future project some additional criteria and
modifications should be considered:



6.2.3
If one wants to be sure that the driver actually is sleepy before a warning
is issued, a new root node consisting of the drivers’ reported KSS could be
added, that prevents the tree from being searched unless the KSS is at least for
example 7
It might be convenient to inhibit the warning system for a certain time after a
warning has been issued
It could be a good idea to modify the tree somewhat, in order to be able to
handle blink detection failure.
Exploration of the dataset
In order to get a better understanding of the data and the classifiers, it could be of
interest to take a deeper look at the selected features and their relationship to each other
and to individual drivers. It should be emphasized that the discussion below concerns
this specific dataset. The aim is to exemplify the challenges of sleepiness classification
and to some extent explain the results in the previous sections.
Figure 25 and Figure 26 show SDLP and fracBlinks from the EOG and the 1-camera,
respectively, for all participants and cases per class (i.e. alert vs. sleepy) in the training
dataset. The Smart Eye based and the EOG based figures are pretty similar (although
the Smart Eye based fracBlinks has a somewhat better performance than the EOG based
counterpart, Table 9). It can be seen that both SDLP and fracBlinks tend to be larger for
sleepy than for alert cases, but also that there is a substantial overlap between alert and
sleepy cases. A classifier based on SDLP and fracBlinks only will thus not have a very
good performance, regardless of what type of classifier is used.
ViP publication 2011-6
49
0.8
Alert (KSS<=7.5)
Sleepy (KSS>7.5)
0.7
0.6
SDLP
0.5
0.4
0.3
0.2
0.1
0
0
0.1
0.2
0.3
0.4
0.5
0.6
fracBlinks EOG
0.7
0.8
0.9
1
Figure 25 Scatter plot of SDLP and fracBlinksEOG for alert and sleepy cases, training
dataset from the simulator experiment.
0.8
Alert (KSS<=7.5)
Sleepy (KSS>7.5)
0.7
0.6
SDLP
0.5
0.4
0.3
0.2
0.1
0
0
0.1
0.2
0.3
0.4
0.5
0.6
fracBlinks 1CAM
0.7
0.8
0.9
1
Figure 26 Scatter plot of SDLP and fracBlinks1CAM for alert and sleepy cases, training
dataset from the simulator experiment.
Adding estimated KSS values will add a lot of information to the classifier, Figure 27
(where KSSestSR is used). The KSS estimates were found to be the most information
50
ViP publication 6-2011
carrying features in the training dataset and thus, all generated classifiers have this
feature in the root node. The KSSestSWP values were found to work well for the training
dataset, but it gave a very poor performance on the validation dataset. This might be
explained by the fact that the KSSestSWP values were 8–9 for almost all cases in the
validation dataset, since the drivers were severely sleep deprived. In spite of the sleep
deprivation, a majority of the drivers rated their sleepiness lower than 8 (most data was
from the beginning of the driving sessions, before the drivers had become very sleepy) –
perhaps they were excited or nervous about the experiment, which may have counteracted their sleepiness – and as a result, the classifier performed poorly. Since participants in any study that will investigate e.g. driver reactions to sleepiness warning
systems probably will be sleep deprived, using a classifier based on KSSestSWP is
presumably not a very good idea.
When KSSestSWP was replaced by the estimated KSS that was based on the self-reported
KSS (KSSestSR), the performance of the classifier decreased for the training dataset, but
increased for the validation dataset. It is actually a bit surprising that the KSSestSWP based
classifier performs better than the KSSestSR based on the training dataset, since the latter
– at least in theory – better would handle individual differences which are known to
have a great influence in sleepiness related research.
0.8
Alert (KSS<=7.5)
Sleepy (KSS>7.5)
0.7
0.6
SDLP
0.5
0.4
0.3
0.2
0.1
0
3
4
5
6
KSSestSR
7
8
9
Figure 27 Scatter plot of SDLP and KSSestSR for alert and sleepy cases, training dataset
from the simulator experiment.
SDLP, fracBlinksEOG and KSSestSR for the validation dataset are shown in Figure 28.
These plots show a similar pattern as those for the training dataset, however, there
seems to be a larger overlap between alert and sleepy cases, and there are not as many
cases with both high SDLP and high fracBlinksEOG. Perhaps the drivers in the validation
study tended to underestimate their sleepiness (they were expected to be very sleepy) or
perhaps high SDLP values (which is the main difference between the training and the
validation plots) are related to long trips (the trips are longer in the training set than in
ViP publication 2011-6
51
the validation set). A third possible explanation may be that the cases that have both
high SDLP and high fracBlinks values originate from a few individuals.
0.8
0.8
Alert (KSS<=7.5)
Sleepy (KSS>7.5)
0.7
0.6
0.6
0.5
SDLP
SDLP
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0
Alert (KSS<=7.5)
Sleepy (KSS>7.5)
0.7
0
0.1
0.2
0.3
0.4
0.5
0.6
fracBlinks EOG
0.7
0.8
0.9
1
0
3
4
5
6
KSSestSR
7
8
9
Figure 28 Scatter plots of SDLP (y-axes), fracBlinksEOG (x-axis left) and KSSestSR
(x-axis right) for alert and sleepy cases in the validation dataset (i.e. from another
simulator study).
Figure 29 and Figure 30 show SDLP and fracBlinks (EOG and 1-camera based, respecttively) for each participant in the training dataset. It can be seen that these two features
separate alert and sleepy cases very well for a few participants (participants 16, 18, 19
and 20 for EOG based data and participants 8, 12, 18 and 20 for 1-camera-based data)
while there are more or less complete overlaps between alert and sleepy cases for other
participants (participant 17 for EOG and participants 5, 9 and 17 for 1-camera). For
most participants, however, there is a tendency towards higher SDLP and/or higher
fracBlinks (regardless of system) for sleepy than for alert cases. A notable exception is
participant 9 who has lower fracBlinks – at least for the EOG based data – for the sleepy
than for the alert cases. A possible explanation is that this participant conducted the two
driving sessions on different days (because of a simulator failure) and therefore the
EOG electrodes might not have been on exactly the same positions in the two conditions, which could have caused the unexpected result. A similar tendency can be seen
for the 1-camera-based data for participant 1, although this result is a bit more uncertain
since there are very few alert cases.
For some participants, there are notable differences between the EOG based and the 1camera-based plots. For example, EOG based data gives better performance for participant 19, while 1-camera-based data gives better performance for participant 8. For
participant 17, EOG based and 1-camera-based data look very different but none of
them is very good at separating alert and sleepy cases.
52
ViP publication 6-2011
Subject 01
Subject 02
Subject 05
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0
0
0
0.2 0.4 0.6 0.8
0
Subject 08
0.2 0.4 0.6 0.8
0
0
Subject 09
0.2 0.4 0.6 0.8
0
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0
0.2 0.4 0.6 0.8
0
Subject 12
0.2 0.4 0.6 0.8
0
0
Subject 13
0.2 0.4 0.6 0.8
0
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0
0.2 0.4 0.6 0.8
0
Subject 17
0.2 0.4 0.6 0.8
0
0
Subject 18
0.2 0.4 0.6 0.8
0
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2 0.4 0.6 0.8
fracBlinks EOG
0
0
0.2 0.4 0.6 0.8
0
0
0.2 0.4 0.6 0.8
0.2 0.4 0.6 0.8
Subject 20
0.6
0
0
Subject 19
0.6
0
0.2 0.4 0.6 0.8
Subject 16
0.6
0
0
Subject 15
0.6
0
0.2 0.4 0.6 0.8
Subject 11
0.6
0
0
Subject 10
0.6
0
SDLP
Subject 03
0
0
0.2 0.4 0.6 0.8
Alert (KSS<=7.5)
Sleepy (KSS>7.5)
Figure 29 Scatter plots of SDLP and fracBlinksEOG for alert and sleepy cases, per
participant, training dataset.
ViP publication 2011-6
53
Subject 01
Subject 02
Subject 05
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0
0
0.2 0.4 0.6 0.8
0
0
Subject 08
0.2 0.4 0.6 0.8
0
0
Subject 09
0.2 0.4 0.6 0.8
0
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2 0.4 0.6 0.8
0
0
Subject 12
0.2 0.4 0.6 0.8
0
0
Subject 13
0.2 0.4 0.6 0.8
0
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2 0.4 0.6 0.8
0
0
Subject 17
0.2 0.4 0.6 0.8
0
0
Subject 18
0.2 0.4 0.6 0.8
0
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2 0.4 0.6 0.8
fracBlinks 1CAM
0
0
0.2 0.4 0.6 0.8
0
0
0.2 0.4 0.6 0.8
0.2 0.4 0.6 0.8
Subject 20
0.6
0
0
Subject 19
0.6
0
0.2 0.4 0.6 0.8
Subject 16
0.6
0
0
Subject 15
0.6
0
0.2 0.4 0.6 0.8
Subject 11
0.6
0
0
Subject 10
0.6
0
SDLP
Subject 03
0
0
0.2 0.4 0.6 0.8
Alert (KSS<=7.5)
Sleepy (KSS>7.5)
Figure 30 Scatter plots of SDLP and fracBlinks1CAM for alert and sleepy cases, per
participant, training dataset.
Figure 31 and Figure 32 show the sensitivity, specificity and performance per participant for the two classifiers, which further illustrates the individual differences. (It may
appear strange that e.g. participant 16 has a specificity>0 for the 1-camera-based classifier although there are no alert cases in Figure 30. Actually there are alert cases but they
are not shown in Figure 30 since blink data is missing for this participant.)
The plots per participant indicate that it might be possible to improve the performance
of the classifier if relative differences rather than absolute values were used as input
54
ViP publication 6-2011
data. However, such a classifier would be impractical, since it requires that the test
driver conduct a baseline/alert session in order to get reference data.
EOG based classifier
Sensitivity
Specificity
Performance
1.2
1
0.8
0.6
0.4
0.2
0
1
2
3
5
8
9
10 11
Subject
12
13
15
16
17
18
19
20
Figure 31 Sensitivity, specificity and performance per participant for the EOG based
decision tree, training dataset.
1-camera based classifier
Sensitivity
Specificity
Performance
1.2
1
0.8
0.6
0.4
0.2
0
1
2
3
5
8
9
10 11
Subject
12
13
15
16
17
18
19
20
Figure 32 Sensitivity, specificity and performance per participant for the 1-camerabased decision tree, training dataset.
ViP publication 2011-6
55
7
Discussion and conclusions
7.1
The Smart Eye embedded camera system
The blink detection performance of the three systems shows very different results,
where the EOG detects nearly all blinks, the 1-camera system detects about half of the
blinks and the blink detection of the 3-camera system is somewhere in between. It may
not be important to detect all blinks when designing a sleepiness estimator, but if half of
the blinks are lost, it becomes very important that the blink parameters that are extracted
from the detected blinks are correct and that those missing are not due to a systematic
error. The one-minute segment that was selected for analysis from each driving session
(from the real road experiment) should be considered as a best case scenario, since it
was the minute with the highest eyelid quality in a five-minute window, on a road
stretch without entrances and exits. Furthermore, it was rather early in the trip so no
drivers had aborted yet and they had not reached the highest levels of sleepiness. This
implies that the general performance of the camera systems as well as of the EOG may
be lower than what is presented here.
A very positive finding for all three systems is that there were very few false blink
detections. However, as mentioned above, the detection evaluation is done on data
where the participants were relatively alert. When drivers get sleepy, artefacts are often
seen, because of movements, changes in posture, yawning etc., which possibly can have
an effect on blink detection.
A majority of the results indicate that the EOG and the 3-camera system provide data
reflecting similar physiological blink parameters. The number of detected blinks, and
consequently the blink frequency, is much lower in the 3-camera system. However, the
distributions of blink durations are similar, the difference in blink duration between alert
and sleep deprived drivers is similar and the relation between blink duration and KSS is
similar. Looking at individual drivers, the two systems diverge somewhat more, but the
general trend is still that the blink duration increases with sleepiness. Looking at the 1camera system, the situation is almost reversed. The distribution shifts towards shorter
blink durations for sleep deprived drivers and 25% of the drivers have a shorter blink
duration when they report that they are sleepy as compared to alert. In addition,
Figure 17 shows that there is a large gap between drivers with the same KSS value
depending on if the driver is alert or sleep deprived. The result from the one-camera is
not in line with neither the EOG nor the 3-camera nor earlier research within this area.
When comparing results between EOG and camera-based systems, it is not entirely
clear how the electrical nerve/muscle signal events correlate with the visible mechanical
movement of the eye lid. There should likely be some kind of lag, and the lag may also
be different for the opening and closing actions. More knowledge on the physiological
processes behind the blink is needed in order to improve blink detection and to make the
determination of blink duration more robust and reliable, both for the EOG and for the
camera systems.
In conclusion, there are problems with all three systems. The results indicate that the
EOG algorithm detects most blinks but is unable to detect long blinks. The 1-camera
system gives systematically erroneous blink durations and many missed blink
detections. Finally the 3-camera system misses a lot of blinks but is at the same time
more capable of detecting long blinks.
56
ViP publication 6-2011
7.1.1
Recommendations for further development of the 1-camera system
The results revealed some shortcomings of the 1-camera system and also gave some
ideas on how to improve the system:





7.2
In many cases where the 1-camera system failed to detect blinks, the low level
signals such as eyelid distance clearly seemed to contain blink events. An
analysis of why these blinks are not detected would probably give new insights
on how to improve the detection algorithm.
The 1-camera and 3-camera systems both run at 60Hz, but the 3-camera system
also sub-samples the opening and closing flanks. This leads to a less noisy blink
duration signal. The 1-camera system should in the future also do sub-sampling.
The time window constraint for blink events should be expanded in order to not
drop very long eye closures.
The blink duration noise should be reduced by extracting dynamical parameters
from a set of nearby blink events. This might, on the other hand, hide variations
in blink duration that could be of interest, so this approach should be used with
care.
Separate detection criteria should be used for blink duration, blink frequency and
PERCLOS, e.g. a blink whose duration cannot be determined should still be
included in the blink frequency and PERCLOS output signals.
Sleepiness thresholds and classifier
The performance of the two classifiers that were developed is similar to what can be
found in the literature (see Section 2.2) for the training dataset, but worse for the
validation dataset. The results demonstrate the difficulties in creating a generalizable
sleepiness detector with good performance also for unseen datasets.
An interesting result is that the EOG based and the 1-camera-based decision trees have
similar performance. From the camera evaluation on the field data, it could be expected
that a classifier based on the 1-camera system would perform worse than one based on
the EOG, so this is a very positive result. A possible explanation is that the simulator
provides a much more stable environment than the instrumented car that was used in the
field test, with regard to light conditions and surrounding traffic. However, it should be
noted that fracBlinks is the least information-carrying feature included in the trees, after
KSSestSR and SDLP.
From both the camera evaluation using the field data and the development of a sleepiness classifier, it can be concluded that the blink duration from the EOG and the blink
duration from the 1-camera system are not the same measures. Both systems have pros
and cons, but an interesting advantage of the 1-camera system – in addition to its
unobtrusiveness – is its potential to detect long blinks. In the present implementation of
the blink duration algorithm, there is an upper limit of how long a blink can be, but
since long blink related indicators seem promising, future studies on how blinks are
measured and aggregated in an optimal way are recommended.
When looking deeper at the data, clear individual differences were found in how the
drivers responded to sleepiness. For most participants, both SDLP and fracBlinks tended
to increase with sleepiness, but there were large differences in which and how much the
features increased. It could also be seen that the differences between two alert (or
sleepy) drivers in many cases are larger than the difference between alert and sleepy
cases for a single participant.
ViP publication 2011-6
57
Because of the large individual differences, a limitation in the present study is the
relatively small number of participants. Unfortunately, some of the participants that
cancelled their participation could not be replaced by new participants and furthermore,
there were technical problems with the simulator resulting in loss of data. Even 20
participants is relatively few when it comes to developing a classifier, but since the
thresholds and the decision tree are relatively easy to modify, the present study should
be seen as a first step towards a reliable and robust sleepiness detector for simulator
applications.
It is not very probable that the performance of the sleepiness detector would increase
substantially if another classifier was used or if more data was available, but perhaps
that could make the classifier more robust. In order to improve the performance new
and/or more reliable features must be identified and individual differences must be
handled in a better way. One way to go for the simulator application is to use relative
changes instead of absolute levels on indicators, but this will also make the classifier
more complicated to use, since it requires “alert” data as a reference.
7.2.1
Recommendations for SleepEYE II
The results presented in this report will provide a basis for a continuation project called
SleepEYE II, where the identified sleepiness indicators will be applied on field data,
both from the present project and from EuroFOT, which is a large scale field
operational test. It is very likely that the indicators need to be modified or replaced by
other indicators in order to be useful in a field situation.
It is recommended that the 1-camera blink detection is improved before analysing the
field data, of two reasons: first of all because there is a great improvement potential in
blink detection, but also because blink parameters probably are very important
indicators in a real road situation. Since nothing is known about the driver’s sleep and
sleepiness in FOT studies, an estimated KSS value based on SWP will be rather
uncertain. Neither SDLP will be a very good indicator since it is strongly influenced by
road characteristics.
7.3
Contribution to ViP
ViP aims to build a common platform for co-operation and knowledge transfer on
driving simulator related development and applications. This concerns not only the
simulator itself but also tools and methods for simulator studies on human-technology
interactions.
The present project has resulted in a low cost 1-camera eye tracker that can be installed
and used in advanced driving simulators as well as in low-budget, less advanced driving
simulators. An eye tracking system is an essential component in driver state investigations, which is a complex and non-standardized task. A common framework for driver
state investigations will allow for comparisons of results from different studies and
facilitate exchange of knowledge and experiences.
One of the aims of the present project was to develop a sleepiness classifier for simulator applications, based on the 1-camera system. Although the results of the evaluation of
the 1-camera system revealed some shortcomings, the performance of the 1-camerabased classifier was similar to that of the EOG based counterpart.
58
ViP publication 6-2011
The sleepiness classifier is intended to be used in future simulator studies that aim at,
for example, investigating different warning modalities for sleepiness warning systems
or studying driver or driving behaviour under the influence of sleepiness. By using a
sleepiness classifier, the level of sleepiness can be assessed in an objective and
repeatable manner.
The classification algorithm, i.e. the decision tree, that was developed within in this
project is free to use in other studies on driver sleepiness.
7.4
Conclusions
Within the framework of the project a low cost head- and eye tracking 1-camera system
was developed and implemented in a test car and in the VTI driving simulator. The
literature review resulted in a list of feasible indicators for sleepiness detection (see
Table 1) and for distraction (see Appendix B).
The indicators blink frequency and blink duration were implemented in the 1-camera
system and evaluated on data from the field test. It was found that the system missed
many blinks and that the blink duration was not in agreement with the blink duration
obtained from the EOG and from a reference 3-camera system. However, the results
also indicated that it should be possible to improve the blink detection algorithm in the
embedded low cost camera system since the raw data looked well in many cases where
the algorithm failed to identify blinks.
The final set of indicators were an estimated KSS value that was based on the value the
drivers reported before the driving session (KSSestSR), standard deviation of lateral
position (SDLP) and fraction of blinks > 0.15 s (fracBlinks; EOG based and 1-camera
based). An optimal threshold for discriminating between KSS above and below 8 was
determined for each indicator. The performances were in the range of 0.68–0.76.
Two decision trees based on the selected indicators were created: one using
fracBlinksEOG and one using fracBlinks1CAM. The performances of the two trees were
0.82 and 0.83, respectively, on the training dataset, i.e., the overall performance of the
EOG based and the 1-camera-based classifier was similar, although individual
differences could be seen. The performance of the decision tree using fracBlinksEOG
decreased to 0.66 when using a validation dataset from another study, which illustrates
the difficulties in creating a generalized sleepiness classifier.
ViP publication 2011-6
59
References
Bergasa, L.M., Nuevo, J., Sotelo, M.A., Barea, R. and Lopez, M.E. (2006). "Real-time
system for monitoring driver vigilance." IEEE Transactions on Intelligent
Transportation Systems 7(1): 63–77.
Berglund, J. (2007). In-vehicle prediction of driver sleepiness - steering related
variables. Master thesis. Linköping, Sweden.
Boverie, S., Giralt, A. and Le Quellec, J.M. (2008). Diagnostic fusion for in vehicle
driver vigilance assessment. IFAC Proceedings Volumes (IFAC-PapersOnline).
Boyraz, P., Acar, M. and Kerr, D. (2008). "Multi-sensor driver drowsiness monitoring."
Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile
Engineering 222(11): 1857–1878.
Brown, I.D. (1994). "Driver fatigue." Human factors 36(2): 298–314.
Dement, W.C. and Carskadon, M.A. (1982). "Current perspectives on daytime
sleepiness: the issues." Sleep 5: 56–66.
Desmond, P.A., Matthews, G. and Hancock, P.A. (1997). Dimensions of subjective
fatigue states in driving The 14th international conference on alcohol, drugs and traffic
safety.
Dinges, D.F. (1995). "An overview of sleepiness and accidents." Journal of sleep
research 4: 4–14.
Eskandarian, A. and Mortazavi, A. (2007). Evaluation of a smart algorithm for
commercial vehicle driver drowsiness detection. IEEE Intelligent Vehicles Symposium,
Proceedings.
Golz, M., Sommer, D., Chen, M., Trutschel, U. and Mandic, D. (2007). "Feature Fusion
for the Detection of Microsleep Events." Journal of VLSI Signal Processing Systems for
Signal, Image, and Video Technology 49(2): 329–342.
Jammes, B., Sharabaty, H. and Esteve, D. (2008). "Automatic EOG analysis: A first
step toward automatic drowsiness scoring during wake-sleep transitions." Somnologie
12: 227–232.
Jin, S., Park, S.Y. and Lee, J.J. (2007). "Driver fatigue detection using a genetic
algorithm." Artificial Life and Robotics 11(1): 87–90.
Kim, Y. and Hahn, M. (2008). Detecting driver fatigue based on the driver's response
pattern and the front view environment of an automobile. Proceedings of the 2nd
International Symposium on Universal Communication, ISUC 2008.
Liu, C.C., Hosking, S.G. and Lenné, M.G. (2009). "Predicting driver drowsiness
using vehicle measures: Recent insights and future challenges." Journal of Safety
Research 40(4): 239–245.
Marsland, S. (2009). Machine learning: an algorithmic perspective. Boca Raton, FL,
Chapman & Hall/CRC.
Ohsuga, M., Kamakura, Y., Inoue, Y., Noguchi, Y. and Nopsuwanchai, R. (2007).
Classification of blink waveforms toward the assessment of driver's arousal levels – An
EOG approach and the correlation with physiological measures. Lecture Notes in
Computer Science (including subseries Lecture Notes in Artificial Intelligence and
Lecture Notes in Bioinformatics). 4562 LNAI: 787–795.
60
ViP publication 6-2011
Quinlan, J.R. (1986). "Induction of decisions trees." Mach. Learn. 1(March).
Quinlan, J.R. (1993). C4.5: Programs for Machine Learning, Morgan Kaufmann
Publishers.
Sandberg, D. (2008). Analysis and optimization of systems for detecting sleepiness in
drivers. Göteborg, Sweden, Chalmers university.
Sandberg, D. (2011). "The performance of driver sleepiness indicators as a function of
interval length." Submitted.
Sandberg, D., Åkerstedt, T., Anund, A., Kecklund, G. and Wahde, M. (2011).
"Detecting driver sleepiness using optimized nonlinear combinations of sleepiness
indicators." IEEE Transactions on Intelligent Transportation Systems 12(1).
Schleicher, R., Galley, N., Briest, S. and Galley, L. (2008). "Blinks and saccades as
indicators of fatigue in sleepiness warnings: Looking tired?" Ergonomics 51(7):
982–1010.
Shuyan, H. and Gangtie, Z. (2009). "Driver drowsiness detection with eyelid related
parameters by support vector machine." Expert systems with applications 36:
7651–7658.
Sommer, D., Golz, M. and Krajewski, J. (2008). Consecutive detection of extreme
central fatigue. IFMBE Proceedings.
Sommer, D., Golz, M., Trutschel, U. and Edwards, D. (2009). Determination of driver's
hypovigilance from biosignals. ICAART 2009 – Proceedings of the 1st International
Conference on Agents and Artificial Intelligence.
Stern, J.A., Walrath, L.C. and Goldstein, R. (1984). "The endogenous eyeblink."
Psychophysiology 21(1): 22–33.
Vadeby, A., Forsman, Å., Kecklund, G., Åkerstedt, T., Sandberg, D. and Anund, A.
(2010). "Sleepiness and prediction of driver impairment in simulator studies using a
Cox proportional hazard approach." Accident analysis and prevention 42(3): 835–841.
Williamson, A., Feyer, A. and Friswell, R. (1996). "The impact of work practices on
fatigue in long distance truck drivers." Accident analysis and prevention 28(6):
709–719.
Wright, N., Stone, B., Horberry, T. and Reed, N. (2007). A review of in-vehicle
sleepiness detection devices, TRL.
VTI. (2011). "Simulator III." Retrieved September 7, 2011, from
http://www.vti.se/templates/Page____11861.aspx.
Wu, J.D. and Chen, T.R. (2008). "Development of a drowsiness warning system based
on the fuzzy logic images analysis." Expert Systems with Applications 34(2):
1556–1561.
Vural, E., Cetin, M., Ereil, A., Littlewort, G., Bartlett, M. and Movellan, J. (2007).
Drowsy driver detection through facial movement analysis. Lecture Notes in Computer
Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes
in Bioinformatics). 4796 LNCS: 6–18.
Yang, J.H., Mao, Z.H., Tijerina, L., Pilutti, T., Coughlin, J.F. and Feron, E. (2009).
"Detection of driver fatigue caused by sleep deprivation." IEEE Transactions on
Systems, Man, and Cybernetics Part A:Systems and Humans 39(4): 694–705.
ViP publication 2011-6
61
Yang, Y., Sheng, J. and Zhou, W. (2007). The monitoring method of driver's fatigue
based on neural network. Proceedings of the 2007 IEEE International Conference on
Mechatronics and Automation, ICMA 2007.
Zhang, J., Zhu, Z.X., Song, Z.H. and Mao, E.R. (2008). Research on information fusion
on evaluation of driver fatigue. Proceedings – International Symposium on Computer
Science and Computational Technology, ISCSCT 2008.
Åkerstedt, T., Folkard, S. and Portin, C. (2004). "Predictions from the Three-Process
Model of Alertness." Aviation, Space, and Environmental Medicine 75(Suppl 1,
March): A75–A83.
Åkerstedt, T. and Gillberg, M. (1990). "Subjective and obejctive sleepiness in the active
individual." International Journal of Neroscience 52: 29–37.
Åkerstedt, T., Ingre, M., Kecklund, G., Anund, A., Sandberg, D., Wahde, M., Philip, P.
and Kronberg, P. (2009). "Reaction of sleepiness indicators to partial sleep deprivation,
time of day and time on task in a driving simulator - the DROWSI project." Journal of
sleep research.
62
ViP publication 6-2011
Appendix A
Page 1 (18)
Eye movement based driver distraction detection algorithms
Introduction
Driver inattention and distraction are major contributors to road incidents and crashes
(Gordon, 2009; Lee, Young, & Regan, 2009). Even so, drivers continue to engage
themselves in distracting tasks such as using their mobile phones or navigation systems,
eating, grooming and tending to their children. Advanced driver assistance systems may
change this pattern by alerting the driver when his or her attention diverts from the road.
Driver distraction can be defined as “the diversion of attention away from activities
critical for safe driving toward a competing activity” (Lee, et al., 2009). This is a very
general definition where the diversion of attention can be visual, auditive, physical or
cognitive and where the competing activity can be anything from mobile phone usage to
getting lost in thought. New advances in remote eye tracking technology provide a
means to counteract distracted driving in real-time. Eye movements can be used to gain
access to several types of distraction. For example, studies have shown that eye
movements are sensitive not only to visual distraction but also to auditory secondary
tasks (Engström, Johansson, & Östlund, 2005; Recarte & Nunes, 2000; Trent W Victor,
Harbluk, & Engström, 2005).
In general it is taken for granted that attention is located where the gaze is directed. In
most cases this appears to be true (Theeuwes, Kramer, Hahn, & Irwin, 1998; Yantis &
Jonides, 1984), but there are occasions in which the location of overt attention, that is,
where the gaze is directed, and of covert attention, that is, where the actual cognitive
attention is directed, are different (Hafed & Clark, 2002; Hunt & Kingstone, 2003;
Posner, 1980).
Most of the algorithms discussed in this survey follow the idea that attention and gaze
move about the environment in tandem. A special case is when the attention is directed
to internal thoughts, where the resultant eye glance can be described as “vacant staring”
(Fletcher & Zelinsky, 2005; Pohl, Birk, & Westervall, 2007; Trent W Victor, et al.,
2005) or tunnel vision (Hancock, Lesch, & Simmons, 2003; Johnson, Voas, Lacey,
McKnight, & Lange, 2004; Patten, Kircher, Östlund, & Nilsson, 2004), and where the
active scanning is reduced (Recarte & Nunes, 2000; Trent W Victor, et al., 2005; Harry
Zhang, Smith, & Witt, 2006).
The objectives of this report are to:



Summarize current research in eye movement physiology that might be useful
for distraction monitoring.
Summarize available distraction detection algorithms that are based on eye or
head tracking technology.
Suggest improvements to current driver distraction detection algorithms.
Eye movements
Distraction detection algorithms that are based on eye tracking data are usually based on
gaze data points (each registered tracking point) and in rare occasions on fixation data
where saccades have been discarded. In addition to fixations and saccades, there are
ViP publication 2011-6
Appendix A
Page 2 (18)
also other types of eye movements, such as smooth pursuits or glissades, which might
be of interest to a distraction detection algorithm. An example of eye tracking data
containing fixations, saccades and smooth pursuit is illustrated in Figure 33.
Vertical gaze direction (deg)
Horizontal gaze direction (deg)
-2
10
5
0
1869
1869.5
1870
Time (s)
-3
-4
-5
-6
1870.5
1869
1869.5
1870
Time (s)
1870.5
-3
Vertical gaze direction (deg)
Vertical gaze direction (deg)
-2
-4
-5
-6
0
5
10
Horizontal gaze direction (deg)
-2
-4
0
-6
5
1870.5
1870
10
1869.5
1869
Horizontal gaze direction (deg)
Time (s)
Figure 33 Example of eye tracking data containing fixations (blue), saccades (red) and
smooth pursuit (black). The circles represent each gaze data point.
During fixations, which are not really a movement but rather a phase during which the
eyes are kept focused on a static target in the world, visual and cognitive processing
takes place (Salvucci & Goldberg, 2000). Fixations are characterised by low velocities
(usually below 5 °/s) where consecutive gaze registrations are spatially close to each
other. Their typical duration is between 200 and 400 ms (Salvucci & Goldberg, 2000),
even though fixations of 1000 ms and more are possible. Long fixations do however
cause fatigue (Hammoud & Mulligan, 2008). The fixation duration depends on the
current task. For example, the mean fixation duration during visual search is 275 ms
while it increases to 400 ms for tasks requiring hand-eye coordination (Blignaut, 2009).
Fixations are voluntarily kept on the target via closed-loop feedback. Only the first part
of a fixation is devoted to encoding the information, while programming the next
saccade is done in the remaining time (Rayner, 1998).
Saccades are discrete movements that quickly change the orientation of the eyes, such
that the object of interest is projected on the fovea (Krauzlis, 2004). During saccades
little or no visual processing takes place (Salvucci & Goldberg, 2000). Saccades can be
voluntary or reflexive (Krauzlis, 2004). For voluntary saccades it appears to be
necessary that the spatial attention is allocated at the saccade goal (Souto & Kerzel,
2008). Saccades are very fast with a typical velocity of more than 300°/s (Salvucci &
Goldberg, 2000). The saccade velocity increases with saccade amplitude, and can range
from 30 °/s to over 900 °/s for large amplitudes (Carpenter, 1988; Hammoud &
2
ViP publication 6-2011
Appendix A
Page 3 (18)
Mulligan, 2008). There seem to be inter-individual differences in the duration of the
latency for saccade onset, which typically is about 200-250 ms, but shorter for so-called
express-saccades, whose latencies can be 25–75 ms shorter (Krauzlis, 2004). An
interesting aspect related to latencies is that we probably have directed our attention
towards a target already before the eye reaches it. Saccades are open-loop by necessity;
they are so fast that the neural feedback could not have reached them until long after the
overshoot. Therefore, corrective saccades are a common phenomenon. One saccade
cannot be followed by another with an interval shorter than 180 ms (Carpenter, 1988).
During longer saccades eye blinks are likely to occur.
Microsaccades are saccade-like eye movements with amplitudes less than 2° of visual
angle that occurs during fixations. They usually occur at a rate of 1-2 Hz during a
fixation (Betta & Turatto, 2006). Occasional microsaccades can reach velocities up to
30 – 40 °/s, but such high velocities are unusual.
Smooth pursuit eye movements are continuous movements that slowly rotate the eyes
to compensate for the motion of a visual target object with the goal to minimise visual
blur (Krauzlis, 2004; Tavassoli & Ringach, 2009) and to keep the target centred on the
fovea. The typical latency for smooth pursuit is shorter than for saccades and lies at
100-125 ms (Krauzlis, 2004; Tavassoli & Ringach, 2009). Pursuit latencies can be
reduced by visual cues, where latency reduction is greater for cues indicating the
location of the pursuit target as compared to the direction of motion. The first 100 ms of
smooth pursuit eye movements are open-loop, later on they are steered by a negative
feedback loop. The accuracy of pursuit depends on the speed of the target (Murphy,
1978). For faster targets predictive catch-up saccades are necessary. There is no lower
bound for smooth pursuit velocity, but the upper limit is usually set to 80–100 °/s. For
smooth pursuit to occur, a moving stimulus has to be present (Blake & Sekuler, 2006).
The vestibulo-ocular reflex (VOR) is a reflex movement, which keeps the retinal
image stable in the presence of head movements (Hammoud & Mulligan, 2008). It is
based on speed rather than vision and compensates eye rotation for head rotation on any
axis (Crawford & Vilis, 1991). Physiologic nystagmus is a form of involuntary eye
movement that is part of the VOR. It contains a smooth pursuit movement in one
direction and a saccade in the opposite direction and is characteristic for tracking
moving fields.
Most experiments that have been done to establish the different types of eye movements
have been done in the lab for purposes of control. During driving many of those
movements interact, however. The driver himself moves through the world, inside of
the vehicle. Therefore, the outside world can cause a physiologic nystagmus, for
example when the driver is monitoring a constant stream of cars moving in the opposite
direction. Furthermore, the driver is unlikely to keep his head completely still, which
will induce the vestibulo-ocular reflex. Fixating on stationary objects in the
environment can lead to fixations, when the target is far away and/or relatively centred,
or to smooth pursuit movements when the target is close by and placed far to the side.
Objects inside of the car are stationary relative to the driver and can be fixated. Targets
inside the car are quite close to the driver and necessitate vergence, a further eye
movement not described above, where the eyes rotate towards respectively away from
each other. Moving targets, like other road users, can be tracked with the help of smooth
pursuit eye movements.
ViP publication 2011-6
Appendix A
Page 4 (18)
Principles of real-time driver distraction detection
A schematic overview summarizing the structure of most driver distraction detection
algorithms is illustrated in Figure 35. The basis for all algorithms is measures registered
in real-time during driving. They can stem from the driver himself (driver behaviour),
like eye movements or hand movements, or they can be logged from the vehicle
(driving behaviour), like speed or lateral position. Furthermore, situational variables like
time and position can be used (other data). Certain features of these data, like gaze
direction, standard deviation of lateral position or others are extracted and possibly
fused in order to arrive at a continuous measure of the driver’s distraction level. This
output is then used to classify the driver’s state of attention. For most algorithms these
states are visually distracted vs. not visually distracted, while one algorithm has an
additional output of internally/cognitively distracted.
warning algorithm
classification
no warning
warning
strategy
fusion
feature extraction
driving
behaviour
not
distracted
other data
measures
driver
behaviour
distraction detection algorithm
warning
Figure 34 A schematic overview of the common features of most driver distraction
detection algorithms and mitigation systems.
Field relevant of driving
Common for all eye or head movement based distraction algorithms is that they use offroad glances as the basic source of information. The idea is to define a field relevant for
driving, which is basically the area where the driver is looking when he or she is
driving. If a world model is not available, the field relevant for driving can, for example,
be defined as a circle (Engström & Mårdh, 2007; Fletcher & Zelinsky, 2005; T. W.
Victor, 2005) or a rectangle (H. Zhang, Smith & Dufour, 2008), see Figure 35. It is also
possible to select different shapes. In Kircher et al. (2009), a circle where the lower part
was removed was used so that the dashboard would not be included in the field relevant
from driving. Since there is no information about where the driver is looking in the real
4
ViP publication 6-2011
Appendix A
Page 5 (18)
world, the selected field relevant from driving needs to be positioned in the real world
based on statistics of where the driver has been looking. This is often done by centering
the selected shape around the largest peak in the distribution of recent gazes.
Figure 35 Examples of two different approaches to select the field relevant from
driving. The size of the area is defined by the angles in the figure. Typical values are 8°
for the circle (T. W. Victor, 2005) and 24° x 24° for the rectangle (H. Zhang, et al.,
2008).
If the eye tracking systems allows a world model to be used, the field relevant for
driving can be defined based on different zones related to the interior of the car. This
approach is used by Pohl et al. (2007) and Kircher et al. (2009), see Figure 36. In the
latter of these two, the field relevant for driving is defined as the intersection between a
viewing cone of 90 degrees and the vehicle windows. This means that the circular field
relevant for driving concept in Figure 35 is expanded with information about the design
of the car.
Figure 36 Examples of different zones that can be used when defining the field relevant
for driving when a world model is available.
In Figure 37, glances away from the road ahead (i.e. outside the field relevant for
driving) are coloured in red. The duration of these glances away from the road ahead is
the basic source of information that all visual distraction detection algorithms to date are
based upon. If the driver is looking away from the road too often or for too long, the
driver is considered distracted.
ViP publication 2011-6
Appendix A
Page 6 (18)
Figure 37 Example of gaze data points plotted over time. The grey cylinder represents
a circular field relevant for driving plotted over time. Blue gaze data points reside in
the cylinder and are thus defined as on-road while red gaze data points are off-road.
Distraction estimation
The mappings that transform glances away from the road to a continuous distraction
estimate are often very similar. For example, Zhang et al. (2008) used the average
duration of glances away from the road in a 4.3-second wide sliding window, Donmez
et al. (2007) used a weighted sum of the current glance and the average glance duration
in a 3-second sliding window and Victor (2005) used the percentage of on-road gaze
data points in a 60-second sliding window. A slightly different approach is to use a
buffer (Kircher & Ahlstrom, 2009) or a counter (Fletcher & Zelinsky, 2005) that
changes its value when the driver looks away. Here the counter/buffer reaches a
maximum/minimum value when the driver is judged to be too distracted. In the AttenD
algorithm, the driver has a time buffer with a maximum value of two seconds (Kircher
& Ahlstrom, 2009), see Figure 38.
Earlier research have shown that eye glances away from the road rarely exceed a
duration of two seconds (Dukic, Hanson, Holmqvist & Wartenberg, 2005; Kircher,
Kircher, & Ahlstrom, 2009; Wikman, Nieminen & Summala, 1998), and most normal
glances range from about 0.7 seconds to slightly above one second (Kircher, 2007).
Glances away from the road that are longer than two seconds are often considered
dangerous. This threshold was originally determined by Zwahlen et al. (1988) who
stated that glances away from the road with a duration of more than two seconds lead to
unacceptable lane deviations. Also, Klauer et al. (2006) found that glances away from
the road with a duration of more than two seconds more than double the odds of a crash.
In general, drivers use repeated glances instead of extending one single glance, if the
secondary task demands attention for a longer period of time. It has been shown,
however, that repeated glances have more detrimental effects on driving performance
than a single glance of the same duration as one of the repeated glances (Tijerina,
Parmer & Goodman, 1999). It is therefore important that distraction detection
algorithms take glance history into account.
6
ViP publication 6-2011
Appendix A
Page 7 (18)
Figure 38 Example of time trace illustrating the development of the time buffer that is
used in AttenD (Kircher & Ahlstrom, 2009). Three consecutive one-second glances
away from the field relevant for driving (FRD), marked dark gray, with half-second
glances back to the FRD in between. Note that different latency periods are used to
influence the value of the buffer (0.1s before increasing the buffer and 1s before the
buffer starts to decrease when looking in a mirror).
So far, there has been a direct link from the field relevant for driving via the glance
duration to the estimated distraction level. However, it is possible to make this link
fuzzier in an attempt to get a better estimate of distraction. It is reasonable that it
becomes more dangerous the further away from the road centre the driver is looking.
One idea is thus to penalize glances that are far away from the road centre. In the
SafeTE project (Engström & Mårdh, 2007), this was done by the so-called eccentricity
function E(α) = 6.5758 −1/(0.001*α + 0.0152). This is basically a weighting function
that favours glances close to the road centre while penalizing glances with a large gaze
direction angle. The equation is based on a study by Lamble et al. (1999) and is related
to visual behaviour and brake response when a lead vehicle suddenly starts to
decelerate. In cases where a world model is available, it is possible to use different
weights on different objects (Kircher & Ahlstrom, 2009; Pohl, et al., 2007). For
example, the rear view mirrors and the speedometer could have a higher weight as
compared to the field relevant for driving but lower than the middle console or the glove
compartment. Higher weights in this context mean that the distraction estimate will
increase faster while lower weights have the opposite effect.
Distraction decision
The continuous distraction estimate needs to be mapped to a decision whether the driver
is distracted or not. Basically, the driver enters the distracted state when a threshold is
reached and returns to the attentive state when some criteria are fulfilled. The main
difference between different approaches is how to leave the distracted state. One
approach is to require that the driver is looking forward for some minimum time before
he or she is considered to be attentive (Donmez, et al., 2007; Kircher & Ahlstrom, 2009;
Pohl, et al., 2007). The other approach is that it is enough for the driver to look back at
the road to be considered fully attentive (Fletcher & Zelinsky, 2005).
Inhibition criteria
A distraction detection algorithm determines whether a driver is distracted or not, but
when and in which way the driver will be warned for distraction is determined by the
warning strategy. Information about different warning strategies is out of the scope of
ViP publication 2011-6
Appendix A
Page 8 (18)
this review. More information can be found in, for example, Donmez et al. (2009).
However, there are situations when it is not suitable to give distraction warnings. For
instance, if the driver is braking hard he or she is probably aware of the situation and
should not be disturbed by a warning. For this reason, certain criteria can be set up to
inhibit warnings. Common criteria include (Kircher, Kircher & Claezon, 2009):
 Speed
Below 50 km/h gaze behaviour is not very uniform. The
gaze is often outside the FRD without the driver being
distracted.
 Direction indicators Changing lanes and turning can include planned glances
outside the FRD.
 Reverse gear
Reverse engaged means that the driver should look over
the shoulder.
 Brake pedal
No warning should be given while driver is braking, in
order not to interfere with critical driving manoeuvres.
 Steering wheel angle No warning should be given while the driver is engaged
in substantial changes of direction, in order not to
interfere with critical driving manoeuvres.
 Lateral acceleration No warning should be given when the vehicle makes
strong movements, in order not to interfere with critical
driving manoeuvres.
Survey of available distraction detection algorithms
To get a picture of how different distraction algorithms fit into the framework described
in the section on eye movements, all available eye tracking based distraction detection
algorithms that we could find are listed below:


8
Per cent road centre (T. W. Victor, 2005; Trent W Victor, et al., 2005):
o Field relevant for driving: A road centre area which is defined as a
circular area of 16 degrees diameter, centred around the road centre
point. The road centre point was determined as the mode, or most
frequent gaze angle, of each subject’s baseline driving data.
o Distraction estimation: The percentage of gaze data points in a 1-minute
sliding window.
o Distraction decision: Too low PRC values are indicative of visual
distraction, while very high values are indicative of cognitive distraction.
SAVE-IT (Donmez, et al., 2007):
o Field relevant for driving: Away from the road is defined as glances
towards an IVIS-display. Glances directed everywhere else in the world
are considered to be on the road.
o Distraction estimation: A weighted average of the current off-road
glance duration (1) and the average off-road glance duration (2) during
the last 3 seconds. The weighting factor, or relative influence of the
current glance duration, is the variable . These factors together
determine the momentary value of distraction  with  = 1 + (1 - )2,
where  = 0.2.
o Distraction decision: The driver is considered slightly distracted if  ≥ 2
and severly distracted if  ≥ 2.5.
ViP publication 6-2011
Appendix A
Page 9 (18)




2in6 (Klauer, et al., 2006)
o Field relevant for driving: Video recordings were analysed manually to
determine when a driver looked away from the road.
o Distraction estimation: The cumulative glance duration away from the
road in a 6-second sliding window.
o Distraction decision: The driver was considered distracted if the
distraction estimate exceeded 2 seconds. This threshold gave results that
were significantly related to crash/near-crash involvement.
AttenD (Kircher & Ahlstrom, 2009; Katja Kircher, et al., 2009)
o Field relevant for driving: The intersection of a circular area of 90
degrees diameter and the windows of the car.
o Distraction estimation: A 2-second buffer is decremented when the
driver looks away from the field relevant for driving. The mirrors and the
speedometer are treated with latency periods of one second, meaning that
it is allowed to look in these zones for one second before the buffer starts
to decrease. There is also a latency period of 0.1 second before the buffer
starts to increase when the driver looks back at the road again. Note that
the driver can never be more attentive than a buffer value equal to 2.
o Distraction decision: The driver is considered distracted if the buffer
runs empty.
Fletcher’s algorithm (Fletcher & Zelinsky, 2005)
o Field relevant for driving: Same as per cent road centre above.
o Distraction estimation: Similar to AttenD. A counter increases
continuously over time. When a gaze data point is directed towards the
road ahead, the counter is reset. The distraction estimate is a function of
speed, either as the inverse or the squared inverse.
o Distraction decision: Not defined in the article, but when the distraction
estimate reaches a certain value, a warning is given.
Pohl’s algorithm (Pohl, et al., 2007)
o Field relevant for driving: No particular field relevant for driving.
Different zones in the cockpit are assigned weights that are used in the
distraction estimate. The windows have low weights, mirrors have
intermediate weights and other zones have high weights. Note that the
presented algorithm is based on head movements only.
o Distraction estimation: The mean weight of head direction data points in
a sliding window. Window width is not described.
o Distraction decision: The driver is considered distracted if the distraction
estimate is above a threshold. When the distraction estimate has
decreased for some time, the driver is considered attentive again. The
thresholds are not defined in article.
Except for Fletcher’s algorithm, all algorithms take the recent glance history into
account instead of only focusing on the very last glance off the road. This is based on
the notion that not only single long glances, but also repeated glances are detrimental
for traffic safety (Tijerina, et al., 1999; Tsimhoni, 2003). A summary of the relationship
between glance duration and traffic safety can be found in Kircher (2007). By taking
several glances into account, visual time sharing can be dealt with in a proper way. An
ViP publication 2011-6
Appendix A
Page 10 (18)
example of visual time sharing was illustrated in Figure 38, where three consecutive
glances away from the road lead to a distraction warning.
The time horizon that is utilized by the different algorithms is either of variable duration
or of fixed duration. The per cent road centre, the 2in6, the SAVE-IT and Pohl’s
algorithm uses fixed time horizon, implemented as a sliding window. The durations
range from 3 seconds in the case of SAVE-IT to 60 seconds for certain per cent road
centre implementations. Fletcher’s algorithm and AttenD on the other hand has a
variable time horizon. In Fletcher’s algorithm, the window is dependent on the current
speed and the duration of the last glance away from the forward roadway, and for
AttenD, the time window that is factually contributing to the distraction decision
depends on how long the driver has been below the maximum level of the attention
buffer.
Only one algorithm of those mentioned above, namely the per cent road centre
algorithm, explicitly considers the concept of internal distraction (also called cognitive
distraction). The idea is that high gaze concentrations towards the road centre area is
indicative of cognitive distraction (T. W. Victor, 2005). The other algorithms were
developed with only visual distraction in focus, but theoretically it should be possible to
extend them to take internal distraction into account. One approach could be to use
prolonged glances towards the field relevant for driving as a surrogate measure of the
gaze concentration.
Improvements to current algorithms
Most of the presented algorithms are relatively new and not tested and validated
extensively. Therefore, there is room for improvement, and in this section, different
suggestions are made both for the detection of visual distraction and for the detection of
internal distraction.
Eye movements
Most mentioned algorithms focus on visual distraction only. This type of distraction
means that the driver is not looking where he or she is supposed to be looking, or in the
terminology used in the algorithms, the driver is not looking into the field relevant for
driving. This implies that two steps are necessary to optimise the performance of the
algorithm. Firstly, the definition of the field relevant for driving must be as accurate as
possible at all times, meaning that the field relevant for driving should change shape,
size and position dynamically over time. Secondly, the allowed duration for glances
away from the road must be adapted to the current situation. Thus, for visual distraction
it appears more important to improve those aspects than to consider different parameters
of eye movement types. The theoretical motivation behind this is that visual distraction
is context dependent – it can only be determined if it is known what one should focus
on. An improvement of the definition of the field relevant for driving could thus be
achieved by external sensors, which monitor, analyse and judge the surrounding traffic
and the environment.
Internal distraction, however, is a state that is much less context dependent. Instead of
focusing on the task at hand, one focuses on thoughts, or does not focus at all. Still, it is
likely that the eyes are open, even though the person in question might not actually see
10
ViP publication 6-2011
Appendix A
Page 11 (18)
what he or she is looking at. This phenomenon is often called “looked but failed to see"
in the literature (Sabey & Staughton, 1975; Staughton & Storie, 1977). There are
indications, however, that certain eye movements also can differentiate between internal
distraction and attention. Some of the results presented below are contradictory, but it
might still be useful to investigate how eye movements are related to internal
distraction, and possibly also to visual or other types of distraction.
The fixation duration has been found to vary with cognitive load, but the results are
contradictory. Reyes and Lee (2008) states that fixations are shorter during cognitive
load as compared to baseline driving while Recarte and Nunes (2000) found an increase
in average fixation duration. An important difference between the two studies is that
Reyes study was run in a simulator while Recarte and Nunes conducted a field trial.
Giannopulu et al. (2007) found that a number of fixation and saccade parameters
differed in a computer generated driving scene in a simulator, as compared to the same
scene recorded on video and driven in a simulator.
The gaze concentration on the road centre is higher during cognitive load as compared
to baseline. A number of studies support this finding. The standard deviation of
horizontal, vertical and radial gaze has been found to decrease with increased task
difficulty (Recarte & Nunes, 2000; T. W. Victor, 2005; Trent W Victor, et al., 2005).
Radial gaze is defined here as the l2-norm of the horizontal and vertical gaze
components. Other measures of gaze concentration such as the percent road centre
metric and the size of the visual field provides similar results (Rantanen & Goldberg,
1999; Recarte & Nunes, 2000; Reyes & Lee, 2008; T. W. Victor, 2005). However, if
workload is imposed by a visual display, the visual field increases instead (Harry
Zhang, et al., 2006).
The saccade amplitude has been found to be either larger (May, Kennedy, Williams,
Dunlap, & Brannan, 1990; Reyes & Lee, 2008) or smaller (Recarte & Nunes, 2000)
during cognitive load as compared to baseline. It has also been found that the saccade
speed is faster and the variation in saccade speed is higher during cognitive load
(Reyes & Lee, 2008), but this is a direct consequence of the relationship between the
distance, speed and duration of a saccade (Carpenter, 1988) as was described in the
previous section on eye movements.
Blink frequency decreases for visually demanding secondary tasks, because more
visual information must be processed, whereas memory demanding tasks increase blink
frequency (Liang, Lee & Reyes, 2007).
The studies that investigated internal distraction used a secondary task that induced
cognitive load, but it can be discussed whether this really corresponds to
internal/cognitive distraction (Kircher & Ahlstrom, 2009). Provided that there are
similarities between externally induced cognitive load and internal distraction, the
presented findings should be investigated further.
Further aspects such as parameters related to smooth pursuits and microsaccades,
which have not been considered at all in the literature, could also be of interest. An
important step is to develop a method that allows tapping into true internal distraction,
which occurs without externally imposing cognitive load via a secondary task. This
might also help to explain some of the apparent contradictions above. To be able to use
most of the parameters above successfully, it is necessary, however, to use a
segmentation algorithm that detects the different eye movements reliably from recorded
ViP publication 2011-6
Appendix A
Page 12 (18)
data, as described in the next paragraph. Furthermore, the relationship between these
eye movements and the actual location of attention has to be taken into account.
Segmentation of eye tracking data
To be able to utilize the information that can be found in fixations, saccades and smooth
pursuits, it is essential that reliable real-time algorithms are available for segmentation
of the constant flow of gaze data points. This is not an easy task. For example, it has
been shown that the duration of detected fixations is very dependent on the choice of
parameters in the chosen algorithm. In fact, significant results can be made insignificant
and even reversed with different parameter settings (Blignaut, 2009).
Gaze data points are at least three or four dimensional with one time coordinate and two
or three spatial coordinates. This means that segmentation algorithms can take
advantage of temporal information, spatial information or both. Temporal characteristics that can be employed are, for example, that fixations are rarely less than 100 ms
(Salvucci & Goldberg, 2000) and that the duration between two consecutive saccades is
greater than 180 ms (Carpenter, 1988). Spatial characteristics on the other hand are
often measured via eye movement velocity or eye movement dispersion (Salvucci &
Goldberg, 2000). Velocity and dispersion based algorithms use two conceptually
different approaches to solve the segmentation problem; velocity algorithms try to
detect the saccades and say that everything in between are fixations while dispersion
algorithms try to detect fixations and say that everything in between are saccades.
Dispersion based algorithms identify fixations by looking for samples within a spatially
limited region for a minimum amount of time. The main disadvantage with this
approach is that it is sensitive to noise and drifts in the data. This is an issue since
driving entails a large amount of smooth pursuits, i.e. drifts in the data. Another
disadvantage is that the time resolution of onset and offset of saccades is rather poor
(Nyström & Holmquist, 2009). The problem with drifts in the data could probably be
avoided by allowing a more flexible rule when defining the spatial region, for example,
by allowing the region to be translated in space as a function of time. A suitable preprocessing step is to smooth the gaze data points with, for example, a median filter since
such smoothing algorithms are edge preserving. Other smoothing approaches include
the mean shift algorithm which is basically a multidimensional low-pass filter
implemented with a Gaussian kernel (Santella & DeCarlo, 2004). Closely related to the
dispersion algorithms are other cluster algorithms such as minimum spanning trees
(Salvucci & Goldberg, 2000) or projection clustering (Urruty, Lew, Ihadaddene, &
Simovici, 2007), but they are mostly applicable to finding areas of interest in pictures or
video recordings.
Velocity based algorithms uses the fact that saccadic eye movements are faster than eye
movements related to fixations. Usually, the eye movement velocity is calculated by
differentiating the l2-norm of the spatial coordinates (Salvucci & Goldberg, 2000). Since
eye tracking data is noisy, it is always cumbersome to calculate derivatives since the
differentiation operator amplifies noise. A disadvantage with velocity based algorithms
is thus the lack of robustness. The robustness can be increased for example by using a
hidden Markov model (with the two states saccade and fixation) or by smoothing the
data before the differentiation. In either case, the edges (i.e. the saccades) in the signal
will be blurred, thus giving poor time resolution for the onset and offset of saccades.
Nyström and Holmquist (2009) suggests that the differentiations should be implemented
12
ViP publication 6-2011
Appendix A
Page 13 (18)
via the Savitsky-Golay smooting filter. This is a sensible approach since the derivatives
are performed analytically on a polynomial fitted to the raw data in a least-squares
sense. In this way, it is possible to estimate both the velocity and the acceleration of eye
movements without introducing any additional noise. A prerequisite is of course that the
polynomial fit is accurate enough.
A few things to keep in mind before implementing these algorithms are:




The accuracy and precision of the measurement system needs to be taken into
account. With an accuracy of 3°, there is no meaning to try to extract
microsaccades which have amplitudes of less than 2°.
If saccade velocities are sought, data smoothing with median type filters should
be avoided since they are designed to preserve edges in a way that smooth edges
are sharpened, thus leading to an overestimate of the velocity.
Eye blinks often occurs during longer saccades. Since tracking is lost during eye
blinks, it is important not to calculate saccade velocities when tracking quality is
low.
With high quality data, there are algorithms available to detect smooth pursuit
and microsaccades (Holmqvist et al., 2011). There are however no available
algorithms to detect other events than fixations and saccades in naturalistic
remote eye movement data measured in the field. It should be fairly easy to
adapt current smooth pursuit detection algorithms developed in a laboratory
setting, but as of today, data quality issues render microsaccade detection
impossible.
Driver behaviour
While internal distraction is characterised by the fact that the driver is not interacting
with any obvious external stimuli, there are other types of distraction that could be
indicated by such an interaction. Changes in the facial features that are based on talking,
eating or yawning are candidates that could be of interest for distraction detection.
There is hope that these activities can be detected via the head model built by a remote
eye tracker, and such work is in progress, even though no results are published so far.
Psychophysiological measures can also be subsumed under driver behaviour. Variables
related to heart rate, skin conductance, brain activity, posture, etc. can potentially be
related to distraction and might be considered in the future. Psychophysiological
measures can be divided in two categories; measures of emotional and physical
activation such as heart rate variability and pupil size, and measures that reflect mental
and perceptual processing such as eye movements (Lee, Reyes & McGehee, 2004). The
first category could potentially be used to detect driver distraction due to underload or
attentional withdrawal while the second category is suited for detecting when drivers are
overloaded. It is generally agreed that several different physiological measures are
needed to estimate the level of mental workload (Hankins & Wilson, 1998).
Certain activities that are executed manually, like operating the stereo system, the
heating or a mobile telephone can be indicative of driver distraction. While the
operation of embedded systems can be accessed via the controller area network (CAN)
of the vehicle, it is more difficult to monitor the operation of so-called nomadic devices.
Here, image analysis or a pressure sensitive steering wheel, which can tell whether it is
held with two hands, one hand or no hand, could be used.
ViP publication 2011-6
Appendix A
Page 14 (18)
The driver’s input into the vehicle can also been classified as driver behaviour, as
compared to the vehicle output, which is treated below under driving behaviour. The
results of the relationship between distraction and performance indicators related to
steering wheel and pedal movements differ, depending on how distraction was invoked.
Performance indicators that have been looked at in the past are steering wheel reversal
rate, average proportion of high frequency steering, brake reaction time, steering wheel
angle variation, steering entropy, accelerator release reaction time, throttle hold, and
others (Green, et al., 2007; Harry Zhang, et al., 2006; Joakim Östlund, et al., 2005).
Driving behaviour
Secondary task effects on driving behaviour are often reported in the literature.
Performance indicators that have been used include speed, variability in speed, lateral
position, variability in lateral position, number of lane exceedences, time- and distance
headway, time to collision, and others (Green, et al., 2007; Wierwille, et al., 1996;
Young, Regan, & Lee, 2009; Joakim Östlund, et al., 2005). Note that secondary tasks
are only a subset of distracted driving since it is a conscious choice to perform a
secondary task, and the driver could just as well choose not to perform the task if the
current traffic situation is complex. Driver distraction on the other hand also includes
events where the driver does not realise that he or she engages in activities that are not
relevant for driving.
A benefit of most driving performance indicators is that they are easily measured and
are not intrusive. Unfortunately, they may not be sensitive to low levels of distraction
(Lee, et al., 2004; Zylstra, Tsimhoni, Green & Mayer, 2003), and more importantly,
they are lagging indicators that can only reflect the negative results of distraction in
hindsight. Many of the performance indicators are very task dependent and do not
always vary with the level of distraction in a linear fashion. For example, lateral control
measures often indicate a more precise lateral control behaviour during moderate
cognitive loads, but during visual loads, lane-keeping variation increased (Greenberg, et
al., 2003). Similarly, longitudinal measures often indicate that the driver is increasing
the safety margin (reduced speed, increased headway) during visually demanding tasks
(Greenberg, et al., 2003; Patten, et al., 2004; J. Östlund, et al., 2004) while no effects at
all can be found during cognitive tasks (J. Östlund, et al., 2004).
In Section 5.1 it was mentioned that the field relevant for driving needs to be adapted to
the current driving situation. This also holds true for measures of driver and driving
behaviour. Different performance indicators are not only sensitive to the type of
distraction that is occurring, but also to situational variables such as the road type, road
geometry, surrounding traffic and intersections (Green, et al., 2007). It has been
suggested that the choice of performance indicators should be guided by the research
question under examination (Young, et al., 2009), so it is questionable to what extent
these measures can be used in a distraction detection algorithm for everyday use.
14
ViP publication 6-2011
Appendix A
Page 15 (18)
Conclusions and road map for future research
Available algorithms for eye tracking based driver distraction detection attempt to
detect visual distraction. All algorithms can be fitted in a common framework;
determine if the driver is looking at the road or not, convert this information into a
continuous estimate of (visual) distraction and finally use some rule, often a threshold,
to determine if the estimated level of distraction should be considered distracted or
attentive. The main limitation of these approaches is that they do not take the current
traffic situation into account. This could be done by allowing the field relevant for
driving to change dynamically over time. Future research is needed to (a) determine the
optimal field relevant for driving for different traffic situations and traffic environments
and (b) develop technology to be able to measure the current traffic situation and traffic
environment.
Only one of the available algorithms (per cent road centre) was prepared in order to
detect internal distraction. Suggested measures of internal distraction are based on the
concentration of gazes towards the road centre area, which is higher when the driver is
lost in thought. It has been suggested that other eye movements such as saccades and
microsaccades could be indicative of work load or inattention. Future research is needed
to (a) investigate eye movement physiology during driving, (b) develop remote eye
tracking technology with higher accuracy so that these small and fast eye movements
can be measured, and (c) develop algorithms that reliably and accurately detect different
types of eye movements like fixations, saccades and smooth pursuit from the continuous
data stream.
Other distraction indicators such as lateral and longitudinal control parameters seem to
be very task and situation dependent, and it is questionable whether they can be used in
a general purpose driver distraction detection algorithm. Future research includes fusion
of several data sources, including situational variables, so that the appropriate set of
performance indicators is used at exactly the right place at the right time. Even though it
might be impossible to replace eye movement related indicators completely with driving
related parameters, it would be very valuable to be able to fall back on this type of data
when eye tracking is lost.
References
Betta, E., & Turatto, M. (2006). Are you ready? I can tell by looking at your microsaccades.
NeuroReport, 17(10), 1001-1004.
Blake, R., & Sekuler, R. (2006). Perception (5th ed.). New York: McGraw-Hill.
Blignaut, P. (2009). Fixation identification: The optimum threshold for a dispersion algorithm. Attention,
Perception, and Psychophysics, 71(4), 881-895.
Carpenter, R. H. S. (1988). Movements of the eyes (2nd ed.). London: Pion.
Crawford, J., & Vilis, T. (1991). Axes of eye rotation and Listing's law during rotations of the head.
Journal of Neurophysiology, 65(3), 20-35.
Donmez, B., Boyle, L. N., & Lee, J. D. (2007). Safety implications of providing real-time feedback to
distracted drivers. Accident Analysis and Prevention, 39(3), 581-590.
Donmez, B., Boyle, L. N., & Lee, J. D. (2009). Designing feedback to mitigate distraction. In M. A.
Regan, J. D. Lee & K. L. Young (Eds.), Driver distraction: Theory, effects and mitigation (pp.
519-531). London: CRC PRess.
Dukic, T., Hanson, L., Holmqvist, K., & Wartenberg, C. (2005). Effect of button location on driver's
visual behaviour and safety perception. Ergonomics, 48(4), 399-410.
Engström, J., Johansson, E., & Östlund, J. (2005). Effects of visual and cognitive load in real and
simulated motorway driving. Transportation Research Part F: Traffic Psychology and
Behaviour, 8(2), 97-120.
ViP publication 2011-6
Appendix A
Page 16 (18)
Engström, J., & Mårdh, S. (2007). SafeTE Final Report.
Fletcher, L., & Zelinsky, A. (2005). Driver state monitoring to mitigate distraction. Paper presented at the
International Conference on Driver Distraction, Sydney, Australia.
Giannopulu, I., Bertin, R. J. V., Kapoula, Z., Brémond, R., & Espie, S. (2007). Visuomotor strategies
using driving simulators in virtual and pre-recorded environment. Paper presented at the Road
Safety and Simulation (RSS 2007), Rome, Italy.
Gordon, C. P. (2009). Crash Studies of Driver Distraction. In M. A. Regan, J. D. Lee & K. L. Young
(Eds.), Driver Distraction: Theory, effect and mitigation (pp. 281-304). London: CRC Press,
Taylor & Francis Group.
Green, P. E., Wada, T., Oberholtzer, J., Green, P. A., Schweitzer, J., & Eoh, H. (2007). How do distracted
and normal driving differ: An analysis of the ACAS naturalistic driving data (Technical Report
No. UMTRI-2006-35). Ann Arbor, MI: The University of Michigan Transportation Research
Institute.
Greenberg, J., Tijerina, L., Curry, R., Artz, B., Cathey, L., Grant, P., et al. (2003). Evaluation of driver
distraciton using an event detection paradigm. Journal of the Transportation Research
Board(1843), 1-9.
Hafed, Z. M., & Clark, J. J. (2002). Microsaccades as an overt measure of covert attention shifts. [doi:
DOI: 10.1016/S0042-6989(02)00263-8]. Vision Research, 42(22), 2533-2545.
Hammoud, R. I., & Mulligan, J. B. (2008). Introduction to eye monitoring. In R. I. Hammoud (Ed.),
Passive eye Monitoring (pp. 1-19). Berlin: Springer Verlag.
Hancock, P. A., Lesch, M., & Simmons, L. (2003). The distraction effects of phone use during a crucial
driving maneuver. [doi: DOI: 10.1016/S0001-4575(02)00028-3]. Accident Analysis &
Prevention, 35(4), 501-514.
Hankins, T. C., & Wilson, G. F. (1998). A comparison of heart rate, eye activity, EEG and subjective
measures of pilot mental workload during flight. Aviation Space and Environmental Medicine,
69(4), 360-367.
K. Holmqvist, et al., Eye tracking: a comprehensive guide to methods and measures. Oxford, UK: Oxford
University Press, 2011.
Hunt, A. R., & Kingstone, A. (2003). Covert and overt voluntary attention: linked or independent? [doi:
DOI: 10.1016/j.cogbrainres.2003.08.006]. Cognitive Brain Research, 18(1), 102-105.
Johnson, M., Voas, R., Lacey, J., McKnight, A., & Lange, J. (2004). Living dangerously: Driver
distraction at high speed. [doi:10.1080/15389580490269047]. Traffic Injury Prevention, 5, 1-7.
Kircher, K. (2007). Driver distraction: A review of the literature (VTI Report No. 594A). Linköping,
Sweden: VTI (Swedish National Road and Transport Research Institute).
Kircher, K., & Ahlstrom, C. (2009). Issues related to the driver distraction detection algorithm AttenD.
Paper presented at the 1st International Conference on Driver Distraction and Inattention.
Kircher, K., Kircher, A., & Ahlstrom, C. (2009). Results of a Field Study on a Driver Distraction
Warning System. Linköping, Sweden: VTI (Swedish National Road and Transport Research
Institute).
Kircher, K., Kircher, A., & Claezon, F. (2009). Distraction and drowsiness. A field study (VTI Report
No. 638A). Linköping: VTI.
Klauer, S. G., Dingus, T. A., Neale, V. L., Sudweeks, J., & Ramsey, D. (2006). The impact of driver
inattention on near-crash/crash risk: An analysis using the 100-car naturalistic driving study
data (Technical Report No. DOT HS 810 594). Washington DC: NHTSA.
Krauzlis, R. J. (2004). Recasting the smooth pursuit eye movement system. Journal of Neurophysiology,
91, 591-603.
Lamble, D., Laakso, M., & Summala, H. (1999). Detection thresholds in car following situations and
peripheral vision: implications for positioning of visually demanding in-car displays.
Ergonomics, 42(6), 807-815.
Lee, J. D., Reyes, M. L., & McGehee, D. (2004). SAfety VEhicles using adaptive Interface Technology
(Task 5): A Literature Review of Cognitive Distraction.: University of Iowa
Lee, J. D., Young, K. L., & Regan, M. A. (2009). Defining Driver Distraction. In M. A. Regan, J. D. Lee
& K. L. Young (Eds.), Driver Distraction: Theory, effect and mitigation (pp. 31-40). London:
CRC Press, Taylor & Francis Group.
Liang, Y., Lee, J. D., & Reyes, M. L. (2007). Nonintrusive detection of driver cognitive distraction in real
time using Bayesian networks. Transportation Research Record: Journal of the Transportation
Research Board, 2018, 1-8.
May, J. G., Kennedy, R. S., Williams, M. C., Dunlap, W. P., & Brannan, J. R. (1990). Eye movement
indices of mental workload. Acta Psychologica, 75(1), 75-89.
16
ViP publication 6-2011
Appendix A
Page 17 (18)
Murphy, B. J. (1978). Pattern thresholds for moving and stationary gratings during smooth pursuit eye
movement. Vision Research, 18(5), 521-530.
Nyström, M., & Holmquist, K. (2009). An Adaptive Algorithm for Fixation, Saccade, and Glissade
Detection in Eye-Tracking Data Behavior Research Methods In press.
Patten, C. J. D., Kircher, A., Östlund, J., & Nilsson, L. (2004). Using mobile telephones: cognitive
workload and attention resource allocation. Accident Analysis & Prevention, 36(3), 341-350.
Pohl, J., Birk, W., & Westervall, L. (2007). A driver-distraction-based lane-keeping assistance system.
Proceedings of the Institution of Mechanical Engineers Part I-Journal of Systems and Control
Engineering, 221(I4), 541-552.
Posner, M. I. (1980). Orienting of attention. Quarterly Journal of Experimental Psychology, 32, 3-25.
Rantanen, E. M., & Goldberg, J. H. (1999). The effect of mental workload on the visual field size and
shape. Ergonomics, 42(6), 806-834.
Rayner, K. (1998). Eye movements in reading and information processing. Psychological Bulletin,
124(3), 372-422.
Recarte, M. A., & Nunes, L. M. (2000). Effects of verbal and spatial-imagery tasks on eye fixations while
driving. Journal of Experimental Psychology: Applied, 6(1), 31-43.
Reyes, M. L., & Lee, J. D. (2008). Effects of cognitive load presence and duration on driver eye
movements and event detection performance. Transportation Research Part F, 11, 391-402.
Sabey, B., & Staughton, G. C. (1975). Interacting roles of road environment, vehicle and road user, in
accidents. Paper presented at the 5th International Conference Of The International Association
For Accident Traffic Medicine, London.
Salvucci, D. D., & Goldberg, J. H. (2000). Identifying fixations and saccades in eye tracking protocols.
Paper presented at the Eye Tracking Research and Applications Symposion, Palm Beach
Gardens, FL, USA.
Santella, A., & DeCarlo, D. (2004). Robust clustering of eye movement recordings for quantification of
visual interest. Paper presented at the Symposium on Eye tracking research & applications, San
Antonio, Texas
Souto, D., & Kerzel, D. (2008). Dynamics of attention during the initiation of smooth pursuit eye
movements. Journal of Vision, 8(14:3), 1-16.
Staughton, G. C., & Storie, V. J. (1977). Methodology of an in-depth accident investigation (Survey
Report No. 672): TRRL, Crowthorne, Berkshire.
Tavassoli, A., & Ringach, D. L. (2009). Dynamics of smooth pursuit maintenance. Journal of
Neurophysiology, 102(1), 110-118.
Theeuwes, J., Kramer, A. F., Hahn, S., & Irwin, D. E. (1998). Our Eyes Do Not Always Go Where We
Want Them to Go: Capture of the Eyes by New Objects. Psychological Science, 9(5), 379-385.
Tijerina, L., Parmer, E. B., & Goodman, M. J. (1999). Individual differences and in-vehicle distraction
while driving: A test track study and psychometric evaluation. Paper presented at the 5th ITS
World Congress, Seoul, Korea.
Tsimhoni, O. (2003). Time-sharing of a visual in-vehicle task while driving: findings from the task
occlusion method (Technical Report No. UMTRI-2003-13). Ann Arbor, MI: The University of
Michigan Transportation Research Institute (UMTRI).
Urruty, T., Lew, S., Ihadaddene, N., & Simovici, D. A. (2007). Detecting eye fixations by projection
clustering. ACM Transactions on Multimedia and Computing, 3(4), 23:21-23:20.
Victor, T. W. (2005). Keeping eye and mind on the road. Unpublished Doctoral thesis, Digital
Comprehensive Summaries of Uppsala Dissertations from the Faculty of Social Sciences 9,
Uppsala University, Uppsala.
Victor, T. W., Harbluk, J. L., & Engström, J. A. (2005). Sensitivity of eye-movement measures to invehicle task difficulty. Transportation Research Part F: Traffic Psychology and Behaviour, 8(2),
167-190.
Wierwille, W., Tijerina, L., Kiger, S., Rockwell, T., Lauber, E., & Bittner, A. (1996). Heavy Vehicle
Driver Workload Assessment (Task 4): Review of Workload and Related Research.
Wikman, A. S., Nieminen, T., & Summala, H. (1998). Driving experience and time-sharing during in-car
tasks on roads of different width. Ergonomics, 41(3), 358-372.
Yantis, S., & Jonides, J. (1984). Abrupt visual onsets and selective attention: evidence from visual search.
Journal of Experimental Psychology: Human Perception and Performance, 10(5), 601-621.
Young, L. K., Regan, M. A., & Lee, J. D. (2009). Measuring the effects of driver distraction: Direct
driving performance methods and measures. In M. A. Regan, J. D. Lee & K. L. Young (Eds.),
Driver distraction: Theory, effects and mitigation (pp. 85-105). London: CRC PRess.
ViP publication 2011-6
Appendix A
Page 18 (18)
Zhang, H., Smith, M., & Dufour, R. (2008). A Final Report of SAfety VEhicles using adaptive Interface
Technology (Phase II: Task 7C): Visual Distraction.
Zhang, H., Smith, M. R. H., & Witt, G. J. (2006). Identification of real-time diagnostic measures of visual
distraction with an automatic eye-tracking system. Human Factors, 48(4), 805-821.
Zwahlen, H. T., Adams, C., & DeBald, D. (1988). Safety Aspects of CRT Touch Panel Controls in
Automobiles. Paper presented at the Vision in Vehicles II.
Zylstra, B., Tsimhoni, O., Green, P. A., & Mayer, K. (2003). Driving performance for dialing, radio
tuning, and destination entry while driving straight roads (Technical Report No. UMTRI-200335). Ann Arbor, MI: The University of Michigan Transportation Research Institute.
Östlund, J., Nilsson, L., Carsten, O., Merat, N., Jamson, H., & Jamson, S. (2004). Deliverable 2 - HMI
and Safety-Related Driver Performance. (No. GRD1/2000/25361 S12.319626) Human Machine
Interface And the Safety of Traffic in Europe (HASTE) Project.
Östlund, J., Peters, B., Thorslund, B., Engström, J., Markkula, G., Keinath, A., et al. (2005). Driving
performance assessment - methods and metrics (No. D2.2.5).
18
ViP publication 6-2011
Appendix B
Page 1 (1)
Karolinska Sleepiness Scale
Instruktioner till försöksdeltagare om hur man skattar sömnighet
Karolinska Sleepiness Scale (KSS) är utvecklad för att man ska kunna ge ett mått på hur
sömnig man är. I det försök som du ska delta i kommer du att få skatta din sömnighet på
en 9-gradig skala var femte minut under varje körning, samt även vid några ytterligare
tidpunkter under försöket. Under körningarna kommer du var femte minut att bli
uppmärksammad (via en ljudsignal eller på annat sätt) på att det är dags att ge en
skattning. Du ska då svara genom att tydligt säga den siffra som bäst stämmer överens
med hur sömnig du har känt dig under tiden sedan den senaste skattningen (dvs. de
senaste 5 minuterna). Det minsta värde du kan ange är siffran 1 och den högsta siffran
är 9. Försöksledaren hör din skattning och noterar den. Skalan finns även i bilen som en
påminnelse.
Skalan som du kommer att använda ser ut på följande sätt:
1 – extremt pigg
2 – mycket pigg
3 – pigg
4 – ganska pigg
5 – varken pigg eller sömnig
6 – första tecknen på sömnighet - lätt sömnig
7 – sömnig men ej ansträngande vara vaken
8 – sömnig och något ansträngande att vara vaken
9 – mycket sömnig, mycket ansträngande att vara
vaken, kämpar mot sömnen
ViP publication 2011-6
ViP publication 2011-6
Appendix C
Page 1 (1)
Algorithms
The decision trees that were created can be implemented as a number of nested if
statements:
EOG based decision tree
if KSSestSR>7.9657
sleepy=1;
else
if SDLP>0.2850694
sleepy=1;
else
if FracBlinks<=0.02419355
sleepy=0;
else
if SDLP<=0.1855628
sleepy=0;
else
if KSSestSR<= 6.8558
sleepy=0;
else
sleepy=1;
end
end
end
end
end
1-camera-based decision tree
if KSSestSR>7.9657
sleepy=1;
else
if SDLP>0.2850694
sleepy=1;
else
if FracBlinks<=0.06430868
sleepy=0;
else
if KSSestSR<=6.73392
sleepy=0;
else
if SDLP<= 0.20127
sleepy=0;
else
sleepy=1;
end
end
end
end
end
ViP publication 2011-6
ViP publication 2011-6
ViP
Virtual Prototyping and Assessment by Simulation
ViP is a joint initiative for development and application of driving simulator methodology with a focus on the interaction between humans and
technology (driver and vehicle and/or traffic environment). ViP aims at
unifying the extended but distributed Swedish competence in the field of
transport related real-time simulation by building and using a common
simulator platform for extended co-operation, competence development
and knowledge transfer. Thereby strengthen Swedish competitiveness
and support prospective and efficient (costs, lead times) innovation and
product development by enabling to explore and assess future vehicle
and infrastructure solutions already today.
Centre of Excellence at VTI funded by Vinnova and ViP partners
VTI, Saab Automobile, Scania, Volvo Trucks, Volvo Cars, Bombardier Transportation, Swedish Transport Administration,
Dynagraph, HiQ, Pixcode, SmartEye, Swedish Road Marking Association
www.vipsimulation.se
Olaus Magnus väg 35, SE-581 95 Linköping, Sweden – Phone +46 13 204000
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement