Intelligent automotive safety systems: the third age challenge

Intelligent automotive safety systems: the third age challenge
Loughborough University
Institutional Repository
Intelligent automotive safety
systems: the third age
challenge
This item was submitted to Loughborough University's Institutional Repository
by the/an author.
Additional Information:
• A Doctoral Thesis. Submitted in partial fullment of the requirements for
the award of Doctor of Philosophy of Loughborough University.
Metadata Record:
Publisher:
https://dspace.lboro.ac.uk/2134/18485
c Imran Amin
This work is made available according to the conditions of the Creative
Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BYNC-ND 4.0) licence. Full details of this licence are available at: https://creativecommons.org/licenses/bync-nd/4.0/
Rights:
Please cite the published version.
~
•
Loughborough
University
University Library
AuthorlFiling Title .......
I\r\\N........\.: .. A.: ................ ..
........................................................................................
---
Class Mark .................................. .\. ................................ .
Please note that fines are charged on ALL
overdue items.
00" REFERENC ONLV
0403604079
1111111111111111111111111111111111111111111111
INTELLIGENT AUTOMOTIVE SAFETY SYSTEMS:
THE THIRD AGE CHALLENGE
By
Imran Amin
A Doctoral Thesis
Submitted in partial fulfilment of the requirement
for the award of Doctor of Philosophy
of Loughborough University
December 2006
© Imran Amin 2006
·
~r:'1
t1:£;;
Loughhorough
tini\'tTsity
J:,lh1f2!()!' I.
r---._._.-...
-~
Date
Class
ihrary
--
Jz1!::Qo <t
"\
~~~ 040) ~ 4-oiCJ
..
Abstract
Over 300,000 individual are injured every year by vehicle related accident in the United Kingdom
alone. Government and the vehicle manufacturers are not only bringing new legislation but are also
investing in vehicle safety research to bring this figure down.
A private self-driven car is an important factor in maintaining the independence and quality of life of
the third age individuals. However, since older people brings deterioration of cognitive, physical and
visual abilities, resulting in slower reaction times and lapses while driving. The third age individuals
are involved in more vehicle related accidents than middle aged individuals. This scenario is corrected
by the fact that the number of third age individuals is increasing, especially in developed countries. It is
expected that the percentage of third age individuals in the United Kingdom will increase to 20% of the
total population by 2010.
Several safety systems have been developed by the automotive industry including intelligent airbags,
Electronic Stability Control (ESC) and pre-tensioned seat belts, but nothing has been specifically
developed for the third age related problems.
This thesis proposes a driver posture identification system using low resolution infrared imaging. The
use of a low resolution thermal imager provides a reliable non-contact based posture identification
system at a relatively low cost and is shown to provide robust performance over a wide range of
conditions. The low resolution also protects the privacy of the driver.
In order to develop the proposed safety system an Artificial Intelligent Thermal Imaging algorithm
(AITl) is created in MatLAB. Experimentation is conducted in real and simulated environment, with
human subjects, to evaluate the results of the algorithm.
The result shows that the safety system is able to identify eighteen different driving postures. The
system also provides other valuable information about the driver such as driver physical built, fatigue,
smoking, mobile phone usage, eye-height and trunk stability. It is clear that in incorporating this safety
system in the overall automotive central strategy, better safety for third age individual can be achieved.
This thesis provides various contributions to knowledge including a novel neural network design, a
safety system using low resolution infrared imager and an algorithm that can identify driver posture.
Keywords: Vehicle Safety, Infrared, Third Age, Artificial Neural Network, Image Processing.
Acknowledgements
First I would like to say Alhamdulliah, and thanks to Allah for giving me write this thesis.
Many people are due thanks particularly Mr Andrew J Taylor and Professor Rob Parkin for their
support as supervisors. I would also like to thank all my colleagues at Mechatronics Research Centre
for their help and support.
Special thank to my parents Jawaid Arnin and Shahzadi Urfana for their support, encouragement and
prayers, without which I could not have finished this thesis.
r - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - --- -
-
Intelligent Automotive Safety Systems: The Third Age Challenge
Contents
List of Figures .............................................................................vii
List of Tables ............................................................................... xi
List of Equations ......................................................................... xii
Nomenclature ............................................................................ xiii
Acronyms .................................................................................... xiv
1.
INTRODUCTION ............................................................................... 1
1.1.
Third age drivers ............................................................................................................. 2
1.2.
Motivation for research .................................................................................................. 4
1.3.
Aim of research ............................................................................................................... 5
1.4.
Scope ................................................................................................................................ 6
1.5.
Boundaries of the research ............................................................................................. 7
1.6.
Methodology .................................................................................................................... 7
1.7.
Thesis outline ................................................................................................................... 8
1.8.
Summary ........................................................................................................................ 10
2.
LITERATURE REVIEW ...................................................................12
2.1.
General driver problems .............................................................................................. 12
2.1.1.
Young age dri vers ...................................................................................................... 12
2.1.2.
Driver distraction ........................................................................................................ 13
2.1.3.
Drink and Drugs risk .................................................................................................. 14
2.1.4.
Sleepiness and fatigue ................................................................................................ 15
2.1.5.
Disableddrivers ......................................................................................................... 16
2.1.6.
Driver's Vision .................................................. :........................................................ 17
2.1. 7.
Third age drivers and age related disabilities ............................................................. 17
Vision problems in third age drivers .................................................................................... 20
Driving habits of third age people ....................................................................................... 22
Attentional ability ................................................................................................................ 23
2.1.8.
Safety for third age people: Memory & Motor skills ................................................. 25
Intelligent Automotive Safety Systems: The Third Age Challenge
2.2.
Safety in cars with intelligent sensors .......................................................................... 28
2.2.1.
Primary safety sensors ................................................................................................ 28
Pre tension seat belts ............................................................................................................ 28
Radar and ultrasonic sensors ................................................................................................ 29
Laser scanners ...................................................................................................................... 32
Night vision (Infrared or Near Infrared vision) .................................................................... 33
Occupant position sensor and driver measurement .............................................................. 34
Anti-lock brakes (ABS) ....................................................................................................... 34
Electronic stability control (ESC) ........................................................................................ 35
2.2.2.
Secondary safety sensors ............................................................................................ 35
Airbags ................................................................................................................................. 35
Side airbags (SABs) ............................................................................................................. 38
2.2.3.
Intelligent vehicle systems ......................................................................................... 38
2.2.4.
Human and obstacle tracking ..................................................................................... 39
2.3.
3.
Summary ........................................................................................................................ 42
TECHNICAL BACKGROUND .........................................................43
3.1.
Thermal imaging ........................................................................................................... 43
3.1.1.
NIR (Near Infrared) .................................................................................................... 45
3.1.2.
SWIR (Short wavelength Infrared) ........................................................................... .45
3.1.3.
MWIR (Medium wavelength Infrared) ..................................................................... .45
3.1.4.
LWIR (Long wavelength Infrared) ........................................................................... .45
3.1.5.
FIR (Far Infrared) ....................................................................................................... 46
3.2.
Thermal imagers ........................................................................................................... 46
3.2.1.
Pyroelectric infrared detector ..................................................................................... 46
3.2.2.
IRISYS thennal imager .............................................................................................. 47
3.2.3.
IRISYS thennal imager construction ........................................................................ .48
3.3.
3.3.1.
Thermal imaging applications ..................................................................................... 49
Thennal imagers for human tracking ......................................................................... 53
3.4.
Infrared & Visual Image Acquisition software .......................................................... 56
3.5.
TaUey pressure matrix .................................................................................................. 57
3.6.
Visual camera ................................................................................................................ 60
3.7.
Driving simulator .......................................................................................................... 61
11
Intelligent Automotive Safety Systems: The Third Age Challenge
3.7.1.
Construction ............................................................................................................... 61
3.7.2.
Programming scenarios .............................................................................................. 62
Artificial intelligence and Image processing techniques ............................................ 62
3.8.
3.8.1.
Fuzzy logic ................................................................................................................. 63
3.8.2.
Neural networks ......................................................................................................... 64
Supervised Learning ............................................................................................................ 65
Unsupervised learning ......................................................................................................... 66
Back Propagation Neural Network ...................................................................................... 66
Radial basis neural network ................................................................................................. 68
Summary ........................................................................................................................ 69
3.9.
HYPOTHESIS ..................................................................................71
4.
4.1.
Hypothesis ...................................................................................................................... 71
4.2.
Research question ......................................................................................................... 71
4.2.1.
Secondary research questions ..................................................................................... 71
4.3.
Detailed research layout ............................................................................................... 72
4.4.
Hypothesis validation .................................................................................................... 75
4.5.
Summary ........................................................................................................................ 75
5.
IMAGING TECHNIQUES AND IMAGE PROCESSING
ALGORITHMS ........................................................................................76
5.1.
Imaging techniques ....................................................................................................... 76
5.1.1.
Difference between visual image and infrared thermograph ...................................... 77
5.1.2.
Infrared Image Interpolation ...................................................................................... 78
5.1.3.
Multiple level segmentation ....................................................................................... 79
5.1.4.
Imaging features extraction ........................................................................................ 84
5.1.5.
Image processing analysis tool: CompareIQ2 ............................................................ 84
5.2.
Image processing algorithm ......................................................................................... 86
5.2.1.
Phase 1: Imaging algorithm initial development. ....................................................... 86
Pre processing ...................................................................................................................... 86
Processing ............................................................................................................................ 87
5.2.2.
Phase 2: Advanced approach - Angled IR ................................................................. 92
iii
Intelligent Automotive Safety Systems: The Third Age Challenge
Pre processing ...................................................................................................................... 95
Processing ............................................................................................................................ 95
5.3.
6.
Summary ...................................................................................................................... 104
ARTIFICIAL NEURAL NETWORK ................................................105
6.1.
Comparison of fuzzy logic and artificial neural network ........................................ 105
6.2.
Types of neural network under consideration .......................................................... 106
6.2.1.
Scatter-gram selection method for neural network input vectors ............................. 107
6.3.
Neural network designs .............................................................................................. 114
6.4.
Single neural network designs .................................................................................... 115
6.4.1.
Large input single FFB neural network design ........................................................ 115
6.4.2.
Feature based single FFB neural network design ..................................................... 116
6.4.3.
Feature based Radial Neural network design ........................................................... 117
6.4.4.
Self organized map network design ......................................................................... 118
6.5.
Novel neural design: TNN .......................................................................................... 118
6.5.1.
Three neural network design evaluation ................................................................... 120
6.5.2.
Selection and training of the three neural network design ....................................... 121
6.6.
7.
Summary ...................................................................................................................... 126
EXPERIMENTAL SETUP ..............................................................127
7.1.
Introduction ................................................................................................................. 127
7.2.
Experiment 1: Trials with Infrared camera pointed from the windscreen ............ 127
7.2.1.
Aim of the experiment.. ............................................................................................ 127
7.2.2.
Driving simulator ..................................................................................................... 128
STISIM .............................................................................................................................. 129
Scenario ............................................................................ ;................................................ 134
7.2.3.
Reference sensors ..................................................................................................... 136
Visual Camera ................................................................................................................... 136
Talley Pressure Matrix ....................................................................................................... 137
7.2.4.
Data Acquisition platform ........................................................................................ 138
ANSI C GUI based Software Development ...................................................................... 139
Interfacing sensors ............................................................................................................. 143
iv
Intelligent Automotive Safety Systems: The Third Age Challenge
7.2.5.
Mounting ofIRISYS Imager .................................................................................... 145
7.2.6.
Volunteer drivers for Factory street experiment ....................................................... 148
7.2.7.
Offline data collected for Phase 1 Imaging Algorithm ............................................. 149
7.2.8.
Experimental environment and temperature ............................................................. 150
7.2.9.
Conclusion of experiment 1 ..................................................................................... 151
7.3.
Experiment 2: The experiment with infrared camera mounted on the right at an
angle
152
7.3.1.
Aim of the experiment. ............................................................................................. 152
7.3.2.
Holywell STISIM Driving Simulator ....................................................................... 152
Mounting low cost sensors ................................................................................................ 154
7.3.3.
Angle mounting ofIRlSYS Imager .......................................................................... 158
7.3.4.
56 Channel LED controller ...................................................................................... 159
7.3.5.
Driving simulator scenarios...................................................................................... 162
Scenario 1: Driving task and situational experimental run ................................................ 162
Scenario 2: Urban experimental run .................................................................................. 163
Scenario 3: Rural experimental run ................................................................................... 164
7.3.6.
Volunteer drivers ...................................................................................................... 164
7.3.7.
Conclusion of experiment 2 ..................................................................................... 167
7.4.
Real life video data comparison ................................................................................. 168
7.5.
Experiment 3: Real car experiment using infrared imager..................................... 170
7.5.1.
Aim of the experiment .............................................................................................. 170
7.5.2.
Peugeot 406 driving in controlled area ..................................................................... 170
7.5.3.
Mounting ofIR Imager ............................................................................................ 173
Inverter ............................................................................................................................... 174
7.5.4.
Data Acquisition platform ........................................................................................ 175
7.5.5.
Volunteer drivers ...................................................................................................... 176
7.5.6.
Conclusion of experiment 3 ..................................................................................... 176
7.6.
8.
Summary ...................................................................................................................... 177
RESULTS AND DISCUSSION ......................................................178
8.1.
Introduction ................................................................................................................. 178
8.2.
Results of Neural network as a behaviour modelling ............................................... 178
8.3.
Real life data comparison results ............................................................................... 182
v
Intelligent Automotive Safety Systems: The Third Age Challenge
8.4.
Use of P-codes in an intelligent central safety control.............................................. 184
8.5.
Real car experimentation results ............................................................................... 186
8.6.
Assessing capabilities of the system ........................................................................... 190
8.6.1.
Drowsy driving (Nodding off at steering wheel) ...................................................... 192
Analysis and discussion ..................................................................................................... 193
8.6.2.
Tiredness and fatigue ............................................................................................... 194
8.6.3.
Trunk stability .......................................................................................................... 195
8.6.4.
Out of position driver ............................................................................................... 198
8.6.5.
Eye height. ................................................................................................................ 203
8.6.6.
Head turning and dynamic allocation of attention while driving ............................. 204
8.6.7.
Large and small person or distinct driver features .................................................... 206
8.6.8.
Distance from steering wheel ................................................................................... 210
8.6.9.
Special scenarios ...................................................................................................... 215
8.6.10.
9.
Discussion ........................................................................................................... 216
CONCLUSION AND FURTHER WORK ........................................218
9.1.
Conclusion ................................................................................................................... 218
9.2.
Further work ............................................................................................................... 219
10.
REFERENCES ...........................................................................221
APPENDIX A: RESULT GRAPHS FROM ANN ..............................226
APPENDIX B: MATLAB CODE ..................................................286
APPENDIX C: IDENTIFYING OCCUPANTS .................................294
APPENDIX D: TECHNICAL DATA.............................................306
APPENDIX E: PUBLlCATIONS ..................................................315
VI
Intelligent Automotive Safety Systems: The Third Age Challenge
List of Figures
Figure 1-1 Road accident casualties in UK from 1992 to 2002 (DFT, 2002)
1
Figure 1-2 Research stages w.r.t. time
8
Figure 2-1: Third age suit developed by Loughborough University
27
Figure 2-2: Seat belt pre-tension mechanism explained (source: howstuffworks.com)
29
Figure 2-3: Distance sensors
30
Figure 2-4: Application of High-resolution radar systems in automobile
31
Figure 2-5: Grand Cherokee dual front airbags (©Cherokee motors)
36
Figure 2-6: Ultrasonic occupant position sensor
37
Figure 2-7: Side curtain airbags deployed in crash test car (copyright Honda motors)
38
Figure 2-8: Tracking of a jump while playing volleyball (Chang and Lee, 1997)
40
Figure 3-1: Typical PZT pyroelectric sensor schematic (Schreiter et al., 2006)
47
Figure 3-2 IRISYS imager sizing
49
Figure 3-3: Pedestrian classification using high resolution FLIR thermal image. Hot spots
appear in white (Armitage et al., 2005).
55
Figure 3-4 Medium resolution occupant position detector by (Qinetiq, 2004)
56
Figure 3-5 I-Quire modified version
57
Figure 3-6 Talley Pressure monitor set-up on driving seat
58
Figure 3-7: Talley Pressure monitor control
59
Figure 3-8 Resulting surface of Talley pressure monitor
60
Figure 3-9: Back Propagation Neural network construction
67
Figure 4-1 Flowchart of research programme
73
Figure 5-1: Differences between infrared and visual image
77
Figure 5-2: Four types of interpolated infrared images
79
Figure 5-3: Reduced colour infrared image
80
Figure 5-4: shows multiple layer thresholding on Infrared image showing a person driving a
vehicle.
82
Figure 5-5: Subtraction of thresholded images
83
Figure 5-6: CompareIQ2 screen capture
85
Figure 5-7: Preliminary phase imaging algorithm
86
Figure 5-8: Interpolated infrared image with histogram
88
Figure 5-9: Body segmented infrared image with histogram, inverted histogram shows
segmented region
89
Figure 5-10: Face segmented infrared image with histogram
90
Figure 5-11: Tracking movement
91
vii
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 5-12: tracking path, (A) ideal driving position, (B) putting seatbelt on, (C) rolling
window down.
92
Figure 5-13 Previous and new mounting position of IR Imager
93
Figure 5-14: advanced approach: Angled IR imaging algorithm
94
Figure 5-15: Determination of segmentation value
96
Figure 5-16 Region allocation of infrared image
97
Figure 5-17: Feature selection process
98
Figure 5-18: 'Head region' shown in interpolated infrared image
99
Figure 5-19: 'Torso region' shown in interpolated infrared image
100
Figure 5-20: 'Shoulder and arm region' shown in interpolated infrared image
101
Figure 6-1: Area scatter-gram of head region
110
Figure 6-2: Area/Bounding box scatter-gram of head region
111
Figure 6-3: Head region angle scatter-gram
112
Figure 6-4: Head region area scatter-gram
112
Figure 6-5: Torso region area scatter-gram
113
Figure 6-6: Shoulder and Arm region area scatter-gram
114
Figure 6-7: Segmented image input into multi layered neural network
115
Figure 6-8: Features based input into multi layered neural network
116
Figure 6-9: Feature based radial network design
117
Figure 6-10: Novel neural network design: TNN
119
Figure 6-11 ANN comparison result for 'torso' region
123
Figure 6-12 ANN comparison result for 'head' region
124
Figure 6-13 ANN comparison result for 'arm and shoulder' region
125
Figure 7-1 (A) Shows car simulator running scenario and Infrared camera, (B) Simulation
Control
128
Figure 7-3: STISIM hardware interfacing
130
Figure 7-4: Steering control for STISIM simulator
131
Figure 7-5: Speed control for STISIM simulator
132
Figure 7-6: Speed control with transmission consideration for STISIM Simulator
132
Figure 7-7 Bird's eye view of the intersection scenario
135
Figure 7-8: Logitech® Quickcam® Messenger
137
Figure 7-9: Data acquisition schematic
138
Figure 7-10: IRISYS thermal imaging software
139
Figure 7-11: Previously developed data acquisition software
140
Figure 7-12: Modified data acquisition software
141
Figure 7-13: Interfacing of sensors with IBM-PC for Initial experiment
144
Figure 7-14: During experiment, front mounting position of IRISYS thermal imager
145
Figure 7-15: Custom built G-Clamp stand for mounting IRISYS Thermal imager
146
Figure 7-16 Mounting of web cam and IRISYS Infrared imager for experiment
147
viii
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 7-17: Measuring instruments used in the mounting of the IRISYS imager
148
Figure 7-18: Eleven volunteers visual images and interpolated thermal images
149
Figure 7-19: Holywell test rig during experimentation
153
Figure 7-20: Holywell test rig with STISIM driving simulator
154
Figure 7-21 Difference in construction of potentiometer and encoder
155
Figure 7-22 Shows potentiometer mounting on steering column (A) potentiometer, (B)
timing belt
156
Figure 7-23: Electrical diagrams of potentiometer connections
157
Figure 7-24: During the experiment: changed position of IRISYS thermal imager
158
Figure 7-25: 56 Channel LED controller
160
Figure 7-26: Sony SVO-9620 Video editor and playback
169
Figure 7-27: Peugeot 406 used to conduct experiment
171
Figure 7-28: During the experiment
172
Figure 7-29: Specialized stand used for mounting IRISYS thermal imager inside the vehicle
174
Figure 7-30: Inverter supplies 240 AC volts from a cigarette lighter 24DC to the IRISYS
thermalimager
175
Figure 8-1: Head region FBN neural network simulation result
179
Figure 8-2: Torso region FBN neural network simulation result
180
Figure 8-3: Shoulder and arm region FBN neural network simulation result
181
Figure 8-4: Concept of intelligent central safety control
185
Figure 8-5: Multi channel driving patterns identification form intelligent central safety
control
186
Figure 8-6: Head region FBN neural network results
187
Figure 8-7: Torso region FBN neural network results
188
Figure 8-8: Shoulder and arm region FBN neural network results
189
Figure 8-9 Work plan for detecting drowsy driver
193
Figure 8-10: Drowsy driver driving for over 40 minutes
194
Figure 8-11: Driver touching her face while driving
195
Figure 8-12: Stable driver's trunk swaying effect while going over roundabout. Black line
shows trunk swaying angle.
196
Figure 8-13: Moderately unstable driver's trunk swaying effect while going over
roundabout. Black line shows trunk swaying angle.
196
Figure 8-14: Highly unstable driver's trunk swaying effect while going over roundabout.
Black line shows trunk swaying angle.
197
Figure 8-15: Trunk stability driving pattern based on time based history analysis
198
Figure 8-16: Normal driving position taken by infrared imager
199
Figure 8-17: Normal driving position in Interpolated Infrared image
199
ix
,----------------------------------------------------------------------------------------------------1
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 8-18: Normal driving position in Interpolated Infrared image (with 4 level
thresholding)
200
Figure 8-19: Driver leaning down at 80 degrees
201
Figure 8-20: Driver leaning down at 45 degrees
202
Figure 8-21: Driver putting seat belt on
203
Figure 8-22: Measuring of driver's eye height using centroid height
204
Figure 8-23: 65 Year old driver concentrating/looking down left at gear over 3 seconds 205
Figure 8-26: Driver with different built
208
Figure 8-27: Average built drivers
209
Figure 8-28: Tall driver
210
Figure 8-29: Steering distance measurement using IR Imager
211
Figure 8-30: Trigonometric calculation for calculating distance from steering wheel to
driver
212
Figure 8-31: FOV area calculation
214
Figure 8-32: Infrared images showing distance from steering wheel
214
Figure B-1: CompareIQ2 GUI interface
296
Figure C-1 Eleven volunteers visual images and interpolated thermal images
302
Figure C-2 Volunteers A, B & C
303
Figure C-3 Back Propagation ANN with 'Iogsig' as transfer function and 1 layer
305
Figure C-4 Back Propagation ANN with 'tansig' as transfer function and 1 layer
305
Figure C-5 Back propagation ANN with 'linear' as transfer function and 1 layer
306
Figure C-6 Back propagation ANN with 'Iogsig' transfer function and two hidden layers 307
Figure C-7 Back propagation ANN with 'Iogsig' transfer function and 2 hidden layers
307
Figure C-8 Back propagation ANN with 'linear' transfer function and 2 hidden layers
308
Figure C-9 Nett: Radial Basis ANN; Spread 3, Target Goal 0.0001
309
Figure C-10 Net2: Radial Basis ANN; Spread 2, Target Goal 0.0001
310
Figure C-11 Net3: Radial Basis ANN; Spread 1, Target Goal 0.0001
310
Figure C-12 Net4: Radial Basis ANN; Spread 0.75, Target Goal 0.0001
311
Figure C-13 NetS: Radial Basis ANN; Spread 0.5, Target Goal 0.0001
311
Figure C-14 Net6: Radial Basis ANN; Spread 0.25, Target Goal 0.0001
312
x
,----------------
Intelligent Automotive Safety Systems: The Third Age Challenge
List of Tables
Table 3-1 Applications of thermal imaging (Burnay et aI., 1988, HoIst, 2000)
52
Table 3-2 Talley pressure monitor specifications
58
Table 5-1: Posture and cropped region
102
Table 5-2: Posture Code
103
Table 6-1: Selected features for neural network
111
Table 6-2 Neural network specifications
120
Table 7-1 Steering potentiometer absolute counts
156
Table 7-2 Throttle and brake potentiometer absolute counts
156
Table 7-3: Male volunteer drivers selected for experiment 2 (below 50)
165
Table 7-4: Female volunteer drivers selected for experiment 2 (below 50)
165
Table 7-5: Male volunteer drivers selected for experiment 2 (above 50)
166
Table 7-6: Female volunteer drivers selected for experiment 2 (above 50)
166
Table 8-1 Real life driving task comparison with neural network result
183
Table C-l Radial basis ANN Configuration
309
Xl
Intelligent Automotive Safety Systems: The Third Age Challenge
List of Equations
Equation 1: Plank's radiation law
42
Equation 2: Stefan Boltzman Law of emissivity radiation
42
Equation 3: Artificial neural network synaptic weights
63
Equation 4: Radial basis neural network
66
Equation 5: Lagrange polynomial interpolation
75
Equation 6: Maxima of the function p(x)
91
Equation 7: Calculating driver's distance from the steering wheel
199
Equation 8: Field of view area calculation
200
xii
Intelligent Automotive Safety Systems: The Third Age Challenge
Nomenclature
p = Energy Radiated
A= Wavelength
T= Temperature (Kelvin)
h= Plank's Constant
c = Velocity of light
b = Boltzman Constant
w = Radiated energy
E
= Emissivity
11 = Boltzman constant
T = Temperature (Kelvin)
tj = Desired or target response for ith unit
Yj = Actually produced response for ith unit
E = Error calculated for adjustment of synaptic weights
ii j = Centre of activation functions
qj = Parameter for optimization
p (v t) = Polynomial
a = Activation function
j = Number of neurons
q = Point at which interpolation takes place
P(q) = Interpolated value
fj = Known values on the grid at points (qj)
L j(q) = Lagrange polynomial
Dl = Distance from imager to point of interest
A = Distance between steering wheel axis and thermal imager
D = Distance between driver and steering wheel
Xlll
Intelligent Automotive Safety Systems: The Third Age Challenge
Acronyms
ABS
: Anti-lock brakes
ACC
: Adaptive cruise control
AD
: Alzheimer disease
AI
: Artificial intelligence
AIT!
: Artificial intelligence thermal imaging
ANN
: Artificial neural network
ASCII
: American Standard Code for Information Interchange
ATM
: Automated teller machines
BAE
: British aerospace engineering
BPN
: Back propagation neural network
BS
: British standard
CBA
: Cost benefit analysis
CCD
: Charged coupled device
Cl
: Computational intelligence
CMOS
: Complementary metal-oxide semiconductor
CPU
: Central processing unit
DAC
: Digital acquisition card
DUID
: Driving under the influence of drugs
EOTR
: Eyes of the road
ESC
: Electronic stability control
ESRI
: Ergonomics safety research institute
FFB
: Feed forward back propagation neural network
FIR
: Far Infrared
FLRS
: Forward looking radar sensor
FOV
: Field of view
FPS
: Frames per second
XIV
Intelligent Automotive Safety Systems: The Third Age Challenge
GHz
: Giga hertz
GPS
: Global positioning system
HDD
: Hard Disk Drive
HUD
: Heads up display
HR
: Intermediate infrared
ITS
: Intelligent transport system
IR
: Infrared
LED
: Light emitting diode
LCD
: Liquid crystal display
LWIR
: Long wavelength Infrared
MLP
: Multi layered perceptrons
MOMSSE : Mattis organic mental syndrome screening examination
MWIR
: Medium wavelength infrared
NHTSA
: National highway transport safety administration
NIR
: Near Infrared
NODS
: Near object detection sensors
OOP
: Out of position
RAM
: Random access memory
RBF
: Radial basis function
RBN
: Radial basis neural network
RGB
: Red green blue
RPM
: Revolutions per minute
SAB
: Side airbag
SOM
: Self organized map neural network
SDL
: Simple scenario definition language
SWIR
: Short wavelength Infrared
TRL
: Transport research laboratories
UFPA
: Un-cooled focal plane array
WDM
: Windows Driver Model
WHO
: World health organization
xv
Intelligent Automotive Safety Systems: The Third Age Challenge
1. Introduction
In 2002, 3431 people were killed and 35,976 seriously injured by vehicle accidents in
Great Britain alone (see Figure 1-1) (DFT, 2002). Even though Great Britain has the
lowest traffic related fatalities per capita and per kilometre of travel than any other
developed nation, total casualties are still 300,000 per year as shown in Figure 1-1,
which includes slight injuries. The D.K. government proposed a target of 40 percent
reduction in fatalities and serious injuries by 2010 (DETR, 2000). Improvements in
vehicle design, road safety regulations and road system design led to a 13 percent
decrease in fatalities from 1999 to 2002 (DFT, 2002).
1000000 ,..-................................................................................................................................................................................ _ .................................................................................... ,
.....
-
"
.............. - .......... .................. - .................................. - ................................................. .
337000
322000
314000
100000 + - - - - - - - - - - - - - - - - - - - - - - - '
51000
Cl
~c..
-.
45000
48500
10000 + - - - - - - - - - - - - - - - - - - - - - - - '
o
Cl
.a
E
.
~..
:::J
Z
1000
+-_--_-_-_-_-_-_-_-_-_-_--=.c-=-=-:....:-=-=--=-=-=-=-~-=-=-""'-"-'..."'-=C"'"""_~~ ~---~ ~ ~j~:~ hllIies i
1500
1220
1080
1__
Fatalities
i
Cl
100+---------------------~
S
10+---------------------~
1+---.--.--~-_.--._-_.--._-_._-~
1994
1995
1996
1997
1998
1999
2000
2001
2002
Years
Figure 1-1 Road accident casualties in UK from 1992 to 2002 (DFT, 2002)
Much research is being carried out by the automobile industry to make automobiles
safer with preference now being given to the use of intelligent sensors. Modem
electronics can, for example, help see through night, fog or even forecast an
approaching vehicle at a blind turn. Technologies like millimetre wave radars,
1
Intelligent Automotive Safety Systems: The Third Age Challenge
infrared sensors, satellite tracking, ultrasonics, laser scanners and magnetic sensors
are being rapidly adopted by automobile manufacturers (Dixit, 1998).
Passive safety is nowadays a standard feature in every automobile. When comparing
passive safety with active safety systems findings by the Society of Motor
Manufacturers and Traders reported that 58 percent of people were unaware of any
active safety system (Pull in, 2005). There is no agreed definition of active safety.
However, it can be best described as use of technologies that help to avoid accidents
and improve the safety situation of the car for the occupant and other road users.
Methods discussed in this thesis are mostly related to active safety in cars.
The automotive industry classifies safety into primary and secondary safety. Primary
safety is concerned with prevention of an accident before it happens. For example
intelligent sensors like ABS braking systems, lane departure systems and forward
looking radar systems are primary safety sensors. Secondary safety is concerned with
reduction in the effect and impact of the crash during an accident. The main concern
of a secondary safety system is to prevent damage to the vehicle's passenger cabin
and reduce risk of injury to the occupants. An example of secondary safety sensors
are seat belts, airbags, side airbags.
Drivers under the influence of alcohol, drugs tiredness, fatigue or stress present a
high risk of accidents. The automobile industry is pushing towards active safety
systems that can warn the driver before an accident happens. Thus technologies like
intelligent sensors that can detect out of position drivers, intelligent airbags and
driver position monitoring sensors have started to appear.
1. 1. Third age drivers
The demographic time change in the last century and reductions in mortality and
birth rate (Hitchcock et al., 2001, Dissanayake et al., 2002, WHO, 1998) have
resulted in an increase of the ageing population in the developed countries. Europe,
the United States, Japan and elsewhere have now proportionally a greater number of
2
Intelligent Automotive Safety Systems: The Third Age Challenge
older people than at any other stage in history. In the United Kingdom 18.11 percent
of the population is over retirement age and 35 percent of the labour force is over 45.
It is estimated that 20 percent of the United States population will be over 65 years
and over 70 years in the United Kingdom by 2030 (WHO, 1998). According to the
World Health Organization (WHO) the population of the world over the age of 60
will increase from 580 Million in 1998 to 1000 Million in 2020.
The term "Third Age" applies to those aged 55+, although this definition is not
universally accepted (Reuben, 1993, D. Carr, 1994). The third age term is not simply
used for persons over 55, but the old adage of 'You're as old as you feel' holds true.
Those over 55 year old will clearly have a greater impact on designing needs in the
future society. But as the grey, or 'third age', population increases the products in the
consumer market mostly are not adapting to 'third age' people's living style and their
habits. As for older drivers, it has been established that driving abilities deteriorate
with age, which is a matter of concern for developed nations. The term 'third age' is
used for people who are not only old but also have some kind of impairment, which
may be due to old age or some disability that can occur in old age. Old people,
sometimes even in their sixties or seventies, are more active than younger people.
These old people are not included in the 'third age' definition.
A survey by the National Research Council, U.S.A, (Benekohal et al., 1994)
indicated that 70 percent of older drivers used their cars at least 5 days a week, and a
higher proportion of male drivers than female drivers drove 7 days a week. As age
increases, urban road use increases and highway use decreases. A Transport
Research Laboratory (TRL) report (Simms, May 1992) indicated that drivers over 70
commonly use a car for purposes like shopping or going to the bank. Ninety percent
(90%) make these trips on weekly basis and eighty percent (80%) make a monthly
visit to relatives. One in four fatalities by car accident in the United Kingdom is of
third age people. Statistics also show that elderly drivers are the second most likely
group to have accidents after 17 to 24 year old drivers.
Many researchers suggest that the age factor does cause a decoration in driving
skills. Third age drivers become vulnerable with age (Reuben, 1993). For Older
3
Intelligent Automotive Safety Systems: The Third Age Challenge
drivers it is difficult to navigate while driving (May et al., 2005). Deterioration of
vision, tunnel vision, and restriction of movement in limbs are some of the symptoms
that affect them.
1.2. Motivation for research
The topic for this research is 'intelligent automotive safety systems: the third age
challenge'. 'Third age' people have age-related impairments which may affect their
driving skills. These driver problems include symptoms such as restriction of
movements in limbs and slower reaction time. Vision becomes worse significantly
which requires other body parts to compensate for the deterioration in vision. For
example tunnel vision requires more head movement. Driving is challenging for third
age people and this results in increased errors, lapses and violations while driving.
Due to these impairments some particular accident circumstances can cause fatality
or severe injuries to elderly drivers.
This project is primarily concerned with the investigation of an intelligent safety
system. Driver problems are discussed in detail in later chapters. The motivation for
this research came from the global focus towards inclusive design. As the grey
population increases in developed countries more focus is towards the third age
population. Growing markets and stricter government regulations are major
incentives for designers to design products for third age people.
The 'inclusive design' term is about ensuring that surroundings, products and
services are usable by all ages and abilities. For inclusive design, designers make
sure that the product and service is available to the widest possible audiences. This
includes third age people and disabled people which were previously ignored by
mainstream design practices. New global legislation (BS-7000-6, 2004) was
introduced in 2004. This legislation forces designers, manufacturers and service
providers not to discriminate against people on the basis of age or capabilities.
4
Intelligent Automotive Safety Systems: The Third Age Challenge
Infrared imagers were used previously for research and development in fields like
high technology defence and aerospace applications. As the price of thermal imagers
goes down, other industries are also starting to research into thermal imaging
technology. Low resolution thermal imaging has recently been introduced for
commercial and industrial applications by infrared imager manufacturers. Low
resolution thermal imaging technology is still in its infancy but holds significant
potential. Some of the applications for low resolution thermal imaging include
condition monitoring, automated people-counting, obstruction detection, intelligent
surveillance and intelligent vehicle vision.
1.3. Aim of research
The main aim of this research is to investigate means of identifying driver postures,
movements and behaviours which indicate a high level of risk, particularly for older
or impaired drivers. The identification method will use a low resolution IRISYS
infrared imager making the identification method non-contact, potentially cost
effective and ideally feasible for use in cars produced within the next five years.
Eventually the safety system will provide information to the centrally controlled
safety system.
For the third age and disabled drivers the safety system will be useful if it can
identify restriction of movement in their limbs or neck, or slower reaction times. The
safety system will also be useful for identifying young driver problems that might
include, for example, ignoring crucial driving tasks like looking both ways at an
intersection and dynamic allocation of attention. Additionally the safety system can
be used as a tool that can give information about the driver and their driving patterns
to a researcher. This information will help to identify driver problems, especially for
third age drivers and impaired drivers, when used offline in conjunction with other
sensors. The identification of driving behaviour could also lead to offline comparison
of different driving patterns of young and third age drivers.
5
Intelligent Automotive Safety Systems: The Third Age Challenge
This research will only be concerned with identification of driver postures. Infrared
imaging has been chosen because it is a non-contact method. It is also insensitive to
ambient noise and lighting variations. It seeks body heat which, in the wide range of
conditions encountered in cars, makes infrared imaging a more reliable approach
than other systems such as ultrasonic, visual cameras, radar and laser based sensors.
The infrared imager used for this research is a low cost device with low resolution
which provides much needed privacy to drivers in cars. This is not possible using
conventional visual cameras which may be perceived as being intrusive (see Section
5.1.1 ). A major part of the research will be the development of a safety system based
on a non-contact method using a low-resolution infrared imager, the IRISYS thermal
imager.
1.4. Scope
Currently there is no such low resolution, low cost thermal imaging based safety
system in automobiles that can give information about the driver and passengers. A
few visual based systems are available and are still under research. These visual
systems are mostly being used for intelligent airbag deployment.
By using Thermal Imaging the safety system can identify the driver's movement.
Finding restriction of movements (upper half) for 'third age' and 'impaired' drivers
will give us insight into their driving habits. This can be seen as a new safety sensor
for intelligent automobiles, which can find flaws in the driving habits and warn
before an accident happens. It can be classified mainly as a primary safety system.
However, in addition to this the detection of out of position (OOP) occupants will
make this system also classify as a secondary safety system.
6
Intelligent Automotive Safety Systems: The Third Age Challenge
1.5. Boundaries of the research
The thesis is focused on investigation of the safety system which detects driver
postures and movements. It explores the technical side of the sensor system and the
driving postures.
1. The focus is on the development of a safety system not the development of
the sensor. The safety system will only provide posture related information to
the central safety unit for appropriate action.
2. The safety system will be an offline reporting system.
3. The safety system will be based on driver posture.
1.6. Methodology
The research consists of several stages comprising literature review, technology and
safety system proposal. These stages are then classified into sub stages as shown in
Figure 1-2. The proposed safety system stage is the major part of the research. The
sub-stage: experimentation will involve testing and proving the algorithm developed
in the earlier stage.
7
Intelligent Automotive Safety Systems: The Third Age Challenge
Literature review
. ------------------------------.
....
Stage 2
Technology
,--_ .... -------_ .. ----------_ .. ------
Figure 1-2 Research stages w.r.t. time
A thermal image processing algorithm is developed in stage 2 after finalising the
position of the IR imager. A novel neural network is designed. Test runs are done in
a driving simulator and IR thermographs are acquired using the IRISYS imager. To
these IR thermographs the imaging algorithm is applied and then simulated using the
neural network.
After the 'development of safety system' sub-stage, a broad range of human subjects
is selected based on their age and gender. More experimental runs in the driving
simulator are done. The results are then compared offline and driving patterns are
discussed. The low resolution IRISYS thermal imager is used as the main tool in the
experiments. Infrared acquisition and visual acquisition software, developed by the
author, is used in the experiment sub-stage to acquire thermographs from the IRISYS
imager (Amin, 2003).
1. 7. Thesis outline
The thesis is organized as follows:
8
Intelligent Automotive Safety Systems: The Third Age Challenge
Chapter 1: Introduction
This chapter gives a brief background to the field and explores the motivation behind
the research. It establishes the aims and scope of the research. The methodology of
research is also explained briefly in this chapter.
Chapter 2: Literature Review
This chapter discusses what previous research has been done in the field related to
this research.
Chapter 3: Technical Background
This chapter gives detail about the technologies, sensors and tools that will be used
and related to the research carried out in this thesis
Chapter 4: Hypothesis
This brief chapter gives the research questions which will be addressed in this thesis.
Chapter 5: Imaging Techniques and Image Processing Algorithms
This chapter is concerned with the actual imaging algorithm development and the
techniques that are used during this process.
Chapter 6: Artificial Neural Network
This chapter goes through the design, training and implementing of the neural
networks. Optimization of the neural network is discussed based on the design and
modified accordingly.
Chapter 7: Experimental Setup
The experimental runs are done to evaluate the imaging algorithm. This chapter
explains in detail the kind of experiments that are conducted.
9
Intelligent Automotive Safety Systems: The Third Age Challenge
Chapter 8: Results and Discussion
This chapter gives the results obtained from the neural networks and its discussion.
The chapter also discusses the system capabilities. It will address the research
questions stated in chapter 4 in a systematic manner.
Chapter 9: Conclusions and Further Work
This is the concluding chapter of this thesis and also suggests directions for further
work.
1. 8. Summary
•
Vehicle safety is a major issue of concern even for developed countries.
Governments and automobile manufacturers are taking steps to make roads safer
for drivers and pedestrians. Extensive research is being done in this field to
develop intelligent sensors that will aid the drivers or make their driving safer.
•
There are a group of drivers that are identified as a safety concern on the
road, especially for developed countries. This group is termed the "Third Age"
drivers. This term applied mostly to old drivers who are over the age of 55 with
some exceptions. Their driving abilities deteriorate with age. This age group will
represent around 20% of the population in the near future.
•
This research proposes a non-contact non-intrusive alternative for identifying
driver movements. This system should be an inclusively designed safety system.
•
There is an extensive use of thermal imaging in defence and other industrial
applications. Currently no low resolution intelligent thermal imaging solutions
exist for vehicle safety systems.
10
Intelligent Automotive Safety Systems: The Third Age Challenge
•
The research will be based on technology, proposed safety system design,
experimentation and evaluation of the safety system.
11
Intelligent Automotive Safety Systems: The Third Age Challenge
2. Literature Review
This chapter uses previous literature and research for common driving problems
faced by the drivers. It also discusses previously researched sensors and safety
systems. Most of the problems discussed are due to the physical limitations of the
drivers. Further on this chapter focuses in detail on the Third age driving problems.
Recent developments in vehicle safety systems are then discussed.
2.1. General driver problems
There are many driving problems that are frequently encountered by drivers. Not all
of them can be discussed in this literature review as the list will grow significantly.
Only major driver problems and issues are discussed and related research is
reviewed. These driver problems are interrelated and some topics do overlap but are
discussed from a different perspective as required.
2.1.1.
Young age drivers
According to a research survey (Williams et al., 1997), young beginner drivers are
three times more at risk than middle-aged drivers. In 40 % of the fatal accidents
occurring at night time, 16 to 17 year old drivers are involved. Young drivers are at
elevated risk of an accident when accompanied by multiple passengers. The risk
increases four to five times compared with driving alone.
Recent research (Lucidi et al., 2006, Monarrez-Espino et al., 2006) provides
statistics of accidents by age groups and their causes. This shows that drivers under
12
Intelligent Automotive Safety Systems: The Third Age Challenge
the age of 30 years are at high risk of accidents during the early morning, whereas
drivers from 17 to 24 are 10 times at higher risk at late night driving than at noon.
This higher risk of accidents in this age range is due to less experience and
knowledge of how to cope with fatigue. Young drivers usually overestimate their
driving abilities. The research was questionnaire based in which participants from 18
to 22 years old with driving experience of not more than 2 years were selected.
About 15 % of participants said that they had not driven a car between midnight and
0500 hrs in the last 6 months. Of the ones driving between these hours 46.6%
experience impairment by sleepiness at least once a month. 41.3% of young male
drivers experienced severe sleepiness compared to 27.1 % of female drivers.
2.1.2.
Driver distraction
Driving alone, when not involved in distracting activities, time sharing tasks are
performed concurrently by the driver. These crucial tasks involve staying on the
road, changing lanes, checking mirrors, reacting to changes and maintaining forward
motion. Other secondary tasks are slightly less important like checking speed or road
signs. When the driver is distracted both crucial and secondary tasks suffer.
Mobile phone users are four times more likely to have an accident than an average
driver. Researchers found larger steering movements while doing crucial tasks,
delayed braking patterns and slow reaction to critical signals (DFT, 2002).
An experiment was conducted with the help of 36 young drivers using a mobile
phone while driving on a STISIM driving simulator (Beede et al., 2006). During this
experiment drivers received phone calls using a headphone and speaking pieces.
Participants driving performance is divided into four categories: violations, attention
lapses, driving maintenance and reaction time. Results showed that 67% of the
participants had at least one accident. 61 % of the participants were speeding while
using a mobile phone. Participating drivers took one-third of a second longer to set
off after the car came to a stop sign when engaged in telephone conversation. In a
13
Intelligent Automotive Safety Systems: The Third Age Challenge
separate driving questionnaire 80% of drivers said they engaged in hand held mobile
phone conversation at least once a week with an average of 8.4 minute conversation
everyday. Participants reported an average driving distance of 15 miles per day.
Participants managed to narrow their attention to more crucial driving tasks with not
much concentration given to secondary driving tasks.
2.1.3.
Drink and Drugs risk
A research paper (Pack et al., 1995) regarding illegal drugs intake and its effects
whilst driving confirms that illegal drugs like cannabioids, cocaine, lysergic acid
diethylamide, amphetamine and ecstasy cause symptoms that can cause fatal
accidents. Young teenage drivers are more involved in drugs and drink driving.
Drinking and taking illegal drugs can leave the user with distorted perception,
confusion, blurred vision, anxiety, nausea and over confidence. Also after several
hours the users will experience severe fatigue and tiredness.
Research (Augsburger et al., 2005) conducted in Switzerland shows that illegal drugs
are also known to cause impaired driving skills by affecting attention abilities, visual
acuity, judgement, reaction time, drowsiness etc. Thus multiple medication intakes
also increase the risk of road accidents. Police took blood and urine samples from the
patients who fitted the criteria of being alive at least 24 hours after the accident and
having the documentary proof of DUID (driving under the influence of drugs). The
experiment was conducted with 440 subjects who met the selection criteria. The
results showed that 91 % of drivers who crashed under the influence of drugs are
males. Mean age of the drivers was 28. The most common drug used was
cannabinoids at 59% and ethanol (alcohol intoxicant) was at 49%. Other drugs were
opiates (9%) and cocaine (13%). The authors concluded that suspicion of impaired
drivers is highly correlated to drug analysis in blood and that young male drivers are
at higher risk than female and older drivers.
14
Intelligent Automotive Safety Systems: The Third Age Challenge
2.1.4.
Sleepiness and fatigue
An asleep driver is defined as the driver who fell asleep while driving prior to a crash
(Stutts et al., 2003). A fatigued driver does not necessarily have to be asleep and is
classed as drowsy, or physically tired. Fatigue is defined as temporary loss of
strength due to mental or physical work or tiredness caused by stress. Fatigue will
lead to sleepiness which is a very sleepy state. A mail-based survey was conducted
by the same author in the US, with more than 1400 drivers from North Carolina
involved in the study. In the survey more than 23% of drivers that had accidents in
the past related to fatigue said that drowsiness was not important at all. Drivers
involved in sleep related crashes are more likely to have problems sleeping or have
trouble falling sleep. Long driving trips is also another factor which increases the
likelihood of having a sleep related crash. Nearly 8% of the sleep crashes involved
alcohol intoxicated drowsiness.
It is a well known fact that sleepiness causes driving accidents (Gold et al., 1992,
Stutts et al., 2003, Pack et al., 1995, Bunn et al., 2005). The symptoms of sleepiness
include eye problems, yawning, difficulties staying alert, and task focus (van den
Berg et aI., 2006). The crashes occurring due to drowsiness mostly involve young
persons, night shift or rotating shift workers, persons with undiagnosed or untreated
sleep disorders and drivers under the influence of soporific medications or sedating
medicines. Medical conditions are another factor that causes sleepiness or drowsiness
while driving. Drivers suffering from sleep apnoea are prone to fall asleep up to
700% percent more than regular drivers. This becomes very dangerous when driving
on a motorway. In the U.S 5.1 % of accidents are related to fatigue, drowsiness and
lack of concentration, especially in large vehicles. This is because truck drivers are
always considered at more risk due to their long driving hours. When drivers lack
sleep they are easily distracted and are less alert.
Research published by the BBC (BBC, 2005) describes how they were able to spot
sleepy drivers early without going on the road. The results suggest that drivers that
usually drowse while driving will do the same when driving in a driving simulator.
15
Intelligent Automotive Safety Systems: The Third Age Challenge
As patients with sleep apnoea have very high risk of having an accident, the research
team suggested that driving simulators can be used as a benchmark parameter of
driving performance for sleep apnoea patients. One of the symptoms of sleep apnoea
syndrome is a disorder that causes daytime sleep.
From the above it can be established that sleep or drowsiness can be a critical issue
while driving. This has caused many accidents alone. Taking it further it can be
found that sleep disorders are common in middle aged males. In a survey conducted
by (Krahn et al., 2006) it was found that 49% of middle aged men with heart
conditions suffer from sleep apnoea. Also relating to heart diseases, Javaheri
(Javaheri, 2006) conducted research on sleep related breathing disorders. Sleep
related breathing disorders are known to occur in a patient with heart failure.
Significantly obese drivers also have breathing disorders and develop snoring in
many cases. Snoring can be related to heart condition and sleeping disorders.
2.1.5.
Disabled drivers
In some countries it is required by law that a disabled person should be engaged in
activities in the same level as that of other people (Falkmer et al., 2000). Driving is
one of the most important factors that can increase quality of life by spontaneous
mobility.
TRL conducted a study on disabled drivers' controls and car conversions
(Ergonomics, 1986). Even though the study is quite outdated, it still forms the basis
of car controls used nowadays. The findings from the research report are discussed.
Several types of controls are developed for disabled driver including drive by wire
systems, foot steering wheel systems, horizontal steering systems, knee operated
steering, ultra light steering wheels and shoulder operated brakes. Voice and infrared
sensors are being used for non critical functions like activating GPS and radio. In the
UK rod mounted brakes are most common. Foot steering systems are getting more
and more common and commercially available. Most difficulty in driving comes
16
Intelligent Automotive Safety Systems: The Third Age Challenge
from people who are suffering from neurological problem and severe weakness.
Making controls for this group of disabled drivers is more expensive. There is no
regulation or standard for mounting and installing of car controls, it varies from
individual to individual. Car simulators are also being used to find out drivers' ability
to drive using these special controls.
Stability and psychomotor skills are of utmost importance for disabled drivers
(Geiger et al., 2004). Stress tolerance and reaction time is also one of the major
factors that affects third age drivers and most prominently disabled drivers. Driving
on the road for disabled and elderly people can be a challenging task, physically and
mentally.
2.1.6.
Driver's Vision
Driving is to a very high extent a vision task. Vision impairment, vision obstruction,
and blind spots, identifying obstructions while reversing should be investigated.
Vision occlusion while driving is considered to be a major factor in accidents and
collisions; instrumentations and techniques are created to mimic vision occlusion and
experiments are conducted by ergonomic practitioners to measure the effect of vision
occlusion while driving (Noy et al., 2004, van der Horst, 2004, Baumann et al.,
2004).
2.1.7.
Third age drivers and age related disabilities
An eighty year old woman is four times more likely to die in car crash than a twenty
year old man. Third age drivers have the highest fatality rate of any driver age group.
The third age driver will have more lapses (though not violations) than any other
driver. These lapses will result in not carrying out tasks which are essential in driving
including looking left and right. It is seen from annual statistics that most third age
17
Intelligent Automotive Safety Systems: The Third Age Challenge
that most third age driver accidents occur at junctions. This is due to lack of neck
movement (Hu et aI., 1998).
A driver can take his eyes off the road for a maximum of 1.5 seconds. More than 1.5
seconds is dangerous, especially when driving on a motorway (Parker et al., 2000).
As the reaction time gets slower for older people the time needed to look away
increases. Third age drivers are known to have taken double the time it takes middle
aged drivers to complete a certain task while driving.
A survey was conducted in Manchester,
u.K.
in 1989 of drivers aged 50 or over
(Parker et al., 2000). The survey mentioned three types of driving problem:
1. Errors
2. Lapses
3. Violations
Errors are driving mistakes and can have serious consequences. Lapses are primarily
unintentional failures, which cause embarrassment, but there is no direct impact on
safety. Violations are risky driving behaviours, which the driver engages in
deliberately. Some of the most frequent driving errors, lapses and violations by the
50+ aging community are as follows:
Errors while driving:
•
Unable to estimate the speed while overtaking a vehicle.
•
Braking quickly on a wet/slippery road.
•
While changing lanes, forget to check rear view mirror
18
Intelligent Automotive Safety Systems: The Third Age Challenge
Lapses while driving:
•
Misreading signs and taking a wrong turn from roundabout.
•
Wrong lanes taken while approaching a roundabout.
•
Forgetting where the car is parked.
•
While driving towards destination A, you notice that you are off to
destination B, which is a more usual route.
Violations while driving
•
Disregarding speed limits during non-rush hours.
•
Becoming impatient with a slow driver, in front and undertaking the vehicle.
A passive accident is one in which the driver's vehicle is hit by another vehicle and
vice versa in the case of an active accident. If the active-passive ratio is greater than
one (1) it means that the driver is involved in more active accidents than passive ones
and if the active-passive ratio is below one (1) the driver of that vehicle is less likely
to be at fault as he has been involved in more passive accidents than active accidents
(Parker et aI., 2000).
Looking at age trends over the whole sample in factor scores, they showed that
violations decrease with age. At the age of 50 violations levelled off with the lapses.
As the age increases from 50 to onwards the number of accidents decreases as they
go out less frequently but the number of 'active/passive accidents ratio' increases
considerably from 0.95 for 59years and less to 1.44 for 75years and older (Parker et
al., 2000).
19
Intelligent Automotive Safety Systems: The Third Age Challenge
The potential factors that contribute towards having a vehicle crash by older people
are (Hu et al., 1998):
•
Demographic attributes
•
Limitations or restriction in carrying out physical activities
•
Chronic conditions
•
Physical features
•
Psychosocial features
•
Symptoms
•
Drug usage
•
Health related factors
Vision problems in third age drivers
The size of the useful visual field of work is not always the same. It changes with
situation, time and tasks being conducted (Sanders, 1970). The ability to detect
peripheral signals in more than one task worsens with age. Thus older driver's
performance is much poorer than their younger counterparts.
Research shows that the 40% reduction in visual field by third age people
significantly increases the risk of accidents while driving. Further research reveals
that vision field deterioration is directly related to tunnel vision phenomenon (Roge
et a!., 2003, Seiple et al., 1996).
20
Intelligent Automotive Safety Systems: The Third Age Challenge
A study was done by Roge at al. (Roge et aI., 2002) on visual impairment and states
of vigilance while driving. Car driving is a complex task which involves vision
modality to a high degree. A degree of visual impairment does not necessarily mean
bad driving as the subject will compensate for an artificially generated deficiency. In
the experiments conducted the artificial visual impairment was created by wearing
goggles. The field of view was worsened while performing a similar task like driving
for a prolonged period. The vision signals appear to be in 5 to 20 degrees, and tunnel
vision phenomenon also occurs. In the experiments conducted by Roge the field of
view for the driving simulator was restricted to 50 degrees horizontally and 25
degrees vertically. Also the driving was carried out in a fog scene. In which the
subjects had to follow a car with an average speed of Il00au/hr. During the
experiment the subjects were presented with a peripheral signal, a momentary visual
point in the simulation, at different eccentricities (50, 100, 150 and 200). It's was
also found during the study that as the tunnel vision angle decreases the occupant
becomes drowsy and two subjects fell asleep. Eighty peripheral signals were
presented at different eccentricities every half hour and the subject had to respond by
flashing the head lights to full beam as quickly as possible. Each experimental run
consisted of a 2 hour run without any interruption to the driver. It was noticed that as
the eccentricity of peripheral signals increased the performance deteriorated. For
example male subjects have an accuracy percentage of 78.4% at 5 degrees of
eccentricity which deteriorated to the accuracy percentage of 10.1 % at 20 degrees of
eccentricity. The central task of following the car at an average speed is also
monitored by a central signal. The performance for this central signal deteriorated as
that of peripheral signals. For example only a 10% decrease was shown from the first
half hour run to the fourth half hour run. It was noticed that fewer corrections to the
steering wheel and fewer micro movements are detected with time. Tunnel vision
implies that the probability of perceiving signals by a drowsy driver is not constant
across the whole visual field.
Visual, physical and cognitive functions decline with age. This in result affects daily
tasks, including the ability to drive an automobile safely. A study conducted by
McGwinJr et al. (McGwinJr et al., 2000)
i~volves
visual risk factors in older drivers.
In the study two groups of older subjects were assembled, the first group including
21
Intelligent Automotive Safety Systems: The Third Age Challenge
older drivers with cataracts and the other group without cataracts. Cataract is a
medical condition which leads to visual impairment in the older third age drivers.
This visual impairment includes visual acuity, contrast sensitivity and visual field
sensitivity, thus causing increase difficulty with visual activities of daily living. This
medical condition is curable by surgical means. All subjects selected in this study
were from 55 year old to 85 year old independently living licensed drivers. The first
stage of the study involved collection of demographic information (like age and
gender), driving habits, visual function and cognitive status. Eight driving scenarios
for example driving in rain, rush hour driving, driving alone and making left turns
were selected. The subjects were asked to rate the difficulty from one (1) to five (5),
with five (5) being not difficult and one (1) being extremely difficult. In the second
stage visual functional status of all participants was measured with respect to visual
acuity, contrast sensitivity, disability glare and functional field of view. The visual
measurements were taken from speciality charts like 10gMAR for distance acuity and
the Pelli-Robson contrast sensitivity chart. Useful field of view is measured by
Visual Attention Analyzer, Model 2000 (Visual Resources, Inc., Chicago, IL, USA).
Later cognitive functions were tested in a twenty (20) minutes test. This was done by
the Mattis Organic Mental Syndrome Screening Examination (MOMSSE), which is
designed to assess cognitive function in the elderly. The author validates his
hypothesis that the older drivers with visual impairments have difficulty in specific
driving situations. The author concluded that visual acuity and contrast sensitivity are
the main concern of safety for older drivers with visual impairments.
Driving habits of third age people
Travel by third age people is limited compared with younger people (Cutler et al.,
1992, Siren et al., 2004). Very few studies are available for elderly driving
behaviour, travelling habits and the factors affecting them. A US study shows that
personal transport is not commonly used by the third age people, particularly female
gender and urban residents. Most elderly people prefer using public transport as their
eyesight and mobility deteriorates and they are unlikely to go out at night in their
22
Intelligent Automotive Safety Systems: The Third Age Challenge
cars unless necessary. According to the above study based on the census data of the
US, 1.3% rural farm men who are 65 to 74 do not have access to vehicle, while
57.8% of urban living females over the age of 85 do not have access to personal
transport and like travelling using public transport. There is less desire for frequent
travel and longer distances, and therefore generally less need for mobility.
Attentional ability
To keep the car on the road is crucial for safe driving. An experiment with young and
elderly drivers was conducted by de Waard et al. (de Waard et al., 2004). Visual
information on the driving simulator road was created as follows:
•
No delineation
•
Centre-line; 3 metre long only
•
Centre-line and roadside markers
•
Full delineation; includes centre-line, continuous road edge and road side
markers
•
Full delineation with lampposts
A ghost road was forked from the scenario with only lampposts but no delineation.
8! % of younger drivers followed the centreline from where the ghost road is forked,
while only 50% of the elderly within the ages of 55 to 70 took the centre line. The
rest of them followed the ghost road with only lampposts. Thus older drivers tend to
give much more concentration to road elements to predict the position on the road as
they are easier to see than the centreline. Elderly drivers also have difficulty in
judging and deciding the flow of traffic or in tasks that demand attentional skills (de
23
Intelligent Automotive Safety Systems: The Third Age Challenge
Waard et al., 2004, McGwin Jr. et al., 1998). Elderly drivers were more confused
than other drivers, and wrong turn probability was over 50%.
In a study conducted by Louis et al. (Louis et ai., 2000) two age groups of sixteen
subject drivers were selected for the experiment. The first group was of younger
drivers under the age of 35 and the second group was of older drivers over the age of
55 years. Each group contained an equal number of male and female drivers. A
specially designed test track was used for the experiment which was 7.5 miles long.
The test car was equipped with speed measuring, lane departure detection, roadscene camera and driver eye glance behaviour at 30 Hertz sampling rate. The average
trial time to complete the experiment run for older driver was twice that of younger
drivers. The tasks included using four different types of guidance systems, tuning the
radio and dialling a 10 digit manual mobile number. Total seconds of eyes off the
road (EOTR) for older driver is twice that of younger drivers. Young drivers took an
average of 40 seconds EOTR during the whole test, older drivers took 83 seconds,
for tasks including looking down at the radio, mobile phone and guidance system.
Advanced navigation and information systems can be dangerous for older drivers.
Henderson et al. (Henderson et al., 1999) regard advanced information systems as a
two edged sword for third age drivers. This is because of diminishing perceptual and
cognitive abilities. Usually a normal driver makes small head movements toward the
right and left to get a view of 30 to 35 degrees, secondly the driver adjusts his or her
eye for close vision and reacts to the situation accordingly while driving. Older
drivers take longer to process the information. The reaction time for older drivers is
also slower than younger drivers. Larger head movements to left and right are
required to compensate for visual impairment phenomena like tunnel vision.
Driver attention sharing between road and in-vehicle displays was compared between
young and older drivers by Mourant et al. (Mourant et al., 2000). The experiment
was conducted on a driving simulator. The first age group of ten driving volunteers
ranged from the age of 23 to 46, the second age group was from 58 to 76 also ten
driving volunteers. A total of 26 trials were done for each volunteer, the first trial
being a practise run to familiarize the volunteer with the scene. The response data
was collected by superimposing a random four digit number onto a road scene and
24
Intelligent Automotive Safety Systems: The Third Age Challenge
verbal response from the volunteer is recorded. The superimposing was done on an
in-vehicle display and driving simulator screen, thus causing driver distraction.
When the time between stimuli was 1.6 seconds younger drivers had an accuracy of
99% and older drivers had an accuracy of 89%. As the time is decreased to 1 second
between stimuli the younger drivers had an accuracy of 73% and the older drivers
accuracy reduced to 59%. During the trial runs the older drivers spend 20% more
time outside their driving lane than younger drivers, in a driving simulator. The
authors suggest that for older drivers, the most difficult number to read is
superimposed from the far view rather than the closer in vehicle display. The older
drivers also have difficulty in switching between near and far vision.
2.1.8.
Safety for third age people: Memory & Motor
skills
The skills of third age people deteriorate with age in tasks ranging from driving
(Barrett et al., 2000, Holland et al., 1994) to using Automated teller machines
(ATM) (Adams et al., 1991, A. Rogers et aI., 1997, Rogers et al., 1996) which are
considered easier to use by the younger generation and require no training. This will
make the third age generation reluctant to adapt to new products due to products
involving unfamiliar concepts or procedures, for example use of computerised and
interactive technology (Marquie et al., 1994, Rogers et al., 1996, Park et al., 1999).
A study was conducted by Lundberg et al. (Lundberg et al., 1998) on cognitive
functions of older drivers, as spatial orientation and speed perception are known to
decline with normal ageing. Age related cognitive diseases also create high risk of
automobile crashes, particularly diseases like Alzheimer's Disease (AD). The authors
mentions about impaired and third age driver problems which include driving too
slow, getting lost and taking wrong turns on roundabouts. The main cause for these
problems is cognitive impairments. Third age people with diagnosed cognitive
impairments are at high risk while driving. Sixty nine driver-volunteers, over the age
of 65 took part in experiments. The experiments were conducted to find the extent of
older driver crash involvement due to cognitive impairments. Thirty six drivers had
25
Intelligent Automotive Safety Systems: The Third Age Challenge
suspended licenses with twenty-six involved in car crashes. These drivers were
compared to thirty one drivers with clean licenses. In addition to neurophysiological
examination, a thorough medical examination was conducted which included
visuospatial ability, memory, reaction time, psychomotor speed and diverted
attention. In the suspended license group four distinguished categories were found:
1. Twenty-four (24) drivers violated intersection rules like not stopping at red
traffic lights, not giving right of way, not stopping at Stop signs.
2. Four (4) drivers complained ofloss of vehicle control.
3. Two (2) drivers; the cause for license suspension was speeding.
4. There were also priority violations leading to head-on collision, rear end
collision, running down pedestrians and, crashing into a railway barrier.
It has been noted that car accidents involving the elderly includes cognitive
decrements in memory and visual perceptual skills (Lundberg et aI., 1998, McGwin
Jr. et al., 1998).
Also there are decrements in speech processing skills like loss of hearing, speech
recognition, cognitive inhibition and working memory (Tun et aI., 1997, Sharit et
al.).
Ergonomics practitioners and design engineers have created tools over the years to
over come this deterioration of skills in third age. The hearing aid is one of the most
commonly used aids by the end user to over come speech processing problems
(Smeeth et al., 2002).
26
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 2-1: Third age suit developed by Loughborough University
A third age suit in Figure 2-1 created by Hitchcock et al. (Hitchcock et al., 2001,
BBC, 2004) gave ergonomics practitioners a new tool which helped them to identify
what aspects maybe used in designing a product for third age people. Previously
information systems like USERfit were also available (D. Poulson, 1998, D. Poulson,
1996), but none provided design engineers with the information required to design
the consumer product. The third age suit allowed the designer to try it on before
designing the product. It allowed the designer to feel like a 'third age' person. The
third age suit was developed by means of a thorough review of the physiological
aspects of the ageing process. Movement restrictors are applied to the wrist, elbow,
back of the upper and lower torso, knees and ankles. The suit allowed design
engineers to simulate the experiences of third age people while designing or
optimizing the application.
27
Intelligent Automotive Safety Systems: The Third Age Challenge
2.2. Safety in cars with intelligent sensors
This section explores the recent advances in the field of car safety sensors. The key
focus of the literature review in this section is on active and occupant safety
components in automobiles. The literature discussed in this section is sub divided
into primary and secondary safety.
2.2.1.
Primary safety sensors
Pre tension seat belts
The car manufacturer Volvo made the first seat belt in 1949. Even common seatbelts
nowadays lock up when a sudden force is applied. The pre-tension seat belt activates
a tensioner which pulls tension onto the seat belt prior to maximum impact force
occurring.
The most popular type of pre-tension mechanism for a seat belt is pyrotechnics
based. On collision the ignition creates a pressure that moves the belt webbing back
inside housing. This is achieved usually by a gear rack arrangement as shown in
Figure 2-2 but varies from designer to designer.
28
Intelligent Automotive Safety Systems: The Third Age Challenge
Rack
pinion
arrangement
for retracting
seat belt
Gas canister
Ignition
At the lime of an accidentthe ignition tires pushing the rack up to
rotate the pinioll which retrncu the ~ent belr. (howstllffwork!;.cQln)
Figure 2-2: Seat belt pre-tension mechanism
Radar and ultrasonic sensors
Over the last decade 'Short Range Radars' are increasingly being used in the
automotive industry to create a safe 'intelligent highway control'. 'Short Range
Radars' are mounted on the front and back of cars to detect any obstruction while
driving. These radars are also known as 'Millimetre radar sensors' or 'Short distance
radars' shown in Figure 2-3.
The range of short distance radar sensors is around 20 metres. These radars have
frequency 24GHz with relatively short wavelengths of around 4mm. 'Short Range
Radars' can achieve an accuracy of 10cm to 25cm in determining the position of an
object. Short distance radars are used in parking assistance, pre-crash detection, stop
and go driving, back-up warning, blind spot detection, side impacts and so on These
radars can work easily in deteriorating weather conditions, like rain, fog, snow and
night (Nebot et aI., 1999).
29
Intelligent Automotive Safety Systems: The Third Age Challenge
Short Distance Radar
Ultrasonic distance sensors
Figure 2-3: Distance sensors (Macom, 2006)
Researchers came up with ideas such as autonomous or aided lane changing, sideimpact warning, reverse warning, visual aid and collision avoidance by using radar
technology (Mar et aI., 2003). This radar technology has been developed in
automobiles for not only looking behind a vehicle but also it is being used for 360
degrees envelope coverage of the automobile. Near object detection sensors (NODS)
are also used for short detection of vehicles and people in the vicinity of the vehicle.
Long-range sensors, or Forward Looking Radar Sensors (FLRS), are used for
adaptive cruise control. The Forward Looking Radar Sensor (FLRS) is the most
sophisticated sensor system that can detect more than 150 metres at 77 GHz and
allows adaptive cruise control according to traffic flow, warning systems like precrash detection and some of the vehicle controls, as well by use of signal processing
(Macom, 2006).
Rudin-Brown et al. (Rudin-Brown et al., 2004) demonstrated an application of
FLRS. It is an extension of conventional cruise control. Allowing a vehicle to follow
another vehicle at speed and maintain a constant distance by controlling engine and
brake. A vehicle with ACC (adaptive cruise control system) (see Figure 2-4) will
increase or decrease speed according to the vehicle in front, and measures a distance
with special FLRS radars.
30
Intelligent Automotive Safety Systems: The Third Age Challenge
Lane changing
Blind spot detection
-wall
l.5-m
Reverse Impact Warning
FLRS
lOOm
Figure 2-4: Application of High-resolution radar systems in automobile
Ultra sonic sensors, depicted in Figure 2-3, are also used in the automotive industry
for obstacle detection. Mostly these sensors are used for reverse parking warning
systems. Based on a 40 kHz sound pressure wave, the sensor covers a range of 1 to 3
metres detecting objects. As the measuring angle is far greater than the radar systems
there is too much noise in these sensors from the backgrounds like road and more
angular objects to allow their use for driving situations other than parking.
All these sensors measure distance, but radars can only be used within a limited
angle. They also cannot be used to distinguish between different objects. Here
intelligent video algorithms and machine system may be used. Stereovision is also
used to measure distance between different objects. These videos can tell flow of
movement, distance and relative velocity with computer vision.
Visual cameras are used to find the distance of a driver's head from the steering
wheel using stereo vision for intelligent airbag systems. Reading eye pupil gaze
movements using visual cameras with a Near Infrared filter is a common technique
in which the driver's gaze is tracked to find fatigue and other vision related driver
problems (Boyraz, P., 2006).
31
Intelligent Automotive Safety Systems: The Third Age Challenge
Other intelligent car sensors that involve machine vision include lane changing
sensors used in intelligent cruise control. Autonomous vehicle steering and vision
guidance systems is an active research field. Collision warning systems and obstacle
detection is also done using machine vision techniques (Mar et al., 2003 and Bertozzi
et al., 2000). The current research emphasis is on stereo based vision systems to
bring an extra dimension (depth) into the previously developed vision systems.
Vehicle tracking, detection and classification is now commonly done using machine
vision systems installed on busy highways.
Laser scanners
Laser sensors can measure distance and angle of the object relative to the car.
However they have to be installed on the outside of the vehicle, which is a
disadvantage. The range of these types of sensors is around 50metres.
Laser machine vision is a vast field, In automotive applications the laser sensor is
used particularly for distance measurements, a technique called laser range
measurement or laser range scanning. This is also a substitute for radar sensors but
has been outdated by the use of short wave radar sensors due to the need for
expensive laser signal measurement equipment.
Many automotive steering control systems are based on GPS sensors but an
alternative method for autonomous guidance, when the satellite navigation signals
are blocked, is that of machine vision and laser based radar systems.
Applications that involve laser radar systems include autonomous vehicle navigation,
lateral guidance systems, obstacle detection, autonomous service vehicles, walking
robot foot placement, manufacturing and quality inspection, military and agriculture.
32
Intelligent Automotive Safety Systems: The Third Age Challenge
Night vision (Infrared or Near Infrared vision)
Only one quarter of total driving is done during night time, but half of the traffic
fatalities occur at night. Lack of visual information is the main contributor to these
accidents. In the last five years night vision systems have been developed using a
high spatial resolution infrared thermal imager of resolution 320 x 240 pixels at
30FPS. It consists of an Uncooled Focal Plane Array (UFPA). In a research
conducted by Martinelli et al. (Martinelli et al., 1999) the LCD displays a real-time
infrared monochrome image. An aspheric mirror reflects the image from the LCD
display onto the windscreen where it is viewable by the driver. A human can be
detected at 500 metres and recognized at 135 metres. The video camera records the
reflected image which appears black and white on the screen by a head up display
(HUD) (DaimlerChrysler, 2000).
These sensors, and other sensors can be fused for tracking algorithms and to improve
the reliability of the system. Scenario assessment is required to include most obvious
road hazards and predictions. These night vision systems are expensive and an extra
£10K has to be paid for installation in the car.
Similar systems have been developed using near infrared cameras and infrared LEDS
by car manufacturers (DaimlerChrysler, 2000). These use infrared light to illuminate
the road, which is invisible to the human eye. These infrared illuminated areas are
then seen by cameras with near infrared capabilities. The mounting position of
cameras is usually over the rear-view mirror.
Omron developed a high resolution CMOS camera that can see in low light and
extreme lighting conditions such as tunnels, blinding sunlight and after dark (Oct
2003). This camera is a significant improvement over conventional CCD cameras
and can detect images which are radiated with near infrared light.
33
Intelligent Automotive Safety Systems: The Third Age Challenge
Occupant position sensor and driver measurement
A 3D point measurement system has been developed for automobile drivers
(Stockman et aI., 1997). The aim of this research is to measure posture for
automobile seat design. Special markers are placed on the driver's body and these
markers are focused by two cameras mounted inside a car. These cameras were
calibrated and fixed at known distances to each other. By creating a stereoscopic
vision the author was able to measure points on the human body in 3D space.
It is not necessary to find the exact measurements of the driver posture. Especially
off-the-road driver measurements can mostly benefit designers and ergonomists in
designing something safer. But on-the-road real-time occupant position sensors can
give significant information that can be useful.
Some of the automotive manufacturers (Bruns, 2000, Breed et al., Ghiardi, 1999)
have started to manufacture occupant detection systems. These systems uses load
cells which are installed into the car seat to find the size and the height of the
occupant. Manufacturing costs of these types of active safety systems will increase
the cost of cars significantly.
Anti-lock braking system (ABS)
ABS was designed to give conditional control during an emergency stop or poor road
conditions. ABS is now a standard option in cars. A typical ABS sensor includes a
wheel speed sensor, electronic control unit and hydraulics unit. This becomes a
closed loop circuit. The electronic unit checks and compares each wheel speed
sensor. If the electronics control unit senses a lock up of any wheel it reduces the
amount of pressure to the hydraulic unit at that wheel.
34
Intelligent Automotive Safety Systems: The Third Age Challenge
Electronic stability control (ESC)
Electronic stability control (ESC) is a term for a primary safety system in
automobiles which is designed to improve vehicle handling. First developed by
German automobile manufacturers, the ESC system compares the driver inputs from
steering and braking. The ESC reduces excess power by the engine to stop understeer
and oversteer by finding the lateral acceleration, individual wheel speeds. ESC also
integrates traction control and ABS. ESC has different terms defined by each
automobile manufacturer like electronic control unit, Vehicle Dynamic Control and
Electronic Stability Program.
2.2.2.
Secondary safety sensors
Airbags
First airbags started to roll out in the North American automobile market from 1982
by Mercedes Benz, a German automobile manufacturer. It consisted of a crash
sensor, gas generator, airbag and knee bolster (Scholz et al., 2003). The first
generation of airbags were more aggressive and caused injuries to the out of position,
frail and older drivers. Second generation airbags (Figure 2-5) used less inflation
power than the previous generation of airbags.
35
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 2-5: Grand Cherokee dual front airbags (©Cherokee motors)
While doing a cost benefit analysis on airbags in 1994 (Fildes et al., 1994),
comparison is made between the European standard and the American standard.
European standard airbags have 40 litres capacity with a 24 km/hr firing range
threshold, which is much smaller than the American standard airbag of 70 litre
capacity with a 16km/hr firing range. The European airbag has a slower deployment
rate than that of America and thus is less likely to injure an occupant. American
standard full size airbag is more suited for unbelted occupants. The concept of harm
reduction was introduced by Fildes et al. (Fildes et al., 1994) for quantifying these
benefits. It quantified the unit cost, the frequency of a particular type of injury and
the total cost of injury running into millions of dollars. This cost benefit analysis
(CBA) was very useful for automotive manufacturers who are always keen to lower
the risk and cost. The CBA showed that the American full size airbag has a ratio of
1.17, whereas the European standard airbag has a ratio of 0.98. This suggests that the
American full sized airbag is beneficial to the occupant due to safety reason as well
as car manufacturers.
The ultrasonic based occupant position sensor is a technology which allows the
position sensor to be installed as a real-time system. It is developed by National
Highway Transport Safety Administration, UK NHTSA (Breed et al.). This sensor
system is made up of four ultrasonic sensors as shown in Figure 2-6. Ultrasonic
36
Intelligent Automotive Safety Systems: The Third Age Challenge
sensors are able to find the driver position while driving. Pattern recognition in
conjunction with neural network is applied to estimate the position and velocity of
the occupant by the sensor. The sensor system is made to identify an empty seat, a
baby car seat and other scenarios. It estimates the delay in the ultrasonic pulse which
will help identify the position. The primary application of this position sensor is as an
active feedback system for intelligent airbags.
Ultrasonic sensors
Figure 2-6: Ultrasonic occupant position sensor
A Side-Airbag safety electrostatic capacitive sensor was created by Hubbard et al.
(Hubbard et al., 1999, Fukui et al., 2001). This sensor is installed on the inside of the
car seat. The impression on the car seat can measure height, weight and size of
occupant and inform of pattern. A pattern recognition algorithm is made to identify
adults with different heights and to deploy a side airbag accordingly. The sensor is
unable to give the position of the occupant. However position of occupant is crucial
in some cases, therefore, it is difficult to show the effectiveness of this sensor unless
the position of the occupant is predetermined.
37
Intelligent Automotive Safety Systems: The Third Age Challenge
Side airbags (SABs)
Side airbags (SABs) were introduced much more recently than conventional frontal
airbags. The commercial vehicle manufacturers started incorporating SABs since
1996. SABs gives protection against side impact crashes to the occupants see Figure
2-7.
Figure 2-7: Side curtain airbags deployed in crash test car (copyright Honda motors)
2.2.3.
Intelligent vehicle systems
Third aged drivers when turning at intersections are more likely to collide. Due to
sudden encounters with pedestrians, traffic or on coming cars (Daimon et al., 2003).
Third age driver behaviour in using Advanced Cruise-assist Highway system was
monitored. Advanced Cruise-assist Highway system uses two way communications
with other vehicles on the road using sensors. Experiments for this study were
conducted on a driving simulator. Two scenarios selected involved making a right
turn on a crossing and a pedestrian crossing. Both these scenarios are split into two
experiments. Measurements taken during the experiment to study driver behaviour
were coordinates, speed, time between oncoming vehicles when at an intersection
and visual behaviour using 8mm video camera at 30FPS. The subject drivers were
38
Intelligent Automotive Safety Systems: The Third Age Challenge
then asked to make a right turn and to cross the pedestrian crossing when it was safe
to do so in the driving simulation. Third aged drivers collided 2.5 times more than
young drivers, which is significant. Another important parameter is the minimum
distance between cars when making a right turn without collision. This can be
classed as a near miss, and is more frequent in third age drivers than younger drivers.
This is due to slower reaction times in third age drivers. The visual information
confirmed that third aged drivers took longer for glancing at information than
younger drivers. For the younger drivers the reaction time for all tasks completed
was less than 3 seconds and 75% of tasks were completed in less than 1.5 seconds.
For third age drivers a few tasks took more than 3 seconds to complete. The majority
of tasks took over 2 seconds to complete.
2.2.4.
Human and obstacle tracking
Human tracking technology used by many researchers is not completely novel or has
been used before using different techniques. Sensors such as placing load cells and
strain gauges on a car seat to find weight and position of the occupant, non-contact
sensors like CCD/CMOS cameras, infrared and ultrasonics to find driver's height
and position in real-time have been used. Tracking using intelligent sensors is not
only limited to automobile passengers. Intelligent sensors are being used more and
more everyday to track human movements as well as other obj ect tracking. Previous
research related to tracking using intelligent sensors is investigated in this section.
There are many kinds of detectors that are used for human information. Most
commonly used sensors are CCD cameras. These CCD cameras are high quality and
smaller in size but require a frame grabber card. CCD cameras with image
processing techniques are being used as human information systems. Application of
human information system are in Underground railway surveillance (Chow et al.,
2002), elevators for counting people (Schofield et al., 1997), tourist information
systems (Sacchi et al., 2001), face recognition and eye detection (Zhou et al., 2004).
Some researchers use camera images and develop novel algorithms using image39
Intelligent Automotive Safety Systems: The Third Age Challenge
processing techniques, while other research involves special kinds of sensor or
marker based tracking during the experiment to track the human motion.
Recently tracking different objects, including rigid or non-rigid objects is one of the
most active researches topics (Noyer et al., 2004, Tissainayagam et al., 2003). This
object tracking also involves tracking humans. Most examples of human tracking are
available using monocular image sequences (Noyer et al., 2004). For example
accurate motion tracking of the human body using body markers (Figueroa et aI.,
2003), pedestrian tracking (Pai et al., 2004), or from surveillance cameras (Chow et
al.,2002).
Human tracking is required in sports research for sport motion analysis. This is used
as an entertainment, television, sports motion study and training guide for
professional sportsman and medical research purposes (see Figure 2-8) (Chang et al.,
1997). The software developed by the author uses video sequence from the sports
scene. One video sequence shows tracking of a sportsman playing volleyball, Figure
2-8. From the video tracking it is possible to find information like the highest
possible jump taken by the sportsman, the distance travelled and movement
coordinates.
Figure 2-8: Tracking of a jump while playing volleyball (Chang and Lee, 1997)
40
Intelligent Automotive Safety Systems: The Third Age Challenge
Motion tracking is done by taking images from a camera and analysing those images
in real-time or offline. Tracking of several objects is previously conducted, for
example traffic tracking on a multi lane express way for the purpose of safety and
traffic management (Tai et al., 2004), human tracking to monitor their kinematics,
identification and to study their motion for anthropometry data (Ning et al., 2004).
Some of the applications require more than just 2D motion information, they also
require 3D information. 3D vision tracking is also important in virtual environments
and computer animation applications. There are two types of 3D vision tracking.
Monocular vision uses only one camera. Stereo vision uses multiple cameras. The
multiple camera approach can extract 3D coordinates directly (Sun, 2004).
Another application of tracking is vehicle hazard and obstacle monitoring. A vision
based real-time vehicle detection and recognition system was developed by Ran et al.
(Ran et al., 1999). The sensor system includes a colour CCD camera mounted inside
a vehicle pointing towards the road centre line. The video sequence is segmented and
edge detected. This system used mainly as a lane departure system, can detect
obstructions in the road. However the authors admitted that using CCD cameras will
have problems with lighting and lane detection during night time.
Image or vision based scanning systems generally work at the speeds of 25 Hertz, 30
Hertz, 60 Hertz or better for the intelligent transport systems (ITS). This speed and
distance cannot be achieved by sensors like laser range sensors, millimetre wave
radar sensors and tactical and acoustics sensors. Vision based systems can also
manage to work as lane tracking and obstruction tracking and detection systems. The
only limitations that make vision based system less robust are during fog, snow,
night or direct sun-shine conditions. Thus previously they have been replaced by
millimetre radar sensors (Bertozzi et al., 2000). Infrared based imaging systems are
able to use the flexibility and speed of vision based systems without having the
limitations of a vision based system with an exception of detecting white lines or
obstacles at ambient temperature. The infrared imaging systems can work poor
weather conditions such as night, fog, snow and rain (Flir, 2006).
41
Intelligent Automotive Safety Systems: The Third Age Challenge
2.3. Summary
•
Young drivers are at higher accident risk than any other age group, and the
second most vulnerable age group for accidents is the third age drivers. However
the third age driver accidents are more fatalities than injuries.
•
Accidents risks include driver distraction, drink and drugs (DUID), sleepiness
and other related symptoms.
•
Third age driving problems include vision impairments like tunnel vision,
reduction of visual field, not turning the head sufficiently, attentional abilities and
memory and motor skills deficits leading to errors and lapses.
•
Previous research shows that several intelligent sensors have been developed
like pro-tension seat belts, radar sensors, ABS, ESC, intelligent airbags.
•
Human tracking has been done previously for different applications like
automated people counting, face recognition, traffic control, public transport
surveillance, sports body motion. Most applications were vision based solutions.
42
Intelligent Automotive Safety Systems: The Third Age Challenge
3. Technical Background
3.1. Thermal imaging
All objects emit heat by three means: Conduction, convection and radiation.
1. Conduction, transfers heat through solid objects.
2. Convection, transfers heat through fluids like air and water.
3. Radiation, transfers heat through electromagnetic radiation.
Objects continuously radiate heat with a certain wavelength. This wavelength
depends upon the temperature of the radiating object and its spectral emissivity. As
the object temperature increases the radiation also increases. The radiation emitted
also includes the infrared radiation emission of wavelength between 0.7 micro metres
to 100 micro metres. Small ranges of infrared emission emitted by the objects are
detected by the thermal imagers, which is then made visible as an image.
The concept behind the thermal imager infrared emission detection is the notion that
the black body is a perfect radiator; it emits and absorbs all incident energy. The
energy emission for the black body is the greatest possible energy emission for that
certain temperature. Radiation power emitted by a black body as given by Plank's
radiation law is: (Bumay et aI., 1988)
43
Intelligent Automotive Safety Systems: The Third Age Challenge
P(A T)
,
= 2trhc
A2
2
{ex
(~)-l}-I
(Equation 1)
P AbT
where:
p= Energy Radiated
').,,= Wavelength
T= Temperature (Kelvin)
h= Plank's Constant
c= Velocity of light
b= Boltzman Constant
Real objects are not perfect emitters or absorbers. Thus emissivity (&) of the real
surface is defined as the ratio of thermal radiation emitted by a surface at a given
temperature to that of a black body for the same temperature, spectral and directional
conditions (HoIst, 2000). Thus emissivity of a black body is 1 and all other real
surface emissivities will be between 1 and O. This electromagnetic spectrum range
contains maximum radiative emissions, which are used for thermal imaging
purposes.(S.G.Bumay, 1988,2000).
According to the Stefan Boltzman Law of emissivity radiation:
(Equation 2)
where:
w = Radiated energy
8
= emissivity
b = Boltzman constant
(5.67 x 10-
8
,;
K4 )
T = Temperature (Kelvin)
44
Intelligent Automotive Safety Systems: The Third Age Challenge
Infrared imagers are commercially available that measure infrared radiation. Infrared
imagers are divided into five types based on the range of infrared radiation they can
detect.
3.1.1.
NIR (Near Infrared)
Detection wavelength range is 0.75 micrometre to 1.4 micrometre. These cameras
are used for several applications, for example food quality control measures (Uddin
et aI., 2006), fibre optics in telecommunications, pharmaceuticals, analysis of
chemicals and gasoline, and medicine for blood monitoring and imaging of materials
including tissues (Ciurczak et aI., 2002).
3.1.2.
SWIR (Short wavelength Infrared)
Detection wavelength range is 1.4 micrometre to 3 micrometres. Applications
include health and safety, surveillance, machine vision, night vision, and historical
art inspection. They have much higher frame rates than LWIR and other infrared
imagers and are thus recommended for machine vision applications.
3.1.3.
MWIR (Medium wavelength Infrared)
Detection wavelength range is 3 micrometre to 8 micrometres. It is also called
Intermediate Infrared (HR). Applications for MWIR include process control, nondestructive testing, failure analysis, wildlife study, medical imaging, security and
military use, maintenance and condition monitoring.
3.1.4.
LWIR (Long wavelength Infrared)
Detection wavelength range is 8 micrometre to 15 micrometres. These types of
infrared imagers are ideal for room temperature environments. This infrared imager
45
Intelligent Automotive Safety Systems: The Third Age Challenge
has expensive optics made from germanium, sapphire, or silicon. As LWIR is
suitable for room temperatures human information sensors, people counters,
surveillance equipment and military target applications are ideal applications. High
end LWIR are expensive due to cryogenic cooling requirements, uncooled LWIR
cameras are less expensive and use a micro-bolometer but have slower frame rates.
3.1.5.
FIR (Far Infrared)
Detection wavelength range is 15 micrometre to 1000 micrometres.
3.2. Thermal imagers
Thermal imagers are infrared cameras with a detection range of 0.9 micrometre to 14
micrometre wavelength. Thermography converts thermal radiation into digital
signals which converts it into a visible image. Thermographs are the image maps
created by the thermal imagers.
There are two types of thermal imagers-cooled and un-cooled. Depending upon the
application this selection is made. Un-cooled thermal imagers are most common and
preferred as these thermal imagers are less expensive and require less power. They
cover a spectrum range of 8 micrometres to 12 micrometres and stabilization time
required by the thermal imager is insignificant, therefore output can be collected
straightaway. The disadvantages of un-cooled thermal imagers include less
sensitivity to temperature, and they work only for close distances.
3.2.1.
Pyroelectric infrared detector
Pyroelectric (crystalline) materials produce charge when they undergo thermal
change. When the infrared radiation strikes the pyroelectric detector a charge is
produced. Pyroelectrics are not responsive to steady light input but react only to the
change. Thus a physical shutter effect is required which is also termed a chopper. As
46
Intelligent Automotive Safety Systems: The Third Age Challenge
the pyroelectric detector only gives absolute temperature without the chopper the
background temperature will disappear as pyroelectric devices only provide change
in temperature (Miller, 1994).
Top electrode
Bottom electrode
Membrane
Figure 3-1: Typical PZT pyroelectric sensor schematic (Schreiter et al., 2006)
Common pyroelectric detector materials are triglycine sulphate (TGS), lead zirconate
titanate (PZT), PBTi03, LiTa03 and LiNb0 3.
3.2.2.
IRISYS thermal imager
The IRISYS IRI 1002 is a low resolution low-cost infrared thermal imager
(Monarrez-Espino et al., 2006). The radiation detection range for this image is from
8 micrometre to 14 micrometres, thus classifying it as a long wave IR Imager
(LWIR) (AI-Habaibeh et al., 2003). Temperature range for this infrared imager is20°C to +90°C (with +150°C with reduced accuracy) with +/- 0.5°C error (Irisys,
2002). The original resolution of the imager is sixteen (16) pixels square but is
usually interpolated for better visual analysis. The imager has a maximum frame rate
of eight frames per second (8 FPS). The camera can be interfaced through RS-232C
serial port to the PC. The frame rate can be changed by the writing commands to RS232C port.
47
Intelligent Automotive Safety Systems: The Third Age Challenge
Thermograph acquisition from the IRISYS IRII002 thermal imager has three modes:
1. Acquisition of single infrared frame from imager when command is written
using serial port.
2. In this mode a specified number ofthermographs are required at certain
frame rate for a certain time period. This mode is useful but has a finite frame
limit of255. Therefore command needs to be present if more than 255 frames
are required. It is worth mentioning that the time internal between two serial
write commands cannot be set.
3. The third mode, when activated, sends thermographs at a specified frame rate
continuously.
As the optics is concerned, the field of view is 20° degrees. For example if the
infrared imager was to look at the area of interest from Imetre centre distance, 0.352
metre square will be the viewable region. The focal length 'f of the IRISYS imager
is 17mm and the IRISYS IRII002 is fitted with germanium lens.
The IRISYS IRII002 focal plane array measures relative temperature therefore a
mechanical chopper is necessary to make the 'shutter open' and 'shutter close'. A
frame by frame comparison creates a thermograph of stationary object. Without the
mechanical chopper the stationary object in the field of view of the thermal imager
will fade out into the background (Irisys, 2002).
3.2.3.
IRISYS thermal imager construction
The IRISYS imager is packaged in an aluminium die-casting case of 100mm x
100mm x 60mm, with a weight of less than 1.3kg (see Figure 3-2). The power
required is 12VDC at 300milliampere hours.
48
Intelligent Automotive Safety Systems: The Third Age Challenge
Motor
25mm diameter
36mmheight
·.·~·I
',:
:<
Figure 3-2 IRISYS imager sizing
With the electronics and optics, currently packaged, the IRISYS package could be
reduced up to 50mm x 50mm with 36mm thickness (approximately). The motor
which is being used as a chopper is a considerable factor in increasing package size.
The diameter of the chopper motor is 25mm with a length of 36mm a maximum
speed of 240RPM. If the motor is reduced in size the packaging of the thermal
imager can be reduced significantly.
3.3. Thermal imaging applications
The potential applications of thermal imaging are numerous. There are complete
books devoted to the applications and their description of thermal imaging. Therefore
only limited applications are listed from different areas of interest in Table 3-1.
49
Intelligent Automotive Safety Systems: The Third Age Challenge
Applications
Area 4interest
Gun sights/target
Military and paramilitary
Infrared search and track
Military and paramilitary
Military ground vehicle sensors
Military and paramilitary
Military space sensors
Military and paramilitary
Missile seekers
Military and paramilitary
Tactical missile warning
Military and paramilitary
Perimeter surveillance
Military and paramilitary
Drug interdiction
Military and paramilitary
Law enforcement
Military and paramilitary
Temperature distribution in wind tunnels Industrial, inspection and monitoring
for example temperature distribution on (AerospacelMilitary)
wings, missiles and fuselage
Rocket and jet engine diagnosis
Industrial, inspection and monitoring
(Aerospace)
Shape and temperature distribution in Industrial, inspection and monitoring
exhaust plumes from aircraft jet engines
(Aerospace)
Inspection of gas and fluid relief values
Industrial, inspection and monitoring
(Petrochemical)
Detect faulty components in printed Industrial, inspection and monitoring
circuit boards
(Electrical/electronics)
Breakdown of insulation on power lines
Industrial, inspection and monitoring
(Electrical/electronics)
Monitoring
applications
of
electrical
switchgear Industrial, inspection and monitoring
(Electrical/electronics)
Monitoring of transformers and circuit Industrial, inspection and monitoring
breakers
(Electrical/electronics)
Evaluating and inspecting furnaces / Industrial, inspection and monitoring
Refractory lining and its inspection for (Steel)
50
Intelligent Automotive Safety Systems: The Third Age Challenge
cracks and defects
Industrial, inspection and monitoring
Safety for coal and slag
(Steel)
Monitoring of bearing performance by Industrial, inspection and monitoring
(Mechanical)
measuring friction generated
Optimization of design of mechanical Industrial, inspection and monitoring
parts like improves design of belt and (Mechanical)
pulley to reduce energy loss
Checking
plastic
mould
and
die Industrial, inspection and monitoring
temperature distribution measurement &
(Manufacturing)
check performance of pre-heated plastic
samples
Night VISIon system for commercial Industrial, inspection and monitoring
vehicles
(Automotive)
Inspection of electrically heated car Industrial, inspection and monitoring
windows
(Automotive)
Thermography is used to study moisture Industrial, inspection and monitoring
non-uniformities
in
the
paper
manufacturing process
Determining thermal efficiencies
Monitoring
electrolytic
Industrial, inspection and monitoring
cells
In
Industrial, inspection and monitoring
chemicals like chlorine and fluorine
Inspection of heat exchangers
Industrial, inspection and monitoring
Non destruction testing
Industrial, inspection and monitoring
Steam trap inspection
Industrial, inspection and monitoring
Inspection of process plants
Industrial, inspection and monitoring
Inspection
of
electrically
heated Industrial, inspection and monitoring
windows
Investigation of vascular disorders
Medicine
Oncological investigations
Medicine
Investigation
of
pain,
trauma
and Medicine
inflammatory conditions
51
Intelligent Automotive Safety Systems: The Third Age Challenge
Earth observing sensors
Astronomy, metrology and geo
Interplanetary space sensors
Astronomy, metrology and geo
Weather instruments and cameras
Astronomy, metrology and geo
Environmental/pollution monitoring
Various
Oil pollution control and oil spill
Various
Endangered species monitoring
Various
Night vision for commercial airlines and Various
shipping
Study isotherm patterns of welding
Various
Testing of tyres and temperature profile Various
measurement across tyres
Fire brigades to see through smoke, mists Various
and evacuation purposes
Table 3-1 Applications of thermal imaging (Burnay et aL, 1988, H olst, 2000)
-
-
Use of thermal imaging in the automotive research industry has improved in the last
few years. Night vision HUDs are being installed in high end cornmercial vehicles to
see ahead in night and poor visibility conditions (Martinelli et al., 1999). High cost
thermal imagers are also used for designing a climate control system in automobiles.
A study was conducted by Ghiardi (Ghiardi, 1999) in which a high spatial resolution
thermal imager is focused on the face of the driver. Several scenarios were conducted
in which a driver got in the car fitted with thermal imager. These sc enarios vary from
hot sunny weather to cold rainy weather. An automated climate control system linked
with the thermal imager measured heat patterns from the driver's fa ce and monitored
driver condition over time inside the car. More automotive applications using
thermal imaging are discussed in the night vision sensor section 2.2.1 and thermal
human tracking section 2.2.4.
Infrared imaging can aid in the study of geology. For example fin ding a rise in sea
temperatures, levels of desertification on land, finding clouds and 0 ther geographical
features using satellite imagery. Thermal imaging provides a significant advantage
52
Intelligent Automotive Safety Systems: The Third Age Challenge
by removing the limitation of day and night, thus land, sea and clouds can readily be
identified. Metrological satellites are being used for detecting clouds and their
patterns to study and forecast weather (Pergola et al., 2004, Fisher et al., 2004).
Medical thermography is an actively researched topic. At an ambient temperature an
unclothed healthy person has a temperature of 35 degrees over the chest region
whereas 25 degrees over the feet.
1. Skin surface temperature is based on determination of age of a person. The
warmest temperature is near the head and trunk.
2. The surrounding temperatures have a significant effect on the human body.
3. During exercise the temperature of the human body increase up to 40 degrees
without any illness. But monitoring during exercise the excessive heat is
generated by the active muscles. For example running produces heating
effects in legs.
4. Obesity modifies the temperature distribution. Different thermal patterns are
created in the obese. The fatty areas are the cold regions which modify the
expected heat pattern shown by non-obese people.
3.3.1.
Thermal imagers for human tracking
There has been a growing need for detecting and tracking human bodies using noncontact methods in the field of air-conditioning, lighting, security and others. But
human information is difficult to process if only visual sensing is used, as the
lighting variations give a major challenge making visual cameras inadequate for
processing human movements. Infrared imagers tend to simplify this imaging
problem. The advantage of infrared cameras over visual is their use in night time and
bad weather conditions.
53
Intelligent Automotive Safety Systems: The Third Age Challenge
Infrared tracking is not very common, as many applications require high spatial
resolution infrared cameras. which are expensive (Reynolds et al., 2002). Thus due
to this reason Infrared target tracking is vastly used in air defence applications only
(Yilmaz et al., 2003, Tidrow et al., 2001).
A journal paper by Eveland et al. (Eveland et al., 2003) describes research on
detection and tracking of faces using a thermal imager. For the purpose of
thermograph segmentation the author classified scenes into exposed skin, covered
skin (with clothes, hair etc) and background. After modelling the skin into thermal
scenes the author was able to calculate the probability of exposed skin and covered
skin in the scenes. The author used MWIR and LWIR high spatial resolution thermal
imagers for indoor and outdoor detection and tracking of subjects.
Human observation or human information sensors are actively usmg thermal
imagers. Research conducted by Armitage et al. (2004) used IRISYS thermal
imagers for tracking and counting people. The author argues that using low
resolution low cost IRISYS thennal imager is the preferred choice over visual and
high resolution thennal imagers as it works in any lighting conditions as well as
being low in cost. A very simplistic approach is used by (Chamberlain et al., 2004).
Two people counters are installed which are capable of acquiring 16x16 pixel
thermographs. By using two people counters it is possible to get sub pixel accuracy
for the centroid of the targets.
Another pedestrian monitoring sensor developed by Armitage et al. (2005) provided
a working area of 10 square metres. This was an improvement on the previous sensor
mentioned in Chamberlain et aI., (2004), but real time. The author argues that the
techniques used by infrared imagers are significantly different from visual sensors
due to the thermographs involved. The research used two thermal imagers, IRISYS
and FUR, with a resolution of 16 square pixels and 256 square pixels respectively
(see Figure 3-3). The later thennal imager is used to segment the thermographs into
regions by differentiating between background and people.
54
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 3-3: Pedestrian classification using high resolution FLIR thermal image. Hot spots
appear in white (Armitage et al., 2005).
An occupant detection sensor was developed by Morinaka et al. (Morinaka et aI.,
1998). This occupant detection and movement sensor consists of two parts, an upper
part which is a distance sensor and a lower part which is a pyroelectric infrared
detector made using PBTi0 3 • The pyroelectric infrared detector consists of a
spherical lens and chopper mechanism which is connected to a brushless motor, the
resolution achieved by interpolation was 48 pixels by 180 pixels. The distance was
measured by a four channel distance sensor consisting of near infrared LEDs
transmitter. The experiments were conducted in 5 metres by 6 metres area. The
detection algorithm is based on simple fuzzy logic. However these IR imagers are
very expensive and thus low-resolution low-cost IR imagers are developed for this
purpose. These IR imagers are un-cooled with a mechanical chopper, which is
inserted in front of the detector. Further research on a similar sensor design is
discussed and used by Hashimoto et al. (Hashimoto et al., 2000) and Yoshiike et al.
(Yoshiike et al., 1999).
Other smaller applications of infrared imagers include using it as a night vision
device for tracking and detection of animals, termites and insects (Reynolds et al.,
2002).
55
Intelligent Automotive Safety Systems: The Third Age Challenge
A medium-resolution infrared imager (64 x 64 pixels) was also developed
for
position tracking of occupants for an intelligent airbag system (Qinetiq, 2004) as
shown in Figure 3-4. This sensor developed by BAE system's sister company
'Qinteiq' in collaboration with First Technologies shown in Figure 3-4 named the
'Fungi Thermal Imager'.
Lo\V resolu1ion infrared sensor
detoction of occupant position
Figure 3-4 Medium resolution occupant position detector by (Qinetiq, 2004)
3.4. Infrared & Visual Image Acquisition software
I-Quire software, as shown in Figure 3-5, is used which was developed by the author
for the infrared and visual image acquisition. This software acquires webcam images
and thermal imager in soft real-time. The image frequency in the experiment is set at
2FPS. This image acquisition frequency is selected on the basis of the length of
experiment and reaction time of each volunteer. The image acquisition is done for
the whole length of the simulation scenario. This software is modified extensively for
the experiment to take an unlimited number of images during the experiment, see
section 7.2.4.
56
Intelligent Automotive Safety Systems: The Third Age Challenge
lIo,ot~
li".. ft'iifVlI
~~~
'JoF-..;
ASC."
i
SIIlOJ't(J
Jtl'$'i$lrl...p~
. ~:~:I'i
~4x.4·
Wh 1.19'
r
S~_!n\lI~rg ...
r.e",...,~<l:f"''''
O;'~Ii9It""Qt{d_«-l
•..,
~"H''''.lk..",:<V,<It....
SOlG'9tC>.n~",,_._,-,
Col",.
..
OdM F..1ht !dW<>1" •• , .• c:'1r>;dlU
Sdtcm:I".,1
~._",•• _.,~,.~•••. ,.~_,,_~_••,~~J
0.,. I
r""n""",.,...
" •.,,1
Figure 3-5 I-Quire modified version
3.5. Talley pressure matrix
A Talley pressure monitor is manufactured by Talley Medical for the pressure
monitoring which is used as a design tool for different products and experiments. It is
used to estimate the weight of the occupant. Also the pressure monitor is helpful in
finding the position of the occupant as it finds the pressure of small air pockets. The
pressure monitor is connected through RS-232C port from the IBM-PC. The
specifications of the Talley pressure monitor are shown in Table 3-2.
57
Intelligent Automotive Safety Systems: The Third Age Challenge
Number of air matrix
8
Total number of air pockets
8x12 (96)
Communication
RS-232C
Throughput
9600bps
Parity
o
Table 3-2 Talley pressure monitor specifications
Pressure Matrix arrangement
Figure 3-6 Talley Pressure monitor set-up on driving seat; V-shaped to correspond to driver
legs position
58
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 3-7: Talley Pressure monitor control
The raw data is read into MatLAB. The data is then arranged into a realistic V-shape
as arranged onto the driving seat shown in Figure 3-6. The measurements of the
driving seat are also taken into account. Figure 3-7 shows the control unit of the
Talley pressure monitor. The resulting surface received from the Talley pressure
monitor while driving is shown in Figure 3-8. The whiter the surface the higher the
weight, while black shows no weight.
59
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 3-8 Resulting surface of Talley pressure monitor
The pressure mat has 96 air pockets, which are used to measure the pressure and is
connected to the Talley pressure monitor which pumps the air and measure it. The
values are then output through RS-232C port and stored in ASCII format.
3.6. Visual camera
The main purpose of using a webcam during the experiment was to compare and
verify the results. Visual images provide plenty of information. It can be used to
validate infrared imager thermographs. Using a webcam instead of a high-resolution
visual camera allowed the experiment to be conducted at extremely low costs.
60
Intelligent Automotive Safety Systems: The Third Age Challenge
3. 7. Driving simulator
To simulate the driving in a laboratory without risking safety a low cost driving
simulator was used for the experiments. The STI Driving Simulator (STISIM) by
Systems Technology, Inc is most suitable for research purposes. It is one of the most
stable driving simulation packages around with 40 years of development (Noy et al.,
2004).
The simulation is created using complete vehicle dynamics model in real-time (Wade
Allen et al., 1998). This vehicle model was developed for the National Highway
Traffic Safety Administration. STISIM contains a GAINS file. This file contains
several parameters of vehicle dynamics and visual display transport. For the STISIM
release used, sixty-three (63) parameters are given in the GAINS file. From that you
can set the control to response relation of steering wheel input, lateral and directional
motions and their relationship with throttle and brake. The vehicle dynamics model
contains a steering module, which inputs the steering values calculating the vehicle
path curvature. A speed control module inputs throttle and brake. The speed feeds
into the steering control module. To calculate engine RPM transmission gear ratio is
used.
3.7.1.
Construction of Ford Scorpio test rig
The hardware and software (see Section 7.2.2) was supplied by STISIM, and
installed in a static Ford Scorpio car, Figure 7-1. The car is controlled by force
feedback steering column, accelerator and brake. The steering, brake, accelerator and
speedometer are connected to absolute encoders, which give analogue readings to the
Data Acquisition Card (DAC). This DAC is connected and configured with a PC
with an installed copy of STI Driving Simulator. The DAC board is from CIO-
61
Intelligent Automotive Safety Systems: The Third Age Challenge
DASOS/JR series from Measurement computing Inc and has a capacity of S analogue
and S digital 110 channels (Measurement-Computing, 2001).
The animated driving scene was projected onto a wall, which is 5 metres by 5 metres
and 5 metres (approx.) away from the driver's position. The software then projects
135 degrees field of view of scenario on the wall.
The later set-up was in a custom made rig constructed by ESRI (Ergonomics and
th
Safety Research Institute), which can be adjusted to accommodate 5
to 95
th
percentile range of users. The steering force feedback column and pedals is taken
from a Range-Rover.
3.7.2.
Programming scenarios
Driving tasks and scenarios are defined using a command list of events called simple
scenarios definition language (SDL). The simulation, while running, can collect and
store various parameters. Speed, vehicle curvature, road curvature, vehicle heading
angle, lateral lane position, distance travelled, steering angle, throttle input, brake
input, time, signal indicators and use input based signals can be collected and used
for offline data analysis (Rosenthal et al., 1999). The build 1.1.15 for STISIM used
in the experiments gives 50 command list events, which can be programmed into
simulation scenarios.
3.8. Artificial
intelligence
and
Image
processing
techniques
Digital image processing is currently a very active field of research. Humans have
the ability to analyse and act according to visual information, an ability which is
quite remarkable and unnoticed by humans. To make a machine do the visual
62
Intelligent Automotive Safety Systems: The Third Age Challenge
analysis and act accordingly is a challenging task. As images taken, visualized by
humans and photographed has now taken a complete turn. Now the images are
captured, manipulated and action which is derived from them is made through
computers. Great technological advances have been made to visualize these images
from the eye of machines over the years. This has opened many areas of new
research and merged many engineering disciplines (Jahne, 1999).
Further work is done in particular areas to make the computers more intelligent for
analysing the visual information provided. Now they are able to predict and scan
information almost like humans do. For example, object recognition such as
distinguishing between different kinds of fruits, or maintenance of rail fasteners
(Mazzeo et al., 2004).
In biometrics, the human face(Zhao et al., 2004, Turk et aI., 1991) or finger print
recognition is routinely carried out. Use of artificial intelligence and digital image
processing in medicine (Wu, 2004), traffic control and management (de la Escalera
et aI., 2003) and automated target recognition is well developed (Pasquariello et al.,
1998).
The most commonly used techniques in computer vision artificial intelligence are:
1. Fuzzy Logic
2. Artificial Neural Networks
a. Backpropagation neural network
b. Radial basis neural network
3.8.1.
Fuzzy logic
Fuzzy logic is a mathematical technique for dealing with imprecise data and
problems. The fuzzy logic system makes only true and false (if-else) decisions based
63
Intelligent Automotive Safety Systems: The Third Age Challenge
on rule sets. This system resembles human logic and
IS
a kind of artificial
intelligence.
More information about fuzzy logic based system can be taken from Harris (Harris,
2000, Timothy, 2004)
3.8.2.
Neural networks
Artificial Neural networks (ANN) (Haykin, 1999) are mathematical models that
resemble the biological method of human decision making processes by using
neurons which are interconnected to each other (Lee, 2004): The ANN decision
accuracy depends upon the following two factors:
•
The learning process: The training method is different for each type of neural
network. The supervised and unsupervised learning are the most common
techniques and will be discussed below.
•
An ANN stores information in the form of interconnection strengths between
neurons and the synaptic weight of each neuron.
Artificial neural networks are collections of mathematical models that emulate some
of the observed properties of biological nervous systems and draw on the analogies
of adaptive biological learning. The ANN has a novel architecture that mainly
contains highly interconnected neurons. These neurons contain the activation
functions like 'linear', 'sigmoid', 'logarithmic', 'tangential' and so on, depending
upon the particular application of where the ANN is used.
The main advantage of using neural networks is the full automation of the learning
and classification processes. Therefore, they can be implemented in fully automated
monitoring systems, such as people counting to recognize and classify different
64
-
----
-----------------------------------------------------------------------,
Intelligent Automotive Safety Systems: The Third Age Challenge
patterns without human involvement, thereby, eliminating any error or lapses
associated with human concentration during a repetitive task.
As in nature, the network function is determined largely by the connections between
elements. Some Neural networks are classified as feed-forward while others have a
different architecture like self-organized or supervised network, RAM based neural
network, Radial Basis Neural Network (RBN). The selection of ANN, training
technique and activation function used depends upon the type of data that is being
processed (Ramadan et al., 2004).
Supervised Learning
During supervised learning of an ANN, an input stimulus is applied that results in an
output response. Then this response is compared with a desired output i.e. the target
response. If the actual response differs from the target response, the neural network
generates an error signal, a popular measure of the error 'E' for a single training
pattern, is the sum of square differences i.e.
(Equation 3)
Where,
ti =
desired or target response for ith unit,
Yi = actually produced response for ith unit.
E = Error calculated for adjustment of synaptic weights
The error "E" is then used to calculate the adjustment that should be made to the
network's synaptic weights so that the actual output matches the target output.
65
Intelligent Automotive Safety Systems: The Third Age Challenge
Unsupervised learning
Unsupervised learning does not require a target output. It is usually found in the
context of recurrent and competitive nets. In case of unsupervised learning, there is
no separation of the training set into input and output pairs during the training
session, the neural net receives as its input many different excitations, or input
patterns, and it arbitrarily organizes the patterns into categories. When a stimulus is
later applied, the neural net provides an output response indicating the class to which
the stimulus belongs. If a class cannot be found for the input stimulus, a new class is
generated. However, it should be noted that even though unsupervised learning does
not require a teacher, it requires guidelines to determine how it will form groups.
Grouping may be based on shape, colour, or material consistency or on some other
property of the object.
Back Propagation Neural Network
Back Propagation Neural Networks (refer Figure 3-9), are one of the most commonly
used neural network structures, as they are simple and effective, and have been used
successfully for a wide variety of applications, such as speech or voice recognition,
image pattern recognition, medical diagnosis, and automatic controls.
66
Intelligent Automotive Safety Systems: The Third Age Challenge
Input
Layer
Hidden La yer
Output
Layer
Figure 3-9: Back Propagation Neural network construction
It is a supervised neural network, which consists of "n" numbers of neurons
connected together to form an input layer, hidden layers and an output layer. The
input and output layers serve as nodes to buffer input and output for the model,
respectively, and the hidden layer serves to provide a means for input relations to be
represented in the output. Before any data has been run through the network, the
weights for the nodes are randomly chosen, which makes the network very much like
a newborn's brain, developed but without knowledge. When presented with an input
pattern, each input node takes the value of the corresponding attribute in the input
pattern. These values are then "fired"', at which time each node in the hidden layer
multiplies each attribute value by a weight and adds them together. If this is above
the node's threshold value, it fires a value of "1'''; otherwise it fires a value of "0".
The same process is repeated in the output layer with the values from the hidden
layer, and if the threshold value is exceeded, the input pattern is given the
classification. Once a classification has been given; it is compared to the actual, i.e.
desired classification, and the error is fed back (back propagated) to the neural
network and used to adjust the weights such that the error decreases with each
iteration and the neural model gets closer and closer to producing the desired output
(Ince, 2004, Ramadan et al., 2004, Marengo et al., 2004). This process is known as
67
Intelligent Automotive Safety Systems: The Third Age Challenge
"training". The back propagation neural network used in this study uses a sigmoid
function in the hidden layer and a linear function in the output layer.
Radial basis neural network
A typical radial basis function (RBF) network is a three-layer network: a layer of
input neurons feeding the input vectors into the network; a single hidden layer of
RBF neurons calculating the outcome of the basis functions; and a layer of output
neurons calculating a linear combination of the basis function. The number of input
neurons should be the same as the number of input variables.
RBF networks are often used to solve problems of supervised learning. Supervised
learning is to guess or estimate a function from some example of input-output pairs,
with little or no knowledge of the form of the function. The function is learned from
the samples that a teacher supplies. The training set contains elements that consist of
paired values of the input and the output. The function relation between the input (x)
and output (y) is given by y = f (x), where x is a vector and y is a scalar. The units in
the input layer do not process the information, and they only distribute the input
variables to the hidden layer. Thus, the RBF network can also be considered as a
two-layer network.
RBF neural networks can offer approximation capabilities similar to those of the
multi-layer perceptrons which are basic components of BPN. In general, the radial
basis function method is a global interpolation technique that has good localization
properties (Erfanian Omidvar, 2004), as it avoids the difficulty of local optima by
conducting the training procedure in two steps. The locations of the centre vectors
are found in the first step; then the values of the weights are optimized in the second
step. Therefore, it provides a smooth interpolation of scattered data in arbitrary
dimensions. It has been proven (Park et al., 1991, Park et al., 1993) that radial basis
neural networks with one hidden layer are capable of universal approximation.
Radial basis neural networks can be summarized as follows (Morelli et al., 2004):
68
Intelligent Automotive Safety Systems: The Third Age Challenge
j
f (V, ) =L q;O"; Cliv, - u; IIJ + p (v
I)
1=1
where,
U1 = centre of activation functions
ql
= parameter for optimization
P (v I )
(Equation 4)
=polynomial
0"
=activation function
j
=number of neurons
RBF neural networks have widely been applied to problems of supervised learning,
such as regression (Li et aI., 2004, Loukas, 2001) and pattern recognition (Haddadnia
et aI., 2003).
3.9. Summary
•
Heat is transferred by three means: conduction, convection and radiation.
Thermal imaging can detect heat that travels by means of radiation. Thermal
imagers are used to acquire this radiation data in the form ofthermographs.
•
Thermal imagers are classified based on their temperature measuring range.
Discussed in detail is the IRISYS thermal imager which is a 16x16 array low
resolution thermal imager available at a low cost. It is classified as a LWIR
thermal imager which communicates to an IBM PC using RS-232c port. Human
information sensor and other room temperature applications are based on LWIR
type IR imagers.
•
Software is developed as a data acquisition platform using National
Instruments LabIWindows platform. It acquires, organises and stores IR and
visual data.
69
Intelligent Automotive Safety Systems: The Third Age Challenge
•
This chapter also discusses briefly the tools that will be used during the
experimentation like the STISIM driving simulator, visual camera and Talley
pressure monitor.
•
Types of artificial intelligence are discussed. Working of most common types
of neural networks i.e. BPN and RBN is explained.
70
Intelligent Automotive Safety Systems: The Third Age Challenge
4. Hypothesis
4.1. Hypothesis
The information provided by low resolution infrared imaging can be used as the basis
for a system which can reduce the risks to 3rd age and similarly impaired drivers.
4.2. Research question
Research questions form the basis on which research is planned and conducted.
Every research programme has a hypothesis or a main research question. This
chapter deals with the main research question and secondary research questions.
•
How may we identify driver postures, movements and behaviours which pose
a high level of risk?
4.2.1.
•
Secondary research questions
How to create a low cost safety system which would be non-contact and nonintrusive?
•
How can low resolution infrared imaging be used to find driver's movement
while driving?
•
Can restriction of movement be found in the third age and impaired drivers?
71
Intelligent Automotive Safety Systems: The Third Age Challenge
•
Can the system locate 'out of position' drivers?
•
Can the system detect periods with eyes off the road?
•
Can the system identify drowsy drivers? What can be done to identify signs
of drowsiness while driving?
•
What can be done to make cars safer by considering the driver's height! head
height?
•
How can driving be made safer for third age drivers in the future by using this
system?
The main focus of this research would be on the third age drivers. However, this does
not mean that the proposed safety system is any less useful to other drivers.
4.3. Detailed research layout
Developing an AI (Artificial Intelligence) based thermographic imaging algorithm
(AIT!) and conducting experiments to verify it are interrelated activities. This is due
to the complexity of the process involved in designing an AITI algorithm. Therefore
a research layout is first defined which helps in developing the AITI.
The research programme layout (Figure 4-1) is different from the methodology given
in Chapter 1. The methodology provides a 'bird's eye view' of the research whereas
the research layout defines how the technical work will be conducted and relates to
Stage 2 of the methodology (Figure 1-2).
72
Intelligent Automotive Safety Systems: The Third Age Challenge
IR Position tnclcer
t-------------------- ----------------------------
Track position
Identify Subject
I
,--------- --------
,
:
:
Results
:--------1
,
: k sess resu1ts
t_______________
I-------...J
)<4:
,-------------------------,
_____________________ J
lot
f
~~~~~-~~~~~L
:' _________________________
Change of camera position:1
.1,
Beha.viour Modtller
~----~--------~----------------------,
ANN Model
Comparison with
re allife data
Driving Model
.--------
,:
I
:
, Assess results
:
•__ ..... __ .....
.
--------~
Results
:
,
~---------------------------~I
:
: Steering. brake and throttle :I
,"+--:'T"
. ..ustal1ation of low cost sensors:',I
:
-:::.::::.::..---::.:..-..:.::.::::.-..::.-.:::~::~--=:..-..:::.:.!
Driver task analysis
,--------------------- --------------------------,
t
,
:
,,
Drivertasks
:
,,
,
I
,
,
"
Assess capabilities of system w.r.t
':
driver taskslbehaviours
,
~____________________
Specific tasks
I'_,
Complex tasks I'_"
,
_ __________________________ l
Assess potential for integration in intelligent car
Figure 4-1 Flowchart of research programme
73
Intelligent Automotive Safety Systems: The Third Age Challenge
The research programme is divided into four sections, which are:
1. IR position tracking
2. Behaviour Modeller
3. Driver task analysis
4. Integration
The first section of the research layout helps to identify the limitations and behaviour
of the IRISYS infrared imager. This section will have series of smaller experiments
and will help find an optimal method for tracking driver movement using
thermographic imaging. Each imaging technique will be verified and validated with
experiments, making it an iterative route. The position of the IRISYS thermal imager
is also very crucial and its optimal location will be found in this stage.
A behaviour modeller will be the core section for the development of the AIT!
algorithm. A neural network will be designed that will use successful techniques of
thermographic imaging from the previous section.. Alternatives to neural network
are fuzzy logic, hybrid intelligence and Bayesian network. The main decision
systems are fuzzy logic and neural networks. A comparison of fuzzy logic systems
and neural networks and why neural networks are preferred over fuzzy logic for this
application are discussed in section 6.1 and section 6.2. A well laid out experiment
will be conducted on a driving simulator to validate the AITI algorithm. Validated
results from the experiment will also be compared with real life video driving data.
Another experiment will be conducted with a wider range of subjects. Data from the
experiment will be analysed by the AITI algorithm developed in behaviour modeller.
The third section will also answer the research questions made earlier in this chapter.
It will further discuss the results and scenarios in which the system will be useful.
The 'integration' section will look briefly into the future research work for this
system and integration of this system in vehicles.
74
Intelligent Automotive Safety Systems: The Third Age Challenge
4.4. Hypothesis validation
The research will aim to provide validation that the proposed system can track driver
postures, movements and aspects of behaviour which might affect safety.
4.5. Summary
•
A hypothesis and primary research question are provided, aimed at
determining how to identify driver movements and postures which pose a high
level risk. Other secondary questions are also stated which are related to the
primary research question.
•
An AITI algorithm will be developed and experimentation will be conducted
to evaluate the results of the algorithm. This research is laid out in four stages.
•
The safety system will deliver driver posture detection at the end of the
research.
75
Intelligent Automotive Safety Systems: The Third Age Challenge
5. Imaging techniques and Image
Processing Algorithms
This chapter is divided into two main sections. The first section discusses the
imaging techniques or special techniques developed and subsequently used in the
image processing algorithm. The second describes the actual development of the
image processing algorithm.
5.1. Imaging techniques
The following sub-section describes the imaging techniques used in the development
of an imaging algorithm or the concepts of imaging which are useful in the
understanding.
Infrared imaging is mainly a temperature measurement system. Previously the
primary application for infrared imaging was in the field of defence and military but
since the introduction oflow resolution infrared imagers with lower cost applications
commercial and industrial solutions are being developed and researched. Infrared
imaging is superior to visual systems for tracking or monitoring human and other
living beings as it seeks heat from the body. In an automobile safety context infrared
imaging is used in applications such as minimizing night hazards, and improved
visibility in poor weather conditions, especially in fog, rain and snow.
76
Intelligent Automotive Safety Systems: The Third Age Challenge
5.1.1.
Difference between visual image and infrared
thermograph
A true digital colour image is a 24-bit image. Each pixel value in this image is
specified by 3 values which are red, green and blue in a true colour image. A true
colour image does not use a colour map, the pixel colour and intensity is calculated
by the combination of these three values, whereas a greyscale image is only 8 bit as
there is only one bit-value for each pixel. A colour map is required to render a
greyscale image; it consists of different intensities of grey against each pixel value.
In other words a greyscale image pixel value is equal to the average of three values
of each pixe1 in a true colour image.
The difference between a greyscale image and infrared thermograph is its pixel
value. In an infrared image the intensity value of a pixel is replaced with the
temperature; see Figure 5-1. An infrared thermograph can be considered as an 8-bit
temperature map. For example, consider an infrared image with greyscale infrared
thermograph, the higher the temperature the whiter (hotter) the pixel.
RGB image 1.5 Red, 0.5 Green &
3.6 Blue gives different intensities when
combined makes a single pixel
Infrared
Temperatur
in Celsius
, 5
}
Infrared image gives temperature in
Each pixel instead of colour intensities.
Each temperature can then be assigned
with colour to display an image called
thermogram.
Figure 5-1: Differences between infrared and visual image
77
Intelligent Automotive Safety Systems: The Third Age Challenge
5.1.2.
Infrared Image Interpolation
An interpolation process estimates values of intermediate components of continuous
function in discrete samples. Interpolation is extensively used in image processing to
increase or decrease the image size. There are commonly five types of interpolation
used cubic, spline, nearest, bilinear and hyper-surface (Kulkarni, 1994).
An interpolation technique does not add extra information into the image but can
provide better thermal images for human perception. For bicubic interpolation, the
output pixel value is the weighted average of the pixels in the nearest 4 x 4
neighbourhood. Mathematically, bicubic interpolation can be described as follows:
The Lagrange polynomial interpolation:
3
P(q) = ~:rjLj(q)
(Equation 5)
j=O
(AI-Habaibeh et al., 2003)
Where,
q=Point at which interpolation takes place
P(q)= interpolated value
fj=Known values on the grid at points (qJ
Lj(q)=Lagrange polynomial, for example
The infrared images taken from the infrared imager reqUIre interpolation. The
interpolation is required mostly for recognition of features by the human eye as the
16 x16 pixel image as shown in Figure 5-2 does not give enough visual information.
The linear and spline interpolation are better interpolated functions but due to
78
Intelligent Automotive Safety Systems: The Third Age Challenge
computation complexities and time taken by the spline interpolation the linear
interpolation is preferred as it is the simplest type of interpolation. A single infrared
image, as shown in Figure 5-2, with all four types of interpolation in a greyscale
colormap.
Linear Interpolation
Spline Interpolation
Figure 5-2: Four types of interpolated infrared images
5.1.3.
Multiple level segmentation
The linear and spline infrared interpolated image also provides more image area. The
next stage is to reduce the linearly interpolated image into 4 ranges of temperature as
shown in Figure 5-3.
Based on principals discussed in the above paragraph and also discussed by Eveland
et al. (Eveland et aI., 2003); 100 images were taken of the volunteers. All of these
readings are taken after volunteers had been in room temperature for more than 15
minutes. Room temperature ranged from 19 to 21 degree Celsius. Before going
further the difference between visual and infrared images is discussed in the
following paragraph.
79
Intelligent Automotive Safety Systems: The Third Age Challenge
DOL-I_ _
Reduced to
256 colour
infrared image
Covered skin
area
Lightly covered
or skin
4 colour infrared image
Face area
Figure 5-3: Reduced colour infrared image
Thus these 100 infrared images are interpolated to obtain a bigger visual image.
Using the mean of reference infrared images the room temperature is calculated.
Reference infrared images are created by taking samples of images without any
subject in the image field of view. During these experiments room temperature value
was from 18.5 to 20.5 Degree Celsius. Thus any values from 18.5 to 20.5 are
assigned to the background layer. Going further, after analysing these infrared
samples with visual images four more layers are assigned to the infrared image.
These layers are as follows:
1. Covered skin: this area of infrared includes covered skin, mostly can be
covered with cloths.
2. Hair and skin: The second layer is assigned to hair, hands and areas other
than the face.
80
Intelligent Automotive Safety Systems: The Third Age Challenge
3. Face area: the third and most important layer is the face layer. This mostly
focuses on the area of the face. This temperature range will be used in the
first part of image processing.
4. Face feature: This range being warmer than everything else, focusing on the
mouth and nose region. Sometimes it also focuses on the forehead depending
upon the density of hair on the forehead.
This classification of infrared temperature range is suitable for volunteers who are
working in normal conditions at room temperature of 21 degree Celsius. Thus after
classification at different temperature the multiple layer segmentation is shown in
Figure 5-4. The segmentation was done with the face features being the warmest
region, then the face region temperature, followed by uncovered skin regions. The
covered body background separation is from the cold background which was around
21 degrees Celsius at the time of the experiment. The final interpolated image is
inverted to aid visualization.
81
Intelligent Automotive Safety Systems: The Third Age Challenge
Body background
separation
Hairs, hand
separation
Face separation
Face feature
separation
Figure 5-4: shows multiple layer thresholding on Infrared image showing a person driving a
vehicle.
From the above segmented binary images it can be seen that different features can be
extracted using subtraction of images.
The most important feature that can be found using this image processing technique
is the face contours separation, as shown in Figure 5-4. Using this segmentation
algorithm the face contours like cheek structure or hairstyle is prominent in this
image (Figure 5-5). Thus face contours of each volunteer infrared image are
extracted. All volunteers are looking forward at that moment.
82
Intelligent Automotive Safety Systems: The Third Age Challenge
Subtraction of suitable image
results indicate cloth covered
skin. hairs and cheeks in above
image
Subtraction of suitable image
results indi cate face boundary
including cheeks and hairs and
hands on steering
Figure 5-5: Subtraction of thresholded images
The use of computer based imaging analysis is a good alternative to visual
identification. Researchers have been using image feature analysis to classify and
identify different groups of objects. For example in food sciences researchers have
classified wheat grains based on their colour and sizes, and frozen pizza toppings
inspection before being packaged. In medicine and laboratory counting and
classifying micro-organisms is carried out based on their features like size and shape.
In the field of transport and vehicle safety the detection of pedestrians and obstacles
using non-stereo and stereo vision in visual and night vision cameras, road curvature
recognition based on road markers, driver assistance systems and autonomous
vehicle navigation have been achieved (Mar et al., 2003 and Bertozzi et al., 2000). In
the field of human behaviour and ergonomics study the tracking of driver or athlete
behaviours using visual markers on their body is a common technique. In the field of
defence and military human and vehicle tracking and recognition, target tracking
uses infrared imagery. The list of work done in the field ofimaging is vast and only a
glance is what is achievable is discussed above.
83
Intelligent Automotive Safety Systems: The Third Age Challenge
5.1.4.
Imaging features extraction
Information in images can be classified using Image processmg features.
Classification of information in images is done by extracting different features from
the image. Details of extracting imaging features are discussed in Section 6.2.1 under
the heading of "Scatter-gram selection method neural network input vectors".
5.1.5.
Image processing analysis tool: ComparelQ2
CompareIQ2 as shown in Figure 5-6 is a standard GUI interface for MatLab for the
I-Quire software (image acquisition software used in the experiment). Further details
for I-Quire are available in Amin (Amin, 2003). CompareIQ2 is used to analyse the
experimental data visually and export certain visual and infrared images in MatLab.
This program is a basic platform for the image processing techniques used. It
consists of two windows showing a visual image and the corresponding Infrared
image.
84
Intelligent Automotive Safety Systems: The Third Age Challenge
Visual
reference
I nfrared/lmage
processing region
Read ex eriment
files and
~ence point in
aV',\Orirnont
Figure 5-6: CompareIQ2 screen capture
This GUI interface for MatLAB has proved to be really useful for running analysis
on hundreds of images through the experiment and analysing their results. The
infrared image or image processing technique applied is placed in that certain plot
and then applied sequentially to all infrared images.
85
Intelligent Automotive Safety Systems: The Third Age Challenge
5.2. Image processing algorithm
5.2.1.
Phase 1: Imaging algorithm initial development
The initial algorithm which was designed had two stages: pre-processing and
processing. Each block of the imaging algorithms is discussed individually (Figure
5-7).
Pra-processing
·:
·:;
·
:", • • • • • • • • • • • • • " . . . . . . . . ., • • • • • • • • • • • • 11 • • • • • • • • • •
.:
Reading
infrared
thermograph.
Interpolation:
:
.
.
. . . . . • • • • • • • • . . . '1 . . . . . ",,, • • • " . . . . . . . . . . . . . . . . . . . . . . . . . . . . " .
Processing
·: Multiple level.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . fII . . . . . . . . . . . . . . . . . . . . ,. . . . . . . . . . . . . . . . . . . . . . . . .,.
Feature
extraction
: Segmentation'
:
••• " .. ",. iri. " ........ " •• '* .....
$
............
.Iit.~
..
Tracking plot
~
:
11*"" 11.11 ................... " ••• " ••
Figure 5-7: Preliminary phase imaging algorithm
Pre processing
Pre processing is an important part of an image processing algorithm. It helps the
image processing algorithm by reducing the noise and applies any other image
enhancing processes. The infrared thermographs which are acquired from the
infrared camera contain negligible amount of noise. This is due to the unconventional
imaging mechanism and the germanium lens.
86
Intelligent Automotive Safety Systems: The Third Age Challenge
Reading the Infrared thermograph
Infrared thermograph acquisition is conducted by usmg a RS-232C serial port
communication. Further details of infrared acquisition can be found in Chapter: 7.
The acquired infrared thermographs are stored on the hard disk. MatLAB reads the
infrared protocol stored on the hard disk by the software and interprets it into a visual
representation of infrared thermographs (see Appendix B).
Interpolation
The raw infrared thermograph is 16x16 pixels. Interpolation is required to get a
better visual representation and larger area to work on. Infrared data interpolation is
done to achieve 128 x 128 pixels image height and width.
Interpolation
IS
further described in detail in the Imaging Techniques section,
Chapter 7.
Processing
The processing of the initial algorithm contains three stages; segmentation of infrared
image, extraction of features from the segmented image and track plot of the driver
using those features. These stages are described in detail in the following
subsections:
87
Intelligent Automotive Safety Systems: The Third Age Challenge
Multi level infrared segmentation
The Infrared thermograph histogram, (shown in Figure 5-8) displays three peaks in
which two peaks are distinct, the large peak on the left shows the background which
has less temperature and is represented in black.
Figure 5-8: Interpolated infrared image with histogram
The other peak which is approximately in the centre and covers a greater area than
the background peak is the uncovered and clothed part of the subject showen in the
infrared image (see Figure 5-9).
88
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 5-9: Body segmented infrared image with histogram, inverted histogram shows
segmented region
The third peak on the extreme right represents the hottest region in the infrared
image. This region is the face; a line is drawn in Figure 5-10 showing the position of
face segmented infrared image.
89
Intelligent Automotive Safety Systems: The Third Age Challenge
Face threshold line
Figure 5-10: Face segmented infrared image with histogram
Feature extraction
After segmentation of the infrared image comes the stage of extracting features. See
section 6.2.1, "Scatter-gram selection method neural network input vectors" for more
detail.
Tracking plot
Software is written in MatLAB that can track movement of the volunteers while
driving as shown in Figure 5-11. The thresholded infrared image is that of a subject
90
Intelligent Automotive Safety Systems: The Third Age Challenge
driving, whereas the plots in the right window show the movement of the head within
the field of view of the infrared imager, which is 355x355 sq millimetres.
100
120
20
Rudlp~
40
I.
60
00
~~I
100
120
20
40
60
00
100
120
45
Figure 5-11: Tracking movement
The centre position of the head is used for plotting coordinates. This can be seen
taking a closer look on plots as in Figure 5-12. Label (A) in Figure 5-12 shows the
ideal position of driving. But when the subject comes onto position (A), he puts on a
seat belt and follows path (B) which is traced in Figure 5-12. During driving the
subject comes to a stop (in this particular scenario), rolls down a window, swipes the
card to simulate barrier exit. This motion can be seen from the plot of path (C) in
Figure 5-12.
91
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 5-12: tracking path, (A) ideal driving position, (8) putting seatbeIt on, (C) rolling window
down.
Plotting other well selected infrared image features will also yield similar results.
5.2.2.
Phase 2: Advanced approach - Angled IR
The previous algorithm has some drawbacks. These drawbacks are listed as follows:
•
Mounting position of the IR camera. It blocks the viewing area of the driver.
•
It is only possible to track the driver, no identification of driver posture.
Therefore a new imaging algorithm was developed to eliminate the deficiencies of
the previous algorithm. Figure 5-13 shows the previous position of the IR imager, in
which the camera was mounted in line with the steering column and the mounting
height was equal to the head height of the driver. This method blocks the driver's
view significantly and was not a viable option, therefore an alternative mounting
position was sought. The angled position as shown in Figure 5-13 is used, as in this
position the IR imager can see the forward movements as well as the sideways
movements.
92
Intelligent Automotive Safety Systems: The Third Age Challenge
1
I
.....1 - - - - - - - 1 meter'------.........I
I
( , - __ I
Previous
mounting positio
oflR Imager
I St.erlng Oe.1
\I ~
J,-'~6'Oo
(Outside vehicle
Driver with
obstructed
view
1
-""1
I
New
nting
position of
IR Imager
(Side Pillar)
Figure 5-13 Previous and new mounting position ofIR Imager
Figure 5-14 shows the new changed algorithm before the use of the neural network.
The initial pre processing part is the same as that of the previous algorithm. In
processing the segmentation is based on a histogram. The further image processes
like infrared image splitting and features extraction is different as the infonnation
will be used in a neural network.
93
Intelligent Automotive Safety Systems: The Third Age Challenge
Pre Processing
r'" - -,. - - '" - -
~-I
I
Infrared
Image ....._ ....
.
Reading
infrared
thermograph
,::..
Infrared
Interpolation
I
I
1. __ . __ . -_._...;
Processing
r----'--------'-----'-----'-----'--:
t
1
.
I
Segmentation
based on
histogram
I
I
Division of
infrared image, ,
into three'
.•. regions" '.
Features
extracted from
region 1
Features
extracted from
"region 2
Features
extracted from
region 3
i
I
'"
J
1_. __ ._-;-_. __ . __ . __ . __ ._---_. __ ._-'
Figure 5-14: advanced approach: Angled IR imaging algorithm
94
Intelligent Automotive Safety Systems: The Third Age Challenge
Pre processing
The pre processing algorithm is self explanatory from Figure 5-14. The detail is
already coved in previous sections 5.1.2 and 5.2.1.
Infrared Interpolation
The infrared images taken from the experiment are interpolated from 16 pixel sq to
121 pixels square. As discussed earlier this interpolation on infrared image does not
actually add anymore information into the system but the only advantage is to have a
larger thermograph area to work on.
Processing
Segmentation
The devised adaptive segmentation method is based on the IR histogram (Figure
5-15). This method will compensate for slight temperature changes.
Now consider the histogram as a function called p(x). By taking the limits the
maxima of the function p(x) can be given as:
p(X),O:s;x:S;40
f'(x) = p(x)
(Equation 6)
Solving for the first derivative of p(x) function give two maxima values. The limit
for the equation will then be the x value of f(x) value. Then the minima intensity
value within the limit of those maxima values is the threshold value for that
particular image. This segmentation technique is calculated for each single infrared
95
Intelligent Automotive Safety Systems: The Third Age Challenge
image. Looking for two prominent peaks and taking the minimum value within those
maxima ranges as depicted in Figure 5-15.
Maxima
line
Maxima
line
I
I
500
1
I
I
400
~
.1$
Co
~
I
1
I
~
300
a;
,Q
In:Jfie~nts
•
.
§
z 200
/
-~~~ lA.J'A
V
100
-f')
0
o
I
//\
i/ \
V
25 I
Minima or
thresholding value
Temperature
(Celc!Us) •
\.11
I
~I
•
40
Figure 5-15: Determination of segmentation value
Infrared Image splitting
After interpolation and segmentation, the infrared thermograph is divided into three
regIOns.
The infrared imager is 1 metre away at an angle of 60 degrees to the front left pillar
of the car, see section 7.3. The field of view is 355 millimetre square. Each division
of the thermograph image is based on mainly the three different region of the
occupant (refer Figure 5-16). These are
96
Intelligent Automotive Safety Systems: The Third Age Challenge
•
RI region is the background mostly with rare occurrences of head, shoulders
and hands while driving. The region cover the RI is 178 millimetres by 205
millimetres into 61 pixels by 80 pixels.
•
R2 region is the face or head region. This region will allow focusing only on
head movement and its temperature. The head region also covers 178 millimetres
by 205 millimetres into 61 pixels by 80 pixels.
•
R3 region covers shoulder, ann and hand movements. This region is the
region with most movements whenever driver changes posture or moves slightly.
This section is the lower section of the infrared image which is converted from
152 millimetres by 355 millimetres field of view to 121 pixels by 41 pixels.
R1
R2
R3
Interpolated
ther...... ~'....'.,."'h
Thermograph divided into three
rDl"linn<! R 1. R2 and R3
Figure 5-16 Region allocation of infrared image
Image feature selection
Selection of features from the image is the most vital step for the imaging algorithm
to give useful and accurate results. Therefore a great deal of care is to be taken while
97
Intelligent Automotive Safety Systems: The Third Age Challenge
selecting them. A procedure is devised to find the appropriate feature for neural
network input (see Figure 5-17).
Image processing block
..
....
1-----'-----'--'-----'-----'-----'1
I
Division of 1
.
...
Infrared Image
: ..
._
...
_
.- . - _ - _ - . - _ - _ - . - _
. ""_
infrared image i
into three. !
regions
I
Segmentation
of body' region
f
__
•
__
,
__
•
_ _ .. _ _
*
,--'''''''''''''- --*--*-_.- .-_. __ . __ .
I
Specifications of
driver posture
list which is
being detected
Features
i extracted from
! . region 1
Features
extracted from
region·2
_. __
.1
----*1
Features
extracted from
region 3
c
o
~
ID
-a;
List of features narrowed down by comparing
different infrared images
Cl)
...
.aca
ID
ID
U.
Plotting of features against each other
(using scatter~gram. graphs)
Decision curves drawn and graphs compared
Selected
features from
. region 1
Selected
features from
region 2
Selected
features from
region 3
.
1__ - _. __ • __ • __ . __ • __ . __ •__ , __ • __ ._'
Figure 5-17: Feature selection process
The driver performs the majority of his movements by the upper half of the body.
Therefore the infrared imager is also focused to monitor the upper body half
movements. In future additional information on lower body (feet) can easily be
obtained from pedal sensors like brake, throttle and clutch. Those movements are
head turning, arm movements, torso movements and different combinations of them.
98
Intelligent Automotive Safety Systems: The Third Age Challenge
As the infrared image is divided into three different regions each region is dealt with
separately as far as features selection and recognition is concerned.
1. 'Head region' is focused on head turning movement; this region will find
where the driver is looking (refer Figure 5-18). The three movements for
which the network will be trained are looking left, looking right and looking
ahead (Le. on road).
Figure 5-18: 'Head region' shown in interpolated infrared image
99
Intelligent Automotive Safety Systems: The Third Age Challenge
2. 'Torso region' is indirectly linked to torso movement as it will monitor head
and neck movement in back and forth action (refer Figure 5-19). The three
postures that will be defined in the neural network are no-leaning posture,
leaning posture and looking down posture.
Figure 5-19: 'Torso region' shown in interpolated infrared image
100
Intelligent Automotive Safety Systems: The Third Age Challenge
3. 'Shoulder and arm region' shows two postures that will be trained for the
neural network. There are hands-on steering and hands-off steering wheel
(refer Figure 5-20).
Figure 5-20: 'Shoulder and arm region' shown in interpolated infrared image
A total of 18 different driver postures have been identified from these three regions.
Neural networks will be trained to uniquely identify these postures from thermal
images.
The shoulder and arm region can show if the car is stationary or in a moving state
based on the assumption that driver's hands will be on the steering wheel when he is
driving the vehicle and vice versa. Bigger angle germanium lens can be used to get a
larger FOV to get lower part of steering wheel in focus. The other two infrared image
regions can identify the position of the driver. Detailed discussion of what can be
achieved can be found in the results and discussion chapter 9.
101
Intelligent Automotive Safety Systems: The Third Age Challenge
Table 5-1 gives the idea of what region is for what type of posture movement
detection.
Infrared cropped region
Posture movement
Head region
Turning head
(Left, right and ahead)
Torso region
Back and forth movement
(Straight torso, Leaning and Looking
down)
Shoulder and arm region
Hand on-steering and hands off-steering
Table 5-1: Posture and cropped region
P-code description
For neural network results to be easily readable a numerical value for each region is
allocated which points to a particular type of posture. These numerical values are
then stored in a look-up a table. This look-up table is then used to make up a letter
code which indicates what the driver is doing (see Table 5-2).
102
-
------------------------------
Intelligent Automotive Safety Systems: The Third Age Challenge
ANN Numerical
Posture
P-
output
description
code
RI
I
upright posture
N
RI
2
Leaning
E
RI
3
Looking Down
D
R2
I
Looking Ahead
F
R2
2
Looking Left
L
R2
3
Looking Right
R
R3
I
Hands on Steering
S
R3
2
Hands not on
NS
Region
Steering
Table 5-2: Posture Code
So all three region codes are combined to describe a certain posture for example
N-R-S means upright, looking right with hands on the steering wheel, which is a
posture, for example, if someone is at a roundabout (see later the comparison with
real video data).
Again D-L-NS means looking down on the left side with hands not on the steering
wheel, which means that the driver might be putting the seat belt on or doing
something other than driving.
103
Intelligent Automotive Safety Systems: The Third Age Challenge
5.3. Summary
•
Differences in visual and IR thermographs are compared. Visual images
consist of light intensity values whereas IR thermographs consists of actual
temperatures.
•
Phase 1: Initial imaging algorithm consists of two sections, pre-processing
and processing. Pre-processing includes reading of IR thermographs and their
interpolation.
•
Phase 1 processing includes three processes; segmentation, feature extraction
and tracking. The segmentation conducted is based on the temperature range.
After segmentation, imaging features (centroid, centre X and centre Y) are
extracted which are plotted as a graph. This graph is then used for tracking
movement of the driver.
•
Phase 2: Advanced approach - Angled IR algorithm includes two sections,
pre-processing which is similar to that of the Phase 1 algorithm.
•
Phase 2 processing section, which includes segmentation based on a
thermograph histogram. Later the IR thermograph is split into three image
sections. For each image section image features are extracted. The selection of
features are based on the type of neural network and discussed in the next
chapter.
104
Intelligent Automotive Safety Systems: The Third Age Challenge
6. Artificial Neural Network
Artificial intelligence (AI) is defined as synthetic intelligence by a system which is
generally assumed to be a computer. Artificial intelligence is a well researched area
in the field of computing. There are two branches of AI, Conventional AI and
Computational Intelligence (Cl).
Cl processes include development, learning or training. The most well-known
computational intelligence processes are:
•
Fuzzy logic
•
Artificial neural networks (ANN)
•
Evolutionary or genetic computation
•
Hybrid intelligence networks
6.1. Comparison of fuzzy logic and artificial neural
network
Strengths and weaknesses of fuzzy logic and artificial neural networks are compared
to decide which type of Cl is ideal for imaging processes used in this thesis. Fuzzy
logic is usually used for reasoning with condition of uncertainty. A Fuzzy logic
output decision is based on a continuum of possibilities and embodies the alternative
"maybe" to determine the probability of inclusion in a certain membership class
105
Intelligent Automotive Safety Systems: The Third Age Challenge
whereas neural networks are used for complex decision making. Fuzzy logic decision
curves are mostly linear which is not ideal for differentiating feature recognition.
Pattern recognition is the main strength of ANN. For image processing there are
specialised learning networks that perform better than other neural network types.
Certain ANN can take data without training and classify them appropriately, such as
self organized map.
For some control systems and applications fuzzy logic is preferred and for others an
artificial neural network is used. Artificial neural networks are preferred for imaging
applications and pattern recognition (Dubey et al., 2006) and are used in this thesis.
6.2. Types of neural network under consideration
Complex tasks are performed by neural networks, for example forecasting earnings,
stock trading, fraud detection, hand writing recognition, speech recognition and other
complex decisions. There are several types of neural networks but in this thesis only
three types of networks are compared with each other for ideal results. These neural
networks are the most widely used and are as follows:
1. Back propagation neural network or Feed forward neural network (BPN)
2. Radial based neural network (RBN)
3. Self organized map (SOM)
Each neural network is designed for different purposes. These selected neural
networks are the most common types that are used and known for their pattern
recognition capabilities (AI-Habaibeh et al., 2003, Haykin, 1999).
A scatter-gram is a graph used in statistics to visually display and relate two or more
quantitative variables. In image feature processing using scatter-grams to visually
106
Intelligent Automotive Safety Systems: The Third Age Challenge
relate features is a widespread method that is done before using any kind of decision
making method. Infrared thermograph features are extracted and plotted using
scatter-grams, which allow the selection of features as the inputs of neural networks,
visually before creating the neural network.
6.2.1.
Scatter-gram
selection
method
for
neural
network input vectors
Infrared images were taken of two subjects (male and female). Several segmented
infrared images from each subjects experiment database were selected applied to
eighteen different postures that are required by the neural network to be trained. Each
image is cropped into three regions as described in section 5.2.2. For each posture a
minimum of ten region infrared images and maximum of thirty region infrared
images were taken depending upon the frequency and importance of the driving task.
These blocks of infrared images are then analysed using the image processing
software MATLAB to find the features. The features that are extracted from the
region block are as follows:
•
Area
Find the binary image area in pixel squares.
•
Aspect ratio
Give the aspect ratio of the image. The aspect ratio of an image is its displayed width
(horizontal) divided by its displayed height (vertical).
•
Object's area and bounding box ratio
This features is the ratio of 'area of the object' and 'area of the bounding box'
•
Box width and height ratio
Box XfY is the ratio of the bounding box width (X) and height (Y).
107
Intelligent Automotive Safety Systems: The Third Age Challenge
•
Centre-X
This feature takes the x-coordinate of the centroid of the object.
•
Centre-Y
This feature takes the y-coordinate of the centroid of the object.
•
Density (mean)
Reports mean intensity or density of the object.
•
Angle
This feature reports the angle between the vertical axis and the major axis of the
ellipse equivalent.
•
Holes
Finds and reports the number of holes.
•
Hole Area
This feature reports the area of the hole within an object.
•
Hole Ratio
Hole ratio is determined by Area / (Area + Hole).
•
Major and minor axis
Reports the length of the major axis and minor axis of the ellipse equivalent.
•
Diameter (mean, max and min)
This feature gives the minimum, maximum or mean of the outline points and passing
through the centroid of that object.
108
Intelligent Automotive Safety Systems: The Third Age Challenge
•
Radius (max and min)
Finds and report the minimum and maximum distance from the centroid to the object
perimeter.
•
Perimeter
Gives the outline length of the object, this includes the holes outline length.
•
Radius Ratio
Radius ratio is calculated by minimum radius divided by maximum radius.
•
Roundness
Roundness is calculated by using the formula
Perimeter / (2 x pi x Area)
•
Box width and box height
This gives the object bounding box width and box height.
The scatter-grams (as shown in Figure 6-1) are plotted for features that are listed
above. The values from the thermograph features are plotted to show different groups
visually. For example the Figure 6-1 shows area feature plotted against number of
samples and shows grouping of different driver postures for that particular region.
Only a few of the above features showed distinguishable results that can be used for
pattern recognition in a neural network.
109
Intelligent Automotive Safety Systems: The Third Age Challenge
4500
4000
3500
3000
•••
•••
•
Looking Ahead
~
2500
Cl)
<c
2000
1500
1000
500
0
Number of samples
Figure 6-1: Area scatter-gram of head region
The overlapped features in scatter-grams represent when trained in the neural
network will result in inaccurate result (see Figure 6-2). It can be said with
confidence that a neural network trained with data from Figure 6-1 will yield better
accuracy than the neural network trained with Figure 6-2. This will be true for all
kinds of networks as there is no way of distinguishing between the mixed or
overlapped data.
110
Intelligent Automotive Safety Systems: The Third Age Challenge
••••
0.9
0.8
x
0.7
Looklng Ahead
o
c:c
1:)
c
0.6
:a
5: 0.5
o
c:c
)
0.4
0.3
0.2
0.1
O~-------------------------------------------------J
Number of samples
Figure 6-2: Area/Bounding box scatter-gram of head region
The features used as an input in the neural networks are shown in the following
Table 6-1 (see Figure 5-16):
Infrared cropped region
Selected features
Head region
Angle (Figure 6-3) and Segmented Area
(Figure 6-4)
Torso region
Area (Figure 6-5)
Shoulder and arm region
Area (Figure 6-6)
Table 6-1: Selected features for neural network
111
Intelligent Automotive Safety Systems: The Third Age Challenge
Head Region
60
40
20
C)
C,
~
0
-20
Looking Left
·40
-60
Number of samples
Figure 6-3: Head region angle scatter-gram
Head Region
5ooo.---------------------------------------------~
1000
500
O~----------------------------------------------~
Number of samples
Figure 6-4: Head region area scatter-gram
112
Intelligent Automotive Safety Systems: The Third Age Challenge
Slight overlapping of scatter gram data as in Figure 6-4 i.e. looking left and looking
ahead is not a major issue because the safety system's infrared imager frequency is 4
FPS and therefore the system is analysing 4 thermographs every second. If one
thermograph of looking left is in the looking ahead region the subsequent three may
be in the looking left region.
Figure 6-3 to Figure 6-6 show the scatter-gram for actual training set generated for
the subject. The data is ready for training in a neural network.
Torso Region
~~--------------------------------------------.
4500
• ••
•• •••• •
••
\.
4000
...• ••••
3500
3000
~
•••
•• •
2500
•••••••
••
• ••
•• t/.
~...
• • •••• ••
•
L----.. . . . ._~ ... • +leating
Number of samples
Figure 6-5: Torso region area scatter-gram
113
Intelligent Automotive Safety Systems: The Third Age Challenge
Shoulder & Arm Region
J2500
2000
1500
1000
500
o~--------------------------------------------~
Humber of samples
Figure 6-6: Shoulder and Arm region area scatter-gram
6.3. Neural network designs
Designing a neural network require several parameters to be considered. The types of
inputs required, how many input vectors, number of neurons in a hidden layer if it is
a Feed-Forward Back Propagation Neural Network (FFB), number of hidden layers
if it is a FFB network, number of outputs, how the outputs will represent the results.
There is no certain formula for setting up these parameters but neural network design
optimization is more of an iterative process.
Before starting the design of a neural network the output of the network should be
defined. There are eighteen different positions that needed to be defined by this
neural network. The output can be a single output if a single neural network with
linear function is used. It can also be multiple outputs but then it needed to show how
these eighteen different postures are defined in a mUltiple output neural network.
114
Intelligent Automotive Safety Systems: The Third Age Challenge
6.4. Single neural network designs
6.4.1.
Large input single FFB neural network design
A single binary array was used as a neural network input for the initial neural
network design. This binary array was equal to the number of pixels in the raw
infrared image which is 256. The segmented infrared image is used as an input for
the network. For number of inputs equal to 256 the hidden layer needs to be carefully
laid out. It could be a single hidden layer or several hidden layers. There will be at
least 350 neurons in the network, which is a large network to work with. The training
time would be significantly large for multi layered and self-organized networks and
the network would take significant memory. This type of network design is not
preferred see Figure 6-7.
Segmented infrared
image
ANN Input vector
ANN Hidden layer
Figure 6-7: Segmented image input into multi layered neural network
115
Intelligent Automotive Safety Systems: The Third Age Challenge
6.4.2.
Feature based single
FFB
neural
network
design
The other method is to identify features from the infrared image and use them as an
input for the neural networks. This network design will consume less memory and
CPU power. It will be faster in training and simulation than the previous network
design which is much larger. The features need to be carefully selected, for a six (6)
feature neural network design there will be approximately 15 neurons in a network
(see Figure 6-8).
Area
Angle
Area/Box
Perimetor
Roundness
Center-X
AN N Hidden Layer
Figure 6-8: Features based input into multi layered neural network
116
Intelligent Automotive Safety Systems: The Third Age Challenge
The output needs to be linear as there are eighteen different outputs required by a
single network. The network requires extensive and large training sets to be trained
on. Eighteen different outputs from a single output neuron is to achieve with average
accuracy.
6.4.3.
Feature based Radial Neural network design
The advantages of Radial based function (RBF) neural network design is that they
are constructed in a fraction of the time that it takes linear based neural networks like
FFB to train. The training of a Radial based network (RBN) is done when it is
constructed. On the other hand a RBN takes a lot of memory due to the number of
neurons. The input of the RBN is the same as that of a feature based FBN network
(see Figure 6-9). Features are extracted from the IR imagery. The output neuron is
setup as a linear function to identify eighteen different driving postures.
Area
Angle
RBN
ArealBox
Network
Perimeter
ANN Output
Layor
Roundness
Ccnter·X
ANN Input
vector
Figure 6-9: Feature based radial network design
117
Intelligent Automotive Safety Systems: The Third Age Challenge
6.4.4.
Self organized map network design
Self organised map network is unsupervised learned design. For detecting eighteen
different driver postures self-organized map type network gives various results thus
this type of design is ruled out.
6.5. Three Neural Network design: TNN
This novel neural network design is based on a principle of combining small neural
networks together. This design consists of three neural networks; each neural
network is related to region features as shown in Figure 6-10 (see section 5.2.2 for
explanation on image regions and section 6.2.1 for region features). The results from
all three neural networks are combined together to form a posture code (p-code) (see
section 5.2.2 for p-code details).
118
Intelligent Automotive Safety Systems: The Third Age Challenge
N
Output
Layer
ANN
Layer
Output
Layer
ANN
Hidden
Layer
Output
Layer
ANN
Figure 6-10: Novel neural network design: TNN
119
Intelligent Automotive Safety Systems: The Third Age Challenge
6.5.1.
Three neural network design evaluation
Neural networks for each of the three regions as shown in Figure 6-10 are separate.
This means that there will be three neural networks working simultaneously on a
single thennograph. Furthennore for each region three different types of neural
networks are constructed. They are FFB, RBN and SOM networks. Comparison and
evaluation of all three networks results in finding the best neural network for a
particular region. These types are selected because they have good ability to
differentiate between different parameters. The construction parameters of all nine
neural networks are listed in Table 6-2).
Inputs
Region 1 'RI'
Region 2 'R2'
Region 3 'R3'
1
2
1
(area RI)
(area/angle R2 & area (area R3)
R2)
Target
1 neuron,
I neuron,
1 neuron,
outputs
3 output values
3 output values
2 output values
FFB
Layers 2
Layers 2
Layers 2
Neurons 2
Neurons 5
Neurons 2
Inner Layer: Sigmoid Inner Layer: Sigmoid Inner Layer: Sigmoid
function
function
function
Output Layer: Linear Output Layer: Linear Output Layer: Linear
RBN
function
function
function
Goal: 0.001
Goal: 0.001
Goal: 0.001
Spread constant: 1
Spread constant: 1
Spread constant: 1
Inner Layer: Radial Inner
function
Layer:
Radial Inner Layer:
function
Radial
function
Output Layer: Linear Output Layer: Linear Output Layer: Linear
SOM
function
function
function
Goal: 0
Goal: 0
Goal: 0
Epochs: 25
Epochs: 25
Epochs: 25
Table 6-2 Neural network specifications
120
Intelligent Automotive Safety Systems: The Third Age Challenge
Thus training data is selected to cover all types of motions and the frequency of
motion. The training data for each volunteer driver is three hundred (300) samples
for each region neural network. The simulation data is three times the size of the
training set which is nine hundred (900) samples for each subject.
6.5.2.
Selection and training of the three neural
network design
The actual ANN results are plotted in section 8.2. The different postures are linked
with a numerical postural code. Each numerical value represents a particular posture.
The closer the actual result plot is to the posture code the more accurate is the
detection.
Only one type of ANN is required for each region. For selection of the most
appropriate ANN from FFB, RBN and SOM two volunteer drivers were randomly
selected.
The SOM network gives a range of values. By making the decision lines on the
output graph of the SOM, a posture code can be found.
The RBN network is moderately accurate but gives unexpected results sometimes.
Also it can be seen from the results (Figure 6-11 and Figure 6-12) that looking ahead
is one of the principle tasks that a driver performs. The error is found in the looking
right posture. This error affects the final decision of the posture detection algorithm
as there are 3 different regions, which are combined together to get the final result.
The posture detection algorithm will be running at 4 FPS, i.e. analysing four (4)
thermographs each second. If one thermograph ANN output result shows
abnormality or inaccuracy the remaining three (3) thermographs may be able to
remove the abnormality or inaccuracy_
121
Intelligent Automotive Safety Systems: The Third Age Challenge
The graphs shown in Figure 6-11, Figure 6-12 and Figure 6-13 are the results of
compared neural networks. Each posture code section in the graphs shows three
neural networks. These neural network results can be Zero (0), One (1) or Two (2)
with the exception of the SOM neural network as these networks are self trained and
thus their outputs are uncontrolled. If the result is as expected, i.e. ideal, the resulting
plot will be Zero (0). The results can deviate from Zero (0) up to Two (2), this shows
the error. This does not mean that the neural network is inaccurate based on a single
plot. If a certain neural network deviates from the Zero (0) line this means that
particular network is not suitable for the feature recognition task. It is generally seen
that FFB neural networks are most suitable for feature recognition tasks as they have
previously been trained using supervised learning and contain a linear transfer
function.
122
Intelligent Automotive Safety Systems: The Third Age Challenge
Torso region ANN results
'~'""~~'~"r~""""""'-""-r-"~-'''-r--'~''~--''f'"''~-"-'-'"'T"---J"'-"r""'-=
I
I
FfS
-RBN
--'r-SOM
......................................l .............. ~l ..............
I
I
~.....................................1~ .............. :~ ........ -.... .
I
....... _..................... _._ ... _~.l __ ... __ ....... !t .. .
I
I
I
I
f
·····································t···_·········· 1....
I
I
Posture Code:3
100
Number of simulated samples
150
200
Figure 6-11 ANN comparison result for 'torso' region
The radial basis network gives unreliable results. The self organized map is reliable
except for some data it was unable to process and distinguish and starts hence to give
random values. Therefore a neural network based on this FFB network is strongly
recommended in the second region case.
The FFB network gIves a consistent performance with error percentage not
significantly high.
123
Intelligent Automotive Safety Systems: The Third Age Challenge
Head region ANN results
I
..........Ij .............. .
I
.... _...!.l .......... .
I
I
I
I
I
I
I
I
I
2
······················!··~··············!l···
·······················,···············1
I
I
PCGtUfe Code:3
PootUfe COde:1
100
Number of simulated samples
150
200
Figure 6-12 ANN comparison result for 'head' region
The SOM performed well in the first subject third region but this was not the case
with the second subject (see Figure 6-13), As both cases varied in amount of skin
covered and movement behaviour thus making human posture behaviours
unpredictable, thus the FFB network is the preferred choice for the third region.
124
Intelligent Automotive Safety Systems: The Third Age Challenge
Arm and shoulder region
ANN results .
--FrB
-'l!-RIm
-'V-
SOM
........ ,........................................t-.......................................... .
I
I
1
--- .. ---.---- .. ---.- .. -- .. ).~.------ .. -- .. -- .. --.--.--------I
I
0
~~~~~~~~~~~~~~~~~~~~~~~~~~
I
I
~
.. -... -........... -... -... --.... ~.!.~.
I
I
---------------------.--------... --1-------.. --------------.---.--.. -
I
I
Posture Code:1
Posture Code:2
100
200
Number of simulated samples
Figure 6-13 ANN comparison result for 'arm and shoulder' region
The 'Arm and shoulder' region is easier because only two outputs are required from
a single stream of input data. Thus all networks performed significantly well, with
the SOM and FBN network achieving an accuracy of 100% on all samples. Thus
making FBN network preferred choice being training based neural network.
The neural network for each subject will be trained individually. Therefore the
system can detect the same posture with similar accuracy for all SUbjects. This means
that a person with long hair or short hair will not have errors in the safety system
depending upon their different features. Even though FFB and SOM networks both
125
Intelligent Automotive Safety Systems: The Third Age Challenge
gave accurate results the preferred choice of network in this case is also FFB. The
FFB network is more stable as it was trained on sample data previously. SOM also
gave out random results with unexpected data this kind of behaviour is not expected
from FFB networks.
6.6. Summary
•
Using related literature artificial intelligence techniques like fuzzy logic are
compared with the artificial neural network (ANN). ANN is preferred as these
networks are a more flexible choice than fuzzy logic. An ANN can be trained
over and over again.
•
Three types of neural network are considered and compared. These networks
are FFB, RBN and SOM.
•
Initially the neural network designs are single network based. Several neural
networks designs are shown.
•
The first design shows the input of a large array of pixels in an FFB neural
network with a single output. The second design shows a single FFB neural
network with infrared thermograph features as an input. The other designs were
similar and based on RBN and SOM networks.
•
Finally a novel design of three FFB network design working on split infrared
thermographs is described.
•
Training of the Novel ANN design is described. Also compared are FFB,
RBN and SOM for the novel ANN designed. FFB was the preferred choice for all
three neural networks.
126
Intelligent Automotive Safety Systems: The Third Age Challenge
7. Experimental Setup
7.1. Introduction
This chapter evaluates the safety system; which was designed and developed in
previous two chapters by experimentation. It gives a detailed account of how the
experimentation was conducted.
7.2. Experiment 1: Trials with Infrared camera pointed
from the windscreen
7.2.1.
Aim of the experiment
This was the first experiment conducted for this research. The aim of this initial
experiment was to find the capabilities and limitations of the low resolution IR
imager in a driving environment, i.e. how well a low resolution IRISYS imager can
track a driver's movements. Another result·that is required from the experiment is to
find the accuracy of the infrared imager in terms of measuring driver movements.
A secondary aim of the experiment conducted was to see whether the IRISYS
thermal imager could be used as a vehicle occupant identification sensor. For more
details on this see Appendix C.
127
Intelligent Automotive Safety Systems: The Third Age Challenge
7.2.2.
Driving simulator
The driving simulator was installed in an ergonomics test facility with a STISIM
driving simulator which is discussed in Section 3.7. The driving simulator was built
on a static Ford Scorpio with front projection of 3 metres by 2.5 metres shown in
Figure 7-1. The control room consists of a driver communication system and the
STISIM control computer. The driver is kept in contact via the driver communication
system during the length of the experiment. The data acquisition system was
standalone and will be discussed in detail in Section 7.2.4.
Figure 7-1 (A) Shows car simulator running scenario and Infrared camera, (B) Simulation
Control
128
Intelligent Automotive Safety Systems: The Third Age Challenge
The STISIM system
An overview of the STISIM platform is discussed in section 3.7. In this section
interfacing and programming of scenarios is discussed briefly.
The main controller for the STISIM driving simulator is an IBM PC.
The STISIM simulator hardware requirements are as follows:
•
IBM PC which is 80486 equivalent or better
•
At least 4MB RAM
•
Hard Disk Drive
•
Parallel Port
•
34020 based TIGA graphics board capable of 1024x786 resolution
•
VGA graphics card
•
Digital 110 and AID interface card, CIO-DAS08/JR series from Measurement
computing Inc used.
•
Sound card
•
STISIM Hardware protection key
Figure 7-2 shows the interfacing of sensors with the STISIM driving simulator
controller.
129
Intelligent Automotive Safety Systems: The Third Age Challenge
Steering
Controller
Pedal controller
Display for
Simulator
Projector
Controller's
Display
08-9 or 08-37
Display
Intenace
Input controller card
IBM PC
Figure 7-2: STISIM hardware interfacing
The STISIM driving simulator is programmed using the MS-DOS operating system.
The simulator is designed such that it provides the driver with a very realistic driving
experience, using visual display and audio sounds as feedback to driver actions. For
example crashing a car shows visual as well as audio display to the driver for the
crash scenario.
Three input files are used to vary the scenario elements during the simulation run. All
these files are in ACSII format and can be changed in text editor. These files are as
follows:
130
Intelligent Automotive Safety Systems: The Third Age Challenge
•
STISIM.COL
This file defines the colour displays. The colours defined are for the objects that
are displayed on the roadway scene in the simulation.
•
GAINS FILE
This file is the primary configuration file of STISIM simulator. Parameters in this
file influence the vehicle dynamics, visual display, transport delay, vehicle
handling characteristics.
Vehicle dynamics can be divided into steering, speed control and transmission.
Figure 7-3 shows the steering control of the simulator.
I
Steering angle
a....,;_~_...
<
•
*
. Steenng
activity
in
o
3
3
~
Steering input 1-,_
... _ _......a....,;_~
(Yaw rate/Steering) .::
(1 +(speedxUndersleer cOen
Curvature
error
Vehicle path
curvature
Speed
Figure 7-3: Steering control for STISIM simulator
Figure 7-4 shows the speed control and parameters that influence the speed.
131
.-----------------------------------------------------------------------------Intelligent Automotive Safety Systems: The Third Age Challenge
Throttle acceleration
Throttle
input
Command
' ~ acceleration
Rolling engine
acceleration
. . rl
~t
Brake acceleration
Brake
input
......
Speed
Acceleration
limits
...
".
..
Figure 7-4: Speed control for STISIM simulator
Figure 7-5 shows the speed control with transmission and engme revolution
consideration.
Engine
Speed
..
Transmission
Gear ratio'
i
...
RPM
Smoothing
....
,
Figure 7-5: Speed control with transmission consideration for STISIM Simulator
•
EVENTS FILE
This file, as the name suggests, will describe all the events that will occur during the
simulation run. The events in the file are user definable and there are several
performance measures that can be selected. It is possible to activate several events at
the same time, for example creating an intersection, crossing of pedestrians,
oncoming vehicles and cross traffic. This can be done by using SDL (Scenario define
language) command language. The SDL command language contains two parts, the
first part defines the events and the second part collects data for which performance
measures are selected. It also uses pre-defined events and sub-routines to simplify the
programming procedure.
132
Intelligent Automotive Safety Systems: The Third Age Challenge
The events part of the file follows the general format as:
ON DISTANCE, EVENT SPECIFIER, PARAMETER 1, ... , PARAMETER N,
COMMENTS
The following 'event specifiers' are the most commonly used during the
programming of the event file:
•
•
•
•
•
•
•
A:
Vehicle ahead of subject, from other lane
BLCK:
Display blocks on the display screen
BSAV:
Starts saving dynamics data
ESAV:
Ends saving dynamics data
C:
Add curvature to the roadway
CT:
Cross traffic at intersection
CV:
Control vehicle automatically
•
DL:
Double lane change
•
DI:
Digital input event
•
•
•
•
•
•
•
•
DO:
Digital output event
ES:
End simulation
I:
Display intersection
lA:
Display intersection sign
LS:
Speed limit change
PDE:
Previously defined event
PED:
Pedestrian display
ROAD:
Displays a specific roadway
•
SA:
Displays traffic signal sign
•
SL:
Traffic signal light
The second part of the event file can record performance parameters. Performance
measures used and recorded in the experimental runs are as follows:
133
Intelligent Automotive Safety Systems: The Third Age Challenge
•
Time
•
Speed
•
Vehicle curvature
•
Road curvature
•
Distance travelled
•
Steering wheel angle
•
Throttle acceleration - deceleration
•
Braking acceleration - deceleration
Scenario
The scenario selected for the experiment was urban busy traffic which lasted for 20
minutes approximately. An urban traffic scene was selected because the number of
tasks during driving is much higher than that of motorway driving. The scenario was
created with traffic signals and five (5) intersections at random intervals, see Figure
7-6. The pedestrian crossing is also taken into consideration to simulate real driving
behaviour. Before the experiment the volunteers were instructed about the various
driving tasks, such as intersections, overtaking, lane changes, pedestrian crossings,
looking left or right before making a turn or arriving at the cross road and waiting for
a signal, lane changing manoeuvres, looking at side and rear view mirrors. This was
to ensure that the experiment conducted would be as realistic as possible. For
example as shown in Figure 7-6 the volunteer drives up to the red traffic signal. The
diver has to consider pedestrians crossing; looking left and right after the traffic
signal turns green. Further tasks conducted during the scenario run include putting on
134
Intelligent Automotive Safety Systems: The Third Age Challenge
the seat belt, adjusting mirrors, looking in the rear view mirror, looking left or right,
using a swipe card to simulate entrance or exit of secured car park, mobile phone
usage while driving and using an in-car stereo system or climate control.
Car under simulation stopping at
intersection
Figure 7-6 Bird's eye view of the intersection scenario
The simulation room was kept in darkness to simulate night time and the working of
IR imager in night time. The IR imager exhibited no major change in light or dark
surroundings.
135
Intelligent Automotive Safety Systems: The Third Age Challenge
7.2.3.
Reference sensors
The experiment included two reference sensors.
1. Visual camera
2. Talley pressure Matrix
The primary sensor for the safety system is the IRISYS Imager. A visual camera and
the Talley pressure monitor acted as the reference aids during the experiment. The
webcam and pressure monitor were mounted to find the position of the human
subject and to check the approximate distance of the movements detected by the
thermal imager. The IRISYS thermal imager and web cam were mounted together,
looking from 1 metre away from the subject from the front. As this was an initial
experiment the position of the infrared camera and web cam did not make a major
difference, as the aim was to find the capabilities of the infrared imager.
The pressure mat was placed on the sitting area of the subject for which the pressure
is measured for each air pocket (see section 3.5).
Visual Camera
The visual camera used for the experiment was an entry level CMOS webcam by
Logitech® (see Figure 7-7). The idea was to acquire the visual image and IR
thermograph simultaneously. The camera was mounted directly above the IRISYS
Imager therefore sharing the same FOV.
136
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 7-7: Logitech® Quickcam® Messenger
Talley Pressure Matrix
The Talley pressure matrix was used to help find the position of the driver with
reference to the driving seat. This can also help find the drivers weight and can tell if
the driver is leaning. This pressure monitor is connected to the RS-232C port of the
data acquisition system.
Two data sets from the Talley pressure monitor were taken, the first set at the
beginning of the experiment and another at the end of the experiment. The Talley
pressure monitor takes data by inflating and deflating the air pockets (see section
3.5); this process takes up to a minute, which is relatively slow for real time
processing. This counted as a disadvantage of this sensor and it was excluded from
the later experimental stages because of this slow response timing of the sensor.
Other means like visual camera and manual height and weight readings were taken in
later experiments.
137
Intelligent Automotive Safety Systems: The Third Age Challenge
7.2.4.
Data Acquisition platform
The Data acquisition system was designed based on the experiments that were to
conducted in this research programme. It involves consideration of hardware
interfacing of devices with the acquisition system, software for organizing and
collection of data on disk drives and understanding of IR imager and visual camera
protocols. Figure 7-8 shows the data acquisition platform schematic.
JL---'\J
ca
:::>
\.--_..,~ Cl)
IBM PC
U
N
('I')
N
I
Cl)
Cl::
ActiveX I WDM
Technolo
IRISYS IR Protocol
Visual Bmp image stored
on HDD
IR data stored in ASCII
format on HDD
Figure 7-8: Data acquisition schematic
The following sections will explore each section in more detail.
138
Intelligent Automotive Safety Systems: The Third Age Challenge
ANSI C GUI based Software Development
The software supplied by the IRISYS IRIlO02 thermal imager only measures and
records certain pixels at one time. Therefore online thermal imaging software which
is supplied by IRISYS is very limited, see Figure 7-9. To display and record the
complete thermograph the author had to develop his own software.
Figure 7-9: IRISYS thermal imaging software
The author had previously developed an IRISYS and visual data acquisition system
for automated people counting (refer Figure 7-10) (Amin, 2003). The software
developed was required to deliver data acquisition for small time period for infrared
and visual data. The time interval is not critical between each frame.
139
Intelligent Automotive Safety Systems: The Third Age Challenge
}j~1
~... t.in
I
~<I""-.
f<"~'"~'
<.
g"'" "S:.li
.
I. .,.y~.$tot,
...,,~'''''-.
~:~:=:II'
r; r.~h,iIId{.:''''''''''9f
ro..~ .. ~'·
("~_'/I-
Infrared Image
Web cam Image
Figure 7-10: Previously developed data acquisition software
Significant modifications were done to the existing software to fulfil the
requirements of the experiments (see Figure 7-11).
140
Intelligent Automotive Safety Systems: The Third Age Challenge
...
Videovr"""",
Device ConIlob
~~Ied Oevice
I
~,..tP;~
I
Infrared
Image
!'AUSE)
Time Jot",'a
No, of capl\Jlet
Visual
Image
~~
fittf,~
A5a,-,
BltlARY-
!RI~~J!!~,!,!,j
IS.1S"I'"
32.326h6412Ih129-
,
JI
I
. .
I~~:,:,: '~ (~_downl.".J
• 2dOctF~1
,Q:.,eteOat·1
~~p..~
l.....intent1
Quit
m,1
Figure 7-11: Modified data acquisition software
The following is the list of major modifications that were made:
•
Unlimited acquisition of both infrared and visual data. No time limit.
•
Time interval is decreased, the software can now acquire up to 4 FPS.
•
Previously the software was unstable. Significant changes to the hardware
interfacing code were done to make the software more stable and useable on
different operating systems. Compatible operating systems included Microsoft
windows 2000 and XP.
•
File naming and storing was restructured making data acquisition more
organisable. This is required as vast amounts of data would be stored during the
experiment runs. Therefore batch renaming and storing data in folders is
included.
141
Intelligent Automotive Safety Systems: The Third Age Challenge
•
Simple file naming and directory structure can be performed from software
interface.
The software gives detail status of what task is being performed.
IRISYS Imager: Serial Protocol programming
The IRISYS IRI1002 thermal imager has its own protocol which needed to be
converted into an infrared image. The software has to be compatible with hardware
specifications for the thermal imager .i.e. 1,15,200 baud, 8 data bits, no parity, 1 stop
bit, no handshaking. (See Appendix D)
The IRISYS IRI1002 sends thermal data only when a certain command is written to
it through a serial port. Further details and its construction are discussed in the
section 3.2.2.
Visual Camera acquisition technology
Microsoft® DirectShow® is an architecture for streaming media on the Microsoft
Windows® platform. It supports capture using Windows Driver Model (WDM)
devices or older Video for Windows devices. DirectShow® is integrated with other
Microsoft DirectX® technologies. It detects and uses video and audio acceleration
hardware when available, but also supports systems without acceleration hardware,
therefore can be used with vast variety of compatible IBM-PC running Microsoft
operating system.
Technology used in the software is WDM ActiveXTM which is from MarvelSoft®
called VideoOCX. This control is backward compatible with most Video-forWindows (VFW) devices, such as USB cameras (webcams) and framegrabbers in
conjunction with a CCD camera or camcorder. VideoOCX works smoothly in most·
142
Intelligent Automotive Safety Systems: The Third Age Challenge
ActiveXTM hosting environments, such as LabIWindows and Visual Basic.
Applications range from professional scientific image processing and surveillance to
computer vision and general multimedia programs. This control is independent of the
interface, frame rate of the device or the Microsoft® OS used.
Thus this control gives us the flexibility of making software more flexible as it can
capture data in real-time. This control also gives some image processing capabilities
that are not built into the visual device.
Interfacing sensors
Three sensors were required to be interfaced with the data acquisition PC see Figure
7-12. These sensors are:
•
IRISYS Thermal Imager: Interfaced using DB-9 connector to a Serial COMl
port.
•
Talley pressure monitor: Interfaced using DB-9 connector to a Serial COM2
port.
•
Logitech Messenger webcam: Interfaced using USB 1.1
143
Intelligent Automotive Safety Systems: The Third Age Challenge
PC
08·9 Connector
IRISYS Thermallmager
12 Volts 1.25 Amps
power source: Individual
Lla....-..;....___
CPU ----..
use Connector
Logitech webcam
12 Volts
Power source: CPU
08·9 Connector
Talley Pressure Monitor
24 Volts
Power source: Indlviual
Figure 7-12: Interfacing of sensors with IBM-PC for Initial experiment
144
Intelligent Automotive Safety Systems: The Third Age Challenge
7.2.5.
Mounting of IRISYS Imager
The IRISYS thermal imager was mounted in the test rig by using a custom made
clamping stand (Figure 7-13).
Figure 7-13: During experiment, front mounting position of IRISYS thermal imager
This included placing a custom build clamp and placing it together with the back
face plate of IRISYS thermal imager and a G-Clamp stand on which the IRISYS
thermal imager mounts. The height of the G-Clamp stand is adjustable and it can be
clamped using to an attached at the bottom end see Figure 7-14.
145
Intelligent Automotive Safety Systems: The Third Age Challenge
Custom made
clamp
G-Clamp stand
for IRISYS
Figure 7-14: Custom built G-Clamp stand for mounting IRISYS Thermal imager
The IRISYS thermal imager and Logitech webcam are also joined together with each
such that they have the same FOV see Figure 7-15.
146
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 7-15 Mounting of web cam and IRISYS Infrared imager for experiment
The position and angle of the IRISYS thermal imager is measured by measuring tape
and angle measurement tool respectively. In some special cases a tripod stand is also
used for mounting the IRISYS imager. A level is required to make sure the IRISYS
imager is in horizontal position see Figure 7-16.
147
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 7-16: Measuring instruments used in the mounting of the IRISYS imager
7.2.6.
Volunteer drivers for experiment
Eleven (11) volunteer drivers (see Figure 7-17) are selected for the experiment from
sixteen (16) year old to sixty (60) year old. The selection was based on the volunteer
driver experience, age, hair length and considering other facial features for example
large forehead, beard and glasses. Figure 7-17 shows the visual reference image of
the volunteer driver below which is the interpolated infrared thermograph. All data
displayed is from the experimental data collected using the data acquisition system.
148
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 7-17: Eleven volunteers visual images and interpolated thermal images
7.2.7.
Offline data collected for Phase 1 Imaging
Algorithm
The offline data that was collected from the STISIM simulator run lasts for 25
minutes on average per driver. The experimental data is stored on the hard disk drive
on the data acquisition system which was later transferred to an offline platform
where MatLAB is used for Phase 1 imaging algorithm processing. For more details
on phase 1 imaging algorithm see Section 5.2.1
149
Intelligent Automotive Safety Systems: The Third Age Challenge
7.2.8.
Experimental environment and temperature
During the thermograph acquisition the infrared imager was configured to remain
accurate to 0.1 degrees. This can be obtained by a change in the parameters of the
infrared camera. The experiment was conducted on different days at different times
of the day to include the effect of day and night temperature in the experimentation.
If the infrared camera is focused from the front i.e. the view used previously by
Amin et al., (2004) the minimum temperature goes down up to 22-23 degrees, this is
due to the background temperature of the car which is significantly lower due to steel
construction of the car.
As the infrared imager detects heat emitting from the body, therefore, a limitation of
the infrared imager is the presence of glass. Visual cameras can see through glass,
but infrared imagers cannot. Therefore the infrared imager has to be installed inside
the car so it can look directly at the driver. In this experiment the position of the
infrared imager mounting was from the front looking directly at the driver. This
position was later discarded due to the impracticality of mounting in the real car. The
ideal position of mounting the infrared imager should provide a field of view 300 to
350 square millimetres as this field of view includes the head, shoulders and upper
part of arms in most cases; this is what is intended to be used for this experiment.
The right pillar (for right hand drive) is not considered as the field of view is too
small as it uses a lens of 20 degrees. The other two locations under consideration
were the rear view mirror position and left pillar. The rear view mirror position was
considered but would create a protrusion and safety hazard for the driver and also the
IRISYS thermal imager would need to be fitted with a wider angle lens. Thus the left
pillar position for mounting the infrared imager was selected, as it was an ideal
distance looking from 1 metre away giving a field of view of 14 inch sq at an angle
of 60 degrees measuring from perpendicular to the head restrain of the car seat.
The maximum temperature from thermographs acquired from the infrared imager
taken from 11 volunteers ranged from 37 to 39 degrees. This range is required to find
the background separation which will help binary threshold the thermographs which
150
Intelligent Automotive Safety Systems: The Third Age Challenge
IS
explained in detail in section 5.2.1. Small temperature variations in the
surroundings is also taken into consideration. Thus experiments were conducted at
different times of day and on different days over a long period of time. The large
temperature variations throughout the year are not considered.
7.2.9.
Conclusion of experiment 1
Experiment 1 shows the capability of the IRISYS thermal imager. Even though the
IRISYS thermal imager is not high resolution it can still track the movement of
drivers after application of the imaging algorithm on the experimental data. The
experiment 1 and phase 1 imaging algorithm can track movements but more detailed
driver movement detection can be achieved by changing position of the IRISYS
thermal imager and improving the imaging algorithm. Also the current mounting
position of IRISYS thermal imager is not suitable in practice as it blocks drivers
FOV and causes a safety hazard. Therefore experiment 2 was conducted as a more
detailed and extensive trial based on a changed position of the IRISYS thermal
imager and improved imaging algorithm.
151
Intelligent Automotive Safety Systems: The Third Age Challenge
7.3. Experiment 2: Experiment with infrared camera
mounted on the passenger side screen pillar
7.3.1.
Aim of the experiment
The IRISYS thermal imager was repositioned at an angle during this experiment (see
Section 5.2.2). This allows the experiment to be more realistic than the previous
experiment where the IRISYS mounting was blocking the driver's FOV. Also the
angled position of the thermal imager was expected to improve the previous tracking
algorithm. The number of volunteer drivers who conducted the experiment is also
increased.
7.3.2.
Holywell STISIM Driving Simulator
The STISIM driving simulator was moved from its Factory Street location to a test
rig in the university's Holywell building. Figure 7-18 shows a view of the Holywell
test rig while the experiment was being conducted.
152
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 7-18: HoJywell test rig during experimentation
The Holywell test rig (see Figure 7-19) allowed more flexibility in setting up the
sensors and instrumentation. It also allows adjustment in accordance with the
volunteer driver height and physique and instrument adjustment for each individual.
Setting up the Holywell test rig for STISIM driving simulator involved installing low
cost sensors, setting up a projection screen and balancing the test rig.
153
Intelligent Automotive Safety Systems: The Third Age Challenge
~----------'---P
"
--Projector
"
"
Projection screen
""
""
"
D
""'hl
o
STISIM
IRlmagar " "
........
"
Sensors mounting table
Adjustable steering
column by Range
Instrumer.talioo
/
!zlble
....
Rover
....
Figure 7-19: Holywell test rig with STlSIM driving simulator
Mounting low cost sensors
Potentiometer
The STISIM driving simulator uses potentiometers for the purpose of measuring
rotation and linear motion. It eliminates the high cost of using an encoder.
The construction of a potentiometer is much simpler than that of an encoder (see
Figure 7-20). 5 volts are applied to the potentiometer and voltage is measured as
154
Intelligent Automotive Safety Systems: The Third Age Challenge
shown in Figure 7-20A. This variable analogue signal is then digitised using a DAC
board.
N
Sine
JUl
T
(A) Pctfrticrreter
Figure 7-20 Difference in construction of potentiometer and encoder
Steering column mounting
Mounting involved two timing pulley, one mounting onto the steering column shaft
while the other was fixed to the potentiometer shaft, see Figure 7-21. The timing
pulleys had a 1: 1 ratio as both pulleys were identical (35mm diameter with 36 teeth).
A rotational potentiometer was configured to give 4000 counts when digitized. The
steering was then adjusted to give a reading of around 2000 counts and this point was
selected as the centre position of steering (see Table 7-1). Complete steering wheel
rotation movement was 400 degrees, 200 degrees on each side. Each 200 degrees
gave 450 counts. Which is enough counts to make the simulation work and collect
data. This construction gives an accuracy of 0.44 degrees per count.
155
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 7-21 Shows potentiometer mounting on steering column (A) potentiometer, (B) timing
belt
Table 7-1 Steering potentiometer absolute counts
For the throttle and brake the counts are shown in Table 7-2.
Throttle Min - Max
0- 3975 Counts
Brake Min - Max
0- 3975 counts
Table 7-2 Throttle and brake potentiometer absolute counts
156
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 7-22 shows the electrical diagram for interfacing the potentiometers with the
STISIM DAC board.
DACBoard
o
I!:I
0
0
5
'-omIIrr-~(Brake)
1.5 Volt
2· Analog GND
3· Analog 0 (Steering)
4· Analog I (Brake).
5· Analog 2 (Throttle)
!
Rotary & Linear potentiometers
Figure 7-22: Electrical diagrams of potentiometer connections
157
----
-----------------------------------------------------~
Intelligent Automotive Safety Systems: The Third Age Challenge
7.3.3.
Angled mounting of the IRISYS Imager
The new mounting position of the IRISYS thermal imager was proposed in Section
5.2.2 and a modified version of the imaging algorithm was developed (see Figure
5-13).
Figure 7-23 shows the experimentation conducted with the new position of the
IRISYS thermal imager.
Figure 7-23: During the experiment: new position of IRISYS thermal imager
158
Intelligent Automotive Safety Systems: The Third Age Challenge
7.3.4.
56 Channel LED controller
To study human response times ergonomists use light signals, usually Light Emitting
Diodes (LEDs). This is a very useful technique that is used by ergonomists and
others who require human response timing, for example rescue forces and air force
pilots.
A 56 Channel LED custom built controller (see Figure 7-24) was used for getting
responses from the driver during the experimentation by switching strategically
placed LEDs.
159
Intelligent Automotive Safety Systems: The Third Age Challenge
Back connection panel
Front control panel
Figure 7-24: 56 Channel LED controller
This is required to get realistic posture movements from the driver. Five positions
were carefully selected for setting up LEDs. The LED positions in the Holywell test
rig were as follows:
•
Left blind spot check
•
Left side mirror position
•
Rear view mirror position
160
Intelligent Automotive Safety Systems: The Third Age Challenge
•
Right side mirror position
•
Right blind spot check
During the first and second scenario experimental runs, the appropriate LEDs were
turned on and off to elicit a particular driver response. This allowed the capturing of
a driver's exact movements and posture by the thermal imager. For example take a
left turn at aT-junction; before making the turn in real life the driver checks the rear
view mirror, stops before the give-way line, looks both ways and then makes the left
turn. In the experimental run for this manoeuvre the driver approaches the T-junction
and before stopping the rear view mirror LED will light up, the driver response is to
look at the LED; after stopping at the give-way line the driver looks at the right and
left side-mirror position LEDs as they turn on and off twice and then the driver is
allowed to make the turn.
This equipment was used for the first and second scenario runs only. The LED
controller was controlled manually during these runs.
The third scenario run simulated motorway driving for monitoring fatigue and
drowsiness in drivers. Therefore conditions in which there was no disturbance were
critical to this test.
161
Intelligent Automotive Safety Systems: The Third Age Challenge
7.3.5.
Driving simulator scenarios
Detailed driving simulator scenarios were programmed for this experiment and
divided into three small experiments which were:
•
Scenario 1: Driving task and Situational experimental run
•
Scenario 2: Urban experimental run
•
Scenario 3: Rural experimental run
Scenario 1: Driving task and situational experimental run
In this scenario the most common driving tasks were listed and programmed for five
times in the STISIM simulator. The time for each driving task was noted. The
driving tasks were as follows:
•
T -Junction Right Turn
•
Intersection Left Turn
•
Crossing Roundabout
•
Making U-Turn
•
Emergency stop
•
Putting seat belt on
162
Intelligent Automotive Safety Systems: The Third Age Challenge
•
Reverse parking
LEDs were used to make driver use mirrors while they are driving.
Scenario 2: Urban experimental run
The Urban scenario was focused on city driving styles and was programmed with the
following driving tasks and urban environment in mind:
•
T-junction turns
•
Intersection crossing
•
Pedestrian crossings
•
Merging onto a dual carriageway
•
Traffic signals
•
Speed limit
•
Farced overtaking manoeuvre
•
Fog
•
Emergency stop by the car ahead
This scenario required the driver to be attentive all the time during the run. The
LEDs were flashed during the run to make sure driver looked in all the mirrors. The
aim was to make the urban driving behaviour close to reality.
163
Intelligent Automotive Safety Systems: The Third Age Challenge
Scenario 3: Rural experimental run
This scenario was programmed for monitoring fatigue and sleepiness in drivers while
driving. Therefore a long curving road was programmed in a scenario which lasted
for over 50 minutes. Curtains were drawn over the driving simulator test rig to give
the driver a sense of driving alone. The noise was cut to a minimum and less traffic
was shown on the road. The driver was asked to maintain a speed of 60 Mph.
7.3.6.
Volunteer drivers
A total of twenty (20) volunteer drivers were selected for the experiment, with the
selection based on gender and age. Half of both genders were over the age of 50.
Table 7-3, Table 7-4, Table 7-5and Table 7-6 shows the information about the
subjects selected.
164
Intelligent Automotive Safety Systems: The Third Age Challenge
Male
Subjects
Age
ID
(yrs)
M01
42
Ht
(cm)
153
wt.
Gender
(kg)
73
M
Glasses
Yes
M02
35
172
70
M
Yes
M03
25
175
55
M
No
M04
29
171
85
M
Yes
MOB
27
175
80
M
No
M05
26
182
82
M
Yes
Hair
style
Short
covered
2cm
Big
foreheadshort hair
1cm
Flat black
2cm
Stuck up
from front
1cm
Short
Flat4cm
Small
hair 1cm
with pcap
Beardl
Shave
Beard
Moustache
No
Cap
Yes
Driving
expo
20
Beard
Yes
No
7
Shaved
No
No
1
Shaved
No
No
10
Shaved
No
No
8
Shaved
No
Yes
8
Table 7-3: Male volunteer drivers selected for experiment 2 (below 50)
Female
Subjects
ID
Age
F01
27
Ht.
(cm)
159
Wt.
(kg)
57
Gender
F
Glasses
no
F02
27
165
52
F
no
F03
21
170
58
F
no
F04
24
159
58
F
no
Hair
style
Medium
blond
25cm
Long
curly hair
50cm
Long st
hair
40cm
Pigtail
(tied hair)
30cm
Beardl
Shaved
Moustache
Cap
N/A
N/A
No
Driving
expo
7
N/A
N/A
No
3
N/A
N/A
No
3
N/A
N/A
No
3
Months
Table 7-4: Female volunteer drivers selected for experiment 2 (below 50)
165
Intelligent Automotive Safety Systems: The Third Age Challenge
Male
Subjects
over 50
Age
ID
MM01
54
Ht.
(cm)
180
Wt.
(kg)
80
Gender
M
Glasses
Yes
MM02
50
175
76
M
Yes
MM03
51
178
75
M
No
MM04
MM05
62
56
167
172
73
85
M
M
Yes
Yes
Hair
style
Flat
grey
4cm
Grey
curly
3cm
Curly
2cm
Bald
Short
grey
Beard!
Shave
Shaved
Moustache
No
Cap
No
Driving
expo
37
Shaved
Yes
No
20
Shaved
No
No
30
Shaved
Shaved
No
No
No
No
35
38
Table 7-5: Male volunteer drivers selected for experiment 2 (above 50)
Female
Subjects
over 50
Age
ID
Ht.
(cm)
wt.
(kg)
Gen
-der
Glasses
Hair
style
Beard!
Shave
15cm
grey
black
20cm
grey
white
20cm
Black
20cm
Black
FF01
55
160
52
F
Yes
FF02
53
175
57
F
No
FF03
50
165
87
F
Yes
FF04
52
165
86
F
No
Cap
Driving
expo
N!A
Mou
stach
e
N/A
No
37
N/A
N/A
No
34
N/A
N/A
No
25
N/A
N/A
No
28
Table 7-6: Female volunteer drivers selected for experiment 2 (above 50)
166
Intelligent Automotive Safety Systems: The Third Age Challenge
7.3.7.
Conclusion of experiment 2
Experiment 2 was conducted after repositioning of the IRISYS thermal imager,
change of test rig and other modifications to the tracking algorithm. The volunteer
drivers and driving scenanos are carefully selected based on the information
required. It can be said that this experiment shows the true tracking posture
capabilities of the IRISYS thermal imager. The tracking algorithm is able to identify
eighteen (18) different posture movements which are noteworthy. Real driving video
comparisons and installing and analysing the safety system in a real car were done
afterwards. No major changes to the imaging algorithm and artificial neural network
were done after this experiment.
167
Intelligent Automotive Safety Systems: The Third Age Challenge
7.4. Real life video data comparison
Video footage was obtained of 20 volunteer drivers driving in an urban area and later
onto a motorway. Out of 20, 10 volunteers were below the age of 45, whereas the
other 10 volunteers were over the age of 55. Volunteers were selected as 50% male
and 50% female. Each video lasts for at least 45 minutes depending upon each
volunteer's driving time from point A to point B.
A conventional method of video experiment study was used which involved
professional video equipment, log book and stopwatch usually built-in the recording
equipment. This method is mostly used by ergonomists for various studies. The
process involves a tedious sequence of going through the experimental videos in
slow motion back and forth and recording driver's movements in the log book
together with the time taken to complete the driving task. The equipment used for
analysing the videos was professional editing equipment Sony SVO-9620 (see Figure
7-25).
168
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 7-25: Sony SVO-9620 Video editor and playback
The log created was later compared to the generated p-code, the results of which are
explained in the section 8.3.
169
Intelligent Automotive Safety Systems: The Third Age Challenge
7.5. Experiment 3: Real car experiment using infrared
imager
7.5.1.
Aim of the experiment
Previous experiments in driving simulators and video footage comparison with actual
driving behaviour showed a well developed and stable safety system. Therefore the
aim of this experiment was to implement the IR imaging system in a real car. The
data would still be processed offline. Trunk stability data collection was also possible
from this experiment, due to the inertial forces being present. These inertial forces
were non-existent in the driving simulator experimentation.
This was the last
experiment in a series of experiments conducted during this research.
7.5.2.
Peugeot 406 driving in controlled area
The vehicle selected for experiment was a Peugeot 406 Estate car which is shown in
Figure 7-26.
170
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 7-26: Peugeot 406 used to conduct experiment
The experiment conducted (shown in Figure 7-27) consisted of three trial runs. These
trials were required to obtain trunk stability results which are shown in section 8.6.2.
These trials are as follows:
•
Trial run 1: Normal driving
•
Trial run 2: Moderate trunk stability
•
Trial run 3: Severe truck instability (to represent an older or diabled driver)
171
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 7-27: IR imager position during the experiment
Each experimental trial run consisted of going through a test route and completing
certain driving tasks. These driving tasks were as follows:
•
Negotiate a roundabout
•
After 30 yards turn left
•
Stop at T -Junction, check both sides for traffic
•
Make a right turn
•
Follow the road for 100 yards
•
Follow the road round a left turn
172
Intelligent Automotive Safety Systems: The Third Age Challenge
•
After 50 yards, stop and signal for a right turn
•
Check the rear view mirror and side mirror for traffic
•
Make a right turn
•
Follow the road for another 50 yards and stop
•
Reverse park the car into a parking bay.
7.5.3.
Mounting of IR Imager
A specialised stand was used for mounting the IRISYS thermal imager see Figure
7-28. This stand ensures that the imager is mounted securely and does not move
while the vehicle is in motion.
173
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 7-28: Specialized stand used for mounting IRISYS thermal imager inside the vehicle
Inverter
Supplying power to the IRISYS thermal imager and Data acquisition platform was
also an issue. An inverter shown in Figure 7-29 was used for this purpose. It takes 12
DC Volts at 13 Amperes from a cigarette lighter and supplies 240AC Volts at 150
Watts to the devices.
174
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 7-29: Inverter supplies 240 AC volts from a cigarette lighter 24DC to the IRISYS
thermal imager
7.5.4.
Data Acquisition platform
The Data acquisition platform was a laptop with a serial port (COM I) and with Data
acquisition software installed. The experimental trials were conducted at 2 FPS. No
visual camera or webcam was mounted with the IRISYS thermal imager as the
algorithm had already been tested in experiment 2, see section 7.3.
175
Intelligent Automotive Safety Systems: The Third Age Challenge
7.5.5.
Volunteer drivers
Not many volunteer drivers were required at this stage as the safety system algorithm
had been tested and evaluated before. Therefore only an individual driver carried out
the trial runs whose results are shown in section 8.5.
7.5.6.
Conclusion of experiment 3
Experiment 3 was the last series of experiments conducted for this research. The first
two experiments were the development stage for the algorithm. The third experiment
used the safety system implemented in a real car, thus showing the safety system
capability in its working environment.
176
Intelligent Automotive Safety Systems: The Third Age Challenge
7.6. Summary
•
The aim of the first experiment was to find the capabilities of the IRISYS
thermal imager.
•
The second experiment repositioned the IRISYS thermal imager and changed
to a different test rig for better flexibility. This experiment was significantly large
in terms of number of volunteers and the number of trial runs each volunteer was
asked to conduct.
•
The safety system algorithm developed and tested on data from the second
experiment was capable of identifying detailed posture, movement and
behaviours. This accomplished the guidelines for the safety system.
•
Driving video comparison was conducted by using special purpose video
editing and playback equipment.
•
The third experiment was conducted in a real car (Peugeot 406). To show that
the safety system can be implemented without major problems in practice. The
car was taken for three trial runs. The results are shown in Section 8.5.
177
Intelligent Automotive Safety Systems: The Third Age Challenge
8. Results and discussion
8.1. Introduction
This chapter is divided into two sections. The first section shows the results from the
experiments that were conducted. The results are mostly in the form of graphs which
are the output of neural network simulations. The later section assesses the
capabilities of the safety system by considering different driving situations and
everyday driver problems.
8.2. Results of Neural network as a behaviour modeller
In the following pages the reader will see graphs (from Figure 8-1 to Figure 8-3) that
show the simulated results of the neural network. These results shown below are for
a single volunteer driver, the algorithm used is described in Section 5.2.2 and the
data collected from Experiment 2 is detailed in Section 7.3 The complete results for
20 volunteer drivers are shown in Appendix A.
178
Intelligent Automotive Safety Systems: The Third Age Challenge
.....
(J)
M
(J)
(J)
co
10
co
a;
.....
.....
M
.....
(J)
CD
10
CD
co
.....
10
M
10
....
....
....
(J)
10
~
/11
CD
C.
E
cu
....0/11
.
CD
.c
E
:::l
Z
.....
M
M
M
(J)
N
10
N
~
N
.....
M
(J)
10
10
ci
0
eSUOdS8J uOllelnwls uu"
Figure 8-1: Head region FBN neural network simulation result
179
R2
3.5T-------------~----------------------------------------------------------~
"1
.
.,
~
I'D
QC
I
tool
.,o-l
'".,
o·
Q
Q
I'D
IrQ
=
"1
=
Z
=
.,
=
e:.
=
i
.,
I'D
I'D
Q
~
s·=
'"
Cl)
UI
c
0
Co
UI
I!!
-Looking right
C
- - - Looking ahead
-Looking left
0
;;
.!!!
:::I
E 1.5 +-~~____~'~'~------------------~----------~~~--------~----~--~----~--------~--~---1
'in
c
c
«
-=
;'
o·
.,
I'D
-='"
00
o
O.5+-~----~----------~--------------------------------------------------~
5
9
13
17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
R3
""l
!iQ'
2.5r-----------------------------------------------------------~------~
.,
=
"
QC
I
!1
en
=Q
=
is:
.,"
2+---~--~--------~--~----------~------------~------------~----~~
=
g.
=
.,=
.,a
"
a.9.
Q
=
""l
- - - Hands on steering
=
Z
,
=
.,"=
!.
"=
.
- - - - Hands off steering
(\
I \ I'
1\. I'..
\~I \':;;'J
\
\.;;'
i
.,
Q
~
s·
'"
=
....
o·
.,
=
;'
"'"e.
....
-00
5
9
13 17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
Intelligent Automotive Safety Systems: The Third Age Challenge
8.3. Real life data comparison of results
The video footage of each volunteer was analysed by taking movement description,
time taken and the p-code was judged and noted. A certain pattern of movements
relates to the p-code generated during the video analysis. The movement for each
motion detected can be small, like looking right, or could be a series of movements
to carry out a particular manoeuvre, like putting a seat belt on. This series of
movements will create a pattern for that manoeuvre which is more or less the same
for most drivers. Some of the patterns of motion with p-codes follow. The large
manoeuvres are broken down into smaller movements which the safety system
detects, see Table 8-1.
182
Intelligent Automotive Safety Systems: The Third Age Challenge
2s
1 Putting Handbrake
D-L-S
One hand on steering wheel
Subject brakes and carries out
2 Left Turn (T-Junction
following movement
or Cross Road)
Look Right
3s
N-R-S
Turn Left (Driver looks
1s
N-F-S
Look Right
6s
N-R-S
Turn Left (Driver looks
1s
N-F-S
0.5 s
N-F-S
Less traffic
ahead)
3 Left Turn (T-Junction
or Cross Road)
Heavy traffic from right
ahead)
4 Merge onto Motorway
Looks at speedometer
Small movement undetected
by the system
Rear View mirror
0.5 s
N-F-S
Small movement undetected
by the system
Looks right
1s
N-R-S
On slip road - checking blind
spot
Looks ahead
1s
N-F-S
On slip road - should be
indicating
Looks right
1s
N-R-S
On slip road - checking blind
spot
Looks right -
Lane 1 s
N-F-S
On Motorway
Change
Table 8-1 Real life driving task comparison with neural network result
183
Intelligent Automotive Safety Systems: The Third Age Challenge
Tasks vary considerably in their complexity and also according to the driving
situation at the time. In future multi-sensor fusion will help to identify and categorise
particular tasks. For example the difference between merging onto a motorway and
turning into a junction or roundabout can be found by linking the speedometer with
the current IR system. This will tell the system when the car is stopped, or moving
very slowly, at a junction, signal or roundabout; otherwise if the car is moving and
the driver looks in a certain direction aggressively two or three times it would
indicate a change of lane or merging onto a motorway.
8.4. Use of P-codes in intelligent central safety control
Intelligent central safety control is a device installed in an intelligent car which
combines all intelligent safety systems and intelligent sensors (see Figure 8-4). This
control unit takes processed information from each safety system and sensor like Pcodes, throttle inputs, braking data and tyre pressures. It will create a complex
network of vehicle data flowing between the control unit, safety systems and sensors.
184
Intelligent Automotive Safety Systems: The Third Age Challenge
i...
~
i'
•
IR Posture detection safety sy~tem
,
Intelligent airbags
'-
:;
,
Intelligent central safety
control
~
'-
I'
~
,..
I,
I'
Intelligent controls
(~rakes, clutch, th~otlle ••
;y
FLRS and other parking sensors
,.
'.'
,....
I...
.
'" I'
~
•
•
t
~
•••
"
Figure 8-4: Concept of intelligent central safety control
This central safety control will be capable of identifying even the smallest driving
tasks and determining the risk in a driver's driving pattern. Consider a multi-channel
input of information coming to the intelligent central safety control (see Figure 8-5).
Some information will be needed as instantaneous or single-frame data and some
will need to be acquired over a period to provide a time history for analysis. Each
channel provides useful information to the safety control unit and based on that the
control unit will be able to decide the driving pattern, risk involved and safety
precautions that will be required. This is beyond the scope of this research therefore
the concept is only discussed briefly here.
185
Intelligent Automotive Safety Systems: The Third Age Challenge
Driving patterns
,/\
I
I
Channel A
Nt-S
I
,
,
NRS
I
I
I
NRS
I
I
I
NFNS
P-code
Channel 8
Throttle
ChannelC
FLRS
-+--~~~~~~~--~~--~~----~-----
-,-I
Channel 0
...
Time
Figure 8-5: Multi channel driving pattern identification from intelligent central safety control
8.5. Real car experimentation results
Experiment 3 was conducted in a Peugeot 406 car. The neural network simulation
results of this experiment are shown in Figure 8-6, Figure 8-7 and Figure 8-8. The
details of experiment 3 are explained in Section 7.5.
186
Intelligent Automotive Safety Systems: The Third Age Challenge
c:
.Q
:to
I/)
0
Cl.
"C
III
Ql
c:
0
:e
"C
III
0
Cl.
.s::
c:
I/)
"C
III
Ql
.s::
iii
.s::
en
c:
'2
Z
..J
E
0
III
Ql
Ql
~
0
"C
enc:
c: 0
:;::2
01/)
00
..J Cl.
I I
I
I
I
(
I'-
J
'"
'"
co
'"
\
M
,)
,
i
<.
I
, I,;'"
\
( ;
Il)
co
a;
I'I'-
c:
)
M
,)
I'-
\
J
1
'"
)
.....
CD
Il)
CD
"
....0::
''I;
/;
CD
III
I'-
I
Il)
'\"
"~I
M
,
Il)
,
\
'"v
I'
Il)
v
I
I
~
)
Ql
Q.
E
Cl!
....III0
..
Ql
.c
E
:l
Z
I'-
I1 •
M
M
M
\
I
/
,
'"
/
Il)
N
N
\
" c':
.....
N
/
,)
.., \
I'-
l
M
.....
r:, ,:
...
'"
(
I,
I.
Il)
\'
Il)
N
M
Il)
0
ci
asuodsaJ uOnelnW!s uU'o'
Figure 8-6: Head region FBN neural network results
187
.....................-----------------------------------------R2
3.5~------------------------------------~----------~----~------------~
-Looking right
- - - Looking ahead
-Looking left
O.5+---------------------------------~~--~--------~~--~------------------1
00
00
5
9
13 17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
.......................------------------------------------R3
2.5~----------------------------------------------------------------------~
- - - Hands on steering
- - Hands off steering
00
\0
5
9
13 17 21
25 29 33 37 41
45 49 53 57 61 65 69 73 77 81
Number of samples
85 89 93 97
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 8-6 (for region RI), Figure 8-7 (for region R2) and Figure 8-8 (for region
R3) show the results of simulated neural networks from the real car experiment.
These figures show results with similar accuracy to that of the driving simulator
experiment. Since the safety system was installed in a real car and the results are of
the same accuracy to that of experiments conducted in the driving simulator it can be
said that the safety system can be installed in a real car without any difficulty.
8.6. Assessing capabilities of the system
A list is made of what to expect from the system and what cannot be expected from
the system. The list is broken down into 'achievable', 'unachievable' and
conditional '. The 'achievable but conditional' list can be achieved only if further
work is accomplished.
Achievable
•
Trunk stability
•
Airbag deployment
o Out of position
o Large or small person
o Distance from steering wheel
o Eye Height for airbag deployment
•
Eye Height as visibility issue
•
Head turning
•
Dynamic allocation of attention !Driver distraction
o Talking / attending to passenger
o Mobile phone usage
o Lighting up cigarette
o Adjusting radio or cassette / CD player
190
Intelligent Automotive Safety Systems: The Third Age Challenge
o Other activities in car / looking down
Achievable but conditional
•
Drowsiness and fatigue: physical indicators
•
Reaction time
•
Tunnel vision
•
Movement time / trajectory
•
Time period with attention away from road: physical indicators
•
Impairments
o
Drug abuse
Unachievable
•
Task management
•
Fatigue or sleep deprivation - cognitive effects, eye movements and small
movements.
•
Impairments
o
Detection of alcohol
•
Cold legs and feet in third age people
•
Not wearing seatbelt
•
Time Period with attention away from the road - cognitive factors or small
movements.
191
Intelligent Automotive Safety Systems: The Third Age Challenge
The scenarios discussed below were analysed and altered manually to show the
safety system other capabilities. The achievable capabilities of the safety system are
discussed in further detail below:
8.6.1.
Drowsy driving (Nodding off at steering wheel)
From the literature review it can be said that sleep or drowsiness can be very critical
while driving (see section 2.1.4). It can not only be dangerous for the driver but for
other people on the road at that time as well. Symptoms of sleepiness can be
identified by nodding, yawning, eye alertness and task focus. When driving for
longer periods without interruptions or driving during the times of Ilpm to 7am the
driver starts to get drowsy. While in the sleepy state the critical symptom is nodding
off at the steering wheel. It involves an instant of sleep during which the head drops
forward then head lunges upwards and comes back to the driving position in an
erratic motion. This behaviour was observed during Experiment 2, scenario 3 and is
familiar to many people.
The following work plan in Figure 8-9 and analysis shows how movement and
posture detection system can identify nodding while driving.
192
----.----------------------------------------------------------------------~
Intelligent Automotive Safety Systems: The Third Age Challenge
Behaviour
parameters . .
Monitor driver
movements
Time frame
Activate
.. warning
system
Figure 8-9 Work plan for detecting drowsy driver
Analysis and discussion
The movement and posture detection system was installed in car simulator.
Experiments were conducted; the scenario used in these experiments was over 45
minutes and included a long stretch of straight road. The long stretch of road or rural
driving is known to cause drowsiness if driven for longer periods of time. The
infrared field of view in the experiment is approximately 350mm by 350mm. The
infrared image when interpolated is 120pixels by 120pixels. This means that a driver
moving his head by about 7pixels is equivalent to 20mm approximately.
Drowsy driver was detected by the system as a leaning posture. If the movement
information from the system is linked with time history a certain pattern of nodding
can be seen. The nodding pattern can be similar to the head to lean for fifteen (15) to
thirty-five (35) degrees. The driver usually moves back from nodding position to
normal driving position in an erratic motion. This sleep driving pattern is repetitive
and nodding occurs randomly in a short time frame.
193
Intelligent Automotive Safety Systems: The Third Age Challenge
It can be seen from the Figure 8-10 that during the 45 minutes of experiment a
female subject nodded off. The first frame showed a normal driving position and the
nth frame her head lunges upwards.
.. . 1
(n.2) seconds
Normal driving position
(n) seconds
Driver nodded off
(n+2) seconds
Back 10 nonnal
driving position
...
nme (seconds)
Figure 8-10: Drowsy driver nodding off
8.6.2.
Tiredness and fatigue
Although it is not possible to detect cognitive signs of fatigue or small movements
some repeated physical actions can be a sign of fatigue. For example, in Figure 8-11
driver is shown touching her face. This could be a sign for fatigue and tiredness if the
driver is driving for over an hour, at night time or other conditions linked with driver
fatigue and tiredness. If this information is correlated with time history, the safety
system can make assumptions with much accuracy.
194
Intelligent Automotive Safety Systems: The Third Age Challenge
Interpolated infrared
image
Interpolated infrared
image with 4 level
thresholding
Figure 8-11: Driver touching her face while driving
8.6.3.
Trunk stability
Trunk stability affects racing drivers and average on-road drivers. But for racing
drivers this effect is relatively higher than road drivers. The third age drivers have
frail bodies therefore truck stability will have considerable effect. Especially Third
age drivers are unstable in cars when going around the corners. Their trunks sway
sideways due to inertial forces (Treffner et aI., 2002).
The safety system is able to identify the trunk stability effect if time based analysis is
conducted. Time based analysis involves the study of IR thermographs and p-codes
over a period of time. For example if the driver is swaying unnecessarily over a
certain time frame can be identified as unsafe due to trunk instability. Figure 8-12
shows an average driver going around a roundabout.
195
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 8-12: Stable driver's trunk swaying effect while going over roundabout. Black line shows
trunk swaying angle.
Figure 8-13 and Figure 8-14 shows unstable trunk drivers going around a
roundabout.
Figure 8-13: Moderately unstable driver's trunk swaying effect while going over roundabout.
Black line shows trunk swaying angle.
196
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 8-14: Highly unstable driver's trunk swaying effect while going over roundabout. Black
line shows trunk swaying angle.
Figure 8-15 shows trunk stability driving pattern based on time history. During
driving the driver keep repeating the movement often making the safety system
identify the posture as trunk stability based on time history instead of leaning
posture.
197
------
---------------------------------------------------------------------,
Intelligent Automotive Safety Systems: The Third Age Challenge
Normal drilling (NO)
Slight sway sideways
(SS)
Extreme sway
sideways (ESS)
Time
Figure 8-15: Trunk stability driving pattern based on time based history analysis
The following sections discuss problems which require single frame analysis.
8.6.4.
Out of position driver
Identifying out of position (OOP) driver is a crucial task as far as safety is concerned.
Low resolution infrared imaging can classify OOP drivers. To make the images clear
to the visible eye the images shown are interpolated and infrared images are
thresholded up to four (4) levels see Figure 8-16, Figure 8-17 and Figure 8-18.
198
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 8-16: Normal driving position taken by infrared imager
It can be seen that information in Figure 8-17 and Figure 8-18 are easier to decode
visually than Figure 8-16.
Figure 8-17: Normal driving position in Interpolated Infrared image
199
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 8-18: Normal driving position in Interpolated Infrared image (with 4 level thresholding)
Figure 8-17 and Figure 8-18 shows a driver in normal driving conditions. Figure
8-19 shows a person leaning down at driving seat in a car simulator. From the Figure
8-19 it can be observed that the driver's head is outside the field of view and leaning
towards the steering wheel. It can be assumed that the driver could be reaching for
something on the passenger side. This is a very dangerous position if the airbag
deploys and can cause serious injuries particularly to the driver. To make the system
more flexible and worthwhile further information can be input into the system for
example speed of the vehicle and time driver was in leaning down position.
200
Intelligent Automotive Safety Systems: The Third Age Challenge
Interpolated infrared
image
Interpolated infrared
image with 4 level
thresholding
Figure 8-19: Driver leaning down at 80 degrees
The driver position in Figure 8-20 is similar to Figure 8-19 but the situation is
slightly different. In Figure 8-20 the supposition of worst case scenario is considered;
that the driver attention is away from the road. It could be because the driver
concentration is on the dashboard console or he is trying to use a car radio or CD
player. Again the further infonnation from the time history for example how long the
driver was in this position, road will corroborate the system.
201
Intelligent Automotive Safety Systems: The Third Age Challenge
Interpolated infrared
image
Interpolated infrared
image with 4 revel
thresholding
Figure 8-20: Driver leaning down at 4S degrees
Driver is putting seat belt on in Figure 8-21. At this instant driver took his hands off
the steering wheel and looking down while leaning left towards the seat belt buckle.
Identification of drivers which reverse by looking directly at rear window is also
possible for example reverse parking or parallel parking. The major difference
between Figure 8-21 and reversing will be that the driver hands will be on the
steering wheel. In either of these two cases the airbag need not be deployed as both
tasks are for slow motion or stationary vehicle. If airbag is deployed there will be a
danger of neck injury to the driver. The severity of injury will depend upon the
physical built of the driver.
202
Intelligent Automotive Safety Systems: The Third Age Challenge
Interpolated infrared
image
Interpolated infrared
image with 4 level
thresholding
Figure 8-21: Driver putting seat belt on
8.6.5.
Eye height
Small height of drivers is considered a safety concern, if their seats are not adjusted
appropriately. The appropriate eye height varies from individual to individual but as
a rule of thumb the eye height should be higher than the steering wheel (Porter et al.,
2001).
The eye height of the driver is calculated by finding the centroid of the driver's face.
If the driver height is below a certain limit the complete head is displayed in the IR
FOV. Thus a centroid can be found after segmentation and eye height can be
approximately calculated. If the driver's height is above a certain limit then the eye
height need not to be calculated as tall drivers do not face this risk. Therefore
measuring eye height requires that the driver's seat is adjusted appropriately, which
is shown in Figure 8-22.
203
Intelligent Automotive Safety Systems: The Third Age Challenge
Centroid
height
Figure 8-22: Measuring of driver's eye height using centroid height
8.6.6.
Head
turning
and
dynamic
allocation
of
attention while driving
Dynamic allocation of attention includes several tasks that driver performs
unintentionally. These tasks are unsafe and a risk to driver, passengers and
pedestrians such as talking to passenger face to face, looking outside the side
window at an accident or advertisement board, using a mobile phone or a GPS
system and frequent looking at the speedometer while driving in a speed camera
zone. Not all dynamic allocation of attention can be detected using the safety system
only particular ones that involve turning of head and leaning like talking with
passengers face to face or looking outside the side windows. Some examples of
dynamic allocation of attention that could not be detected by the safety system are
small head movements for looking at the side mirrors and rear view mirror, eye gaze
at the traffic coming from the side without turning the head and small head
movement made to look at the GPS display or CD player display at dashboard.
Stiff neck in older drivers and sports injury drivers could not be detected but driver
head movement can be monitored. For example, if the driver did not look towards
right for some time or when making a turn at T-junction shows that the driver might
204
Intelligent Automotive Safety Systems: The Third Age Challenge
have restricted neck movement. Another example is failure to look at blind spot due
to stiff neck while merging onto a motorway.
Head turning is a common body motion when driving a car. For example when
crossing an intersection, going through roundabouts, making turns and stopping at TJunctions, all these driver's tasks require head movement to left or right. Most of
these driving tasks are done when the car is stationary. Driver glancing for a fraction
of a second to left or right is permitted during driving but looking left or right for
longer than two (2) seconds is classified as hazardous driving (Klauer et al., 2006).
The scenario is shown in Figure 8-23, a third age driver is looking down for over two
and half (2.5) seconds at the traffic signal.
Interpolated
Infrared Image
4 Level
thresholded
Infrared Image
Figure 8-23: 65 Year old driver concentrating/looking down left at gear over 3 seconds
The driver was trying to change to lower gear before he moves off from the traffic
signal. The driver was driving in the car simulator therefore posed no such threat.
But this kind of driver's lapses when driving on actual road can put the driver and
other people on road at risk.
205
Intelligent Automotive Safety Systems: The Third Age Challenge
8.6.7.
Driver characteristics or distinct features
The infrared imager can identify drivers with different characteristics. Driver while
smoking can be prominent without difficulty using infrared imagery as shown in
Figure 8-24. A hot spot is noticed in Figure 8-24 which shows the flame of the
cigarette. In another situation (see Figure 8-25) driver is seen using a mobile phone.
In Figure 8-25 driver is shown holding one hand close to his ear for several seconds
while driving. This will not always be the case as hands free kits are common
nowadays. This can be advantageous as the system under consideration will spot
drivers using mobile phones without hands free kit. As previous research shows that
using mobile phones without hands free kit is as precarious as driving under the
influence of drugs or alcohol.
Figure 8-24: Driver smoking (Front view)
206
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 8-25: Driver using mobile phone (Front view)
In addition to classifying diverse postures and positions the system can obtain
information about the physical build of a driver without intruding on their privacy.
For example in Figure 8-26 there is a substantial difference between slim build and
muscular build of drivers. The hottest area in the infrared image in Figure 8-26 is the
frontal face area. For a muscular built person the frontal face area increases
significantly, approximately twice as that of slim built one.
207
Intelligent Automotive Safety Systems: The Third Age Challenge
Tall female driver (ht 1.65m)
with slim built
Average height male driver (ht
1.71m) with large muscular built
(wearing spectacles)
Figure 8-26: Drivers with different build
Average built drivers are shown in Figure 8-27. The frontal face area is
approximately equal for both drivers. The hair length can sometimes differentiate
between some drivers, for example most female drivers have longer hair than male
drivers. Based on this assumption the airbag and other automotive secondary safety
systems can acquire information about the driver gender and build.
208
Intelligent Automotive Safety Systems: The Third Age Challenge
Average height male driver (ht
1.75m) with average built
(wearing spectacles)
Average height female driver (ht
1.58m) with average built
(wearing spect..1cles)
Figure 8-27: Average built drivers
To accommodate 95 percentile design for intelligent airbags and other safety system
the driver height can also be very useful. Tall and short persons can be identified, as
the field of view for infrared imager stays constant (see Figure 8-28). This will not
hamper the other capabilities of the system for that particular driver, for example
identification of driver position and behaviour.
209
Intelligent Automotive Safety Systems: The Third Age Challenge
Tall male driver {ht 1.97m) with
slim-average built
Figure 8-28: Tall driver
8.6.8.
Distance from steering wheel
The distance from steering wheel to driver can be measured by using Image
processing and optics calculations. Mounting of the infrared imager is shown in
Figure 8-29. The angle between the driver and Infrared imager facilitate the
measurement of steering wheel and the driver positions.
210
---~--------------
Intelligent Automotive Safety Systems: The Third Age Challenge
I
I
I~X-ll1
I
1
I
1
I
I
I
1,,-...
/
"\
\-
"
I
I ...
.
0 1
.
-:-
--
..
Driver
;""
;""
-<
20FOV
Mounting position
of
IR Imager
(Side Pillar)
Figure 8-29: Steering distance measurement using IR Imager
211
Intelligent Automotive Safety Systems: The Third Age Challenge
Figure 8-30: Trigonometric calculation for calculating distance from steering wheel to driver
Now by using Figure 8-30 where 'A' is the distance between the steering and
thermal imager and 'z' is an image plane parallel to the thermal imager FOV. Then
'D' and 'x' can be found by the following expression:
212
-----------------------------------------------------------,
Intelligent Automotive Safety Systems: The Third Age Challenge
A= CxSin300
C=
A
Sin300
D=CxCos300
D=
D=
A
Sin300 x Cos300
A
(~x~?{)
D= 4A
J3
Assume rjJ ~ 30°
Z is x world coordinate from the thermal imager
:. X=ZxCos300
X=
J3
2
:.I(D+x) =2ZAI
(Equation 7)
To calculate the field of view area covered by the infrared imager the following
calculation was used (see Figure 8-31).
T
8_(fov1 )1
.lan ID
Where
8 =10°
Dl =1 metre
(Equation 8)
=Dl x tan 8
fovl =Ix tan (10)
fovl
fovl
= 0.1763
FOV=fovlx2
FOV =0.352square metre
213
Intelligent Automotive Safety Systems: The Third Age Challenge
-n
0
<
0
Ql
t..>
01
I\)
(.I)
.0
3Cl)
-
" "I
Figure 8-31: FOV area calculation
Figure 8-32 shows the FOV calculation of infrared thermal imager; the double sided
arrows in the figure give an idea about the distance. The distance of one metre is an
approximation. The infrared image on the left shows the driver driving very close to
the steering wheel.
Figure 8-32: Infrared images showing distance from steering wheel
214
Intelligent Automotive Safety Systems: The Third Age Challenge
8.6.9.
Special scenarios
Lane merging hazard in third age
This particular situation is selected because it is a high risk involved for older drivers
due to their restriction of movement and slow reaction time.
•
Sixty year old driving at night time in vehicle A.
•
While merging onto a motorway
•
Heavy vehicle passing and decided to give way to the heavy vehicle by
checking side mirror
•
Tried to pull in behind heavy vehicle, but fail to check blind spot which
comes under driving lapses (see section 2.1.7 for third age driver problems).
•
The vehicle B just behind the heavy vehicle, which driver in Vehicle A fail to
notice. Unless the driver in vehicle B adjusts the speed there could be a chance
for accident.
Restriction of neck movement can be a single issue not only in third agers. But it can
even be a professional driver with neck strain. These kinds of movements can be
identified by the safety system if time based history method is used which is
explained in section 8.6.2.
Multiple drivers
Young drivers are at elevated risk when accompanied by multiple passengers. The
risk increases 4-5 times, than driving alone. The safety system if installed in a
different position or wider angled lens can detect multiple passengers in a car.
215
---------------------------------------------------------------,
Intelligent Automotive Safety Systems: The Third Age Challenge
8.6.10. Discussion
The safety system is shown that it can identify eighteen different driver movements
and interpret it into p-codes. Also the safety system capabilities can be enhanced
with some modifications. It is capable of identifying drowsy driver and trunk
stability if time history based analysis is conducted. On the other hand OOP driver,
eye height and driver's physique can be detected by adding another imaging
algorithm. Trigonometric calculations in section 8.6.8 showed that the distance from
the steering wheel to the driver can be found using IR thermographs. Head turning
has been identified using p-codes.
The safety system cannot detect small movements of driver's head and eye gaze. Eye
gaze can be anything from looking in the side and rear view mirrors to scenes on the
road. Since the thermal imager is low resolution it is difficult to detect small
movements made by the driver's head.
The IR safety system will be a part of intelligent central safety control. This safety
system is a non-intrusive driver movement and position monitor. It is better than
other methods which are either visual or contact based. The integration of IR safety
system, low cost sensors, multi-sensor systems and existing sensors will make a
complex safety network in the vehicle making intelligent cars even safer.
The initial focus of this research was, as the title states, on detecting problems
typically encountered by third age drivers. As the work progressed it became
apparent that means of detecting problems or high risk behaviours for third age
drivers would also apply to drivers of any age, especially those with impairments due
to disability, injury or simple tiredness. Therefore the research would inevitably have
wider application than first thought. The third age theme was, however, maintained
as a useful focus to direct the research, but the results have delivered an approach
which can be beneficial for all drivers, irrespective of age. Future implementation of
the system may therefore be described as 'inclusive design'.
216
Intelligent Automotive Safety Systems: The Third Age Challenge
8. 7. Summary
•
This chapter addresses the research questions that were generated at the start
of research.
•
It also evaluates the capability and potential of the safety system.
217
Intelligent Automotive Safety Systems: The Third Age Challenge
9. Conclusions and Further work
9.1. Conclusions
•
The contribution to knowledge made by this thesis includes the identification
of driver postures, movements and behaviour, use of a Iow resolution IR imager
as an intelligent vehicle safety system and a novel design of a neural network
used for thermal imaging.
•
This thesis studies driver problems and current automotive safety systems and
highlights the fact that no safety system exists nowadays that can identify driver
postures, movements and behaviours which may pose a high level of risk to the
driver. Such a safety system is especially required by third age people and
disabled drivers due to physical and cognitive impairments.
•
Infrared imaging has been shown to provide a successful, non contact and
non intrusive method for identifying driver movement and postures. The IRISYS
thermal imager selected as the primary sensor is relatively low cost and low
resolution, offering advantages in terms of driver privacy. It has been shown to
provide very robust performance over a wide range of conditions.
•
An algorithm has been developed based on thermal image processmg
techniques and a novel neural network design. The algorithm was verified on a
series of experiments and was shown to be robust.
•
Experimentation has been conducted in a simulated environment using the
STISIM car simulator system and later in a 'Peugeot 406' car. The data
acquisition was conducted using software custom-built in LablWindows CV!.
218
Intelligent Automotive Safety Systems: The Third Age Challenge
The experimentation involved a wide spectrum of human subjects from 16 years
old to 67 years old. The scenario for experimentation involved urban as well as
rural runs. The system had therefore been tested with a wide range of variables.
•
The main focus of the work on third age drivers subsequently delivered an
approach that is inclusive and can ultimately be beneficial to all drivers at times
irrespective of age.
•
The safety system as developed using the IRISYS thermal imager and the
algorithm is able to identify eighteen different driver postures and movements.
Several other general and special driving cases that are high risk for the driver
were also identified using the same technique.
9.2. Further work
•
Driver posture and movement is a very complex task. It vanes from
individual to individual. Therefore a detailed study of driver posture and
movement related to high risk situations is required.
•
Extending the safety system implementation and experiment from cars to
large commercial vehicles which include public buses and trucks would be
beneficial. This will allow the detection of fatigue and other problems
III
professional drivers and hence increase the safety of commercial vehicles.
•
The safety system can be extended as a tool for helping ergonomists for
studying driver's behaviour by video. The conventional method of studying
driver videos includes tedious work of long hours of manual playback and noting
each driver task with respect to time. The system can be used to detect driver
behaviour and show only the useful behaviour at the user's request.
219
Intelligent Automotive Safety Systems: The Third Age Challenge
•
Further improvement in the safety system can be achieved by sensor fusion.
For example fusing with a visual camera which tracks eye pupil movements. This
will allow more advanced movement and posture detection and prediction.
•
Implementing novelty detection techniques will improve the artificial neural
network. Novelty detection will make the system more robust when unrecognised
things are displayed to the neural network.
•
Real-time training of the artificial neural networks can be implemented. This
will improve the safety system prediction and accuracy significantly as the safety
system will be training while taking the system offline.
•
On the physical design side, improving the packaging of the IRISYS thermal
imager is required so that it can be installed in a vehicle with ease.
•
The imaging algorithm and ANN should be implemented in embedded form,
thus eliminating the need for offline data collection and analysis.
220
Intelligent Automotive Safety Systems: The Third Age Challenge
10. References
(Qct 2003) Safety: Advanced Cameras and Image-processing.Automotive
Engineer, 32.
A. Rogers, W., Kristen Gilbert, D. and Cabrera, E. F. (1997) An analysis of
automatic teller machine usage by older adults: A structured interview
approach Applied Ergonomics, 28(3), 173-180. ISSN: 0003-6870.
Adams, A. S. and Thieben, K. A. (1991) Automatic tel/er machines and the
older population Applied Ergonomics, 22(2), 85-90.ISSN: 0003-6870.
AI-Habaibeh, A. and Parkin, R. (2003) An autonomous low-cost infrared
system for the on-line monitoring of manufacturing processes using
novelty detection International Journal of Advanced Manufacturing
Technology, 22(3-4),249-258. ISSN: 0268-3768.
Amin, I. (2003) Sensor fusion of visual and infrared system for monitoring
people. Wolfson school of Mechanical and Manufacturing Engineering,
187.
Amin, I. J., Parkin, R. and Taylor, A. J. (2004) In-Cabin Occupant Tracking
Using a Low Cost Infra Red System In: Proceedings Mechatronics and
Robotics 2004, Aachen, Germany, Institute of Electrical and
Electronics Engineers Inc. pp. 259-264. ISBN: 3-938153-50-X.
Armitage, A., Binnie, D., Chamberlain, T., Nilsson, M. and Rutter, M. (2005)
Tracking pedestrians using visible and infrared systems In: 37th
Annual Conference of UTSG, Bristol, UK, Institute for Transport
Studies, Leeds pp. 37.
Augsburger, M., Donze, N., Menetrey, A., Brossard, C., Sporkert, F., Giroud,
C. and Mangin, P. (2005) Concentration of drugs in blood of
suspected impaired drivers Forensic Science International, 153(1), 1115. ISSN: 0379-0738.
Barrett, J. and Kirk, S. (2000) Running focus groups with elderly and disabled
elderly participants Applied Ergonomics, 31(6), 621-629. ISSN: 00036870.
Baumann, M., Keinath, A., Krems, J. F. and Bengler, K. (2004) Evaluation of
in-vehicle HMI using occlusion techniques: experimental results and
practical implications Applied Ergonomics, 35(3), 197-205. ISSN:
0003-6870.
221
Intelligent Automotive Safety Systems: The Third Age Challenge
BBC (2004) A suit that simulates old age is helping architects design more
user-friendly hospitals. BBC News [Internet] Available from:
<http://news.bbc.co.ukl1/hi/health/3538220.stm> [Accessed 9 Aug,
2005]
BBC (2005) Test can spot 'sleepy' drivers BBC News [Internet] Available
from: <http://news.bbc.co.ukl1/hi/health/4383224.stm> [Accessed 10
Aug, 2005]
Beede, K. E. and Kass, S. J. (2006) Engrossed in conversation: The impact
of cell phones on simulated driving performance Accident Analysis &
Prevention, 38(2), 415-421. ISSN: 0001-4575.
Benekohal, R. F., Michaels, R. M., Shim, E. and Resende, P. T. V. (1994)
Effects of aging on older drivers' travel characteristics Transportation
Research Record, (1438),91-98. ISSN: 0361-1981.
Bertozzi, M., Broggi, A. and Fascioli, A. (2000) Vision-based intelligent
vehicles: State of the art and perspectives Robotics and Autonomous
Systems, 32(1), 1-16. ISSN: 0921-8890.
Boyraz, P. (Oct 2006) Active risk management system for occupant safety.
Wolfson school of Mechanical and Manufacturing Engineering.
Breed, D. S., Summers, L., Carlson, J. and Koyzreff, M. (2001) Development
of an occupant position sensor system to improve frontal crash
protection In: 17th International Technical Conference on the
Enhanced Safety of Vehicles, Amsterdam, National Highway Traffic
Safety Administration pp. 1-10. 325: 325. DOT HS 809 220
Bruns, B. (2000) Occupant position sensing Sensors (Peterborough, NH),
17(12),34-35. ISSN: 0746-9462.
Bunn, T. L., Slavova, S., Struttmann, T. W. and Browning, S. R. (2005)
Sleepiness/fatigue and distraction/inattention as factors for fatal
versus nonfatal commercial motor vehicle driver injuries Accident
Analysis & Prevention, 37(5), 862-869. ISSN: 0001-4575.
Burnay, S. G., Williams, T. L. and Jones, C. H. (1988) Applications of thermal
imaging, lOP Publishing Ltd. ISBN: 0852744218.
Chamberlain, T., Armitage, A., Rutter, M. and Binnie, D. (2004) Using low
resolution thermal sensors in stereo to measure pedestrian movement
In: PREP 2004, UK, IEEE.
Chang, C.-w. and Lee, S.-y. (1997) A Video Information System for Sport
Motion Analysis Journal of Visual Languages & Computing, 8(3), 265287. ISSN: 1045-926X.
222
Intelligent Automotive Safety Systems: The Third Age Challenge
Chow, T. W. S. and Cho, S.-Y. (2002) Industrial neural vision system for
underground railway station platform surveillance Advanced
Engineering Informatics, 16(1), 73-83. ISSN: 1474-0346.
Ciurczak, E., Drennen, J. and Drennen Ill, J. K. (2002) Pharmaceutical and
Medicinal Applications of Near-infrared Spectroscopy, Marcel-Dekker,
Inc, New York. ISBN: 0824794532.
Cutler, S. J. and Coward, R. T. (1992) Availability of personal transportation
in households of elders: age, gender, and residence differences The
Gerontologist, 32(1), 77-81. ISSN: 0016-9013.
D. Carr, T. W. J., D.J. Madden and H.J. Cohen (1994) The effect of age on
driving skills Journal of the American Geriatrics Society. ISSN: 00028614.
D. Poulson, M. A. a. S. R. (1996) Userfit: A Practical Handbook on UserCentred Design for Assistive Technology.
D. Poulson, S. R. (1998) Userfit: A Framework for User Centred Design in
Assistive Technology. Technology and Disability Technology and
disability,
DaimlerChrysler (2000) Infrared-laser night vision system from Daimler
Chrysler Increases visibility at night DaimlerChrysler.com [Internet]
Available from: <http://www.daimlerchrysler.com/dccom/O.O-5-7180-19815-1-0-0-0-0-0-243-7165-0-0-0-0-0-0-0,OO.html> [Accessed 12 Aug,
2005]
Daimon, T. and Kawashima, H. (2003) Study of aged drivers behaviors when
using the Advanced Cruise-Assist Highway Systems (AHS) at an
intersection JSAE Review, 24(4), 393-402. ISSN: 0389-4304.
de la Escalera, A., Armingol, J. M. and Mata, M. (2003) Traffic sign
recognition and analysis for intelligent vehicles Image and Vision
Computing, 21(3), 247-258. ISSN: 0262-8856.
de Waard, D., Steyvers, F. J. J. M. and Brookhuis, K. A. (2004) How much
visual road information is needed to drive safely and comfortably?
Safety Science, 42(7), 639-655. ISSN: 0925-7535.
DETR (2000) Tomorrow's Roads: Safer for Everyone. The Government's
Road Safety Strategy and Casualty Reduction Targets for 2010.
DFT (2002) Department of Transport Report.Department for Transport.
ISSN: 011552701 X.
Dissanayake, S. and Lu, J. J. (2002) Factors influential in making an injury
severity difference to older drivers involved in fixed object-passenger
car crashes Accident Analysis & Prevention, 34(5), 609-618. ISSN:
0001-4575.
223
Intelligent Automotive Safety Systems: The Third Age Challenge
Dixit, R. (1998) Microwave and millimeterwave applications in automotive
electronics--Trends Journal of the Franklin Institute, 335(1), 13-21.
ISSN: 0016-0032.
Dubey, B. P., Bhagwat, S. G., Shouche, S. P. and Sainis, J. K. (2006)
Potential of Artificial Neural Networks in Varietal Identification using
Morphometry of Wheat Grains Biosystems Engineering, 95(1), 61.
ISSN: 1537-5110.
Erfanian Omidvar, A. (2004) Configuring radial basis function network using
fractal scaling process with application to chaotic time series
prediction Chaos, Solitons & Fractals, 22(4), 757-766. ISSN: 09600779.
Ergonomics, A. (1986) Car controls for disabled drivers.Applied Ergonomics.
Eveland, C. K., Socolinsky, D. A. and Wolff, L. B. (2003) Tracking human
faces in infrared video Image and Vision Computing, 21 (7), 579-590.
ISSN: 0262-8856.
Falkmer, T., Henriksson, P., Gregersen, N. P. and Bjurulf, P. (2000) Driver
education for persons with cerebral palsy - a retrospective study of
educational problems Transportation Research Part F: Traffic
Psychology and Behaviour, 3(1), 15-27. ISSN: 1369-8478.
Figueroa, P. J., Leite, N. J. and Ba rros , R. M. L. (2003) A flexible software for
tracking of markers used in human motion analysis Computer
Methods and Programs in Biomedicine, 72(2), 155-165. ISSN: 01692607.
Fildes, B. N., Cameron, M. H., Vulcan, A. P., Digges, K. H. and Taylor, D.
(1994) Airbag and facebag benefits and costs Accident Analysis &
Prevention, 26(3), 339-346. ISSN: 0001-4575.
Fisher, J. I. and Mustard, J. F. (2004) High spatial resolution sea surface
climatology from Landsat thermal infrared data Remote Sensing of
Environment, 90(3), 293-307. ISSN: 0034-4257.
Flir (2006) PathFindlR, Flir Systems Inc, www.flir.com
Fukui, T., Koyota, N., Isonaga, K., Kamiji, K. and Ino, T. (2001) Occupant
position detection system (OPDS) for side airbag system JSAE
Review, 22(1), 69-74. ISSN: 0389-4304.
Geiger, E., Brunnauer, A. and Laux, G. (2004) Course and stability of
psychomotor functionsrelated to car driving skills in schizophrenic
patients: A pilot study European Neuropsychopharmacology, 14,
S252-S253. ISSN: 0924-977X.
224
Intelligent Automotive Safety Systems: The Third Age Challenge
Ghiardi, G. L. (1999) Occupant thermal comfort evaluation Proceedings of
SPIE - The International Society for Optical Engineering Proceedings
of the 1999 Thermosense XXI, Apr 6-Apr 8 1999, 3700, 324-331.
ISSN: 0277-786X.
Gold, D. R, Rogacz, S., Bock, N., Tosteson, T. D., Baum, T. M., Speizer, F.
E. and Czeisler, C. A (1992) Rotating shift work, sleep, and accidents
related to sleepiness in hospital nurses American Journal Of Public
Health, 82(7), 1011-1014. ISSN: 0090-0036.
Haddadnia, J., Faez, K. and Ahmadi, M. (2003) A fuzzy hybrid learning
algorithm for radial basis function neural network with application in
human face recognition Pattern Recognition, 36(5), 1187-1202. ISSN:
0031-3203.
Harris, J. (2000) An Introduction to Fuzzy Logic Applications, Springer.
ISBN: 0-792-36325-6.
Hashimoto, K., Tsuruta, T., Morinaka, K. and Yoshiike, N. (2000) High
performance human information sensor Sensors and Actuators A:
Physical, 79(1),46-52. ISSN: 0924-4247.
Haykin, S. (1999) Neural Networks: A Comprehensive Foundation, Prentice
Hall, Ontario, Canada. ISBN: 0-132-73350-1.
Henderson, S. and Suen, S. L. (1999) ITS: a two-edged sword for older
drivers? In: 78th Annual Meeting of the Transportation Research
Board, Washington, TRB publication pp. 1-14.
Hitchcock, D. R, Lockyer, S., Cook, S. and Quigley, C. (2001) Third age
usability and safety - An ergonomics contribution to design
International Journal of Industrial Ergonomics, 55(4),635-643. ISSN:
0169-8141.
Holland, C. A and Rabbitt, P. M. A. (1994) The problems of being an older
driver: comparing the perceptions of an expert group and older drivers
Applied Ergonomics, 25(1), 17-27. ISSN: 0003-6870.
Hoist, G. C. (2000) Common Sense Approach to Thermallmaging, SPIEInternational Society for Optical Engine. ISBN: 0819437220.
Hu, P. S., Trumble, D. A, Foley, D. J., Eberhard, J. W. and Wallace, R B.
(1998) Crash risks of older drivers: a panel data analysis Accident
Analysis & Prevention, 30(5), 569-581. ISSN: 0001-4575.
225
Intelligent Automotive Safety Systems: The Third Age Challenge
Hubbard, J. E. J. and Burke, S. E. (1999) Spatially distributed smart skin seat
sensor for high-resolution real-time occupant position tracking
Proceedings of SPIE - The International Society for Optical
Engineering Proceedings of the 1999 Smart Structures and Materials Industrial and Commercial Applications of Smart Structures
Technologies, Mar 2-Mar 4 1999,3674, 104-117. ISSN: 0-8194-31486 0277-786X.
Ince, R. (2004) Prediction of fracture parameters of concrete by Artificial
Neural Networks Engineering Fracture Mechanics, 71 (15), 2143-2159.
ISSN: 0013-7944.
Irisys (2002) IRISYS IRI1001 Handheld thermallmager.
Jahne, B. (1999) Digital Image Processing: Concepts, Algorithms, and
Scientific Applications, Springer. ISBN: 3-540-62724-3.
Javaheri, S. (2006) Sleep disorders in systolic heart failure: A prospective
study of 100 male patients. The final report International Journal of
Cardiology, 106(1), 21-28. ISSN: 0167-5273.
Kato, S., Minobe, N. and Tsugawa, S. (2003) Applications of inter-vehicle
communications to driver assistance system JSAE Review, 24(1), 915. ISSN: 0389-4304.
Klauer, S. G., Dingus, T. A., Neale, V. L., Sudweeks, J. D. and Ramsey, D. J.
(2006) The Impact of Driver Inattention on Near-Crash/Crash Risk: An
Analysis Using the 100-Car Naturalistic Driving Study Data, Report
No. DOT HS 810 594. 226.
Krahn, A. D., Yee, R., Erickson, M. K., Markowitz, T., Gula, L. J., Klein, G. J.,
Skanes, A. C., George, C. F. P. and Ferguson, K. A. (2006)
Physiologic Pacing in Patients With Obstructive Sleep Apnea: A
Prospective, Randomized Crossover Trial Journal of the American
College of Cardiology, 47(2), 379-383. ISSN: 0735-1097.
Kulkarni, A. D. (1994) Artificial Neural Networks for image understanding,
Van Nostrand Reinhold, ISBN: 0-442-00921-6
Lee, T.-L. (2004) Back-propagation neural network for long-term tidal
predictions Ocean Engineering, 31(2), 225-238. ISSN: 0029-8018.
Li, Q., Chen, X. and Hu, Z. (2004) Quantitative structure-property relationship
studies for estimating boiling points of alcohols using calculated
molecular descriptors with radial basis function neural networks
Chemometrics and Intelligent Laboratory Systems, 72(1), 93-100.
ISSN: 0169-7439.
226
Intelligent Automotive Safety Systems: The Third Age Challenge
Louis, T., Edwin, P. and Goodman, M. J. (1999) Driver Workload
Assessment of Route Guidance System Destination Entry While
Driving: A Test Track Study. Proceedings of the 5th ITS World
Congress, Berlin, Germany: VERTIS.
Loukas, Y. L. (2001) Radial basis function networks in liquid chromatography:
Improved structure-retention relationships compared to principal
components regression (PCR) and nonlinear partial least squares
regression (PLS) Journal of Liquid Chromatography and Related
Technologies, 24(15), 2239-2256. ISSN: 1082-6076.
Lucidi, F., Russo, P. M., Mallia, L., Devoto, A., Lauriola, M. and Violani, C.
(2006) Sleep-related car crashes: Risk perception and decisionmaking processes in young drivers Accident Analysis & Prevention,
38(2), 302-309. ISSN: 0001-4575.
Lundberg, C., Hakamies-Blomqvist, L., Almkvist, O. and Johansson, K.
(1998) Impairments of some cognitive functions are common in crashinvolved older drivers Accident Analysis & Prevention, 30(3), 371-377.
ISSN: 0001-4575.
Macom (2006) Long and Short distance radar sensors [Internet] Available
from: <www.macom.com> [Accessed 15 Feb, 2006]
Mar, J., Lin, F.-J., Lin, H.-T. and Hsu, L.-C. (2003) The car following collision
prevention controller based on the fuzzy basis function network. Fuzzy
Sets and Systems, 139(1), 167-183. ISSN: 0165-0114.
Marengo, E., Bobba, M., Robotti, E. and Lenti, M. (2004) Hydroxyl and acid
number prediction in polyester resins by near infrared spectroscopy
and artificial neural networks Analytica Chimica Acta, 511 (2), 313-322.
ISSN: 0003-2670.
Marquie, J. C., Thon, B. and Baracat, B. (1994) Age influence on attitudes of
office workers faced with new computerized technologies: A
questionnaire analysis Applied Ergonomics, 25(3), 130-142. ISSN:
0003-6870.
Martinelli, N. S. and Seoane, R. (1999) Automotive night vision system
Proceedings of SPIE - The International Society for Optical
Engineering Proceedings of the 1999 Thermosense XXI, Apr 6-Apr 8
1999,3700,343-346. ISSN: 0277-786X.
May, A., Ross, T. and Osman, Z. (2005) The design of next generation invehicle navigation systems for the older driver Interacting with
Computers, 17(6), 643-659. ISSN: 0953-5438.
Mazzeo, P. L., Nitti, M., Stella, E. and Distante, A. (2004) Visual recognition
of fastening bolts for railroad maintenance Pattern Recognition
Letters, 25(6), 669-677. ISSN: 0167-8655.
227
Intelligent Automotive Safety Systems: The Third Age Challenge
McGwin Jr., G., Owsley, C. and Ball, K. (1998) Identifying crash involvement
among older drivers: agreement between self-report and state records
Accident Analysis & Prevention, 30(6), 781-791. ISSN: 0001-4575.
McGwinJr, G., Chapman, V. and Owsley, C. (2000) Visual risk factors for
driving difficulty among older drivers Accident Analysis & Prevention,
32(6), 735-744. ISSN: 0001-4575.
Measurement-Computing (2001) CI0-DAS08/JR Analog 1/0 and Digital I/o
Board User's manual.
Miller, J. L. (1994) Principles of Infrared Technology: A practicle guide to the
state of the art, Kluver. ISBN: 041209861X.
Monarrez-Espino, J., Hasselberg, M. and Laflamme, L. (2006) First year as a
licensed car driver: Gender differences in crash experience Safety
Science, 44(2), 75-85. ISSN: 0925-7535.
Morelli, M. J. M. J., Montagna, G., Nicrosini, 0., Treccani, M., Farina, M. and
Amato, P. (2004) Pricing financial derivatives with neural networks
Physica A: Statistical Mechanics and its Applications, 338(1-2), 160165. ISSN: 0378-4371.
Morinaka, K., Hashimoto, K., Tanaka, S. and Yoshiike, N. (1998) Human
information sensor Sensors and Actuators A: Physical, 66(1-3), 1-8.
ISSN: 0924-4247.
Mourant, R. R., Tsai, F., AI-Shihabi, T. and Jaeger, B. K. (2001) Divided
Attention Ability of Young and Older Drivers, Proceedings of the 80th
Annual Meeting of the Transportation Research Board, Washington,
DC.
Nebot, E. M. and Durrant-Whyte, H. (1999) A high integrity navigation
architecture for outdoor autonomous vehicles Robotics and
Autonomous Systems, 26(2-3), 81-97. ISSN: 0921-8890.
Ning, H., Tan, T., Wang, L. and Hu, W. (2004) Kinematics-based tracking of
human walking in monocular video sequences Image and Vision
Computing, 22(5), 429-441. ISSN: 0262-8856.
Noy, Y. I. Y. I., Lemoine, T. L., Klachan, C. and Burns, P. C. (2004) Task
interruptability and duration as measures of visual distraction Applied
Ergonomics, 35(3), 207-213. ISSN: 0003-6870.
Noyer, J.-C., Lanvin, P. and Benjelloun, M. (2004) Non-linear matched
filtering for object detection and tracking Pattern Recognition Letters,
25(6),655-668. ISSN: 0167-8655.
228
Intelligent Automotive Safety Systems: The Third Age Challenge
Pack, A. I., Pack, A. M., Rodgman, E., Cucchiara, A., Dinges, D. F. and
Schwab, C. W. (1995) Characteristics of crashes attributed to the
driver having fallen asleep Accident Analysis & Prevention, 27(6), 769775. ISSN: 0001-4575.
Pai, C.-J., Tyan, H.-R., Liang, Y.-M., Liao, H.-Y. M. and Chen, S.-W. (2004)
Pedestrian detection and tracking at crossroads Pattern Recognition,
37(5), 1025-1034. ISSN: 0031-3203.
Park, D. C., Morrell, R. and Shifren, K. (1999) Aging and the controlled and
automatic processing of medical information and medical intentions,
Lawrence Erlbaum Associates. ISBN: 0-805-82889-3.
Park, F. and Sandberg, I. W. (1993) Approximation and radial-basis-function
networks Neural Computation, 5(2),305. ISSN: 0899-7667.
Park, F. and Sandberg, I. W. (1991) Universal Approximation using RadialBasis Function Networks Neural Computation, 3(2), 246-257. ISSN:
0899-7667.
Parker, D., McDonald, L., Rabbitt, P. and Sutcliffe, P. (2000) Elderly drivers
and their accidents: the Aging Driver Questionnaire Accident Analysis
& Prevention, 32(6), 751-759. ISSN: 0001-4575.
Pasquariello, G., Satalino, G., la Forgia, V. and Spilotros, F. (1998)
Automatic target recognition for naval traffic control using neural
networks Image and Vision Computing, 16(2), 67-73. ISSN: 02628856.
Pergola, N., Tramutoli, V., Marchese, F., Scaffidi, I. and Lacava, T. (2004)
Improving volcanic ash cloud detection by a robust satellite technique
Remote Sensing of Environment, 90(1), 1-22. ISSN: 0034-4257.
Porter, J. M. and Porter, C. S. (2001) Occupant Accommodation: An
Ergonomics Approach An Introduction to Modern Vehicle Design, 233276. ISSN: 0750650443.
Pullin, J. (2005) Drivers take a back seat. Professional Engineering, 18(22),
34-36.
Qinetiq (2004) Occupant Position Sensing using a passive heat-sensitive
camera In: UK, Qinetiq & First Technology
Ramadan, Z., Hopke, P. K., Johnson, M. J. and Scow, K. M. (2004)
Application of PLS and Back-Propagation Neural Networks for the
estimation of soil properties Chemometrics and Intelligent Laboratory
Systems, 75(1), 23-30. ISSN: 0169-7439.
229
Intelligent Automotive Safety Systems: The Third Age Challenge
Ran, B. and Uu, H. X. (1999) Development of A Vision-Based Vehicle
Detection and Recognition System For Intelligent Vehicles,
Transportation Research Record No. 1679, Intelligent Transportation
Systems, Vehicle-Highway Automation, and Artificial Intelligence,
p130-138. ISBN: 0-309-07105-4
Reuben, D. B. (1993) Assessment of older drivers Clinics in Geriatric
Medicine, 9(2),449-459. ISSN: 0749-0690.
Reynolds, D. R. and Riley, J. R. (2002) Remote-sensing, telemetric and
computer-based technologies for investigating insect movement: a
survey of existing and potential techniques Computers and Electronics
in Agriculture, 35(2-3),271-307. ISSN: 0168-1699.
Roge, J., Pebayle, T., Hannachi, S. E. and Muzet, A. (2003) Effect of sleep
deprivation and driving duration on the useful visual field in younger
and older subjects during simulator driving Vision Research, 43(13),
1465-1472. ISSN: 0042-6989.
Roge, J., Pebayle, T., Kiehn, L. and Muzet, A. (2002) Alteration of the useful
visual field as a function of state of vigilance in simulated car driving
Transportation Research Part F: Traffic Psychology and Behaviour,
5(3), 189-200. ISSN: 1369-8478.
Rogers, W. A., Cabrera, E. F., Walker, N., Gilbert, D. K. and Fisk, A. D.
(1996) Survey of automatic teller machine usage across the adult life
span Human Factors, 38(1), 156-166. ISSN: 0018-7208.
Rosenthal, T. J., Parseghian, Z., Alien, R. W. and Stein, A. C. (1999) User
Guide STISIM:The low cost driving simulator.
Rudin-Brown, M., C. and Parker, H. A. (2004) Behavioural adaptation to
adaptive cruise control (A CC): implications for preventive strategies
Transportation Research Part F: Traffic Psychology and Behaviour,
7(2),59-76. ISSN: 1369-8478.
S.G.Burnay, T. L. W., C.H.Jones (1988) Applications of thermal imaging, lOP
Publishing Ltd. ISBN: 0-852-74421-8
Sacchi, C., Gera, G., Marcenaro, L. and Regazzoni, C. S. (2001) Advanced·
image-processing tools for counting people in tourist site-monitoring
applications Signal Processing, 81(5), 1017-1040. ISSN: 0165-1684.
230
Intelligent Automotive Safety Systems: The Third Age Challenge
Sanders, A F. (1970) Some aspects of the selective process in the functional
visual field.Ergonomics, 13(1), 101-17. ISSN: 0014-0139.
Schofield, A J., Stonham, T. J. and Mehta, P. A (1997) Automated people
counting to aid lift control Automation in Construction, 6(5-6), 437-445.
ISSN: 0926-5805.
Scholz and Juergen, H. (2003) In Air Bag Development and Performance,
Product Code: PT-88, SAE International, pp. 251-259. ISBN: 978-07680-1119-7.
Schreiter, M., Gabl, R., Lerchner, J., Hohlfeld, C., Delan, A, Wolf, G., Bluher,
A., Katzschner, B., Mertig, M. and Pompe, W. (2006) Functionalized
pyroelectric sensors for gas detection Sensors and Actuators B:
Chemical, In Press
Seiple, W., Szlyk, J. P., Yang, S. and Holopigian, K. (1996) Age-related
functional field losses are not eccentricity dependent Vision Research,
36(12), 1859-1866. ISSN: 0042-6989.
Sharit, J., Czaja, S. J., Nair, S. and Chin Chin, L. (2003) Effects of Age,
Speech Rate, and Environmental Support in Using Telephone Voice
Menu Systems Human Factors, 45(2), 234-251. ISSN: 0018-7208.
Simms, B. (May 1992) Characteristics and driving patterns of drivers over
seventy In: Transport Research Laboratory 6th International
conference, TRL
Siren, A. and Hakamies-Blomqvist, L. (2004) Private car as the grand
equaliser? Demographic factors and mobility in Finnish men and
women aged 65+ Transportation Research Part F: Traffic Psychology
and Behaviour, 7(2), 107-118. ISSN: 1369-8478.
Smeeth, L., Fletcher, A. E., Siu-Woon Ng, E., Stirling, S., Nunes, M., Breeze,
E., Bulpitt, C. J., Jones, D. and Tulloch, A (2002) Reduced hearing,
ownership, and use of hearing aids in elderly people in the UK--the
MRC Trial of the Assessment and Management of Older People in the
Community: a cross-sectional survey The Lancet, 359(9316), 14661470. ISSN: 0140-6736.
Stockman, G., Chen, J.-L., Cui, Y. and Reynolds, H. (1997) Measuring body
points on automobile drivers using multiple cameras Image and Vision
Computing, 15(4), 317-329. ISSN: 0262-8856.
Stutts, J. C., Wilkins, J. W., Scott Osberg, J. and Vaughn, B. V. (2003) Driver
risk factors for sleep-related crashes Accident Analysis & Prevention,
35(3), 321-331. ISSN: 0001-4575.
231
Intelligent Automotive Safety Systems: The Third Age Challenge
Sun, T .-L.-L. (2004) Conics-enhanced vision approach for easy and low-cost
3D tracking Pattern Recognition, 37(7), 1441-1450. ISSN: 0031-3203.
Tai, J.-C., Tseng, S.-T., Lin, C.-P. and Song, K.-T. (2004) Real-time image
tracking for automatic traffic monitoring and enforcement applications
Image and Vision Computing, 22(6), 485-501. ISSN: 0262-8856.
Technologies, Omega. (2000) Transactions in measurement and
control. Non-Contact temperature measurement.
Tidrow, M. Z. and Dyer, W. R. (2001) Infrared sensors for ballistic missile
defense Infrared Physics & Technology, 42(3-5), 333-336. ISSN:
1350-4495.
Timothy, J. R. (2004) Fuzzy Logic with Engineering Applications, John Wiley
& Sons. ISBN: 0470860758.
Tissainayagam, P. and Suter, D. (2003) Contour tracking with automatic
motion model switching Pattern Recognition, 36(10), 2411-2427.
ISSN: 0031-3203.
Treffner, P., Barrett, R. and Petersen, A. (2002) Stability and skill in driving
Human Movement Science, 21(5-6), 749-784. ISSN: 0167-9457.
Tun, P. A. and Wingfield, A. (1997) Language and communication:
Fundamentals of speech communication and language processing in
old age, Academic Press, San Diego. ISBN: 0122576802.
Turk, M. and Pentland, A. (1991) Eigenfaces for recognition Journal of
Cognitive Neuroscience [Special Issue: Face perception], 3(1), 71-86.
ISSN: 0898-929X.
Uddin, M., Okazaki, E., Ahmad, M. U., Fukuda, Y. and Tanaka, M. (2006)
NIR spectroscopy: A non-destructive fast technique to verify heat
treatment of fish-meat gel, Food Control, 17(8), 660-664. ISSN: 09567135.
van den Berg, J. and Landstrom, U. (2006) Symptoms of sleepiness while
driving and their relationship to prior sleep, work and individual
characteristics Transportation Research Part F: Traffic Psychology
and Behaviour, 9(3), 207. ISSN: 1369-8478.
van der Horst, R. (2004) Occlusion as a measure for visual workload: an
overview of TNO occlusion research in car driving Applied
Ergonomics, 35(3), 189-196. ISSN: 0003-6870.
Wade Alien, R., Rosenthal, T. J., Aponso, B. L., Klyde, D. H., Anderson, F.
G., Hogue, J. R. and Chrstos, J. P. (1998) Low cost PC based driving
simulator for prototyping and hardware-in-the-Ioop applications, SAE
Paper 980222 In: Proceedings of the 1998 SAE International
232
Intelligent Automotive Safety Systems: The Third Age Challenge
Congress and Exposition, Detroit, MI, USA, SAE, Warrendale, PA,
USA pp. 35-47.
WHO (1998) Population ageing - A public health Challenge, WHO Fact Sheet
No. 135.
Williams, A. F. and Preusser, D. F. (1997) Night driving restrictions for
youthful drivers: a literature review and commentary Journal of Public
Health Policy, 18(3),334-45. ISSN: 0197-5897.
Wu, H.-H. P. (2004) Patient information extraction in digitized X-ray imagery
Image and Vision Computing, 22(3), 215-226. ISSN: 0262-8856.
Yilmaz, A., Shafique, K and Shah, M. (2003) Target tracking in airborne
forward looking infrared imagery Image and Vision Computing, 21 (7),
623-635. ISSN: 0262-8856.
Yoshiike, N., Morinaka, K, Hashimoto, K, Kawaguri, M. and Tanaka, S.
(1999) 360[deg] direction type human information sensor Sensors and
Actuators A: Physical, 77(3), 199-208. ISSN: 0924-4247.
Zhao, Z.-Q., Huang, D.-S. and Sun, B.-Y. (2004) Human face recognition
based on multi-features using neural networks committee Pattern
Recognition Letters, 25(12),1351-1358. ISSN: 0167-8655.
Zhou, Z.-H. and Geng, X. (2004) Projection functions for eye detection
Pattern Recognition, 37(5), 1049-1056. ISSN: 0031-3203.
233
~
R1
~
~
0
~
~
I
to
==
~
.,Cl.to
o·
IJQ
=
2.5
..
III
c
Co
III
I!!
f
"
0
2
\/'
.... '"
'"
'\
\
I
"
~<
"
,-",.
-Normal head position
- - - Leaning head position
-Looking down head position
.(,
0
~
.' .1,
:;
I
E 1.5
iil
c
c
0-
)('
»
:;;0
(1)
en
s::::
;:::;
01
"'CS
:r
en
.,.....
0
-::C'
c:(
(1)
::::J
(Q
-<
I1
c
»
"'CS
"'CS
3
»z
O.5~--------~----------------~----~--------~------------~----~
5
9
13 17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
z
.....................----------------------------------------R1
3.5~----------------------------------------------------------------~
2.5+-----~----~--------------~--~----------------~--~~------~~
Cl)
Cl)
~,
o
~
l!!
__ ,~~ ___ ~,~~,,'~-~---,~--~~~J---,~-~-"~~,----~
, , ' ,
c
,
.
"
;'"
2
-
Normal head position
c
o
- - - Leaning head position
~ 1.5+---------------~--~--~--------------------------~----~~----~----~
-Looking down head
position
~
:;
c
c
<
O.5+---~--~--~------~--~------------~----------~--------------~
5
9
13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
Number of samples
R1
3.5~----------------------------------------------------~------~
2.5+-----~--------------------------------------~------------~----~
GI
Cl)
c
o
~
I!! 2+-------------~----~--------~------~~~----~------------------------~ ----------------------~
Normal head position
r
c
- - - Leaning head position
-Looking down head position
o
~
:;
~ 1.5+---------------------------------------------------------------------~
c
c
«
5
IV
w
0"'1
9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
Number of samples
.......................----------------------------------------R1
3.5~------------------------------------------------------------~
2.5+-------~~~~--~--~~----~----~----~----~~--~----~----4
Cl
III
C
o
a.
III
I!
-
1i
- - - Leaning head position
-Looking down head position
c
o
'5
~ 1.5+-~----~--~~--~----~~--------~----~~----~--------~--------~~
c
c
et
O.5+-------------~------------~----------------------------~----~
5
9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
Number of samples
Normal head position
Ri
3.5~------------~--------------------------------------------------~
Normal head position
- - - Leaning head position
-Looking down head position
5
N
VJ
00
9
13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
Number of samples
.....................-----------------------------------------Ri
3.5r-----------------------------------------------~----------~
2.5+-~--~--~----~------------~------~--~~--~----~----~~--~
Cl)
1/1
C
o
c.
e
c
-Normal head position
- - - Leaning head position
-Looking down head position
o
~
"5
~ 1.5+---~----~--~~------------~----~~----------------~----~----~
c
c
<I:
O.5+-~------------------------------~~--------------~--~------~~
5
N
I.;.)
\0
9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
Number of samples
R1
3.5~----------------------------------------------------------------~
3~----------------------------------------------------~~
2.5
CD
III
C
0
..
0.
III
CD
..
2
1'-\/
Ir
11
c
0
<11
:i
E 1.5
iii
c
c
.....
I,
1I
1I
\
\I
<I:
O.5+---------------------~~--~----~------------------~----~----4
5
IV
.j::o.
o
9
13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
Number of samples
-Normal head position
- - - Leaning head position
-Looking down head position
Ri
3.5~---------------------------------------------.~-.------------------~
3~------------------------------------------------------~
2.5
ID
UI
C
0
Co
UI
ec
2
-Normal head position
- - - Leaning head position
-Looking down head position
0
~
"5
E
u; 1.5
c
c
<
5
-
N
~
9
13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
Number of samples
R1
3.5r------------------------------------------------------------,
3~------~~~--~------~~----~----------~----~----~
2.5
CD
UJ
C
0
Co
UJ
ec
2
I f
\l
\I
\I
0
;;
10
-=E
iii
c
c
<I
I \1
II
-Normal head position
- - - Leaning head position
-Looking down head position
\
I1
11
1.5
••
<
''','<'<
V
J
O.5+---------------~----------~--------~~------~--------~---4
5
N
..j:>.
N
9
13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
Number of samples
R1
3.5~----------------------------------------------------------------~
3~------------------------------------------------------~
2.5
,f
Cl)
1/1
C
0
Cl.
1/1
ec
2
\
,"
r
"
0
,+
:;:;
10
:i
E 1.5
Uj
"\/ \.'
\1
\'
"I
C
c
<I:
O.5+-----------------~--~--------~----------------~~--------~--4
5
IV
~
w
9
13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
Number of samples
Nonnal head position
- - - Leaning head position
-Looking down head position
R1
3.5~----------------------------------------------------------------~
2.5
ID
UI
C
0
C-
I'
UI
. ....
"\,.,---,..._ .......
J\ I' ,..
\".
."
,----
...
..... ,, __ ..:.. __ __\J'_..:.. __
J
1\
'-'"
I!!
-Normal head position
- - - Leaning head position
-Looking down head position
c
0
~
~
E
u; 1.5
c
c
"'
O.5+-----~----------------------------------------------------------4
5
9
13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
Number of samples
......
.....
::l
~
R1
ciQ0
(D
.....
::l
:>
:::
3.5
.....
0
3
0
.....
<0
3
(D
Cl)
~
~
.....
'<:
2.5
Cl)
'<:
Ul
.....
(D
3
Q)
UI
c
0
a.
UI
I!!
~
2
-Normal head position
- - - Leaning head position
-Looking down head position
c
0
~
:;
E 1.5
Vi
c
c
>-l
::r
(D
>-l
::r
::;0
c..
:>
(JQ
c:(
(D
n
::r
en
::l
(JQ
(D
0.5
5
9
13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
Number of samples
R1
3.5r-------------------------~---------------------------------,
2.5
Cl)
1/1
C
0
Cl.
1/1
ec
('
2
Normal head position
- - - Leaning head position
-Looking down head position
0
~
:i
E 1.5
Ui
c
c
<
5
IV
~
a..
9
13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
Number of samples
R1
3.5r--------------------------------------------------------------.
3~------------------------------------------------------~
2.5
<11
1/1
•
C
0
CL
1/1
ec
2
[\
"'./\
I'
".
I
.
'\'" "
-Normal head position
- - - Leaning head position
-Looking down head position
0
i
"5
E 1.5
Ui
c
c
<
O.5+-----------------~------------------------------------~--------4
5
N
~
.....:J
9
13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
Number of samples
R1
3.5~----------------------------------------------------------~
3~------------------------------------------------------~
2.5
CD
Cl)
C
0
..
Co
Cl)
CD
2
Normal head position
- - - Leaning head position
-Looking down head position
C
0
;;
10
:i
E 1.5
iii
c
c
0(
O.5+-------------------------------------~------~--~------------~
5
9
13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
Number of samples
R1
3.5~----------------------------------------------------------------~
2.5+---~----~----------~--~------~~--~~----~----------------4
CD
III
C
o
Cl.
~
Normal head position
- - - Leaning head position
-Looking down head position
C
o
~
:;
~ 1.5+---~~----~------------~----------~----~~------~--~~----4
c
c
et
5
9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
Number of samples
......................-----------------------------------------R1
3.5~--------------------------------------------~--------------~
2.5+-~--------------------------~--------2-----~----2---~--------4
Cl)
UI
•
C
o
c..
e
c
2+-~-;--~----~~~----~~~------~----------~~~~--~1~
,~ ,
A
I IJ\J,- __ I f \ \_._",J...\___
.
\ JI \li I\ /.JI. \\ . _ .
.... '\ ~__ 1\'- ___ ....r \. ___ ,-_/'_,/\_
1 \f'\__
o
- - - Leaning head position
-Looking down head position
i
"5
.~ 1.5+-~------------2---------2---------------------------2---------~
C
C
<I:
O.5+-------------------------------------------------------~----~
5
N
VI
o
- - Normal head position
9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
Number of samples
.......................---------------------------------------R1
3.5~----------------------------------------------------------------~
2.5
OD
III
C
0
..
Cl.
III
OD
2
,.
I'
Normal head position
- - - Leaning head position
-Looking down head position
C
0
;::
ns
:i
E 1.5
Cii
c
c
«
5
N
Vl
9
13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
Number of samples
R1
3.5~----------------------------------------------------~--------~
3~----------------------------------------------------~~
2.5+---~--~------------------~----~----~----------------~------~
III
(/)
r::
o
Cl.
~
2
,~_~~~/,
_______
,~,r~'-~
__
~~
-,..;
................
--,,,,---
r::
o
i
'5
~ 1.5+-~--~--~--------------------~~------------------------~----------4
r::
r::
<C
O.5+-------~--------------------~--~------------------~----------~
5
IV
VI
IV
9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
Number of samples
Normal head position
- - - Leaning head position
-Looking down head position
......
::s
.....
~
R1
ciQ"
(1)
::s
.....
:>
s::
3.5
.....
0
S
0
.....
<"
3
(1)
r/J
III
~
.....
'-<
2.5
r/J
'-<
tIl
.....
S
tIl
CD
1/1
C
(1)
0
..
Q,
1/1
CD
2
Nonnal head position
- - - Leaning head position
-Looking down head position
c
0
;;
III
:i
E 1.5
Ui
c
c
~
(1)
~
::r
::;"
Q-
:>
(JQ
<
(1)
(")
::r
a
~
::s
(JQ
(1)
0.5
5
9
13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
Number of samples
~
~
R2
~
0
Z
3.5
N
I
.,1-3
.,
0
[I}
0
3
tD
IJQ
0'
=
2.5
.
Q,
Cl)
;>
s::
0
S
0
~'
(l)
C/.J
Pl
~
«.....
C/.J
«Cl>
(l)
S
CIl
2
-Looking right
- - Looking ahead
-Looking left
c
o
~
"5
.51/1
~
<iQ'
(l)
::s
...
CD
1/1
C
0
1/1
-...
......
::s
1.5
c
c
<I:
...,
::s...,
(l)
::s-
a:
;>
(JQ
(l)
n
::sa
CD
::s
(IQ
(l)
0.5
5
~
.j::o
9
13 17 21
25
29
33 37 41
45 49
53 57 61
Number of samples
65 69 73 77 81
85 89 93
97
R2
3.5~---------------------------------------------------------------------'
2.5
Cl)
UI
c
0
Q.
UI
I!!
2
c
1I
I. \
\1. I
\ I I
---
I I
I" I
-
}\
,'
1
/ 11./
\1
t L.
I
0
"IfI
~
"5
E 1.5
'iii
c
c
I I
II
\
-Looking right
- - - Looking ahead
-Looking left
11
11
0(
5
9
13 17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
R2
3.5~------------------------------------------------------------------------'
2.5+-~~--~----~------------~----~--------~~--~~--------------~~~
CD
III
1\ ,
...... -
C
o
a.
e
I
-Looking right
- - - Looking ahead
-Looking left
c
o
~
3
~ 1.5+-~--~~--~--~----~------------------~----~----~----~----------~
c
c
et
5
IV
VI
0\
9
13 17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
R2
3.5r-------------------------------------------------------------------~
3+n,·rv~"nnrr~·~~~~~f~I'~------~~~---r.~~nr~,~~~~,---~
2.5~l-tl~(--tHI~--~~.--II-,f~~-A~W~---L---4,~~;-:----~:----~,--T~-41HI~'~,:I\I~I.-~~~--~
~
GI
' , .
8.
~
;
\
1
, ...
\1' ....
'"
,
II ; '
'\'~\' , , \ , " \
l\l\y..:..I\,.../.I~
,\
1\
\//"'\,,,1,1"\,\1,
2
,1\
r ',' '
'U
:~/\", I
,/>.,-' ",'\1\
,',;;
',1'\'
c
o
~
:;
E
'iij
+-______________________~_______~""'~~~_~;'__________~__~________________~
1,5
','
. , ' < ;
c
c
et
"
O.5+---------~-----~;--------~--~'~"--~------~--~----~--------------~--1
5
9
13 17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
-Looking right
- - - Looking ahead
-Looking left
R2
3.5r-------------------~--------------------------------------~----~
2.5+-----------~~~------~----~--~----------~------~--------------~~
GI
en
c
o
CL
2 __ - ____ - ________ _
~
,
'
C
;1;'
~~
----~----~--,--,-~---------,
~
C
o
~
'5
~ 1.5~----------~--~----------~----------~----------------~----~------~
c
c
<
O.5+-----------~------~----------------~----------------------~------~~
5
IV
VI
00
9
13 17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
-Looking right
- - - Looking ahead
-Looking left
.....
::s
......
~
ciQ.
R2
(1)
::s
......
3.5
:>
t:
......
0
:3
0......
3
=;:-
(1)
rJJ
~
~
~
2.5
rJJ
'<
Cl>
......
(1)
III
III
c
:3
0
Co
III
l!!
Cl>
2
-Looking right
- - - Looking ahead
-Looking left
c
0
i
:i
E
u; 1.5
c
c
~
::r
(1)
~
::r
=1.
0-
:>
(IQ
<
(1)
(J
::r
a
n
::s
(IQ
(1)
0.5
5
9
13
17 21
25 29
33 37 41
45 49 53 57 61
Number of samples
65 69 73
77 81
85 89 93 97
R2
3~--------------------------------------------~------------------~
2.5+-----------------------------------~---------------------------------i
2
Cl)
m
c
0
cm
I!!
c
0
~
:;
E
-Looking right
- - - Looking ahead
-Looking left
1.5
Ui
c
c
o!(
O.5+---~------------------------------------------~----~~----~------~--4
5
IV
0'1
o
9
13 17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
......
::l
.....
~
R2
uo"
(1)
::l
.....
3.5
>
SO
S
0
.....
:;"
3
(1)
m
~
~
.....
'<
2.5
m
'<
tIl
.....
(1)
S
tIl
Cl)
III
C
0
c.
..
III
Cl)
2
d
c
0
~
:;
\i
\
E 1.5
iii
c
c
-Looking right
- - - Looking ahead
-Looking left
>-3
::r
(1)
>-3
::;"
::r
Co
>
0Cl
<
(1)
(1
::r
~
(;"
::l
0Cl
(1)
0.5
5
N
0\
...-
9
13
17 21
25 29
33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
.....................-----------------------------------------R2
3.5~~~----------------------------~----------------------------------~
3~--~------------------~~~~------------------~--~~----~
2.5+-~~------------------------~~--~----------------------~--~~~~
c'
r--, ,___ , ;:. ____
G)
en
c
o
Co
en
c.:.. _ _ _ _ _ _ ,
,
___
, __ ,
, _____
"1.
, ____ ,
- - - - - ....
c'w-,
2~~--------__----Y---~V~--~--~~----~/~------~\.r---~L---~L-------~ ~__________~
I!!
-Looking right
- - - Looking ahead
-Looking left
v
c
o
1ii
:;
~ 1.5+-~--~--~----------------------~--~--------------------------------~------~
c
c
<C
5
9
13
17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
R2
3.5r-----------------------------------------------------------------~
3
c'"
2.5
J
fI
l\ I \
I \I \
Cl)
en
c
0
CL
.
en
Cl)
, :I
I
,\: I
t 1\ I \ d
:\
\; 1 ' 1
I
2
. .' \J
c
0
il
:i
E 1.5
iii
c
c
<
~~ ~
O.5+-----------~----------------------~----~~~----------------------~~
5
9
13 17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
-Looking right
- - - Looking ahead
-Looking left
......
....::s
~
R2
ciQ"
G
g
3.5
:>
s=
.....
0
S
0
.....
3
I
-
: I1
2.5
JI
0
.
11
I1
I'
c
CL
fl
/1
CD
UI
UI
CD
.
J\
2
I I,
"
~"
G
~,
",
r:n
III
(jjl
~
r:n
.1,
"
: I I
, I"
...
~
....
VI
,:
G
S
: I. \
VI
-Looking right
- - - Looking ahead
-Looking left
c
0
~
:i
E 1.5
Ui
c
c
~
::r
G
~
Er.
'"1
Q..
"'"
:>
(IQ
c(
G
()
::r
a(i"
::s
(IQ
G
0.5
5
9
13
17 21
25 29
33 37 41
45 49 53 57 61
Number of samples
65 69 73
77 81
85 89 93 97
R2
3.5~----------------------------------------------------------------~
3
2.5+-----------------------~----------------------------------------------~
CD
III
C
o
a.
~
c
o
~
"5
~ 1.5+-------~------------~~----~----~----~--------------------~--~-------4
c
c
c(
O.5+---------------~----------------------------------------------------~~
5
9
13 17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
-Looking right
- - - Looking ahead
-Looking left
R2
3.5~----------------------------------------------------------------------~
2.5+-------------------------------~~--------~----~--------~~--~----~
1/1
,..
o
I \
Cl)
C
c.
e
-Looking right
- - - Looking ahead
-Looking left
c
o
i
:i
~ 1.5+-~--------~--~~------------------~----~--~~--~------------~-------4
c
c
c(
5
IV
0'\
0'\
9
13 17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
R2
3.5~------~~-----------------------------------------------------------'
2.5+-----~----------------------------------------------------------~-----;
GI
UI
C
o
a.
~
,
,;,~
':
2fL~~T,~~_=~~~~~~~~~·~~~~/~-~~~~~~~~~~~~r-~~~~~~~~~~~~~~"~=··w~~·~~
-Looking right
- - - Looking ahead
-Looking left
c
o
~
"3
~ 1.5+---------~------~~------~------~~----------~------~----------~~------~
c
c
0(
5
9
13 17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
......................---------------------------------------
R2
3.5~------------------------~--------------------------------------~----~
2.5
GI
en
c
Co
en
~
\
,\
I \
h '\
0
I!!
.
,
2
,\
1\
I \
I \ I \
...
I..
,
1\
-Looking right
- - - Looking ahead
-Looking left
c
0
~
~
E 1.5
Ui
c
c
c(
O.5+-----------------------------~~--~~----------------~------------~~
IV
0\
00
5
9
13 17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
R2
3.5~----------------------------------------------------------------------~
3~------------~------~--~--~--~----~--------------------~
2.5
Cl)
UI
C
0
Co
UI
I!!
-..,
2
, / .....
-""~'-
c
0
~
'5
E 1.5
Cii
c
c
et
/
I
O.5+-------~----~--------------------~--------~~--------~----~--------4
5
9
13 17 21
25 29 33 37 41
45 49
53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
-Looking right
- - - Looking ahead
-Looking left
R2
3.5r-----------------------------------------------------------------------~
.,'
2.5+-------------------------------------~------~------~------------~--~
CD
UI
C
o
Q,
~
2+-~~------~~--~~------~--~------------------~~------------------~~ ~------------~
-Looking right
- - - Looking ahead
-Looking left
C
o
~
: i i
.',
~ 1.5+-----------------------------------~----------~------------~--------~
.: .
c
c
et
}
...
-...
-
-
O.5+-~.--.------------------------~----~--------------------------------------~
5
9
13 17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
......
::s
.....
~
R2
ciQ"
(l)
a
:>
s::
3.5
.....
0
s0
.....
::;:(l)
3
CZl
p)
(tl
.....
'<
2.5
CZl
'<
en
.....
CD
UI
(l)
sen
C
0
CL
UI
l!!
2
-Looking right
- - - Looking ahead
-Looking left
c
0
;:;
ca
:;
E 1.5
~
(l)
.....:J
e:
'"i
c..
Cii
c
c
:>
(JQ
<
(l)
n
::l"'
e.
CD
::s
(JQ
(l)
0.5
5
-
IV
-...l
9
13 17 21
25 29
33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
R2
3.5r----------------------------------------------------------------------,
2.5+---------------------------------------~------~~------------~~------4
'
,
"
Cl)
UI
C
o
0.
~
1\ ,.,.- ....... _-_..;....:. __ "_ .... - - _ -
".
.
'
...;.;,_..-,':'::'_,.,.",...;.!.,'}\."- .., .... -:_, ................'"'-",-, ......... - .......... _,
2+-------~----~~----------------~------~~--~~~--------~~--~--~
-Looking right
- - - Looking ahead
-Looking left
C
o
~
:i
~ 1.5+-~~------------~------------~~----~~----~~------~~----------------~~
c
c
«
O.5+-------------~----------------------------~--~----~----------------~
5
9
13 17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
.......................---------------------------------------......
::s
......
~
R2
ciQ.
G
::s
......
3.5
;>
t:
......
0
30
......
<.
3
G
CZI
$l)
(p
~
2.5
CZI
'<
tI>
......
CD
G
Cl!
3tI>
c
0
Co
..
Cl!
CD
2
-Looking right
- - - Looking ahead
-Looking left
c
0
~
:i
E
u; 1.5
>-3
::r
G
;t.
'"I
0-
;>
c
c
(IQ
«
G
n
::r
~
~
::s
(IQ
G
0.5
5
9
13
17 21
25 29
33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
~
9
R3
o
z
2.5~--------------------------------------------------------------------~
~
I
>
.,
8
~
=
Q.
2+-~----------~----------~~--------~----------~~--~------~~--~
fIl
==
.,cs::
o
t\)
CD
UI
C
..o
~
[ 1.5+-~----------------------~~------------------------------~~--------~----~~~
(JQ
UI
I!!
- - Hands on steering
- - Hands off steering
c
o
~
:;
E
iii
c
c
cC
O.5+-------------------------~--------~------~------------------~~--~
5
9
13 17 21
25 29 33 37 41 45 49 53 57 61 65 69 73 77 81
Number of samples
85 89 93 97
=
......................------------------------------------------R3
2.5~------------------------------------------~--------------------~~
CD
III
[C
1.5+-~------------~~--~----~----------------~--~~--~~----~~----------~~
e
c
o
~
:;
E
I
iii
c
c
',
'
t'\
[' r;,,\
v
'"
/.v\
,/
t\
1\ .
I LJ,
I
,,'
'\
..... .J
1\,,/\
~
.\
~
O.5+---------------------~----------------------~----~----------~----~
5
9
13 17 21
25 29 33 37 41 45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
- - - Hands on steering
---- Hands off steering
R3
2.5r-----------------------------~------------------------------------_.
2+-~------------------------~----------------------~~------~--~
CD
1/1
C
~ 1.5+-----~--------------------~--~--------~--------------------------------------~
~
c
- - - Hands on steering
- - Hands off steering
o
ii
:;
E
"iii
c
c
'\
\1
•
c(
I·v \
\ J.
I
\I
.\1
O.5+-~--------~------~------~--------~----------------------------~~
5
9
13 17 21
25 29 33 37 41 45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
R3
2.5r--------------------------------------------------------------------,
~
2
Cl)
VI
C
~ 1.5+-----~--~----~~----~~--------------~--------~----------~------~------~
e
- - - Hands on steering
- - - Hands off steering
c
o
~
:;
E
Ui
c
c
<
O.5+-~------------------------------------------------~------------~~-4
5
9
13 17 21 25 29 33 37 41 45 49 53 57 61
Number of samples
65 69 73 77 81 85 89 93 97
R3
2.5~--------------------------------------------------------------------~
- - - Hands on steering
- - Hands off steering
0.5+-----------------------------------------------------------------------4
5
N
-....)
00
9
13 17 21
25 29 33 37 41 45 49 53 57 61 65 69 73 77 81
Number of samples
85 89 93 97
......
g
~
R3
ciQ"
~
::l
.....
2.5
:>
=
.....
0
3
0
.....
c".,
<t
~
2
Cl)
~
~
, ~
~
Cl)
'<
Cl>
.....
3
CD
UI
C
0
Co
~
1.5
~
UI
I!!
- - - Hands on steering
- - Hands off steering
c
0
;;;
...,
=r...,
~
~
=r::;"
c
c
:>
(JQ
E
u;
c..
<
~
x
<1;
()
"
=r-
a(D
::l
0.5
(Jq
~
5
9
13 17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
R3
2.5r---------------------------------------------------------------~
2+---~----------------------~----~----~~--------------------------~
CD
UI
C
~ 1.5+---~--------~--------------------------~------~--------~--------------~--~
UI
I!!
- - - Hands on steering
- - Hands off steering
c
o
~
~
E
'iii
c
c
<C
O.5+---~----~--------------------------------------------~------~----~
5
IV
00
o
9
13 17 21
25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97
Number of samples
......
::s
.....
~
R3
ciQ"
(1)
::s
.....
2.5
)-
s::
.....
0
S
0
.....
<"
(1)
(/j
2
III
(t>
.....
'<
(/j
'<
tIl
.....
(1)
S
tIl
Cl)
III
C
0
CL
1.5
III
!!!
.
- - - Hands on steering
- - Hands off steering
c
0
co
"5
E
...,
...,
a:
::r
(1)
'"1
0..
"iii
c
c
)OQ
c(
(1)
()
::r
E-
n
::s
"
0.5
OQ
(1)
5
N
00
9
13 17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
R3
2.5r--------------------------------------------------------------------,
CD
CII
e
~
1.5
/'/
I
CII
CD
0
11
Il.
"5
II'.
1I
0-
e
i
E
, }
\I"
\/"
<
.
\I \I
t
,
I 11 /
L
... IU 1 1/
1(1/
,
\I ,
,
~
f \ 1\ • , .
I ~: III " 1\ 11
111 ,\1\/1
1
I. \
iii
e
e
I
I
I
~
/1
11
'I
~\/ltll
f
~
,#I , ,I" .
. /1 '1 I.~ .
11/1/
I
I • I I I
1/, /\ I I I
r
I
t;
/111/ t
,.
11 V
'<
O.5+---------~------------------------~--------~--------------~~-----4
5
IV
00
IV
9
13 17 21
25 29 33 37 41 45 49 53 57 61
Number of samples
65 69 73 77 81 85 89 93 97
- - - Hands on steering
- - Hands off steering
......
::s
......
2:
oq.
R3
(D
::s
......
>
e:
......
2.5
0
30
......
:;:r
(D
r:/l
2
!l)
(jjl
q
r:/l
'<
Cl)
.-+
CD
(D
UI
e
0
..
(l.
3
1.5
~
....,
UI
CD
- - - Hands on steering
- - Hands off steering
e
o
~
"5
E
'iii
::s(D
....,
::s::;.
0..
>
e
e
{JQ
et
(D
()
::sr:?
0::s
(IQ
,
0.5
(D
5
IV
00
VJ
9
13 17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
R3
2.5r--------------------------------------------------------------------,
CD
UI
C
~ 1.5+---~----------------------------~~--------------------------------------------~
e
c
o
~
"5
E
Ui
- - - Hands on steering
- - - - Hands off steering
t
,"
"
\
\
I \
c
c
c(
5
IV
00
~
9
13 17 21
25 29 33 37 41 45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
R3
2.5r--------------------------------------------------------------------,
2~--~~------~----~--------~----~------------~~~------~--~
- - - Hands on steering
- - Hands off steering
O.5+-----------------~------------~------~------------~----------------4
5
IV
00
VI
9
13 17 21
25 29 33 37 41 45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
......
.....
::l
~
aq"
R3
(1)
a
>
s::
.....
2.5
0
3
.....
0
~t
(1)
(/)
2
~
~
.....
'<
(/)
'<
CIl
.....
3
CD
III
c
0
.
Q.
III
(1)
1.5
~
CD
- - - Hands on steering
- - Hands off steering
c
0
;:;
ca
:;
E
'iii
0-3
::r
a
-
c
c
0-3
::r
(1)
>
(JQ
et
(1)
(i
::r
e:.
0'
::l
(JQ
0.5
(1)
5
N
00
0\
9
13 17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
R3
2.5r---------------------------------------------------------------~
CD
UI
&1.5+---------------------------~--------------------------------------------------~
c
UI
f!!
c
- - - Hands on steering
- - - Hands off steering
o
ii
'5
E
iii
c
c
<C
O.5+---------------~----------------------------------------------------~
5
IV
00
-...1
9
13 17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81 85 89 93 97
-....£:
: :l
R3
ciQ"
~
::l
....
2.5
....>-
$::=
0
3
....0
:;"
~
en
Il)
2
....et'
'<
en
'<
en
....
GI
1/1
c
0
D-
~
3
1.5
en
1/1
f!!
- - - Hands on steering
- - Hands off steering
c
0
:;
"5
E
iii
c
c
...:J
::r
~
...:J
::r
::;"
0-
:>
(JQ
<C
~
(1
::r
g.
0::l
0.5
(JQ
~
5
IV
00
00
9
13 17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
......
::s
.....
£:
R3
ciQ"
(1)
::s
.....
2.5
;
:>
s::
.....
0
S
0
.....
,
:;::"
(1)
en
2
Pl
et>
.....
'<
en
'<
Cl>
.....
(1)
S
Cl>
CD
III
C
0
a. 1.5
III
CD
...c
- - - Hands on steering
- - Hands off steering
0
~
:;
cK
,.
E
Vi
c
c
et
. . -0
,,",,'
/
0..
/
:>
(JQ
'"
:".
"
(1)
"'t
'1'
..... :;,J
...,
::r...,
a:
:-1.
(1)
()
::r-
e:.
er
::s
0.5
(JQ
(1)
5
IV
00
\0
9
13 17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
R3
2.5r-----------------------------------------~----------~-------------,
2+---~--~~~~--~~~--~~--------~~----------------------~----~
III
Cl)
c
~ 1.5+-~--~------------~------~------------------~------~--~~~----~~~------4
~
- - - Hands on steering
- - - Hands off steering
c
o
~
'5
E
'iii
c
'v\
1 _
,
! It'
c
<
O.5+-----------~------------------------------------------------~------~
5
N
\0
o
9
13 17 21
25 29 33 37 41 45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
R3
2.5r-------------------------------------~--------------------------,
\
\
'\
It
I~
11
\
\ " ...r .... 1 ..
....
- - - Hands on steering
- - Hands off steering
.....
I \
I ,
I
r ...
O.5+---~----~--~--~~~~------------~--~--------~~--------------~
5
9
13 17 21
25 29 33 37 41 45 49 53 57 61
Number of samples
65 69 73 77 81 85 89 93 97
.......
::l
~
R3
ciQ"
(1)
::l
2.5
:>
SO
S
0
::;::(1)
m
2
I»
(t>
'<
m
'<
(Il
CD
UI
C
0
Co
UI
(1)
S
....,
1.5
(Il
I!!
- - - Hands on steering
- - Hands off steering
c
0
~
ca
~
::r
(1)
--l
::r
[
:>
(IQ
E
iii
c
c
<C
(1)
(1
::r
a
G
::l
(IQ
0.5
(1)
5
9
13 17 21
25 29 33 37 41
45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
R3
2.5~--------------------------------------------------------------------~
2+---~--------------~----~~----~----~----------~------~~~
ID
UI
C
~ 1.5-~--------~------~------~------~--------------------~--------~--~----~--~
El
- - - Hands on steering
- - - - Hands off steering
c
o
~
:;
E
Ui
c
c
«
O.5+-----------~------------------------------~------~~~------~-----4
5
9
13 17 21
25 29 33 37 41 45 49 53 57 61
Number of samples
65 69 73 77 81
85 89 93 97
Intelligent Automotive Safety Systems: The Third Age Challenge
Appendix B: MatLAB Code
This appendix describes the important function that were written during this
research. These functions are written in MatLAB M-Code language.
1. The ReadIquire2gui.m function reads the DAT files created by the Data
Acquisition software for infrared, visual and internal thermal imager
temperature data and imports it into MatLAB work space.
function [Iquire, filel, numberimg] = ReadIquire2gui
[filename, pathname] = uigetfile ( ... %Shows the user interface to read files & path
name
{'* .dat', 'I-Quire Files'; ...
'*.txt', 'I-Quire txt files'; ...
'*.*', 'All Files'}, ...
'Pick a file');
file 1= [pathname, filename]; %Puts file and pathname together
ii_eof= 0;
i=l;
fid = fopen(filel);
%Opens File
iffid -=-1
while ii eof <= 0
% Reads Infrared data
a = fgetl(fid);
b = sscanf(a,'%d');
Iquire(i).Infrared = reshape(b,16,16);
% Arranges IR data into a 16xl6
Iquire(i).Infrared = Iquire(i).Infrared/IO; % divides the multiple of Infrared by 10
% Reads Internal temperature
c=fgetl(fid);
% These three lines below get Internal temperature of
IRISYS & put in a data
294
Intelligent Automotive Safety Systems: The Third Age Challenge
d=fliplr(strtok(fliplr(c»);
Iquire(i).Itemp=sscanf(d, '%d');
% Reads Visual File data
alphabmp=fget1(fid);
% These 3 lines below get Bmp File name
Iquire(i).Bmp = [pathname, alphabmp, '.bmp']; %puts together the pathname as well
tmp=fget1(fid);
tmp=fget1(fid);
ii_eof= feof(fid);
i=i+1;
end
end
fclose(fid);
numberimg = i-I;
2. The CompareIQ2.m (see Figure C-I) function reads the infrared data from
the DAT file and compares the infrared thermograph before and after image
processes. The GUI interface go through the experimental data easily and
quickly and can import data to MatLAB workspace with ease.
295
Intelligent Automotive Safety Systems: The Third Age Challenge
;
2'14
I
(>I'O'1 Q-1
Rood ~()At',1t
~
.......
I
~
0
R2
IU
'62'
IOU
·ZS.I~!S
"",U.I
....er:!
le
...u.,
Figure B-1: CompareIQ2 GUI interface
% This Function executes on slider movement.
function slider2_Callback(bObject, eventdata, bandIes)
handles.slider_value = fIX(get(handles.slider2, 'Value'»; % GETS SLIDER VALUE
axes(handles.infrared);
infl =fliplr((handles.iq(handles. slider_value ).Infrared»;
handles.infl28=interp2(infl,3 );
rl=handles.infl28(1:80,1:61); %Section for RI - TORSO
rl_b=rl>30.5;
rl_area=sum(sum(rl_b»; %area rl
r2=handles.infl28(1:80,60:121); %Section for R2 - HEAD
r2_b=r2>30.5;
r2_b=bwareaopen(r2_b,200);
r2_area=sum(sum(r2_b»; %area r2
296
Intelligent Automotive Safety Systems: The Third Age Challenge
r2_b=bwlabel(r2_b);a=regionprops(r2_b,'orientation'); %finds angle for head
r2_aa=r2_area/a.Orientation;
%arealangle r2
r3=handles.infl28(81: 121,:); %Section for R3 - ARM I SHOULDER
r3_b=r3>27;
r3_area=sum(sum(r3_b»; %area r3
imagesc(interp2(infl ,3»;
colormap('gray');
%now being used for cropped region
axes(handles.r 1);
imagesc(rl_b);
% displays rl file
axes(handles.r2);
imagesc(r2_b);
% displays r2 file
axes(handles.r3);
imagesc(r3 _b);
% displays r3 file
handles.r 1_area=r I_area;
handles.r2_ area=r2_area;
handles.r2_ aa=r2_ aa;
handles.r3 _area=r3 _area;
set(handles.edit3, 'String' ,handles.rl_area);
set(handles.edit4, 'String' ,handles.r2_area);
set(handles. edi t5, 'String' ,handles.r2_ aa);
set(handles.edit6, 'String',handles.r3 _area);
set(handles.edit2, 'String' ,handles.slider_value);
guidata(hObject,handles);
297
Intelligent Automotive Safety Systems: The Third Age Challenge
3. The Annprocess.m function reads data from the MS Excel XLS format files
constructs, trains and simulates the ANN. Then writes the results in the same
XLS file in appropriate space. The m-code is changed depending upon the
network that is being trained.
function ann process
[filename, pathname] = uigetfile ( ... %Shows the user interface to read files & path
name
{'*.xls', 'Excel File'; ...
'*.*', 'All Files'}, ...
'Pick a file');
filel= [pathname, filename]; %Puts file and pathname together
%xls sheetl =rI, sheet2 =aa r2, sheet3 =area r2, sheet4 = r3
%%%%%%%%%%%%%Rl SECTION%%%%%%%%%%%%%
training_set! = xlsread(file 1,'sheet! ','d2:d51 I); %gets the first training set for rl
target_set 1 = xlsread(filel,'sheetl ','e2:e51 I); %gets the first target set for rl
training_set2 = xlsread(file 1,'sheet! ','f2:f51 I); %gets the second training set for rl
target_set2 = xlsread(filel,'sheet! ','g2:g51 I); %gets the second target set for rl
training_set3 = xlsread(filel,'sheetl ','h2:h51 I); %gets the third training set for rl
target_set3 = xlsread(filel,'sheetl ','i2:i51 I); %gets the third target set for rl
!iimulation_setl = xlsread(filel,'sheetl ',j2:jl0l I); %gets first simulation set for rl
simulation_ set2=xlsread(file 1,'sheetl ','12: 11 0 1');%gets second simulation set for r 1
simulation_set3 = xlsread(filel,'sheetl ','n2:nl0l ');%gets third simulation set for rl
training_set_rl = [training_set!; training_set2; training_set3]';
target_set_rl = [target_setl; target_set2; target_set3]';
simulation_ set_rl = [simulation_ setl; simulation_ set2; simulation_ set3]';
298
Intelligent Automotive Safety Systems: The Third Age Challenge
% FOR RI ANN FF NETWORK
netJl
=
newff([rl_min rl_max],[2 1], Cof the 0;
% for RI
creates ANN FF
network
netJ1.trainParam.epochs = SO;
net_rl. trainParam.goal=O.O 1;
netJl = train(netJl, training_set_rl, target_set_rl);
% y = sim(net_rl, training_set_rl); % To simulate rl training set enable this
output_set_rl = sim(net_rl, simulation_set_rl); %simulates the ann fib for rl
output_setl = (output_set_rl(I:100»';
output_set2 = (output_setJl(101:200»';
output_set3 = (output_set_rl(201:300»';
xlswrite(filel, output_setl, 'sheetl ','k2:klOl I); %write the first output set for rl
xlswrite(filel, output_set2, 'sheetl','m2:ml0l'); %write the second output set for rl
xlswrite(filel, output_set3, 'sheetl ','02:0101 I); %write the third output set for rl
%%%%%%%%%%%%%%%%%R2 SECTION%%%%%%%%%%%%%
%%%%FORAA
training_setl
=
xlsread(filel,'sheet2','d2:dSl I); %gets the first training set for r2
target_setl = xlsread(filel,'sheet2','e2:eSl I); %gets the first target set for r2
training_set2 = xlsread(filel,'sheet2','f2:fSl I); %gets the second training set for r2
target_set2 = xlsread(filel,'sheet2','g2:gS1 I); %gets the second target set for r2
training_set3 = xlsread(filel,'sheet2','h2:hS1 I); %gets the third training set for r2
target_set3
= xlsread(file 1,'sheet2','i2:iSl I);
%gets the third target set for r2
simulation_setl = xlsread(filel,'sheet2',j2:jl0l I); %gets first simulation set for r2
simulation_set2 = xlsread(filel,'sheet2','12:1101 I); %gets second simulation set for r2
simulation_ set3 = xlsread(file 1,'sheet2','n2:nlO 1I); %gets third simulation set for r2
299
Intelligent Automotive Safety Systems: The Third Age Challenge
training_set_r2_aa = [training_set1; training_set2; training_set3]';
target_setJ2_aa = [target_set!; target_set2; target_set3]';
simulation_set_r2_aa = [simulation_set1; simulation_set2; simulation_set3],;
%%%% FOR ANGLE
training_set! = xlsread(file1,'sheet3','d2:d51'); %gets the first training set for r2
target_set! = x1sread(file1,'sheet3','e2:e51'); %gets the first target set for r2
training_set2 = xlsread(file 1, 'sheet3', 'f2:f51'); %gets the second training set for r2
target_set2 = xlsread(file1,'sheet3','g2:g51'); %gets the second target set for r2
training_set3 = xlsread(file1,'sheet3','h2:h51'); %gets the third training set for r2
target_set3 = xlsread(filel,'sheet3','i2:i51'); %gets the third target set for r2
simulation_set! = xlsread(filel,'sheet3',j2:jI01'); %gets first simulation set forr2
simulation_set2 = xlsread(filel,'sheet3','12:1101'); %gets second simulation set for r2
simulation_set3 = xlsread(file1,'sheet3','n2:nlOl'); %gets third simulation set for r2
training_set_r2_angle = [training_set!; training_set2; training_set3]';
target_setJ2_angle = [target_set1; target_set2; target_set3]';
simulation_set_r2_angle = [simulation_set!; simulation_set2; simulation_set3],;
training_set_r2 = [training_set_r2_aa; training_set_r2_angle];
target_set_r2 = [target_set_r2_aa; target_set_r2_angle];
simulation_set_r2 = [simulation_set_r2_aa; simulation_set_r2_angle];
% FOR R2 ANN RB NETWORK
net_r2 = newrb(training_set_r2, target_set_r2, 0.01, I); %for R2 creates a ANN
RBN network, goal =0.01, spread = 1
% Y = sim(netJ2, training_set_r2); % To simulate r2 training set enable this
output_set_r2 = sim(net_r2, simulation_set_r2); %simulates the ann rbn for r2
output_set1 = (output_setJ2(l : 100))';
output_set2 = (output_set_r2(101:200))';
output_set3 = (output_setJ2(201:300))';
300
Intelligent Automotive Safety Systems: The Third Age Challenge
xlswrite(filel, output_set!, 'sheet2','k2:klOl'); %write the first output set for r2
xlswrite(filel, output_set2, 'sheet2','m2:mlOl '); %write the second output set for r2
xlswrite(filel, output_set3, 'sheet2','o2:o101 '); %write the third output set for r2
%%%%%%%%%%%%R3 SECTION%%%%%%%%%%%%%%%%%
training_set! = xlsread(filel,'sheet4','d2:d51 '); %gets the first training set for r3
target_set 1 = xlsread(filel,'sheet4','e2:e51 '); %gets the first target set for r3
training_set2 = xlsread(filel,'sheet4','f2:f51 '); %gets the second training set for r3
target_set2 = xlsread(filel,'sheet4','g2:g51 '); %gets the second target set for r3
simulation_set! = xlsread(filel,'sheet4',J2:j101 '); %gets first simulation set for r3
simulation_set2 = xlsread(filel,'sheet4','12:1101 '); %gets second simulation set for r3
training_set_r3 = [training_set!; training_set2]';
target_set_r3 = [target_set!; target_set2]';
simulation_set_r3 = [simulation_setl; simulation_ set2]';
% FOR R3 ANN FF NETWORK
netJ3 = newff([r3_min r3_max],[3 1],. In); % for R3 creates ANN FF network
netJ3.trainParam.epochs = 50;
netJ3.trainParam.goal=0.01;
net_r3 = train(net_r3, training_setJ3, target_set_r3);
% y = sim(net_r3, training_set_r3); % To simulate r3 training set enable this
output_set_r3 = sim(netJ3, simulation_setJ3); %simulates the ann fib for r3
output_set! = (output_setJ3(1:100»';
output_set2 = (output_set_r3(101:200»';
xlswrite(filel, output_set 1, 'sheet4','k2:kl0l '); %write the first output set for r1
xlswrite(filel, output_set2, 'sheet4','m2:m101'); %write the second output set for r1
301
Intelligent Automotive Safety Systems: The Third Age Challenge
Appendix C: Identifying occupants
Objectives that are required to be achieved are identification of subjects and their
positions while driving.
Recognition of different subjects that volunteered in the experiment is required using
infrared imager. Eleven subjects are selected with somewhat different face features,
height and hairstyles shown in Figure C-l. The aim is to distinguish between groups
of subjects are ideally differentiate them individually.
Figure C-l Eleven volunteers visual images and interpolated thermal images
For the ease of image processing and neural network analysis the tasks are divided
into two main sections.
302
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -------
Intelligent Automotive Safety Systems: The Third Age Challenge
1. Subject identification and recognition
2. Position tracking and recognition of tasks being performed
The identification is based on the mUltiple temperature thresholding which is shown
in (Amin et al., 2004). The face segmented image acquired is then used to take
further measurements. An example is shown how the identification of volunteers is
done. As seen from the Figure C-2 volunteers A, Band C all infrared images which
are threshold as discussed in (Amin et al., 2004). A datum height is created anyone
higher or lower than this height is placed in two different groups. It takes into
account the height and inclination of the car seat. Also the volunteer is wearing
glasses or not. As in the Figure C-2, Volunteer (B) is placed in above reference
height group where as Volunteer (A) and (C) are in the below reference height group.
Also Volunteer (A) and (B) are not wearing glasses, but volunteer (C) is wearing
glasses. Thus similarly different groups are created that can distinguish between the
eleven (11) volunteer of the experiments.
Height
reference
(9)
Figure C-2 Volunteers A, B & C
303
Intelligent Automotive Safety Systems: The Third Age Challenge
These parameters which measured are as follows:
1. above or below reference height
2. wearing glasses or not
3. median of Talley pressure monitor
4. head height and head width ratio (in pixels)
5. circularity of the thresholded image
6. particular features (in some cases)
7. filled area
As only three to four relations are enough to identify between a small group of
volunteers during experiment. Many more quantitative features are created that can
be measured and help to identify the individuals from the infrared images. Neural
networks are used to differentiate between them.
Experiment 1: Neural Network for identification of subjects
Many back propagation neural network and radial basis neural network
configurations are applied to identify the eleven volunteers in the infrared image. At
the moment the infrared images features which are extracted using multiple
thresholding technique is used as an input for the neural networks. For output the
subjects are assigned identification numbers from one (1) to eleven (11).
Back propagation ANN results
The output results from the back propagation neural networks for identification of
subjects are shown from Figure C-3 to Figure C-8:
304
Intelligent Automotive Safety Systems: The Third Age Challenge
Back Propagation ANN, 1 Layer, 'Iogsig' Function
Cl
ti
Ql
:D"
::s
en
Sample Numbers
Figure C-3 Back Propagation ANN with 'Iogsig' as transfer function and 1 layer
Back Propagation ANN, 1 Layer,
~ansig'
Function
Cl
....
U
QJ
:ii'
:::I
en
10
20
40
·50
60
70
Sample Numbers
Figure C-4 Back Propagation ANN with 'tansig' as transfer function and 1 layer
305
Intelligent Automotive Safety Systems: The Third Age Challenge
Back Propagation ANN, 1 Layer, 'linear' transfer function
12
11
1[]
9
B
0
ti
ID
:rr
7
6
::J
(fl
5
4
3
2
Sam le Numbers
Figure C-5 Back propagation ANN with 'linear' as transfer function and 1 layer
Back propagation ANN (BPN) simulation results as shown in Figure C-, Figure C-.
All three Back propagation neural networks are small and contain one (1) hidden
layer. The input vector is 256 where as the optimized hidden layer neurons are found
to be around 300 to 350 neurons. It can be seen that from Figure C- and Figure Cthat 'logsig' and 'tansig' are much better transfer functions.
Further two (2) hidden layered ANN are created. The results from these back
propagation neural networks are shown in Figure C-6 to Figure C-S.
306
Intelligent Automotive Safety Systems: The Third Age Challenge
Back Propagation ANN, 2 Layer, 1ogsig' transfer function
12
11
10
9
8
-
Cl
u
Cl
:D'
:::s
UJ
7
6
5
4
3
2
1
0
0
10
20
30
40
50
60
70
Sample Numbers
Figure C-6 Back propagation ANN with 'Iogsig' transfer function and two hidden layers
Back Propagation ANN, 2 Layers I 'ansig' transfer function
12r-----~----~----~----~r_----~----~----_,
11
9
8
o
-
7
~
6
UJ
5
:D'
:::s
4
3
10
20
30
40
50
60
70
Sample Numbers
Figure C-7 Back propagation ANN with 'Iogsig' transfer function and 2 hidden layers
307
Intelligent Automotive Safety Systems: The Third Age Challenge
Back Propagation ANN, 2 Layers; 'linear' franfer function
Sample Numbers
Figure C-8 Back propagation ANN with 'linear' transfer function and 2 hidden layers
It can be seen from Figure C-6, Figure C-7 and Figure C-8, that 'logsig' transfer
function performed much better in our circumstance. But as the complexity of the
network and hidden layer increases there is not much influence on the result. Thus a
simple single hidden layer neural network with 'logsig' transfer function is sufficient
to achieve this result.
Radial Based ANN results
Six (6) radial based neural networks are created and simulated. The configuration of
these six (6) neural networks is shown in Table C-l.
308
Intelligent Automotive Safety Systems: The Third Age Challenge
Netl
Target Goal
0.0001
Spread
3
Net2
Target Goal
0.0001
Spread
2
Net3
Target Goal
0.0001
Spread
1
Net4
Target Goal
0.0001
Spread
0.75
Net5
Target Goal
0.0001
Spread
0.5
Net6
Target Goal
0.0001
Spread
0.25
Table C-l Radial basis ANN Configuration
The simulation results from each network are shown in Figure C-9 to Figure C-14.
Radial Basis ANN, Spread 3, Target Goal 0.0001
12.-----~-----r-----T----_.r_----~----_r----_,
11
:3
10
20
30
40
50
60
70
Sample Numbers
Figure C-9 Nett: Radial Basis ANN; Spread 3, Target Goal 0.0001
309
Intelligent Automotive Safety Systems: The Third Age Challenge
Radi.al Basis ANN. Spread 2. Target Goal 0.0001
12
11
10
9
8
0
7
ti
Ql
6
:0'
:::J.
UJ
5
4
3
2
Sample Numbers
Figure C-I0 Net2: Radial Basis ANN; Spread 2, Target Goal 0.0001
Radial Basis ANN. Spread 1. Target Goal 0,0001
12
11
~
10
Q)
:0'
:::J
UJ
7
4
3
2
n
~Ir
6
5
h if
r\
.,
~
u
-
I1
~~I~
~
1\
10
~V' ~h
I/' h
I.<:l A
20
In
r.,
f) \
30
~
I
Ir
IrI- ~m~
r
/:
r--
/
8
u
.------.
~
9
0
,......,
nnl'/!
~
rn
~
I-i
1\ If'
Ir
f-----
~
I\Jl M V\~~lr\~ '"
I.--
40
50
60
70
Sam le Numbers
Figure C-B Net3: Radial Basis ANN; Spread 1, Target Goal 0.0001
310
Intelligent Automotive Safety Systems: The Third Age Challenge
Radial Basis ANN, Spread 0.75, Target Goal 0.0001
12
11
r-
r--
10
r-
-
9
r-
-
r-
8
Cl
7
U
Ql
6
:.ci'
:3
en
--,
-
r-
r-
,-,
fl
5
llMl
'r
4
!---,
,MVIlll
11
3
-1 N!
2
l;iV,{JJ 'A
o
0
10· .•••
,\~ ,~
"
'~Aln
~
20
30
40
Sample Numbers
Ir ,. . . .,
~r~ '"
A
50
7o
60
Figure C-12 Net4: Radial Basis ANN; Spread 0.75, Target Goal 0.0001
Radial Basis ANN, Spread 0.5, Target Goal 0.0001
12
11
r-
,....--,
10
r-
h
9
r-
-
h
8
l-
-
F- --,
Cl
7
U
Ql
6
"
5
h
4
,-,
:.ci'
r-
:3
en
3
'I
2
1
~ Jr-Il
10
'A
N
~
·20
'A~
1
--,
n
(ft~ If['
30
40
. Sample Numbers
A
lA lA
50
~~
60
70
Figure C-13 NetS: Radial Basis ANN; Spread 0.5, Target Goal 0.0001
311
Intelligent Automotive Safety Systems: The Third Age Challenge
Radial8asis ANN, Spread 0.25, Target Goal 0.0001
12
11
..-
..-
10
r-
"-
9
<-
10-
8
Cl
U
Ql
:.D'
::J
Cl)
<-
I--
7
~Io-
<-
I-
6
1..-
5
I-
4
1..-
IUUI~
3
n
il
2
~:A(I II' -
lA
10
......,
20
30
40
lA
rI
50
If
I---.
lA
II'
, 1111111
60
70
Sample Numbers
Figure C-14 Net6: Radial Basis ANN; Spread 0.25, Target Goal 0.0001
As the plots of simulation results shown in Figure C-9 to Figure C-14 the spread
increases the results improve and when decreases the limit gets tighter.
312
Intelligent Automotive Safety Systems: The Third Age Challenge
ANN Conclusion for experiment 1
Back propagation takes longer time to train but a much simple back propagation
neural network with only one hidden layer and 'logsig' transfer function and 'linear'
transfer function as an output gives very similar result to that of high spread (around
3 or slightly higher) radial basis neural network. These two neural networks give
higher accuracy than the rest of the neural networks tested on the thresholded
infrared images.
To make these neural networks much more accurate a much intelligent approach of
processing the infrared image should be done before feeding infrared images into the
neural network. For example different features like height, pressure and white pixel
count will achieve much accurate result.
313
Intelligent Automotive Safety Systems: The Third Age Challenge
Appendix D: Technical Data
314
ISP 30074 Issue 1
17/05/01
InfraRed Integrated Systems Ltd
Towcester Mill . Towcester
Northants NN126AD· UK
Tel:
+441327357824
Fax:
+44 1327 357825
e-mail: [email protected]
IRI 1002 Online Thermal Imager Serial Protocol
Prepared by:
Graham lones
Senior Engineer
Authorised by:
Steve Porter
Chief Engineer
Mike Mansi
Head of Thermal Imaging
Authorised by:
M. K. Robinson
Quality Manager
Controlled distribution:
Document Control
This copy received by:
This document is uncontrolled except when printed on yellow paper.
IIssue
No.
Date
C.N.No.
1
16112/02
CN0679
The copyright in this document is the property of infraRed Integrated Systems Ltd. (IRISYS). The document is supplied by IRISYS on the
express understanding that it is to be treated as confidential and that it may not be copied, used or disclosed to others in whole or in part for
any purpose except as authorised in writing by IRISYS. Unless IRISYS has accepted a contractual obligation in respect of the permitted use
of the infonnation and data contained herein, such information and data is provided without responsibility and IRISYS disclaims all liability
arising from its use. The copyright and the foregoing restrictions on reproduction, use, and disclosure extend to all media in which this
infonnation may be embodied.
ISP 30074 Iss 1 IRISYS Online Thermal Imager Serial Protocol.pdf
09:37 17112102
Page 1 of 8
ISP 30074 Issue 1
17/05/01
Contents
lIntroduction
3
2Hardware Specification
3
3Packet Format
3
4Definition of Basic Commands
3
4.1Basic Commands Transmitted to Thermal Imager
4
4.2Basic Commands Received from Thermal Jmager
5
SDefinition of Advanced Commands
7
5.1Description of Calibration Sets
7
5.2Description of Startup Mode setting
7
5.3Advanced Commands Transmitted to Thermal Imager
8
5.4Advanced Commands Received from Thermal Imager
8
Use, duplication or disclosure of data contained on this sheet is subject to the restrictions on the title page of this document.
ISP 30074 Iss 1 IRISYS Online Thermal Imager Serial Protocol.pdf
09:37 17/12/02
Page 2 of 8
ISP 30074 Issue 1
17/05/01
At power on, the Identity packet is automatically sent.
4.1 Basic Commands Transmitted to Thermal Imager
Packet Type
OxaO
Oxab
Packet Size
(Data)
No data
3 bytes
Packet Name and Data Composition
Stop imager
Send Temperature data (Multi-frame with resolution "x")
• 1 byte - Resolution (1 = 1K, 10 = O.1K)
• 1 byte - Number of Frames (0 = continuous, 1 to 255 no of frames)
• 1 byte - Frame Rate over N (1 th- transmit every frame, ... 255 every 255 th frame, 0 = every 256 frame)
Table 1 Packet Data Definition - Basic Commands to Thermal Jmager
Use, duplication or disclosure of data contained on this sheet is subject to the restrictions on the title page of this document.
ISP 30074 Iss 1 IRISYS Online Thermal Imager Serial Protocol.pdf
09:37 17/12/02
Page 4 of 8
ISP 30074 Issue 1
17/05/01
4.2 Basic Commands Received from Thermal Imager
Packet Data contents
Ox05
Packet Size
(Data)
517 bytes
Ox06
517 bytes
Ox24
18 bytes
Packet Type
Packet Name and Data Composition
Temperature data with temperature sensor value, (Res lK)
• 1 byte - Calibration Set used
• 2 bytes - serial number: (only lower 2 bytes of serial number,
see identity packet for full serial number).
• 512 bytes = 256 values of 16 bit unsigned temperature data
(Upper byte, Lower byte) in Kelvin. Resolution lK.
• 2 byte temperature sensor value
(UPPER BYTE - Integer temperature, LOWER BYTE - Fixed
point fractional part of temperature).
Lower byte (Bits 3.. 0) contain unused status flags
Temperature data with temperature sensor value, (Res O.lK)
• 1 byte - Calibration Set used
• 2 bytes - serial number: (only lower 2 bytes of serial number,
see identity packet for full serial number).
• 512 bytes = 256 values of 16 bit unsigned temperature data
(Upper byte, Lower byte) in Kelvin. Resolution O.lK.
• 2 byte temperature sensor value
(UPPER BYTE - Integer temperature, LOWER BYTE - Fixed
point fractional part of temperature).
Lower byte (Bits 3.. 0) contain unused status flags
Identity packet
• 3 bytes serial number (High byte, middle byte, lowest byte)
• 2 bytes hex version number - Upper byte is the major version
number, Lower byte is the minor version number, eg ver1.a2=
OxOl0xa2
• 1 byte - specifies the number of calibration sets in the imager
• 1 byte - indicates which calibration set it is using. (1 to n; where
n is the number of calibration sets in the imager)
• 1 byte - Default serial resolution (only used by the imager if
startup mode = continuous)
• 1 byte - Default frame rate over n (only used by the imager if
startup mode = continuous)
• 1 byte = Default startup mode - Startup in silent mode or in
continuous mode. (Silent = 0; Continuous = 1)
• 8 bytes = (Dummy chars reserved for future use).1 byte specifies which calibration set to fetch the description from. (1
to n; where n is the number of calibration sets in the imager)
Table 2 Packet Data Definition - Basic Commands received from Thermal Imager
The temperature sensor reading in the data packet gives an indication of the temperature inside the imager
enclosure.
Use, duplication or disclosure of data contained on this sheet is subject to the restrictions on the title page of this document.
ISP 30074 Iss 1 IRISYS Online Thermal Imager Serial Protocol.pdf
09:37 17112/02
Page 5 of 8
ISP 30074 Issue 1
17/05/01
Shortform guide to the IRI 1002 Online Thermal Imager Serial Protocol- Basic Commands
Hardware Specification
From the Thermal Ima2er
115200,8,n,I,no (115200 baud, 8 data bits, no parity, 1 stop bit, no handshaking)
RS 232 levels +- 12v
To the Thermal
IResolution:
I or 10
No. of~es to send:
0: continuous
L-~--1I: single frame
Imager
."\255: 255 frames
Example Packets
~
"" ~acket Bytes
Command
Send IK Resolution
Single Frame, Full
Rate
(10bytes)
Data Update rate:
1: every frame =>
255: every 255th fr
0:ev~256thframe
Command
Temperature packet
(O.IK resolution) with
temperature sensor value,
calibration set I, serial
number 129.
~\
.... 256 pixel values total
=305.3K)
=512 bytes,
OB,ED (upper, lower bytes combined gives OBED
.... 256 pixel values total
=3053d =305.3K)
=512 bytes,
ID,40 (temperature sensor value of 29.25),
(1O bytes)
(10 bytes)
SendO.lK
Resolution,
Continuous Data, Full Decimal- 170,187,171,10,0,1,0,0,170,204
11,237 (upper byte, lower byte combined gives 3053
Hex - AA,BB,02,06,Ol,OO,11, (packet preamble 7 bytes),
Decimal-170,187,171,I,I,I,O,O,170,204
Hex - AA,BB,AB,OA,Ol,Ol,OO,OO,AA,CC
Decimal-170,187,2,6,I,O,129, (packet preamble 7 bytes),
0,0,170,204 (packet completion 4 bytes)
/
Decimal-170,187,171,IO,I,I,O,O,170,204
(525 bytes total packet size)
29,64 (temperature sensor value of 29.25; 2 bytes),
Hex - AA.BB,AB,OI,OI,Ol,OO,OO,AA.CC
SendO.lK
Resolution, Single
Frame, Full Rate
Packet Bytes
OO,OO,AA,CC (packet completion 4 bytes)
Temperature packet {IK (525 bytes total packet size)
resolution) with
Decimal- 170,187,2,5,1,0,129, (packet preamble 7 bytes),
temperature sensor value,
1,49 (upper byte, lower byte combined gives 305 305K)
serial number 129.
.... 256 pixel values total 512 bytes,
=
=
29,64 (temperature sensor value of 29.25; 2 bytes),
Rate
Hex - AA.BB.AB,OA,OO,Ol,OO,OO,AA.CC
SendO.lK
Resolution,
Continuous Data,
Every 81b frame (- 1
per sec).
(1O bytes)
Hex- AA,BB,02,05,Ol,OO,l1, (packet preamble 7 bytes),
Decimal- 170,187,171,10,0,8,0,0,170,204
01,31 (upperbyte,lowerbyte combined gives 0131
Hex - AA.BB,AB,OA,OO,08,OO,OO,AA.CC
.... 256 pixel values total
0,0,170,204 (packet completion 4 bytes)
=305d =305K)
=512 bytes,
ID,40 (temperature sensor value of 29.25),
OO,OO,AA,CC (packet completion 4 bytes)
Use, duplication or disclosure of data contained on this sheet is subject to the restrictions on the title page of this document.
ISP 30074 !ss 1 IRISYS Online Therma1lmager Serial Protoco1.pdf
09:37 17/12102
Page 6 of 8
ISP 30074 Issue 1
17/05101
5 Definition of Advanced Commands
In addition to the basic start and stop commands, the Thermal Imager also has some advanced commands that
can be used to set the imager to various states.
List of commands
i. Request Identity Packet
ii. Request Number of Calibration Sets
iii. Request Calibration Set Description
iv. Use Specified Calibration Set
v. Set Startup Mode
To Imager
To Imager
To Imager
To Imager
To Imager
List of received packets
i. Number of Calibration Sets in Imager
ii. Calibration Set Description
From Imager
From Imager
5.1 Description of Calibration Sets
The Themal Imager can be calibrated with a number of temperature ranges. These options will have been
specified at purchase of the unit. Each range has a "Calibration Set" associated with it. The Imager will be
programmed with a default calibration set (temperature range). The user software can modify this through the
serial interface. Each calibration set has a text description associated with it so that the user can decide which set
is appropriate to use.
When a calibration set is used, the imager will automatically store this setting in non-volatile memory so that it
will then be the default one used by the imager, even after a power down cycle. The non-volatile memory has a
finite number of guaranteed write cycles (currently 100,000) after which, the default setting will not be stored
correctly. However, the imager will continue to switch calibration sets correctly.
Applicable Commands: Request Number of Calibration Sets, Request Calibration Set Description, Number of
Calibration Sets in imager, Calibration Set Description, Use Specified Calibration Set.
Suggested usage:
After power up during installation:
Request Number Of Calibration Sets;
Request Calibration Set Description for all calibration sets (1 to n)
Operator 1 installer to decide which one is best, send Use Specified Calibration Set command.
Request a single temperature packet and check "Calibration Set Used" byte is as expected.
(This setup is valid for that particular imager - serial number shown in Identity Packet.)
5.2 Description of Startup Mode setting
By default, the Thermal Imager will start up in "Silent" mode. However, the customer can set the Thermal
Imager to startup automatically in "Continuous" mode when the imager powers up. This setting is stored in nonvolatile memory and so is only required to be set at installation.
The options are the same as for the Temperature Packets: Serial Packet resolution lK or O.1K, FrameRateOverN
and Startup Mode. The "Number Of Frames" command is not required as it will be outputting continuous frame
data.
The imager will first output it's Identity Packet before carrying out any startup mode requirements.
Applicable Commands: Set Startup Mode
5.3 Advanced Commands Transmitted to Thermal Imager
Use. duplication or disclosure of data contained on this sheet is subject to the restrictions on the title page of this document.
ISP 30074 Iss 1 IRISYS Online Thermal Imager Serial Protocol.pdf
09:37 17112102
Page 7 of 8
ISP 30074 Issue 1
17/05/01
Ox96
Ox93
Ox94
Packet Size
(Data)
No data
No data
1 byte
Ox95
1 byte
Ox97
3 bytes
Packet Type
Packet Data Composition
Request Identity packet
Request Number of Calibration Sets
Request the description for a specified calibration set
• 1 byte - specifies which calibration set to fetch the description
from. (1 to n; where n is the number of calibration sets in the
imager)
Use Specified Calibration Set
• 1 byte - tells the imager which calibration set to use. (1 to n; where
n is the number of calibration sets in the imager)
Set Startup Mode
• 1 byte - specifies the default serial resolution 1 (1K), 10 (O.lK)
(only used by the imager if startup mode =continuous)
• 1 byte - specifies the default Framerate Over N. (0, or 1 to 255)
(only used by the imager if startup mode =continuous)
• 1 byte - Default startup mode - Startup in silent mode or in
continuous mode. (Silent =0; Continuous = 1)
Table 3 Packet Data Definition - Advanced Commands to Thermal Imager
5.4 Advanced Commands Received from Thermal Imager
Ox12
Packet Size
(Data)
1byte
Ox13
65 bytes
Packet Type
Packet Data Composition
Number of calibration sets stored in the imager
• 1 byte - specifies the number of sets in the imager
Description for the specified calibration set
• 1 byte - specifies the set number (1 to n; where n is the number of
calibration sets in the file. "0" means no set of that number)
• 64 bytes; this is the description for the specified calibration set
Table 4 Packet Data Definition - Advanced Commands to Thermal Imager
Use, duplication or disclosure of data contained on this sheet is subject to the restrictions on the title page of this document.
ISP 30074 Iss 1 IRISYS Online Thenna1 Imager Serial Protocol.pdf
09:37 17/12/02
Page 8 of 8
Intelligent Automotive Safety Systems: The Third Age Challenge
Appendix E: Publications
315
In-cabin occupant tracking using a low-cost infrared system
ImranJ Amin
Andrew J Taylor
Mechatronics Research Centre
Loughborough University
HolywelI Building, Holywell Way
Loughborough
LEll 3UZ
[email protected]
Wolfson School of Mechanical and Manufacturing Eng.
Loughborough University
Loughborough
LE113TU
aj. [email protected]
RMParkin
Mechatronics Research Centre
Loughborough University
Holywell Building, Holywell Way
Loughborough
LEll 3UZ
[email protected]
Abstract - Vehicles in future will be safer and more intelligent,
able to make appropriate and autonomous decisions to improve
safety. From intelligent cruise control to intelligent airbag
deployment conventional CCD cameras and computer vision
are being used increasingly. Two major issues relating to the use
of CCD cameras for the development of intelligent computer
vision systems are the difficulty of human detection and
invasion of privacy while driving. In this paper a low cost
infrared system is proposed as a potentially practical solution
for in-cabin monitoring of driver activities in an intelligent car.
I. INTRODUCTION
More than 3,500 people are killed and 35,000 seriously
injured in cars annually in the United Kingdom [1].
Automobile safety is now an actively researched topic and
one of the major concerns of the automobile industry, driven
partly by increasingly strict regulations on vehicle safety.
Mechatronic systems have become central to improving
automobile safety on the road. Vehicle safety sensors may
be broadly categorised according to location:
1. External vehicle safety sensors
2. In-cabin passenger safety sensors
Safety sensors mounted externally [2-7] include CCD
cameras with image processing and stereoscopic vision,
ultrasonic sensors to find distance and relative speed of
obstructing objects, and FLRS (Forward looking radar
sensors) to aid driver awareness of obstructing objects in
conditions of low visibility, such as fog, night or heavy rain.
Short range radar systems aid in reverse parking, obstruction
detection and blind spot detection.
Internal sensors may be focused on the driver, and/or
passengers, for in-cabin safety. The aim of this research is to
develop a cost-effective and robust system of occupant
tracking to enable the implementation of a variety of safety
systems and strategies. In particular the use of thermal
imaging for tracking occupant movements is being developed
as part of a system to measure their ability of movement.
This will help differentiate elderly, disabled or injured
people. By identifying particular difficulties in movement it
is proposed that measures may then be taken to make their
journey safer and easier.
Many advanced safety systems will require knowledge of
occupant position, movement and behaviour over time. For
example intelligent air bags are being developed to deploy
according to subject physique and position [7,9]. Position
tracking is sometimes achieved by stereoscopic vision while
others use single CCD cameras with image processing.
However automatic tracking of occupants in-cabin using
visual imaging encounters difficulties in distinguishing
occupants from the background and from sensitivity to
lighting variations, which in driving applications will
inevitably be extreme. Another important consideration is the
invasion of privacy associated with any continuous visual
monitoring. Ultrasonic [8] and contact based sensors being
developed to monitor occupant position [9-11] do not have
this problem but are more limited in their potential for
obtaining qualitative information.
It can be said that for in-cabin passenger tracking high
spatial resolution infrared (IR) imagers can make human
detection and tracking much easier, since they use emitted
rather than reflected radiation. However the cost of the
equipment becomes prohibitively high - a high spatial
resolution infrared camera costs more than £9,000. The
recent development of low-cost, low-resolution infrared
cameras has reduced the cost problem and provides the
ability to distinguish humans from backgrounds [12].
Low-resolution infrared thermographs [13, 14] and visual
images [15] have been used successfully for human detection
and other research. One advantage of using low-resolution
visual and thermographic images is the speed of processing,
making the system faster and cheaper. It has been shown
that the computer does not require high resolution images in
order to detect useful information from the image [15].
Another advantage of IR with regard to privacy is that
position tracking can be done with no visual features or
markings distinguishable on the image, making the system
more socially acceptable.
The approach adopted is to use a low cost, low resolution
IR camera which has recently become available at a cost of
less than one tenth of the normal higher resolution systems.
The camera will be small enough, with some repackaging, to
be mounted near the rear view mirror or on the 'A' pillar
with the driver's head and shoulders in view. The experiment
described in this paper was carried out with the IR camera
together with a CMOS webcam visual camera for
comparative purposes. The procedure is shown in Figure I.
Stage3: Infrared
I '
Stage 4: Result
rr---------I
11 Multiple Thresholding 11
I Comparison of Visual I
11
1~ Image with plots of I
I Finding Image properties I
I Occupant movement I
1i~Jlli>~sslD&...
1
:1
Plottingofcentroid
I:
I
I
Prediction of
Occupant movement
L
----------
I
I
J
Fig. 1 Overview of occupant tracking system
The work forms part of the early stages of a research
programme to fully explore the potential capability and uses
of low resolution thermal imaging for in-cabin occupant
tracking and safety systems.
11. INFRARED THERMOGRAPHY
All objects continuously emit radiation at a rate and with a
wavelength distribution that depends upon the temperature of
the object and its spectral emissivity.
A black body is an object that absorbs all incident
radiation and is a perfect radiator. The total radiation emitted
by a black body is given by Stefan's Law, which states that
'total radiation is (surface area) times (4th power of the
temperature)'[I6], mathematically expressed as:
Q=
Where £5
Ill. EXPERIMENTAL WORK
The experiment was conducted on a driving simulator
with a number of drivers following the same sequence of
driving activities. The CMOS visual camera had a resolution
288x288 pixels. Both cameras were mounted togethe:
directly in front of the driver as a baseline trial to provide
data for image processing and to assess the potential of the
imaging system as a whole. The effects of camera position
are being investigated as the next stage of the research.
A. Volunteer Selection
Eleven volunteers were selected on the basis of different
height, build, hairstyle, face structure and spectacles.
or.
4
WK .
=5.67 E -102
IS
m
relay spatial detail while a colour image reveals temperature
differentiation. When looking at an object using the infrared
imager the object is compared to a black body, an ideal
radiator with an emittance of 1. Thermal imagers find heat
transfer on the surface not beneath. An excellent example is
that of the human face [19].
The infrared device used in this experiment is an IRISYS
IRIlOOI thermal imager employing a 16xl6 pyroelectric
array, with a temperature range of -20°C to +90°C (+I50°C
with a reduced accuracy) [20]. The thermal imager views the
scene with a rotating disc module and imaging optics, and
communication is via RS-232 through an IBM-PC. The
imager can capture up to eight frames per second. A
germanium lens is used, instead of a regular glass lens, with a
20° field of view used in this case, thus from one metre the
thermal imager has a viewable region of 0.352 metres square.
the Stefan-Boltzman
constant and T is the absolute temperature.
The energy emitted by a black body is the maximum
theoretically possible for a given temperature. Objects that
are not black bodies emit only a fraction of black body
radiation. As the temperature increases the energy emitted at
any wavelength increases and the wavelength of peak
emission decreases. The thermal or infrared region contains a
waveband from 2 to 15 micrometres. This electromagnetic
spectrum range contains the maximum radiative emissions,
which are used for thermal imaging purposes [17,18].
The infrared thermal imaging device is a different
approach from other heat measuring devices, creating an
image termed a thermogram which provides mapping of
apparent temperatures. A black and white thermogram will
Fig. 2 Eleven volunteers and their corresponding infrared image
B. Driving Simulator
A fully interactive driving simulator, the STI Driving
Simulator by Systems Technology Inc, was used for the
experiment. The STI Driving Simulator is very suitable for
research purposes as it is one of the most stable driving
simulation packages available, with 40 years of development.
This simulator uses PC Microsoft® Windows based control
and customised simulations can be created using a very basic
script. The simulation is projected by a data projector onto a
4m by 3m screen and driven by the controls inside a Ford
Scorpio car. The steering, brake, accelerator and speedometer
are connected to absolute encoders, which give analogue
readings to the Data Acquisition Card (DAC). This DAC is
connected and configured in the PC with an installed copy of
the STI Driving simulator software.
verification purposes during the image processing analysis.
E. Image Acquisition software
'I-Quire' software [21], developed by the author to
perform the task of data acquisition for the thermal imager,
and Video for Windows (VFW) supported the visual devices.
The platform used for development is National Instruments
LabWindows/CVI which has an ANSI C environment. This
software can acquire up to 4 frames per second saving bmp
and infrared data simultaneously. The duration of the image
acquisition can be from 1 second to unlimited and can be
paused during the acquisition.
[email protected] IU€t5§!,lb
T_~.
1'Iro.o:I~
:pr-~
.,....----~'~"~-~.-
(A)
;Jltr~ :!-j..,
l·
Fig. 3 (A) Simulation during experiment with image acquisition
system, (B) Simulation control PC and Supervision PC
The DAC takes 3 inputs in the form of analogue signals
from steering wheel movement, brake pedal and accelerator.
The projector simulation is also shown on the supervision PC
that is used for autopilot mode, centring of steering with the
screen and review of scenarios before running experiments.
The simulation control PC delivers the main control of the
simulation software and graphics. The encoder counts from
the DAC are read directly from the STI Driving Simulator.
"""I'
1·=~
J ''-'1'.,
( ~J~;~:
l1
I
'rtlMdll,lfsqf..... ~j
' ..... '... 1
- .... 1
u.._"'
....
~
Fig. 5 Acquisition software used in the experiment
The image frequency used in the experiment was 2 frames
per second and image acquisition was done for the whole
length ofthe simulation.
F. Experimentation
Fig. 4 Sensors mounted focusing on volunteer during experiment
C. Scenarios
Scenarios for the STI driving simulator are created using a
basic script language. In this experiment a single scenario
was used, the duration of which ranged from 350 seconds to
500 seconds. The scenario starts from an urban area with a
single lane and continues into heavy traffic, intersection
crossings, traffic signals, pedestrians crossing the street, hills
and bends. Further on the scenario develops into a long
straight dual lane expressway until the session ends.
D. Sensors
For this experiment the IRISYS infrared imager and
Webcam were mounted together on the driving simulator at a
distance of one metre away from the subject. The Webcam
provides an essential visualization tool for comparison and
Im:ter
~I
Fig. 6 Experimental setup
The experiment was conducted with an ambient
temperature of 20 degrees and a trial run was undertaken by
each volunteer before the start of the experiment. Around
800 visual images and thermograms were taken for each
driver. During the experiment certain instructions were given
to e.g. perform overtaking manoeuvres, slow down, look
right and left at intersections and to simulate a crash situation
by moving the head onto the steering wheel.
IV. IMAGE PROCESSING
The infrared images taken are analysed using MatLAB.
These images are linearly interpolated from 16xl6 pixe1s to
121x121 pixels. Interpolation does not add any extra
infonnation into the image but helps low-resolution infrared
images to be visually analysed and gives a greater number of
pixe1s to work on.
Four types of different temperature ranges are found in the
images that can be thresholded. These are:
1. Background
2. Covered skin (with clothes or hair)
3. Face
3. Eyes, mouth and forehead.
segmented images. The software shows the plotted centroid
with the last thresholded image. It can be seen from Fig. 9
that different regional classification is able to broadly
classify the tasks of the driver.
20
40
60
80
100
20
~.od\qAI.
40
I
60
80
f'1oIl...g.
100
I
120
20
40
60
BO
&9
Background
eliminaoon
Subject looking forward/driving
Subject putting seatbelt on
Head and covered skin
(notice rermval of seat belt on
right side because the
fm1>erature decreases)
Face (without hair)
Eyes and
mouth
Fig. 7 Multi thresholding of interpolated infrared images
Thresholded images for face separation are used further in
the imaging tracking analysis. Facing forward is taken as a
reference image from which motions are tracked. A
comparison of driving task using visual and infrared
thresholded image is shown Fig. 8.
Fig. 9 Plotting of centroid from the thresholded images
Also it can be seen from Fig. 10 that the subject is wearing
glasses. At the start of the journey the subjects put on a
seatbe1t, then drove mostly straight but encountering
intersections, thus the centroids deviate to the left or right as
the head turns.
Measurement of the subject's head movement is by simple
calculation as the field of view in infrared is 352 by 352 mm.
It can be seen from Fig. 11 that the distance between the far
left centroid and the centroid for looking forward, is 20
pixels. Thus by simple mathematics the furthest head
movement is calculated to be 116 mm in the case shown
below.
wearing glasses
Mostly looking forward
20
x
x
20
R.od lqi.
40
I
60
80
f'1oIl_
100
I
120
120
130
Wearing seatbelt
Fig. 8 Comparison of different driving tasks of occupant using
conventional camera and multiple thresholded infrared images
Software is written to plot the centroid of the face-
Fig. 10 Subject with glasses driving
VI. ACKNOWLEDGMENTS
The authors gratefully acknowledge the contributions of
A.FJuna, FJunejo and the participating volunteers.
VII. REFERENCES
EN.REFLIST
~
I
I
I
I
I
I
I
I
I
~
)(
X
~
)(
x
xx
.
j'"
116 mm
Fig. 11 Measuring distance from the centroids plot
Currently MatLAB is used for offline analysis as the system
is in the experimental stages. The real time system will be
implemented subsequently using National Instruments/CVI
language due to the speed of data acquisition and image
processing required.
V. RESULTS AND OBSERVATIONS
Experimental data from eleven volunteers, each containing
800 samples of infrared, was applied to the tracking
algorithm. The thresholded infrared samples were then
compared with visual data. Thus it can be said that in-cabin
tracking using Iow-resolution infrared images has been
achieved. The system can reliably locate the occupant with
an accuracy of +/-15 mm.
Although the information received from the infrared
sensor is two-dimensional it can give results comparable in
accuracy with ultrasonic position sensors or contact based
sensors, as the contact based sensors only track position
based on seating position of the occupant. In comparison
with visual image detection the infrared is far superior in
detecting human motion within a wide range of conditions.
Use of Iow resolution thermal imaging for tracking
occupant movements is now being developed as a key part of
a multi-sensor system to measure drivers' capabilities and
limitations of movement. Together with driving task analysis
this will help differentiate elderly, disabled or injured people.
By identifying limitations or difficulties of a particular
person measures may be taken to make their journey safer
and easier. A number of other potential applications and
benefits of infrared imaging are also being identified as part
of this work.
Elsevier Editorial System(tm) for Measurement
Manuscript Draft
Manuscript Number:
Title: Automated people counting by using low-resolution infrared and visual cameras
Article Type: Research Paper
Section/Category:
Keywords: people-counting, imaging, infrared, artificial neural network (ANN)
Corresponding Author: Mr. Imran Amin,
Corresponding Author's Institution: Loughborough University
First Author: Imran Amin
Order of Authors: Imran Amin; Andrew Taylor; Faraz Junejo; Amin AI-Habaibeh; Robert M Parkin
Manuscript Region of Origin:
Abstract: Non-contact counting of people in a specified area has many applications for safety, security and
commercial purposes. Visible sensors have inherent limitations for this task, being sensitive to variations in
ambient lighting and colours in the scene. Infrared imaging can overcome many of these problems but
normally hardware costs are prohibitively expensive. A system for counting people in a scene using a
combination of low cost, low resolution visual and infrared cameras is presented in this paper. The aim of
this research was to assess the potential accuracy and robustness of systems using low resolution images.
This approach results in considerable savings on hardware costs, enabling the development of systems
which may be implemented in a wide range of applications. The results of eighteen experiments show that
the system can be accurate to within 3% over a wide range of lighting conditions.
Elsevier Editorial System(tm) for Measurement
Manuscript Draft
Manuscript Number:
Title: Automated people counting by using low-resolution infrared and visual cameras
Article Type: Research Paper
Section/Category:
Keywords: people-counting, imaging, infrared, artificial neural network (ANN)
Corresponding Author: Mr. Imran Amin,
Corresponding Author's Institution: Loughborough University
First Author: Imran Amin
Order of Authors: Imran Amin; Andrew Taylor; Faraz Junejo; Amin AI-Habaibeh; Robert M Parkin
Manuscript Region of Origin:
Abstract: Non-contact counting of people in a specified area has many applications for safety, security and
commercial purposes. Visible sensors have inherent limitations for this task, being sensitive to variations in
ambient lighting and colours in the scene. Infrared imagil}g can overcome many of these problems but
normally hardware costs are prohibitively expensive. A system for counting people in a scene using a
combination of low cost, low resolution visual and infrared cameras is presented in this paper. The aim of
this research was to assess the potential accuracy and robustness of systems using low resolution images.
This approach results in considerable savings on hardware costs, enabling the development of systems
which may be implemented in a wide range of applications. The results of eighteen experiments show that
the system can be accurate to within 3% over a wide range of lighting conditions.
Elsevier Editorial System(tm) for Measurement
Manuscript Draft
Manuscript Number:
Title: Automated people counting by using low-resolution infrared and visual cameras
Article Type: Research Paper
Section/Category:
Keywords: people-counting, imaging, infrared, artificial neural network (ANN)
Corresponding Author: Mr. Imran Amin,
Corresponding Author's Institution: Loughborough University
First Author: Imran Amin
Order of Authors: Imran Amin; Andrew Taylor; Faraz Junejo; Amin AI-Habaibeh; Robert M Parkin
Manuscript Region of Origin:
Abstract: Non-contact counting of people in a specified area has many applications for safety, security and
commercial purposes. Visible sensors have inherent limitations for this task, being sensitive to variations in
ambient lighting and colours in the scene. Infrared imaging can overcome many of these problems but
normally hardware costs are prohibitively expensive. A system for counting people in a scene using a
combination of low cost, low resolution visual and infrared cameras is presented in this paper. The aim of
this research was to assess the potential accuracy and robustness of systems using low resolution images.
This approach results in considerable savings on hardware costs, enabling the development of systems
which may be implemented in a wide range of applications. The results of eighteen experiments show that
the system can be accurate to within 3% over a wide range of lighting conditions.
nU5cript : Automated counting paper _ no fig
Automated people counting by using low-resolution infrared and visual
cameras
I. J. Amin, A. J. Taylor, F. Junejo, A. Al-Habaibeh, , R. M. Parkin
The Wolfson School of Mechanical and Manufacturing Engineering
Loughborough University
Loughborough
Leicestershire
LEl13TU
Abstract
Non-contact counting ofpeople in a specified area has many applications for safety, security
and commercial purposes. Visible sensors have inherent limitations for this task, being
sensitive to variations in ambient lighting and colours in the scene. Infrared imaging can
overcome many ofthese problems but normally hardware costs are prohibitively expensive.
A system for counting people in a scene using a combination oflow cost, low resolution
visual and infrared cameras is presented in this paper. The aim of this research was to assess
the potential accuracy and robustness of systems using low resolution images. This approach
results in considerable savings on hardware costs, enabling the development of systems which
may be implemented in a wide range of applications. The results of eighteen experiments
show that the system can be accurate to within 3% over a wide range oflighting conditions.
Keywords: people-counting, imaging, infrared, artificial neural network (ANN)
Notation
p
Energy Radiated
..t
Wavelength
T
Temperature (Kelvin)
h
Plank's Constant
Cc
Velocity oflight
b
Boltzman Constant
lV
Radiated energy
E
Emissivity
..,
Boltzman constant
P(q)
Interpolated value
L;(q)
Lagrange polynomial
q
Point at which interpolation takes place
/;
Known values on the grid at points (qj)
t;
Desired or target response for ith unit
y;
Actually produced response for ith unit
E
Error for a single training pattern in neural network
n
Transfer function output value
°lh
Output thresholded image
CXth
Thresholding value
r
Original image
d
Constant
c
Constant
q
Grayscale image
x
Infrared image
t5
Average body heat
m
Thresholded infrared image
2
1. Introduction
Automated counting is an active research topic in many areas including biology [1],
medicine, quality control and industrial machine vision processes amongst others. There are
many situations where it is useful or essential to count people and numerous automated
people-counting systems have been developed over the years. A variety of contact based
sensors are in use, such as pedestrian barriers on entrances to public buildings and gateways.
Most commercially available non-contact based counters use infrared beams or ultrasonic
sensors, and specialized human information sensors are also developed for this task [2].
However the most commonly used non-contact system still remains the visual camera [3-5].
One present disadvantage with visual counting systems is the cost- a high spatial
resolution visual camera and a frame grabber required for the system are still fairly expensive
items. However a more fundamental problem, even with high spatial resolution cameras,
remains the inaccuracy associated with visual detection of people. If a person is wearing the
same shades of grey as the background it is difficult to distinguish between the background
and the clothes. Also there are no reliable ways of distinguishing with accuracy a person from
similar objects. These objects in the background are one ofthe main concerns, commonly
raising false alarms in many automated people counting systems. Generally it can be said that
background separation is not an easy task. Furthermore visual automated counting systems
can only work in the presence of ambient lighting such as an office environment, sunlight, or
other types oflighting. In case of emergencies, such as fife or blackouts, the system will
malfunction during evacuation of the building and thus could be rendered useless at crucial
times. Similar situations can occur with exterior use of visual people counters [5], there will
be false alarms during night time ifthere is no special lighting arrangement in the area under
consideration.
Thus a system is proposed to overcome these problems by using a low cost infrared
thermal imager together with a visual camera. The visual camera uses an image-processing
algorithm that can distinguish between people and objects with an accuracy of about 12%.
3
This system, developed by Schofield et al [4], uses visual automated counting but can be
modified easily to accommodate low spatial resolution visual images. The working principle
is based on the background training of visual images using a neural network.
2. Thermallmaging
Thermal infrared (IR) imaging sensors respond to emitted, more than reflected, radiation.
All objects emit heat by three means: conduction, convection and radiation.
Conduction transfers heat through solid objects; convection transfers heat through fluids;
radiation transfers heat through electromagnetic radiation.
Objects continuously radiate heat with certain wavelengths, dependent upon the
temperature of the radiating object and its spectral emissivity. As the object temperature
increases the radiation increases. The radiation emitted includes the infrared emission which
consists of electromagnetic wavelengths between 0.7 Jlm to 100 Jlm. Small ranges of infrared
emission from the objects are detected by the thermal imager and then made visible as an
image in the form of a thermogram - a mapping of apparent temperatures.
The concept behind infrared emission detection of the thermal imager is the assumption
that a black body is a perfect radiator; it emits and absorbs all incident energy. The energy
emission for the black body is the greatest possible for energy emission for that particular
temperature. Radiation power emitted by a black body as given by Plank's radiation law [6]
is:
p(A,T) =
21t~~/ [exp(~~ )-Ir
p= Energy Radiated
A= Wavelength
T= Temperature (Kelvin)
h= Plank's Constant
Cc =
Velocity of light
b= Boltzman Constant
4
[I]
Real objects are not perfect emitters or absorbers. Thus emissivity (E) of the real
surface is defined as the ratio of thermal radiation emitted by a surface at given temperature to
that ofa black body for the same temperature, spectral and directional conditions [7,8]. Thus
the emissivity ofa black body is 1 and all other real surface emissivities will be between 1
andO.
According to the Stefan Bo Itzman Law of emissivity radiation:
W=EIlT4
[2]
w = Radiated energy
E = emissivity
" = Boltzman constant
(5. 1067x
8
;
~
)
T = Temperature (Kelvin)
Thermal imaging converts thennal radiation into a digital signal which is then
converted into a visible image. This study uses a newly developed thermal imager of type
IRYSIS IRI 1001. This offers many advantages including low cost, wide temperature
measurement range and the capability to capture images on an IBM-PC via an RS-232C port.
The thennal imager is housed in an aluminium casing of 100 mm by 100 mm complete with
optics, pyroelectric detector [9], chopping motor and optical modulator. It has a temperature
measurement range of-20 to 90°C with an accuracy of+/-O.l °C [10]. Although it is a low
resolution, 16 x 16 pixel, thermal imager it can be used to display images of up to 128x128
pixels using bilinear or bicubic interpolation. The interpolation process estimates values of
intermediate components ofcontinuous function in discrete samples. An interpolation
technique does not add extra information into the image but can provide better thennal images
for human perception. For bicubic interpolation, the output pixel value is the weighted
average of the pixels in the nearest 4 x 4 neighbourhood. Mathematically, bicubic
interpolation can be described as follows:
Let L; be a third degree polynomial. The Lagrange polynomial interpolation is given by [11]
5
3
P(q)
=L/;L;(q)
[3]
;=0
Where,
q =Point at which interpolation takes place
P(q)= interpolated value
/; =Known values on the grid at points (q;)
L;(q) =Lagrange polynomial, for example
L;(q)
=rr:,.;,k=O(q-qk)/(q; -qk)
Previous research [4] has shown that low resolution visual images give similar visual
information to that of high resolution devices, as the visual information will be processed by
computer. A similar approach is used for low-resolution thermal images. A low resolution
thermal imager will cost much less than a typical high resolution thermal imager, around one
tenth of the cost, and will be much smaller than a conventional thermal imager. Additionally
the low resolution imager is specially designed for embedded systems, where data can be
directly streamed through an RS232 connection to the computer for on-line monitoring and
off-line analysis.
3. Neural Networks
An artificial neural network (ANN) is an information-processing paradigm inspired by the
way in which the densely interconnected, parallel structure of the human brain processes
information. Neural networks resemble the human brain in the following two ways:
A neural netwotk acquires knowledge through learning.
A neural netwotk's knowledge is stored within inter-neuron connection strengths
known as synaptic weights.
Artificial neural networks are collections of mathematical models that emulate some of the
observed properties of biological nervous systems and draw on the analogies of adaptive
biologicalleaming. The key element of the ANN paradigm is the novel structure of the
6
infonnation processing system. It is composed ofa large number of highly interconnected
processing elements that are analogous to neurons and are tied together with weighted
connections that are analogous to synapses.
The main advantage of using a neural network is the full automation ofthe learning and
classification processes, allowing them to be implemented in fully automated monitoring
systems, such as people counting, to recognize and classifY different patterns without human
involvement. This eliminates any error or lapses associated with human concentration during
a repetitive task.
Neural networks are composed ofsimple elements operating in parallel, inspired by biological
nervous systems as mentioned previously. As in nature, the network function is determined
largely by the connections between elements. Some Neural networks are classified as feedforward while others are recurrent (Le., implement feedback) depending on how data is
processed through the network. Another way of classifYing neural network types is by their
method ofleaming or training, as some ofthe neural networks employ supervised training
while others are referred to as unsupervised or self-organizing networks. The selection of
supervised or unsupervised network is greatly dependent on the data to be processed for the
training of the network.
During supervised learning of an ANN an input stimulus is applied that results in an output
response. Then this response is compared with a desired output i.e. the target response. If the
actual response differs from the target response, the neural network generates an error signal.
A popular measure ofthe error 'E' for a single training pattern, is the sum of square
differences Le [12].
[4]
where,
t; = desired or target response for ith unit,
y; = actually produced response for ith unit.
7
The error "E" is then used to caIcu late the adjustment that should be made to the network's
synaptic weights so that the actual output matches the target output.
In contrast to supervised learning the case ofunsupelVised learning does not require a
teacher; i.e. no target output is required. It is usually found in the context of recurrent and
competitive nets. In the case of unsupervised learning there is no separation of the training set
into input and output pairs during the training session, the neural net receives as its input
many different excitations, or input patterns, and it arbitrarily organizes the patterns into
categories. When a stimulus is later applied the neural net provides an ou tput response
indicating the class to which the stimulus belongs. If a class cannot be found for the input
stimulus, a new class is generated. However, it should be noted that even though unsupervised
learning does not require a teacher, it requires guidelines to determine how it will form
groups. Grouping may be based on shape, colour, or material consistency or on some other
property of the object [12, 13].
3.1. The 8ackpropagation Neural Network
Backpropagation Neural Networks are one ofthe most commonly used neural network
structures, as they are simple and effective, and have been used successfully for a wide
variety ofapplications such as speech or voice recognition, image pattern recognition,
medical diagnosis, and automatic controls. It is a supervised neural network, consisting of"n"
numbers of neurons connected together to form an input layer, hidden layers and an output
layer. The input and output layers selVe as nodes to buffer input and output for the model and
the hidden layer serves to provide a means for input relations to be represented in the output.
Before any data has been run through the network, the weights for the no des are randomly
chosen, which makes the network very much like a newborn's brain, developed but without
knowledge. When presented with an input pattern each input node takes the value ofthe
corresponding attribute in the input pattern. These values are then "fired"', at which time each
node in the hidden layer multiplies each attribute value by a weight and adds them together. If
this is above the node's threshold value, it fires a value of "1 "'; otherwise it fIres a value of
8
"0". The same process is repeated in the output layer with the values from the hidden layer,
and ifthe threshold value is exceeded, the input pattern is given the classification. Once a
classification has been given it is compared to the actual, i.e. desired, classification and the
error is fed back (backpropagated) to the neural network and used to adjust the weights such
that the error decreases with each iteration and the neural model gets closer and closer to
producing the desired output. This process is known as "training". The back propagation
neural network used in this study uses a sigmoid function in the hidden layer and a linear
function in the output layer [12]. Both functions can be expressed respectively as follows:
3.2. The RAM based Neural Network
Most conventional neural network training procedures, as mentioned above, are used to
develop the required behaviour in a learning system, having assumed that the 'weight'
parameters in which the system's knowledge is stored can be positive or negative and
unboundedly large in size. These analogue weights, and the algorithms by which they are
adapted, are not well suited to hardware implementation. However, in this study, a sequential
(RAM based) neural network has been used which uses binary weights, i.e. Oil values, stored
in RAM memory blocks which themselves play the role of the 'neurons' in the system. This
approach, sometimes called 'weightless neural computing'. has many advantages over other
neural networks, such as fast network training. It uses 'one-shot' learning procedures vel)'
different from the iterative ones of conventional neural networks and furthermore they can
operate well on low resolution images. In addition to this, in the case of RAM based neural
networks, the bit-stream communication between RAM neurons, rather than being a
hindrance to the system when learning, is actively beneficial in promotinggeneraiisation.
This refers to the neural network producing reasonable outputs for inputs not encountered
during training (learning), whereas, other networks have to introduce such a 'blurring' of the
input (so that in effect a wider range of patterns are seen during training) in a much more
artificial way [14].
9
RAM based Neural Network Architecture:
As shown in Figure 5, the basic architecture is as follows:
•
the input vector is divided into parts; each part is connected to the address inputs ofa
I-Bit-RAMunit.
•
The output of all the RAMs within one discriminator are summed. The number of
discriminators needed in a network is determined by the number of classes which
need to be distinguished by the network.
The I-Bit-RAM unit, is a device which can store one bit o finformation for each input
address. A control input is available to switch the mode of the RAM between 'Write' and
'Read' for learning and recall. Initially all memory units are set to '0'. During the learn
('Write') mode the memory is set to 'I' for each supplied address; in the recall ('Read') mode
the output is returned for each supplied address, either '1' (ifthe pattern was learned) or '0' (if
the pattern was not learned).
The discriminator is the device which performs the generalization. It consists of several
RAMs and one node which sums the outputs of the RAMs in recall mode. The discriminator
is connected to the who le input vector; each RAM within the discriminator is connected to a
part of this vector, so that each input bit is connected to exactly one RAM. The connections
are preferably chosen by random.
4. Experimental work
The experiment was conducted by mounting the low-cost visual imaging device (Webcam)
and the IRISYS IRH 001 thermal imager looking vertically down. Markers are placed on the
floor under consideration so that both infrared imager and visual camera are sharing the same
information. The visual imager has a much wider field of view than the thermal imager, thus
only a cropped visual view is taken into consideration.
Special software was developed using National Instruments LabWindows/CVI [15]. This
software communicates with the visual imager using a USB 1.1 interface and the infrared
imager using RS-232C. The data is stored omine for further analysis. The software is flexible
10
enough to store at different frame rates and different resolutions, and also displays the data
which is being stored.
The resolution selected for VGA is 320x240 pixels while infrared resolution is fixed at 16xl6
pixels .. The images are taken at 4 frames per second (FPS), even though the infrared imager
is capable of up to 8FPS, as here analysis is based mostly on individual images rather than
time-based imaging analysis.
Three control experiment scenarios were used. Each scenario was based on six
experiments with differences in position, movement of subjects and different lighting
conditions. The background images with no subjects were also taken each time. Each
experiment conducted contained around 150 visual and infrared samples of data stored on a
hard disk. The length of each experiment varied from 3 to 5 minutes depending upon the
subjects involved, and during all experiments the data acquisition software was kept running.
The three scenarios were as follows:
4.1. Elevator camera (static)
In this scenario ten volunteers were involved which resulted in thirty tests. This simulates the
elevator surveillance camera with a restriction of any volunteer leaving the scene during the
length of each test. During each test volunteers are asked to stand for five seconds at random
positions in the area which is being monitored. Also the number of volunteers increased as the
test progressed. The maximum number of volunteers in tests was five and each test was
repeated five times with random selection of volunteers.
4.2. Gate Camera
This scenario simulates the gate camera for counting. Volunteers were asked to enter the
scene from one side and leave on the opposite side. Thirty tests were conducted, with each
test repeated five times with a random selection of volunteers. Two special conditions, i.e. one
11
person standing within the gate for a certain period of time and one person stopping and
returning to where helshe entered from, were included in these tests.
4.3. Elevator camera (dynamic)
This scenario simulates the actual elevator surveillance camera. The volunteers in this
scenario are allowed to leave and enter the scene but only from one side which is the elevator
door. The maximum number of volunteers in tests was ten. During the test volunteers were
given specific instructions when to enter or leave the scene. It also simulates the peak timing
as well as off-peak timing during the day. As in the previous scenarios all volunteers were
selected randomly for each test to maintain the validity of the final result.
5. Image processing strategy
The visual system used is lower cost than traditional CCTV cameras, around III oth ofthe
cost. The low cost CMOS sensor used by the visual system also develops a noise factor,
which presents a major issue to be considered during the visual analysis. Thus images with
simple subtraction with respect to the reference scene do not provide a consistent image in our
case which can be thresholded.
The visual analysis carried out is very similar to that done by Schofield et al [4] except that
the equipment used is low cost. The thermal imaging analysis is also done separately. The
results of each analysis are then further compared to increase the accuracy ofthe system.
These showed thatthe system can be developed to be capable of counting in a smoked filled
room and other emergency situations, which is not the case with conventional visual counting
systems. In the following sections both visual and infrared data are analysed separately and
then the combined results are discussed.
5.1. Visual Analysis
The visual analysis uses the background identification technique employed by Schofield et al
[4]. This system avoids any standard approach which fails to take light variations into
12
account, hence it is independent oflight intensities in the image. Thus this process is chosen
for visual analysis for the development of our system with some modifications, for example
we do not require location information in an image as it is not necessary in the proposed
application. The visual counting system developed should have the following characteristics:
Accuracy
Approximately 10%
Error
Maximum of +/-1 error in 4 to 10 people in a
scene
Lighting conditions
Adaptable to any indoor lighting conditions
Adaptability
Most scenes in indoors bu ildings
Table 1. Design guidelines for visual counting system
5.1.1. Stage 1a: Pre-Processing
The pre-processing stage for visual analysis consists of re sizing and thresholding. The initial
image acquired from the experiment is 288x288 pixels. This is then reduced to 72x72 pixels.
The reduction in resolution allows faster processing and a faster counting rate with negligible
degradation in the thresholding result For example, for the initial image of288x288 pixels
thresholding takes about 45 seconds using a fast processing speed while 72x72 pixels takes
only about 25 seconds using MatLAB. This will improve significantly after final
development of the system using a programming language such as C or C++.
Following resizing a reference image from each experiment is taken. Reference images are
merely background images with no people in the scene. These reference images are
thresholded not by the constant greyscale value but by applying adaptive local thresholding.
The neighbouring pixel will allow the intensity ofpixels to be compared with each other. If
the comparison ofthese pixels is high, up to a certain value set by another variable' CXth
pixel is turned black otherwise white. Thus it can be mathematically expressed as:
13
"
the
Where
ath = Thresholding value
r = original image
0th
= output thresholded image
i=lto72
j=lto72
Here' cx th ' is the global thresholding value of the image being processed. To calculate this
value l/3ro pixel values ofimages are randomly selected. The difference ofintensities of these
pixels is taken from their diagonal neighbour. Here two constants 'c' and od' are introduced in
the thresho Iding value of' cx th '. After summing all of the intensity difference values the fmal
value is multiplied by a constant 'c', which is less than 1. The value acquired is then added to
the constant value of'd'. The thresholding expression is mathematically expressed as:
Where
a 1h = Thresholding value
d = Constant
c = Constant
q = Grayscale image
k = 1 to 72 (random values)
I = 1 to 72 (random values)
The optimal values of'c' and od' are found by experimenting with the visual images taken
during the experiment.
14
- - - - - - - - - - - -
5.1.2. Stage 2a: Background Identification
The background identification is based on the RAM based neural network creation and
training of that network. Only background images are trained using this network.
The thresholded image is divided into 4x4-sections, with 18 sections in each row and 324
altogether in the 72x72 pixel image. We consider each 4x4-section containing 16 pixels
divided further into four sections, which are termed sub-sections. These sub-sections,
containing four pixels each, are then randomly selected, and this selection remains the same
over the life ofa neural network. These randomly selected sub-sections are used as the
addresses of RAM. For each 4x4 section created and randomly selected 4 sub-sections create
a single classifier.
Training ofa RAM based network is done by reading the 4 pixels from each group in 4x4
section outputs 1 to the RAM of that certain address as shown in Figure 5. For example ifthe
value ofthe 4 randomly selected pixels is 010 1 then it outputs 1 to the corresponding memory
output of that address. Then it starts summing up all values in the memory addresses, which
are specific for each individual 4 pixel group. Thus for every section of the image seen it
outputs 1 into the RAM ofthat section address. It goes on until all the background samples
are trained for that network. There is no reason to run the samples again through the network
for a background already seen, as the resu It will always be the same for that particular image.
To simulate the image using a trained network a thresholded sample ofthe image is fed into
the network. The sample image is then divided into the random sections, which are the same
as that ofthe trained network. The addresses of sample images are compared with the trained
network valu es. If the network has already seen the same section during training it ou tputs
'1 " if the network hasn't seen anything like the section it outputs '0'. An output image is
constructed with 1 's as the background and O's as the unseen object during the training. After
inverting the image the unseen objects or people then appear as a cluster of 1 's in that image.
51 reference samples are used for training ofthe RAM-based neural network.
15
A 5x5 section is scanned over the output image by the neural network. For highest counts
found in 5x5 sections in the image a count is incremented, and the 3x3 section in the middle is
set to zero whereas the 16 outside values are halved. This process is continued until a certain
cut-offvalue is achieved for the image. An optimized cut-off value is found by comparing the
result found with the actual result
5.2. Infrared Analysis
For the development of the low-resolution infrared counting system certain guidelines were
laid down as follows:
The infrared analysis system developed will be used in conjunction with the visual system but
can be used as a stand-alone system with very slight modifications.
Accuracy
Approximately 5%
Error
Maximum of+l-l error in 4 to 10 people in a
scene
To most indoor building conditions and
Adaptability
objects in scene (except extreme temperature
conditions, e.g. +50 oCelsius)
Completely insensitive to lighting variations
Lighting Variations
in a scene
Table 2. Design guidelines for Infrared Counting system
5.2.1. Stage 1b: Pre-processing
Infrared data taken from the experiment are taken offline into MATLAB. The raw
infrared data taken from the experiment is interpolated to find the 'average body heat'.
The temperature of a person is generally higher than the background, except in very hot
16
areas such as desert, but as this experiment is conducted inside a building we can assume
a reasonably consistent temperature difference. Average heat ofthe background image in
this experiment is found to be:
averageheat
L,-,-p_ix_el_s _ 2430C I .
= . eslUS
=
256
The internal temperature ofthe IRISYS® infmred camem remains 32.375 0 Celsius. Thus
the overaII temperature ranges for the duration of our experiment remain within
min bodytemp
=27° C
maxbodytemp=32°C
0= 29.5°Celsius
where
0= Average body heat
The 'avemge body heat' calculated from the infrared data is then used as the thresholding
value for the experiments conducted. This 'avemge body heat' varies upon weather
conditions and location of the experiment, such as whether it is conducted indoors or
outdoors.
Infrared images ofl6x16 pixels are processed using the foIIowing equation:
{
[m ](j,k) = 0lifx~O
ifx~o
where
x = Infrared image
0= Avemge body heat
j=16;
k=16;
Let x be an element ofthe original matrix ofl6x16 elements from the infrared imager, m
is an element ofthe thresholded matrix and 0 is the average body heat.
17
The infrared images after thresholding at average body heat give a distinguishable result
that can be used for object recognition. Butthis is true only for small numbers of people
as when the area under consideration becomes crowded then the algorithm becomes
unreliable and hence further processing is necessary.
5.2.2. Stage 2b: Back Propagation Neural Network
For infrared image counting neural network areas are selected as the images are small, up
to 16x16 pixels. After thresholding the infrared images are trained on back propagation
neural networks.
A back propagation neural network is created. The specification for the fmal network
selected is as follows:
Inputs
256
Hidden Layer
Hidden Layer Neurons
280
Hidden Layer Function
Sigmoid Function
Output Layer Neurons
Output Layer Function
PureLin Function
Training Performance goal achieved
0.00642496
Epochs
500
Learning rate
0.005
Training Samples
360
Table 3. Configuration ofoptimiLed neural network for Infrared Analysis
18
Training of the backpropagation neural network is done by using twenty (20) samples
from all eighteen (18) experiments as fed into the network.
6. Results
6.1.1. Infrared neural network simulation and results
The results acquired from the infrared data are plotted in the form of percentage error,
with the error plot based on the simulation of200 samples selected from each of18
experiments. The error tends to increase as the numberofpeople counted in the scene
increases though there is much scatter and the error does not continue to rise as the
numbers approach the maximum often.
6.1.2. Visual RAM based neural network simulation and result
As for visual images, the system is within 5% for less than six people in each scene. But as
the actual number of people in each scene increases the error percentage increases to around
12%. This is due to people standing very close to each other, as would be the case for
example at peak time in elevators. To overcome this error in the system infrared and visual
results are combined.
6.1.3. Combined results of visual and infrared systems
It can be seen from the above results that an infrared system is capable of predicting a high
density of people with high accuracy, whereas a visual system has proved to be more reliable
for predicting lower densities of people. Therefore, in order to optimize the overall accuracy
of the system, fusion of results from thermal and visual systems is carried out by taking
percentage error and shifting the weight of results with less error percentage. As a result, as
shown in Figure 9, the maximum percentage error has been reduced to 3%, even for scenarios
containing a high density of people.
19
Conclusions
Combining two automated counting systems, visual and infrared, has been shown to give
significant improvements in accuracy. The percentage error on % is far more accurate than
either the visual RAM based system alone. This percentage error remains at 3% for more than
six people in the experiment with both visual and infrared sensing. This was not the case with
the visual counting system working without the infrared camera. The low resolution, low cost
infrared imager can provide slightly less accurate but vel)' reliable counting in low or zero
light conditions, making it suitable for emergency situations.
References
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
Marotz, 1., e. Lubbert, and W. Eisenbe{ss], Effective object recognition for automated
counting ofcolonies in Petri dishes (automated colony counting)I. Computer Methods and
Programs in Biomedicine, 2001. 66(2-3): p. 183-198.
Morinaka, K., et aL, Human information sensor. Sensors and Actuators A: Physica~ 1998.
66(1-3): p. 1-8.
Chow, T. W.S. and S.-Y. Coo, Industrial neural vision system for underground railway station
platform surveillance. Advanced Engineering Informatics, 2002. 16(1):p. 73-83.
Schofield, A.J., T.1. Stonham, and P.A. Mehta, Automatedpeople counting to aid lift control.
Automation in Construction, 1997. 6(5-6): p. 437-445.
Sacch~ C., et al, Advanced image-processing toolsfor counting people in tourist sitemonitoringapplications*I. Signal Processing, 2001. 81(5):p. 1017-1040.
Bumay, S.G., T.L. Williarns, and C.R Jones,Applications ofThermal Imaging. 1998, Bristo~
Great Britain: IO P Publishing Ltd.
Kato, S., N. Minobe, and S. Tsugawa, Applications ofinter-vehicle communications to driver
assistance system. JSAE Revi:w, 2003. 24(1): p. 9-15.
Hoist, G.C., Common Sense Approach to thermal Imaging. 2000: SPIE Opti.:alEngineering
Press.
Milier, J.L., Principles ofInfrared Tedmology: A practicle guide to the state ofthe art. 1994:
Van Nstrand Reinhold.
IRISYS: The Affordable ThermalImager, InfraRed Integrated Systems Ltd.
AI-Habaibeh, A. and R. Parldn, An autonomous low-cost infrared systemfor the on-line
monitoring ofmanufocturing processes using novelty detection. International Journal of
Advanced Manufacturing Technology, 2003.22(3-4): p. 249-258.
Demant, C., B. Streicher-Abe~ and P. Waszkewitz, Backpropagation training, in Industrial
Image Processing: Visual Quality Control in Manufocturing. 1999, Springer-Verhg TeIos.
Gonzaiez, R.e. and R. E. Woods, Object recognition, in Digital Image Processing. 2002,
Prenti.:e Hall.
Haykin, S., Neural Networks: A Comprehensive Foundation, ed. Second. 1999: Prenti.:e Hall.
MartineIl~ N.S. and R. Seoane, Automotive night vision system. Proceedings ofSPIE - The
International Soci:ty for Optical Engileering Proceedings of the 1999 Thermosense XXI, Apr
6-Apr 8 1999,1999.3700: p. 343-346.
20
List of figures
FIGURE 1. MATHEMATICAL EXPRESSIONS FOR THE TRANSFER FUNCTIONS
FIGURE 2 EXPERIMENTAL SETUP AND DA TA ACQUISITION SYSTEM
FIGURE 3. OVERVIEW OF THE MODIFIED PEOPLE-COUNTING SYSTEM
FIGURE4. THE UPPER TWO IMAGES SHOW NO PEOPLE STANDING WHILE THE
LOWER TWO IMAGES SHOW EIGHT PEOPLE STANDING IN THE SCENE
FIGURE 5. EXAMPLE OF RAM NEURAL NETWORK TRAINING
FIGURE 6. THE BACKPROPAGATION NEURAL NETWORK
FIGURE 7. INFRARED IMAGE NEURAL NETWORK ERROR PERCENTAGE
FIGURE 8. VISUAL RAM BASED SIMULATION RESULT COMP ARISON
FIGURE 9. COMBINED PERCENTAGE ERROR OF VISUAL AND INFRARED SIGNAL
21
gure(s) fig1
ick here to down load high resolution image
~f:n
--~
·1
QC)-
1
~ -l+e- n
UMII Function
+1
Sigm()id Function
C(n1) = aflt
where a=1
igure(s) fig2
.:.::k here to down load high resolution image
.~~~~q~]
-
~~
l~kilfM!J1Jf
f
I
I
r~
I
.,'
f
Ct~:I\"-{t ;>{tt'tHlt
gure(s) fig3
ick here to download high resolution image
~~!~ l ~ f!'.e.:£f~~$sing
:I
~~~! ~~!!t.:e.fge~ssing
Tbresholding
I :
:I TIlfesbolding I:
:I
Res%6
I
•
.. _ _ _ _ _ _ _ _ _ _ _ "t
t~~~e)J.!. ~ !C!~9'~)[email protected]£~ti..~
!I
Divide into sections
I
: RAt.}I NetworJ~ creation
:I
Training
I:
1
:
):
1__ - _. ___________ •• ___ I
SJ!&.e}]=...q~ £.c.!;~
Locate object.
I:
:....
I -Reshape
I:
..... _--_ ..
t
I
l~t~J!':..q~
:I
:I
B.P. Nthvork creation
r~U!~)E:..q~!~ .£~'!!.
______________ J
:1 IN
etwork ResultC
auntl :
:
!
I:
;::=====
.
.
•.
1._-----------,
I
Getcour.t
,
Training
I:
I:
4 _______ •• ________ •
I ...
:I
________•
r §~ge-4: f:.!'!!'.P!t!S2l1_ •
: Resul
:
'----------...,~ 1;;O~dCount.l:
! :,."-- - --- - - --- .'':'.'
- - - - - -
gure{s) fig4
ick here to down load high resolution image
Visual Image
Thresholded Image
-
gure(s) figS
lick here to down load high resolution image
0000
0001
l~+=::=:71t--_Ol_O_l
_..........-.. . . .
RAM Address
Randomly selected
4 sections
0
0
0010
0011
0100
0101
0110
0111
1000
1001
1010
0
0
1
0
0
0
0
0
1011
0
1100
1101
1110
1111
0
0
0
0
0
gure(s) fig6
lick here to download high resolution image
infrared im age
1 neuron output layu
Purdin Function
I...
---~
Threshold
Reshape
:;'Jre(s): fig7
lick here to down load high resolution image
'·\~r\' .. In;'! pe
<'tNf V" 1\1; hlal ('dum
5
5
J
0
"
B
.E
...
14
...
;;
0
0
0
0
0
.4
lot
lot
:E""
U
0
:0
LS
0
o,~
0
I
9
)
"
5
(.
,\;ul.lll1umh:r ofjh7C'I'l:
7
S
10
I
Igure(s): figS
I;;,;k here to down load high resolution image
\'",u~lllA\1 t>,l,\l!'ll ~imllbtcJ
counl
1~
0
~
::
~
to
0
0
...c
-::
'J
&:!""
-
0
s
- "!
;f'
:i!
1:
~
c..
~
(,
~
~
0
...
.;
'"
::
::
c
-
..
0
0
v
'"
"~
~
0
~
Ucal
0
0
0
0
I
.
'\
3
.
5
tl
Achl.11 fUlml-cr "fpciJpk
7
R.A'"
h;lWd
to
gure(s): fig9
lick here to down load high resolution image
r'('!'CCti!
ent!f{)h",ualillld inbred I'-:"'I'lc C()U1"I1
J
0
:<~
0
"l)
!1
e
.5
"l)
,.c
..
~
1.I"i
-.g
0
L~
0
0
"'"
0
t%
..~
;;"
0
n.
it:
~
c
...
,::;
0
0
0
'1
8
0.5
(l
0
.0.$
I
"
*
J
.J
5
ACltlJl1"Itlm~ef
(j.
"rt'eep\:
.J
10
Driver tracking and posture detection using
low resolution infrared sensing
I. J. Amin, A.J. Taylor and R. M. Parkin
School of Mechanical and Manufacturing Engineering,
Loughborough University, Loughborough, UK
Abstract
Intelligent sensors are playing an ever-increasing role in automotive safety. This
paper describes the development of a low-resolution infrared (IR) imaging system to
continually track and identify driver postures and movements. The resolution of the
imager is unusually low at 16 by 16 pixels. An image processing technique has been
developed using neural networks operating on a segmented thermographic image to
categorise driver postures. The system is able to reliably identify 18 different driver
positions and results have been verified experimentally with 20 subjects driving in a
car simulator. IR imaging offers several advantages over visual sensors; it will
operate in any lighting conditions and is less intrusive in terms of the privacy of the
occupants. Hardware costs for the low-resolution sensor are an order of magnitude
lower than conventional IR imaging systems. The system has been shown to have
the potential to play a significant role in future intelligent safety systems.
Keywords: Infrared sensing, Artificial Neural Network (ANN), machine vision,
intelligent safety system
1
1
INTRODUCTION
This paper discusses research into safety systems based on the use of a lowresolution infrared sensor for driver posture detection and position tracking. The
ability to detect driver position and movement will be a key enabling factor in the
development of a number of current and future safety systems. These may range
from detecting out-of-position (OOP) occupants for safe airbag deployment to timebased monitoring of behaviour, for example to detect periods with attention away
from the road or evidence of drowsiness.
This research has initially been focussed on the needs and problems of older, or
'third-age', drivers. Transportation by private car is an important factor for the
independence and quality of life of many older people. Demographic changes and
reductions in birth rate have resulted in a large increase in the ageing population of
industrialised countries. At present one third of the population in these nations is over
55 [1, 2] and, with the average age rising, the number of older drivers will continue to
increase for the foreseeable future. The deterioration of driving abilities with
advancing age is therefore a cause for concern [3]. Deterioration of cognitive,
physical and visual abilities leads to increasing risks for older people [4], although
these are mitigated to some extent by fewer and shorter journeys and a tendency to
avoid night driving.
The term 'third-age' is generally applied to those over 55, although this definition is
not universally accepted [3, 5]. The adage 'you are as old as you feel' holds true and
people in their 60's and 70's may be more active than some younger people.
However it is clear that, as the 'grey' population increases, the needs of the over 55's
will have a greater impact on the design of many products in the future.
Problems encountered by third-age drivers typically include reading traffic signs and
signals, observing road markings, reading the instrument panel, changing lanes or
2
merging in high speed traffic, turning the head whilst parking or reversing, making Uturns and turning at crossroads or T-junctions. Many tasks that involve neck and
trunk movements are restricted in older people [6].
Cars must be designed for use by the full spectrum of the adult population. The
subject of automotive ergonomics and safety is clearly complex and a detailed
discussion of older drivers' problems and characteristics is beyond the scope of this
paper.
It is reasonable to assume that many of the difficulties and risks typically encountered
by this older group will also apply to other groups, of any age, whose driving may be
affected by physical impairments. The aim of the research described in this paper is
to provide the basis for a system which, ultimately, will make driving safer and more
pleasant for any impaired, as well as able-bodied, driver.
Additionally the system may be of benefit to Ergonomists, since observing and
analysing driver postures and movements can be difficult and time-consuming. Many
current procedures involve marker-based visual systems and others use load cells or
pressure mats installed in the car seat [7,8], methods which are not entirely practical
for use in real-time in a real car environment. Load cells and pressure mats are not
practical to use in the experiment because if the driver had a wallet in his back
pockets it would create false readings other factors are limited availability and high
cost. The IR system may provide a convenient and effective ergonomics tool for
driver movement and behaviour analysis.
Infra-red (IR) imaging offers the ability to work in any ambient lighting conditions, a
major advantage over visual cameras. The IR sensor used throughout this research
programme is an IRISYS IRI1 002 thermal imager with a resolution of 16 by 16 pixels.
The advantages offered by this particular sensor include the relatively low cost when
compared with high resolution IR cameras (hundreds rather than many thousands of
3
pounds) and the small size. Also, importantly, the low resolution protects the privacy
of the car occupants as only indistinct images composed of areas of colour or grey
levels are obtained.
2
INFRARED THERMOGRAPHY
An infrared imager measures infrared radiation emitted by objects - light with a
wavelength in the range of 0.78 to 100 Ilm. This particular range is unseen by the
naked human eye, however infrared imagers with different specifications can capture
particular ranges of infrared wavelengths.
The principle behind infrared emission detection is based on the assumption that the
black body is a perfect radiator, emitting and absorbing all energy that is incident on
it. The energy emitted by a black body is the highest possible energy emission for
that particular temperature. As real objects are not perfect black bodies, i.e. a perfect
absorber or emitter, the emissivity of the real surface is the ratio of the thermal
energy emission from the surface to that of a black body at the same condition as
that of the real body [9,10].
The infrared imager used in this paper is a long wavelength infrared imager (LWIR),
also termed a far infrared imager. It can measure infrared radiation from 8 to 24 J.lm
wavelength and has a low resolution of 16x16 pixels. This low resolution device is
approximately one tenth of the cost of a conventional infrared imager, allowing
infrared imaging to be considered in areas other than military and defence usage.
4
3
3.1
IMAGING ALGORITHM
Methodology
In this study a driver posture tracking system is developed. The tracking algorithm
has three stages of processing as shown in Figure 1. The processing stage makes
use of artificial neural networks (ANN). The detected posture is converted into a
code-based description of the driver's position after processing, referred to as a 'pcode'. The data comes from the low resolution infrared imager installed inside a car
at a suitable location and, in this study, the thermographs from the infrared imager
were taken for offline analysis. Experiments were conducted in a STISIM® car
simulator to verify the algorithm developed.
3.2
Pre processing
Data Acquisition software
The bespoke data acquisition software used was developed by the authors for both
the infrared and visual image acquisition using National Instruments
LabWindows/CVI. The user interface is shown in Figure 2. This software acquires
webcam images and thermographs in software based real-time. The image
acquisition frequency in the experiment was set at 2 frames per second (FPS),
selected on the basis of the length of experiment. Image acquisition was done for the
whole length of the simulation scenario. File naming and storing is structured as a
vast amount of data must be stored during each experimental run, therefore batch
renaming and storing data in folder options are included. The DAT file generated by
the data acquisition software is read in MatLAB using a function written by the author.
5
Infrared Interpolation
The interpolation process estimates values of intermediate components of a
continuous function in discrete samples. Interpolation is extensively used in image
processing to increase or decrease the image size. An interpolation technique does
not add extra information into the image but provides a better image for human
perception. In this case interpolation simply provides a larger thermograph area to
work on, as a 16 x16 pixel image, as shown in Figure 3, does not provide sufficient
visual information. There are commonly five types of interpolation used: cubic, spline,
nearest, linear and hyper-surface [11].
The cubic and spline interpolation are superior to other interpolated functions but due
to computation complexities, and time taken by the spline interpolation, linear
interpolation is preferred, as it is the simplest type of interpolation and is faster as
less computation is required. No significant advantage would have been gained by
using one of the other methods. A single infrared image is shown in Figure 3 with
four types of interpolation in greyscale.
3.3
Processing
Segmentation
The devised adaptive segmentation method is based on the IR histogram. This
method will compensate for slight temperature changes. The histogram of
thermographs taken from the experiment contains two peaks on each extreme
(Figure 4). The peak with the lower intensity values (black) represents the
background and the peak with the higher intensity values (white) represents the
subject.
6
Region allocation
After interpolation and segmentation of the infrared thermograph it is divided into
three regions, based upon the field of view and the position of the driver, as shown in
Figure 5. The full image size is 121 x 121 pixels.
Splitting of the IR image into three regions is a novel approach used as the basis for
an algorithm which is designed to maximise the information gained from the limited
resolution thermograph. The three regions are associated with different parts of the
driver's body and generally indicate different types of activity. These regions are
labelled as 'Torso region R1', 'Head region R2' and 'Arm and shoulder region R3'.
With the infrared imager one metre away mounted on the front left (passenger side)
'A' pillar of the car the field of view is 355mm square approximately.
'Torso region R1' is termed as such because any activity in this region necessitates
trunk movement, and therefore it becomes the focus when the driver is leaning or
looking down. This part of the infrared image is about 180mm by 200mm in size, or
61 pixels by 80pixels. The 'Head region R2' focuses on head movements and is the
most critical area. Its size in the infrared image is again 180mm by 200mm divided
into 61 pixels by 80pixels. The 'Arm and shoulder region R3' looks for arm and
shoulder movements with respect to the steering wheel. This is the region where
most movements are recorded, whenever the driver changes posture or moves the
steering wheel. The location of region R3 is the lower part of the infrared image with
a150mm by 355mm field of view, containing 121 pixels by 41 pixels.
P-code description
For neural network results to be easily readable a numerical value for each region is
allocated which points to a particular type of posture. These values are then looked
up in Table 1 to identify a unique letter code, or 'p-code', which describes the body
position in each region.
7
Region
Posture
ANN
P-code
description
Numerical
output
R1
1
Upright posture
N
R1
2
Leaning forward
E
R1
3
Looking Down
D
R2
1
Looking Ahead
F
R2
2
Looking Left
L
R2
3
Looking Right
R
R3
1
Hands on Steering
S
R3
2
Hands not on
NS
Steering
Table 1. Posture codes
All three region codes are then combined to describe a particular posture. For
example N-R-S indicates an upright position and looking right with hands on the
steering wheel, typical of a posture adopted when entering a roundabout or at a 'T'
junction (see the later comparison with real video data). In comparison D-L-NS would
indicate a driver looking down to the left side with hands not visible on the steering
wheel, which means that the driver might be putting on a seat belt or accessing a
dashboard compartment and therefore, if the movement is any more than
momentary, the car should not be in a moving state.
Feature extraction
Selection of features from the image is a vital step to enable the imaging algorithm to
give useful and accurate results. Therefore a great deal of care needs to be taken in
8
feature selection and a procedure has been devised to find each appropriate feature
for neural network input, see Figure 6.
Each of the three regions of the IR image is dealt with separately as far as feature
selection and recognition is concerned. The three categories of posture that will be
defined in the neural network are non-leaning postures, leaning postures and looking
down postures. 'Head region - R2' is considered the most critical. The three main
movements identified in this region are looking ahead, Le. normal driving, looking
right and looking left. In the 'Shoulder and arm region - R3' two positions are
identified for training the neural network, these are hands-on and hands-off the
steering wheel
In region - R2, the first posture, Le." looking ahead", numerous thermograph samples
are taken for each volunteer. Similar samples are also obtained for two other
postures, Le. looking left and looking right. Two imaging features, Le. angle and area,
from the above mentioned thermograph images are used to distinguish between
three postures.
Neural network construction and training
The neural network for each region is separate, hence there are three neural
networks working simultaneously on a single thermograph. Furthermore for the
purpose of comparing the types of neural network for each region three different
types of neural network are constructed. These are multi-layered perceptron, radial
basis network and a self-organized map network. Comparison and evaluation of all
three networks results in determining the best neural network for that particular
region. These three types are selected because each has good ability to differentiate
between different parameters [12].
Training data is selected to cover all types of motion and the frequency of motion.
Three hundred training samples for each region are taken, Le. one hundred samples
9
for each different output. All types of network constructed for each region and for
each subject contain a common set of input data and a simulation set three times
larger to verify the results i.e. nine hundred simulation samples in all. The
construction of all nine neural networks is shown in Table 2.
Torso region 'R1'
Head region 'R2'
Arm and shoulder
region'R3'
Inputs
1
2
2
(area R1)
(area/angle R2 &
(area R3)
area R2)
MLP
Layers 2, Neurons 2
Layers 2, Neurons 5
Layers 2, Neurons 5
Multi layer
Inner Layer: Sigmoid
Inner Layer: Sigmoid
Inner Layer: Sigmoid
perceptron
function
function
Output Layer: Linear
Output Layer: Linear
function
Output Layer: Linear
function
function
function
Goal: 0.001
Goal: 0.001
Goal: 0.001
RBN
Spread constant: 1
Spread constant: 1
Spread constant: 1
Radial basis
Inner Layer: Radial
Inner Layer: Radial
Inner Layer: Radial
network
function
function
Output Layer: Linear
Output Layer: Linear
function
SOM
Goal: 0, Epochs: 25
function
Output Layer: Linear
function
Goal: 0, Epochs: 25
function
Goal: 0, Epochs: 25
Self organizing
map
Table 2. Neural Network construction specifications
10
4
EXPERIMENTATION
The objective of the experiment was to acquire representative information from the
infrared imager to test and validate the driver posture tracking methodology.
The conditions in the experiments were arranged to be as close as possible to that of
driving a real car. Driving simulator scenarios were developed to enable subjects to
interact and drive as realistically as possible.
Driving volunteers
20 subjects were studied during this experiment with 10 male and 10 female drivers,
with ages ranging from 17 to 65 years old. All subjects had driving experience, with
the older drivers having the most experience.
IRISYS Imager: Serial Protocol programming
The IRISYS IRI1002 thermal imager has its own protocol which needed to be
converted into an infrared image. The software has to be compatible with hardware
specifications for the thermal imager .i.e. 1,15,200 baud, 8 data bits, no parity, 1 stop
bit and no handshaking.
Infrared acquisition software
Infrared acquisition software, developed by the author, was used in this experiment
and has also previously been used by Amin et al [13, 14]. This software acquires
webcam images and thermographs in real-time. The image frequency in the
experiment was set at 2 FPS. This image acquisition frequency was selected on the
basis of the length of experiment and image acquisition was carried out for the whole
length of the simulation scenario.
11
Mounting of the IRISYS imager
The infrared imager was the main instrument used to acquire driver information
during the experiment. The infrared imager and visual camera were mounted close
together on a test rig. The camera was positioned on the left (passenger side)
windscreen pillar making an angle of 60 Degrees with the longitudinal direction. The
lens field of view (FOV) was 20 degrees and the IR camera was mounted 1 metre
away approximately from the seat head rest. With the image capturing 355 mm
square from the angled position, it manages to acquire the face, shoulders and arm
with the hands on the upper half of the steering wheel.
Visual camera
A visual (CMOS) web camera was used for the purpose of adjusting the field of view
of the IR imager and for verification of the results taken from the IR imager after
processing.
Driving simulator
The driving simulator comprised a custom built test rig with front projection screen
and adjustable driving controls (Figure 7). A separate control room housed a driver
communication system and the STISIM® driving simulation software on an IBM PC.
The driver was kept in contact with the researcher by means of the driver
communication system during the length of the experiment.
The STISIM® driving simulator was programmed for the MS-DOS operating system.
The simulator is designed such that it provides the driver with realistic driving
experience using both the visual display and audio effects as feedback to driver
actions.
12
Scenario
The scenario selected for the experiment involved urban busy traffic, lasting for
approximately 20 minutes. An urban traffic scene was selected because the number
of tasks during driving is much higher than for motorway or trunk road driving. The
scenario involved a number of tasks performed by the volunteer as well as going
through the driving simulation. These tasks included putting a seat belt on, adjusting
mirrors, looking in the rear view mirror, looking left or right, using a swipe card to
simulate entering a secured car park, mobile phone usage while driving and using an
in-car stereo system or climate control. These tasks were in addition to other
conventional driving tasks, such as looking left or right before making a turn, arriving
at traffic lights and waiting for a signal, lane changing manoeuvres and looking at
side and rear view mirrors.
A second scenario was conducted for monitoring fatigue and sleepiness in drivers
while driving for longer periods. A long road with curves was programmed in a
scenario which lasted for over 50 minutes and curtains were drawn over the
simulator test rig to give the driver a sense of driving alone. Noise was cut to a
minimum, less traffic was shown on the road and the drivers were asked to maintain
a constant speed of 60 mph.
5
5.1
EXPERIMENTAL RESULTS
ANN Simulation
The simulation set was three times the size of the training data set. Simulation set
results are plotted in Figures 8, 9 and 10. The plots are for a single human subject,
chosen to represent an average identification accuracy rate. The different postures
13
in the graphs are linked with the numerical postural code, or p-code - the closer the
actual result plot is to the posture code the more accurate is the detection.
Each driver volunteer is considered individually for ANN training, therefore the
system can detect the same posture with similar accuracy for all subjects. This
means that a person with long hair or short hair will not have different accuracies in
the system due to the thresholding of thermographs and individual driver training. as
there are 3 different regions, which are combined together to get the final result. ~
~~!~;gel~ctioJit~9Ji!tlji1~m be fu~nip~!3LEE.§H~~t1~grJ~
m:~rmogfaQtis ea~~~.£Q.~ one thetin~!iiygE!sultiStiOl!§
5Snormalif
or!ina'cciira~tRe
'fe,,:aifUH'liltFih:te
ra 'nsmaJii. able t:::m
~.*_~~,_~!!!!Y,_~,
__"..•~_",~",,~_."'~~~W
~~'k"~'~"'"(3\ttia~
MJ,_,~9.J!J2!J.:_~'~')!k'_.*_~".~
__
[§m~Y~.Jh~~_~,rrQIm~!!!Jt~QX:rl}~~£!1J:~11TJjJ!,OlaJs~JD4~I~~Y~!.§[email protected]~
r~iQ!§.LTtie;safe!y syst~mfin "I-$al )if~ 6~s :tcDie:8 re~1 tirne;wtiictiJW9Uld Qr~
[[forffiatloDJQJfi~: ihtelllg)m[q~ntr~afE1tY~YsI~m·tq~prgyiae··feeg6aQ[1~.anvifj
5.2
Real life data comparison with results
Video footage of 20 volunteers was taken driving in urban areas and later on a
motorway. Ten volunteers were below the age of 45, whereas the other 10 were over
the age of 55. The groups of volunteers were 50% male and 50% female. Each
video lasted for at least 45 minutes depending upon the time eacti volunteer took to
drive from start to finish.
The video footage of each volunteer was analysed by taking a movement description
and the time taken and also the p-code was identified and noted. A particular pattern
of movements relates to ttie p-code generated during the video analysis. The
movement for each motion detected may be a single event, for example like looking
right, or it could be a series of movements to carry out a particular manoeuvre, like
14
putting on a seat belt. This series of movements will create a pattern for that
manoeuvre which is more or less the same for most drivers. A few examples of the
patterns of motion with p-codes follow in Table 3. The larger manoeuvres are broken
down into smaller movements which the safety system detects.
Task
Time
P-Code
Comments
1
Applying Handbrake
2s
D-L-S
One hand on steering wheel
2
Left Turn (T-Junction
Subject brakes and carries
or Cross Road)
out following movement
Look Right
3s
N-R-S
Turn Left (Driver looks
1s
N-F-S
Look Right
6s
N-R-S
Turn Left (Driver looks
1s
N-F-S
Less traffic
ahead)
3
Left Turn (T-Junction
or Cross Road)
Heavy traffic from right
ahead)
Table 3. Examples of driver positions found from the videos
15
6
DISCUSSION
Low resolution infrared imaging has been shown to be an effective means of
monitoring driver posture and movements to provide a number of important physical
indicators of driving behaviour. IR imaging is a practical and non-intrusive method of
occupant tracking which does not invade the privacy of individuals in the way that
visual imaging may be perceived to do and, therefore, should be ethically acceptable.
The system using neural network analysis has been shown to be robust. IR imaging
has the advantage of being entirely independent of lighting conditions. The low
resolution allows hardware costs to be kept down and further reductions will be
possible by design for mass production to achieve appropriate cost levels for midrange and, ultimately, smaller cars.
The low resolution means that this safety system cannot detect eye gaze or small
movements of the driver's head. Eye gaze can be anything from looking in the side
and rear view mirrors to observing scenes on the road.
The imaging system as developed can identify 18 different driver postures or
movements and reliably interpret these in the form of p-codes. Single frame analysis
can then provide valuable information on occupancy and position, for example OOP
drivers, the driver's physique and eye height estimates, all of which are important
considerations for safe airbag deployment. Further analysis based on time histories
will be able to identify a range of high risk situations. For example impaired
movements can cause difficulty in turning the head sufficiently for adequate vision at
junctions and for checking of blind spots. Poor trunk stability can cause lack of
confidence and degraded car control. These are typical problems encountered by
many older people or people impaired by disability or injury.
Although it may be used in a stand-alone mode the greatest potential of the IR
imaging system will be realised as part of an integrated multi-sensor intelligent safety
16
system. The interpretation or significance of events and behaviours will clearly
depend to an extent on the state of the vehicle, for example whether it is stationary or
in motion, driving straight or turning. Simply linking the IR system with road speed,
throttle and braking information is straightforward and low cost and can provide a rich
data source which can be used to identify high risk situations or behaviours, such as
evidence of drowsiness or time with attention away from the road while the vehicle is
in motion.
This research has shown that a low resolution 16 by 16 pixellR imager can play a
significant role in the next generation of intelligent safety systems.
REFERENCES
1.
World Health Organisation, Population ageing - A public health Challenge,
WHO Fact Sheet No. 135, 1998.
2.
Dissanayake, S. and Lu, J. J. Factors influential in making an injury severity
difference to older drivers involved in fixed object-passenger car crashes. Accident
Analysis & Prevention, 2002. 34(5), 609-618.
3.
Carr, D., Jaskson, T. W., Madden, D. J. and Cohen, H. J. The effect of age
on driving skills. Journal of the American Geriatrics Society., 1994.
4.
Lundberg, C., Hakamies-Blomqvist, L., Almkvist, O. and Johansson, K.
Impairments of some cognitive functions are common in crash-involved older drivers.
Accident Analysis & Prevention., 1998.30(3).371-377.
5.
Reuben, D.B., Assessment of older drivers., Clinics in Geriatric Medicine,
1993. 9(2), 449-459.
6.
Smith, D.B.D., Meshkati. N. and Robertson, M.M. The older driver and
passenger. Automotive Ergonomics, Ed. Peacock, P. and Karwowski, W. Taylor and
Francis, London, 1993, p. 461
17
7.
Andreoni, G., Santambrogio, G. C., Rabuffetti, M. and Pedotti, A. Method
for the analysis of posture and interface pressure of car drivers. Applied Ergonomics,
2002,33(6), 511-522.
8.
Rakheja, S., Haru, I. and Boileau, P. E. Seated occupant apparent mass
characteristics under automotive postures and vertical vibration. Journal of Sound
and Vibration, 2002, 253(1), 57-75.
9.
Burney, S.G., Williams, T. W. and Jones, C. H. Applications of Thermal
Imaging, 1988, Institute of Physics Publishing.
10.
Hoist, G.C. Common Sense Approach to Thermallmaging, 2000 (SPIE-
International Society for Optical Engineers).
11.
Kulkarni, A.D. Artificial Neural Networks for image understanding, 1994.
12.
AI-Habaibeh, A. and Parkin, R. M. An autonomous low-cost infrared system
for the on-line monitoring of manufacturing processes using novelty detection.
International Journal of Advanced Manufacturing Technology, 2003, 22(3-4), 249258.
13.
Amin, I. J., Taylor, A. J. and Parkin, R. M. In-cabin occupant tracking using
a low-cost infrared system. In Proceedings of the IEEE Mechatronics and Robotics
Conference, Aachen, Germany, 2004.
14.
Amin, I.J. Sensor fusion of visual and infrared system for monitoring people.
MSc thesis, Wolfson School of Mechanical and Manufacturing Engineering,
Loughborough University, 2003, p. 187.
18
APPENDIX 1
Notation
ANN
artificial neural network
known values on the grid at points (qj)
FPS
frames per second
IR
infrared
Lj(q)
Lagrange polynomial
LWIR
long wavelength infrared
MLP
Multi-layer perceptron
OOP
out of position
p-code
a unique letter code describing a body position
P(q)
interpolated value
q
point at which interpolation takes place
R1
segmented region of the image forward of the driver
R2
segmented region of the image containing the driver's head
R3
the lower segmented region of the image
RBN
Radial basis network
SOM
Self organizing map
19
List of figure captions
Figure 1. System flow chart for the position tracking algorithm
Figure 2. Infrared data acquisition software
Figure 3. Four types of interpolated infrared image
Figure 4. Histogram of an infrared thermograph
Figure 5. Region allocation within the infrared image
Figure 6. Feature selection process
Figure 7. Driving simulator test rig
Figure 8. ANN simulation result for torso region R1
Figure 9. ANN simulation result for head region R2
Figure 10. ANN simulation result for arm and shoulder region R3
Table 1. Posture codes
Table 2. Neural Network construction specifications
Table 3. Examples of driver positions found from the videos
20
Processina
_._._._.-._._._._.,
1
1
Pre-processing
_._._._._._._._._.,
Reading
infrared
thermograph
Infrared
interpolation
1
1
1
1
H
•
.-.-.-.-.-.-.-.-.-.~
Results
Binarization
1_-----_
ANN
~.-.-.-.-.-.-.-.-.~
Region
allocation
Feature
extraction
ANN
1
results
Compare with
real life
results
_._._._._._._._._.J
construction,
training and
simulation
Figure 1. System flow chart for the position tracking algorithm
21
-
D...i:.C"""""
i:~~'1 ti.. P,.....~ I
~
11<"--." ,"" d~'
~
:er--
!oI<dr,*,*1
t!~o~1
n._I"~
,.",
Figure 2. Infrared data acquisition software
22
Nearest Interpolation
Cubic Interpolation
Linear Interpolation
Spline Interpolation
Figure 3. Four types of interpolated infrared image
23
Maxima
line
Maxima
line
I
500
400
!J)
Qi
x
.0.
'0
300
Q;
.0
E
:I
z 200
100
0
0
25 I
Minima or
thresholding value
Temperature
(Celcius) ~
Figure 4. Histogram of an infrared thermograph
24
R2
R1
R3
Thermograph divided into three regions R1,
R2 and R3
Interpolated thermograph
Figure 5. Region allocation within the infrared image
25
Figure 6. Feature selection process
Image processing block
r--------------------------------------1
1
1
1
1
1
1
1
1
1
Infrared
image
Segmentation
of
body region
Division of
infrared image
into three
regions
1 ______ - - - - - - - - - - - - - - - - - - - - - - - - - . - - - - - -
r-----~----------~-----------~----Features
Features
Features
extracted from extracted from
extracted from
region 1
region 2
region 3
Specifications of
driver posture
list which is
being detected
( P- codes)
I
~
I
1
1
1
1
1
1
1
1
1
,
-.
r::
o
t5ID
List of features narrowed down by comparing
different infrared images
(j)
en
~
:::l
I
caID
LL.
Plotting of features against each feature
(using scatter-gram, graphs)
I
Decision curves drawn and graphs compared
I
I
Selected
features from
region 1
Selected
features from
region 2
I
Selected
features from
region 3
1
26
STISIM
ir----r' controller
"
Visual
Garrera
""
"
Infrared
Irrnger
" ""
""
""
""
""
""
"" "
"
Sensors rrountin table
I
Self -<:entring
rrechanism
Linear
rig adjustrrent rrechanisms
Figure 7. Driving simulator test rig
27
Figure 8. ANN simulation result for torso region R1
3.5
Looking down head position
3.0
~
ID
...
Y"-
...
y
~-v--"'Y
'Y '"
(/)
c:
0
a.
(/)
~
c:
2.5
Leaning head position
2.0
.1'\
A
/V'-v /I"
V
~
"'- 1\
A
~"
,,/\
I
0
... .r-'\
A
:;::::;
ro
:::J
E
'00
c:
c:
/V"\
I
,,/\~
v
1.5
Normal head position
1.0
~
0.5
0
o
I
I
I
25
50
75
Number of samples
28
100
Figure 9. ANN simulation result for head region R2
3.5
Looking left
3.0
Q)
(/)
c:::
0
Co
2.5
(/)
~
c:::
2.0
0
:;:::.
Cl]
S
E
"(j)
c:::
c:::
«
1.5
1.0
0.5
0
0
25
75
50
Number of samples
29
100
Figure 10. ANN simulation result for arm and shoulder region R3
2.5
Hands off steering
2.0
r-
-
_....
"V ....
-
.,....
--
--
ID
(/)
C
0
a.
(/)
1.5
~
Hands on steering
c
0
:.;:::.
CO
1.0
"S
E
'00
c
c
«
0.5
o
I
o
25
I
I
75
50
Number of samples
30
100
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement