Illumination technique for optical dynamic range

Illumination technique for optical dynamic range
Illumination technique for optical dynamic
range compression and offset reduction
C. Koch, S.-B. Park , T.J. Ellis and A. Georgiadis
City University, London, EC1V OHB, UK
BMW AG, D-80788 Munich, Germany
FH Nordostniedersachsen, D-21339 Lüneburg, Germany
Abstract
This paper presents a novel illumination technique for image processing in
environments which are characterized by large intensity fluctuations and hence
a high optical dynamic range (HDR). This proposal shows how by combining
a set of images, flashed with different radiant intensities but with a constant
exposure time for the imager, a single image can be produced with a compressed dynamic range and a simultaneously reduced offset.
This makes it possible to capture high dynamic range scenes without using a high dynamic range camera. This technique can be used as the first
signal processing step to simplify the segmentation in applications such as:
face recognition, interior surveillance, vehicle occupant detection or motion
detection in general.
1 Introduction
Mainstream CCD based (and most of the emerging CMOS based image sensors) provide
an optical dynamic range of
dB. This dynamic range is sufficient for scenes with
homogeneous illumination and without extreme contrasts. Extreme contrasts may occur
when operating an imager in direct sunlight or within scenes containing areas of high
brightness and deep shadow, e.g. a building entrance. Other classic examples are looking
from a dimly lit room through a window towards a bright outdoor scene, the direct view
into an operating light bulb or a vehicle interior (see Fig. 5).
Usually it is possible to adjust common imagers either to the very bright areas or to
the dark areas in the scene by adapting several of the imager’s parameters, e.g. exposure
time, lens aperture or by the use of optical filters. Nevertheless in extreme dynamic environments there will be areas in the scene which are over or under exposed, resulting in lost
image detail. In particular CCD based imagers tend to suffer this problem because only a
small number of over exposed pixels can yield large saturated areas (due to blooming and
smearing). This well known limitation is shown in Fig. 4.
There are various ways to extend the limited dynamic range of an imager. One approach uses several images of the same scene taken with different exposure times which
are then assembled into one high dynamic image [2, 1]. Another approach is to use an
imager with intelligent pixels to choose the optimal exposure time individually [5]. Both
techniques have the disadvantage of a high processing requirement in order to produce
an output, and as such their system design is specialized for industrial applications such
293
as visual welding inspection. Hence, these imagers are expensive and the frame rate is
usually slower when compared to mainstream designs.
An alternative and well known way to reduce the influence of ambient illumination
and thereby to reduce the range of brightness variation within a scene is to utilize supplementary illumination in combination with an optical band pass filter. Using a filter with
a center wavelength (CWL) at the wavelength of the supplementary illumination source
blocks all irradiation from the ambient illumination outside the transmission wavelength
band. A suitable wavelength range ( ) for such an illumination is the near infrared
(NIR), because it is not visible to the human eye which is important for many surveillance
tasks. Also most cameras based on silicon are still sensitive at these wavelengths and the
spectral power density of the main disturbance noise source (the sun) decreases signifi
cantly after
nm. This effect can be improved by utilizing a strong flashed light
source which increases the signal-to-noise ratio (SNR), suppresses the noise induced by
the environment and minimizes its influence on the image.
However problems with strong flashing can occur if large areas have to be illuminated,
because this can require an illumination source whose optical intensity can reach the
threshold values for eye safety [3]. This paper describes a solution to this problem without
the need for special high dynamic range (HDR) cameras and without violating eye safety
limitations whilst retaining the important scene details. Section 2 gives an introduction to
offset reduction due to ambient illumination and section 3 describes mathematically the
definition of optical dynamic range. This leads to section 4 where our new double flash
algorithm is introduced. Section 5 illustrates the proposed dynamic range compression
with an example calculation and describes some preliminary experimental results.
2 Offset reduction
In order to reduce the necessary pre-processing steps for image analysis, it is obviously
preferable to capture a sequence consisting of frames (
) without global
or local illumination fluctuations, i.e. without varying illumination offset and noise. This
means that the irradiance which is integrated over the exposure time ( ) of the imager
should be constant for every frame and hence the grey level of an arbitrary pixel at position
within a digital image
only varies if the scene has changed, i.e. only if the
reflectivity of the surface
and not the ambient illumination has changed.
"!
$#&% '$(
$ )*+,
-.*/$012
3
3
(1)
This would simplify most image processing steps, e.g. background updating for motion
detection [6], thus decreasing the necessary computation time and reducing the cost of the
processing hardware, and is therefore a further step towards real-time image processing.
One way to achieve this offset and noise reduction is to acquire one image
with
only ambient illumination (
) at frame and a second image
with additional
7546
294
98;: <.>=
546
546 6 ) from a supplementary light source (e.g. NIR) at frame $ .
"!
;546 $#&% 546 $(
(2)
"!
;8;: <.>= $#&% " 546 $ 6 $(
(3)
.6 ;546 ;8;: <.>= (4)
Assuming that the scene is static; hence is constant over all frames. This means that
the captured greylevels within the images are only a function of which illuminates the
scene. Thus, the difference between the images 546 and ;8;: <.>= yields only the received
radiant power of the local illumination which is constant (see eqn. (5). The variable offset
of the ambient illumination source 7546 is eliminated and the output sequence 96 illumination (
is thus exempt from light fluctuations [4].
546
.6
" 5 46
6
6 9 6 7
const.
const.
(5)
(6)
546
This method is illustrated in Fig. 1. It shows the intensity difference between frame
and
of the same pixel over a sequence of 60 frames. The sequence begins with
small illumination fluctuations and ends with a double-peak which was induced using a
powerful external light source directed at the scene for a short time. Referring to eqn.
(5), the calculated difference between
and
should ideally be constant. The
plot of differs marginally from an (ideal) straight line due to sensor noise and small
illumination changes of
between both input images. The sequence was produced
with a linear HDR camera [1] (optical range of up to dB, see section 5). The following
conditions must hold:
Dynamic: The camera has to cover the whole optical dynamic range of the scene
;8;: <.>=
546
.6
7546
;8;: <.>=
and must not saturate or under-expose any pixel, otherwise the subtraction of consecutive images yields an unrepresentative result. This means that for an environment with large irradiance differences in the spatial and time domain, a HDR
camera is required.
9546
98;: <.>=
Speed: The time gap between
and consecutive
should be kept as small
as possible to minimize the effect of changing ambient irradiation
or surface
reflectivity
, caused by moving objects, which results in an inaccurate output
image.
546
Reach: The whole scene which has to be analyzed must be within the scope of the
local illumination . This condition occurs in closed environments, e.g. within
a vehicle interior or building.
6
3 Dynamic range compression
The utilization of supplementary illumination in section 2 has the additional effect upon
the flashed image
such that the optical dynamic range within the image is compressed compared to the image without supplementary illumination. This is explained in
the following paragraphs.
98;: <.>=
295
14000
12000
I_sun
I_flash
I_nir
Grey Level (linear)
10000
8000
6000
4000
2000
0
0
5
10
15
20
25
30
35
Time [Frames]
40
45
50
55
60
Figure 1: Principle of offset reduction
generated by a digital imager represents the photogenThe image intensity
erated current of each pixel. This current is a function of incident irradiance, sensor
offset, gain and sensor noise. The dynamic range for an image sensor (
) is commonly
defined as the ratio of its largest nonsaturating signal ( ) to the standard deviation
of the noise under dark conditions ( ) [7].
9=
9=
9= <
6
(7)
Assume a scene which exhibits a wide
in both the time and spatial domains due to
fluctuating ambient illumination and different interior surface materials, or within rooms
with large windows. For simplification, assume that is proportional to the incident
irradiance
and that the imager is located in such a way that there is no irradiation
directly incident on the sensor. Hence, the
within the image (spatial domain) or
between two frames at time and
(time domain) will be determined by the product
of the reflectivity of the interior surface
and the occurring maximum and minimum
irradiance , respectively.
9=
< 6
9=
! #"
$% &('*)
(8)
"
$% &('*) +
(9)
6
<
Within a scene without supplementary illumination, the irradiance , and are
determined only by ambient illumination sources. In vehicles and buildings etc. this
ambient source is primarily the sun
.
7546
< 6 296
546- < 54-6 6
(10)
In the case that the image is illuminated with a supplementary illumination
is added to the ambient
.
- - 7546
8;: <.>= < 8;: <.>= 6 546 <
546 6
8;: <.>= , 6 6
6
(11)
If eqn. (10) and (11) are inserted into eqn. (8) eqn. (12) results which shows that the
optical dynamic range of a scene with supplementary illumination (
) is lower
than the scene without (
). This effect is independent of the reflectance
which is shown in eqn. (13).
8;: <.>=
546
% $$
$
$
546
8;: <.>=
% $$
$$
5 46- < 546- 6
5 46- < 546- 6
) $ % ) $ ) '
6 6 (12)
) $ % ) $ ) '
546- <
6 546- <
( 546- 6 6 54-6 6
(13)
This means that by using supplementary illumination which brightens the whole scene,
the optical dynamic range of the scene is compressed. The power of determines if
areas that were formerly too dark or too bright (and thus out of the dynamic range of the
imager) can now be accurately captured (see Fig. 4). Fig. 2 illustrates this effect. It shows
the same offset reduction plot with identical data as Fig. 1 but with logarithmic scaling
of the grey values to highlight the changes to the dynamic range of
and
.
However, if the offset reduction is applied, an input image without local illumination for
determining
is still necessary for calculating the offset free output. This image might
still feature a high dynamic range. Hence, for analyzing HDR environments as described
in the introduction, a HDR camera is still essential, if the offset reduction should be used.
A solution to this problem is our proposed double flash technique which is described in
the following section.
6
546
8;: <.>=
7546
4 Double flash
We introduce a new approach which combines the advantages of both offset reduction
and dynamic range compression by illuminating two input images with different radiant
intensities. Both input images
and
are compressed in their optical
dynamic range due to the supplementary illumination, and the output image is also free
of illumination fluctuations. The output image of the double flash is calculated as
follows:
98;: <.>= =
;8;: <.>= = ;8;: <.>= : ! 8 ;8;: <.>= :
8
"!
$#&% " 546 $ 6 = $(
"!
$#&% " 546 $ 6 : $(
;8;: <.>= = ;8;: <.>= : 297
(14)
(15)
(16)
100000
DR_flash
Grey Level (log)
10000
1000
100
I_sun
I_flash
I_nir
DR_sun
10
0
5
10
15
20
25
30
35
Time [Frames]
40
45
50
55
60
6
Figure 2: Compressed dynamic range due to supplemental illumination . The double
arrows indicate the difference between
and
.
546
8;: <.>=
546
The influence of the ambient illumination
has according to eqn. (17) been eliminated.
const. (17)
const.
(18)
! " 546
6 = " 546 6 : 6 = 6 : 8 6 = 6 :
The difference to the basic offset reduction in eqn. (5) is that 76 : operates as the gain
for the minimum radiation 6 which was previously at 9546 and only defined by the
ambient illumination 7546 (see eqn. (5) and (10)). The original dynamic range of both
images are thus compressed by 6 according to eqn. (12).
8;: <.>= = * < 54-6 < 6 = (19)
8;: <.>= = * 6 54-6 6 6 = (20)
8;: <.>= : < 54-6 < 6 : (21)
8;: <.>= : 6 54-6 6 6 : (22)
This leads to the effect that the maximum dynamic range of the double flash output image
(
) is now defined by the ratio between the adjustable local illumination
intensity
and . This is shown in eqn. (23) and (24) where eqn. (19)
(22) are
inserted into the basic dynamic range definition of eqn. (8).
(23)
6 =
8
6 :
) $ ) $ ) $ ) $ )
)
8
$ ) $ ) ) $ ) ) $ ) (24)
The timing diagram of the trigger logic for this illumination technique is shown in Fig.
3. High power light emitting diodes could be used as illumination sources. The diodes
298
hi1
NIR
low1
hi
low
0
Logic
Read-out
1
0
Trigger
1
(a)
(b)
40
60
0
0
20
80
100
Time t [ms]
120
140
160
180
Figure 3: Camera timing for the double flash: One frame is equal to for read-out and transfer to the PC.
76 =
200
ms, including ms
6 :
operate with alternating high and low currents and emit and , respectively.
In the case of a camera with a global and synchronous shutter it is far more efficient to
activate the LED illumination only when the imager pixels are integrating light. The offtime when the image is read out can be used to allow the LED’s to cool off and also to
reduce the total power consumption. Hence, it is possible to run the LED’s with higher
pulse currents resulting in the use of smaller diodes with no reduction in radiant power.
The absolute function which is applied in eqn. (5) and (17) implies that the illumination
order of and , i.e. the capturing to
and
respectively, is
arbitrary. Thus, although the offset reduction and double flash need two input images, they
produce one output image for every new input after the second captured image because
only an irradiance change between subsequent frames is required. The last captured
image at time can be compared to the current image for computing the next output image and hence suffer only a 1 frame delay.
6 :
6 =
98;: <.>= :
$ ;8;: <.>= =
8
5 Experiments
Fig. 4 shows a scene in a laboratory which includes high irradiance differences, generated
by a bright spot light from the right. This spot light in combination with the ambient
illumination simulates
. A mainstream CCD-based camera with a single integration
time (and thus limited dynamic range) is employed to capture the scene. The integration
time in Fig. 4(a) was optimized for reading the labels of the power supply. By extending
the integration time in Fig. 4(b), the silhouette of a dark ring on top of the floppy disk
becomes visible but most of the image is over-exposed. The imager was not able to
capture the scene without losing detail with a single exposure time. A supplementary
illumination as used in section 3 and 4 in combination with a shortened integration
7546
6
299
(a) Short exposure
(b) Long exposure
(c) With
6 (proposed)
Figure 4: Example of a optical high dynamic range scene. Image (c) indicates the result
of the proposed dynamic range compression by supplementary illumination.
time (Fig. 4(c)) yields the capability to capture the labels as well as the dark ring on
top of the floppy with just one exposure time. The primary direction and intensity of the
disturbing light source from the right remained untouched for every image. The radiance
emmitted by the supplementary illumination brightens the entire scene (
), i.e. in
both the dark areas and bright areas. Hence, to capture the bright areas an integration
time reduction was necessary though the additional radiance was sufficient to shift the
image detail of the dark areas into the dynamic range of the camera. The following
calculation illustrates our experimental
results. Assuming
a scene with a dynamic range
of
- and
- , which was measured within a
passenger car interior on a sunny summer day.
The aim of the car interior surveillance is to recognize, classify and track the occupants for reasons of safety and convenience. Such a vehicle interior represents a HDR
environment due to the varying influences from the sun intensity and impact angle through
the windows. Our measurements of the interior yield a raw optical dynamic range of up
to dB which is outside the capabilities of mainstream imagers. To capture the interior
8;: <.>=
546
6,
11
546 < 1
(a) Short exposure
(b) Long exposure
(c) Double flash (proposed)
Figure 5: Car occupant detection via CMOS camera. Image (c) shows the result of the
proposed double flash: Reduced dynamic, without ambient offset and FPN.
across a wide range of illumination situations without losing image detail, it was neces
sary to employ a linear HDR camera [1] with a dynamic range of dB. Fig. 1 and 2
were created with the data from such a camera. Our aim was to find a way of employing
a mainstream imager with limited dynamic range to capture the vehicle interior without
losing the interior details. We solved this problem by using the double flash approach as
300
proposed in section 4 (see Fig. 5). A supplementary offset of , pro
duced by a NIR illumination source, compresses the image dynamic to dB (see eqn.
(25)). This is equivalent to a maximum contrast of
and thus corresponds to a
resolution of 10 bits which is within the range of mainstream CMOS imagers.
5 46'
8;: <.>= 76 11
%
% dB
%
) % dB
6 (25)
(26)
To illustrate the double flash effect, we extend the calculation of eqn. (25). The final
dB.
output image of the double flash should show a maximum dynamic of
This is equal to a maximum contrast of , which can be displayed and stored with 8
bits. The maximum dynamic range of mainstream CMOS imagers was already specified
for the first example
with dB and thus leads to a necessary supplementary illumination
of
. This is the minimum illumination necessary to reduce the dynamic range
of the scene to the requirements of the given imager. Hence, we label it , because
a more powerful local illumination merely yields a stronger dynamic compression
and is therefore within the detection range of the imager.
8,
11
6 :
6 =
!
)
%
)
% %
) % with 6 : 6 = #"
8
) (27)
A further advantage with the double flash approach arises if the CMOS imager suffers
from fixed pattern noise (FPN) [5]. FPN is an individual offset from each pixel caused
by slight unwanted variations of active pixels, i.e. transistor characteristics. To minimize
the FPN of the camera, the captured data can be compared internally or externally to
an offset-map for correcting the final output. This FPN correction can be performed by
firmware and is thus relatively fast but still takes time.
Due to the fact that the individual FPN’s represent an offset which is constant over
time, it will be eliminated if two subsequent images are subtracted from each other.
Hence, the output image sequence of a double flash system improves the final picture
quality for imagers which suffer FPN and the internal or external FPN correction can be
disabled. The results of the double flash is shown in Fig. 5 which shows the passenger and
driver seat of a car. The CMOS camera was equipped with an optical NIR band pass as
described in section 1 which cuts off wavelengths in the visible range. A strong halogen
lamp (illuminating through the sun roof) simulates
and causes a bright region on the
infant seat. The CMOS camera was not able to capture details of the infant and the driver
seat within the same image just by varying the exposure time or aperture. Furthermore
strong FPN effects are visible which might distort a texture analysis of the scene. Fig. 5
(c) was acquired by using the double flash approach: Texture of both infant and driver seat
are visible due to the compressed dynamic range. The influence of ambient illumination
(the halogen lamp) was eliminated and the image quality was considerably improved due
to the reduced FPN.
546
301
6 Summary
7546
6
546 <
6 =
The irradiance
- and
- in a given high dynamic range scene is fixed and
usually requires a HDR camera for capturing all the scene details. By using two supplementary flashes and , it is possible to influence the optical dynamic
range of the scene so that a scene with high dynamic range (
) can be captured
by commonly available cameras with limited optical dynamic range (
) without sacrificing image detail and with synchronous offset reduction. The grey-level of a pixel
within the output image sequence varies only if the scene changes and not as a
result of fluctuations in the ambient illumination levels. This significantly simplifies any
subsequent image processing.
An additional advantage referring to the otherwise necessary HDR cameras is that
the supplementary illumination increases the brightness of the scenes. This enables the
camera system in principle to run with a higher framerate because the integration time
can be shortened due to the higher minimum irradiance level resulting in a greater signalto-noise ratio (SNR). The major shortcoming of the double flash is that its application is
limited to environments where the whole scene to be analyzed must be within the scope of
the local illumination . The image pre-processing described in eqn. (16) is relatively
simple and could be performed by firmware within the camera or integrated onto the
sensor if CMOS technology is used. This yields an increase in the processing speed of
the entire image processing system. Finally the costs for a mass produced camera with
limited dynamic range in combination with simple trigger logic for the illumination are
less than for a smart HDR camera.
6 :
546
8 8
6
References
[1] J. Huppertz, R. Hauschild, B.J. Hosticka, et al, ”Fast CMOS Imaging with High
Dynamic Range”, IEEE Workshop on CCD’s and Advanced Image Sensors, 1997.
[2] S. Mann, R.W. Picard, ”On being ’undigital’ with digital cameras: Extending Dynamic Range by Combining Differently Exposed Pictures”, Society for Imaging Science and Technology, Annual Conf., May 1995.
[3] Int. Norm ISO 825-1, ”Maximum Permissable Exposure Limits”, 1993.
[4] S-B. Park, A. Teuner, B.J. Hosticka, G. Triftshaeuser, ”An interior compartment
protecting system based on motion detection using CMOS imagers”, Proc. of Int.
Conf. in Intelligent Vehicles, October 1998.
[5] B. Schneider, H. Fischer, S. Benthien, et al, ”TFA Image Sensors: From the One
Transistor Cell to a Locally Adaptive High Dynamic Range Sensor”, Technical Digest of International Electron Devices Meeting, December 1997.
[6] K. Toyama, J. Krumm, B. Brumitt, B. Meyers, ”Wallflower: Principles and practice
of Background maintenance”, Int. Conf. on Computer Vision, September 1999.
[7] D. Yang and A. El Gamal, ”Comparative analysis of SNR for image sensors with
widened dynamic range”, Proc. of SPIE, February 1999.
302
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement