This article was published in an Elsevier journal. The attached copy

This article was published in an Elsevier journal. The attached copy
This article was published in an Elsevier journal. The attached copy
is furnished to the author for non-commercial research and
education use, including for instruction at the author’s institution,
sharing with colleagues and providing to institution administration.
Other uses, including reproduction and distribution, or selling or
licensing copies, or posting to personal, institutional or third party
websites are prohibited.
In most cases authors are permitted to post their version of the
article (e.g. in Word or Tex form) to their personal website or
institutional repository. Authors requiring further information
regarding Elsevier’s archiving and manuscript policies are
encouraged to visit:
Author's personal copy
J. Vis. Commun. Image R. 18 (2007) 359–365
Adaptive dynamic range camera with reflective liquid crystal
Hidetoshi Mannami
, Ryusuke Sagawa a, Yasuhiro Mukaigawa a, Tomio Echigo b,
Yasushi Yagi a
The Institute of Scientific and Industrial Research, Osaka University, 8-1 Mihogaoka, Ibaraki, Osaka 567-0047, Japan
Department of Engineering Informatics, Faculty of Information and Communication Engineering, Osaka Electro-Communication University,
18-8 Hatsu-cho, Neyagawa, Osaka 572-8530, Japan
Received 15 November 2006; accepted 12 June 2007
Available online 21 June 2007
Wide dynamic range images (WDRIs) are needed for capturing scenes which include drastic lighting changes. This paper presents a
method to widen the dynamic range of a camera by using a reflective liquid crystal. The system consists of a camera and a reflective liquid
crystal placed in front of the camera. By controlling the attenuation ratio of the liquid crystal, scene radiance of each pixel is controlled
adaptively. After applying the control, the original scene radiance is derived from the attenuation ratio of the liquid crystal and the radiance obtained by the camera. We have implemented a prototype system and conducted experiments in a scene that includes drastic lighting changes. These lighting changes require that we control the radiance of each pixel independently. We show how WDRIs are obtained
by calculating the original scene radiance from these results.
2007 Elsevier Inc. All rights reserved.
Keywords: Reflective liquid crystal; Adaptive radiance control; Wide dynamic range camera
1. Introduction
Digital cameras generally represent brightness information using 8 bits (256 level) in each color channel. In the
real world, there is great variety in the brightness of scenes,
ranging from direct sunlight to the deepest shadow. When
capturing scenes that include drastic lighting changes, the
stronger light saturates the receiving elements and the
actual radiance cannot be obtained. This makes many computer vision problems more difficult. Thus the problem of
obtaining ‘‘Wide Dynamic Range Images’’ (WDRIs) has
attracted much attention from researchers.
Before presenting our approach, we include a brief summary of existing techniques for widening the dynamic
range of a camera. Most of the proposed methods involve
capturing a scene using different exposures [1–5]. This is
done by capturing the scene in a series of sequential frames
and varying the exposure time for each. However, motion
Corresponding author.
E-mail address: [email protected] (H. Mannami).
1047-3203/$ - see front matter 2007 Elsevier Inc. All rights reserved.
in both the camera and the scene makes the registration
of these sequential images more difficult. Naturally these
approaches have limitations for dynamic scenes.
Some methods implement image sensors which have
arithmetic circuits on CMOS. Such circuits add outstanding
features to image sensors. Image sensors which have logarithmic response, have been proposed in [6,7]. Oi et al. have
developed a sensor which adaptively adjusts an extra capacitor to bright areas [8]. Intensity can be measured based on
the time taken to charge a certain amount of incident light
energy [9]. An idea that enhances reflectance of surfaces
(e.g., texture) rather than brightness of regions has been
proposed in [10]. All these methods expand the dynamic
range of an image detector itself, and as such we refer to
them as hardware approaches. The notion of composing
an acceptance surface by receiving elements of different
exposures is introduced in [11–15]. In these approaches, a
number of receiving elements are considered as a single
group, and thus spatial resolution is reduced.
Some novel approaches have also been proposed in
[16,17], where the exposure of each pixel is adaptively
Author's personal copy
H. Mannami et al. / J. Vis. Commun. Image R. 18 (2007) 359–365
controlled. These methods use a camera combined with a
device that can change the attenuation of its pixels independently. The device controls incident radiance by changing
the attenuation based on the radiance. This enables the
camera to use the proper exposure. These methods do
not, however, reduce spatial resolution as they only control
the exposure of conventional cameras. While there are
limitations in scenes with rapid motions, the methods are
applicable, in principle, to dynamic scenes. Moreover, since
these methods use conventional cameras, they can be
combined with the hardware approaches mentioned above.
As a device to control radiance, a transmissive liquid
crystal is used in [16], and a DMD is used in [17]. However,
each of these devices has its own limitations. In transmissive liquid crystals, the driving circuit between the liquid
crystal elements prevents the focus being on the liquid crystal plane. Hence, pixel level radiance control cannot be
achieved. Since DMDs work with time division, precise
radiance control is difficult when DMDs are combined with
a camera that has a fast shutter speed.
In this paper, we present a wide dynamic range camera
system that uses a reflective liquid crystal. The system
adaptively widens the dynamic range of the camera by controlling the incident radiance using the reflective liquid
2. Wide dynamic range imaging through radiance control
In this paper, we introduce a method to widen the
dynamic range using a reflective liquid crystal. We begin
with an introduction to the principle of widening the
dynamic range of a camera, and describe devices for this
2.1. Basic principle to widen dynamic range
Our method widens the dynamic range using a system
that consists of a camera and an attenuation device. The
in–out ratio of the attenuation device can be controlled
for each pixel independently. The method consists of two
2.1.1. Adaptive attenuation control in response to incident
In the first step, the system adaptively controls the incident radiance. As shown in Fig. 1, incident light enters the
image detector through the attenuation device. The attenuation ratio of each element is independently controlled
based on the measured radiance of the corresponding
image pixel. Since incident light passes through the attenuation device before entering the image detector, an increment in the attenuation ratio results in a decrement of
the received radiance. This means that receiving elements
can avoid saturation by increasing the attenuation ratio
of the corresponding attenuation elements. Thus, the range
of acceptable light intensity for the system is expanded
adaptively. Note that this function is different from some
image detector
Fig. 1. Incident light is adaptively controlled by changing attenuation
ratios based on incident radiance.
camera adjustments such as gain control and aperture
adjustment. The function does not affect areas which
receive relatively small levels of radiance. The measured
radiance It at the time t is expressed as
I t ¼ L t At ;
where Lt is the original incident radiance and At is the
attenuation ratio of the device. When a device is controlled
in real-time, this method is applicable to scenes that include
dynamic scene changes.
2.1.2. Restoring original radiance
After controlling the incident radiance, it is possible to
calculate the original radiance, Lt, from the measured radiance and the attenuation ratio as follows,
Lt ¼ I t =At :
Since attenuation is controlled for each receiving pixel, original radiance is also restored in each receiving pixel
according to Eq. (2). The restored image is the same as
the measured image with the exception of the radiance
range, which can be considerably wider than the dynamic
range of the image detector. Considering its radiance value,
the restored image is known as a WDRI.
In addition, the attenuation ratio, At, should be controlled adequately to obtain scene radiance that includes
only small quantization errors. We describe the control
algorithm for the attenuation ratio in Section 3.2.
2.2. Devices for radiance control
Incident radiance is controlled by using devices that can
control the attenuation ratio at the pixel level. A short summary of such devices follows.
Transmissive liquid crystal: Transmissive liquid crystals
are a popular device used in LCDs and several other
products. A model of the device is shown in Fig. 2a. This
device controls polarization states, and thus achieves
contrast, by transmitting incident lights. Because the
device transmits light, driving circuits must exist
between the liquid crystal elements. This however,
reduces the aperture ratio of the device and causes a disadvantage in contrast ratio.
Reflective liquid crystal: This device consists of liquid
crystal between a semiconductor chip and a glass plate.
A model of the device is shown in Fig. 2b. A well-known
Author's personal copy
H. Mannami et al. / J. Vis. Commun. Image R. 18 (2007) 359–365
drive circuit
drive circuit
micro-mirror array
Fig. 2. Simple models of light controlling devices: (a) transmissive liquid crystals, (b) reflective liquid crystals and (c) DMDs. The properties of each
device, as used in our system, are compared in Table 1.
implementation is LCoS (Liquid Crystal on Silicon),
which also achieves contrast by controlling polarization
states, but, in this device, incident lights are reflected.
The device can achieve a high contrast ratio because
driving circuits can be set on the back.
Digital Micro-mirror Device (DMD): DMD is a microelectro-mechanical system that has a tiled micro-mirror
array. As shown in Fig. 2c, the device controls the attenuation by controlling the direction of the reflected light.
The latest generation of DMDs can switch orientations
in a few microseconds, thus enabling modulation of incident light with very high precision.
For realizing attenuation control in combination with a
camera, the device must satisfy the following specifications:
• ability to control attenuation for each imaging pixel,
• compatibility with a high-speed camera.
If the focus is on the attenuator, a problem arises in the
use of transmissive liquid crystals. Their driving circuit is so
large that shadows appear on the captured image when the
focus is on the liquid crystal. This problem makes radiance
measurement more difficult. While the problem may be
solved by special optical devices, such as micro-lens arrays,
the solution increases the complexity of the system. Moreover, when light passes through liquid crystal cells, the cells
produce a diffraction effect, which causes slight blurring in
the captured images [17]. Since DMDs work with time division, they cannot achieve full contrast when combined with
a camera that has a fast shutter speed. Thus a DMD is not
a suitable device to be combined with a camera. In addition, the mechanism tends to be complex because of the
need to synchronize DMD switching and the camera
All the problems mentioned above can be solved by
using reflective liquid crystals. The driving circuit can be
ignored because it does not appear in the light path. Moreover, the blurring effect does not occur because incident
light is reflected at the device. The system also achieves
good performance when combined with a camera because
the principle used to achieve contrast is polarization and
not time division.
The discussion above is summarized in Table 1. Based
on the comparison of each device with respect to the
required specifications, reflective liquid crystals are the
most appropriate device for our purpose.
3. Adaptive dynamic range camera using reflective liquid
We now present our method for a wide dynamic range
camera using reflective liquid crystal. We describe both
the composition of the system and the algorithm for radiance control using a reflective liquid crystal.
3.1. Composition of the system
The basic optical layout of the system is shown in
Fig. 3a. The incident light is first focused onto the liquid
crystal plane by the objective lens. Then the reflected light
is refocused onto the image detector by the camera lens.
Such focusing enables attenuation at a pixel level. The
effect of attenuation is denoted by Eq. (1).
The basic system has a problem in that the focal
length of the lenses is constrained and is required to be
longer than the size of beam splitter. This affects the
objective lens especially and reduces the versatility of
the system. Lenses having a short focal length are thus
unacceptable for use in the system, irrespective of any
user request to use various lenses (e.g., fish-eye lens, telephoto lens, etc.).
We improve the versatility of the system by using relaylenses as shown in Fig. 3b. A relay-lens is a system that
relays images with changing scaling to the other point of
the optical system. As the relay-lens can relay an image
onto the liquid crystal, the system can contain an objective
lens fixed outside the relay-lens. The relay-lens system is
composed of two lenses. A pair of lenses (left and right
in Fig. 3b) focuses incident lights onto the liquid crystal
plane, and another pair (left and bottom in Fig. 3b)
focuses the reflected light from the liquid crystal to the
image detector. Since the image detector is subjected to
attenuated light without blurring, attenuation is achieved
at the pixel level. When considering all the components,
except the objective lens, as one group, the concept of a
camera with radiance control is realized: the focused
Author's personal copy
H. Mannami et al. / J. Vis. Commun. Image R. 18 (2007) 359–365
Table 1
Comparison of each device
liquid crystal
Reflective liquid
Attenuation control at
pixel level
Combination with highspeed camera
Works well
Works well
Not recommended
At1 1
At2 1
image using the objective lens is refocused on the image
detector with the effect of attenuation. Additionally, the
system is downsized for the efficient placement of these
3.2. Control algorithm of reflective liquid crystal
The concept of adaptive attenuation control is represented in Fig. 4. Let us assume that three pixels are subjected to radiance L1, L2 and L3, respectively, and the
attenuation values of the pixels are equal (i.e.,
A1t ¼ A2t ¼ A3t ). The strong light L3 causes saturation. The
weak light L1 is measured with massive quantization errors
however the measured radiance is below the saturation
level. Thus, the quality of the WDRIs obtained is lowered
by these errors. The solution to acquiring more precise
WDRIs is to control the attenuation properly: saturation
is suppressed by incrementing the attenuation from A3t to
A3tþ1 , the quantization error is reduced by decrementing
the attenuation from A1t to A1tþ1 . Note that the time for controlled attenuation is t + 1 since the measured radiance It is
needed for proper control.
We calculate attenuation based on the measured radiance It. When attenuation is properly controlled, the measured radiance at the next time frame, t + 1, will be the
optimal radiance Iopt. The optimal radiance Iopt is now
defined in order to reconstruct the scene radiance precisely.
The optimal attenuation is clearly found from Eq. (1).
I opt
Atþ1 ¼ min
At ; 1 ;
It þ e
where e is a small number.
liquid crystal
beam splitter
Fig. 4. The concept of adaptive attenuation control. Attenuation control
enables the image detector to accept various ranges of radiance with the
least quantization errors.
The next problem that arises is: what is the desired radiance? The most precise WDRI requires controlled radiance
just below the saturation level. However, this method of
radiance control is a trade-off for a weakness in the radiance increment, i.e., a small increment may cause saturation and therefore the actual radiance can never be
obtained. We define Iopt as the median of the camera radiance while taking this trade off into consideration. That is,
I opt ¼
I max þ I min
where Imax and Imin are the maximum and minimum radiance values, respectively, of the camera’s dynamic range.
4. Experimental results
In this section, we describe the prototype system and
present the experimental results.
4.1. Prototype system
We implemented the advanced system presented in
Fig. 3b. The prototype system is shown in Fig. 5. The
LCoS used as the reflective liquid crystal is a Brillian
Z86D-3 model with 800 · 600 elements. A PointGreyResearch Flea 8 bit monochrome camera with 1024 · 768 pix-
liquid crystal
At3 1
Fig. 3. Imaging system using reflective liquid crystal: (a) basic composition, (b) advanced system with relay-lenses. The incident light is first focused onto
the liquid crystal plane. The reflected light is refocused onto the image detector.
Author's personal copy
H. Mannami et al. / J. Vis. Commun. Image R. 18 (2007) 359–365
els is used. A non-polarizing beam splitter and polarization
filters are used to achieve LCoS contrast in the prototype
system. Off-line calibrations were done for both the geometric and radiometric properties of the system. For the
Fig. 8. Comparison of system outputs obtained from Fig. 7. The radiance
outputs are calculated from Eq. (2).
Fig. 5. Implementation of the prototype system. The system consists of a
monochrome camera, reflective liquid crystal, and lenses used as relay and
objective lenses. The system is covered by a jig for capturing.
geometric calibration, mapping between LCoS and CCD
pixels was obtained using homography. Radiometric calibration involves the relationship between LCoS control
and its attenuation ratio. This relationship is important
for proper radiance control. The radiometric calibration
was done by measuring the actual radiance changes that
occur in response to LCoS control. Fig. 6 shows the results
of the radiometric calibration.
Output ratio
4.2. Dynamic range of the prototype system
We evaluated the dynamic range of the prototype system by measuring the camera outputs for incident radiance, which was varied by controlling the shutter speed
of the camera. The system adjusts the attenuation of the
LCoS to avoid saturation with maximum effort. To compare with a conventional camera, the same measurement
was done with the attenuation control turned off. The
results of the measurements are shown in Fig. 7, while
Fig. 8 shows a comparison of the outputs. These results
clearly show that our system is able to capture scenes that
have widely differing radiance information. Our system
achieves a 45.2 times wider range of output than a conventional one.
Control information of the LCoS
Fig. 6. The output ratio of the LCoS. LCoS attenuation response for 8 bit
control. The response is measured at the upper and lower parts of the
camera coordinates. The vertical axis represents the attenuation ratio of
the LCoS. The horizontal axis represents the depth of attenuation control.
A depth of 255 implies the highest attenuation, similar to the color
representation used in many image formats.
Fig. 7. Captured radiance and attenuation are measured for incident radiance: (a) our system, (b) conventional camera. The incident radiance is controlled
precisely by the programmed shutter speed control; that is, in hundreds of steps. The vertical axis represents the depth of the camera radiance and
attenuation. The highest depth of 255 corresponds to the highest value as described in Fig. 6. In order to compare the system with a conventional camera,
the measurement was done with the attenuation function turned off. Thus, attenuation is not recorded in (b).
Author's personal copy
H. Mannami et al. / J. Vis. Commun. Image R. 18 (2007) 359–365
Fig. 9. An outdoor scene taken through a window. (a) Captured image with uniform highest attenuation (initial scene), (b) controlled attenuation for the
scene, (c) captured image after attenuation control. We captured a window from the inside as an example of a scene that includes drastic lighting changes.
Fig. 10. A color movie captured using the prototype system with a color camera. Scene radiance is controlled adaptively in the order of scene changes.
However, rapid camera movement makes artifacts appear in the penultimate image.
Fig. 11. The WDRI obtained by the proposed system. The WDRI is represented by simple image processing for display on an 8-bit monitor: (a and b) the
range of interest is changed: (a) for a dark region and (b) for a bright region, (c) log scale representation. Images (a) and (b) are approximately equivalent
to the images captured when the camera exposure is changed. Both bright and dark regions are clearly visible in these images, namely a stuffed bear and a
box as the dark regions, and the outdoor buildings as the bright regions.
Fig. 12. The WDRIs obtained from Fig. 10. These images are represented in log-scale similar to Fig. 11(c). The leftmost image has lower dynamic range
since the radiance had not been controlled. The artifacts that appeared in the penultimate image of Fig. 10 appear as motion blur in this figure.
4.3. Adaptive radiance control
The function for attenuation control was tested by capturing scenes with drastic lighting changes. The results are
shown in Figs. 9 and 10. Fig. 9 shows the results of an outdoor scene taken through a window. Fig. 9a and c shows
the captured images before and after radiance control,
respectively, and Fig. 9b shows the mask image used for
Author's personal copy
H. Mannami et al. / J. Vis. Commun. Image R. 18 (2007) 359–365
the control. From the results, we see that scene radiance is
controlled adaptively, except in regions subjected to especially strong light, e.g., the specular reflection on a metal
rack. While the dynamic range of the system is about 50
times higher than that of a conventional camera, the prototype system still has limits. We are of the opinion that specular reflection is too strong to capture with our prototype
system. We also intend extending the capacity of the system
by including some of the other approaches described in
Section 1. Though the outdoor scene is not clear in the captured image, it is possible to obtain it from the mask image
used for radiance control.
Fig. 10 shows some frames of color movie footage. In
this figure, the captured images and mask images are
stored in the upper and lower columns, respectively. In
place of the monochrome camera in the prototype system,
we have used a color camera with the same resolution in
this experiment. The scene is captured by moving the camera by hand. Even with the color camera, we control the
radiance simply by changing the attenuation according
to Eq. (2). The radiance is obtained by converting the
color image to gray scale. Since the radiance is not
controlled in the leftmost image, the mask image has no
scene information, and the bright region in the captured
image (e.g., the sunlit wall) is saturated. In our opinion,
the radiance of rest images is controlled as well as that
shown in Fig. 9, except for the penultimate image. This
image includes artifacts created by the LCoS. Because of
the difference in bright regions between adjacent frames,
the radiance control does not work as well when the scene
includes rapid motion.
4.4. Results of wide dynamic range imaging
WDRI is obtained by calculating the original radiance
according to Eq. (2). The results of restoring WDRI are
shown in Figs. 11 and 12. Here, the restored scene is the
same as that shown in the previous section.
In Fig. 11, the original radiance is derived using the
measured radiance (Fig. 9c) and the corresponding attenuation (Fig. 9b). WDRIs have to be compressed somehow to
be displayed on a usual 8-bit monitor. Here, we have represented the WDRI with simple image processing; the
range of interest is changed in Fig. 11a and b, and the
WDRI is converted to a log-scale in Fig. 11c. Fig. 11a
and b shows approximately the same information as the
images captured by changing camera exposure. In
Fig. 11c, we can see both bright and dark regions at the
same time.
Fig. 12 shows the WDRIs obtained from Fig. 10. As the
penultimate image in Fig. 10 has artifacts, the WDRI
obtained from these images also has artifacts. The artifacts
are observed because the FPS is low with respect to the
rapid motion. The artifacts appear as motion blur.
These results show that our system has the ability to represent a considerably large amount of radiance information.
5. Conclusion
In this paper, we have presented an adaptive dynamic
range camera that uses a reflective liquid crystal as a
device to control incident radiance. The camera system
controls the incident radiance for each pixel by controlling attenuation of the liquid crystal. We have constructed a prototype system and verified its applicability
through experiments. As the system can be considered
as a normal camera as shown in Fig. 3b, it is possible
to incorporate other WDR imaging techniques reviewed
in Section 1 and realize even wider dynamic range. Our
current system is however, larger than we would wish
for, as it requires complex optics, as shown in Fig. 3b.
In future work, with the application of some ingenuity,
the system will hopefully be shrunk (for example, by
using Fiber Optic Plate (FOP)).
[1] B.C. Madden, Extended Intensity Range Imaging, Technical Report
MS-CIS-93-96, University of Pennsylvania, 1993.
[2] S. Mann, R. Picard, Being ‘undigital’ with digital cameras: extending
dynamic range by combining differently exposed pictures, in: Proc.
IS&T’s 48th Annual Conf., 1995, pp. 422–428.
[3] P. Debevec, J. Malik, Recovering high dynamic range radiance maps
from photographs, in: Proc. SIGGRAPH, 1997, pp. 369–378.
[4] T. Mitsunaga, S.K. Nayar, Radiometric self calibration, in: Proc.
CVPR, vol. 1, 1999, pp. 374–380.
[5] S.B. Kang, M. Uyttendaele, S. Winder, R. Szeliski, High dynamic
range video, ACM Trans. Graph. 22 (3) (2004) 319–325.
[6] L.W. Lai, C.H. Lai, Y.C. King, A novel logarithmic response CMOS
image sensor with high output voltage swing and in-pixel fixedpattern noise reduction, IEEE Trans. Sens. J. 4 (2004) 122–126.
[7] S. Kavadias, B. Dierichx, D. Scheffer, A. Alaerts, D. Uwaerts, J.
Bogaerts, A logaritmic response CMOS image sensor with on-chip
calibration, Int. J.Solid-State Circuits 35 (8) (2000) 1146–1152.
[8] R. Oi, K. Aizawa, Wide dynamic range imaging by sensitivity
adjustable CMOS image sensor, in: Proc. ICIP, vol. 2, 2003, pp. 583–
[9] V. Brajovic, T. Kanade, A VLSI sorting image sensor: global
massively parallel intensity-to-time processing for low-latency adaptive vision, IEEE Trans. Robotics Automation 15 (1) (1999) 67–75.
[10] V. Brajovic, Brightness perception, dynamic range and noise: a
unifying model for adaptive image sensors, in: Proc. CVPR, vol. 2,
2004, pp. 189–196.
[11] R.J. Handy, High dynamic range CCD detector/imager, US patent
4623928 (1986).
[12] M. Konishi, M. Tsugita, M. Inuiya, K. Masukane, Video camera,
imaging method using video camera, method of operating video
camera, image processing apparatus and method, and solid-state
electronic imaging device, US patent 5420635 (1995).
[13] R.A. Street, High dynamic range segmented pixel sensor array, US
patent 5789737 (1998).
[14] S.K. Nayar, T. Mitsunaga, High dynamic range imaging: Spatially
varying pixel exposures, in: Proc. CVPR, vol. 1, 2000, pp. 472–479.
[15] S.K. Nayar, G. Narasimhan, Assorted pixels: Multi-sampled imaging
with structual models, in: Proc. ECCV, vol. 4, 2002, pp. 636–652.
[16] S.K. Nayar, V. Branzoi, Adaptive dynamic range imaging: optical
control of pixel exposures over space and time, in: Proc. ICCV, vol. 2,
2003, pp. 1168–1175.
[17] S.K. Nayar, V. Branzoi, T. Boult, Programmable imaging using a
digital micromirror array, in: Proc. CVPR, vol. 1, 2004, pp. 436–443.
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF