This article was published in an Elsevier journal. The attached copy is furnished to the author for non-commercial research and education use, including for instruction at the author’s institution, sharing with colleagues and providing to institution administration. Other uses, including reproduction and distribution, or selling or licensing copies, or posting to personal, institutional or third party websites are prohibited. In most cases authors are permitted to post their version of the article (e.g. in Word or Tex form) to their personal website or institutional repository. Authors requiring further information regarding Elsevier’s archiving and manuscript policies are encouraged to visit: http://www.elsevier.com/copyright Author's personal copy J. Vis. Commun. Image R. 18 (2007) 359–365 www.elsevier.com/locate/jvci Adaptive dynamic range camera with reﬂective liquid crystal Hidetoshi Mannami b a,* , Ryusuke Sagawa a, Yasuhiro Mukaigawa a, Tomio Echigo b, Yasushi Yagi a a The Institute of Scientiﬁc and Industrial Research, Osaka University, 8-1 Mihogaoka, Ibaraki, Osaka 567-0047, Japan Department of Engineering Informatics, Faculty of Information and Communication Engineering, Osaka Electro-Communication University, 18-8 Hatsu-cho, Neyagawa, Osaka 572-8530, Japan Received 15 November 2006; accepted 12 June 2007 Available online 21 June 2007 Abstract Wide dynamic range images (WDRIs) are needed for capturing scenes which include drastic lighting changes. This paper presents a method to widen the dynamic range of a camera by using a reﬂective liquid crystal. The system consists of a camera and a reﬂective liquid crystal placed in front of the camera. By controlling the attenuation ratio of the liquid crystal, scene radiance of each pixel is controlled adaptively. After applying the control, the original scene radiance is derived from the attenuation ratio of the liquid crystal and the radiance obtained by the camera. We have implemented a prototype system and conducted experiments in a scene that includes drastic lighting changes. These lighting changes require that we control the radiance of each pixel independently. We show how WDRIs are obtained by calculating the original scene radiance from these results. 2007 Elsevier Inc. All rights reserved. Keywords: Reﬂective liquid crystal; Adaptive radiance control; Wide dynamic range camera 1. Introduction Digital cameras generally represent brightness information using 8 bits (256 level) in each color channel. In the real world, there is great variety in the brightness of scenes, ranging from direct sunlight to the deepest shadow. When capturing scenes that include drastic lighting changes, the stronger light saturates the receiving elements and the actual radiance cannot be obtained. This makes many computer vision problems more diﬃcult. Thus the problem of obtaining ‘‘Wide Dynamic Range Images’’ (WDRIs) has attracted much attention from researchers. Before presenting our approach, we include a brief summary of existing techniques for widening the dynamic range of a camera. Most of the proposed methods involve capturing a scene using diﬀerent exposures [1–5]. This is done by capturing the scene in a series of sequential frames and varying the exposure time for each. However, motion * Corresponding author. E-mail address: [email protected] (H. Mannami). 1047-3203/$ - see front matter 2007 Elsevier Inc. All rights reserved. doi:10.1016/j.jvcir.2007.06.002 in both the camera and the scene makes the registration of these sequential images more diﬃcult. Naturally these approaches have limitations for dynamic scenes. Some methods implement image sensors which have arithmetic circuits on CMOS. Such circuits add outstanding features to image sensors. Image sensors which have logarithmic response, have been proposed in [6,7]. Oi et al. have developed a sensor which adaptively adjusts an extra capacitor to bright areas . Intensity can be measured based on the time taken to charge a certain amount of incident light energy . An idea that enhances reﬂectance of surfaces (e.g., texture) rather than brightness of regions has been proposed in . All these methods expand the dynamic range of an image detector itself, and as such we refer to them as hardware approaches. The notion of composing an acceptance surface by receiving elements of diﬀerent exposures is introduced in [11–15]. In these approaches, a number of receiving elements are considered as a single group, and thus spatial resolution is reduced. Some novel approaches have also been proposed in [16,17], where the exposure of each pixel is adaptively Author's personal copy 360 H. Mannami et al. / J. Vis. Commun. Image R. 18 (2007) 359–365 controlled. These methods use a camera combined with a device that can change the attenuation of its pixels independently. The device controls incident radiance by changing the attenuation based on the radiance. This enables the camera to use the proper exposure. These methods do not, however, reduce spatial resolution as they only control the exposure of conventional cameras. While there are limitations in scenes with rapid motions, the methods are applicable, in principle, to dynamic scenes. Moreover, since these methods use conventional cameras, they can be combined with the hardware approaches mentioned above. As a device to control radiance, a transmissive liquid crystal is used in , and a DMD is used in . However, each of these devices has its own limitations. In transmissive liquid crystals, the driving circuit between the liquid crystal elements prevents the focus being on the liquid crystal plane. Hence, pixel level radiance control cannot be achieved. Since DMDs work with time division, precise radiance control is diﬃcult when DMDs are combined with a camera that has a fast shutter speed. In this paper, we present a wide dynamic range camera system that uses a reﬂective liquid crystal. The system adaptively widens the dynamic range of the camera by controlling the incident radiance using the reﬂective liquid crystal. 2. Wide dynamic range imaging through radiance control In this paper, we introduce a method to widen the dynamic range using a reﬂective liquid crystal. We begin with an introduction to the principle of widening the dynamic range of a camera, and describe devices for this purpose. 2.1. Basic principle to widen dynamic range Our method widens the dynamic range using a system that consists of a camera and an attenuation device. The in–out ratio of the attenuation device can be controlled for each pixel independently. The method consists of two steps. 2.1.1. Adaptive attenuation control in response to incident radiance In the ﬁrst step, the system adaptively controls the incident radiance. As shown in Fig. 1, incident light enters the image detector through the attenuation device. The attenuation ratio of each element is independently controlled based on the measured radiance of the corresponding image pixel. Since incident light passes through the attenuation device before entering the image detector, an increment in the attenuation ratio results in a decrement of the received radiance. This means that receiving elements can avoid saturation by increasing the attenuation ratio of the corresponding attenuation elements. Thus, the range of acceptable light intensity for the system is expanded adaptively. Note that this function is diﬀerent from some image detector Lt attenuation device It At controller Fig. 1. Incident light is adaptively controlled by changing attenuation ratios based on incident radiance. camera adjustments such as gain control and aperture adjustment. The function does not aﬀect areas which receive relatively small levels of radiance. The measured radiance It at the time t is expressed as I t ¼ L t At ; ð1Þ where Lt is the original incident radiance and At is the attenuation ratio of the device. When a device is controlled in real-time, this method is applicable to scenes that include dynamic scene changes. 2.1.2. Restoring original radiance After controlling the incident radiance, it is possible to calculate the original radiance, Lt, from the measured radiance and the attenuation ratio as follows, Lt ¼ I t =At : ð2Þ Since attenuation is controlled for each receiving pixel, original radiance is also restored in each receiving pixel according to Eq. (2). The restored image is the same as the measured image with the exception of the radiance range, which can be considerably wider than the dynamic range of the image detector. Considering its radiance value, the restored image is known as a WDRI. In addition, the attenuation ratio, At, should be controlled adequately to obtain scene radiance that includes only small quantization errors. We describe the control algorithm for the attenuation ratio in Section 3.2. 2.2. Devices for radiance control Incident radiance is controlled by using devices that can control the attenuation ratio at the pixel level. A short summary of such devices follows. Transmissive liquid crystal: Transmissive liquid crystals are a popular device used in LCDs and several other products. A model of the device is shown in Fig. 2a. This device controls polarization states, and thus achieves contrast, by transmitting incident lights. Because the device transmits light, driving circuits must exist between the liquid crystal elements. This however, reduces the aperture ratio of the device and causes a disadvantage in contrast ratio. Reﬂective liquid crystal: This device consists of liquid crystal between a semiconductor chip and a glass plate. A model of the device is shown in Fig. 2b. A well-known Author's personal copy H. Mannami et al. / J. Vis. Commun. Image R. 18 (2007) 359–365 361 output output drive circuit output drive circuit micro-mirror array Fig. 2. Simple models of light controlling devices: (a) transmissive liquid crystals, (b) reﬂective liquid crystals and (c) DMDs. The properties of each device, as used in our system, are compared in Table 1. implementation is LCoS (Liquid Crystal on Silicon), which also achieves contrast by controlling polarization states, but, in this device, incident lights are reﬂected. The device can achieve a high contrast ratio because driving circuits can be set on the back. Digital Micro-mirror Device (DMD): DMD is a microelectro-mechanical system that has a tiled micro-mirror array. As shown in Fig. 2c, the device controls the attenuation by controlling the direction of the reﬂected light. The latest generation of DMDs can switch orientations in a few microseconds, thus enabling modulation of incident light with very high precision. For realizing attenuation control in combination with a camera, the device must satisfy the following speciﬁcations: • ability to control attenuation for each imaging pixel, and • compatibility with a high-speed camera. If the focus is on the attenuator, a problem arises in the use of transmissive liquid crystals. Their driving circuit is so large that shadows appear on the captured image when the focus is on the liquid crystal. This problem makes radiance measurement more diﬃcult. While the problem may be solved by special optical devices, such as micro-lens arrays, the solution increases the complexity of the system. Moreover, when light passes through liquid crystal cells, the cells produce a diﬀraction eﬀect, which causes slight blurring in the captured images . Since DMDs work with time division, they cannot achieve full contrast when combined with a camera that has a fast shutter speed. Thus a DMD is not a suitable device to be combined with a camera. In addition, the mechanism tends to be complex because of the need to synchronize DMD switching and the camera shutter. All the problems mentioned above can be solved by using reﬂective liquid crystals. The driving circuit can be ignored because it does not appear in the light path. Moreover, the blurring eﬀect does not occur because incident light is reﬂected at the device. The system also achieves good performance when combined with a camera because the principle used to achieve contrast is polarization and not time division. The discussion above is summarized in Table 1. Based on the comparison of each device with respect to the required speciﬁcations, reﬂective liquid crystals are the most appropriate device for our purpose. 3. Adaptive dynamic range camera using reﬂective liquid crystal We now present our method for a wide dynamic range camera using reﬂective liquid crystal. We describe both the composition of the system and the algorithm for radiance control using a reﬂective liquid crystal. 3.1. Composition of the system The basic optical layout of the system is shown in Fig. 3a. The incident light is ﬁrst focused onto the liquid crystal plane by the objective lens. Then the reﬂected light is refocused onto the image detector by the camera lens. Such focusing enables attenuation at a pixel level. The eﬀect of attenuation is denoted by Eq. (1). The basic system has a problem in that the focal length of the lenses is constrained and is required to be longer than the size of beam splitter. This aﬀects the objective lens especially and reduces the versatility of the system. Lenses having a short focal length are thus unacceptable for use in the system, irrespective of any user request to use various lenses (e.g., ﬁsh-eye lens, telephoto lens, etc.). We improve the versatility of the system by using relaylenses as shown in Fig. 3b. A relay-lens is a system that relays images with changing scaling to the other point of the optical system. As the relay-lens can relay an image onto the liquid crystal, the system can contain an objective lens ﬁxed outside the relay-lens. The relay-lens system is composed of two lenses. A pair of lenses (left and right in Fig. 3b) focuses incident lights onto the liquid crystal plane, and another pair (left and bottom in Fig. 3b) focuses the reﬂected light from the liquid crystal to the image detector. Since the image detector is subjected to attenuated light without blurring, attenuation is achieved at the pixel level. When considering all the components, except the objective lens, as one group, the concept of a camera with radiance control is realized: the focused Author's personal copy 362 H. Mannami et al. / J. Vis. Commun. Image R. 18 (2007) 359–365 Table 1 Comparison of each device Transmissive liquid crystal Reﬂective liquid crystal DMD Attenuation control at pixel level Combination with highspeed camera Diﬃcult Works well Possible Works well Possible Not recommended L2 L1 1 t A L3 scene radiance At2 At3 At1 1 At2 1 image using the objective lens is refocused on the image detector with the eﬀect of attenuation. Additionally, the system is downsized for the eﬃcient placement of these lenses. 3.2. Control algorithm of reﬂective liquid crystal The concept of adaptive attenuation control is represented in Fig. 4. Let us assume that three pixels are subjected to radiance L1, L2 and L3, respectively, and the attenuation values of the pixels are equal (i.e., A1t ¼ A2t ¼ A3t ). The strong light L3 causes saturation. The weak light L1 is measured with massive quantization errors however the measured radiance is below the saturation level. Thus, the quality of the WDRIs obtained is lowered by these errors. The solution to acquiring more precise WDRIs is to control the attenuation properly: saturation is suppressed by incrementing the attenuation from A3t to A3tþ1 , the quantization error is reduced by decrementing the attenuation from A1t to A1tþ1 . Note that the time for controlled attenuation is t + 1 since the measured radiance It is needed for proper control. We calculate attenuation based on the measured radiance It. When attenuation is properly controlled, the measured radiance at the next time frame, t + 1, will be the optimal radiance Iopt. The optimal radiance Iopt is now deﬁned in order to reconstruct the scene radiance precisely. The optimal attenuation is clearly found from Eq. (1). I opt Atþ1 ¼ min At ; 1 ; ð3Þ It þ e where e is a small number. reflective liquid crystal beam splitter scene object image detector camera Fig. 4. The concept of adaptive attenuation control. Attenuation control enables the image detector to accept various ranges of radiance with the least quantization errors. The next problem that arises is: what is the desired radiance? The most precise WDRI requires controlled radiance just below the saturation level. However, this method of radiance control is a trade-oﬀ for a weakness in the radiance increment, i.e., a small increment may cause saturation and therefore the actual radiance can never be obtained. We deﬁne Iopt as the median of the camera radiance while taking this trade oﬀ into consideration. That is, I opt ¼ I max þ I min ; 2 ð4Þ where Imax and Imin are the maximum and minimum radiance values, respectively, of the camera’s dynamic range. 4. Experimental results In this section, we describe the prototype system and present the experimental results. 4.1. Prototype system We implemented the advanced system presented in Fig. 3b. The prototype system is shown in Fig. 5. The LCoS used as the reﬂective liquid crystal is a Brillian Z86D-3 model with 800 · 600 elements. A PointGreyResearch Flea 8 bit monochrome camera with 1024 · 768 pix- reflective liquid crystal objective lens lens At3 1 objective lens beam splitter imaginary detector lens lens lens image detector camera system Fig. 3. Imaging system using reﬂective liquid crystal: (a) basic composition, (b) advanced system with relay-lenses. The incident light is ﬁrst focused onto the liquid crystal plane. The reﬂected light is refocused onto the image detector. Author's personal copy H. Mannami et al. / J. Vis. Commun. Image R. 18 (2007) 359–365 12000 10000 Radiance els is used. A non-polarizing beam splitter and polarization ﬁlters are used to achieve LCoS contrast in the prototype system. Oﬀ-line calibrations were done for both the geometric and radiometric properties of the system. For the 363 8000 6000 4000 2000 0 0 20 40 60 Fig. 8. Comparison of system outputs obtained from Fig. 7. The radiance outputs are calculated from Eq. (2). Fig. 5. Implementation of the prototype system. The system consists of a monochrome camera, reﬂective liquid crystal, and lenses used as relay and objective lenses. The system is covered by a jig for capturing. geometric calibration, mapping between LCoS and CCD pixels was obtained using homography. Radiometric calibration involves the relationship between LCoS control and its attenuation ratio. This relationship is important for proper radiance control. The radiometric calibration was done by measuring the actual radiance changes that occur in response to LCoS control. Fig. 6 shows the results of the radiometric calibration. Output ratio 1 0.8 4.2. Dynamic range of the prototype system 0.6 We evaluated the dynamic range of the prototype system by measuring the camera outputs for incident radiance, which was varied by controlling the shutter speed of the camera. The system adjusts the attenuation of the LCoS to avoid saturation with maximum eﬀort. To compare with a conventional camera, the same measurement was done with the attenuation control turned oﬀ. The results of the measurements are shown in Fig. 7, while Fig. 8 shows a comparison of the outputs. These results clearly show that our system is able to capture scenes that have widely diﬀering radiance information. Our system achieves a 45.2 times wider range of output than a conventional one. 0.4 0.2 0 0 64 128 192 255 Control information of the LCoS Fig. 6. The output ratio of the LCoS. LCoS attenuation response for 8 bit control. The response is measured at the upper and lower parts of the camera coordinates. The vertical axis represents the attenuation ratio of the LCoS. The horizontal axis represents the depth of attenuation control. A depth of 255 implies the highest attenuation, similar to the color representation used in many image formats. Fig. 7. Captured radiance and attenuation are measured for incident radiance: (a) our system, (b) conventional camera. The incident radiance is controlled precisely by the programmed shutter speed control; that is, in hundreds of steps. The vertical axis represents the depth of the camera radiance and attenuation. The highest depth of 255 corresponds to the highest value as described in Fig. 6. In order to compare the system with a conventional camera, the measurement was done with the attenuation function turned oﬀ. Thus, attenuation is not recorded in (b). Author's personal copy 364 H. Mannami et al. / J. Vis. Commun. Image R. 18 (2007) 359–365 Fig. 9. An outdoor scene taken through a window. (a) Captured image with uniform highest attenuation (initial scene), (b) controlled attenuation for the scene, (c) captured image after attenuation control. We captured a window from the inside as an example of a scene that includes drastic lighting changes. Fig. 10. A color movie captured using the prototype system with a color camera. Scene radiance is controlled adaptively in the order of scene changes. However, rapid camera movement makes artifacts appear in the penultimate image. Fig. 11. The WDRI obtained by the proposed system. The WDRI is represented by simple image processing for display on an 8-bit monitor: (a and b) the range of interest is changed: (a) for a dark region and (b) for a bright region, (c) log scale representation. Images (a) and (b) are approximately equivalent to the images captured when the camera exposure is changed. Both bright and dark regions are clearly visible in these images, namely a stuﬀed bear and a box as the dark regions, and the outdoor buildings as the bright regions. Fig. 12. The WDRIs obtained from Fig. 10. These images are represented in log-scale similar to Fig. 11(c). The leftmost image has lower dynamic range since the radiance had not been controlled. The artifacts that appeared in the penultimate image of Fig. 10 appear as motion blur in this ﬁgure. 4.3. Adaptive radiance control The function for attenuation control was tested by capturing scenes with drastic lighting changes. The results are shown in Figs. 9 and 10. Fig. 9 shows the results of an outdoor scene taken through a window. Fig. 9a and c shows the captured images before and after radiance control, respectively, and Fig. 9b shows the mask image used for Author's personal copy H. Mannami et al. / J. Vis. Commun. Image R. 18 (2007) 359–365 the control. From the results, we see that scene radiance is controlled adaptively, except in regions subjected to especially strong light, e.g., the specular reﬂection on a metal rack. While the dynamic range of the system is about 50 times higher than that of a conventional camera, the prototype system still has limits. We are of the opinion that specular reﬂection is too strong to capture with our prototype system. We also intend extending the capacity of the system by including some of the other approaches described in Section 1. Though the outdoor scene is not clear in the captured image, it is possible to obtain it from the mask image used for radiance control. Fig. 10 shows some frames of color movie footage. In this ﬁgure, the captured images and mask images are stored in the upper and lower columns, respectively. In place of the monochrome camera in the prototype system, we have used a color camera with the same resolution in this experiment. The scene is captured by moving the camera by hand. Even with the color camera, we control the radiance simply by changing the attenuation according to Eq. (2). The radiance is obtained by converting the color image to gray scale. Since the radiance is not controlled in the leftmost image, the mask image has no scene information, and the bright region in the captured image (e.g., the sunlit wall) is saturated. In our opinion, the radiance of rest images is controlled as well as that shown in Fig. 9, except for the penultimate image. This image includes artifacts created by the LCoS. Because of the diﬀerence in bright regions between adjacent frames, the radiance control does not work as well when the scene includes rapid motion. 4.4. Results of wide dynamic range imaging WDRI is obtained by calculating the original radiance according to Eq. (2). The results of restoring WDRI are shown in Figs. 11 and 12. Here, the restored scene is the same as that shown in the previous section. In Fig. 11, the original radiance is derived using the measured radiance (Fig. 9c) and the corresponding attenuation (Fig. 9b). WDRIs have to be compressed somehow to be displayed on a usual 8-bit monitor. Here, we have represented the WDRI with simple image processing; the range of interest is changed in Fig. 11a and b, and the WDRI is converted to a log-scale in Fig. 11c. Fig. 11a and b shows approximately the same information as the images captured by changing camera exposure. In Fig. 11c, we can see both bright and dark regions at the same time. Fig. 12 shows the WDRIs obtained from Fig. 10. As the penultimate image in Fig. 10 has artifacts, the WDRI obtained from these images also has artifacts. The artifacts are observed because the FPS is low with respect to the rapid motion. The artifacts appear as motion blur. These results show that our system has the ability to represent a considerably large amount of radiance information. 365 5. Conclusion In this paper, we have presented an adaptive dynamic range camera that uses a reﬂective liquid crystal as a device to control incident radiance. The camera system controls the incident radiance for each pixel by controlling attenuation of the liquid crystal. We have constructed a prototype system and veriﬁed its applicability through experiments. As the system can be considered as a normal camera as shown in Fig. 3b, it is possible to incorporate other WDR imaging techniques reviewed in Section 1 and realize even wider dynamic range. Our current system is however, larger than we would wish for, as it requires complex optics, as shown in Fig. 3b. In future work, with the application of some ingenuity, the system will hopefully be shrunk (for example, by using Fiber Optic Plate (FOP)). References  B.C. Madden, Extended Intensity Range Imaging, Technical Report MS-CIS-93-96, University of Pennsylvania, 1993.  S. Mann, R. Picard, Being ‘undigital’ with digital cameras: extending dynamic range by combining diﬀerently exposed pictures, in: Proc. IS&T’s 48th Annual Conf., 1995, pp. 422–428.  P. Debevec, J. Malik, Recovering high dynamic range radiance maps from photographs, in: Proc. SIGGRAPH, 1997, pp. 369–378.  T. Mitsunaga, S.K. Nayar, Radiometric self calibration, in: Proc. CVPR, vol. 1, 1999, pp. 374–380.  S.B. Kang, M. Uyttendaele, S. Winder, R. Szeliski, High dynamic range video, ACM Trans. Graph. 22 (3) (2004) 319–325.  L.W. Lai, C.H. Lai, Y.C. King, A novel logarithmic response CMOS image sensor with high output voltage swing and in-pixel ﬁxedpattern noise reduction, IEEE Trans. Sens. J. 4 (2004) 122–126.  S. Kavadias, B. Dierichx, D. Scheﬀer, A. Alaerts, D. Uwaerts, J. Bogaerts, A logaritmic response CMOS image sensor with on-chip calibration, Int. J.Solid-State Circuits 35 (8) (2000) 1146–1152.  R. Oi, K. Aizawa, Wide dynamic range imaging by sensitivity adjustable CMOS image sensor, in: Proc. ICIP, vol. 2, 2003, pp. 583– 586.  V. Brajovic, T. Kanade, A VLSI sorting image sensor: global massively parallel intensity-to-time processing for low-latency adaptive vision, IEEE Trans. Robotics Automation 15 (1) (1999) 67–75.  V. Brajovic, Brightness perception, dynamic range and noise: a unifying model for adaptive image sensors, in: Proc. CVPR, vol. 2, 2004, pp. 189–196.  R.J. Handy, High dynamic range CCD detector/imager, US patent 4623928 (1986).  M. Konishi, M. Tsugita, M. Inuiya, K. Masukane, Video camera, imaging method using video camera, method of operating video camera, image processing apparatus and method, and solid-state electronic imaging device, US patent 5420635 (1995).  R.A. Street, High dynamic range segmented pixel sensor array, US patent 5789737 (1998).  S.K. Nayar, T. Mitsunaga, High dynamic range imaging: Spatially varying pixel exposures, in: Proc. CVPR, vol. 1, 2000, pp. 472–479.  S.K. Nayar, G. Narasimhan, Assorted pixels: Multi-sampled imaging with structual models, in: Proc. ECCV, vol. 4, 2002, pp. 636–652.  S.K. Nayar, V. Branzoi, Adaptive dynamic range imaging: optical control of pixel exposures over space and time, in: Proc. ICCV, vol. 2, 2003, pp. 1168–1175.  S.K. Nayar, V. Branzoi, T. Boult, Programmable imaging using a digital micromirror array, in: Proc. CVPR, vol. 1, 2004, pp. 436–443.
* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project