Closed-Loop Feedback Illumination for Optical Inverse

Closed-Loop Feedback Illumination for Optical Inverse
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,
VOL. 17,
NO. 6,
JUNE 2011
857
Closed-Loop Feedback Illumination for Optical
Inverse Tone-Mapping in Light Microscopy
Oliver Bimber, Member, IEEE Computer Society, Daniel Klöck, Toshiyuki Amano, Member, IEEE,
Anselm Grundhöfer, and Daniel Kurz
Abstract—In this paper, we show that optical inverse tone-mapping (OITM) in light microscopy can improve the visibility of specimens,
both when observed directly through the oculars and when imaged with a camera. In contrast to previous microscopy techniques, we
premodulate the illumination based on the local modulation properties of the specimen itself. We explain how the modulation of uniform
white light by a specimen can be estimated in real time, even though the specimen is continuously but not uniformly illuminated. This
information is processed and back-projected constantly, allowing the illumination to be adjusted on the fly if the specimen is moved or
the focus or magnification of the microscope is changed. The contrast of the specimen’s optical image can be enhanced, and highintensity highlights can be suppressed. A formal pilot study with users indicates that this optimizes the visibility of spatial structures
when observed through the oculars. We also demonstrate that the signal-to-noise (S/N) ratio in digital images of the specimen is higher
if captured under an optimized rather than a uniform illumination. In contrast to advanced scanning techniques that maximize the S/N
ratio using multiple measurements, our approach is fast because it requires only two images. This can improve image analysis in digital
microscopy applications with real-time capturing requirements.
Index Terms—Computer graphics, picture/image generation, display algorithms, image processing, enhancement.
Ç
1
INTRODUCTION AND MOTIVATION
I
N light microscopy (or optical microscopy), visible light is
either transmitted through or reflected from a specimen
before it is observed or recorded. In bright field microscopy,
the illumination light is modulated in intensity and color
depending on the specimen’s transmission or reflection
properties before it enters the objective lens. The resolution
of this method is limited by the contrast and the wavelength
of visible light. Several other imaging and illumination
techniques are commonly employed in light microscopes to
enhance contrast, such as dark field, phase contrast,
(differential) interference contrast, fluorescence, oblique,
and Rheinberg illumination. By exploiting refraction,
diffraction, interference, or fluorescence of light, these
methods are applied mainly to make objects, for instance,
biological structures such as cells, visible that are otherwise
invisible. Compared to bright field microscopy, the optical
images produced with such contrast techniques usually
give an unnatural appearance to the observed specimens. A
considerable degree of experience is required to interpret
. O. Bimber is with the Institute of Computer Graphics, Johannes Kepler
University Linz, Altenberger Straße 69, Science Park I, 3rd Floor,
MT0323, 4040 Linz, Austria. E-mail: [email protected]
. D. Klöck is with the Brandenburg Technical University Cottbus,
Münchner Str. 27B, 82467 Garmisch-Partenkirchen, Germany.
E-Mail: [email protected]
. T. Amano is with the Graduate School of Information Science, Nara
Institute of Science and Technology, 8916-5 Takayama, Ikoma, Nara 6300101, Japan. E-mail: [email protected]
. A. Grundhöfer and D. Kurz are with the Bauhaus-University Weimar,
Germany. E-Mail: {grundhoe, kurz}@uni-weimar.de.
Manuscript received 24 Aug. 2009; revised 1 Feb. 2010; accepted 6 July 2010;
published online 29 July 2010.
Recommended for acceptance by T. Möller.
For information on obtaining reprints of this article, please send e-mail to:
[email protected], and reference IEEECS Log Number TVCG-2009-08-0185.
Digital Object Identifier no. 10.1109/TVCG.2010.104.
1077-2626/11/$26.00 ! 2011 IEEE
correctly the resulting gradient and false color images. In
addition, specimens that are too thick for transmitted light
require a reflected illumination, for which only a few means
of contrast enhancement are currently available. Since a
detailed discussion of these techniques is beyond the scope
of this paper, interested readers are referred to microscopy
text books, such as [16], and to online sources.1 In this
paper, we focus entirely on enhancing classical bright field
microscopy.
In domains that utilize bright field microscopy, such as
microscopic surgery, forensic analysis, materials science, and
bioimaging, contrast is often too low or too high, making
observations directly through the oculars problematic, since
the capabilities of the human visual system are limited. Low
contrast can result from low-contrast specimens or a high
degree of scattering within the specimen. Excessively high
contrast can be the result of specular highlights on the
specimen. Furthermore, the acquisition of extremely low- or
high-contrast images is problematic in many digital microscopy applications that rely on robust image analysis.
In this paper, we describe how the visibility of specimens
observed through the oculars can be enhanced, and how the
signal-to-noise (S/N) ratio of captured images can be
increased using a technique we refer to as optical inverse
tone-mapping (OITM). In both cases, we control the microscope illumination spatially in real time, as illustrated in
Fig. 1. This allows highly interactive and dynamic optical
examinations and image analysis tasks that involve specimen movements and changing observation parameters,
such as focus and magnification. This is essential in many
applications of light microscopy and related domains,
including endoscopy and microscopic surgery.
1. http://www.microscopyu.com/, http://www.olympusmicro.com/,
http://zeiss-campus.magnet.fsu.edu/.
Published by the IEEE Computer Society
858
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,
Fig. 1. Schematic drawing of optical inverse tone-mapping in bright field
microscopy: The illumination is premodulated by the spatial light
modulator of a projector, based on the camera-recorded local modulation properties of the specimen itself. It is modulated a second time
when transmitted or reflected by a specimen. This results in a contrastenhanced optical image that can be recorded with a camera or observed
through the oculars. The drawing illustrates an example of reflected
illumination. Prototypes for reflected and transmitted illumination are
presented in Section 5.
Unlike the contrast techniques listed above, our approach allows visibility enhancement while preserving the
natural appearance of the specimens. It holds particular
potential for real-time applications that depend on bright
field microscopy and require a reflected illumination.
2
RELATED WORK AND CONTRIBUTION
2.1 Spatial Light Modulation in Optical Microscopy
Spatial light modulators (SLMs), such as LCDs and DMDs,
are used at different stages of the microscope optical train to
generate or enhance contrast. Fig. 2 complements the
following summary with an overview of a simplified
microscope optical train.
SLMs are integrated at the objective back focal plane to
apply phase-stepping contrast techniques [22] or pointspread-function shaping to reduce aberrations in confocal
microscopy [20]. They generate annulus masks dynamically
at the condenser aperture plane to support imaging and
illumination techniques, such as dark field, Rheinberg, and
oblique illumination, but also optical staining without the
need to change physical filters mechanically [26]. SLMs are
also, as in our case, focused at the field aperture plane. In
fluorescence microscopy, for instance, this allows different
areas of the specimen to be excited individually by using
manually defined binary illumination patterns [28], [9]. This
method avoids high scattering and color artifacts that are
caused by autofluorescence. In addition, dynamic field
apertures (instead of static ones) are also employed in
modern ophthalmological devices for examining the eyeballs and for protecting the retina from bright light during
eye surgery. Light field microscopy [13] captures 4D radiant
light fields by placing a microlens array at the intermediate
image plane of the imaging path. The incident light field
can be controlled by using a second microlens array at the
intermediate image plane of the illumination path and a
projector [14]. Controlling the incident light field makes
possible reproducing techniques such as dark field and
oblique illumination, and also allows directing and focusing
light rays at selected positions and depths. Using a
mechanism similar to that described in [9] this can also
reduce negative effects caused by autofluorescence.
VOL. 17,
NO. 6,
JUNE 2011
Fig. 2. Diagram of simplified microscope optical train: The condenser
focuses the field aperture into the field plane and into the intermediate
image plane (imaging path) and, at the same time, focuses the light
sources into the condenser aperture plane and into the objective back
focal plane (illumination path). For simplicity, the example shows a
transmitted illumination configuration.
None of the techniques described above premodulate
the illumination in intensity or color based on the local
modulation properties of the specimen itself. We, however,
show that this can improve the visibility of the resulting
optical image when observed through the oculars or
captured in digital images. Microscopy techniques that
apply SLMs for scanning, measuring, superresolution
imaging, and the like already exist. We enhance optical
images in real time, rather than produce enhanced digital
images using a substantial number of measurements.
2.2
High Dynamic Range Imaging and Optical
Tone-Mapping
High dynamic range (HDR) imaging is frequently applied in
microscopy. Like conventional multiexposure techniques
[8], alternative HDR imaging methods preattenuate the scene
radiance on a per-pixel basis with SLMs before it reaches the
camera sensor. In [15], an LCoS panel is used for premodulation. Multiple images are recorded while adapting the
response of the LCoS panel iteratively until no sensor pixel
remains saturated. Using the final camera image and
attenuation values, the original scene radiance can be
reconstructed. For optical tone-mapping, SLMs are also
employed to attenuate the incoming scene radiance to
the dynamic range of the camera sensor [17], [18], [11]. Some
approaches need multiple recording iterations [11], while
others support video-rate capturing [17]. However, such
techniques normally do not produce high dynamic range
images. Rather than a contrast enhancement, a contrast
reduction of the scene radiance is usually achieved, and thus
unsaturated low dynamic range (LDR) images are captured.
Optical tone-mapping is also used in optical see-through
devices rather than in imaging applications [29]. In this case,
contrast enhancement of the scene radiance is possible.
A fundamental limitation of HDR imaging and most
optical tone-mapping techniques is that the contrast of the
recorded or observed optical images themselves cannot be
increased before being imaged or viewed. Such methods
ensure only that the entire contrast range of the scene is
BIMBER ET AL.: CLOSED-LOOP FEEDBACK ILLUMINATION FOR OPTICAL INVERSE TONE-MAPPING IN LIGHT MICROSCOPY
captured or attenuated to the dynamic range of the camera.
Although digital contrast techniques, such as inverse tonemapping [5], could be applied after recording, imaging
techniques alone cannot counter contrast reductions that
result from light modulation by the specimen. Furthermore,
details within high-intensity reflection regions also cannot be
revealed after imaging under a uniform illumination. This
also holds true for the optical see-through tone-mapping
technique in [29], since contrast enhancement or highlight
suppression is performed after the light is modulated with
the scene. Overcoming these problems is only possible if the
illumination can be controlled before it is modulated by a
scene or, as in our context, by a specimen. For the recording
of high dynamic range radiance maps in microscopy, this
was explored in [1], where multiple images were captured at
the same exposure but with successively decreasing local
illumination in saturated regions. However, another essential limitation of HDR imaging and optical tone-mapping is
that most techniques require capturing multiple images. This
prevents them from recording dynamic scenes (e.g., moving
specimens) or scenes under changing imaging conditions
(e.g., adapting magnification and focus). This limitation also
applies to programmable array microscopy [28], where the
illumination path and the imaging path can be spatially
modulated simultaneously.
Our technique, however, allows observing and recording
contrast-enhanced optical images of specimens in real time
and in interactive and dynamic situations. Low contrast that
results from the modulation of light by the specimen can be
amplified, since the irradiant illumination rather than the
imaged radiance is modulated. Furthermore, highlighting
relevant structures and suppressing the background leads to
a decreased amount of scattering and hence to an additional
gain in contrast. In addition, appropriate illumination
control makes it possible to image and observe instantly
details in regions of surface- and subsurface reflections.
2.3 Projector-Based Illumination
Projectors have been used for spatial light modulation in
backlights of high dynamic range displays that employ
LCDs as front modulators [27]. Here, light is modulated
twice; once by the backlight projector and a second time by
the LCD front panel. This is generally referred to as doublemodulation. It leads to high contrast and high dynamic range
images if both modulators can be controlled with respect to
the displayed content. A variety of other approaches exist
that realize HDR displays based on this principle by
applying different kinds of SLM, such as Liquid Crystal
Displays (LCDs), Digital Micromirror Devices (DMDs),
Light-Emitting Diodes (LEDs), and Liquid Crystal on
Silicon (LCoS). Several projector-camera techniques exist
that improve the quality of video presentations projected
onto nonoptimized screens. They aim to compensate for
artifacts that are caused by textured, colored, or nonplanar
surfaces. Some of them are related to our approach. In [24],
for instance, displaying overlapping images from different
angles using multiple projectors compensates for highlights
on specular screen surfaces. The closed-loop photometric
adaptation method described in [10] is another example,
which estimates the optimal compensation image to be
projected onto a textured surface. It does not imply a static
registration between projector and surface.
859
The techniques described in [6], [4] are closely related,
since they also double-modulate precaptured images on
physical surfaces. In these cases, video projectors are
registered precisely to paper printouts. By computing
accurately the printed and the projected images, the method
in [6], for instance, allows displaying high dynamic range
images, such as those required for radiological visualizations. These were the first applications of OITM that used a
projector-based illumination for contrast enhancement.
2.4 Summary of Contributions
In addition to applying OITM to microscopy, we present
three essential contributions: First, we introduce a closedloop feedback framework that allows OITM to be applied in
real time, thus enabling interactive and dynamic optical
examinations and image analysis tasks. We explain the
implications of OITM in the context of complex light
modulation caused by microscopic specimens and present
solutions that make interactive applications of OITM more
robust. Second, we demonstrate that the S/N ratio in
images that are optically contrast-enhanced with OITM
before recording is higher than in images that are captured
under uniform illumination. In contrast to advanced
scanning or smoothing techniques that maximize the S/N
ratio by means of multiple measurements, our approach is
fast because it requires only two images. Third, we present
a formal pilot study with users, which indicates that OITM
outperforms a uniform illumination in directly perceiving
small and large specimen details through the oculars.
3
OPTICAL INVERSE TONE-MAPPING
3.1 Basic Principle
To enhance contrast with OITM, we can capture the
specimen under uniform (white) illumination, contraststretch the recorded image, and then project the result back
to the specimen, as illustrated in Fig. 1. Thus, contrast
amplification of the optical image is achieved by doublemodulation: The back-projected image of the specimen is
modulated a second time by the specimen itself. In light
microscopy, this is feasible under both transmitted and
reflected illumination. Prototype configurations for these
cases are presented in Section 5. Fig. 3 shows two specimens
for which the physical contrast was boosted optically with
OITM. We achieved increases in physical contrast of up to a
factor of 16 (e.g., for the cricket skin shown in Fig. 3d) and
up to 11,000:1 (e.g., when contrast-boosting the pyrite
sample shown in Fig. 10).
The behavior of OITM can be explained using the
principles of light transport. As illustrated in Fig. 4, we
assume that camera and projector optics are perfectly
coaligned in the microscope optical train (symbolically
represented by a single lens). In this case, each camera/
projector-pixel pair on the shared image plane can both emit
and integrate light rays (at the same optically folded solid
angle) that intersect at the adjusted focal plane, but may
continue in nonopaque specimen volumes. Light can be
transported from each projector pixel p to each camera pixel c
while being modulated on the way. This modulation can be
described by the forward light transport, with !i;j representing the transport from pi to cj (given that Helmholtz
reciprocity is valid, then !i;j ¼ !j;i ):
860
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,
VOL. 17,
NO. 6,
JUNE 2011
Fig. 3. Stretching physical contrast by optical inverse tone-mapping: (a) A cricket skin captured under reflected illumination. The heat-maps illustrate
the measured physical contrast under the same peak luminance (b) for uniform illumination and (c) and (d) for different contrast-stretched backprojected illuminations. The honey bee was captured under transmitted illumination: (e) uniform white and (f) contrast-stretched back-projected. The
histogram profiles are illustrated for both cases (g). All measurements were taken from captured HDR images. The numbers indicate the contrast
measured and the contrast increase relative to a uniform illumination.
2
3 2
!1;1
c1
4...5 ¼ 4 ...
!1;n
cn
32 3
. . . !n;1
p1
. . . . . . 54 . . . 5:
. . . !n;n
pn
ð1Þ
If T denotes the symmetric light transport matrix and
½p1 ; . . . ; pn % ¼ ½1; . . . ; 1% ¼ p describes a uniform illumination,
then the resulting camera image ½c1 ; . . . ; cn % ¼ c can be
computed according to c ¼ T p ¼ T 1, with 1 indicating an
all-ones vector. We can assume that the camera image is
proportional to the optical image that can be observed
through the oculars. If this image is back-projected with
OITM, the new resulting camera image is c0 ¼ T c ¼
T ðT 1Þ ¼ T 2 1. This means that, in this case, the result of OITM
is equivalent to squaring the entire light transport (with T 2
also being symmetric). For specimens that cause little
scattering (e.g., thin or opaque specimens), T is sparse and
Fig. 4. OITM light transport: (a) Coaligned projector-camera pixels are
focused on a plane inside a specimen volume. Light can be transported
from all projector pixels through the volume (where it interacts) to all
camera pixels (where it is integrated). If the interaction of light is
dominant (e.g., due to a high degree of scattering), then OITM becomes
inefficient but does not geometrically bias the resulting optical image.
This is shown using transparent 500-600 "m thick polystyrene microspheres that are illuminated under uniform white (b) and (e) and under
OITM (c) and (f) illumination. For a single bead on an opaque
background, a contrast increase of a factor of 6.9 can be achieved (b)
and (c). For a 1 cm thick pile of beads, the contrast increase drops to a
factor of 1.2. The reason for this is the significantly larger amount of
scattering inside the volume of the bead pile. The gradient inlays and
scanline plots in (d)-(g) illustrate that, in both cases, features and their
gradient profiles are preserved, while only gradient magnitudes vary due
to contrast enhancement.
OITM is efficient, since T 2 enhances the contrast of the direct
light transport (e.g., caused by direct reflection or transmission). When more scattering occurs (e.g., caused by specimens with a high degree of volume scattering or strong
surface interreflections), then T is denser, and OITM becomes
less effective. Compared to a uniform illumination, OITM
preserves gradient profiles of features and therefore does not
geometrically bias the resulting optical image. If an additional
contrast-stretching function (f) is applied to c before backprojecting it, then T is not simply squared, but OITM results
in a more complex light transport: c0 ¼ T fðcÞ ¼ T fðT 1Þ.
Nonetheless, gradient profiles are preserved if f is monotonically increasing. Two examples with different degrees of
scattering are illustrated in Figs. 4b, 4c, 4d, 4e, 4f, and 4g.
3.2 Increasing S/N Ratio
Especially, in photon-limited situations in which photon
noise is the major source of noise (which applies to most
cases in light microscopy), the signal-to-noise ratio can be
increased by stretching the physical contrast optically with
OITM before capturing images that are processed further.
Depending on the specimen, we achieve a S/N gain of up to
a factor of 3-4 in our experiments with our current
prototype. Fig. 5 illustrates an example.
We measured the S/N ratio in each image with Newberry’s analytical equation [21],[16], which is frequently
used in digital microscopy:
Fig. 5. Increasing S/N ratio: Capturing a low-contrast edge (1.22:1) under
white light (a) results in an image with a low S/N ratio (g). Contraststretching and back-projecting this image instead of using a uniform
illumination enhances the signal through double-modulation with the
specimen (c). A higher S/N ratio (i) leads to improved results in digital
image processing techniques, such as automatic thresholding [23].
Thresholding results (b), (d), and (f) show the deviations from the ground
truth (h). Many images have to be recorded and averaged under uniform
illumination for reducing noise at a constant signal level (e), (i), (j), and (k)
to achieve similar results (f) and (l). The contrast edge was created on
Kodak photo paper using a dye-sublimation heat transfer printer.
BIMBER ET AL.: CLOSED-LOOP FEEDBACK ILLUMINATION FOR OPTICAL INVERSE TONE-MAPPING IN LIGHT MICROSCOPY
pffiffiffiffiffiffi
Co
ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
ffi;
q
S=N ¼
1
n#2
n#2
þ
þ
g
Co
pCo
861
ð2Þ
where 1 dB ¼ 20 log S=N, Co ¼ Cf ' Cb is the object count
in analog to digital units (ADU) with Cf and Cb being the
pixel sums within the selected regions that define the object
fore- and background; n is the number of pixels in the object
area, p is the number of pixels in the background area, #2 is
the variance of background pixels (in ADU 2 ), and g is the
camera’s electron gain in electrons/ADU (in 1=ADU).
Before back-projecting, we can also contrast-stretch the
initial image (M) recorded under white light, using nonlinear tone-mapping:
"
" "
###$
L ' MIN ' ðMAX ' MINÞo
L0 ¼ min 1; max 0;
;
ðMAX ' MINÞw
ð3Þ
where L is the original luminance and MAX ' MIN is the
original luminance range of M. A luminance window can be
selected in this range by defining its offset (o 2 ½0; 1%) from
MIN and its width (w 2%0; 1 ' o%). It allows the relevant
luminance range in M to be extracted. Values within the
window are first normalized to ½0; 1% and are then remapped
exponentially by $. Thus, w, o, and $ can be adjusted
individually to achieve the highest possible contrast without
causing saturation in local regions. The tone-mapped
luminance L0 is recombined with the chrominance in M if a
colored illumination is required. Otherwise, only L is
displayed in the dynamic range of the projector. Note that
contrast-stretching M boosts both the signal and the noise
level in the illumination. Double-modulation with the
specimen, however, amplifies the signal level more than
the noise level, since the noise pattern is not present in the
specimen. Furthermore, minimizing scattering (e.g., by the
specimen) and defocus (e.g., due to the microscope’s limited
depth of field), and digital noise compensation reduce the
high-frequency noise components in the illumination.
The S/N gain for a sliced tissue sample and various
exposure times is presented in Fig. 6. Note that we used a
Point Gray Dragonfly 2 CCD camera (37e=ADU) for these
measurements to satisfy Nyquist’s criterion for sampling
the illumination pattern by the camera signal. In this case,
the projector pixel size (3:15 "m) is less than half the
composite camera pixel size (7:05 "m) on the field plane.
As shown in Fig. 6, the S/N ratio in images that are
optically contrast-enhanced before recording is always
higher than in images that are captured under uniform
illumination, even when employing additional digital
contrast techniques. This also applies to HDR images
computed from the individual exposures and to their
tone-mapped LDR counterparts. For the example shown
in Fig. 6, we measured an S/N gain of 27:9 dB/13:6 dB ¼
2:1 in the resulting HDR images, and a gain of 17:2 dB/
11:5 dB ¼ 1:5 in linearly tone-mapped LDR versions that
were computed from the corresponding HDR images.
An alternative to stretching the signal with OITM is
smoothing the noise level by averaging multiple images
recorded under uniform illumination. This increases the S/N
ratio by decreasing noise, while the signal remains at the
same level, since contrast is not enhanced. For low-contrast
Fig. 6. S/N ratio gain at different exposures (top): Comparing uniform
white and contrast-stretched back-projected illumination with $ ¼ 1:0. For
overexposed images S/N is 0. The close-ups visualize the S/N ratio for
uniform and projected illumination at the same signal level. Comparing
optical contrast-enhancement before recording and digital contrastenhancement after recording (bottom): For digital contrast-enhancement,
we applied contrast-limited adaptive histogram equalization (CLAHE)
and Photoshop’s nonlinear contrast operator with two different contrast
settings. Slight S/N variations after digital contrast amplification are due
to regional pixel saturations. The specimen is a sliced tissue sample.
specimens, however, smoothing requires capturing significantly more images to reach an adequate S/N ratio than
OITM, which needs just two recordings, as shown in Fig. 5.
For high-contrast specimens, the noise level is nearly
irrelevant compared to the signal level for detecting features,
and neither smoothing nor OITM would lead to noteworthy
improvements. Compared to bias noise, heat noise, and
thermal noise, the level of photon noise is dominant in light
microscopy. A contrast edge p
between
two
pffiffiffi intensities a and b
ffiffiffi
(a > b) can be detected if a ' a > b þ b. Although a bright
uniform illumination can also increase the S/N, it maybe
avoided to prevent heat damage to the specimen. OITM does
the
not only increase the contrast ratio (a:b) of p
pffiffibut
ffi it
ffiffiffi signal,
also decreases the photon noise level ( a and b) by
adapting the illumination intensity regionally.
An increased S/N ratio is beneficial to image analysis in
digital microscopy, as the thresholding examples in Figs. 5
and 7 show. For automatic thresholding, we used the usual
method by Otsu [23], which chooses a threshold to
minimize the intraclass variance of black and white pixels.
In digital microscopy, more advanced scanning techniques
can be applied to compose digital images with a high S/N
ratio by sampling the specimen with a relatively large series
of structured illumination patterns. The advantage of our
method is that an increased S/N ratio can be achieved by
capturing only two images. Although the quality of multisampling is likely to be higher, fast OITM can be beneficial
in applications with real-time requirements.
3.3 Enhancing Visibility of Spatial Structures
Equation 3 stretches all entries within the selected luminance window to the displayable range of the projector
(linearly or nonlinearly, depending on the value of $). Small
contrast differences in bright or dark regions can be further
amplified with a gamma compression ($ < 1) or a gamma
expansion ($ > 1). This, however, can lead to extremely
862
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,
Fig. 7. Automatic thresholding [23] examples: HDR images were
captured under uniform illumination and under contrast-stretched
back-projected illumination ($ ¼ 1:0). Different exposures are shown
for the sliced tissue sample in (a) and (c), while the results of automatic
thresholding of the HDR images are presented in (b) and (d).
Improvements by automatic thresholding for a low-contrast edge of a
conductor path (e) are also shown in (f) and (g). The difference to the
ground truth edge (solid line) is considered to be inversely proportional
to the fill factor of the area between the solid line and the dashed line
(compared to an entirely filled area in the best case).
high local contrast values in the resulting optical image.
While a stretched physical contrast can be beneficial in
digital microscopy applications as long as the camera
provides the required dynamic range, it is normally not
advantageous when visibility is to be improved for direct
observations through the oculars. The human visual
system’s dynamic range for simultaneous perception without readaptation is limited. Dark details neighboring bright
ones are visually suppressed if their contrast is too high.
In scientific imaging, such as medical imaging, CLAHE
[25], [30] is often applied to improve the visual detection of
spatial structures in images. A histogram equalization
spreads out the most frequent luminance values instead of
stretching the entire luminance range exponentially. In the
context of OITM, we investigated the efficiency of CLAHE
for contrast-stretching the illumination pattern to enhance
the visibility of spatial structures which are observed
directly through the oculars. Furthermore, we coupled
CLAHE with decorrelation stretch (DS) [2] to also maximize
the difference between chrominances.
As before, all computations were carried out based on the
white-light modulation image M. We computed CLAHE
using the perceptually uniform luminance channel L* of M
in the CIE L*a*b* color space. CLAHE examines the
histogram of intensities for N ( M subimages (called tiles of
size tx ) ty ) and calculates equalized mappings as lookup
tables by cumulating the input histograms. These histograms
are modified versions of the subimages’ original histograms
in which, at each intensity level, the contrast enhancement
induced by the method is limited to a user-selectable
maximum referred to as contrast limit (cl). Once the
mappings are calculated, neighbor subimage mappings are
bilinearly interpolated to avoid boundary artifacts.
VOL. 17,
NO. 6,
JUNE 2011
Fig. 8. Computing optimal tile size for CLAHE: First, M (a) and (f) is
smoothed (b) and (g) and binarized (c) and (h). The average size of the
bounding boxes of all connected components in the binarized image is
used for tx and ty (d) and (i). The results of CLAHE on M with the
computed tile sizes (inlays are magnified 4() are presented in (e) and
(j). These images were back-projected for OITM.
Thus, CLAHE requires setting three parameters: cl, tx ,
and ty . We derive the optimal initial setting for cl from the
pilot study in Section 6. We can, however, compute directly
the optimal tile size tx ; ty , which should match the size of
the observed features and therefore depends on the specimen. CLAHE gives good results unless the difference
between feature size and tile size is extreme, in which case
excessively small tiles cause noise and excessively large tiles
cause banding. Since specimens normally differ in feature
size, we use the following steps to estimate a mean feature
size (illustrated for two examples in Fig. 8):
We first apply a smoothing operator to reduce noise in
M. Then, we binarize the result and compute the connected
components as described in [12]. The height and width of
the average bounding box over all connected compounds is
used for tx and ty . Note that, since the average feature size
throughout one specimen usually remains fairly constant,
we can compute the tile size initially and keep it fixed for
the specimen.
To enhance the color contrast while preserving the
relative sense of hue, saturation, and intensity, we apply
DS to the RGB channels of M, which includes removing the
interchannel correlation found in M.
The derivation of the transformation matrix that decorrelates the RGB values of M is explained in [3]. An
additional linear contrast-stretch is applied by defining a
tolerance parameter (0 * tol < 0:5) that saturates the upper
and lower RGB ranges (0 . . . tol and 1 ' tol::1) in the final DS
result. By modifying tol, the degree of saturation can be
adjusted. Fig. 9 illustrates an example.
Since CLAHE increases the luminance contrast and DS
increases the color contrast, we combined both techniques
as follows: We transformed the result of DS to the CIE
L*a*b* color space, composed a new image by using L* from
CLAHE and a*,b* from DS, and finally converted the
composition back to the RGB color space.
BIMBER ET AL.: CLOSED-LOOP FEEDBACK ILLUMINATION FOR OPTICAL INVERSE TONE-MAPPING IN LIGHT MICROSCOPY
Fig. 9. Decorrelation stretch for increasing color contrast: In this
example, a tolerance value of tol ¼ 0:48 was chosen to enforce high
RGB saturation. This maximizes the amplification of color contrast of the
optical image (c) after applying DS to M (a) and back-projecting the
saturated result (b).
This resulted in a contrast technique with a total of three
adjustable parameters (while tx , ty are computed): tol, cl,
and an additional luminance scale factor l, which we apply
to L* to increase and decrease the overall brightness. We
compared the effects of these parameters on visual
improvement and derived their optimal initial values in
the course of a pilot study described in Section 6.
3.4 Suppressing Highlights
Besides low contrast, another issue that is particularly
problematic for reflected illumination in optical microscopy
are highlights on the specimen resulting from high-intensity
surface- and subsurface reflections. As can be seen in the
examples shown in Figs. 10 and 11, most details below or
next to such highlights are lost under uniform illumination.
This is disadvantageous not only for direct observations
through the oculars, but also for image capturing and
analysis in digital microscopy.
Given that the white-light modulation M is known, such
highlights can be suppressed effectively. For example,
entries of M that are above a high-intensity threshold can
be set to a low-intensity magnitude. Since both values are
specimen-dependent, they are user-defined. This reduces
highlights by explicitly clipping the local illumination
Fig. 10. Highlight clipping: Pyrite cubes embedded in rock under uniform
illumination, captured at different exposures (a) and (b) and under
projected illumination that reduces bright reflections and enhances
contrast in the background (c). Moist tissue under uniform illumination
(d) and under projected illumination with suppressed highlights (e).
863
Fig. 11. Inverted illumination: River shrimp shell that reflects high light
intensities under uniform illumination (a). Separating global (G) and
direct (D) illumination using the technique described in [19] reveals
surface reflections and subsurface reflections (b) and (c). Projecting the
complementary of M (i.e., 1 ' M) balances the contrast in the final
image and reduces highlights caused by global and direct components
at the same time (d). Note that 1 ' M is equivalent to 1 ' G ' D, and
that we project only the luminance channel.
signal that causes high-intensity reflections. All entries in
M that are not clipped can still be contrast-enhanced as
explained above. Fig. 10 illustrates that this simple
technique reveals details which are otherwise obscured by
highlights under uniform illumination.
Projecting the complementary of M (i.e., 1 ' M) is yet
another simple and efficient way of balancing contrast and
consequently suppressing highlights. In contrast to highlight clipping, a suppression threshold and an attenuation
magnitude need not to be defined manually, and attenuation transitions are smooth. As in highlight clipping, an
additional contrast enhancement can be achieved by
weighting a contrast-amplified version of M with (1 ' M).
Note that neither details below surface reflections nor
details above or below subsurface reflections can be captured
in HDR images or in low exposure LDR images under a
uniform illumination, since they are superimposed by the
highlight signal. They can be revealed only by reducing the
direct illumination regionally, as shown in Fig. 11.
More powerful segmentation methods can be applied to
emphasize relevant foreground structures and to suppress
irrelevant background structures. This serves not only to
increase contrast and enhance visibility, since scattering is
reduced in the suppressed background regions, but may
also support optically augmented focus and context
visualizations. As in contrast enhancement, suppression of
highlights and background is fast with OITM because only
one image must be captured to compute an optimized
illumination pattern.
However, having to remeasure M by recording the
specimen under a uniform white illumination every time
the specimen is moved or the microscope’s focus or
magnification are changed would disrupt the microscopy
workflow. A continuous and uninterrupted handling is
essential for many applications, including microscopic
surgery. In the next section, we describe how the whitelight modulation can be estimated continuously while
illuminating the specimen with an arbitrary pattern that is
produced by one of the OITM techniques explained earlier.
This makes possible implementing a projector-camera
closed-loop feedback that supports dynamic and interactive
microscopy operations.
864
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,
Fig. 12. Estimating white-light modulation: The illumination patterns (Ii )
and the corresponding camera images (Ci ) are the basis for estimating
the specimen modulation (Miþ1 ) under pure white light, which is used to
compute the illumination pattern (Iiþ1 ) in the next frame. In this example,
the illumination pattern is always the spectrum-scaled, estimated whitelight modulation itself (Iiþ1 ¼ Miþ1 ) T p;c ð1Þ). Hence, comparing each Iiþ1
with the ground truth recording in C0 becomes possible. In the ideal case,
they are equal. The specimen was moved between frames 1 and 2.
4
CLOSED-LOOP FEEDBACK ILLUMINATION
4.1 Estimating White-Light Modulation
We estimate the white-light modulation, M, used for
illumination computations in the next frame (i þ 1) based
on an elementwise division of the camera image C captured
at the current frame (i):
Miþ1 ¼ Ci =
_
_
Ii ;
ð4Þ
where I is the corresponding illumination image that
contains the estimated illumination power transferred to
each color channel (c) of the camera using a calibrated
projector-camera transfer function T p;c :
_
Iic ¼
X $
p2fr;g;bg
% 1 X
$ %
T p;c ð0Þ;
T p;c Iip ' T p;c ð0Þ þ
3 p2fr;g;bg
ð5Þ
where p 2 fr; g; bg and c 2 fr; g; bg are the projector and
camera color channels, respectively, and I is the original
illumination pattern in the projector’s color space. T p;c is the
illumination power transfer from projector color channel p to
camera color channel c, and T p;c ð0Þ is the estimated projector
black-level in the camera’s color space. The calibration of T p;c
is explained in Section 5.2. Thus, M estimates the specimen
modulation with pure white light. We explain our prototypes that apply a three-panel LCD projector for premodulating the microscope’s halogen lamp in Section 5. This,
however, does not produce a perfectly homogeneous
spectrum. Nevertheless, we can compute the appearance of
the estimated white-light modulation by multiplying M with
the projector’s white light response in the camera’s color
space: M ) T p;c ð1Þ. Thus, a camera image captured under
projected white light should have an appearance similar to
the estimated white-light modulation scaled by the projector’s spectrum in the camera’s color space.
Fig. 12 illustrates an example of four subsequent frames.
The specimen was moved between frames 1 and 2. Despite
the misregistered illumination in this frame, the white-light
modulation was estimated correctly and the updated
illumination pattern was registered automatically in the
next frame. In this example, the spectrum-scaled estimated
white-light modulation (I ¼ M ) T p;c ð1Þ) was back-projected,
VOL. 17,
NO. 6,
JUNE 2011
Fig. 13. Temporal filtering: When estimating white-light modulation while
enhancing contrast, alternating artifacts appear where direct pixel
correspondence between projector and camera is lost due to interreflections or defocus (a) and (b). As they are contrast-enhanced
continuously, their strength increases from frame to frame (c) and (f).
Averaging two subsequent estimations results in stable images that are
free of artifacts (d) and (g). Note that (d) and (g) appear more blurred
than (c) and (f). The reason for this are the high-frequency artifacts in the
unfiltered images. These artifacts are not part of the original images
(compare (c) and (d) with (e), for example). The specimen in (g) is
regionally defocused because of the microscope’s limited depth-of-field
and not because of temporal filtering.
while M represented the estimation under pure white light.
Comparing the ground truth camera image captured under
white projected light (C0 ¼ I1 ) with the corresponding
estimations (I2 and I3 ) demonstrated that our method
computes the white-light modulation well. It became
apparent that the contrast of the estimations is slightly
lower than in the captured image under physical white light
projection (i.e., C0 ; I1 versus I2 ; I3 ). The reason for this is that
the optical images produced in back-projection exceed the
dynamic range of the camera in a single shot (C1::3 ). This
leads to a slightly underestimated contrast in M. We can
counter this problem either by using a fast high dynamic
range camera or by subsequently enhancing the contrast of
M as explained above.
4.2 Handling Overestimation
In principle, we can estimate the white-light modulation and
update the illumination with a delay of only one frame, but at
full frame rate. In practice, however, complex modulations
that include interreflections, regional defocus due to the
microscope’s shallow depth-of-field, and movements of the
specimen lead to problems that must be overcome.
If a direct pixel correspondence between projector and
camera is lost because of misregistrations caused by defocus
or scattering, our closed-loop feedback estimation together
with a contrast enhancement produces alternation artifacts,
as illustrated in Fig. 13. The reason for this is that a single
projector pixel that is defocused or scattered is captured by
multiple camera pixels. These camera pixels are, in turn,
registered with other projector pixels. Thus, the direct
relation between local irradiant illumination and corresponding local radiant reflectance that is assumed in (4) is invalid,
which leads to an overestimation of white-light modulation
values due to unknown indirect light transport. Consequently, the corresponding illumination values are updated
wrongly, which influences the modulation estimation and the
BIMBER ET AL.: CLOSED-LOOP FEEDBACK ILLUMINATION FOR OPTICAL INVERSE TONE-MAPPING IN LIGHT MICROSCOPY
865
Fig. 14. Examples of our feedback loop for different illumination patterns (I): contrast enhancement using (3) and $ ¼ 1:0 (rows (a) and (c)). Note that
in these cases the contrast of the optical images exceeds the dynamic range of the single-shot camera images (C). This leads to a lower contrast in
the estimated modulation (M ) T p;c ð1Þ). Inverted illumination (row (b)) and highlight clipping (row (d)). Note that, since the contrast in C is balanced in
these cases, the contrast of the estimated modulation approaches the contrast of the corresponding images of the specimen when captured under
uniform illumination. Reconstructed highlight regions (arrows) are clipped in the estimations.
illumination computations in the next frame, and so on. The
resulting artifacts alternate from frame to frame while their
strength increases over time (because of contrast enhancement) until they become binary and eventually cover the
entire field.
Unfortunately, we cannot separate the direct and indirect
light transport components of M in one frame to overcome
this problem. As explained in [19], multiple images must be
captured under a structured illumination to measure the
indirect component. This, however, is not feasible under an
unobtrusive closed-loop feedback illumination. In classical
control theory, solutions exist to regulate such underdamped situations. They cannot be applied in our case,
since no reference output is given.
One positive characteristic of these artifacts is that their
amplitude reverses in two subsequent frames, as can be seen
in Fig. 13. Therefore, applying temporal filtering by aver0
aging two subsequent modulations (Miþ1
¼ ðMi þ Miþ1 Þ=2)
cancels the artifacts. Note that this way we do not increase
the delay of our feedback loop by more than one frame.
Movements of the specimen in between two frames, for
instance, also lead to temporal misregistrations that can
cause an overestimation of M for regions that are illuminated
at low intensities, as is the case in highlight-clipped areas. If,
for example, darkened areas in the illumination pattern that
reduce highlights before a movement are projected onto
nonhighlight regions after a movement, we cannot provide a
robust estimation of M for these regions.
We overcome this problem as follows: If the estimated
modulation is high for an entry in M, this suggests either a
highlight at this position that must be compensated, or M
was overestimated due to the limitations explained above.
We can distinguish these two cases by evaluating in addition
the corresponding camera response. If it is below a certain
threshold, the large modulation in M is due to overestimation. This assumption is valid, since even the projector’s
black-level together with potential contributions from indirect illumination will lead to a camera response that is higher
in highlight regions than in nonhighlight regions. In this case,
the invalid entries in M are replaced by their nearest valid
entries in M. The required threshold is computed automatically and updated for each current frame using the grayscale histogram computed for all camera pixels belonging to
clipped highlight regions of the previous frame. The threshold is the intensity at the minimum that follows the first
gradient decline in the histogram. This represents the lowest
intensity reflected most by highlight regions only. Lower
intensities are reflected by nonhighlight regions that are
illuminated incorrectly, since they are misregistered briefly
after movement. Fig. 14 illustrates several feedback-loop
examples with different illumination patterns that are
computed for contrast enhancement, highlight clipping,
and inverted illumination.
866
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,
VOL. 17,
NO. 6,
JUNE 2011
Fig. 15. Prototype configurations (a) for reflected illumination and (b) for transmitted illumination. The numbers indicate the magnification factors of
the individual components. The green arrows illustrate the illumination paths.
5
PROTOTYPES
5.1 Optical Configuration
A Zeiss Axio Imager D1 served as a basic platform for all
our experiments. This is a modular upright light microscope that offers the possibility of experimenting with
different configurations (cf. Fig. 15). A Sony VPL-AW 15
three-panel LCD projector was used for light modulation.
Its original UHP lamp and its objective were replaced with
the original microscope halogen lamp and new objective
optics. The halogen lamp emits a more uniform spectrum
than the UHP lamp, and the new objective optics (a
combination of a Nikon 50 mm F1.2 digital camera objective
and a Zeiss 44 M52 ( 0;75 variable microscope camera
adapter) match the pupils of projector and microscope
imaging or illumination ports. We experimented with
linearized Point Gray Flea 2 and Point Gray Dragonfly 2
CCD cameras attached to the top camera path deflection
port. They were synchronized electronically with the
projector to ensure frame-precise exposures.
For reflected illumination, the projector was attached to
the infinite side camera path deflection port (cf. Fig. 15a). In
this case, the light rays emitted by the projector and the light
rays captured by the camera or observed through the
oculars are parallel within the microscope optics, and are
projected and imaged through the same microscope
objective. Thus, projector, camera, and oculars share the
same optical axis and focal plane. For transmitted illumination, the projector was attached to the microscope via the
illumination port at the back normally used for conventional
transmitted illumination (cf. Fig. 15b). The projector image
must be focused using the condenser optics. Both the
camera image and the image observed through the oculars
must still be focused using the microscope objective.
5.2 Calibration
Projector and camera must be calibrated to measure their
geometric and radiometric offsets. Even if both devices are
optically coaligned, different resolutions, chip/sensor sizes,
and orientations around the optical axis can cause geometric misregistration. For reflected illumination, this is
measured by projecting and capturing a Gray code on a
surface mirror placed below the microscope objective. For
transmitted illumination, the Gray code can be projected
directly and captured in the empty field plane if projector
and camera are focused at this plane.
The pixel correspondences were stored in a lookup table
for real-time image warping during runtime. The average
misregistration of a projector pixel is currently one fifth of the
size of a camera pixel. In addition, we measured the
projector-camera transfer functions for all color channels
and their combinations, which led to a total of nine transfer
functions (T p;c ) that indicate the illumination power transfer
from projector color channel p to camera color channel c. To
this end, we projected all intensities of all color channels p
(onto the surface mirror for reflected illumination or directly
at the field plane for transmitted illumination), measured the
corresponding responses in all camera channels c, and stored
them in lookup tables (T p;c ). These lookup tables were used
for real-time color correction during runtime as explained in
Section 4.1. The exact implementation details of projectorcamera calibration are beyond the scope of this discussion.
Interested readers are referred to [7].
Furthermore, brightness and color inhomogeneity in the
illumination were corrected by capturing one image (W )
under white illumination and one image (D) under dark
illumination (projector black-level) on a surface mirror.
During runtime, these images were used to apply a flatfield correction [16] to the projected illumination:
I0 ¼ m
ðI ' T '1 ðDÞÞ
;
' T '1 ðDÞÞ
ðT '1 ðW Þ
ð6Þ
where I 0 is the corrected and I is the original illumination
with mean pixel value m.
For reflected illumination and a five-fold objective
magnification, the projector pixel size on the field plane is
3:15 "m. The physical pixel size of the Point Gray Flea 2
camera is 1:1 "m on the field plane. The composite camera
pixel size (i.e., after demosaicing the Bayer pattern) is
approximately 5:18 "m. The pixel sizes for the Point Gray
Dragonfly 2 camera are 2:3 "m (physical) and approx.
7:05 "m (composite). These sizes decrease with increasing
objective magnifications. For transmitted illumination, the
pixel footprint of the projector on the field plane is 1:64 "m.
The calibration remains constant if projector and camera
share the same focal plane—no matter if the specimen is
moved, the microscope refocused, or its magnification
changed.
BIMBER ET AL.: CLOSED-LOOP FEEDBACK ILLUMINATION FOR OPTICAL INVERSE TONE-MAPPING IN LIGHT MICROSCOPY
867
TABLE 1
Specimen Influence on Parameters (Fcritic ¼ Fð0:05;2;122Þ )
uniform illuminations: one with the full possible brightness
(full) and the other one with a brightness that leads to
roughly the same adaptation luminance (adapt) as the
previously observed OITM illumination. The different specimens are shown in Fig. 16: one with low contrast for
evaluating OITM’s impact on perceived contrast enhancement, one with color structures for evaluating OITM’s impact
on perceived color contrast enhancement, and one with lowreflectance for evaluating OITM’s impact under low lighting
conditions. The task for each subject consisted of two passes
that were repeated for each of the three specimens.
Fig. 16. Visibility improvements: Each diagram shows the ordinal
ranking of the visibility of small and large structures (1 ¼ best), and the
percentage averaged over the votes of all subjects. The first choices are
indicated by an asterisk. The evaluated specimens are presented in the
first row. The direct comparison of the two contrast techniques for each
specimen is illustrated in the second row. The comparison between
CLAHE and the two uniform lighting conditions is shown in the third row,
and the comparison between CLAHE+DS and the two uniform lighting
conditions is shown in the fourth row.
6
PILOT STUDY
6.1 General Procedure
Suppressing highlights or increasing the S/N ratio using a
controlled illumination leads to noticeable improvements.
Thus far, however, it is still unclear if an enhancement of the
visibility of spatial structures with OITM, as explained in
Section 3.3, is in fact beneficial in light microscopy. This is
hard to predict, since we trade the higher contrast of a
spatially modulated illumination for the increased brightness
of a uniform illumination. An illumination pattern computed
with the techniques described above always results in a lower
adaptation luminance than a full uniform illumination.
To prove that OITM indeed leads to visibility improvements and to investigate the optimal initial parameterization of the contrast technique explained in Section 3.3, we
conducted a pilot study with a total of 41 subjects (30 male,
11 female, 21-44 years of age with an average of 27 years).
This study was carried out with the prototype configurations for reflected illumination.
After adapting to the brightness of the microscope, each
subject was asked to observe three different specimens under
four different lighting conditions. For two of these lighting
conditions, we used OITM with the following contrast
techniques applied to M: CLAHE on the luminance channel
only (no colors projected), and CLAHE+DS with an additional decorrelation of the chrominance, as explained in
Section 3.3. The remaining two lighting conditions were two
6.2 Contrast Parameter
In the first pass, each subject was asked to adjust the contrast
parameter for a specimen until the visibility of both small
and large structures was optimal. This was repeated for each
of the two contrast techniques. Thus, the cl and l parameters
were adjusted first for CLAHE, and then tol and l were
adjusted for CLAHE+DS (while keeping cl from the previous
adjustment). While looking through the oculars, a mouse
was used to adjust the parameters in pairs, and its movements were mapped to x, y changes. Note that it was always
possible to readjust the original luminance of the illumination image by modifying the parameter l. For a given value of
l ('255 * l * 255), the luminance was remapped to a range
of ½l; 255% if l > 0, or to a range of ½0; 255 þ l% if l * 0.
After a Lilliefors test to validate that all our measurements were normally distributed, we conducted one-way
ANOVA tests with the specimen as factor and the selected
parameters as values to find out whether the observed
specimen strongly influenced the parameter selection. We
chose an alpha level of 0.05. The results in Table 1 show that
only the adjusted luminance is influenced by the type of
specimen. The contrast parameters cl and tol are specimenindependent. To determine whether the contrast parameters
are influenced by the adjusted luminance level, we performed unbalanced one-way ANOVA tests with four
luminance intervals (½'255; '25%, ½'25; 25%, ½25; 75%, and
½75; 255%) as factor and the contrast parameters as values.
Again, an alpha level of 0.05 was chosen. The results in
Table 2 reveal that cl and tol are also independent of the
adjusted luminance level. Therefore, we chose the average of
the medians from each sample (i.e., cl ¼ 0:26 and tol ¼ 0:25)
as initial parameters for the contrast technique explained in
Section 3.3. They can be fine-tuned any time during runtime.
The luminance value l clearly depends on the reflectance of
the specimen. For l, we also chose the average of the medians
from each sample for both contrast techniques (i.e., l ¼ 42:0
for CLAHE and l ¼ 49:33 for CLAHE+DS) as initial values.
868
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,
TABLE 2
Influence of Luminance on Parameters (Fcritic ¼ Fð0:05;2;40Þ for tol
(Low Reflectance), Fcritic ¼ Fð0:05;3;40Þ for All Other Parameters)
7
VOL. 17,
NO. 6,
JUNE 2011
CONCLUSION
However, we expect that, depending on the specimen, l may
require significant readjustment.
7.1 Discussion
For applications that require direct observation of the
optical image through the oculars, OITM holds the potential
to overcome the contrast constraints of human visual
perception. For digital microscopy applications, the speed
of OITM is probably one essential advantage over more
advanced multiexposure scanning and imaging techniques.
In our current implementation, the entire closed-loop
feedback computations together with contrast enhancement
via nonlinear tone-mapping (3) and highlight clipping
requires a total of 35:6 ms on a GeForce GTX 285. CLAHE
and DS have not yet been ported to the GPU. They require
18 and 144 ms, respectively, on an Intel Core2Duo 3.0 GHz.
All computations scale linearly (OðNÞ) with the resolution
(N) of the projector-camera system.
6.3 Visibility Improvements
After optimal adjustment of the parameters in the first pass,
each subject was asked to compare the visibility of all four
lighting conditions in the second pass. Thus, the subjects
were initially able to switch freely between CLAHE and
the two uniform illuminations (adapt was matched to the
adaptation luminance produced by CLAHE). They were
asked to rank the visibility of small and large structures for
each of these illumination cases (with ordinal ranks from 1
to 3; 1 ¼ best). This was repeated for CLAHE+DS. Finally,
the subjects could switch between CLAHE and CLAHE+DS
for a direct ranking. Fig. 16 illustrates the results averaged
over all subjects.
Fig. 16 shows that OITM always outperforms a uniform
illumination—for all evaluated specimens and with respect
to the visibility of small and large structures. While a higher
adaptation luminance is generally better for uniform lighting conditions, a contrast-enhanced illumination improves
visibility, even if its overall brightness is lower than that of a
uniform illumination.
An enhancement of color contrast in addition to an
enhancement of luminance contrast did not lead to
significant perceptual improvements in our experiments,
not even for specimens with colored structures. The reason
for this might be that luminance contrast is the dominant
factor in detecting structures. Since it did not change
between CLAHE and CLAHE+DS, the detectability of
structures remained unaffected. For small structures, a full
uniform illumination outperformed CLAHE+DS. We think
that the reason for this might be chromostereopsis: Colors
with different wavelengths are refracted and focused
differently by the eye, which leads to visual defocus when
strongly contrasting colors appear side by side. This is
particularly problematic when resolving small features.
However, we believe that an enhancement in color
contrast can improve the visual exploration of complex
colored structures, since it provides a clearer segmentation between components. This was not evaluated in the
course of our pilot study, because visual exploration tasks
are usually more application-dependent and therefore less
generalizable.
7.2 Limitations
Several limitations can be attributed to constraints of the
components used: The dynamic range of the camera must
be high for capturing and estimating the white-light
modulation of the specimen with adequate contrast. A high
projector contrast and less scattering within the microscope
optics will lead to additional improvements. Due to stray
light remaining within the microscope optics, this contrast
is as low as 105:1 in our current prototype.
When measured at the microscope’s intermediate image
plane, the objective resolving power is 2:24 "m and the
projector’s pixel footprint is 31:5 "m for reflected illumination and 16:4 "m for transmitted illumination. Assuming a
visual accuracy of the eye of 7:25 "m on the intermediate
image plane, the size of the projector pixels is currently above
the resolving power of the human eye. This can be solved by
using a higher-resolution SLM or by optimizing the magnifications of oculars and objective. However, diffraction sets
the limit at approximately 1,000-fold magnification for the
visible range in light microscopy. As a result, resolving
features smaller than 0:2 "m is not possible. This is also the
lower limit of the SLM’s pixel footprint in the visible range.
For transmitted illumination, the geometric and radiometric registration and the defocus of the projector and the
camera are neither identical nor constant. The reason for this
is that the camera image is captured through the microscope
objective, while the projector image is displayed through the
condenser optics. This means that the focus of the projector
and the camera must be adjusted individually to the same
field plane (i.e., the same plane of the specimen). Projectorcamera calibration, as outlined above, must be carried out
every time the microscope’s parameters (focus and magnification) or the specimens are changed. A precise mechanical
adjustment of the condenser with respect to the chosen
magnification and focus settings would solve this problem.
This, however, is currently not supported by our prototype.
The biggest limitation of our current prototype is that the
peak luminance of the 100 W halogen illumination is
reduced to 14 percent by the LCD panels and the additional
optics of the projector. To support a better contrast
sensitivity of high-frequency samples, we would need to
employ a higher adaptation luminance. In addition, the
exposure time of the camera must be high for samples with
low reflection properties. Currently, this results in relatively
BIMBER ET AL.: CLOSED-LOOP FEEDBACK ILLUMINATION FOR OPTICAL INVERSE TONE-MAPPING IN LIGHT MICROSCOPY
slow update rates of 3-5 fps for our feedback loop, since
camera exposures of up to 250 ms take a large share of the
processing time. All of these problems can be addressed by
optimizing the light throughput. Using a reflective instead
of a transmissive SLM, such as a DMD, will certainly
improve this situation significantly.
While the above drawbacks can be overcome by
improving the microscope optics, the specimen itself sets
general limitations. OITM becomes less efficient, the more
the specimen scatters light.
7.3 Future Work and Potential Applications
We believe that an extension of real-time optical inverse tonemapping to light-field microscopy [13], [14] can hold
additional potential. Capturing, processing, and projecting
a 4D light field instead of one single 2D slice of it (as done in
our case) will open up further possibilities through the
beneficial effect of a higher angular resolution in illumination
and imaging. This may allow better results to be achieved for
specimens with an excessive amount of scattering. The
spatial resolution of light-field microscopy, however, is
currently too low for most applications in light microscopy.
Thus far, we profit from the native resolving power of
conventional microscopes, and our OITM implementation is
limited only by the resolution of the camera and the SLM.
A possible alternative to our estimation of the specimen’s
modulation is to measure it continuously in wavelengths that
are not used for illumination. The long wavelength of infrared
(IR) light reduces the resolving power of a microscope and
increases volume scattering significantly, and the short
wavelength of ultraviolet (UV) light can destroy living cells
and is therefore usually avoided. Moreover, color cannot be
detected and processed with IR and UV light. Wavelength
multiplexing in the visible range might be a better alternative
that must be explored in future.
Further studies are required to investigate additional
contrast techniques for optimal OITM illumination and
their quantitative impact on contrast perception in more
controlled user experiments. These techniques, however,
might be specific to individual applications and specimens.
In general, we envisage benefits from OITM in a variety
of fields that apply light microscopy and similar techniques.
The contrast of tissue, for example, could be optimized in
real time for microscopic surgery and endoscopy. In
addition, specular highlights caused by liquids could be
suppressed precisely. This may also be beneficial to other
samples that cause specular highlights, such as crystals in
mineralogy and conductor elements in electrical circuit
inspections. In some cases, low-contrast specimens may no
longer require thin slicing or chemical processing (e.g.,
staining or decalcification as done for bones) to enhance
contrast using a transmitted illumination technique, since
OITM supports reflected illumination. For material science,
OITM could be of advantage for real-time applications that
process and observe low-contrast polymers, such as PVC.
REFERENCES
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]
[18]
[19]
[20]
[21]
[22]
[23]
ACKNOWLEDGMENTS
The authors thank Anton Moffat and Enrico Geissler of Carl
Zeiss AG Jena, and Sebastian Thiele and Ferry Häntsch of
the Bauhaus-University Weimar for their support.
869
[24]
A.A. Adeyemi, N. Barakat, and T.E. Darcie, “Applications of
Digital Micro-Mirror Devices to Digital Optical Microscope
Dynamic Range Enhancement,” Optics Express, vol. 17, no. 3,
pp. 1831-1843, 2009.
R.E. Alley, “Decorrelation Stretching as an Aid to Image
Interpretation,” Int’l J. Remote Sensing, vol. 8, pp. 1253-1254,
1987.
R.E. Alley, “Algorithm Theoretical Basis Document for Decorrelation Stretch,” technical report, Ver. 2.2, Jet Propulsion Laboratory,
Aug. 1996.
T. Amano and H. Kato, “Real World Dynamic Appearance
Enhancement with Procam Feedback,” Proc. Int’l Workshop
Projector-Camera Systems (Poster), 2008.
F. Banterle, P. Ledda, K. Debattista, and A. Chalmers, “Inverse
Tone Mapping,” Proc. Conf. Computer Graphics and Interactive
Techniques in Australasia and Southeast Asia, pp. 349-356, 2006.
O. Bimber and D. Iwai, “Superimposing Dynamic Range,” ACM
Trans. Graphics, vol. 27, no. 5, pp. 1-8, 2008.
M. Brown, A. Majumder, and R. Yang, “Camera-Based Calibration
Techniques for Seamless Multi-Projector Displays,” IEEE Trans.
Visualization and Computer Graphics, vol. 11, no. 2, pp. 193-206,
2005.
P.E. Debevec and J. Malik, “Recovering High Dynamic Range
Radiance Maps from Photographs,” Proc. ACM SIGGRAPH,
pp. 369-378, 1997.
A.F. Desimone and B. Crary, “Spatial Light Modulator Apparatus,” Int’l Patent WO 03/040798 A1, May 2003.
K. Fujii, M.D. Grossberg, and S.K. Nayar, “A Projector-Camera
System with Real-Time Photometric Adaptation for Dynamic
Environments,” Proc. IEEE CS Conf. Computer Vision and Pattern
Recognition, vol. 1, pp. 814-821, 2005.
C. Gao, N. Ahuja, and H. Hua, “Active Aperture Control and
Sensor Modulation for Flexible Imaging,” Proc. IEEE Int’l Conf.
Computer Vision and Pattern Recognition, pp. 1-8, 2007.
R.M. Haralick and L.G. Shapiro, Computer and Robot Vision, vol. 1,
pp. 28-48, Addison-Wesley, 1992.
M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light
Field Microscopy,” ACM Trans. Graphics, vol. 25, no. 3, pp. 924934, 2006.
M. Levoy, Z. Zhang, and I. McDowall, “Recording and Controlling the 4D Light Field in a Microscope,” J. Microscopy, vol. 235,
pp. 144-162, 2009.
H. Mannami, R. Sagawa, Y. Mukaigawa, T. Echigo, and Y. Yagi,
“High Dynamic Range Camera Using Reflective Liquid Crystal,”
Proc. IEEE Int’l Conf. Computer Vision, pp. 1-8, 2007.
D.B. Murphy, Fundamentals of Light Microscopy and Electronic
Imaging. Wiley-Liss, 2001.
S.K. Nayar and V. Branzoi, “Adaptive Dynamic Range Imaging:
Optical Control of Pixel Exposures Over Space and Time,” Proc.
IEEE Int’l Conf. Computer Vision, pp. 1168-1175, 2003.
S.K. Nayar, V. Branzoi, and T.E. Boult, “Programmable Imaging
Using a Digital Micromirror Array,” Proc. IEEE CS Conf. Computer
Vision and Pattern Recognition, pp. I-436-I-443, 2004.
S.K. Nayar, G. Krishnan, M.D. Grossberg, and R. Raskar, “Fast
Separation of Direct and Global Components of a Scene Using
High Frequency Illumination,” ACM Trans. Graphics, vol. 25, no. 3,
pp. 935-944, 2006.
M.A.A. Neil, T. Wilson, and R. Juskaitis, “A Wavefront
Generator for Complex Pupil Function Synthesis and Point
Spread Function Engineering,” J. Microscopy, vol. 197, no. 3,
pp. 219-223, 2000.
M.V. Newberry, “Signal-to-Noise Considerations for Sky-Subtracted CCD Data,” Publications of the Astronomical Soc. of the
Pacific, vol. 103, pp. 122-130, 1991.
A.Y.M. Ng, C.W. See, and M.G. Somekh, “Quantitative Optical
Microscope with Enhanced Resolution Using a Pixelated Liquid
Crystal Spatial Light Modulator,” J. Microscopy, vol. 214, no. 3,
pp. 334-340, 2004.
N. Otsu, “A Threshold Selection Method from Gray-Level
Histograms,” IEEE Trans. Systems, Man, and Cybernetics, vol. 9,
no. 1, pp. 62-66, Jan. 1979.
H. Park, M.-H. Lee, B.-K. Seo, H.-C. Shin, and J.-I. Park,
“Radiometrically-Compensated Projection onto Non-Lambertian
Surface Using Multiple Overlapping Projectors,” Proc. Pacific-Rim
Symp. Image and Video Technology, pp. 534-544, 2006.
870
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,
[25] S.M. Pizer, E.P. Amburn, J.D. Austin, R. Cromartie, A. Geselowitz,
T. Greer, B.M. ter Haar Romeny, J.B. Zimmerman, and K.
Zuiderveld, “Adaptive Histogram Equalization and Its Variations,” Computer Vision, Graphics and Image Processing, vol. 39,
pp. 355-368, 1987.
[26] E.C. Samson and C.M. Blanca, “Dynamic Contrast Enhancement
in Widefield Microscopy using Projector-Generated Illumination
Patterns,” New J. Physics, vol. 9, no. 10, pp. 363-377, 2007.
[27] H. Seetzen, W. Heidrich, W. Stuerzlinger, G. Ward, L. Whitehead,
M. Trentacoste, A. Ghosh, and A. Vorozcovs, “High Dynamic
Range Display Systems,” Proc. ACM SIGGRAPH, pp. 760-768,
2004.
[28] P.J. Verveer, Q.S. Hanley, P.W. Verbeek, L.J. van Vliet, and T.M.
Jovin, “Theory of Confocal Fluorescence Imaging in the Programable Array Microscope,” J. Microscopy, vol. 189, no. 3, pp. 192-198,
1998.
[29] G. Wetzstein, D. Luebke, and W. Heidrich, “Optical Image
Processing Using Light Modulation Displays,” to be published
in Computer Graphics Forum.
[30] K. Zuiderveld, “Contrast Limited Adaptive Histogram Equalization,” Graphics Gems IV, P.S. Heckbert, ed., Chapter VIII.5, pp. 474485, Academic Press, 1994.
Oliver Bimber has led the Augmented Reality
Group at the Bauhaus-University Weimar until
March 2010. Since October 2009, he is heading
the Institute of Computer Graphics at the
Johannes Kepler University Linz, Austria. He is
a member of the IEEE Computer Society. For
more information, see www.jku.at/cg.
Daniel Klöck received the Dipl-Inf degree from
Brandenburg Technical University Cottbus in
2009. He has been involved in research projects
in the fields of image processing, AI, growing
artificial societies, AAL, and computational geometry among others. Currently, he is working in
the gaming industry, employed at qforge GmbH.
VOL. 17,
NO. 6,
JUNE 2011
Toshiyuki Amano received the PhD degree in
engineering from Osaka University, Japan. He is
an assistant professor in the Graduate School of
Information Science at Nara Institute of Science
and Technology, Japan. His research interests
include projector-camera system, pattern recognition, and image processing. He is a member of
the IEEE and the IEEE Computer Society.
Anselm Grundhöfer received the diploma
(MSc) degree in media system sciences from
Bauhaus-University Weimar in 2006. Currently,
he is a PhD candidate at Bauhaus-University
Weimar. His research interests include projector-camera systems, real-time image processing, computer vision, and display technologies.
Daniel Kurz received the MSc degree in media
system sciences from the Bauhaus-University
Weimar in early 2010. Currently, he is working in
the Research and Development Group of
metaio, Munich. His research interests include
augmented reality, computer vision, and computer graphics.
. For more information on this or any other computing topic,
please visit our Digital Library at www.computer.org/publications/dlib.
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement