Light-Field Microscopy with a Consumer Light-Field - Manao

Light-Field Microscopy with a Consumer Light-Field - Manao
Light-Field Microscopy with a Consumer Light-Field Camera
Lois Mignard-Debise
INRIA, LP2N
Bordeaux, France
Ivo Ihrke
INRIA, LP2N
Bordeaux, France
http://manao.inria.fr/perso/˜mignard/
Abstract
presents a hurdle for experimenting with the technology. In
this article, we demonstrate the use of the Lytro camera,
an inexpensive consumer-grade light-field sensor, for microscopic work. We achieve an inexpensive and accessible
means of exploring light-field microscopy with good quality, albeit at a reduced optical magnification.
As we show, the major problem in combining the Lytro
and additional magnification optics (in addition to f-number
matching), is the loss of information due to spatial vignetting. Our main finding is the possibility of using the
Lytro in what we term an inverse regime: in this setting the
camera picks up a virtual object that is located far behind its
imaging optics. To our knowledge, this is the first time that
such a light-field imaging mode is described.
We investigate two different setups based on this inverse
regime that do not suffer from spatial vignetting: 1) Our
first option enables the use of the Lytro camera in conjunction with an unmodified microscope by designing an optical
matching system. 2) The second option uses the Lytro behind a standard SLR lens in a macrography configuration to
achieve macro light-field photography.
The paper is organized as follows. In Section 3, we study
the compatibility of both optical systems involved in the
imaging process: the light-field camera and the microscope.
We discuss the implications of the combination of both systems in terms of both spatial and angular resolution. We
then explore the Lytro main optical system and show how to
adapt it to the microscopic imaging context, Sects. 5 and 6.
Finally, we evaluate and compare the different solutions and
present application scenarios, Sect. 7.
We explore the use of inexpensive consumer lightfield camera technology for the purpose of light-field microscopy. Our experiments are based on the Lytro (first generation) camera. Unfortunately, the optical systems of the
Lytro and those of microscopes are not compatible, leading to a loss of light-field information due to angular and
spatial vignetting when directly recording microscopic pictures. We therefore consider an adaptation of the Lytro optical system.
We demonstrate that using the Lytro directly as an ocular replacement, leads to unacceptable spatial vignetting.
However, we also found a setting that allows the use of the
Lytro camera in a virtual imaging mode which prevents the
information loss to a large extent. We analyze the new virtual imaging mode and use it in two different setups for implementing light-field microscopy using a Lytro camera. As
a practical result, we show that the camera can be used for
low magnification work, as e.g. common in quality control,
surface characterization, etc. We achieve a maximum spatial resolution of about 6.25µm, albeit at a limited SNR for
the side views.
1. Introduction
Light-field imaging is a new tool in the field of digital
photography. The increasing interest is shown by the recent
development of several hardware systems on the consumer
market (Lytro, Raytrix, Picam), and applications in the research domain (stereo-vision, panoramic imaging, refocusing). The commercial systems are reliable, functional and
inexpensive. However, they are designed for the imaging
of macroscopic objects. In this article, we explore the use
of commercial light-field cameras for microscopic imaging
applications.
The light-field microscope has been introduced and improved by Levoy et al. [9, 10, 2]. While its conceptual details are well understood, its practical implementation relies on the fabrication of a custom micro-lens array, which
2. Related Work
Light-field imaging requires the acquisition of a large
number of viewpoints of a single scene. Two types of approaches exist, either using multiple sensors or a single
sensor in conjunction with temporal or spatial multiplexing
schemes.
Taking a picture with a conventional camera is similar to
a 2D slicing of the 4D light-field. Repeating this operation
with a planar array of cameras offers sufficient data to esti1
Photographs of specimens
Lytro
Camera
Second
Lens
First
Lens
Microscope
Objective
Object:
Resolution
Target
Figure 1. Left: Microscope setup with the Lytro camera on top of two additional SLR lenses. Images of several samples have been taken
under different illumination conditions. Magnification is 3.0. Top and bottom left: brushed steel from scissors blade with lighting from the
right and bottom. Top middle left: Scratched surface of a piece of metal. Bottom middle left: Plastic surface with highly retro-reflective
properties. The material is made of micro bubbles of transparent plastic that are invisible to the naked eye. Top middle right: Fabric with
a hexagonal structure. The lighting is coming from the side and casts shadows and strong highlights on the three-dimensional structure
of the fabric. Bottom middle right: Highly retro-reflective material from security reflective tape. Top right: Tonsil tissue with bright field
illumination. Bottom right: Pins of a electronic component on a circuit board.
mate the light-field function. Calibration, synchronization,
as well as the available bandwidth of the camera hardware
are the major determining features of this approach. More
details can be found, e.g., in [25, 23]. The strongest limitation of this approach for microscopic applications is the
large size of the corresponding setups.
An alternative method for light-field capture is to multiplex the different views onto a single sensor.
Temporal multiplexing is based on taking several pictures
over time after moving the camera around static scenes. The
movements can be a translation or rotation of the camera.
Alternatively, mirrors [5, 19] can be moved to generate additional virtual viewpoints. Another alternative implementation are dynamic apertures [11].
Spatial multiplexing allows to record dynamic scenes.
Parallax barriers and integral imaging [12] are historically
the first approaches to spatially multiplex the acquisition
of a light-field, trading spatial resolution for angular resolution. A modern elaboration of this approach where the
sensor and a micro-lens array are combined to form an incamera light-field imaging system is the Hand-Held Plenoptic Camera [17]. Alternatively, sensor masks [22, 21] or a
light pipe [15] can be arranged such that in-camera lightfields can be recorded. Other methods use external arrays
of mirrors instead of lenses [7], or external lens arrays [4].
Microscopy is a vast subject and many different illumination and observation schemes have been developed in the
past. A general overview is given in [16]; a comprehensive review of microscopy techniques, including light-field
microscopy, for the neuro-sciences can be found in [24].
Light-field microscopy was introduced by Levoy et al. [9]
and later augmented with light-field illumination [10]. Recently, addressing the large spatial resolution loss implicit
in LFM, the group has shown that computational superresolution can be achieved outside the focal plane of the
microscope [2, 3]. Another super-resolution scheme is combining a Shack-Hartmann wavefront sensor and a standard
2D image to compute a high-resolution microscopic lightfield [14]. LFM has been applied to polarization studies of
mineral samples [18] and initial studies for extracting depth
maps from the light-field data have been performed in microscopic contexts [8, 20]. Most of the work today uses the
same optical configuration that was introduced in the original implementation [9]. With this article, we aim at providing an inexpensive means of experimenting with LFM.
3. Background & Problem Statement
3.1. Light-field Microscopy
The main function of an optical microscope is to magnify small objects so that they can be observed with the
naked eye or a camera sensor. Light-field capabilities such
as changing the viewpoint, focusing after taking the picture, and achieving 3D reconstruction of microscopic sam-
ples rely on the number of view points that can be measured
from a scene. This number is directly linked to the objectside numerical aperture NAo of the imaging system that is
used as an image-forming system in front of the micro-lens
array of the light-field sensor and the micro-lens f-number.
The object-side numerical aperture is defined as
NAo = n sin(α),
(1)
where n is the index of the material in object space (usually air, i.e. n = 1). The numerical aperture quantifies the
extent of the cone of rays originating at an object point and
being permitted into the optical system (see Fig. 2). Microscope objectives usually have a high NAo , because it is
directly linked to better optical resolution and a shallower
depth of field. A high NAo is also important for light-field
microscopy as the base-line of the light-field views is directly linked to it.
Details on how to design a light-field microscope can be
found in [9]. The most important aspect is that the f-number
of the micro-lens array matches the f-number of the microscope objective. The f-number F of any optical system is
defined as the ratio of its focal length f over the diameter D
of its entrance pupil (see Fig.2)
f
(2)
F = .
D
For a microscope of magnification Mmicroscope and numerical aperture NAo , a more appropriate equation taking the
finite image distance into account can be derived [1] from
Eq. 2:
Mmicroscope
Fmicroscope =
.
(3)
2NAo
The majority of microscope objectives has an f-number between 15 and 40. In our experiments, we use a 10× objective with an f-number of F = 20.
Figure 2. Properties of an optical system. An optical system is
defined by its focal length f , its principal planes H and H 0 , and
the diameter D of its entrance pupil. Two conjugate planes define
a unique magnification M , the ratio of the image size y 0 over the
object size y. All optical equations can be found in [1].
3.2. Lytro Features
The Lytro camera is made of an optical system that is
forming an image in the plane of a micro-lens array that
is, in turn, redirecting the light rays to a sensor. It has
a 3280 × 3280 pixel CMOS sensor with 12-bit A/D and
1.4µm × 1.4µm pixels [6]. Each micro-lens has a diameter of 14µm which is equivalent to 10 pixels. The microlenses are packed on a hexagonal lattice (see Fig. 3 (left)).
The effective spatial resolution is therefore 328×328 pixels
whereas the angular sampling rate is 10 × 10 values. The
Lytro’s main objective lens has a fixed f-number of F = 2
and features an 8× optical zoom. We explore its potential
as an imaging parameter in Sect. 5. Another important feature of the objective lens is that it can focus from 0mm to
infinity.
Figure 3. Left: Image of the micro-lens array taken with a microscope with top illumination. We can see the hexagonal structure of
the micro-lens array. The bright dot in the middle of each hexagon
is an image of the light source reflected by the surface of the microlens. Right: The Lytro camera without its optics.
3.3. F-number Mismatch
A prerequisite for non-vignetted imaging (see also
Sect. 5) is that the f-number of the micro-lens array and that
of the microscope objective match. This is the solution employed in conventional light-field microscopy [9, 18, 10]. A
custom F = 20 micro-lens array (MLA) is typically employed as it is compatible with a large number of existing
microscope objectives. However, this F = 20 MLA is not
readily available and has to be custom manufactured.
Since the Lytro is designed for macroscopic imaging, its
f-number (F = 2) is not adapted to the microscopic situation. If we were to use the Lytro micro-lens array as is,
the angular sampling in one direction would be divided by
the ratio of the f-number of the two systems and only one
pixel would be lit under each micro-lens (instead of approximately one hundred) (see Fig. 5 (bottom)). This would remove any interest for light-field purposes since only a single view would be recorded and 99% of the sensor would
remain unused, see Sect. 5 for examples.
Therefore, in order to successfully use the Lytro camera
for microscopic imaging, the f-numbers of the two optical
systems need to be adapted. There is, however, a tradeoff. The optical invariant, a fundamental law of optics (see
e.g. [1]), states that for two conjugate planes, the product
between the sine of the angle at which light rays reach a
conjugate plane (α and α0 ) and the size of the object in this
plane (y and y 0 ) is equal at both planes.
y n sin(α) = y 0 n0 sin(α0 ),
4. Sensor Coverage
(4)
n and n0 are the optical index of the media on both sides of
the optical system. For air, the index is equal to 1.
We therefore opt for an optical demagnification scheme,
decreasing y 0 , to increase the angular size of the cone of
light rays α0 that is incident on the Lytro’s light-field sensor.
Theoretically, we need to divide the microscope objective’s
f-number by 10 to reach the same f-number as the Lytro
camera. An immediate consequence from Eq. 3 is that the
combination of all optical elements must therefore have a
magnification divided by 10, i.e. we are aiming to convert
the system to unit-magnification. Due to the small size of
the Lytro’s micro-lenses, the optical resolution of the system is still satisfactory, even at this low magnification (see
Sect. 7).
The magnification of the combined system Mf inal can
be written as the product of the magnifications of each individual system:
Mf inal = Mmicroscope Mlens1 ... MlensN MLytro ,
Vignetting: When using two or more optical systems in
conjunction, some light rays are lost because the pupils of
the different systems do not match each other. This effect
is called vignetting. Generally, there are two types of vignetting: spatial and angular vignetting.
Spatial vignetting directly translates into a loss of field
of view, which may reduce the image size at the sensor (see
Fig. 5 (top)). Angular vignetting (see Fig. 5 (bottom)) occurs when the cone of rays permitted through one of the
systems is smaller than for the other system, e.g. due to a
stop positioned inside the system. Angular vignetting is not
an issue in a standard camera : it only affects the exposure
and is directly linked to the depth of field of the camera. In
a light-field camera, however, it is crucial to minimize angular vignetting in order to prevent the loss of directional
light-field information.
(5)
where Mlensi , i=1..N indicates the magnification of N
to-be-designed intermediate lens systems. The microscope
objective has a fixed magnification of Mmicroscope = 10,
whereas the lowest magnification setting of the Lytro has a
value of MLytro = 0.5. The resulting Mf inal = 5 without additional optical components (N = 0) is too large to
prevent angular information loss.
We explore two different options (see Fig. 4) to implement the adapted system. The first option (see Sect. 5) is
to demagnify the image of the microscope with additional
lenses (N = 2). This solution lets us use the microscope
and the Lytro camera unmodified. The second option is
to remove the microscope, replacing it by an SLR lens in
macro-imaging mode. Here, we compare a setup with and
without the Lytro optics (see Sect. 6).
Figure 5. Top: Spatial vignetting occurs when light rays from the
object (in red) do not pass through the second lens. For nonvignetted imaging, light rays (in blue) converging to a point at the
sensor edge should include all the red light rays emerging from
the object. Bottom: Representation of the sampling of a cone of
light rays by a F = 2 micro-lens. Green rays symbolize the angular cone that can be acquired by the micro-lens, while blue rays
emerge from an F = 20 optical system. As can be seen from the
figure, angular vignetting prevents an effective sensor utilization.
We propose to measure the vignetting in terms of its
adaptation to the recording light-field sensor. An ideal optical system that is adapted to a particular sensor would fully
cover all its sensor elements. Since the raw pixel resolution
is divided into spatial and angular parts, the sensor coverage
csensor can be approximately expressed as
Figure 4. The diagram shows our two proposed solutions to
achieve low magnification. The first option (first row) keeps the
camera and the microscope intact. The second option (second row)
replaces the microscope with an SLR lens.
csensor [%] = cspatial [%] × cangular [%],
(6)
where cspatial is the spatial coverage of the sensor in percent, and cangular is the angular coverage of one micro-lens
sub-image, also in percent. In our experimental validation,
we measure the spatial coverage in the center view and the
angular coverage in the center lenslet. This choice is motivated by the simpler estimation of the relevant coverage areas as compared to using the side views/edges of the field.
The measure can be considered to be an approximation of
the upper bound of the system space-bandwidth product, i.e.
the optical information capacity of the system [13].
Object :
LCD Display
Objective
Lytro camera in
inverse regime
Objective
Image Plane
Lytro
Object Plane
4.1. Unmodified Use of the Lytro’s Main Optics
The main optics of the Lytro camera, i.e. the optics
without the micro-lens array, is designed to avoid angular
vignetting when imaging onto the micro-lens array, i.e. the
micro-lens array and the main optics have been designed
with the same f-number of F = 2. We have observed
that the main optical system can be used in two different
ways with a microscope. These two imaging regimes can
be used differently in designing an optical matching system.
Regular regime: The Lytro camera can image a plane
as close as the first surface of its optics for a zoom level of
1×. This minimal focus distance increases with the zoom
level. In order to use the Lytro with the microscope, it has
to be positioned such that the near focus of the camera
is placed at the image plane of the microscope objective.
Since the image size y 0 of the microscope objective is
rather large (typically around 50mm × 50mm) whereas the
Lytro’s entrance pupil is only ≈ 20mm in diameter, spatial
vignetting incurs a loss of sensor coverage of up to 94% as
shown in Figure 7 (left). The angular vignetting is stronger
with only 16% angular coverage.
Inverse regime: We discovered that, in a specific configuration where the camera is set to focus to the closest
possible plane for a zoom level of 1×, the camera can enter
into a virtual object regime. The camera is then able to image an object plane that is located behind its first lens, i.e. in
the direction of the sensor (see Fig. 6). This configuration
enables the positioning of the camera close to the microscope objective and therefore reduces the spatial vignetting
since a larger number of rays can be captured by the lens
surface (see Fig. 7 (right). This mode of operation inverts
the image.
For both imaging regimes described above, the magnification MLytro (≈ 0.5) is not low enough for achieving a
good angular coverage. While the spatial vignetting problem can be successfully addressed with the inverse regime,
the angular vignetting can only be dealt with by using a low
magnification optical system. We therefore investigate the
different options.
5. The Lytro Microscope
Our first option to achieve the matching of the f-numbers
discussed in Section 3.3, consists in designing an optical de-
Figure 6. Top: Setup for the inverse regime configuration. Bottom:
Distance of Lytro focus plane to its front lens with the variation of
zoom. An abrupt change of position of the zoom lens group occurs
for a zoom level of 3.7×, enabling to switch between the regular
and inverse regime.
unmodified Lytro main optics
regular regime
inverse regime
Figure 7. Direct imaging through a 10× microscope objective with
an unmodified Lytro. The object is a blue LCD display. It consists of a square black grid that is separating the different pixels
of 0.5mm × 0.5mm size. We hypothesize the white dots to be
bubbles inside the liquid crystal. The large images show the equivalent camera image computed from the light-field, while the small
images on the bottom right show a close-up on the micro-lens images of the raw sensor data for different zoom-levels. Spatial and
angular vignetting are easily observed in the equivalent camera
and micro-lens images, respectively. The zoom level setting is 1×
(regular regime) on the left and 5× (inverse regime) on the right.
The spatial vignetting is strong in the regular regime (6% of spatial coverage), while it is greatly improved in the inverse regime
(100% of spatial coverage). Angular coverage is similar in both
cases (cangular ≈ 25%).
magnification system (placed between the microscope objective and the Lytro) that increases the angular extent of
the light (see Eq. 4). This solution keeps desirable properties like the large numerical aperture of the microscope and
its fixed working distance, while at the same time, the Lytro
camera can remain unmodified. The major task is to find a
good trade-off between the vignetting and the magnification
of the resulting light-field microscope.
Our best solution along this direction employs two
lenses. This configuration serves two goals: 1) to decrease
the magnification successively, simplifying the task of each
individual lens, and 2) to move the image behind the Lytro
camera so that it can be used in the inverse regime which
offers a better spatial coverage cspatial . The ray-diagram in
Fig. 8 illustrates the two-lens setting: an intermediate image
that is slightly demagnified is created in front of the microscope. Then, the second lens creates a further demagnified
image behind the Lytro camera. The Lytro camera is operated in its inverse imaging regime in order to pick up this
virtual image. We determine the positions and focal lengths
of the additional lenses using the following equation derived
from the thin lens formula:
M −1
(7)
x=f
M
The focal length f and the position of the lens x are chosen
according to the desired magnification M (negative because
the image is inverted). In our implementation, the first additional lens has a focal length of 50mm and is put close
to the microscope objective. The second additional lens has
a focal length of 85mm and is put close to the Lytro. The
effect of using two lenses is that the individual focal lengths
are larger and the aberrations are reduced.
Figure 8. Ray-diagram of the system using two additional lenses.
Each sub-system is indicated in a different color and has to be
interpreted independently from the other sub-systems. Their operation can be understood in sequence: the objective images object
AB to A0 B 0 , the first additional lens images A0 B 0 to A1 B1 so that
the second lens images A1 B1 to A2 B2 behind the Lytro camera.
Finally, the Lytro’s main optics in the inverse regime images this
virtual object A2 B2 to its sensor in the plane A00 B 00 .
6. Macro Lytro
Our second option is to use a single SLR camera lens in
front of the Lytro to achieve unit magnification. (Mf inal =
MSLR MLytro = 1). As for the microscope, and because
the focal length of the SLR lens is large (50mm and 100mm
in our experiments), we want to use the Lytro in the inverse
regime to keep the spatial coverage as high as possible. In
practice, the Lytro is set as close as possible to the SLR
lens. A variation of this setup is to remove the Lytro optics,
and only use the SLR lens so that Mf inal = MSLR = 1.
These designs have only one or two optical components and
relieve the hurdle of undoing the work of the microscope
objective with many lenses. However, the SLR lens is not
specifically designed for the magnification of close objects
and its aperture is not meant to be maintained at a constant
value for the micro-lens array. The relations used to establish Eq. 3 are not valid in this macro-configuration. Instead,
the relevant quantity is the working f-number Fw [1] which
is the f-number modified by the magnification M :
1
= (1 − M )F
(8)
Fw =
2NAi
where NAi is the image-side numerical aperture NAi =
n0 sinα0 . The minimal value of working f-number that can
be achieved with a camera lens is close to Fw = 2. It is
reached for a limit f-number of F = 1 and a magnification
of M = −1. This condition would be optimal for the Lytro
micro-lens array. However, commercial lenses usually have
a limit f-number between 1.4 and 3.5 increasing the working
f-number to between 2.8 and 7.0.
Compared to the light-field microscope, on the one hand,
this setup is more versatile. The magnification can be set
to the desired value by simply moving the object and the
micro-lens array. It does not require the difficult alignment
of several optics. On the other hand, this setup does not
benefit from the structure of the microscope that already includes lighting and moving the sample through micrometer
stages in three dimensions. In addition, the working distance is not fixed which changes the magnification as well
as the object-side numerical aperture when moving the sample. However, the strong point of this design is its accessibility. Building a light-field macrography setup is done
quickly and without the need for a deep understanding of
the operating principles of a microscope.
7. Results
Before showing results, we describe and compare the
different implementations. The different sensor coverages
as well as their spatio-angular coverage values can be found
in Fig. 9. It is clearly visible that directly using the Lytro
camera in its regular imaging regime is unsuitable for microscopic light-field imaging. The inverse imaging regime
improves on the spatial coverage, but the angular coverage
is limited. The best combination is achieved with a 50mm
SLR lens in front of the Lytro which yields the best overall
sensor coverage csensor .
7.1. Resolution Test Chart
In order to compare the resolution of the two different
techniques, we use a 1951 USAF resolution test chart. The
results of the experiments are summarized in Fig. 10.
The first option (Sect. 5) ”Lytro Microscope” was implemented using a Canon 50mm SLR lens and a Nikon 85mm
Figure 9. Combined results of the experiments from Sect. 5 and
Sect. 6.
degrades the image. The aberrations are reduced in the SLR
+ Lytro case as compared to SLR + MLA, since the magnification of the SLR lens is lower in this setting. It is most
noticable in the side views since, for these views, imaging is
performed through the outer pupil regions of the SLR lens.
We suspect that the chromatic aberration is introduced by
the SLR lens because it is not intended for macro-imaging.
Using a dedicated macro-lens instead would likely remove
this effect.
Object:
Resolution
target
Figure 10. This table summarizes the results from the experiments
of Sect. 5 and Sect. 6. Theoretical resolution is derived from the
micro-lens pitch. The micro-lens footprint is the size of a microlens in object space.
SLR lens as additional lenses. Those lenses were put on
top of a Leitz Ergolux microscope using an objective of
magnification 10× with an object-side numerical aperture
NAo = 0.2. This microscope has a lens tube with a magnification of 0.8× so the f-number F = 20. The images
have been taken with a magnification of 2.88, i.e. a microlens covers 4.87µm in object space (see Fig. 11 (top)). The
spatial coverage is above 99% but due to the large magnification the angular coverage is low (between 9% and 25%).
The resolution is between 80 and 90 line pairs per mm.
The resolution indicated above is computed for the center view. It decreases with further distance from the center.
A loss of image quality due to aberrations can be observed.
They are introduced because the observed area is larger
than usual for the microscope objective. Microscope objectives are typically designed so that only a reduced inner
portion of the full field is very well corrected. In addition,
our matching lenses introduce further aberrations. Since the
angular vignetting is strong, the contrast of viewpoints far
from the center is low. It should be noted that even viewpoints inside the vignetted area can be computed, albeit at a
poor signal-to-noise ratio (see Fig. 11).
The second option (Sect. 6) was implemented in three
ways: two times with the Lytro placed behind two different
lenses, a 50mm and a 100mm Canon SLR lens (referred as
SLR + Lytro), and with the Lytro micro-lens array without
the Lytro main optics, see Fig. 3 (right), placed behind the
50mm lens (referred as SLR + MLA) (see Fig. 12). Magnifications from 1.26 to 2.34 were achieved. The spatial coverage is always 100% and angular coverage is good (up to
70%). In this case, chromatic aberrations are present which
SLR Lens
100mm
Lytro
Camera
Figure 12. Setup from the SLR + Lytro experiment using a 100mm
SLR lens (100mm SLR + Lytro).
7.2. Microscopic Sample
The most direct application of the light-field microscope
is the study of microscopic samples. The low magnification
and the large field of view allow us to see in detail an object
area that is between 1.5mm×1.5mm and 3.5mm×3.5mm
with a magnification of 2.88× and 1.3× respectively. Cell
tissues or rough surfaces of different materials have a structure close to the millimeter so high magnification is not always necessary to analyze them.
We illustrate this technique in Fig. 1 (right). Several images of microscopic specimen were taken with the same settings as in the previous section. The magnification is 2.88×
and we can clearly see the structure of different kind of surfaces that are invisible to the naked eye.
The light-field data allows for the reconstruction of the
depth of the sample when the number of views is sufficient. We took a picture of a ground truth aluminium staircase with stairs of 1.00mm width and 0.50mm height with
an accuracy of ±5µm with the 50mmSLR + Lytro setup.
We obtained the depth map in Fig. 13 using a modified
variational multi-scale optical flow algorithm for light-field
depth estimation [15]. Although only a small slice of the
staircase is in focus (the depth of field is 1mm), the depth
can be computed outside of this area. Essentially, out-offocus regions are naturally considered as a different scale
by the algorithm, so, the estimation of the depth of the closest and furthest steps is correct. This behavior nicely interacts with the scene properties since the parallax is larger in
out-of-focus regions. The optical system can be seen as supporting the part of the algorithm that is handling large dis-
Figure 11. Set of different viewpoints of the resolution target with the center view in the middle (the red dot in the top left inset indicates
the position of the view). Top: Images taken with the ”Lytro Microscope” (Sect. 5). The magnification is 2.88. Bottom: Images taken with
the ”50mmSLR + Lytro” (Sect. 6). A red-green color shift due to strong chromatic aberrations can be observed in the side views. Note that
the top row has a higher resolution: it shows the pattern that is visible in the center of the bottom row (level 4 and 5). The contrast of the
images of the same row have been set to a similar level for comparison.
placements. Since the detailed properties of the Lytro main
optics are unknown, it is necessary to adjust the scale of the
computed depth. We use the aluminium staircase as reference. We find the affine transformation between the depth
map and the staircase model by fitting planes to match the
stairs. After the transformation, we measure an RMS error
of 75µm for vertical planes and 17µm for horizontal planes.
The difference is due to the different slopes of the horizontal
and vertical steps with relation to the camera view direction.
The magnification is equal to 1.32. We also applied this
depth reconstruction to a daisy head (see Fig. 13). In practice, the depth inside a cube of about 3.5 × 3.5 × 3.5mm3
can be estimated, which is rather large for a microscopic
setting.
8. Discussion and Conclusion
We have developed and tested several adaptations of
the Lytro consumer light-field camera to enable an entrylevel experimentation with light-field microscopy. While
the fixed f-number of the Lytro’s micro-lens array prevents
its direct use with a standard microscope (regular regime),
it is possible to trade the overall system magnification for
light-field features and to avoid spatial vignetting with the
inverse imaging regime. Lytro microscopy is therefore an
option for low-magnification work as is common in industrial settings, or for investigations into the meso- and largescale micro-structure of materials. Even though an optical
magnification between 1 and 3, as achieved in this work, appears to be low, the small size of the micro-lenses still yields
a decent optical resolution of up to 6.25µm in object space
which already shows interesting optical structures that are
imperceivable by the naked eye.
Figure 13. Sub-view (top left) and computed depth map (top middle) of a daisy flower. Sub-view (bottom left) and computed depth
map (bottom middle) of the aluminium staircase. The bottom right
picture is a 3D visualization of the depth map after the calibrating
and the top right picture is a projection of a region in the middle of
the calibrated depth map onto the xz plane showing the plane fits
in red.
For the future, we would like to investigate image-based
denoising schemes for the vignetted side-views, as well as
algorithmic developments for structure recovery. In terms
of applications, the imaging of micro-and meso-BRDFs and
their relation to macroscopic BRDF models appears to be an
interesting development.
Acknowledgements
We would also like to thank Patrick Reuter for his
helpful comments. This work was supported by the
German Research Foundation (DFG) through EmmyNoether fellowship IH 114/1-1 as well as the ANR
ISAR project of the French Agence Nationale de la
Recherche.
References
[1] M. Born and E. Wolf. Principles of optics. Pergamon Press,
1980. 3, 6
[2] M. Broxton, L. Grosenick, S. Yang, N. Cohen, A. Andalman, K. Deisseroth, and M. Levoy. Wave Optics Theory and
3-D Deconvolution for the Light Field Microscope. Optics
Express, 21(21):25418–25439, 2013. 1, 2
[3] N. Cohen, S. Yang, A. Andalman, M. Broxton, L. Grosenick,
K. Deisseroth, M. Horowitz, and M. Levoy. Enhancing the
performance of the light field microscope using wavefront
coding. Optics express, 22(20):24817–24839, 2014. 2
[4] T. Georgiev and C. Intwala. Light field camera design for
integral view photography. Technical report, Adobe System,
Inc, 2006. 2
[5] I. Ihrke, T. Stich, H. Gottschlich, M. Magnor, and H.-P. Seidel. Fast incident light field acquisition and rendering. Journal of WSCG (WSCG’08), 16(1-3):25–32, 2008. 2
[6] J. Kuc̆era.
Lytro meltdown.
http://optics.
miloush.net/lytro/Default.aspx, 2014. 3
[7] D. Lanman, D. Crispell, M. Wachs, and G. Taubin. Spherical catadioptric arrays: Construction, multi-view geometry,
and calibration. In 3D Data Processing, Visualization, and
Transmission, Third International Symposium on, pages 81–
88. IEEE, 2006. 2
[8] J.-J. Lee, D. Shin, B.-G. Lee, and H. Yoo. 3D Optical
Microscopy Method based on Synthetic Aperture Integral
Imaging. 3D Research, 3(4):1–6, 2012. 2
[9] M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz.
Light field microscopy. In ACM Transactions on Graphics
(TOG), volume 25, pages 924–934. ACM, 2006. 1, 2, 3
[10] M. Levoy, Z. Zhang, and I. McDowall. Recording and Controlling the 4D Light Field in a Microscope using Microlens
Arrays. Journal of Microscopy, 235:144–162, 2009. 1, 2, 3
[11] C.-K. Liang, T.-H. Lin, B.-Y. Wong, C. Liu, and H. H. Chen.
Programmable aperture photography: Multiplexed light field
acquisition. ACM Trans. on Graphics, 27(3):1–10, 2008. 2
[12] G. Lippman. Épreuves réversibles photographies intégrales.
CR Acad. Sci, 146:446–451, 1908. 2
[13] A. W. Lohmann, R. G. Dorsch, D. Mendlovic, Z. Zalevsky,
and C. Ferreira. Space-Bandwidth Product of Optical Signals
and Systems. Journal of the Optical Society of America A,
13(3):470–473, 1996. 5
[14] C.-H. Lu, S. Muenzel, and J. Fleischer. High-resolution
Light-Field Microscopy. In Computational Optical Sensing
and Imaging, pages CTh3B–2. Optical Society of America,
2013. 2
[15] A. Manakov, J. Restrepo, O. Klehm, R. Hegedüs, H.-P. Seidel, E. Eisemann, and I. Ihrke. A Reconfigurable Camera
Add-On for High Dynamic Range, Multispectral, Polarization, and Light-Field Imaging. ACM Trans. on Graphics
(SIGGRAPH’13), 32(4):article 47, 2013. 2, 7
[16] D. B. Murphy and M. W. Davidson. Fundamentals of Light
Microscopy and Electronic Imaging. John Wiley & Sons,
2012. 2
[17] R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz,
and P. Hanrahan. Light field photography with a handheld plenoptic camera. Computer Science Technical Report
CSTR, 2(11), 2005. 2
[18] R. Oldenbourg. Polarized Light Field Microscopy: An Analytical Method using a Microlens Array to Simultaneously
Capture both Conoscopic and Orthoscopic Views of Birefringent Objects. Journal of Microscopy, 231(3):419–432,
2008. 2, 3
[19] Y. Taguchi, A. Agrawal, S. Ramalingam, and A. Veeraraghavan. Axial light field for curved mirrors: Reflect your perspective, widen your view. In Computer Vision and Pattern
Recognition (CVPR), 2010 IEEE Conference on, pages 499–
506. IEEE, 2010. 2
[20] M. Thomas, I. Montilla, J. Marichal-Hernandez,
J. Fernandez-Valdivia, J. Trujillo-Sevilla, and J. RodriguezRamos. Depth Map Extraction from Light Field Microscopes. In 12th Workshop on Information Optics (WIO),
pages 1–3. IEEE, 2013. 2
[21] A. Veeraraghavan, R. Raskar, A. Agrawal, R. Chellappa,
A. Mohan, and J. Tumblin. Non-Refractive Modulators for
Encoding and Capturing Scene Appearance and Depth. In
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1–8, 2008. 2
[22] A. Veeraraghavan, R. Raskar, A. Agrawal, A. Mohan, and
J. Tumblin. Dappled Photography: Mask Enhanced Cameras For Heterodyned Light Fields and Coded Aperture Refocussing. ACM Trans. on Graphics (TOG), 26:69, 2007. 2
[23] B. Wilburn, N. Joshi, V. Vaish, E.-V. Talvala, E. Antunez,
A. Barth, A. Adams, M. Horowitz, and M. Levoy. High
performance imaging using large camera arrays. In ACM
Trans. on Graphics (TOG), volume 24, pages 765–776.
ACM, 2005. 2
[24] B. A. Wilt, L. D. Burns, E. T. W. Ho, K. K. Ghosh, E. A.
Mukamel, and M. J. Schnitzer. Advances in Light Microscopy for Neuroscience. Annual Review of Neuroscience,
32:435, 2009. 2
[25] J. C. Yang, M. Everett, C. Buehler, and L. McMillan. A
real-time distributed light field camera. In Proceedings of
the 13th Eurographics workshop on Rendering, pages 77–
86. Eurographics Association, 2002. 2
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement