Improving Resolution and Depth-of-Field of Light Field

Dappled Photography: Mask Enhanced Cameras for
Heterodyned Light Fields and Coded Aperture Refocusing
Ashok Veeraraghavan, Ramesh Raskar, Amit Agrawal,
Ankit Mohan
&
Jack Tumblin
Mitsubishi Electric Research Labs (MERL), Cambridge, MA, USA
Northwestern University, IL
Coded Exposure [Raskar, Agrawal, Tumblin SIGGRAPH 2006]
Coded Exposure (Flutter Shutter) Camera
Raskar, Agrawal, Tumblin [Siggraph2006]
Coding in Time: Shutter is opened and closed
Blurring == Convolution
Sharp Photo
Blurred Photo
PSF == Sinc Function
ω
Traditional Camera: Shutter is OPEN: Box Filter
Coded Exposure
Sharp Photo
Blurred Photo
PSF == Broadband Function
Preserves High Spatial
Frequencies
Flutter Shutter: Shutter is OPEN and CLOSED
Traditional
Deblurred
Image
Coded
Exposure
Deblurred
Image
Image of
Static
Object
How to handle focus
blur?
Coded Exposure
Coded Aperture
Temporal 1-D broadband code:
Motion Deblurring
Spatial 2-D broadband mask:
Focus Deblurring
Point light
source (LED)
In Focus Photo
Out of Focus Photo: Open Aperture
Lens and defocus
Image of a point
light source
Lens’ aperture
Lens
Camera
sensor
Point
spread
function
Focal plane
Slide Credit: Levin et. al
Lens and defocus
Image of a
defocused point
light source
Lens’ aperture
Object
Lens
Camera
sensor
Point
spread
function
Focal plane
Slide Credit: Levin et. al
Lens and defocus
Image of a
defocused point
light source
Lens’ aperture
Object
Lens
Camera
sensor
Point
spread
function
Focal plane
Slide Credit: Levin et. al
Lens and defocus
Image of a
defocused point
light source
Lens’ aperture
Object
Lens
Camera
sensor
Point
spread
function
Focal plane
Slide Credit: Levin et. al
Lens and defocus
Image of a
defocused point
light source
Lens’ aperture
Object
Lens
Camera
sensor
Point
spread
function
Focal plane
Slide Credit: Levin et. al
Out of Focus Photo: Coded Aperture
Blurred Photos
Open Aperture
Coded Aperture, 7 * 7 Mask
Deblurred Photos
Open Aperture
Coded Aperture, 7 * 7 Mask
Captured Blurred Photo
Full Resolution
Digital Refocusing
Blur Estimation & Segmentation
•Defocus blur dependent on depth
•Assumptions
•Layered Lambertian Scene
•Constant blur within each layer
k=1
Deblur at
different blur
sizes k
Captured Blurred Photo
k = 10
Define Cost Function
k=1
k = 10
Deblurred Images
k=1
k = 10
Cost Function Images
Likelihood Error: (Blurred image - Sharp Image * PSFk)2
Gradient Error: Natural Image Statistics, Gradient Kurtosis
Blur Estimation & Segmentation == Labeling
•
Graph cuts for labeling
k=1
K=1
k = 10
Error Images
K=7
Captured
Photo
Reblur
Deblur, k = 7
Fusion
Less is More
Blocking Light == More Information
Coded Exposure
Coding in Time
Coded Aperture
Coding in Space
Flexible Depth of Field
Photography
Nagahara, Kuthirammal, Zhou, and Nayar
ECCV 2008
Slide-deck credit: Nagahara et al.
Hardware Setup
Captured Image
Aperture f/1.4, Exposure 0.36 sec
Deblurred EDOF image
Single “traditional” Image
Aperture f/1.4, Exposure 0.36 sec
Single image with same EDOF
Aperture f/8, Exposure 0.36 sec
Captured Image
Aperture f/1.4, Exposure 0.36 sec
Deblurred EDOF image
Single “traditional” Image
Aperture f/1.4, Exposure 0.36 sec
Single image with same EDOF
Aperture f/8, Exposure 0.36 sec
Tunable focus ring
Discontinuous DOF
Discontinuous DOF
Aperture f/11
Discontinuous DOF
Aperture f/1.4
Tilted DOF
Image from normal camera
Aperture f/1.4
Tilted DOF
Aperture f/1.4
Non-planar DOF
Image from a normal camera
Aperture f/1.4
Non-planar DOF
Aperture f/1.4
Multi-Aperture Photography
Paul Green – MIT CSAIL
Wenyang Sun – MERL
Wojciech Matusik – MERL
Frédo Durand – MIT CSAIL
Motivation
Depth of Field Control
Portrait
Landscape
Large Aperture
Shallow Depth of Field
Small Aperture
Large
Field
Depth of
http://photographertips.net
Depth and Defocus Blur
sensor
lens
plane of focus
circle of
confusion
subject
rays from point in focus converge to single pixel
defocus blur depends on distance from plane of focus
Defocus Blur & Aperture
sensor
plane of focus
lens
aperture
circle of
confusion
subject
defocus blur depends on aperture size
http://photographertips.net
Goals
Aperture size is a critical parameter
for photographers
■ post-exposure
depth of field control
■ extrapolate shallow depth of field
beyond physical aperture
Outline
Multi-Aperture Camera
– New camera design
– Capture multiple aperture settings
simultaneously
Applications
– Depth of field control
– Depth of field extrapolation
– (Limited) refocusing
Related Work
Computational Cameras
– Plenoptic Cameras
• Adelson and Wang ‘92
• Ng et al ‘05
• Georgiev et al ‘06
– Split-Aperture Camera
Georgiev et al‘06
Adelson and Wang ‘92
• Aggarwal and Ahuja ‘04
– Optical Splitting Trees
• McGuire et al ‘07
– Coded Aperture
• Levin et al ’07
• Veeraraghavan et al ’07
McGuire et al ‘07
Aggarwal and Ahuja ‘04
– Wavefront Coding
• Dowski and Cathey ‘95
Depth from Defocus
– Pentland ‘87
Levin et al ’07
Veeraraghavan et al ’07
Plenoptic Cameras
Lens Aperture
Capture 4D LightField
v
– 2D Spatial (x,y)
– 2D Angular (u,v Aperture)
Trade resolution for flexibility after
capture
u
– Refocusing
– Depth of field control
Lenslet Array
– Improved Noise Characteristics
Subject
Sensor (x,y)
Lens (u,v)
1D vs 2D Aperture Sampling
Aperture
v
u
2D Grid
Sampling
http://photographertips.net
1D vs. 2D Aperture Sampling
Aperture
v
45 Samples
Aperture
4 Samples
u
2D Grid
Sampling
1D “Ring”
Sampling
http://photographertips.net
Optical Design Principles
3D sampling
– 2D spatial
– 1D aperture size
– 1 image for each “ring”
Aperture
http://photographertips.net
Sensor
Aperture Splitting
Goal: Split aperture into 4 separate optical paths
– concentric tilted mirrors
– at aperture plane
Tilted Mirrors
Aperture Splitting
Mirrors
Focusing lenses
Sensor
Incoming light
Tilted Mirrors
Aperture Splitting
Ideally at aperture plane
, but not physically possible!
Solution: Relay Optics to create virtual aperture plane
Photographic
RelayLens
system
Aperture splitting optics
X
Aperture Plane
New Aperture Plane
Optical Prototype
mirrors
main lens
relay optics
Mirror Close-up
lenses
tilted mirrors
SLR Camera
Sample Data
Raw data from our camera
Point Spread Function Occlusion
inner
ring 1
ring 2
Ideally would be rings
Gaps are from occlusion
outer
combined
Outline
Multi-Aperture Camera
– New camera design
– Capture multiple aperture settings
simultaneously
Applications
– Depth of field control
– Depth of field extrapolation
– Refocusing
DOF Navigation
I0
I2
I1
I3
DOF Extrapolation?
I
0
I1
I2
Approximate defocus blur as
convolution
?
I n  I 0  K ( n )
Depends on depth and
aperture size
K ( n )
- Circular aperture blurring
kernel
I3
IE
What is
E
at each pixel in
I E?
DOF Extrapolation Roadmap
estimate blur
fit model
Largest
physical
aperture
Blur size
capture
I0
I1
I2
IE
I
Aperture Diameter
3
extrapolate blur
Defocus Gradient
σ
Defocus blur
IE
Blur proportional to
I3
aperture diameter
I
2
I
I0 1
Blur size
σ
D
do
ds
sensor distance

Defocus Gradient
Largest physical
aperture
focal length
Aperture Diameter D
G is slope of this line
Defocus Gradient Map
 (d s  f )d o  fd s 

D
G
fdo


distance
( d s  f ) d oobject
 fd
s
G
fd o
aperture diameter
Optimization
solve for discrete defocus gradient values G at
each pixel
Data term
D (G )   I i  I 0  K (G N1i )
i
Graph Cuts with spatial regularization term
Smallest Aperture Image
Defocus Gradient Map
Depth of Field Extrapolation
Synthetic Refocusing
Modify gradient labels and re-synthesize image
gradient map
“refocused” map
extrapolated f/1.8
“refocused” synthetic
f/1.8
Discussion
■ Occlusion
■ Could help depth discrimination (coded aperture)
■ Difficult alignment process
■ Mostly because prototype
■ Refocusing limited by Depth of Field
■ helped by depth-guided deconvolution
■ Texture required for accurate defocus gradient map
■ Not critical for depth of field and refocus
74
4D Frequency Analysis of
Computational Cameras for
Depth of Field Extension
Anat Levin1,2
Sam Hasinoff 1 Paul Green1
Frédo Durand 1
Bill Freeman1
1MIT CSAIL 2Weizmann Institute
75
Defocus blur in a standard lens
At focus depth,
sharp
Away from focus
depth, blurred
Small aperture – increased depth of field but noisy
76
Depth 1: sharp but
noisy
Depth 2: sharp but
noisy
77
Extended depth of field cameras
input
Modified optics
output
Deconvolution
Extended DOF cameras: remove blur computationally
and design optics to make that easy
In this talk
• How much can depth of field be extended?
• New lens extending depth of field
78
79
The lattice-focal lens
Our design: assembly of subsquares with
different focal powers
each element focuses on a different depth
toy lattice-focal lens

2
with 4 elements

E s ( x , y )  

A8 / 3
 S 4 / 31/ 3 |  x , y |
80
Hardware construction
Proof of concept
• 12 subsquares cut
from plano-convex
spherical lenses
• Attached to main lens
extra focal power
needed very low
• Modest DOF
extension with only
12 subsquares
81
Depth estimation
• Defocus kernels vary with depth
defocus kernels at
different depths
• Depth estimation as for the coded aperture camera
[Levin et al. 07]
input
depth map
82
Standard lens reference
Lattice-focal lens
84
Standard lens reference
Lattice-focal lens
86
Standard lens reference
Results
Lattice-focal lens
88
Application: Refocusing from single captured image
89
Application: Refocusing from single captured image
90
Application: Refocusing from single captured image
The lattice-focal lens – limitations
• Depth estimation needed for deblurring
• Only capture part of the 4D light field
spectrum
• Subsquare size and focal power
optimized for a given focusing range
• Higher spectrum than previous designs,
but does not reach the upper bound
91
Download PDF