# Depth and defocus Admin Obtaining depth Depth from Stereo Active

```5/13/2008
• Next week:
Depth and defocus
Lecture 11
Obtaining depth
• Multiple viewpoints (Stereo)
• Special camera (last lecture on May 9th)
• Active methods
– Anat Levin on Matting
– Tuesday 11am on her SIGGRAPH 2008 paper
• Projects
– Can everyone send me ½ page telling me where they are/what they’ve done
– Projects due Friday 9th 4pm (last class)
Depth from Stereo
• Huge amount of literature in Computer Vision
• Good survey paper: A Taxonomy and Evaluation of Dense Two‐Frame Stereo Correspondence Algorithms. Daniel Scharstein & Richard Szeliski
– Change scene illumination
• Passive methods (require scene texture)
– Defocus analysis
– Multiple images
– Single image
Active methods
• Illuminate scene with light
– Laser range scan
– Binary pattern
http://www.cs.washington.edu/homes/curless/publications/cg99.pdf
Estimated Depth Map
1 of 2 input images Ground truth
Depth map
Active methods
• Illuminate scene with light
– Binary or colored patterns
– Can use LCD projectors
Rapid Shape Acquisition Using Color Structured Light and Multi‐pass Dynamic Programming by Li Zhang, Brian Curless, Steven M. Seitz
1
5/13/2008
Active methods
– 3D Photography using shadows in dual‐space geometry (Bouguet, J.Y & Perona, P.)
Defocus &
Depth of field
Slides from Fredo Durand (MIT)
Circle of confusion
circle of confusion
From Basic Photographic Materials and Processes, Stroebel et al.
Depth of focus
Size of permissible circle?
• Assumption on print size, viewing distance, human
vision
– Typically for 35mm film: diameter = 0.02mm
• Film/sensor resolution
(8μ photosites for high-end SLR )
• Best lenses are around 60 lp/mm
• Diffraction limit
From Basic Photographic Materials and Processes, Stroebel et al.
2
5/13/2008
Depth of field: Object space
Depth of field: more accurate view
• Simplistic view: double cone
– Only tells you about the value of one pixel
– Things are in fact a little more complicated to asses
circles of confusion across the image
– We're missing the magnification factor
(proportional to 1/distance and focal length)
• Backproject the image onto the plane in focus
– Backproject circle of confusion
– Depends on magnification factor
• Depth of field is slightly asymmetrical
sensor
Conjugate of
circle of confusion
Point in focus
lens
Point in focus
lens
Object with texture
Depth of field
Depth of field: more accurate view
Deriving depth of field
• Backproject the image onto the plane in focus
– Backproject circle of confusion
– Depends on magnification factor ~ f/D
•
•
•
•
Circle of confusion C, magnification m
Simplification: m=f/D
Focusing distance D, focal length f, aperture N
As usual, similar triangles
D
~f
D
C
CD/f
f/N
CD/f
lens
d2
d1
Deriving depth of field
Deriving depth of field
D-d1
D
f/N
CD/f
d1
f/N
CD/f
d1
d2
3
5/13/2008
Deriving depth of field
Depth of field and aperture
N2C2D2 term can often be neglected when DoF is small
(conjugate of circle of confusion is smaller than lens aperture)
• Linear: proportional to f number
• Recall: big f number N
means small physical aperture
D
f/N
f/N
CD/f
d2
d1
CD/f
d1
d2
DoF & aperture
Depth of field & focal length
• http://www.juzaphoto.com/eng/articles/depth_of_field.htm
• Recall that to get the same image size,
we can double the focal length and the distance
• Recall what happens to physical aperture size when
we double the focal length for the same f number?
– It is doubled
24mm
f/2.8
50mm
f/32
Depth of field & focal length
DoF & Focal length
• Same image size (same magnification),
same f number
DoF
• Same depth of field!
• http://www.juzaphoto.com/eng/articles/depth_of_fiel
d.htm
Wide-angle
g
lens
DoF
50mm f/4.8
Telephoto
lens (2x f), same aperture
200mm f/4.8
(from 4 times farther)
4
5/13/2008
Important conclusion
Important conclusion
• For a given image size and a given f number, the
depth of field (in object space) is the same.
• Might be counter intuitive.
• For a given image size and a given f number, the
depth of field (in object space) is the same.
– The depth of acceptable sharpness is the same
• But background far far away looks more blurry
g
more
Because it ggets magnified
• Plus, usually, you don't keep magnification constant
• Very useful for macro where DoF is critical.
critical You
can change your working distance without affecting
depth of field
• Now what happens to the background blur far far
away?
Effect of parameters
Is depth of field a blur?
aperture
focusing distance
focal length
From applied photographic optics
Bokeh
• Depth of field is NOT a
convolution of the image
• The circle of confusion
varies with depth
• There are interesting
occlusion effects
• (If you really want a
convolution, there is one,
but in 4D space… more
From Macro Photography
Shape depends on aperture
• Pattern of out‐of‐focus blur
• Also on location within image
http://www.bobatkins.com/photo
graphy/technical/bokeh.html
http://www.bobatkins.com/photography/technical/bokeh.html
5
5/13/2008
Comparison of lenses
Mirror Lens
• Blur pattern
http://photo.net/learn/optics/mirrors/tamron500‐8a.jpg
http://www.bobatkins.com/photography/technical/bokeh.html
Confocal Stereo
Depth from defocus – Multiple images
• Multiple sensors
S. Nayar
• Hasinoff & Kutulakos ECCV’06
• Vary aperture and focus of lens
– Multiple images • Time multiplex
– Need static scene
Output #1: Depth map
Image and Depth from a
Conventional Camera with a
Coded Aperture
Single input image:
Anat Levin, Rob Fergus, Frédo
Durand, William Freeman
MIT CSAIL
6
5/13/2008
Output #1: Depth map
Single input image:
O
Output
#2:
#2 All-focused
All f
d iimage
Lens and defocus
Image of a point
light source
Lens’ aperture
Lens
Camera
sensor
function
Focal plane
Lens and defocus
Lens and defocus
Image of a
defocused point
light source
Lens’ aperture
Object
Lens
Camera
sensor
Object
function
Focal plane
Image of a
defocused point
light source
Lens’ aperture
Lens
Camera
sensor
function
Focal plane
7
5/13/2008
Lens and defocus
Lens and defocus
Image of a
defocused point
light source
Lens’ aperture
Object
Camera
sensor
Lens
Image of a
defocused point
light source
Lens’ aperture
Object
Camera
sensor
Lens
function
function
Focal plane
Focal plane
Depth and defocus
Challenges
Out of focus
• Hard to discriminate a smooth scene from defocus blur
?
Out of focus
Depth from defocus:
Infer depth by analyzing
local scale of defocus blur
• Hard to undo defocus blur
Input
In focus
Ringing with conventional
deblurring algorithm
Related Work
Key contributions
• Exploit prior on natural images
• Depth from (de)focus
e.g.
• Plenoptic/ light field cameras
e.g.
Pentland, Chaudhuri, Favaro et al.
- Improve deconvolution
Adelson and Wang, Ng et al.
- Improve depth discrimination
• Wave front coding
e.g.
• Coded apertures for light gathering:
e.g.
• Blind Deconvolution
e.g.
Cathey & Dowski
Natural
• Coded aperture (mask inside lens)
- make defocus patterns different from
Unnatural
F i
Fenimore
and
dC
Cannon
Kundur and Hatzinakos , Fergus et al, Levin
natural images and easier to discriminate
Never recover both depth AND full resolution image from a single image
Except: Veeraraghavan, Raskar, Agrawal, Mohan, Tumblin SIGGRAPH07
optimize debluring while we optimize depth discrimination
8
5/13/2008
Defocus as local convolution
Defocus as local convolution
y = fk ⊗ x
Calibrated blur kernels at
different depths
Input defocused
image
Local
sub-window
Input defocused
image
y = fk ⊗ x
Depth k=2:
y = fk ⊗ x
Depth k=3:
y = fk ⊗ x
Challenges
Try deconvolving local input windows with different scaled filters:
= ⊗
= ⊗
Sharp
sub-window
Depth k=1:
Overview
= ⊗
Calibrated
blur kernels
at depth k
?
?
?
• Hard to deconvolve even
when kernel is known
Input
Larger scale
Correct scale
• H
Hard
d to identify
id if
correct scale:
Smaller scale
Somehow: select best scale.
Deconvolution is ill posed
?
?
?
Richardson-Lucy deconvolution
algorithm
= ⊗
Larger scale
= ⊗
Correct scale
= ⊗
Smaller scale
Deconvolution is ill posed
f ⊗ x = y
f ⊗ x = y
Solution 1:
⊗
?
⊗
=
?
=
Solution 2:
⊗
?
=
9
5/13/2008
Idea 1: Natural images prior
Deconvolution with prior
Natural
| f ⊗ x − y |2 + λ ∑i ρ (∇xi )
x = arg min
What makes images special?
Unnatural
Convolution error
Image
⊗
?
Derivatives prior
2
_
+
Low
Equal convolution error
⊗
?
2
_
+
High
Comparing deconvolution algorithms
Comparing deconvolution algorithms
(Non blind) deconvolution code available online:
http://groups.csail.mit.edu/graphics/CodedAperture/
Input
Richardson-Lucy
ρ (∇ x ) = ∇ x
ρ (∇ x ) = ∇ x
2
0 .8
Gaussian prior
Sparse prior
Recall: Overview
(Non blind) deconvolution code available online:
http://groups.csail.mit.edu/graphics/CodedAperture/
Input
ρ (∇ x ) = ∇ x
Richardson-Lucy
2
ρ (∇ x ) = ∇ x
0 .8
Gaussian prior
Sparse prior
Idea 2: Coded Aperture
Try deconvolving local input windows with different scaled filters:
• Mask (code) in aperture plane
= ⊗
Larger scale
= ⊗
C
Correct
t scale
l
= ⊗
Smaller scale
?
?
?
Somehow: select best scale.
Challenge: smaller scale not so different than correct
- make defocus patterns different from
natural images and easier to discriminate
Conventional
aperture
Our coded
aperture
10
5/13/2008
Solution: lens with occluder
Solution: lens with occluder
Image of a
defocused point
light source
Aperture pattern
Object
Lens
Camera
sensor
Object
Lens with coded
aperture
function
function
Focal plane
Focal plane
Solution: lens with occluder
Solution: lens with occluder
Image of a
defocused point
light source
Aperture pattern
Object
Lens with coded
aperture
Camera
sensor
Image of a
defocused point
light source
Aperture pattern
Object
Lens with coded
aperture
function
Focal plane
function
Solution: lens with occluder
Image of a
defocused point
light source
Aperture pattern
Lens with coded
aperture
Camera
sensor
Image of a
defocused point
light source
Aperture pattern
Object
function
Focal plane
Camera
sensor
Focal plane
Solution: lens with occluder
Object
Camera
sensor
Lens with coded
aperture
Camera
sensor
function
Focal plane
11
5/13/2008
Convolution- frequency domain representation
spectrum
Larger scale
spectrum
Correct scale
scale
0
Frequency
0
Frequency
0
Frequency
Filter, 2nd scale
0
0
spectrum
Filter, correct scale
Frequency
Division by zero
⇒spatial ringing
ω
ω
Frequency
=
Frequency
Output spectrum has zeros
?
Frequency
=
Observed image
0
0
spectrum
spectrum
spectrum
Filter, wrong scale
?
ω
2nd observed image
⇔frequency multiplication
where filter spectrum has zeros
0
0
Estimated image
Estimated image
Frequency
0
=
Division by zero with a conventional aperture?
spectrum
=
Observed image
Frequency
0
Estimated image
spectrum
spectrum
spectrum
Filter, correct scale
spectrum
Frequency
1st observed image
Sharp Image
Coded aperture: Scale estimation and division by zero
0
=
0
Spatial convolution
?
spectrum
Filter,
Smaller scale
Estimated image
Frequency
0
1st
spectrum
Coded
spectrum
Conventional
Sharp Image
spectrum
Coded aperture- reduce uncertainty in scale identification
spectrum
Why coded?
Filter, wrong scale
ω
Frequency
0
0
Filter Design
Frequency
division of tiny value by zero
⇒ no spatial ringing
?
Frequency
ω
Frequency
ω
=
Frequency
Zero frequencies- pros and cons
Analytically search for a pattern maximizing discrimination
between images at different defocus scales (KL-divergence)
Previous talk:
Account for image prior and physical constraints
See paper for details
0
Score
Our solution:
0
0
No zero frequencies:
More discrimination
between scales
+
-
Less discrimination
between scales
Sampled aperture patterns
Conventional
aperture
Filter can be
easily inverted
Weaker depth
discrimination
Enhanced Cameras for Heterodyned
Light Fields and Coded Aperture
Refocusing Ashok Veeraraghavan
Mitsubishi Electric Research Labs
(MERL)‫כ‬, Cambridge, MA Ankit
Mohan† Jack Tumblin‡
0
Include zero frequencies:
+
+
Zeros improve depth discrimination
Inversion difficult
with image priors
12
5/13/2008
Regularizing depth estimation
Try deblurring with 10 different aperture scales
x = arg min | f ⊗ x − y |2
Convolution error
Derivatives prior
2
_
⊗
Depth results
+ λ ∑i ρ (∇xi )
+
Keep minimal error scale in each local window + regularization
200
235
245
255
265
275
285
295
305
Input
Regularizing depth estimation
Local depth estimation
Regularized depth
Sometimes, manual intervention
200
200
235
235
245
245
255
255
265
265
275
275
285
285
295
295
305
Input
305
Local depth estimation
Input
Local depth estimation
200
235
235
235
245
245
245
255
255
255
265
265
265
275
275
275
285
285
285
295
295
295
305
305
Regularized depth
305
After user corrections
Regularized depth
Input
All focused results
13
5/13/2008
Close-up
All-focused
(deconvolved)
Original image
All-focus image
Input
All-focused
(deconvolved)
Close-up
Comparison- conventional aperture result
Ringing due to wrong scale estimation
Original image
All-focus image
Naïve sharpening
14
5/13/2008
Comparison- coded aperture result
Application: Digital refocusing from a single image
Application: Digital refocusing from a single image
Application: Digital refocusing from a single image
Application: Digital refocusing from a single image
Application: Digital refocusing from a single image
15
5/13/2008
Application: Digital refocusing from a single image
Application: Digital refocusing from a single image
Coded aperture: pros and cons
Deconvolution code available
+ Image AND depth at a single shot
+ No loss of image resolution
+ Simple modification to lens
- Depth is coarse
http://groups.csail.mit.edu/graphics/CodedAperture/
unable to get depth at untextured areas,
might
g need manual corrections.
+ But depth is a pure bonus
- Loss some light
+ But deconvolution increases depth of field
• Fill in feedback forms
50mm f/1.8:
\$79.95
Cardboard:
\$1
Tape:
\$1
Depth acquisition:
priceless
16
```