5/13/2008 Admin • Next week: Depth and defocus Lecture 11 Obtaining depth • Multiple viewpoints (Stereo) • Special camera (last lecture on May 9th) • Active methods – Anat Levin on Matting – Tuesday 11am on her SIGGRAPH 2008 paper • Projects – Can everyone send me ½ page telling me where they are/what they’ve done – Projects due Friday 9th 4pm (last class) Depth from Stereo • Huge amount of literature in Computer Vision • Good survey paper: A Taxonomy and Evaluation of Dense Two‐Frame Stereo Correspondence Algorithms. Daniel Scharstein & Richard Szeliski – Change scene illumination • Passive methods (require scene texture) – Defocus analysis – Multiple images – Single image Active methods • Illuminate scene with light – Laser range scan – Binary pattern http://www.cs.washington.edu/homes/curless/publications/cg99.pdf Estimated Depth Map 1 of 2 input images Ground truth Depth map Active methods • Illuminate scene with light – Binary or colored patterns – Can use LCD projectors Rapid Shape Acquisition Using Color Structured Light and Multi‐pass Dynamic Programming by Li Zhang, Brian Curless, Steven M. Seitz 1 5/13/2008 Active methods • Use shadows: – 3D Photography using shadows in dual‐space geometry (Bouguet, J.Y & Perona, P.) Defocus & Depth of field Slides from Fredo Durand (MIT) Circle of confusion circle of confusion From Basic Photographic Materials and Processes, Stroebel et al. Depth of focus Size of permissible circle? • Assumption on print size, viewing distance, human vision – Typically for 35mm film: diameter = 0.02mm • Film/sensor resolution (8μ photosites for high-end SLR ) • Best lenses are around 60 lp/mm • Diffraction limit From Basic Photographic Materials and Processes, Stroebel et al. 2 5/13/2008 Depth of field: Object space Depth of field: more accurate view • Simplistic view: double cone – Only tells you about the value of one pixel – Things are in fact a little more complicated to asses circles of confusion across the image – We're missing the magnification factor (proportional to 1/distance and focal length) • Backproject the image onto the plane in focus – Backproject circle of confusion – Depends on magnification factor • Depth of field is slightly asymmetrical sensor Conjugate of circle of confusion Point in focus lens Point in focus lens Object with texture Depth of field Depth of field: more accurate view Deriving depth of field • Backproject the image onto the plane in focus – Backproject circle of confusion – Depends on magnification factor ~ f/D • • • • Circle of confusion C, magnification m Simplification: m=f/D Focusing distance D, focal length f, aperture N As usual, similar triangles D ~f D C CD/f f/N CD/f lens d2 d1 Deriving depth of field Deriving depth of field D-d1 D f/N CD/f d1 f/N CD/f d1 d2 3 5/13/2008 Deriving depth of field Depth of field and aperture N2C2D2 term can often be neglected when DoF is small (conjugate of circle of confusion is smaller than lens aperture) • Linear: proportional to f number • Recall: big f number N means small physical aperture D f/N f/N CD/f d2 d1 CD/f d1 d2 DoF & aperture Depth of field & focal length • http://www.juzaphoto.com/eng/articles/depth_of_field.htm • Recall that to get the same image size, we can double the focal length and the distance • Recall what happens to physical aperture size when we double the focal length for the same f number? – It is doubled 24mm f/2.8 50mm f/32 Depth of field & focal length DoF & Focal length • Same image size (same magnification), same f number DoF • Same depth of field! • http://www.juzaphoto.com/eng/articles/depth_of_fiel d.htm Wide-angle g lens DoF 50mm f/4.8 Telephoto lens (2x f), same aperture 200mm f/4.8 (from 4 times farther) See also http://luminous-landscape.com/tutorials/dof2.shtml 4 5/13/2008 Important conclusion Important conclusion • For a given image size and a given f number, the depth of field (in object space) is the same. • Might be counter intuitive. • For a given image size and a given f number, the depth of field (in object space) is the same. – The depth of acceptable sharpness is the same • But background far far away looks more blurry g more Because it ggets magnified • Plus, usually, you don't keep magnification constant • Very useful for macro where DoF is critical. critical You can change your working distance without affecting depth of field • Now what happens to the background blur far far away? Effect of parameters Is depth of field a blur? aperture focusing distance focal length From applied photographic optics Bokeh • Depth of field is NOT a convolution of the image • The circle of confusion varies with depth • There are interesting occlusion effects • (If you really want a convolution, there is one, but in 4D space… more about this in ten days) From Macro Photography Shape depends on aperture • Pattern of out‐of‐focus blur • Also on location within image http://www.bobatkins.com/photo graphy/technical/bokeh.html http://www.bobatkins.com/photography/technical/bokeh.html 5 5/13/2008 Comparison of lenses Mirror Lens • Blur pattern http://photo.net/learn/optics/mirrors/tamron500‐8a.jpg http://www.bobatkins.com/photography/technical/bokeh.html Confocal Stereo Depth from defocus – Multiple images • Multiple sensors S. Nayar • Hasinoff & Kutulakos ECCV’06 • Vary aperture and focus of lens – Multiple images • Time multiplex – Need static scene Output #1: Depth map Image and Depth from a Conventional Camera with a Coded Aperture Single input image: Anat Levin, Rob Fergus, Frédo Durand, William Freeman MIT CSAIL 6 5/13/2008 Output #1: Depth map Single input image: O Output #2: #2 All-focused All f d iimage Lens and defocus Image of a point light source Lens’ aperture Lens Camera sensor Point spread function Focal plane Lens and defocus Lens and defocus Image of a defocused point light source Lens’ aperture Object Lens Camera sensor Object Point spread function Focal plane Image of a defocused point light source Lens’ aperture Lens Camera sensor Point spread function Focal plane 7 5/13/2008 Lens and defocus Lens and defocus Image of a defocused point light source Lens’ aperture Object Camera sensor Lens Image of a defocused point light source Lens’ aperture Object Camera sensor Lens Point spread function Point spread function Focal plane Focal plane Depth and defocus Challenges Out of focus • Hard to discriminate a smooth scene from defocus blur ? Out of focus Depth from defocus: Infer depth by analyzing local scale of defocus blur • Hard to undo defocus blur Input In focus Ringing with conventional deblurring algorithm Related Work Key contributions • Exploit prior on natural images • Depth from (de)focus e.g. • Plenoptic/ light field cameras e.g. Pentland, Chaudhuri, Favaro et al. - Improve deconvolution Adelson and Wang, Ng et al. - Improve depth discrimination • Wave front coding e.g. • Coded apertures for light gathering: e.g. • Blind Deconvolution e.g. Cathey & Dowski Natural • Coded aperture (mask inside lens) - make defocus patterns different from Unnatural F i Fenimore and dC Cannon Kundur and Hatzinakos , Fergus et al, Levin natural images and easier to discriminate Never recover both depth AND full resolution image from a single image Except: Veeraraghavan, Raskar, Agrawal, Mohan, Tumblin SIGGRAPH07 optimize debluring while we optimize depth discrimination 8 5/13/2008 Defocus as local convolution Defocus as local convolution y = fk ⊗ x Calibrated blur kernels at different depths Input defocused image Local sub-window Input defocused image y = fk ⊗ x Depth k=2: y = fk ⊗ x Depth k=3: y = fk ⊗ x Challenges Try deconvolving local input windows with different scaled filters: = ⊗ = ⊗ Sharp sub-window Depth k=1: Overview = ⊗ Calibrated blur kernels at depth k ? ? ? • Hard to deconvolve even when kernel is known Input Larger scale Correct scale • H Hard d to identify id if correct scale: Smaller scale Somehow: select best scale. Deconvolution is ill posed ? ? ? Ringing with the traditional Richardson-Lucy deconvolution algorithm = ⊗ Larger scale = ⊗ Correct scale = ⊗ Smaller scale Deconvolution is ill posed f ⊗ x = y f ⊗ x = y Solution 1: ⊗ ? ⊗ = ? = Solution 2: ⊗ ? = 9 5/13/2008 Idea 1: Natural images prior Deconvolution with prior Natural | f ⊗ x − y |2 + λ ∑i ρ (∇xi ) x = arg min What makes images special? Unnatural Convolution error Image ⊗ ? Derivatives prior 2 _ + Low Equal convolution error gradient ⊗ ? Natural images have sparse gradients 2 _ + High put a penalty on gradients Comparing deconvolution algorithms Comparing deconvolution algorithms (Non blind) deconvolution code available online: http://groups.csail.mit.edu/graphics/CodedAperture/ Input Richardson-Lucy ρ (∇ x ) = ∇ x ρ (∇ x ) = ∇ x 2 0 .8 “spread” gradients “localizes” gradients Gaussian prior Sparse prior Recall: Overview (Non blind) deconvolution code available online: http://groups.csail.mit.edu/graphics/CodedAperture/ Input ρ (∇ x ) = ∇ x Richardson-Lucy 2 ρ (∇ x ) = ∇ x 0 .8 “spread” gradients “localizes” gradients Gaussian prior Sparse prior Idea 2: Coded Aperture Try deconvolving local input windows with different scaled filters: • Mask (code) in aperture plane = ⊗ Larger scale = ⊗ C Correct t scale l = ⊗ Smaller scale ? ? ? Somehow: select best scale. Challenge: smaller scale not so different than correct - make defocus patterns different from natural images and easier to discriminate Conventional aperture Our coded aperture 10 5/13/2008 Solution: lens with occluder Solution: lens with occluder Image of a defocused point light source Aperture pattern Object Lens Camera sensor Object Lens with coded aperture Point spread function Point spread function Focal plane Focal plane Solution: lens with occluder Solution: lens with occluder Image of a defocused point light source Aperture pattern Object Lens with coded aperture Camera sensor Image of a defocused point light source Aperture pattern Object Lens with coded aperture Point spread function Focal plane Point spread function Solution: lens with occluder Image of a defocused point light source Aperture pattern Lens with coded aperture Camera sensor Image of a defocused point light source Aperture pattern Object Point spread function Focal plane Camera sensor Focal plane Solution: lens with occluder Object Camera sensor Lens with coded aperture Camera sensor Point spread function Focal plane 11 5/13/2008 Convolution- frequency domain representation spectrum Larger scale spectrum Correct scale scale 0 Frequency 0 Frequency 0 Frequency Filter, 2nd scale 0 0 spectrum Filter, correct scale Frequency Division by zero ⇒spatial ringing ω ω Frequency = Frequency Output spectrum has zeros ? Frequency = Observed image 0 0 spectrum spectrum spectrum Filter, wrong scale ? ω 2nd observed image ⇔frequency multiplication where filter spectrum has zeros 0 0 Estimated image Estimated image Frequency 0 = Division by zero with a conventional aperture? spectrum = Observed image Frequency 0 Estimated image spectrum spectrum spectrum Filter, correct scale spectrum Frequency 1st observed image Sharp Image Coded aperture: Scale estimation and division by zero 0 = 0 Spatial convolution ? spectrum Filter, Smaller scale Estimated image Frequency 0 1st spectrum Coded spectrum Conventional Sharp Image spectrum Coded aperture- reduce uncertainty in scale identification spectrum Why coded? Filter, wrong scale ω Frequency 0 0 Filter Design Frequency division of tiny value by zero ⇒ no spatial ringing ? Frequency ω Frequency ω = Frequency Zero frequencies- pros and cons Analytically search for a pattern maximizing discrimination between images at different defocus scales (KL-divergence) Previous talk: Account for image prior and physical constraints See paper for details 0 Score Our solution: 0 0 No zero frequencies: More discrimination between scales + - Less discrimination between scales Sampled aperture patterns Conventional aperture Filter can be easily inverted Weaker depth discrimination Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing Ashok Veeraraghavan Ramesh Raskar Amit Agrawal Mitsubishi Electric Research Labs (MERL)כ, Cambridge, MA Ankit Mohan† Jack Tumblin‡ 0 Include zero frequencies: + + Zeros improve depth discrimination Inversion difficult Inversion made possible with image priors 12 5/13/2008 Regularizing depth estimation Try deblurring with 10 different aperture scales x = arg min | f ⊗ x − y |2 Convolution error Derivatives prior 2 _ ⊗ Depth results + λ ∑i ρ (∇xi ) + Keep minimal error scale in each local window + regularization 200 235 245 255 265 275 285 295 305 Input Regularizing depth estimation Local depth estimation Regularized depth Sometimes, manual intervention 200 200 235 235 245 245 255 255 265 265 275 275 285 285 295 295 305 Input 305 Local depth estimation Input Local depth estimation 200 235 235 235 245 245 245 255 255 255 265 265 265 275 275 275 285 285 285 295 295 295 305 305 Regularized depth 305 After user corrections Regularized depth Input All focused results 13 5/13/2008 Close-up All-focused (deconvolved) Original image All-focus image Input All-focused (deconvolved) Close-up Comparison- conventional aperture result Ringing due to wrong scale estimation Original image All-focus image Naïve sharpening 14 5/13/2008 Comparison- coded aperture result Application: Digital refocusing from a single image Application: Digital refocusing from a single image Application: Digital refocusing from a single image Application: Digital refocusing from a single image Application: Digital refocusing from a single image 15 5/13/2008 Application: Digital refocusing from a single image Application: Digital refocusing from a single image Coded aperture: pros and cons Deconvolution code available + Image AND depth at a single shot + No loss of image resolution + Simple modification to lens - Depth is coarse http://groups.csail.mit.edu/graphics/CodedAperture/ unable to get depth at untextured areas, might g need manual corrections. + But depth is a pure bonus - Loss some light + But deconvolution increases depth of field Admin • Fill in feedback forms – Can someone collect and return to 305WWH 50mm f/1.8: $79.95 Cardboard: $1 Tape: $1 Depth acquisition: priceless 16

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement