Video Stabilization and Rectification for Handheld Cameras

Video Stabilization and Rectification for Handheld Cameras
Video Stabilization and Rectification for Handheld Cameras
Chao Jia
Ph.D. Defense
Committee Members:
Prof. Alan Bovik
Prof. Constantine Caramanis
Prof. Brian L. Evans (supervisor)
Prof. Donald Fussell
Prof. Todd Humphreys
2014-04-04
Outline
•  Introduction to Video Stabilization and Rectification
•  Camera-Gyroscope Calibration for Motion Estimation
•  Offline 3D Rotation Smoothing
•  Online (Real-Time) 3D Rotation Smoothing
•  Conclusion
2  Introduction
Video Recording by Handheld Cameras (Smart phones, Tablets…)
Compactness
Everywhere & Anytime
Easy Sharing
Better User Experience (touchscreen)
.
.
.
Poor Video Quality Sometimes …
Camera Motion
CMOS Sensor
Inter-frame Jitter
Rolling Shutter Effect
Handheld Camera Video Stabilization & Rectification
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
3  CMOS Sensor & Rolling Shutter Effect
•  CMOS sensors in almost every
smart phone camera
–  Lower power consumption
–  Faster data throughput
Each row is captured under a different
camera pose if the camera is moving.
Rolling Shutter Effect Under Fast Camera Panning
Image from P.E. Forssén, E. Ringaby, J. Hedborg CVPR 2012 Tutorial
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
4  Rolling Shutter Rectification is Necessary
High frequency camera shake will cause non-rigid rolling shutter distortion
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
5  Video Stabilization & Rectification
•  Stabilization: Removing unwanted jitter (inter-frame correction)
•  Rectification: Removing rolling shutter effect (intra-frame correction)
Camera Pose
Estimation
Camera Pose
Regeneration
Camera pose for
each row
Fix a unique pose/frame
Motion smoothing
Introduction |
Calibration |
Offline Smoothing |
Frame
Synthesis
Apply the correction
Online Smoothing | Conclusions
6  Motion Model Selection
•  2D motion: apparent pixel displacements
Translation
Similarity
Euclidean
Affine
Projective
•  3D real camera motion
Rotation
Degrees of Freedom
Full (Rotation + Translation)
Motion Model Estimation
Complexity
Smoothing
Effectiveness
Correction Complexity
2D
high
low
low
3D Full
high
high
high
3D Rotation
low
(using gyro)
high
low
(projective transform)
•  No approximation in 3D rotational stabilization (proposed method)
-  We are not assuming pure camera rotation
-  Translation is kept as is, and not smoothed
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
7  Thesis Contributions
Œ  Online Camera-Gyroscope Calibration & Synchronization
  Offline 3D Rotation Smoothing
Ž  Online (real-time) 3D Rotation Smoothing
Camera Pose
Estimation
Camera Pose
Regeneration
Frame
Synthesis
Thesis Statement
For handheld cameras with CMOS sensors, videos can be
rectified and then stabilized either online or offline, with the
camera motion estimated directly from gyroscope readings
after effective sensor calibration.
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
8  Outline
•  Introduction to Video Stabilization and Rectification
•  Camera-Gyroscope Calibration for Motion Estimation
•  Offline 3D Rotation Smoothing
•  Online (Real-Time) 3D Rotation Smoothing
•  Conclusion
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
9  Camera Motion (Rotation) Estimation
Vision-based (feature points/pixel intensities)
• Not robust to large moving objects, motion blur, etc.
• Highly complicated for rolling shutter camera
Gyro-based [Karpenko report 2011]
• High sampling frequency (suitable for rolling shutter)
• Independent of video quality
• Sensor calibration & synchronization needed
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
10  Parameters to Estimate
Intrinsic parameters f, cx , cy
bias bg
RS readout time tr
relative orientation qc
delay (use gyro as reference) td
•  Rows are captured sequentially in rolling shutter cameras
feature u
framestamp ti
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
11  Gyro-camera Calibration
•  Previous work on camera-gyro calibration
–  Simultaneous Localization and Mapping (SLAM)
•  Calibrate both gyro & accelerometer
–  Projective matching
•  Assume pure rotation
•  Calibrate only gyro
Image from [Hwangbo IJRR 2011]
Method
Computational Complexity
General Motion?
Online?
SLAM-based
high
✓
✓
Projective matching
low
✗
✗
•  What makes good gyro-camera calibration for video stabilization
–  Online
–  Simple (no need to estimate translation or scene structure)
–  General motion (non-zero translation)
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
12  Coplanarity Constraint
•  How can we get rid of translation?
Epipolar Constraint
Image from [Kneip ECCV 2012]
Coplanarity constraint of global shutter camera
Constraint for rotation only!
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
13  Rolling Shutter Coplanarity Constraint
Assume translation direction is constant
frame i
frame i+1
Pick any reference time
Coplanarity constraint of rolling shutter camera
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
14  EKF-based Online Calibration & Synchronization
•  State Prediction
–  Copy state variables except for gyro bias (random walk process).
•  State Update
–  Use coplanarity constraints as “implicit” measurements
All variables in the state can be related to the constraint
•  Run time: 7fps on 2.3GHz PC
–  Matlab implementation
–  Not necessary to run on every pair of adjacent frames
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
15  Running sequence (random rotation & translation)
•  Estimate Error
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
16  Panning sequence (almost zero translation)
•  Estimate Error
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
17  Outline
•  Introduction to Video Stabilization and Rectification
•  Camera-Gyroscope Calibration for Motion Estimation
•  Offline 3D Rotation Smoothing
•  Online (Real-Time) 3D Rotation Smoothing
•  Conclusion
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
18  3D Rotation Matrix sequence
•  Manifold of 3D Rotation Matrices
–  Special Orthogonal Group
–  An embedded submanifold in R9 (dimension = 3)
Tangent Space
Tangent Vector
Generalization of Straight
Line in Euclidean Space
Geodesic Curve
•  Minimizing Geodesic & Geodesic Distance
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
19  3D Rotation Sequence Smoothing
•  Offline Motion Smoothing
•  Motion Smoothing as Discrete Curve Fitting
Regression on
Regression on
A Balance between Fitting and Smoothness
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
20  Motion Smoothing as Regression
•  First-Order Difference Penalty
Regression using 2D
Motion Model
[Grundmann CVPR 2011]
Local Low-pass
Filtering on Manifold
[Hanning IWMV 2011]
Introduction |
Calibration |
• Convex optimization in Euclidean space
• 2D Motion
• 3D Rotation
• Not globally optimal
• Cannot include additional constraints (unless ad-hoc)
Offline Smoothing |
Online Smoothing | Conclusions
21  Constrained Motion Smoothing
•  Inevitably some pixels are not visible after view change
Correction by
image warping
Hard Constraint: All of the pixels in the cropped
new frame should be visible in the original frame.
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
22  Constrained Manifold Regression
•  Constraint approximation: a geodesic convex set on the manifold
Geodesic balls centered
at the original rotation
•  Constrained Motion Smoothing
min
{Ri }
N
X
1
i=1
2
d2g (R̃i , Ri ) + ↵
N
X1
i=1
1 2
d (Ri , Ri+1 ), s.t. {Ri } 2 ⌦
2 g
⌦ = ⌦1 ⇥ ⌦2 ⇥ · · · ⇥ ⌦N
•  Variable: sequence of rotation matrices
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
23  Optimization
•  Manifold Optimization
Optimization on Euclidean
space [Bertsekas 1999]
• Orthogonality as additional constraints
• Non-convex objective + Non-convex set
(Poor guarantee of convergence or optimality)
Optimization on manifold
(Proposed)
• Geodesic-convex
• Gradient-related iterative algorithms
(Similar convergence properties)
• Gradient & Hessian computation needs
Riemannian geometry
•  From Unconstrained to Constrained
Unconstrained
Constrained on a convex set
steepest gradient descent
gradient projection
scaled gradient descent (Newton,
Quasi-Newton, Congjugate Gradient)
two-metric projection
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
24  Convergence Analysis
•  Convergence comparison for a
video with 478 frames
•  Run time of each iteration of 2metric scaled gradient projection
•  Matlab implementation @2.3GHz
•  ~7.5ms /iteration/frame
~39.0ms /iteration/frame
for gradient projection
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
25  Experimental Results – vs. YouTube
Original Video
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
26  Experimental Results – vs. YouTube
Original Video
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
27  Outline
•  Introduction to Video Stabilization and Rectification
•  Camera-Gyroscope Calibration for Motion Estimation
•  Offline 3D Rotation Smoothing
•  Online (Real-Time) 3D Rotation Smoothing
•  Conclusion
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
28  Online (Real-Time) Motion Smoothing
•  Why online video stabilization
–  Real-time delivery: video conferencing, broadcasting, etc.
–  Improved user experience: WYSIWYG
–  More efficient compression
•  Classical approaches for 2D motion models
–  (1st order low-pass) IIR filtering
–  Kalman filtering with constant-velocity (CV) model
•  Extension to 3D rotation smoothing
–  Euclidean space à SO(3) manifold
–  Ad-hoc projection for black-border constraint
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
29  IIR-like 3D Rotation Smoothing
•  First-Order IIR filtering
SO(3) ✗
SO(3) ✓
Euclidean distance à Geodesic distance
spherical linear
interpolation
(SLERP)
•  Ad-hoc projection
Move closer to the original rotation
if necessary for black-border constraint
1.54ms/frame
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
30  UKF-based 3D Rotation Smoothing
•  Constant-Velocity Model (widely used in target tracking)
smoothed
rotation
angular acceleration
(process noise)
angular
velocity
Dynamic Model
original
rotation
jitter (measurement noise)
Measurement Model
•  Hard to solve on SO(3)
•  Nonlinear on Euclidean space
•  Solved approximately by
unscented Kalman filter (UKF)
6.97ms/frame
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
31  Experimental Results – 2D vs. 3D KF
Original Video
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
32  Experimental Results – 2D vs. 3D IIR
Original Video
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
33  Conclusions
•  Video Stabilization & Rectification for Handheld Cameras
–  CMOS image sensors (rolling shutter effects)
–  Equipped with gyroscopes
Camera-Gyroscope
Calibration
• Online calibration & synchronization
• No need to estimate translation
Offline 3D Rotation
Smoothing
• Stabilization as regression on manifold
• Convex approximation of constraint
• Manifold optimization
Online 3D Rotation
Smoothing
• IIR-like smoothing
• UKF (unscented Kalman Filter)-based smoothing
• Ad-hoc projection for constraint
Introduction |
Calibration |
Offline Smoothing |
Online Smoothing | Conclusions
34  Thanks!
35  References
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
[Bertsekas 1999] D. Bertsekas, Nonlinear Programming, Athena Scientific, 1999.
[Civera JFR 2010] J. Civera, O.G. Grasa, A.J. Davison, and J.M.M. Montiel, “1-point RANSAC for
extended Kalman filtering: Application to real-time structure from motion and visual odometry,”
Journal of Field Robotics, vol. 27, issue 5, pp. 609-631, Sep. 2010
[Gallier 2012] J. Gallier, Notes on Differential Geometry and Lie Groups. University of
Pennsylvania, 2012.
[Grundmann CVPR 2011] M. Grundmann, V. Kwatra, and I. Essa, “Auto-directed video stabilization
with robust L1 optimal camera paths,” in Proc. IEEE Conf. on Computer Vision and Pattern
Recognition, Jun. 2011.
[Hanning IMWV 2011] G. Hanning, N. Forslow, P.-E. Forssen, E. Ringaby, D. Tornqvist, and J.
Callmer, “Stabilizing cell phone video using inertial measurement sensors,” in Proc. IEEE Intl.
Workshop on Mobile Vision, Nov. 2011.
[Hwangbo IJRR 2011] M. Hwangbo, J.-S. Kim, and T. Kanade, “Gyro-aided freature tracking for a
moving camera: fusion, auto-calibration and GPU implementation,” Intl. Journal Robotics Research,
vol. 30, no. 14, pp. 1755-1774, 2011.
[Karpenko report 2011] A. Karpenko, D. Jacobs, J. Baek, and M. Levoy, “Digital video stabilization
and rolling shutter correction using gyroscopes,” Tech. Rep., Stanford University, Mar. 2011.
[Kneip ECCV 2012] L. Kneip, R. Siegwart, and M. Pollefeys, “Finding the exact rotation between
two images independently of the translation,” in Proc. European Conf. Computer Vision, Oct. 2012.
[Morimoto CVPR 1997] C. Morimoto, and R. Chellapa, “Fast 3D stabilization and mosaic
construction,” in Proc. IEEE Conf. on Computer Vision and Pattern Recognition, Jun. 1997.
[Pollefeys IJCV 2004] M. Pollefeys, L. Van Gool, M. Vergauwen, F. Verbiest, K. Cornelis, J. Tops, and
R. Koch, “Visual modeling with a hand-held camera,” Intl. Journal of Computer Vision, vol. 59, no.
3, pp. 207-232, 2004.
36  Personal Publications
• 
Journal Papers
–  C. Jia and B.L. Evans, “Real-time Motion Smoothing for Video Stabilization via Constrained
Multiple-Model Estimation”, IEEE Trans. on Circuits and Systems for Video Technology, to be
submitted.
–  C. Jia and B.L. Evans, “Online Camera-Gyroscope Auto-Calibration for Cellphones”, IEEE Trans.
on Image Processing, In revision.
–  C. Jia and B.L. Evans, “Constrained 3D Rotation Smoothing via Global Manifold Regression for
Video Stabilization”, IEEE Trans. on Signal Processing, Accepted with minor revision.
–  V. Raveendran, P. Bhamidipati, X. Luo, X. Huang and C. Jia, “Mobile Multipath Cooperative
Network for Real-time Streaming”, Signal Processing: Image Communication, Vol 27, Feb 2012
• 
Conference Papers
–  C. Jia, Z. Sinno and B.L. Evans, “Real-Time 3D Rotation Smoothing for Video Stabilization”,
Asilomar Conf. on Signals, Systems, and Computers, Nov. 2-5, 2014, Pacific Grove, CA, to be
submitted.
–  C. Jia and B.L. Evans, “Online Calibration and Synchronization of Cellphone Camera and
Gyroscope”, Proc. IEEE Global Conference on Signal and Information Processing (GlobalSIP),
Dec. 3-5, 2013, Austin, TX.
–  C. Jia and B.L. Evans, “3D Rotational Video Stabilization using Manifold Optimization”, Proc.
IEEE Int. Conf. on Acoustics, Speech, and Signal Processing (ICASSP), May. 26-31,2013,
Vancouver, Canada. (Google Travel Grant Winner)
–  C. Jia and B.L. Evans, “Probabilistic 3D Motion Estimation for Rolling Shutter Video
Rectification from Visual and Inertial Measurements”, Proc. IEEE Int. Workshop on Multimedia
Signal Processing (MMSP), Sep. 17-20, 2012, Banff, Canada. (Top 10% Best Paper Award)
–  C. Jia and B.L. Evans, “Patch-based Image Deconvolution via Joint Modeling of Sparse Priors”,
Proc. IEEE Int. Conf. on Image Processing (ICIP), Sep. 11-14, 2011, Brussels, Belgium.
37  
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertising