Abstract

CMOS sensors employ a row-wise acquisition mechanism while imaging a scene, which can result in undesired motion artifacts known as rolling shutter (RS) distortions in the captured image. Existing single image RS rectification methods attempt to account for these distortions by using either algorithms tailored for a specific class of scenes that warrants information of intrinsic camera parameters or a learning-based framework with known ground truth motion parameters. In this paper, we propose an end-to-end deep neural network for the challenging task of single image RS rectification. Our network consists of a motion block, a trajectory module, a row block, an RS rectification module, and an RS regeneration module (which is used only during training). The motion block predicts the camera pose for every row of the input RS distorted image, while the trajectory module fits estimated motion parameters to a third-order polynomial. The row block predicts the camera motion that must be associated with every pixel in the target, i.e., RS rectified image. Finally, the RS rectification module uses motion trajectory and the output of a row block to warp the input RS image to arrive at a distortion-free image. For faster convergence during training, we additionally use an RS regeneration module that compares the input RS image with the ground truth image distorted by estimated motion parameters. The end-to-end formulation in our model does not constrain the estimated motion to ground truth motion parameters, thereby successfully rectifying the RS images with complex real-life camera motion. Experiments on synthetic and real datasets reveal that our network outperforms prior art both qualitatively and quantitatively.

© 2020 Optical Society of America

Deep coded exposure: end-to-end co-optimization of flutter shutter and deblurring processing for general motion blur removal

Zhihong Zhang, Kaiming Dong, Jinli Suo, and Qionghai Dai
Photon. Res. 11(10) 1678-1686 (2023)

Shuffled rolling shutter for snapshot temporal imaging

Esteban Vera, Felipe Guzmán, and Nelson Díaz
Opt. Express 30(2) 887-901 (2022)

Deep learning for signal clock and exposure estimation in rolling shutter optical camera communication

Cristo Jurado-Verdu, Victor Guerra, Jose Rabadan, and Rafael Perez-Jimenez
Opt. Express 30(12) 20261-20277 (2022)

Compressive temporal imaging using a rolling shutter camera array

Felipe Guzmán, Pablo Meza, and Esteban Vera
Opt. Express 29(9) 12787-12800 (2021)

100,000 frames-per-second compressive imaging with a conventional rolling-shutter camera by random point-spread-function engineering

Gil Weinberg and Ori Katz
Opt. Express 28(21) 30616-30625 (2020)