Skip to main content
SpringerLink
Log in

Rolling Shutter Camera: Modeling, Optimization and Learning

  • Review
  • Published:
Machine Intelligence Research Aims and scope Submit manuscript

Abstract

Most modern consumer-grade cameras are often equipped with a rolling shutter mechanism, which is becoming increasingly important in computer vision, robotics and autonomous driving applications. However, its temporal-dynamic imaging nature leads to the rolling shutter effect that manifests as geometric distortion. Over the years, researchers have made significant progress in developing tractable rolling shutter models, optimization methods, and learning approaches, aiming to remove geometry distortion and improve visual quality. In this survey, we review the recent advances in rolling shutter cameras from two aspects of motion modeling and deep learning. To the best of our knowledge, this is the first comprehensive survey of rolling shutter cameras. In the part of rolling shutter motion modeling and optimization, the principles of various rolling shutter motion models are elaborated and their typical applications are summarized. Then, the applications of deep learning in rolling shutter based image processing are presented. Finally, we conclude this survey with discussions on future research directions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. R. Hartley, A. Zisserman. Multiple View Geometry in Computer Vision, 2nd ed., Cambridge, UK: Cambridge University Press, 2003.

    MATH  Google Scholar 

  2. Y. C. Dai. H. D. Li. L. Kneip. Rolling shutter camera relative pose: Generalized epipolar geometry. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA, pp. 4132–4140, 2016. DOI: https://doi.org/10.1109/CVPR.2016.448.

  3. C. Albl, Z. Kukelova, T. Pajdla. R6P-rolling shutter absolute pose problem. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, pp. 2292–2300. 2015. DOI: https://doi.org/10.1109/CVPR.2015.7298842.

  4. Y. Z. Lao, O. Ait-Aider. Rolling shutter homography and its applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43. no. 8. pp. 2780–2793. 2021. DOI: https://doi.org/10.1109/TPAMI.2020.2977644.

    Article  Google Scholar 

  5. B. Fan, Y. C. Dai. Inverting a rolling shutter camera: Bring rolling shutter images to high framerate global shutter video. In Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, Montreal, Canada, pp. 4208–4217. 2021}. DOI: https://doi.org/10.1109/ICCV48922.2021.00419

    Google Scholar 

  6. B. B. Zhuang, Q. H. Tran, P. Ji, L. F. Cheong, M. Chandraker. Learning structure-and-motion-aware rolling shutter correction. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Long Beach, USA, pp. 4546–4555, 2019. DOI: https://doi.org/10.1109/CV-PR.2019.00468.

    Google Scholar 

  7. Z. H. Zhong, M. D. Cao, X. Sun, Z. R. Wu, Z. Y. Zhou, Y. Q. Zheng, S. Lin, I. Sato. Bringing rolling shutter images alive with dual reversed distortion. In Proceedings of the 17th European Conference on Computer Vision, Springer, Tel Aviv, Israel, pp. 233–249, 2022. DOI: https://doi.org/10.1007/978-3-031-20071-714.

    Google Scholar 

  8. Y. Z. Lao, O. Ait-Aider, A. Bartoli. Solving rolling shutter 3D vision problems using analogies with non-rigidity. International Journal of Computer Vision, vol 129. no. 1, pp. 100–122, 2021. DOI: https://doi.org/10.1007/s11263-020-01368-1.

    Article  MathSciNet  MATH  Google Scholar 

  9. J. H. Kim, C. Cadena, I. Reid. Direct semi-dense SLAM for rolling shutter cameras. In Proceedings of IEEE International Conference on Robotics and Automation, Stockholm, Sweden, pp. 1308–1315. 2016. DOI: https://doi.org/10.1109/ICRA.2016.7487263.

  10. D. Schubert, N. Demmel, L. von Stumberg, V. Usenko, D. Cremers. Rolling-shutter modelling for direct visual-inertial odometry. In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, Macau, China, pp. 2462–2469. 2019. DOI: https://doi.org/10.1109/IROS40897.2019.8968539.

    Google Scholar 

  11. B. B. Zhuang, L. F. Cheong, G. H. Lee. Rolling-shutter-aware differential SfM and image rectification. In Proceedings of IEEE International Conference on Computer Vision, Venice, Italy, pp. 948–956. 2017. DOI: https://doi.org/10.1109/ICCV.2017.108.

  12. O. Saurer, M. Pollefeys. G. H. Lee. Sparse to dense 3D reconstruction from rolling shutter images. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 3337–3345. 2016. DOI: https://doi.org/10.1109/CVPR.2016.363.

  13. S. Im, H. Ha, G. Choe, H. G. Jeon, K. Joo, I. S. Kweon. Accurate 3D reconstruction from small motion clip for rolling shutter cameras. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41. no. 4. pp. 775–787. 2019. DOI: https://doi.org/10.1109/TPAMI.2018.2819679.

    Article  Google Scholar 

  14. J. Hedborg, P. E. Forssen, M. Felsberg, E. Ringaby. Rolling shutter bundle adjustment. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Providence. USA, pp. 1434–1441. 2012. DOI: https://doi.org/10.1109/CVPR.2012.6247831.

  15. M. Meingast, C. Geyer, S. Sastry. Geometric models of rolling-shutter cameras. [Online], Available: https://arxiv.org/abs/0503076, 2005.

  16. B. Fan, Y. C. Dai, M. Y. He. SUNet: Symmetric undistortion network for rolling shutter correction. In Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, Montreal, Canada, pp. 4521–4530, 2021. DOI: https://doi.org/10.1109/ICCV48922.2021.00450.

    Google Scholar 

  17. O. Saurer, K. Köser, J. Y. Bouguet, M. Pollefeys. Rolling shutter stereo. In Proceedings of IEEE International Conference on Computer Vision, Sydney, Australia, pp. 465–472, 2013. DOI: https://doi.org/10.1109/ICCV.2013.64.

  18. O. Saurer, M. Pollefeys, G. H. Lee. A minimal solution to the rolling shutter pose estimation problem. In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, Hamburg, Germany, pp. 1328–1334, 2015. DOI: https://doi.org/10.1109/IROS.2015.7353540.

    Google Scholar 

  19. P. E. Forssén, E. Ringaby. Rectifying rolling shutter video from hand-held devices. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, USA, pp. 507–514, 2010. DOI: https://doi.org/10.1109/CVPR.2010.5540173

  20. V. Rengarajan, A. N. Rajagopalan, R. Aravind. From bows to arrows: Rolling shutter rectification of urban scenes. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 2773–2781, 2016. DOI: https://doi.org/10.1109/CVPR.2016.303.

  21. L. Magerand, A. Bartoli, O. Ait-Aider, D. Pizarro. Global optimization of object pose and motion from a single rolling shutter image with automatic 2D-3D matching. In Proceedings of the 12th European Conference on Computer Vision, Springer, Florence, Italy, pp. 456–469, 2012. DOI: https://doi.org/10.1007/978-3-642-33718-5_33.

    Google Scholar 

  22. K. Wang, B. Fan, Y. C. Dai. Relative pose estimation for stereo rolling shutter cameras. In Proceedings of IEEE International Conference on Image Processing, Abu Dhabi, UAE, pp. 463–467, 2020. DOI: https://doi.org/10.1109/ICIP40778.2020.9191254.

  23. O. Ait-Aider, N. Andreff, J. M. Lavest, P. Martinet. Simultaneous object pose and velocity computation using a single view from a rolling shutter camera. In Proceedings of the 9th European Conference on Computer Vision, Springer, Graz, Austria, pp. 56–68, 2006. DOI: https://doi.org/10.1007/117440475.

    Google Scholar 

  24. O. Ait-Aider, F. Berry. Structure and kinematics triangulation with a rolling shutter stereo rig. In Proceedings of the 12th International Conference on Computer Vision, IEEE, Kyoto, Japan, pp. 1835–1840, 2009. DOI: https://doi.org/10.1109/ICCV.2009.5459408.

    Google Scholar 

  25. C. Albl, Z. Kukelova, T. Pajdla. Rolling shutter absolute pose problem with known vertical direction In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 3355–3363, 2016 DOI: https://doi.org/10.1109/CVPR.2016.365

  26. C. Albl, A Sugimoto, T Pajdla. Degeneracies in rolling shutter SfM In Proceedings of the 14th European Conference on Computer Vision, Springer, Amsterdam, The Netherlands, pp. 36–51, 2016 DOI: https://doi.org/10.1007/978-3-319-46454-13.

    Google Scholar 

  27. Z. Kukelova, C. Albl, A Sugimoto, T. Paadla Linear solution to the minimal absolute pose rolling shutter problem. In Proceedings of the 14th Asian Conference on Computer Vision, Springer, Perth, Australia, pp. 265–280, 2019. DOI: https://doi.org/10.1007/978-3-030-20893-6_17.

    Google Scholar 

  28. C. Albl, Z. Kukelova, V. Larsson, T. Pajdla. Rolling shutter camera absolute pose. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 6, pp. 1439–1452, 2020. DOI: https://doi.org/10.1109/TPAMI.2019.2894395.

    Article  Google Scholar 

  29. Z. Kukelova, C. Albl, A. Sugimoto, K. Schindler, T. Pajdla. Minimal rolling shutter absolute pose with unknown focal length and radial distortion. In Proceedings of the 16th European Conference on Computer Vision, Springer, Glasgow, UK, pp. 698–714, 2020. DOI: https://doi.org/10.1007/978-3-030-58558-7_41.

    Google Scholar 

  30. C. Albl, Z. Kukelova, V. Larsson, M. Polic, T. Pajdla, K. Schindler. From two rolling shutters to one global shutter. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Seattle, USA, pp. 2502–2510, 2020. DOI: https://doi.org/10.1109/CVPR42600.2020.00258.

    Google Scholar 

  31. K. Wang, C. H. Liu, K. X. Wang, S. J. Shen. Depth estimation under motion with single pair rolling shutter stereo images. IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 3160–3167, 2021. DOI: https://doi.org/10.1109/LRA.2021.3063695.

    Article  Google Scholar 

  32. E. Ringaby, P. E. Forssén. Efficient video rectification and stabilisation for cell-phones. International Journal of Computer Vision, vol. 96, no. 3, pp. 335–352, 2012. DOI: https://doi.org/10.1007/s11263-011-0465-8.

    Article  Google Scholar 

  33. E. Ito, T. Okatani. Self-calibration-based approach to critical motion sequences of rolling-shutter structure from motion. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, pp. 4512–4520, 2017. DOI: https://doi.org/10.1109/CVPR.2017.480.

  34. Y. Z. Lao, O. Ait-Aider. A robust method for strong rolling shutter effects correction using lines with automatic feature selection. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 4795–4803, 2018. DOI: https://doi.org/10.1109/CV-PR.2018.00504.

    Google Scholar 

  35. P. Purkait, C. Zach. Minimal solvers for monocular rolling shutter compensation under ackermann motion. In Proceedings of IEEE Winter Conference on Applications of Computer Vision, Lake Tahoe, USA, pp. 903–911, 2018. DOI: https://doi.org/10.1109/WACV.2018.00104.

  36. C. R. Lee, J. H. Yoon, M. G. Park, K. J. Yoon. Gyroscope-aided relative pose estimation for rolling shutter cameras, [Online], Available: https://arxiv.org/abs/1900.06770, 2019.

  37. J. Hedborg, E. Ringaby, P. E. Forssén, M. Felsberg. Structure and motion estimation from rolling shutter video. In Proceedings of IEEE International Conference on Computer Vision Workshops, Barcelona, Spain, pp. 17–23, 2011. DOI: https://doi.org/10.1109/ICCVW.2011.6130217.

  38. B. B. Zhuang, Q. H. Tran. Image stitching and rectification for hand-held cameras. In Proceedings of the 16th European Conference on Computer Vision, Springer, Glasgow, UK, pp. 243–260, 2020. DOI: https://doi.org/10.1007/978-3-030-58571-6_15.

    Google Scholar 

  39. S. Im, H. Ha, G. Choe, H. G. Jeon, K. Joo, I. S. Kweon. High quality structure from small motion for rolling shutter cameras. In Proceedings of IEEE International Conference on Computer Vision, Santiago, Chile, pp. 837–845, 2015. DOI: https://doi.org/10.1109/ICCV.2015.102.

  40. B. Fan, Y. C. Dai, K. Wang. Rolling-shutter-stereo-aware motion estimation and image correction. Computer Vision and Image Understanding, vol. 213, Article number 103296, 2021. DOI: https://doi.org/10.1016/j.cviu.2021.103296.

  41. B. Fan, Y. C. Dai, Z. Y. Zhang, K. Wang. Differential SfM and image correction for a rolling shutter stereo rig. Image and Vision Computing, vol. 124, Article number 104492, 2022. DOI: https://doi.org/10.1016/j.imavis.2022.104492.

  42. Y. Z. Lao, O. Ait-Aider, H. Araujo. Robustified structure from Motion with rolling-shutter camera using straightness constraint. Pattern Recognition Letters, vol. 111, pp. 1–8, 2018. DOI: https://doi.org/10.1016/j.patrec.2018.04.004.

    Article  Google Scholar 

  43. B. Triggs, P. F. McLauchlan, R. I. Hartley, A. W. Fitzgibbon. Bundle adjustment–A modern synthesis. In Proceedings of the International Workshop on Vision Algorithms, Springer, Corfu, Greece, pp. 298–372, 2000. DOI: https://doi.org/10.1007/3-540-44480-7_21.

    Chapter  Google Scholar 

  44. L. Oth, P. Furgale, L. Kneip, R. Siegwart. Rolling shutter camera calibration. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Portland, USA, pp. 1360–1367, 2013. DOI: https://doi.org/10.1109/CVPR.2013.179.

  45. B. Y. Liao, D. L. Qu, Y. F. Xue, H. Q. Zhang, Y. Z. Lao. Revisiting rolling shutter bundle adjustment: Toward accurate and fast solution, [Online], Available: https://arx-iv.org/abs/2209.08503, 2022.

  46. S. Baker, D. Scharstein, J. P. Lewis, S. Roth, M. J. Black, R. Szeliski. A database and evaluation methodology for optical flow. International Journal of Computer Vision, vol. 92, no. 1, pp. 1–31, 2011. DOI: https://doi.org/10.1007/s11263-010-0390-2.

    Article  Google Scholar 

  47. H. C. Longuet-Higgins, K. Prazdny. The interpretation of a moving retinal image. Proceedings of the Royal Society B: Biological Sciences, vol. 208, no. 1173, pp. 385–397, 1980. DOI: https://doi.org/10.1098/rspb.1980.0057.

    Google Scholar 

  48. B. Fan, Y. C. Dai, Z. Y. Zhang, M. Y. He. Fast and robust differential relative pose estimation with radial distortion. IEEE Signal Processing Letters, vol. 29, pp. 294–298, 2022. DOI: https://doi.org/10.1109/LSP.2021.3134593.

    Article  Google Scholar 

  49. V. Rengarajan, Y. Balaji, A. N. Rajagopalan. Unrolling the shutter: CNN to correct motion distortions. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, pp. 2345–2353, 2017. DOI: https://doi.org/10.1109/CVPR.2017.252.

  50. S. C. Su, W. Heidrich. Rolling shutter motion deblurring. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, pp. 1529–1537, 2015. DOI: https://doi.org/10.1109/CVPR.2015.7298760.

  51. P. Purkait, C. Zach, A. Leonardis. Rolling shutter correction in Manhattan world. In Proceedings of IEEE International Conference on Computer Vision, Venice, Italy, pp. 882–890, 2017. DOI: https://doi.org/10.1109/ICCV.2017.101.

  52. A. Patron-Perez, S. Lovegrove, G. Sibley. A spline-based trajectory representation for sensor fusion and rolling shutter cameras. International Journal of Computer Vision, vol. 113, no. 3, pp. 208–219, 2015. DOI: https://doi.org/10.1007/s11263-015-0811-3.

    Article  Google Scholar 

  53. C. Kerl, J. Stückler, D. Cremers. Dense continuous-time tracking and mapping with rolling shutter RGB-D cameras. In Proceedings of IEEE International Conference on Computer Vision, Santiago, Chile, pp. 2264–2272, 2015. DOI: https://doi.org/10.1109/ICCV.2015.261.

  54. F. Bai, A. Sengupta, A. Bartoli. Scanline homographies for rolling-shutter plane absolute pose. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, New Orleans, USA, pp. 8983–8992, 2022. DOI: https://doi.org/10.1109/CVPR52688.2022.00879.

    Google Scholar 

  55. B. Vandeportaele, P. A. Gohard, M. Devy, B. Coudrin. Pose interpolation for rolling shutter cameras using non uniformly time-sampled B-splines. In Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, SciTePress, Porto, Portugal, pp. 286–293, 2017. DOI: https://doi.org/10.5220/0006171802860293.

    Chapter  Google Scholar 

  56. S. Lovegrove, A. Patron-Perez, G. Sibley. Spline fusion: A continuous-time representation for visual-intertial fusion with application to rolling shutter cameras. In Proceedings of the British Machine Vision Conference, BMVA Press, Bristol, UK, pp. 93.1–93.12, 2013. DOI: https://doi.org/10.5244/C.27.93.

    Google Scholar 

  57. J. Mo, J. Islam, J. Sattar. Learning rolling shutter correction from real data without camera motion assumption, [Online], Available: https://arxiv.org/abs/2011.03106, 2020.

  58. J. Z. Huai, Y. K. Zhuang, Y. Lin, G. Jozkow, Q. C. Yuan, D. Chen. Continuous-time spatiotemporal calibration of a rolling shutter camera-IMU system. IEEE Sensors Journal, vol. 22, no. 8, pp. 7920–7930, 2022. DOI: https://doi.org/10.1109/JSEN.2022.3152572.

    Article  Google Scholar 

  59. X. L. Lang, J. J. Lv, J. X. Huang, Y. K. Ma, Y. Liu, X. X. Zuo. Ctrl-VIO: Continuous-time visual-inertial odometry for rolling shutter cameras. IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 11537–11544, 2022. DOI: https://doi.org/10.1109/LRA.2022.3202349.

    Article  Google Scholar 

  60. E. B. Dam, M. Koch, M. Lillholm. Quaternions, Interpolation and Animation, Technical Report DIKU-TR-98/5, Department of Computer Science, University of Copenhagen, Copenhagen, Denmark, 1998.

    Google Scholar 

  61. C. H. Zhao, B. Fan, J. W. Hu, Q. Pan, Z. Xu. Homography-based camera pose estimation with known gravity direction for UAV navigation. Science China Information Sciences, vol. 64, no. 1, Article number 112204, 2021. DOI: https://doi.org/10.1007/s11432-019-2690-0.

  62. C. H. Zhao, B. Fan, J. W. Hu, L. M. Tian, Z. Y. Zhang, S. J. Li, Q. Pan. Pose estimation for multi-camera systems. In Proceedings of IEEE International Conference on Unmanned Systems, Beijing, China, pp. 533–538, 2017. DOI: https://doi.org/10.1109/ICUS.2017.8278403.

  63. S. C. Zhou, R. Yan, J. X. Li, Y. K. Chen, H. J. Tang. A brain-inspired SLAM system based on orb features. International Journal of Automation and Computing, vol. 14, no. 5, pp. 564–575, 2017. DOI: https://doi.org/10.1007/s11633-017-1090-y.

    Article  Google Scholar 

  64. C. L. Wang, T. M. Wang, J. H. Liang, Y. C. Zhang, Y. Zhou. Bearing-only visual SLAM for small unmanned aerial vehicles in GPS-denied environments. International Journal of Automation and Computing, vol. 10, no. 5, pp. 387–396, 2013. DOI: https://doi.org/10.1007/s11633-013-0735-8.

    Article  Google Scholar 

  65. Y. Z. Lao, O. Ait-Aider, A. Bartoli. Rolling shutter pose and ego-motion estimation using shape-from-template. In Proceedings of the 15th European Conference on Computer Vision, Springer, Munich, Germany, pp. 477–492, 2018. DOI: https://doi.org/10.1007/978-3-030-01216-8_29.

    Google Scholar 

  66. O. Ait-Aider, A. Bartoli, N. Andreff. Kinematics from lines in a single rolling shutter image. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, USA, 2007. DOI: https://doi.org/10.1109/CVPR.2007.383119.

  67. R. M. Haralick, D. Lee, K. Ottenburg, M. Nolle. Analysis and solutions of the three point perspective pose estimation problem. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Maui, USA, pp. 592–598, 1991. DOI: https://doi.org/10.1109/CVPR.1991.139759.

  68. M. Panda, B. Das, B. Subudhi, B. B. Pati. A comprehensive review of path planning algorithms for autonomous underwater vehicles. International Journal of Automation and Computing, vol. 17, no. 3. pp. 321–352. 2020. DOI: https://doi.org/10.1007/s11633-019-1204-9.

    Article  Google Scholar 

  69. Y. Yang, F. Qiu, H. Li, L. Zhang, M. L. Wang, M. Y. Fu. Large-scale 3D semantic mapping using stereo vision. International Journal of Automation and Computing, vol. 15, no. 2. pp. 194–206. 2018. DOI: https://doi.org/10.1007/s11633-018-1118-y.

    Article  Google Scholar 

  70. Q. Qi, Q. D. Li, Y. Q. Cheng, Q. Q. Hong. Skeleton marching-based parallel vascular geometry reconstruction using implicit functions. International Journal of Automation and Computing, vol. 17, no. 1. pp. 30–43. 2020. DOI: https://doi.org/10.1007/s11633-019-1189-4.

    Article  Google Scholar 

  71. B. Fan, K. Wang, Y. C. Dai, M. Y. He. RS-DPSNet: Deep plane sweep network for rolling shutter stereo images. IEEE Signal Processing Letters, vol. 28, pp. 1550–1554, 2021. DOI: https://doi.org/10.1109/LSP.2021.3099350.

    Article  Google Scholar 

  72. P. D. Liu, Z. P. Cui, V. Larsson, M. Pollefeys. Deep shutter unrolling network. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Seattle, USA, pp. 5940–5948. 2020. DOI: https://doi.org/10.1109/CVPR42600.2020.00598.

  73. S. Vasu, M. R. M. Mohan, A. N. Rajagopalan. Occlusion-aware rolling shutter rectification of 3D scenes. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 636–645. 2018. DOI: https://doi.org/10.1109/CVPR.2018.00073.

  74. H. C. Wu, L. Xiao, Z. H. Wei. Simultaneous video stabilization and rolling shutter removal. IEEE Transactions on Image Processing, vol. 30. pp. 4637–4652. 2021. DOI: https://doi.org/10.1109/TIP.2021.3073865.

    Article  Google Scholar 

  75. Z. H. Zhong, Y. Q. Zheng, I. Sato. Towards rolling shutter correction and deblurring in dynamic scenes. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Nashville, USA, pp. 9215–9224. 2021. DOI: https://doi.org/10.1109/CVPR46437.2021.00910.

  76. K. Praveen, T Lokesh Kumar, A. N. Rajagopalan. Deep network for rolling shutter rectification, [Online], Available: https://arxiv.org/abs/2112.06170. 2021.

  77. B. Fan, Y. C. Dai, Z. Y. Zhang, Q. Liu, M. Y. He. Context-aware video reconstruction for rolling shutter cameras. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, New Orleans, USA, pp. 17551–17561. 2022. DOI: https://doi.org/10.1009/CVPR52688.2022.01705.

  78. B. Fan, Y. C. Dai, H. D. Li. Rolling shutter inversion: Bring rolling shutter images to high framerate global shutter video. IEEE Transactions on Pattern Analysis and Machine Intelligence, to be published. DOI: https://doi.org/10.1109/TPAMI.2022.3212912.

  79. M. D. Cao, Z. H. Zhong, J. H. Wang, Y. Q. Zheng, Y. J. Yang. Learning adaptive warping for real world rolling shutter correction. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, New Orleans, USA, pp. 17764–17772. 2022. DOI: https://doi.org/10.1109/CVPR52688.2022.01726.

  80. C. K. Liang, L. W. Chang, H. H. Chen. Analysis and compensation of rolling shutter effect. IEEE Transactions on Image Processing, vol. 17. no. 8. pp. 1323–1330. 2008. DOI: https://doi.org/10.1109/TIP.2008.925384.

    Article  MathSciNet  Google Scholar 

  81. S. Baker, E. Bennett, S. B. Kang, R. Szeliski. Removing rolling shutter wobble. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, USA, pp. 2392–2399. 2010. DOI: https://doi.org/10.1109/CV-PR.2010.5539932.

  82. M. Grundmann, V. Kwatra, D. Castro, I. Essa. Calibration-free rolling shutter removal. In Proceedings of IEEE International Conference on Computational Photograph, Seattle, USA, pp. 1–8. 2012. DOI: https://doi.org/10.1109/ICCPhot.2012.6215213.

  83. A. Punnappurath, V. Rengarajan, A. N. Rajagopalan. Rolling shutter super-resolution. In Proceedings of IEEE International Conference on Computer Vision, Santiago, Chile, pp. 558–566. 2015. DOI: https://doi.org/10.1109/ICCV.2015.71.

  84. M. Meilland, T. Drummond, A. I. Comport. A unified rolling shutter and motion blur model for 3D visual registration. In Proceedings of IEEE International Conference on Computer Vision, Sydney, Australia, pp. 2016–2023, 2013. DOI: https://doi.org/10.1109/ICCV.2013.252.

  85. J. Park, K. Ko, C Lee, C. S. Kim. BMBC: Bilateral motion estimation with bilateral cost volume for video interpolation. In Proceedings of the 16th European Conference on Computer Vision, Springer, Glasgow, UK, pp. 109–125. 2020. DOI: https://doi.org/10.1007/978-3-030-58568-6_7.

  86. W. B. Bao, W. S. Lai, C. Ma, X. Y. Zhang, Z. Y. Gao, M. H. Yang. Depth-aware video frame interpolation. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Long Beach, USA, pp. 3698–3707. 2019. DOI: https://doi.org/10.1109/CVPR.2019.00382.

  87. R. Zhang, P. Isola, A. A. Efros, E. Shechtman, O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 586–595. 2018. DOI: https://doi.org/10.1109/CVPR.2018.00068.

  88. A. Dosovitskiy, G. Ros, F. Codevilla, A. M. López, V. Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning, Mountain View, USA, vol. 78, pp. 1–16. 2017.

    Google Scholar 

  89. Z. X. Wang, X. Ji, J. B. Huang, S. Satoh, X. Zhou, Y. Q. Zheng. Neural global shutter: Learn to restore video from a rolling shutter camera with global reset feature. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, New Orleans, USA, pp. 17773–17782. 2022. DOI: https://doi.org/10.1109/CVPR52688.2022.01727.

  90. X. Y. Zhou. P. Q. Duan, Y. Ma, B. X. Shi. EvUnroll: Neuromorphic events based rolling shutter image correction. In Proceedings of IEEE/CVF conference on Computer Vision and Pattern Recognition, New Orleans, USA, pp. 17754–17763. 2022. DOI: https://doi.org/10.1109/CVPR52688.2022.01725.

  91. Y. H. Hu, S. C. Liu, T. Delbruck. V2E: From video frames to realistic DVS events. In Proceedings of IEEE/CVF conference on Computer Vision and Pattern Recognition Workshops, IEEE, Nashville, USA, pp. 1312–1321. 2021. DOI: https://doi.org/10.1109/CVPRW53098.2021.00144.

  92. J. H. Kim. Y. Latif, I. Reid. RRD-SLAM: Radial-distorted rolling-shutter direct SLAM. In Proceedings of IEEE International Conference on Robotics and Automation, Singapore, pp. 5148–5154. 2017. DOI: https://doi.org/10.1109/ICRA.2017.7989602.

  93. D. Schubert, N. Demmel, V. Usenko, J. Stückler, D. Cremers. Direct sparse odometry with rolling shutter. In Proceedings of the 15th European Conference on Computer Vision, Springer, Munich, Germany, pp. 699–714, 2018. DOI: https://doi.org/10.1007/978-3-030-01237-3_42.

    Google Scholar 

  94. J. Mo, J. Islam, J. Sattar. IMU-assisted learning of single-view rolling shutter correction. In Proceedings of the Conference on Robot Learning, London, UK, vol. 164, pp. 861–870, 2021.

    Google Scholar 

  95. S. Tourani, S. Mittal, A. Nagariya, V. Chari, M. Krishna. Rolling shutter and motion blur removal for depth cameras. In Proceedings of IEEE International Conference on Robotics and Automation, Stockholm, Sweden, pp. 5098–5105, 2016. DOI: https://doi.org/10.1109/ICRA.2016.7487715.

Download references

Acknowledgements

This work was supported in part by National Natural Science Foundation of China (Nos. 62271410, 61901387 and 62001394), the Fundamental Research Funds for the Central Universities, China, and the Innovation Foundation for Doctor Dissertation of Northwestern Polytechnical University, China (No. CX2022046).

Author information

Authors and Affiliations

  1. School of Electronics and Information, Northwestern Polytechnical University, Xi’an, 710129, China

    Bin Fan, Yuchao Dai & Mingyi He

Authors
  1. Bin Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuchao Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mingyi He
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yuchao Dai.

Ethics declarations

The authors declared that they have no conflicts of interest to this work.

Additional information

Colored figures are available in the online version at https://link.springer.com/journal/11633

Bin Fan received the B. Sc. degree in statistics and the M. Eng. degree in control science and engineering from Northwestern Polytechnical University, China in 2016 and 2019, respectively. He is currently a Ph. D. degree candiclate in information and communication engineering with School of Electronics and Information, Northwestern Polytechnical University (NPU), China. He was selected to the CVPR 2022 Doctoral Consortium (the only one among Chinese universities). He co-organized the ACCV 2022 tutorial on the topic of rolling shutter cameras. He has published some papers in TPAMI, CVPR, ICCV, TCSVT, CVIU, IVC, etc.

His research interests include computer vision, image processing, 3D reconstruction, and deep learning, especially regarding the rolling shutter camera.

Yuchao Dai received the B. Eng., M. Eeg. and Ph. D. degrees all in signal and information processing from Northwestern Polytechnical University, China in 2005, 2008 and 2012, respectively. He is currently a professor with School of Electronics and Information, Northwestern Polytechnical University (NPU), China. He was an ARC DECRA fellow with the research school of engineering at Australian National University, Australia. He won the Best Paper Award in IEEE CVPR 2012, the Best Paper Award Nominee at IEEE CVPR 2020, the DSTO Best Fundamental Contribution to Image Processing Paper Prize at DICTA 2014, the Best Algorithm Prize in NRSFM Challenge at CVPR 2017, the Best Student Paper Prize at DICTA 2017, the Best Deep/Machine Learning Paper Prize at APSIPA ASC 2017. He served as Area Chair in CVPR, ICCV, ACM MM, ACCV, etc. He serves as Publicity Chair in ACCV 2022.

His research interests include structure from motion, multiview geometry, low-level computer vision, deep learning, compressive sensing, and optimization.

Mingyi He received the B. Eng. degree in electronic engineering and the M. Eng. degree in signal and systems from Northwestern Polytechnical University (NPU), China in 1982 and 1985, respectively, and the Ph. D. degree in signal and information processing from Xidian University, China in 1994. Since 1985, he has been with School of Electronics and Information, NPU, where he has been a full professor since 1996 and appointed as a chief professor of SIP in 1998. He was the (co)recipient of the 2012 CVPR Best Paper Award, the 2017 APSIPA ASC Best Deep/Machine Learning Paper Award, and the 2017 DICTA Best Student Paper Award. He was a recipient of the Government Lifelong Subsidy from the State Council of China and the Baosteel Outstanding Teacher Award in 2017. He received awards from the IEEE Signal Processing Society in 2014, APSIPA in 2019, China Remote Sensing Committee in 2023, Journal of Image and Graphs in 2022, Signal Processing in 2023, the Chinese Institute of Electronics in 2018 and 2020, and the Shaanai Institute of Electronics in 2020. He has acted as the general chair or the TPC (co)chair and the area chair for over 30 national and international conferences. He was also an Associate Editor of the IEEE Transactions on Geoscience and Remote Sensing and APSIPA SIP and a Guest Editor of the IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. He is Fellow of CIE and Vice President of APSIPA (2021–2024).

His research interests focus on advanced machine vision and intelligent processing, including signal and image processing, computer vision, hyper-spectral remote sensing, 3D information acquisition and processing, and neural network artificial intelligence.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fan, B., Dai, Y. & He, M. Rolling Shutter Camera: Modeling, Optimization and Learning. Mach. Intell. Res. 20, 783–798 (2023). https://doi.org/10.1007/s11633-022-1399-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11633-022-1399-z

Keywords

Search

Navigation