research-article

Event-guided Frame Interpolation and Dynamic Range Expansion of Single Rolling Shutter Image

Authors:

Yinqiang ZhengAuthors Info & Claims

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Pages 3078 - 3088

https://doi.org/10.1145/3581783.3612093

Published: 27 October 2023 Publication History

Abstract

In the presence of abrupt motion, the pushbroom scanning mechanism of a rolling shutter (RS) camera tends to bring undesirable distortion, which is recently shown to be beneficial for high-speed frame interpolation. Although promising results have been reported by using multiple consecutive RS frames, to interpolate intermediate distortion-free frames from a single RS image is still an open question, due to the existence of multiple motions that can account for the recorded distortion. Another limitation of RS cameras in complex dynamic scenarios lies in the dynamic range, since traditional ways of multiple exposure for high dynamic range (HDR) imaging will fail due to alignment issues. To deal with these two challenges simultaneously, we propose to use an event camera for assistance, which has much faster temporal response and wider dynamic range. Since there does not exist learning data for this brand new imaging setup, we first build a quad-axis imaging system to capture a realistic dataset called REG-HDR, with pairs of fully aligned RS image and its associated events, as well as their corresponding high-speed HDR GS images. We also propose a flow-based network for frame interpolation, compounded with an attention-based fusion network for dynamic range expansion. Experimental results have verified the effectiveness of our proposed algorithm and the superiority of using realistic data for this challenging dural-purpose enhancement task.

References

[1]

Mingdeng Cao, Zhihang Zhong, Jiahao Wang, Yinqiang Zheng, and Yujiu Yang. 2022. Learning Adaptive Warping for Real-World Rolling Shutter Correction. In Proc. of IEEE Conf. Comput. Vis. Pattern Recog. 17785--17793.

[2]

Haoyu Chen, Minggui Teng, Boxin Shi, Yizhou Wang, and Tiejun Huang. 2022. A Residual Learning Approach to Deblur and Generate High Frame Rate Video With an Event Camera. IEEE Trans. Multimedia (2022).

Digital Library

[3]

Bin Fan and Yuchao Dai. 2021. Inverting a rolling shutter camera: bring rolling shutter images to high framerate global shutter video. In Proc. of Int. Conf. Comput. Vis. 4228--4237.

[4]

Bin Fan, Yuchao Dai, and Mingyi He. 2021. Sunet: symmetric undistortion network for rolling shutter correction. In Proc. of Int. Conf. Comput. Vis. 4541--4550.

[5]

Bin Fan, Yuchao Dai, Zhiyuan Zhang, Qi Liu, and Mingyi He. 2022. Context-Aware Video Reconstruction for Rolling Shutter Cameras. In Proc. of IEEE Conf. Comput. Vis. Pattern Recog. 17572--17582.

[6]

Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, et al. 2020. Event-based vision: A survey. IEEE Trans. Pattern Anal. Mach. Intell., Vol. 44, 1 (2020), 154--180.

Digital Library

[7]

Jin Han, Yixin Yang, Peiqi Duan, Chu Zhou, Lei Ma, Chao Xu, Tiejun Huang, Imari Sato, and Boxin Shi. 2023. Hybrid high dynamic range imaging fusing neuromorphic and conventional images. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).

Digital Library

[8]

Jin Han, Yixin Yang, Chu Zhou, Chao Xu, and Boxin Shi. 2021. Evintsr-net: Event guided multiple latent frames reconstruction and super-resolution. In Proc. of Int. Conf. Comput. Vis. 4882--4891.

[9]

Jin Han, Chu Zhou, Peiqi Duan, Yehui Tang, Chang Xu, Chao Xu, Tiejun Huang, and Boxin Shi. 2020. Neuromorphic camera guided high dynamic range imaging. In Proc. of IEEE Conf. Comput. Vis. Pattern Recog. 1730--1739.

[10]

Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7132--7141.

[11]

Yuhuang Hu, Shih-Chii Liu, and Tobi Delbruck. 2021. v2e: From video frames to realistic DVS events. In Proc. of IEEE Conf. Comput. Vis. Pattern Recog. 1312--1321.

[12]

Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. 2020. Rife: Real-time intermediate flow estimation for video frame interpolation. arXiv preprint arXiv:2011.06294 (2020).

[13]

Noor A Ibraheem, Mokhtar M Hasan, Rafiqul Z Khan, and Pramod K Mishra. 2012. Understanding color models: a review. ARPN Journal of science and technology, Vol. 2, 3 (2012), 265--275.

[14]

Yongcheng Jing, Yiding Yang, Xinchao Wang, Mingli Song, and Dacheng Tao. 2021. Turning frequency to resolution: Video super-resolution via event cameras. In Proc. of IEEE Conf. Comput. Vis. Pattern Recog. 7772--7781.

[15]

Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In Proc. of Eur. Conf. Comput. Vis. Springer, 694--711.

[16]

Hanme Kim, Ankur Handa, Ryad Benosman, Sio-Hoi Ieng, and Andrew Davison. 2014. Simultaneous mosaicing and tracking with an event camera. In Proc. of Brit. Mach. Vis. Conf.

[17]

Taewoo Kim, Jungmin Lee, Lin Wang, and Kuk-Jin Yoon. 2022. Event-guided Deblurring of Unknown Exposure Time Videos. In Proc. of Eur. Conf. Comput. Vis.

Digital Library

[18]

Lichtsteiner, P; Posch, C; Delbruck, T. 2008. A 128× 128 120 dB 15 μs latency asynchronous temporal contrast vision sensor. IEEE Journal of Solid State Circuits, Vol. 43, 2 (2008), 566--576.

[19]

Dave Litwiller. 2001. CCD vs. CMOS. Photonics spectra, Vol. 35, 1 (2001), 154--158.

[20]

Peidong Liu, Zhaopeng Cui, Viktor Larsson, and Marc Pollefeys. 2020. Deep shutter unrolling network. In Proc. of IEEE Conf. Comput. Vis. Pattern Recog. 5941--5949.

[21]

Liyuan Pan, Cedric Scheerlinck, Xin Yu, Richard Hartley, Miaomiao Liu, and Yuchao Dai. 2019. Bringing a blurry frame alive at high frame-rate with an event camera. In Proc. of IEEE Conf. Comput. Vis. Pattern Recog.

[22]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Proc. of Adv. Neural Inform. Process. Syst., Vol. 32 (2019).

[23]

Chiara Plizzari, Mirco Planamente, Gabriele Goletto, Marco Cannici, Emanuele Gusso, Matteo Matteucci, and Barbara Caputo. 2022. E2 (GO) MOTION: Motion Augmented Event Stream for Egocentric Action Recognition. In Proc. of IEEE Conf. Comput. Vis. Pattern Recog.

[24]

Henri Rebecq, René Ranftl, Vladlen Koltun, and Davide Scaramuzza. 2019a. Events-to-video: Bringing modern computer vision to event cameras. In Proc. of IEEE Conf. Comput. Vis. Pattern Recog. 3857--3866.

[25]

Henri Rebecq, René Ranftl, Vladlen Koltun, and Davide Scaramuzza. 2019b. High speed and high dynamic range video with an event camera. IEEE Trans. Pattern Anal. Mach. Intell., Vol. 43, 6 (2019), 1964--1980.

[26]

Vijay Rengarajan, Yogesh Balaji, and AN Rajagopalan. 2017. Unrolling the shutter: Cnn to correct motion distortions. In Proc. of IEEE Conf. Comput. Vis. Pattern Recog. 2291--2299.

[27]

Cedric Scheerlinck, Henri Rebecq, Daniel Gehrig, Nick Barnes, Robert Mahony, and Davide Scaramuzza. 2020. Fast image reconstruction with an event camera. In Proc. of the IEEE Winter Conference on Applications of Computer Vision. 156--163.

[28]

Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).

[29]

Chen Song, Qixing Huang, and Chandrajit Bajaj. 2022. E-CIR: Event-Enhanced Continuous Intensity Recovery. In Proc. of IEEE Conf. Comput. Vis. Pattern Recog.

[30]

Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. 2018. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition. 8934--8943.

[31]

Stepan Tulyakov, Alfredo Bochicchio, Daniel Gehrig, Stamatios Georgoulis, Yuanyou Li, and Davide Scaramuzza. 2022. Time Lens: Event-based Frame Interpolation with Parametric Non-linear Flow and Multi-scale Fusion. In Proc. of IEEE Conf. Comput. Vis. Pattern Recog. 17755--17764.

[32]

Stepan Tulyakov, Daniel Gehrig, Stamatios Georgoulis, Julius Erbach, Mathias Gehrig, Yuanyou Li, and Davide Scaramuzza. 2021. Time lens: Event-based video frame interpolation. In Proc. of IEEE Conf. Comput. Vis. Pattern Recog. 16155--16164.

[33]

Bishan Wang, Jingwei He, Lei Yu, Gui-Song Xia, and Wen Yang. 2020. Event Enhanced High-Quality Image Recovery. In Proc. of Eur. Conf. Comput. Vis.

Digital Library

[34]

Lin Wang, Yo-Sung Ho, Kuk-Jin Yoon, et al. 2019. Event-based high dynamic range image and very high frame rate video generation using conditional generative adversarial networks. In Proc. of IEEE Conf. Comput. Vis. Pattern Recog. 10081--10090.

[35]

Ziwei Wang, Yonhon Ng, Cedric Scheerlinck, and Robert Mahony. 2021. An asynchronous kalman filter for hybrid event cameras. In Proc. of Int. Conf. Comput. Vis.

[36]

Fang Xu, Lei Yu, Bishan Wang, Wen Yang, Gui-Song Xia, Xu Jia, Zhendong Qiao, and Jianzhuang Liu. 2021. Motion deblurring with real events. In Proc. of Int. Conf. Comput. Vis. 2583--2592.

[37]

Qingsen Yan, Dong Gong, Qinfeng Shi, Anton van den Hengel, Chunhua Shen, Ian Reid, and Yanning Zhang. 2019. Attention-guided network for ghost-free high dynamic range imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1751--1760.

[38]

Zhihang Zhong, Mingdeng Cao, Xiao Sun, Zhirong Wu, Zhongyi Zhou, Yinqiang Zheng, Stephen Lin, and Imari Sato. 2022. Bringing rolling shutter images alive with dual reversed distortion. arXiv preprint arXiv:2203.06451 (2022).

[39]

Zhihang Zhong, Yinqiang Zheng, and Imari Sato. 2021. Towards rolling shutter correction and deblurring in dynamic scenes. In Proc. of IEEE Conf. Comput. Vis. Pattern Recog. 9219--9228.

[40]

Xinyu Zhou, Peiqi Duan, Yi Ma, and Boxin Shi. 2022. EvUnroll: Neuromorphic Events Based Rolling Shutter Image Correction. In Proc. of IEEE Conf. Comput. Vis. Pattern Recog. 17775--17784.

[41]

Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniilidis. 2019. Unsupervised event-based learning of optical flow, depth, and egomotion. In Proc. of IEEE Conf. Comput. Vis. Pattern Recog. 989--997.

[42]

Bingbing Zhuang, Quoc-Huy Tran, Pan Ji, Loong-Fah Cheong, and Manmohan Chandraker. 2019. Learning structure-and-motion-aware rolling shutter correction. In Proc. of IEEE Conf. Comput. Vis. Pattern Recog. 4551--4560.

[43]

Yunhao Zou, Yinqiang Zheng, Tsuyoshi Takatani, and Ying Fu. 2021. Learning to reconstruct high speed and high dynamic range videos from events. In Proc. of IEEE Conf. Comput. Vis. Pattern Recog. 2024--2033.

Cited By

Han XZhang ZWu YZhang XWu ZCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Event Traffic Forecasting with Sparse Multimodal DataProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680706(8855-8864)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680706
Yin ZMa MLin GZheng YCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Exploring Data Efficiency in Image Restoration: A Gaussian Denoising Case StudyProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680603(2564-2573)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680603
AliAkbarpour HMoori AKhorramdel JBlasch ETahri O(2024)Emerging Trends and Applications of Neuromorphic Dynamic Vision Sensors: A SurveyIEEE Sensors Reviews10.1109/SR.2024.35139521(14-63)Online publication date: 2024
https://doi.org/10.1109/SR.2024.3513952
Show More Cited By

Index Terms

Event-guided Frame Interpolation and Dynamic Range Expansion of Single Rolling Shutter Image
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Image and video acquisition
        Computational photography

Recommendations

UniINR: Event-Guided Unified Rolling Shutter Correction, Deblurring, and Interpolation
Computer Vision – ECCV 2024
Abstract
Video frames captured by rolling shutter (RS) cameras during fast camera movement frequently exhibit RS distortion and blur simultaneously. Naturally, recovering high-frame-rate global shutter (GS) sharp frames from an RS blur frame must ...
High dynamic range video

Typical video footage captured using an off-the-shelf camcorder suffers from limited dynamic range. This paper describes our approach to generate high dynamic range (HDR) video from an image sequence of a dynamic scene captured while rapidly varying the ...
Event-Based Image Enhancement Under High Dynamic Range Scenarios
Computer Vision – ACCV 2024
Abstract
Event cameras, as bio-inspired vision sensors with a high dynamic range, are capable of addressing the problems of local overexposure or underexposure that conventional frame-based cameras encounter in scenarios with high dynamic range or ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

October 2023

9913 pages

ISBN:9798400701085

DOI:10.1145/3581783

General Chairs:
Abdulmotaleb El Saddik
University of Ottawa, Canada & MBZUAI, UAE
,
Tao Mei
HiDream.ai, China
,
Rita Cucchiara
University of Modena and Reggio Emilia, Italy
,
Program Chairs:
Marco Bertini
University of Florence, Italy
,
Diana Patricia Tobon Vallejo
Unversidad de Medellin, Colombia
,
Pradeep K. Atrey
University at Albany, State University of New York, USA
,
M. Shamim Hossain
M. Shamim Hossain (King Saud University, KSA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

JST, the establishment of university fellowships towards the creation of science technology innovation
ROIS NII Open Collaborative Research
JSPS KAKENHI

Conference

MM '23

Sponsor:

SIGMM

MM '23: The 31st ACM International Conference on Multimedia

October 29 - November 3, 2023

Ottawa ON, Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
264
Total Downloads

Downloads (Last 12 months)120
Downloads (Last 6 weeks)13

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Han XZhang ZWu YZhang XWu ZCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Event Traffic Forecasting with Sparse Multimodal DataProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680706(8855-8864)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680706
Yin ZMa MLin GZheng YCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Exploring Data Efficiency in Image Restoration: A Gaussian Denoising Case StudyProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680603(2564-2573)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680603
AliAkbarpour HMoori AKhorramdel JBlasch ETahri O(2024)Emerging Trends and Applications of Neuromorphic Dynamic Vision Sensors: A SurveyIEEE Sensors Reviews10.1109/SR.2024.35139521(14-63)Online publication date: 2024
https://doi.org/10.1109/SR.2024.3513952
Chang YXiaokaiti YLiu YFan BHuang ZHuang TShi B(2024)Towards HDR and HFR Video from Rolling-Mixed-Bit Spikings2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.02373(25117-25127)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.02373
Zhong ZKrishnan GSun XQiao YMa SWang J(2024)Clearer Frames, Anytime: Resolving Velocity Ambiguity in Video Frame InterpolationComputer Vision – ECCV 202410.1007/978-3-031-73414-4_20(346-363)Online publication date: 25-Oct-2024
https://doi.org/10.1007/978-3-031-73414-4_20

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten