research-article

Semantically-Consistent Dynamic Blurry Image Generation for Image Deblurring

Authors:

Yong XiaAuthors Info & Claims

MM '22: Proceedings of the 30th ACM International Conference on Multimedia

Pages 2547 - 2555

https://doi.org/10.1145/3503161.3548106

Published: 10 October 2022 Publication History

Abstract

The training of deep learning-based image deblurring models heavily relies on the paired sharp/blurry image dataset. Although many works verified that synthesized blurry-sharp pairs contribute to improving the deblurring performance, it is still an open problem about how to synthesize realistic and diverse dynamic blurry images. Instead of directly synthesizing blurry images, in this paper, we propose a novel method to generate semantic-aware dense dynamic motion, and employ the generated motion to synthesize blurry images. Specifically, for each sharp image, both the global motion (camera shake) and local motion (object moving) are considered given the depth information as the condition. Then, a blur creation module takes the spatial-variant motion information and the sharp image as input to synthesize a motion-blurred image. A relativistic GAN loss is employed to assure the synthesized blurry image is as realistic as possible. Experiments show that our method can generate diverse dynamic motion and visually realistic blurry images. Also, the generated image pairs can further improve the quantitative performance and generalization ability of the existing deblurring method on several test sets.

Supplementary Material

MP4 File (MM22-fp1508.mp4)

The presentation video of the paper ''Semantically-Consistent Dynamic Blurry Image Generation for Image Deblurring'', includes our motivation, the detailed method, and qualitative/quantitative results.

Download
30.25 MB

References

[1]

Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. 2021. Adabins: Depth estimation using adaptive bins. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4009--4018.

[2]

Tim Brooks and Jonathan T Barron. 2019. Learning to synthesize motion blur. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6840--6848.

[3]

Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. 2021. Pre-trained image processing transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12299--12310.

[4]

Sunghyun Cho and Seungyong Lee. 2009. Fast motion deblurring. ACM Transactions on graphics (TOG), Vol. 28, 5 (2009), 145.

[5]

Sung-Jin Cho, Seo-Won Ji, Jun-Pyo Hong, Seung-Won Jung, and Sung-Jea Ko. 2021. Rethinking coarse-to-fine approach in single image deblurring. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 4641--4650.

[6]

Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. 2017. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision. 764--773.

[7]

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020).

[8]

Rob Fergus, Barun Singh, Aaron Hertzmann, Sam T Roweis, and William T Freeman. 2006. Removing camera shake from a single photograph. In ACM SIGGRAPH 2006 Papers. 787--794.

Digital Library

[9]

Hongyun Gao, Xin Tao, Xiaoyong Shen, and Jiaya Jia. 2019. Dynamic scene deblurring with parameter selective sharing and nested skip connections. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3848--3856.

[10]

Dong Gong, Jie Yang, Lingqiao Liu, Yanning Zhang, Ian Reid, Chunhua Shen, Anton Van Den Hengel, and Qinfeng Shi. 2017. From motion blur to motion flow: a deep learning solution for removing heterogeneous motion blur. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2319--2328.

[11]

Jiaya Jia. 2007. Single image motion deblurring using transparency. In 2007 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1--8.

[12]

Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision. Springer, 694--711.

[13]

Alexia Jolicoeur-Martineau. 2018. The relativistic discriminator: a key element missing from standard GAN. arXiv preprint arXiv:1807.00734 (2018).

[14]

Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jivr 'i Matas. 2018. Deblurgan: Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8183--8192.

[15]

Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. 2019. Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better. In Proceedings of the IEEE International Conference on Computer Vision. 8878--8887.

[16]

Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. 2017. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3883--3891.

[17]

Jinshan Pan, Zhe Hu, Zhixun Su, Hsin-Ying Lee, and Ming-Hsuan Yang. 2016b. Soft-segmentation guided object motion deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 459--468.

[18]

Jinshan Pan, Zhe Hu, Zhixun Su, and Ming-Hsuan Yang. 2016a. $ l_0 $-regularized intensity and gradient prior for deblurring text images and beyond. IEEE transactions on pattern analysis and machine intelligence, Vol. 39, 2 (2016), 342--355.

[19]

Jinshan Pan, Deqing Sun, Hanspeter Pfister, and Ming-Hsuan Yang. 2016c. Blind image deblurring using dark channel prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1628--1636.

[20]

Dongwon Park, Dong Un Kang, Jisoo Kim, and Se Young Chun. 2020. Multi-Temporal Recurrent Neural Networks For Progressive Non-Uniform Single Image Deblurring With Incremental Temporal Training. In European Conference on Computer Vision. Springer, 327--343.

[21]

Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. 2019. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2337--2346.

[22]

Kuldeep Purohit and AN Rajagopalan. 2020. Region-Adaptive Dense Network for Efficient Motion Deblurring. In AAAI. 11882--11889.

[23]

Jaesung Rim, Haeyun Lee, Jucheol Won, and Sunghyun Cho. 2020. Real-world blur dataset for learning and benchmarking deblurring algorithms. In European Conference on Computer Vision. Springer, 184--201.

Digital Library

[24]

Christian J Schuler, Michael Hirsch, Stefan Harmeling, and Bernhard Schölkopf. 2015. Learning to deblur. IEEE transactions on pattern analysis and machine intelligence, Vol. 38, 7 (2015), 1439--1451.

[25]

Ziyi Shen, Wenguan Wang, Xiankai Lu, Jianbing Shen, Haibin Ling, Tingfa Xu, and Ling Shao. 2019. Human-aware motion deblurring. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 5572--5581.

[26]

Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).

[27]

Maitreya Suin, Kuldeep Purohit, and AN Rajagopalan. 2020. Spatially-attentive patch-hierarchical network for adaptive motion deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3606--3615.

[28]

Yu-Wing Tai, Ping Tan, and Michael S Brown. 2010. Richardson-Lucy deblurring for scenes under a projective motion path. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 33, 8 (2010), 1603--1618.

[29]

Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia. 2018. Scale-recurrent network for deep image deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8174--8182.

[30]

Phong Tran, Anh Tuan Tran, Quynh Phung, and Minh Hoai. 2021. Explore Image Deblurring via Encoded Blur Kernel Space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11956--11965.

[31]

Fu-Jen Tsai, Yan-Tsung Peng, Yen-Yu Lin, Chung-Chi Tsai, and Chia-Wen Lin. 2022. Stripformer: Strip Transformer for Fast Image Deblurring. arXiv preprint arXiv:2204.04627 (2022).

[32]

Oliver Whyte, Josef Sivic, and Andrew Zisserman. 2014. Deblurring shaken and partially saturated images. International journal of computer vision, Vol. 110, 2 (2014), 185--201.

Digital Library

[33]

Oliver Whyte, Josef Sivic, Andrew Zisserman, and Jean Ponce. 2012. Non-uniform deblurring for shaken images. International journal of computer vision, Vol. 98, 2 (2012), 168--186.

Digital Library

[34]

Li Xu and Jiaya Jia. 2010. Two-phase kernel estimation for robust motion deblurring. In European conference on computer vision. Springer, 157--170.

[35]

Li Xu, Shicheng Zheng, and Jiaya Jia. 2013. Unnatural l0 sparse representation for natural image deblurring. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1107--1114.

Digital Library

[36]

Yufei Xu, Qiming Zhang, Jing Zhang, and Dacheng Tao. 2021. ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias. In NeurIPS.

[37]

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. 2021. Multi-stage progressive image restoration. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 14821--14831.

[38]

Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz. 2019. Deep stacked hierarchical multi-patch network for image deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5978--5986.

[39]

Jiawei Zhang, Jinshan Pan, Jimmy Ren, Yibing Song, Linchao Bao, Rynson WH Lau, and Ming-Hsuan Yang. 2018. Dynamic scene deblurring using spatially variant recurrent neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2521--2529.

[40]

Kaihao Zhang, Wenhan Luo, Yiran Zhong, Lin Ma, Bjorn Stenger, Wei Liu, and Hongdong Li. 2020. Deblurring by realistic blurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2737--2746.

[41]

Qiming Zhang, Yufei Xu, Jing Zhang, and Dacheng Tao. 2022b. ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond. arXiv preprint arXiv:2202.10108 (2022).

[42]

Youjian Zhang, Chaoyue Wang, Stephen John Maybank, and Dacheng Tao. 2021. Exposure Trajectory Recovery from Motion Blur. IEEE Trans. Pattern Anal. Mach Intell. (2021).

[43]

Youjian Zhang, Chaoyue Wang, and Dacheng Tao. 2022a. Neural Maximum A Posteriori Estimation on Unpaired Data for Motion Deblurring. arXiv preprint arXiv:2204.12139 (2022).

[44]

Shicheng Zheng, Li Xu, and Jiaya Jia. 2013. Forward motion deblurring. In Proceedings of the IEEE international conference on computer vision. 1465--1472.

Digital Library

Cited By

Wang PZhu YXue DYan QSun JYoon SZhang Y(2024)Take a prior from other tasks for severe blur removalComputer Vision and Image Understanding10.1016/j.cviu.2024.104027245(104027)Online publication date: Aug-2024
https://doi.org/10.1016/j.cviu.2024.104027
Yang HLi FZhang L(2024)6-DOF Motion Blur Synthesis and Performance Evaluation of Object DetectionPattern Recognition10.1007/978-3-031-78444-6_1(1-17)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1007/978-3-031-78444-6_1
Chen LWang YChen YEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)End-to-end XY Separation for Single Image Blind DeblurringProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612149(1273-1282)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612149
Show More Cited By

Index Terms

Semantically-Consistent Dynamic Blurry Image Generation for Image Deblurring
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Machine learning
2. Networks
  1. Network architectures

Recommendations

Deep non-blind deblurring network for saturated blurry images
Abstract
Non-blind image deblurring has attracted a lot of attention in the field of low-level vision. However, the existing non-blind deblurring methods cannot effectively deal with a saturated blurry image. The key point is that the degradation model of ...
Spatially variant defocus blur map estimation and deblurring from a single image

A blur map estimation method using edge information is proposed.The blur map is segmented into multiple superpixels according to the image contours.Ringing artifacts and noise are detected and removed after deconvolution. In this paper, we propose a ...
Disparity-based space-variant image deblurring

Obtaining a good-quality image requires exposure to light for an appropriate amount of time. If there is camera or object motion during the exposure time, the image is blurred. To remove the blur, some recent image deblurring methods effectively ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '22: Proceedings of the 30th ACM International Conference on Multimedia

October 2022

7537 pages

ISBN:9781450392037

DOI:10.1145/3503161

General Chairs:
João Magalhães
NOVA University of Lisbon, Portugal
,
Alberto del Bimbo
University of Florence, Italy
,
Shin'ichi Satoh
National Institute of Informatics, Japan
,
Nicu Sebe
University of Trento, Italy
,
Program Chairs:
Xavier Alameda-Pineda
Inria, Grenoble, France
,
Qin Jin
Renmin University of China, China
,
Vincent Oria
New Jersey Institute of Technology, USA
,
Laura Toni
University College London, UK

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 October 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Key Research and Development Program of Shaanxi Province
Science and Technology Innovation 2030 ??Brain Science and Brain-like Research? Major Project
National Natural Science Foundation of China

Conference

MM '22

Sponsor:

SIGMM

MM '22: The 30th ACM International Conference on Multimedia

October 10 - 14, 2022

Lisboa, Portugal

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
228
Total Downloads

Downloads (Last 12 months)33
Downloads (Last 6 weeks)1

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang PZhu YXue DYan QSun JYoon SZhang Y(2024)Take a prior from other tasks for severe blur removalComputer Vision and Image Understanding10.1016/j.cviu.2024.104027245(104027)Online publication date: Aug-2024
https://doi.org/10.1016/j.cviu.2024.104027
Yang HLi FZhang L(2024)6-DOF Motion Blur Synthesis and Performance Evaluation of Object DetectionPattern Recognition10.1007/978-3-031-78444-6_1(1-17)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1007/978-3-031-78444-6_1
Chen LWang YChen YEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)End-to-end XY Separation for Single Image Blind DeblurringProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612149(1273-1282)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612149
Mehta RRavi SSingh V(2023)Robustness and Convergence of Mirror Descent for Blind DeconvolutionICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP49357.2023.10096350(1-5)Online publication date: 4-Jun-2023
https://doi.org/10.1109/ICASSP49357.2023.10096350

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten