Abstract
Non-uniform image deblurring is an ill-posed problem. Previous research efforts attempt to solve this problem by increasing the number of scales processed in the model, including but not limited to multi-scale methods, multi-patch methods, and atrous convolution. However, these methods are still subject to the fixed geometric structures, which are inherently unable to adequately handle complex blur. This paper proposes a novel residual block called Deform-ResBlock that is composed of traditional convolution and deformable convolution to enhance the model’s capability of modeling geometric transformations. Then, we design parallel multi-scale convolution streams composed of densely Deform-ResBlock for extracting multi-scale features. Finally, we apply the multi-patch approach stacking two stages to deblur images gradually. The overall method is named deformable multi-scale fusion network (DMSFN). Compared to the previous methods, our method combines the advantages of multi-scale and multi-patch approaches and has better modeling geometric transformation capability. Extensive experimental results on the GoPro, HIDE, and RealBlur datasets demonstrate that the proposed method performs favorably against the state-of-the-art in the non-uniform image deblurring.
Similar content being viewed by others
References
Brehm S, Scherer S, Lienhart R (2020) High-resolution dual-stage multi-level feature aggregation for single image and video deblurring. In: 2020 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp 1872–1881. https://doi.org/10.1109/CVPRW50498.2020.00237
Chakrabarti A (2016) A Neural Approach to Blind Motion Deblurring. In: Leibe B, Matas J, Sebe N, Welling M (eds) Computer Vision – ECCV 2016. Springer International Publishing, Cham, pp 221–235
Dai J, Qi H, Xiong Y, Li Y, Zhang G, Hu H, Wei Y (2017) Deformable convolutional networks. In: IEEE international conference on computer vision, ICCV 2017, Venice, Italy, 22-29 Oct 2017, pp 764–773. https://doi.org/10.1109/ICCV.2017.89
Gao H, Tao X, Shen X, Jia J (2019) Dynamic scene deblurring with parameter selective sharing and nested skip connections. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)
Gupta A, Joshi N, Lawrence Zitnick C, Cohen M, Curless B (2010) Single image deblurring using motion density functions. In: Daniilidis K, Maragos P, Paragios N (eds) Computer vision – ECCV 2010. Springer, pp 171–184
Hu Z, Cho S, Wang J, Yang M-H (2014) Deblurring low-light images with light streaks. In: 2014 IEEE conference on computer vision and pattern recognition, pp 3382–3389. https://doi.org/10.1109/CVPR.2014.432
Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 2261–2269. https://doi.org/10.1109/CVPR.2017.243
Jin Q, Meng Z, Pham TD, Chen Q, Wei L, Su R (2019) Dunet: a deformable network for retinal vessel segmentation. Knowl-Based Syst 178:149–162. https://doi.org/10.1016/j.knosys.2019.04.025
Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: Bengio Y, LeCun Y (eds) 3rd International conference on learning representations, ICLR. arXiv:http://arxiv.org/abs/1412.6980
Kupyn O, Budzan V, Mykhailych M, Mishkin D, Matas J (2018) Deblurgan: Blind motion deblurring using conditional adversarial networks. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, pp 8183–8192. https://doi.org/10.1109/CVPR.2018.00854
Kupyn O, Martyniuk T, Wu J, Wang Z (2019) Deblurgan-v2: deblurring (orders-of-magnitude) faster and better. In: 2019 IEEE/CVF international conference on computer vision (ICCV), pp 8877–8886. https://doi.org/10.1109/ICCV.2019.00897
Lei P, Todorovic S (2018) Temporal deformable residual networks for action segmentation in videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)
Liao X, Zhang X (2017) Multi-scale mutual feature convolutional neural network for depth image denoise and enhancement. In: 2017 IEEE visual communications and image processing (VCIP), pp 1–4. https://doi.org/10.1109/VCIP.2017.8305145
Loshchilov I, Hutter F (2017) SGDR: Stochastic Gradient Descent with Warm Restarts. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net. https://openreview.net/forum?id=Skq89Scxx
Ma T-H, Huang T-Z, Zhao X-L, Lou Y (2017) Image deblurring with an inaccurate blur kernel using a group-based low-rank image prior. Inf Sci 408:213–233. https://doi.org/10.1016/j.ins.2017.04.049
Ma C, Zhang J, Xu S, Meng W, Xi R, Kumar GH, Zhang X (2018) Accurate blind deblurring using salientpatch-based prior for large-size images. Multimed Tools Appl
Nah S, Kim TH, Lee KM (2017) Deep multi-scale convolutional neural network for dynamic scene deblurring. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 257–265. https://doi.org/10.1109/CVPR.2017.35
Pan J, Sun D, Pfister H, Yang M-H (2016) Blind image deblurring using dark channel prior. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 1628–1636. https://doi.org/10.1109/CVPR.2016.180
Park D, Kang DU, Kim J, Chun SY (2020) Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training. In: Vedaldi A, Bischof H, Brox T, Frahm J-M (eds) Computer vision – ECCV 2020. Springer, pp 327–343
Park D, Kim J, Chun SY (2019) Down-Scaling with Learned Kernels in Multi-Scale Deep Neural Networks for Non-Uniform Single Image Deblurring. CoRR, arXiv: http://arxiv.org/abs/1903.10157
Rim J, Lee H, Won J, Cho S (2020) Real-world blur dataset for learning and benchmarking deblurring algorithms. In: Vedaldi A, Bischof H, Brox T, Frahm J-M (eds) Computer vision – ECCV 2020. Springer, pp 184–201
Schuler CJ, Hirsch M, Harmeling S, Schölkopf B (2014) Learning to deblur. arXiv:http://arxiv.org/abs/1406.7444
Shen Z, Wang W, Lu X, Shen J, Ling H, Xu T, Shao L (2019) Human-aware motion deblurring. In: 2019 IEEE/CVF international conference on computer vision (ICCV), pp 5571–5580. https://doi.org/10.1109/ICCV.2019.00567
Shi W, Caballero J, Huszár F, Totz J, Aitken AP, Bishop R, Rueckert D, Wang Z (2016) Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. IEEE Computer Society, pp 1874–1883. https://doi.org/10.1109/CVPR.2016.207
Suin M, Purohit K, Rajagopalan AN (2020) Spatially-attentive patch-hierarchical network for adaptive motion deblurring. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 3603–3612. https://doi.org/10.1109/CVPR42600.2020.00366
Sun J, Cao W, Xu Z, Ponce J (2015) Learning a convolutional neural network for non-uniform motion blur removal. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7–12, 2015. IEEE Computer Society, pp 769–777. https://doi.org/10.1109/CVPR.2015.7298677
Sun Y, Zhang Y, Liu S, Lu W, Li X (2020) Image super-resolution using supervised multi-scale feature extraction network. Multimed Tools Appl:(4), pp 1–14
Tang S, Xie X, Luan X, Xia M, Liu P (2017) Spatial-scale-based blur kernel estimation for blind motion deblurring. In: 2017 International conference on security, pattern analysis, and cybernetics (SPAC), pp 256–261. https://doi.org/10.1109/SPAC.2017.8304286
Tang K, Xu D, Liu H, Zeng Z (2020) Context module based multi-patch hierarchical network for motion deblurring. Neural Process Lett:(13), pp 1–16
Tao X, Gao H, Shen X, Wang J, Jia J (2018) Scale-recurrent network for deep image deblurring. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)
Tian C, Xu Y, Zuo W, Zhang B, Fei L, Lin C-W (2021) Coarse-to-fine cnn for image super-resolution. IEEE Trans Multimed 23:1489–1502. https://doi.org/10.1109/TMM.2020.2999182
Tian Y, Zhang Y, Fu Y, Xu C (2020) TDAN: temporally-deformable alignment network for video super-resolution. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 3357–3366. https://doi.org/10.1109/CVPR42600.2020.00342
Tsai F-J, Peng Y-T, Tsai C-C, Lin Y-Y, Lin C-W (2022) BANet: A Blur-Aware Attention Network for Dynamic Scene Deblurring. IEEE Transactions on Image Processing 31:6789–6799. https://doi.org/10.1109/TIP.2022.3216216
Wang X, Chan KCK, Yu K, Dong C, Change Loy C (2019) Edvr: video restoration with enhanced deformable convolutional networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) workshops
Wang Y, Yin S, Basu A (2021) A multi-scale attentive recurrent network for image dehazing. Multimed Tools Appl: (11), pp 1–27
Whyte O, Sivic J, Zisserman A, Ponce J (2010) Non-uniform deblurring for shaken images. In: 2010 IEEE computer society conference on computer vision and pattern recognition, pp 491–498. https://doi.org/10.1109/CVPR.2010.5540175
Wu H, Qu Y, Lin S, Zhou J, Qiao R, Zhang Z, Xie Y, Ma L (2021) Contrastive learning for compact single image dehazing. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 10551–10560. https://doi.org/10.1109/CVPR46437.2021.01041
Xu D, Chu Y, Sun Q (2020) Moiré pattern removal via attentive fractal network. In: 2020 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp 1943–1952. https://doi.org/10.1109/CVPRW50498.2020.00244
Xu X, Liu H, Li Y, Zhou Y (2016) Image deblurring with blur kernel estimation in rgb channels. In: 2016 IEEE international conference on digital signal processing (DSP), pp 681–684. https://doi.org/10.1109/ICDSP.2016.7868645
Xu L, Zheng S, Jia J (2013) Unnatural l0 sparse representation for natural image deblurring. In: 2013 IEEE conference on computer vision and pattern recognition, pp 1107–1114. https://doi.org/10.1109/CVPR.2013.147
Ye M, Lyu D, Chen G (2020) Scale-iterative upscaling network for image deblurring. IEEE Access 8:18316–18325. https://doi.org/10.1109/ACCESS.2020.2967823
Yu X, Xu F, Zhang S, Zhang L (2014) Efficient patch-wise non-uniform deblurring for a single image. IEEE Trans Multimed 16(6):1510–1524. https://doi.org/10.1109/TMM.2014.2321734
Zamir SW, Arora A, Khan S, Hayat M, Khan FS, Yang M-H, Shao L (2020) Learning enriched features for real image restoration and enhancement. In: Vedaldi A, Bischof H, Brox T, Frahm J-M (eds) Computer vision – ECCV 2020. Springer, pp 492–511
Zamir SW, Arora A, Khan SH, Hayat M, Khan FS, Yang M, Shao L (2021) Multi-stage progressive image restoration. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 14821–14831. https://doi.org/10.1109/CVPR46437.2021.01458
Zhang H, Dai Y, Li H, Koniusz P (2019) Deep stacked hierarchical multi-patch network for image deblurring. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)
Zhang C, Kim J (2019) Object detection with location-aware deformable convolution and backward attention filtering. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)
Zhang F, Lu W, Liu H, Xue F (2018) Natural image deblurring based on l0-regularization and kernel shape optimization. Multimed Tools Appl 77 (20):26239–26257
Zhang K, Luo W, Zhong Y, Ma L, Stenger B, Liu W, Li H (2020) Deblurring by realistic blurring. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 2734–2743. https://doi.org/10.1109/CVPR42600.2020.00281
Zhang J, Pan J, Ren J, Song Y, Bao L, Lau RWH, Yang M-H (2018) Dynamic scene deblurring using spatially variant recurrent neural networks. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, pp 2521–2529. https://doi.org/10.1109/CVPR.2018.00267
Zhang Q, Yang Y (2021) Sa-net: shuffle attention for deep convolutional neural networks. In: IEEE international conference on acoustics, speech and signal processing, ICASSP, pp 2235–2239. https://doi.org/10.1109/ICASSP39728.2021.9414568
Zhao G, Liu J, Jiang J, Wang W (2017) A deep cascade of neural networks for image inpainting, deblurring and denoising. Multimed Tools Appl
Zhao Z, Xiong B, Gai S, Wang L (2020) Improved deep multi-patch hierarchical network with nested module for dynamic scene deblurring. IEEE Access PP(99):1–1
Zhao B, Zhang W, Ding H, Wang H (2013) Non-blind image deblurring from a single image. Cogn Comput 5 (1):3–12. https://doi.org/10.1007/s12559-012-9139-2
Acknowledgements
This research is partly supported by the Key R&D Programs in Jiangsu Province of China (No. BE2021703 and BE2022768), and partly supported by the National Natural Science Foundation of China under Grant T2225025.
We thank the Big Data Computing Center of Southeast University for providing the facility support on the numerical calculations in this paper.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
This research is partly supported by the Key R&D Programs in Jiangsu Province of China (No. BE2021703 and BE2022768), and partly supported by the National Natural Science Foundation of China under Grant T2225025.
The authors declare that they have no known personal relationships that could have appeared to influence the work reported in this paper.
The datasets generated and analysed during the current study are available from the corresponding author on reasonable request.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zhang, Z., Chen, Y., Zhu, A. et al. Deformable multi-scale fusion network for non-uniform single image deblurring. Multimed Tools Appl 82, 45621–45638 (2023). https://doi.org/10.1007/s11042-023-14818-y
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-023-14818-y