Skip to main content
SpringerLink
Log in

Deformable multi-scale fusion network for non-uniform single image deblurring

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Non-uniform image deblurring is an ill-posed problem. Previous research efforts attempt to solve this problem by increasing the number of scales processed in the model, including but not limited to multi-scale methods, multi-patch methods, and atrous convolution. However, these methods are still subject to the fixed geometric structures, which are inherently unable to adequately handle complex blur. This paper proposes a novel residual block called Deform-ResBlock that is composed of traditional convolution and deformable convolution to enhance the model’s capability of modeling geometric transformations. Then, we design parallel multi-scale convolution streams composed of densely Deform-ResBlock for extracting multi-scale features. Finally, we apply the multi-patch approach stacking two stages to deblur images gradually. The overall method is named deformable multi-scale fusion network (DMSFN). Compared to the previous methods, our method combines the advantages of multi-scale and multi-patch approaches and has better modeling geometric transformation capability. Extensive experimental results on the GoPro, HIDE, and RealBlur datasets demonstrate that the proposed method performs favorably against the state-of-the-art in the non-uniform image deblurring.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Brehm S, Scherer S, Lienhart R (2020) High-resolution dual-stage multi-level feature aggregation for single image and video deblurring. In: 2020 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp 1872–1881. https://doi.org/10.1109/CVPRW50498.2020.00237

  2. Chakrabarti A (2016) A Neural Approach to Blind Motion Deblurring. In: Leibe B, Matas J, Sebe N, Welling M (eds) Computer Vision – ECCV 2016. Springer International Publishing, Cham, pp 221–235

  3. Dai J, Qi H, Xiong Y, Li Y, Zhang G, Hu H, Wei Y (2017) Deformable convolutional networks. In: IEEE international conference on computer vision, ICCV 2017, Venice, Italy, 22-29 Oct 2017, pp 764–773. https://doi.org/10.1109/ICCV.2017.89

  4. Gao H, Tao X, Shen X, Jia J (2019) Dynamic scene deblurring with parameter selective sharing and nested skip connections. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  5. Gupta A, Joshi N, Lawrence Zitnick C, Cohen M, Curless B (2010) Single image deblurring using motion density functions. In: Daniilidis K, Maragos P, Paragios N (eds) Computer vision – ECCV 2010. Springer, pp 171–184

  6. Hu Z, Cho S, Wang J, Yang M-H (2014) Deblurring low-light images with light streaks. In: 2014 IEEE conference on computer vision and pattern recognition, pp 3382–3389. https://doi.org/10.1109/CVPR.2014.432

  7. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 2261–2269. https://doi.org/10.1109/CVPR.2017.243

  8. Jin Q, Meng Z, Pham TD, Chen Q, Wei L, Su R (2019) Dunet: a deformable network for retinal vessel segmentation. Knowl-Based Syst 178:149–162. https://doi.org/10.1016/j.knosys.2019.04.025

    Article  Google Scholar 

  9. Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: Bengio Y, LeCun Y (eds) 3rd International conference on learning representations, ICLR. arXiv:http://arxiv.org/abs/1412.6980

  10. Kupyn O, Budzan V, Mykhailych M, Mishkin D, Matas J (2018) Deblurgan: Blind motion deblurring using conditional adversarial networks. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, pp 8183–8192. https://doi.org/10.1109/CVPR.2018.00854

  11. Kupyn O, Martyniuk T, Wu J, Wang Z (2019) Deblurgan-v2: deblurring (orders-of-magnitude) faster and better. In: 2019 IEEE/CVF international conference on computer vision (ICCV), pp 8877–8886. https://doi.org/10.1109/ICCV.2019.00897

  12. Lei P, Todorovic S (2018) Temporal deformable residual networks for action segmentation in videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)

  13. Liao X, Zhang X (2017) Multi-scale mutual feature convolutional neural network for depth image denoise and enhancement. In: 2017 IEEE visual communications and image processing (VCIP), pp 1–4. https://doi.org/10.1109/VCIP.2017.8305145

  14. Loshchilov I, Hutter F (2017) SGDR: Stochastic Gradient Descent with Warm Restarts. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net. https://openreview.net/forum?id=Skq89Scxx

  15. Ma T-H, Huang T-Z, Zhao X-L, Lou Y (2017) Image deblurring with an inaccurate blur kernel using a group-based low-rank image prior. Inf Sci 408:213–233. https://doi.org/10.1016/j.ins.2017.04.049

    Article  Google Scholar 

  16. Ma C, Zhang J, Xu S, Meng W, Xi R, Kumar GH, Zhang X (2018) Accurate blind deblurring using salientpatch-based prior for large-size images. Multimed Tools Appl

  17. Nah S, Kim TH, Lee KM (2017) Deep multi-scale convolutional neural network for dynamic scene deblurring. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 257–265. https://doi.org/10.1109/CVPR.2017.35

  18. Pan J, Sun D, Pfister H, Yang M-H (2016) Blind image deblurring using dark channel prior. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 1628–1636. https://doi.org/10.1109/CVPR.2016.180

  19. Park D, Kang DU, Kim J, Chun SY (2020) Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training. In: Vedaldi A, Bischof H, Brox T, Frahm J-M (eds) Computer vision – ECCV 2020. Springer, pp 327–343

  20. Park D, Kim J, Chun SY (2019) Down-Scaling with Learned Kernels in Multi-Scale Deep Neural Networks for Non-Uniform Single Image Deblurring. CoRR, arXiv: http://arxiv.org/abs/1903.10157

  21. Rim J, Lee H, Won J, Cho S (2020) Real-world blur dataset for learning and benchmarking deblurring algorithms. In: Vedaldi A, Bischof H, Brox T, Frahm J-M (eds) Computer vision – ECCV 2020. Springer, pp 184–201

  22. Schuler CJ, Hirsch M, Harmeling S, Schölkopf B (2014) Learning to deblur. arXiv:http://arxiv.org/abs/1406.7444

  23. Shen Z, Wang W, Lu X, Shen J, Ling H, Xu T, Shao L (2019) Human-aware motion deblurring. In: 2019 IEEE/CVF international conference on computer vision (ICCV), pp 5571–5580. https://doi.org/10.1109/ICCV.2019.00567

  24. Shi W, Caballero J, Huszár F, Totz J, Aitken AP, Bishop R, Rueckert D, Wang Z (2016) Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. IEEE Computer Society, pp 1874–1883. https://doi.org/10.1109/CVPR.2016.207

  25. Suin M, Purohit K, Rajagopalan AN (2020) Spatially-attentive patch-hierarchical network for adaptive motion deblurring. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 3603–3612. https://doi.org/10.1109/CVPR42600.2020.00366

  26. Sun J, Cao W, Xu Z, Ponce J (2015) Learning a convolutional neural network for non-uniform motion blur removal. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7–12, 2015. IEEE Computer Society, pp 769–777. https://doi.org/10.1109/CVPR.2015.7298677

  27. Sun Y, Zhang Y, Liu S, Lu W, Li X (2020) Image super-resolution using supervised multi-scale feature extraction network. Multimed Tools Appl:(4), pp 1–14

  28. Tang S, Xie X, Luan X, Xia M, Liu P (2017) Spatial-scale-based blur kernel estimation for blind motion deblurring. In: 2017 International conference on security, pattern analysis, and cybernetics (SPAC), pp 256–261. https://doi.org/10.1109/SPAC.2017.8304286

  29. Tang K, Xu D, Liu H, Zeng Z (2020) Context module based multi-patch hierarchical network for motion deblurring. Neural Process Lett:(13), pp 1–16

  30. Tao X, Gao H, Shen X, Wang J, Jia J (2018) Scale-recurrent network for deep image deblurring. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)

  31. Tian C, Xu Y, Zuo W, Zhang B, Fei L, Lin C-W (2021) Coarse-to-fine cnn for image super-resolution. IEEE Trans Multimed 23:1489–1502. https://doi.org/10.1109/TMM.2020.2999182

    Article  Google Scholar 

  32. Tian Y, Zhang Y, Fu Y, Xu C (2020) TDAN: temporally-deformable alignment network for video super-resolution. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 3357–3366. https://doi.org/10.1109/CVPR42600.2020.00342

  33. Tsai F-J, Peng Y-T, Tsai C-C, Lin Y-Y, Lin C-W (2022) BANet: A Blur-Aware Attention Network for Dynamic Scene Deblurring. IEEE Transactions on Image Processing 31:6789–6799. https://doi.org/10.1109/TIP.2022.3216216

    Article  Google Scholar 

  34. Wang X, Chan KCK, Yu K, Dong C, Change Loy C (2019) Edvr: video restoration with enhanced deformable convolutional networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) workshops

  35. Wang Y, Yin S, Basu A (2021) A multi-scale attentive recurrent network for image dehazing. Multimed Tools Appl: (11), pp 1–27

  36. Whyte O, Sivic J, Zisserman A, Ponce J (2010) Non-uniform deblurring for shaken images. In: 2010 IEEE computer society conference on computer vision and pattern recognition, pp 491–498. https://doi.org/10.1109/CVPR.2010.5540175

  37. Wu H, Qu Y, Lin S, Zhou J, Qiao R, Zhang Z, Xie Y, Ma L (2021) Contrastive learning for compact single image dehazing. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 10551–10560. https://doi.org/10.1109/CVPR46437.2021.01041

  38. Xu D, Chu Y, Sun Q (2020) Moiré pattern removal via attentive fractal network. In: 2020 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp 1943–1952. https://doi.org/10.1109/CVPRW50498.2020.00244

  39. Xu X, Liu H, Li Y, Zhou Y (2016) Image deblurring with blur kernel estimation in rgb channels. In: 2016 IEEE international conference on digital signal processing (DSP), pp 681–684. https://doi.org/10.1109/ICDSP.2016.7868645

  40. Xu L, Zheng S, Jia J (2013) Unnatural l0 sparse representation for natural image deblurring. In: 2013 IEEE conference on computer vision and pattern recognition, pp 1107–1114. https://doi.org/10.1109/CVPR.2013.147

  41. Ye M, Lyu D, Chen G (2020) Scale-iterative upscaling network for image deblurring. IEEE Access 8:18316–18325. https://doi.org/10.1109/ACCESS.2020.2967823

    Article  Google Scholar 

  42. Yu X, Xu F, Zhang S, Zhang L (2014) Efficient patch-wise non-uniform deblurring for a single image. IEEE Trans Multimed 16(6):1510–1524. https://doi.org/10.1109/TMM.2014.2321734

    Article  Google Scholar 

  43. Zamir SW, Arora A, Khan S, Hayat M, Khan FS, Yang M-H, Shao L (2020) Learning enriched features for real image restoration and enhancement. In: Vedaldi A, Bischof H, Brox T, Frahm J-M (eds) Computer vision – ECCV 2020. Springer, pp 492–511

  44. Zamir SW, Arora A, Khan SH, Hayat M, Khan FS, Yang M, Shao L (2021) Multi-stage progressive image restoration. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 14821–14831. https://doi.org/10.1109/CVPR46437.2021.01458

  45. Zhang H, Dai Y, Li H, Koniusz P (2019) Deep stacked hierarchical multi-patch network for image deblurring. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  46. Zhang C, Kim J (2019) Object detection with location-aware deformable convolution and backward attention filtering. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  47. Zhang F, Lu W, Liu H, Xue F (2018) Natural image deblurring based on l0-regularization and kernel shape optimization. Multimed Tools Appl 77 (20):26239–26257

    Article  Google Scholar 

  48. Zhang K, Luo W, Zhong Y, Ma L, Stenger B, Liu W, Li H (2020) Deblurring by realistic blurring. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 2734–2743. https://doi.org/10.1109/CVPR42600.2020.00281

  49. Zhang J, Pan J, Ren J, Song Y, Bao L, Lau RWH, Yang M-H (2018) Dynamic scene deblurring using spatially variant recurrent neural networks. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, pp 2521–2529. https://doi.org/10.1109/CVPR.2018.00267

  50. Zhang Q, Yang Y (2021) Sa-net: shuffle attention for deep convolutional neural networks. In: IEEE international conference on acoustics, speech and signal processing, ICASSP, pp 2235–2239. https://doi.org/10.1109/ICASSP39728.2021.9414568

  51. Zhao G, Liu J, Jiang J, Wang W (2017) A deep cascade of neural networks for image inpainting, deblurring and denoising. Multimed Tools Appl

  52. Zhao Z, Xiong B, Gai S, Wang L (2020) Improved deep multi-patch hierarchical network with nested module for dynamic scene deblurring. IEEE Access PP(99):1–1

    Article  Google Scholar 

  53. Zhao B, Zhang W, Ding H, Wang H (2013) Non-blind image deblurring from a single image. Cogn Comput 5 (1):3–12. https://doi.org/10.1007/s12559-012-9139-2

    Article  Google Scholar 

Download references

Acknowledgements

This research is partly supported by the Key R&D Programs in Jiangsu Province of China (No. BE2021703 and BE2022768), and partly supported by the National Natural Science Foundation of China under Grant T2225025.

We thank the Big Data Computing Center of Southeast University for providing the facility support on the numerical calculations in this paper.

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Southeast University, Nanjing, 210096, China

    Zhizhou Zhang, Yang Chen & Hanxi Liu

  2. School of Computer Science and Technology, Nanjing Tech University, Nanjing, 211816, China

    Aichun Zhu

Authors
  1. Zhizhou Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Aichun Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hanxi Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhizhou Zhang.

Ethics declarations

This research is partly supported by the Key R&D Programs in Jiangsu Province of China (No. BE2021703 and BE2022768), and partly supported by the National Natural Science Foundation of China under Grant T2225025.

The authors declare that they have no known personal relationships that could have appeared to influence the work reported in this paper.

The datasets generated and analysed during the current study are available from the corresponding author on reasonable request.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Z., Chen, Y., Zhu, A. et al. Deformable multi-scale fusion network for non-uniform single image deblurring. Multimed Tools Appl 82, 45621–45638 (2023). https://doi.org/10.1007/s11042-023-14818-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14818-y

Keywords

Search

Navigation