Skip to main content
SpringerLink
Log in

Two-stage single image Deblurring network based on deblur kernel estimation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Image deblurring for dynamic scenes is a serious challenge in computer vision. Motion blur is caused by camera shaking or object movement during the exposure time. Many photos cannot be reproduced at the moment they were taken, its contents cannot be restored if motion blur occurs. In this article, we proposed a deblurring system that uses a two-stage convolutional neural network (CNN) to achieve image deblurring through a joint learning strategy. The first-stage network predicts the deblur kernel of each pixel and pre-deblurs the input image, and then the second-stage network directly predicts clear images based on U-Net architecture. In the first-stage network, the deblur kernel uses the surrounding information to restore the centre pixel, which can effectively remove the tiny motion blur. To additionally deal with large motion blur, we extend the second-stage network is used to compensate for the limited receptive field of the first-stage deblurring kernel. We evaluate the proposed method on benchmark blur datasets. Experimental results show that the proposed method can produce better results than state-of-the-art methods, both quantitatively and qualitatively. The proposed method can achieve the best PSNR at 32.59db, 27.21db and 31.96db for the GOPRO, Köhler, and Su datasets, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Ahmed AH, Zou Q, Nagpal P, Jacob M (n.d.) Dynamic Imaging using Deep Bi-linear Unsupervised Representation (DEBLUR). accepted by IEEE Trans Med Imag. https://doi.org/10.1109/TMI.2022.3168559

  2. Anwar S, Barnes N (2019) Real Image Denoising with Feature Attention. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp 3155–3164

  3. Brooks T, Mildenhall B, Xue T, Chen J, Sharlet D, Barron JT (2019) Unprocessing Images for Learned Raw Denoising. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 11028–11037

  4. Cho S, Lee S (2009) Fast motion deblurring. In: ACM SIGGRAPH Asia 2009 papers, pp 1–8

  5. Ciregan D, Meier U, Schmidhuber J (2012) Multi-Column Deep Neural Networks for Image Classification. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)., pp 3642–3649

  6. Dai T, Cai J, Zhang Y, Xia S, Zhang L (2019) Second-Order Attention Network for Single Image Super-Resolution. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 11057–11066

  7. Fergus R, Singh B, Hertzmann A, Roweis ST, Freeman WT 2006 Removing camera shake from a single photograph. In: ACM SIGGRAPH 2006 Papers, pp 787–794

  8. Gao H, Tao X, Shen X, Jia J (2019) Dynamic scene deblurring with parameter selective sharing and nested skip connections. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3848–3856

  9. Gupta A, Joshi N, Zitnick CL, Cohen M, Curless B (2010) Single image deblurring using motion density functions. In: European conference on computer vision, Springer pp 171–184

  10. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  11. Hirsch M, Schuler CJ, Harmeling S, Schölkopf B (2011) Fast removal of non-uniform camera shake. In: 2011 International Conference on Computer Vision: IEEE, pp 463–470

  12. Jia X, De Brabandere B, Tuytelaars T, Gool LV (2016) Dynamic filter networks. Adv Neural Inf Proces Syst 29:667–675

    Google Scholar 

  13. Jo Y, Oh SW, Kang J, Kim SJ (2018) Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3224–3232

  14. Kim TH, Lee KM (2014) Segmentation-free dynamic scene deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp 2766–2773

  15. Kim TH, Ahn B, Lee KM (2013) Dynamic scene deblurring. In: Proceedings of the IEEE International Conference on Computer Vision, pp 3160–3167

  16. Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

  17. Köhler R, Hirsch M, Mohler B, Schölkopf B, Harmeling S (2012) Recording and playback of camera shake: Benchmarking blind deconvolution with a real-world database. In: European conference on computer vision,: Springer, pp 27–40

  18. Lai W-S, Huang J-B, Hu Z, Ahuja N, Yang M-H (2016) A comparative study for single image blind deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1701-1709

  19. Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z, Shi W (2017) Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 105–114

  20. Liang CH, Chen YA, Liu Y-C, Hsu WH (2022) Raw image Deblurring. IEEE Trans Mult 24:61–72. https://doi.org/10.1109/TMM.2020.3045303

    Article  Google Scholar 

  21. Lim B, Son S, Kim H, Nah S, Lee KM (2017) Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 136–144

  22. Lim S, Kim J, Kim W (2020) Deep Spectral-Spatial Network for Single Image Deblurring. IEEE Signal Processing Letters (SPL), vol 27. pp 835–839,

  23. Milletari F, Navab N,Ahmadi S (2016) V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation. In: 2016 Fourth International Conference on 3D Vision (3DV), 25–28 pp 565–571

  24. Nah S, Kim TH, Lee KM (2017) Deep Multi-Scale Convolutional Neural Network for Dynamic Scene Deblurring. In : 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp 257–265

  25. Pan J, Hu Z, Su Z, Lee H-Y, Yang M-H (2016) Soft-segmentation guided object motion deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 459–468

  26. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention: Springer, pp 234–241

  27. Schuler CJ, Hirsch M, Harmeling S, Schölkopf B (2016) Learning to Deblur. IEEE Trans Patt Anal Mach Intell (TPAMI) 38(7):1439–1451

    Article  Google Scholar 

  28. Shan Q, Jia J, Agarwala A (2008) High-quality motion deblurring from a single image. Acm Trans Graphics (tog) 27(3):1–10

    Article  Google Scholar 

  29. Shin H, Roth HR, Gao M, Lu L, Xu Z, Nogues I, Yao J, Mollura D, Summers RM (2016) Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans Med Imaging 35(5):1285–1298

    Article  Google Scholar 

  30. Sim H, Kim M (2019) A Deep Motion Deblurring Network Based on Per-Pixel Adaptive Kernels with Residual Down-Up and Up-Down Modules. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 16–17.pp 2140–2149

  31. Su S, Delbracio M, Wang J, Sapiro G, Heidrich W, Wang O (2017) Deep video deblurring for hand-held cameras." In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1279–1288

  32. Suin M, Purohit K, Rajagopalan AN (2020) Spatially-Attentive Patch-Hierarchical Network for Adaptive Motion Deblurring. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 3603–3612

  33. Sun J, Wenfei C, Zongben X, Ponce J (2015) Learning a Convolutional Neural Network for Non-Uniform Motion Blur Removal. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 7–12 pp 769–777

  34. Tao X, Gao H, Shen X, Wang J, Jia J (2018) Scale-Recurrent Network for Deep Image Deblurring. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 8174–8182

  35. Tien H, Yang H, Shueng P, Chen J (2021) Cone-beam CT image quality improvement using Cycle-Deblur consistent adversarial networks (Cycle-Deblur GAN) for chest CT imaging in breast cancer patients. Sci Rep 11(1):1133. https://doi.org/10.1038/s41598-020-80803-2

    Article  Google Scholar 

  36. Tong T, Li G, Liu X, Gao Q (2017) Image super-resolution using dense skip connections. In: Proceedings of the IEEE international conference on computer vision,, pp 4799–4807

  37. Wang Z, Simoncelli EP, Bovik AC (2003) Multiscale structural similarity for image quality assessment. In: The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, vol. 2: Ieee, pp 1398–1402

  38. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  39. Yang W, Tan RT, Feng J, Liu J, Guo Z, Yan S (2017) Deep joint rain detection and removal from a single image. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 1685-1694

  40. Ye M, Lyu D, Chen G (2020) Scale-iterative upscaling network for image Deblurring. IEEE Access 8:18316–18325. https://doi.org/10.1109/ACCESS.2020.2967823

    Article  Google Scholar 

  41. Zhang K, Zuo W, Chen Y, Meng D, Zhang L (2017) Beyond a Gaussian Denoiser: residual learning of deep CNN for image Denoising. IEEE Trans Image Process (TIP) 26(7):3142–3155

    Article  MathSciNet  MATH  Google Scholar 

  42. Zhang Y, Li K, Li K, Wang L, Zhong B, Fu Y (2018) Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European conference on computer vision (ECCV), pp 286–301

  43. Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y (2018) Residual dense network for image super-resolution.In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2472–2481

  44. Zhang H, Dai Y, Li H, Koniusz P (2019) Deep stacked hierarchical multi-patch network for image deblurring. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 5978–5986

  45. Zhang H, Sindagi V, Patel VM (2020) Image De-raining using a conditional generative adversarial network. IEEE Trans Circuits Syst Video Technol (TCSVT) 30(11):3943–3956

    Article  Google Scholar 

  46. Zou W, Jiang MO, Zhang Y, Chen L, Lu Z, Wu Y (2021) SDWNet: A Straight Dilated Network with Wavelet Transformation for image Deblurring. In: 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), Montreal, BC, Canada, pp 1895-1904

Download references

Acknowledgements

This study was funded by the ministry of Science and Technology in Taiwan under MOST 108-2221-E-011-130-.

The authors would like to thank Mr. Ding-Wei Hu, who participated in revising the manuscript.

Author information

Authors and Affiliations

  1. Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, No. 43, Keelung Rd., Da’an Dist, Taipei, Taiwan

    Ying Cheng Lu, Tzu Pu Liu & Chang Hong Lin

Authors
  1. Ying Cheng Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tzu Pu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chang Hong Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chang Hong Lin.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Data availability

The datasets analyzed during the current study are available from references [17, 24, 31, and [18], respectively.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lu, Y.C., Liu, T.P. & Lin, C.H. Two-stage single image Deblurring network based on deblur kernel estimation. Multimed Tools Appl 82, 17055–17074 (2023). https://doi.org/10.1007/s11042-022-14116-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-14116-z

Keywords

Search

Navigation