Skip to main content
SpringerLink
Log in

WTransU-Net: Wiener deconvolution meets multi-scale transformer-based U-net for image deblurring

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Deblurring is a classical image restoration problem. Although recent methods have shown promising deblurring performance, most methods still cannot effectively balance the texture details restoration and model complexity. In order to improve the performance of deblurring, some models are designed to be more complex. In this work, a simple and efficient Wiener deconvolution and multi-scale transformer-based U-Net (WTransU-Net) is proposed to tackle these problems. First, the proposed Wiener feature extraction module uses explicit Wiener deconvolution to extract the Wiener features in the deep feature space. Then, the obtained Wiener features are input into a multi-scale feature reconstruction module which only embeds one transformer refining block in each scale of the U-Net to deblur the image from local and global perspectives. In addition, a multi-scale hybrid loss function is designed to train the WTransU-Net in an end-to-end manner to better learn the content and texture details. The experimental results on benchmark datasets show that compared with the state-of-the-art deblurring methods, the proposed WTransU-Net can achieve better performance with fewer artifacts in terms of quantitatively and qualitatively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data availability

This article uses public datasets, which are available on their official website

References

  1. Karacan, K., Uyar, T., Tunga, B., et al.: A novel multistage CAD system for breast cancer diagnosis. Signal Image Video Process. 17, 1–10 (2023)

    Article  Google Scholar 

  2. Nie, C., Qadar, M.A., Zhou, S., et al.: Transnational image object detection datasets from nighttime driving. Signal Image Video Process. 17, 1–9 (2022)

    Google Scholar 

  3. Chong Mou, Qian Wang, and Jian Zhang., et al. (2022): Deep generalized unfolding networks for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2204–13348 (2022)

  4. Rim, J., Kim, G., Kim, J., et al.: Realistic blur synthesis for learning image deblurring. In: European Conference Computer Vision LNCS,v13667, pp. 487–503( 2022)

  5. Shyam, P., Kim, K.S., Yoon, K.J.: GIQE: Generic image quality enhancement via Nth order iterative degradation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2077–2087(2022)

  6. Kaufman, A., Fattal, R.: Deblurring using analysis-synthesis networks pair. In: IEEE Conference Computer Vision Pattern Recognition, pp. 5811–5820 (2020)

  7. Fang, Y., Zhang, H., Wong, H.S., et al.: A robust non-blind deblurring method using deep denoiser prior. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 735–744 (2022)

  8. Guan, Z., Tsai, E.H.R., Huang, X., et al.: Non-blind deblurring for fluorescence: A deformable latent space approach with Kernel parameterization. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision ,pp. 711–719 (2022)

  9. Dong, J., Pan, J., Sun, D., et al.: Learning data terms for non-blind deblurring. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 748–763 (2018)

  10. Vasu, S., Maligireddy, V.R., Rajagopalan, A.N.: Non-blind deblurring: Handling Kernel uncertainty with CNNs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3272-3281 (2018)

  11. Son, H., Lee, S.: Fast non-blind deconvolution via regularized residual networks with long/short skip-connections, In: 2017 IEEE International Conference on Computational Photography (ICCP), 1–10 (2017)

  12. Chakrabarti A.: A neural approach to blind motion deblurring. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, 11-14, 2016, Proceedings, Part III 14 2016 , pp. 221–235(2016)

  13. Sun, J., Cao, W., Xu, Z., et al.: Learning a convolutional neural network for non-uniform motion blur removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 769–777 (2015)

  14. Sankhe, P.D., Patil, M., Margaret, M.: Deblurring of grayscale images using inverse and Wiener filter. In: Proceedings of the International Conference and Workshop on Emerging Trends in Technology, 145–148 (2011)

  15. Mbarki, Z., Seddik, H., Braiek, E.B.: Non blind image restoration scheme combining parametric wiener filtering and BM3D denoising technique. In: 2018 4th International Conference on Advanced Technologies for Signal and Image Processing (ATSIP), pp. 1–5 (2018)

  16. Wiener, N., Mathematician, C., et al.: Extrapolation, interpolation, and smoothing of stationary time series: with engineering applications. MIT press, Cambridge (1949)

    Book  Google Scholar 

  17. Krishnan, D., Fergus, R.: Fast image deconvolution using hyper-Laplacian priors. Adv. Neural Inf. Process. Syst. 22, 1033–1041 (2009)

    Google Scholar 

  18. Levin, A., Weiss, Y., Durand, F., et al.: Understanding and evaluating blind deconvolution algorithms. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1964–1971 (2009)

  19. Levin, A., Fergus, R., Durand, F., et al.: Image and depth from a conventional camera with a coded aperture. Proc ACM SIGGRAPH Conf Computer Graph. 26(3), 70 (2007)

    Article  Google Scholar 

  20. Schmidt, U., Jancsary, J., Nowozin, S., et al.: Cascades of regression tree fields for image restoration. IEEE Trans. Pattern Anal. Mach. Intell. 38(4), 677–689 (2015)

    Article  Google Scholar 

  21. Roth, S., Black, M.J.: Fields of experts. Int. J. Comput. Vis. 82(2), 205–229 (2009)

    Article  MATH  Google Scholar 

  22. Zoran, D., Weiss, Y.: From learning models of natural image patches to whole image restoration. In: 2011 International Conference on Computer Vision , pp. 479–486 (2011)

  23. Xu, L., Ren, J.S., Liu, C., et al.: Deep convolutional neural network for image deconvolution. Adv. Neural Inf. Process. Syst. 27, 1790–1798 (2014)

    Google Scholar 

  24. Ren, W., Zhang, J., Ma, L., et al.: Deep non-blind deconvolution via generalized low-rank approximation. Adv. Neural Inf. Process. Syst. 31, 297–307 (2018)

    Google Scholar 

  25. Zamir, S.W., Arora, A., Khan, S., et al.: Multi-stage progressive image restoration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14816–14826 (2021)

  26. Zhang, Z., Cheng, Y., Suo, J., et al.: INFWIDE: Image and feature space Wiener deconvolution network for non-blind image deblurring in low-light conditions. IEEE Trans. Image Process. 32, 1390–1402 (2023)

    Article  Google Scholar 

  27. Nah, S., Hyun Kim, T., Mu Lee, K.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3883–3891 (2017)

  28. Dong, J., Roth, S., Schiele, B.: Deep wiener deconvolution: Wiener meets deep learning for image deblurring. Adv. Neural Inf. Process. Syst. 33, 1048–1059 (2020)

    Google Scholar 

  29. Vaswani, A., Shazeer, N., Parmar, N., et al.: Attention is all you need. Adv. Neural Inf. Process, Syst (2017)

  30. Zamir, S.W., Arora, A., Khan, S., et al.: Restormer: Efficient transformer for high-resolution image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5728–5739 (2022)

  31. Corbineau, M.C., Bertocchi, C., Chouzenoux, E., et al.: Learned image deblurring by unfolding a proximal interior point algorithm. In: 2019 IEEE International Conference on Image Processing (ICIP), pp. 4664–4668(2019)

  32. Leung, C., Lu, W.S.: A modified Wiener filter for the restoration of blurred images. In: Proceedings of IEEE Pacific Rim Conference on Communications Computers and Signal Processing, PACRIM - Proc., 1: 166–169(1993)

  33. Cho, S.J., Ji, S.W., Hong, J.P., et al.: Rethinking coarse-to-fine approach in single image deblurring. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4641–4650 (2021)

  34. Wang, Z., Cun, X., Bao, J., et al.: Uformer: A general u-shaped transformer for image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.17683–17693(2022)

  35. Abuolaim, A., Brown, M.S.: Defocus deblurring using dual-pixel data. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part X 16, pp. 111–126(2020)

  36. Suin, M., Purohit, K., Rajagopalan, A.N.: Spatially-attentive patch-hierarchical network for adaptive motion deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3606–3615 (2020)

  37. Zhang, D., Wang, X.: Dynamic multi-scale network for dual-pixel images defocus deblurring with transformer. In: 2022 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6 (2022)

  38. Wang, Z., Simoncelli, E.P., Bovik, A.C.: Multiscale structural similarity for image quality assessment. In: The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, Vol. 2, pp. 1398–1402 (2003)

  39. Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Lecture Notes Computer Science, pp. 234–241 (2015)

  40. Ma, K., Duanmu, Z., Wu, Q., et al.: Waterloo exploration database: New challenges for image quality assessment models. IEEE Trans. Image Process. 26(2), 1004–1016 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  41. Martin, D., Fowlkes, C., Tal, D., et al.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. IEEE Int. Conf. Comput. Vis. 2, 416–423 (2001)

    Google Scholar 

  42. Shen, Z., Wang, W., Lu, X., et al.: Human-aware motion deblurring. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5572–5581 (2019)

  43. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, (2014)

  44. Tao, X., Gao, H., Shen, X., et al.: Scale-recurrent network for deep image deblurring. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8174–8182 (2018)

  45. Zhang, H., Dai, Y., Li, H., et al.: Deep stacked hierarchical multi-patch network for image deblurring. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5978–5986 (2019)

  46. Kim, K., Lee, S., Cho, S.: MSSNet: Multi-scale-stage network for single image deblurring. LNCS 13802, 524–539 (2023)

    Google Scholar 

  47. Gong, D., Zhang, Z., Shi, Q., et al.: Learning deep gradient descent optimization for image deconvolution. IEEE Trans. Neural Netw. Learn. Syst. 31(12), 5468–5482 (2020)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

This work was supported by Foundation of HubeiKey Laboratory of Metallurgical Industry Process System Science (No. Y202008) and National Natural Science Foundation of China (No. 61671338).

Author information

Authors and Affiliations

  1. College of Science, Wuhan University of Science and Technology, Wuhan, 430081, China

    Shixin Zhao, Yuanxiu Xing & Hongyang Xu

  2. Hubei Province Key Laboratory of Systems Science in Metallurgical Process, Wuhan, 430081, China

    Shixin Zhao, Yuanxiu Xing & Hongyang Xu

Authors
  1. Shixin Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuanxiu Xing
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hongyang Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

SZ and YX proposed the algorithm and wrote the main manuscript, and HX participated in the design of the network. All authors reviewed the manuscript.

Corresponding author

Correspondence to Yuanxiu Xing.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This article does not involve ethical issues.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, S., Xing, Y. & Xu, H. WTransU-Net: Wiener deconvolution meets multi-scale transformer-based U-net for image deblurring. SIViP 17, 4265–4273 (2023). https://doi.org/10.1007/s11760-023-02659-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-023-02659-z

Keywords

Search

Navigation