Skip to main content
SpringerLink
Log in

MLTDNet: an efficient multi-level transformer network for single image deraining

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Single image rain streaks removal is a great challenging task in computer vision due to the uncertainty of the shape and size of rain streaks. Current methods attempted to adopt complex optimization processes or progressive refinement schemes. But these methods cause a significant impact on the efficiency of many real-time demanding applications. To address this problem, we propose a multi-level transformer deraining network which is an efficient single image rain removal model. Specifically, an efficient deraining network is constructed to extract rain streaks. We then employ cascade networks to extract feature information from deep high-level to shallow low-level layers. In addition, the multi-head self-attention mechanism is applied to extracting global information in the feature map at each level, which can highly improve the representational ability for rain streaks. Experimental results on both synthetic and real-world datasets have demonstrated the efficacy of our method, which uses less time costs and obtains comparable results in comparison to the state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Kim J, Sim J, Kim C (2015) Video deraining and desnowing using temporal correlation and low-rank matrix completion. IEEE Trans Image Process 24(9):2658–2670. https://doi.org/10.1109/TIP.2015.2428933

    Article  MathSciNet  MATH  Google Scholar 

  2. Chen H, Wang Y, Guo T, Xu C, Deng Y, Liu Z, Ma S, Xu C, Xu C, Gao W (2021) Pre-trained image processing transformer. In: IEEE conference on computer vision and pattern recognition, pp. 12299–12310

  3. Ma T, Huang T, Zhao X, Lou Y (2017) Image deblurring with an inaccurate blur kernel using a group-based low-rank image prior. Inf Sci 408:213–233. https://doi.org/10.1016/j.ins.2017.04.049

    Article  Google Scholar 

  4. Alajarmeh A, Salam RA, Abdulrahim K, Marhusin MF, Zaidan AA, Zaidan BB (2018) Real-time framework for image dehazing based on linear transmission and constant-time airlight estimation. Inf Sci 436–437:108–130. https://doi.org/10.1016/j.ins.2018.01.009

    Article  MathSciNet  Google Scholar 

  5. Kang L, Lin C, Fu Y (2012) Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans Image Process 21(4):1742–1755. https://doi.org/10.1109/TIP.2011.2179057

    Article  MathSciNet  MATH  Google Scholar 

  6. Li Y, Tan RT, Guo X, Lu J, Brown MS (2016) Rain streak removal using layer priors, pp. 2736–2744

  7. Fu X, Huang J, Zeng D, Huang Y, Ding X, Paisley JW (2017) Removing rain from single images via a deep detail network. In: IEEE conference on computer vision and pattern recognition, pp. 1715–1723

  8. Li X, Wu J, Lin Z, Liu H, Zha H (2018) Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: Computer Vision - ECCV 2018 - 15th European Conference. Lecture Notes in Computer Science, vol. 11211, pp. 262–277

  9. Yang W, Tan RT, Feng J, Liu J, Guo Z, Yan S (2017) Deep joint rain detection and removal from a single image. In: IEEE conference on computer vision and pattern recognition, pp. 1685–1694

  10. Yang Y, Lu H (2019) Single image deraining via recurrent hierarchy enhancement network. In: Amsaleg, L., Huet, B., Larson, M.A., Gravier, G., Hung, H., Ngo, C., Ooi, W.T. (eds.) Proceedings of the 27th ACM international conference on multimedia, MM, pp. 1814–1822

  11. Ding J, Guo H, Zhou H, Yu J, He X, Jiang B (2021) Distributed feedback network for single-image deraining. Inf Sci 572:611–626. https://doi.org/10.1016/j.ins.2021.02.080

    Article  MathSciNet  Google Scholar 

  12. Li S, Araujo IB, Ren W, Wang Z, Tokuda EK, Junior RH, Junior RMC, Zhang J, Guo X, Cao X (2019) Single image deraining: a comprehensive benchmark analysis. In: IEEE conference on computer vision and pattern recognition, pp. 3838–3847

  13. Wang Y, Song Y, Ma C, Zeng B (2020) Rethinking image deraining via rain streaks and vapors. In: Computer vision—ECCV 2020—16th European conference. Lecture Notes in Computer Science, vol. 12362, pp. 367–382

  14. Wei Y, Zhang Z, Wang Y, Xu M, Yang Y, Yan S, Wang M (2021) Deraincyclegan: Rain attentive cyclegan for single image deraining and rainmaking. IEEE Trans Image Process 30:4788–4801. https://doi.org/10.1109/TIP.2021.3074804

    Article  Google Scholar 

  15. Zhu J, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: IEEE international conference on computer vision, pp. 2242–2251

  16. Wei Y, Zhang Z, Wang Y, Zhang H, Zhao M, Xu M, Wang M (2021) Semi-deraingan: A new semi-supervised single image deraining. In: 2021 IEEE international conference on multimedia and expo (ICME), pp. 1–6 . IEEE

  17. Wei Y, Zhang Z, Zhang H, Hong R, Wang M (2019) A coarse-to-fine multi-stream hybrid deraining network for single image deraining. In: IEEE international conference on data mining, pp. 628–637

  18. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems 30: Annual conference on neural information processing systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998–6008

  19. Wang H, Xie Q, Zhao Q, Liang Y, Meng D (2021) Rcdnet: An interpretable rain convolutional dictionary network for single image deraining. CoRR https://arxiv.org/abs/2107.06808

  20. Guo Q, Sun J, Juefei-Xu F, Ma L, Xie X, Feng W, Liu Y, Zhao J (2021) Efficientderain: Learning pixel-wise dilation filtering for high-efficiency single-image deraining, pp. 1487–1495

  21. Garg K, Nayar SK (2004) Detection and removal of rain from videos. In: IEEE computer society conference on computer vision and pattern recognition, pp. 528–535

  22. Ren W, Tian J, Han Z, Chan AB, Tang Y (2017) Video desnowing and deraining based on matrix decomposition. In: IEEE conference on computer vision and pattern recognition, pp. 2838–2847

  23. Liu J, Yang W, Yang S, Guo Z (2019) D3r-net: dynamic routing residue recurrent network for video rain removal. IEEE Trans Image Process 28(2):699–712. https://doi.org/10.1109/TIP.2018.2869722

    Article  MathSciNet  MATH  Google Scholar 

  24. Wang D, Cui C (2017) Stochastic configuration networks ensemble with heterogeneous features for large-scale data analytics. Inf Sci 417:55–71. https://doi.org/10.1016/j.ins.2017.07.003

    Article  Google Scholar 

  25. Wang D, Li M (2017) Stochastic configuration networks: fundamentals and algorithms. IEEE Trans Cybern 47(10):3466–3479. https://doi.org/10.1109/TCYB.2017.2734043

    Article  Google Scholar 

  26. Ji Y, Zhang H, Gao F, Sun H, Wei H, Wang N, Yang B (2022) LGCNET: A local-to-global context-aware feature augmentation network for salient object detection. Inf Sci 584:399–416. https://doi.org/10.1016/j.ins.2021.10.055

    Article  Google Scholar 

  27. Yang W, Tan RT, Wang S, Fang Y, Liu J (2021) Single image deraining: From model-based to data-driven and beyond. IEEE Trans Pattern Anal Mach Intell 43(11):4059–4077. https://doi.org/10.1109/TPAMI.2020.2995190

    Article  Google Scholar 

  28. Zhang H, Patel VM (2018) Density-aware single image de-raining using a multi-stream dense network. In: IEEE conference on computer vision and pattern recognition, pp. 695–704

  29. Zhang H, Sindagi V, Patel VM (2020) Image de-raining using a conditional generative adversarial network. IEEE Trans Circuits Syst Video Technol 30(11):3943–3956. https://doi.org/10.1109/TCSVT.2019.2920407

    Article  Google Scholar 

  30. Li R, Cheong L, Tan RT (2019) Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 1633–1642. Computer Vision Foundation / IEEE, ??? . https://doi.org/10.1109/CVPR.2019.00173

  31. Mirza M, Osindero S (2014) Conditional generative adversarial nets. CoRR https://arxiv.org/abs/1411.1784

  32. Zhang X, Zhou X, Lin M, Sun J (2018) Shufflenet: An extremely efficient convolutional neural network for mobile devices. In: IEEE conference on computer vision and pattern recognition, pp. 6848–6856

  33. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE conference on computer vision and pattern recognition, pp. 770–778

  34. Fan Z, Wu H, Fu X, Huang Y, Ding X (2018) Residual-guide network for single image deraining. In: ACM multimedia conference on multimedia conference, pp. 1751–1759

  35. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, Uszkoreit J, Houlsby N (2021) An image is worth 16x16 words: transformers for image recognition at scale

  36. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612. https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  37. Ren D, Zuo W, Hu Q, Zhu P, Meng D (2019) Progressive image deraining networks: a better and simpler baseline. In: IEEE conference on computer vision and pattern recognition, pp. 3937–3946

  38. Wang T, Yang X, Xu K, Chen S, Zhang Q, Lau RWH (2019) Spatial attentive single-image deraining with a high quality real rain dataset. In: IEEE conference on computer vision and pattern recognition, pp. 12270–12279

  39. Qian R, Tan RT, Yang W, Su J, Liu J (2018) Attentive generative adversarial network for raindrop removal from a single image. In: IEEE conference on computer vision and pattern recognition, pp. 2482–2491

  40. Huynh-Thu Q, Ghanbari M (2008) Scope of validity of PSNR in image/video quality assessment. Electron Lett 44(13):800–801

    Article  Google Scholar 

  41. Yang W, Tan RT, Feng J, Guo Z, Yan S, Liu J (2020) Joint rain detection and removal from a single image with contextualized deep networks. IEEE Trans Pattern Anal Mach Intell 42(6):1377–1393. https://doi.org/10.1109/TPAMI.2019.2895793

    Article  Google Scholar 

  42. Wei W, Meng D, Zhao Q, Xu Z, Wu Y (2019) Semi-supervised transfer learning for image rain removal. In: IEEE conference on computer vision and pattern recognition, pp. 3877–3886

  43. Wang C, Xing X, Wu Y, Su Z, Chen J (2020) DCSFN: deep cross-scale fusion network for single image rain removal. In: MM ’20: The 28th ACM international conference on multimedia, pp. 1643–1651

  44. Guo Z, Hou M, Sima M, Feng Z (2021) Derainattentiongan: unsupervised single-image deraining using attention-guided generative adversarial networks. Signal Image Video Process 16(1):185–192

    Article  Google Scholar 

  45. Kingma DP, Ba J (2015) Adam: a method for stochastic optimization

Download references

Acknowledgements

This work was supported in part by the project of State Grid Shaanxi Electric Power Company Limited under Grant No. SGSNKYOOSPJS2100221, the Shenzhen Science and Technology Program under Grant no. JCYJ20210324131203009, and the HITSZ-J&A Joint Laboratory of Digital Design and Intelligent Fabrication under Grant no. HITSZ-J&A-2021A01.

Author information

Authors and Affiliations

  1. School of Electrical Engineering, Xi’an Jiaotong University, Xi’an, 710048, Shaanxi, China

    Feng Gao, Shengchang Ji & Jie Guo

  2. Department of Computer Science, Harbin Institute of Technology, Shenzhen, 518055, Guangdong, China

    Xiangyu Mu, Chao Ouyang & Kai Yang

  3. State grid Shaanxi Electric power Research Institute, Xi’an, 710100, Shaanxi, China

    Feng Gao, Haokun Wei & Nan Wang

  4. Jiangsu Fudi Electric Technology Co., Ltd., Nantong, 226253, Jiangsu, China

    Lei Ma

  5. School of Architecture, Harbin Institute of Technology, Shenzhen, 518055, Guangdong, China

    Biao Yang

Authors
  1. Feng Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiangyu Mu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chao Ouyang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kai Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shengchang Ji
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jie Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Haokun Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Nan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Lei Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Biao Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiangyu Mu.

Ethics declarations

Declarations

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gao, F., Mu, X., Ouyang, C. et al. MLTDNet: an efficient multi-level transformer network for single image deraining. Neural Comput & Applic 34, 14013–14027 (2022). https://doi.org/10.1007/s00521-022-07226-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07226-0

Keywords

Search

Navigation