Skip to main content

Advertisement

Springer Nature Link
Log in

Deep single image deraining using an asymmetric cyclic generative and adversarial framework

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

In reality, rain and fog are often present simultaneously, which can greatly reduce the clarity and quality of scene images. However, most unsupervised single image deraining methods primarily concentrate on removing rain streaks and disregard the presence of fog, which often leads to low deraining performance. In addition, these methods generate samples that are too similar and lack diversity, resulting in poor performance when dealing with complex rain scenes. To address the above issues, we propose a novel Asymmetric Cyclic Generative and Adversarial Framework (ACGF) for single image deraining in which the deraining model, which consists of a Rain-fog2Clean (R2C) transformation block and a Clean2Rain-fog (C2R) transformation block, is trained on both synthetic and real rainy images to simultaneously capture both rain streaks and fog features. To better characterize combined rain–fog features in the R2C block, we propose an attention-based rain–fog feature extraction (ARFE) network to exploit the self-similarity of global and local rain–fog information by learning spatial feature correlations. Furthermore, to improve the translational capacity of the C2R block and the diversity of the model on the synthetic rain conversion path, we design a rain–fog feature decoupling and reorganization (RFDR) network by embedding a rainy image degradation model and a mixed discriminator to preserve richer texture details. Extensive experiments on benchmark rain–fog, rain and fog datasets show that our ACGF outperforms state-of-the-art deraining methods.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Algorithm 1
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Xu J, Wang W, Wang H, Guo J (2020) Multi-model ensemble with rich spatial information for object detection. Pattern Recog 99:107098

    Article  Google Scholar 

  2. Chen J, Chen Y, Li W, Ning G, Tong M, Hilton A (2021) Channel and spatial attention based deep object co-segmentation. Knowl-Based Syst 211:106550

    Article  Google Scholar 

  3. Hu X, Fu C-W, Zhu L, Heng P-A (2019) Depth-attentional features for single-image rain removal. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8022–8031

  4. Li R, Cheong L-F, Tan RT (2019) Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1633–1642

  5. Nayar SK, Narasimhan SG (1999) Vision in bad weather. In: Proceedings of the seventh IEEE international conference on computer vision, vol 2, pp 820–827. IEEE

  6. Yang W, Tan R.T, Feng J, Liu J, Guo Z, Yan S (2017) Deep joint rain detection and removal from a single image. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1357–1366

  7. Lan R, Hu X, Pang C, Liu Z, Luo X (2020) Multi-scale single image rain removal using a squeeze-and-excitation residual network. Appl Soft Comput 92:106296

    Article  Google Scholar 

  8. Wang H, Wu Y, Xie Q, Zhao Q, Liang Y, Zhang S, Meng D (2021) Structural residual learning for single image rain removal. Knowl-Based Syst 213:106595

    Article  Google Scholar 

  9. Wan Y, Cheng Y, Shao M, Gonzàlez J (2022) Image rain removal and illumination enhancement done in one go. Knowl-Based Syst 252:109244

    Article  Google Scholar 

  10. Wei W, Meng D, Zhao Q, Xu Z, Wu Y (2020) Semi-supervised transfer learning for image rain removal. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  11. Wei Y, Zhang Z, Wang Y, Xu M, Wang M (2021) Deraincyclegan: Rain attentive cyclegan for single image deraining and rainmaking. IEEE Trans Image Process PP(99)

  12. Yang W, Tan RT, Feng J, Guo Z, Yan S, Liu J (2020) Joint rain detection and removal from a single image with contextualized deep networks. IEEE Trans Pattern Anal Mach Intell 42:1377–1393

    Article  Google Scholar 

  13. Li R, Tan RT, Cheong LF (2020) All in one bad weather removal using architectural search. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  14. Zhu H, Peng X, Zhou JT, Yang S, Chanderasekh V, Li L, Lim J-H (2019) Singe image rain removal with unpaired information: A differentiable programming perspective. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 9332–9339

  15. Ye Y, Yu C, Chang Y, Zhu L, Zhao X-L, Yan L, Tian Y (2022) Unsupervised deraining: Where contrastive learning meets self-similarity. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 5821–5830

  16. Jiang N, Luo J, Lin J, Chen W, Zhao T (2023) Lightweight semi-supervised network for single image rain removal. Pattern Recog 137:109277

    Article  Google Scholar 

  17. Zhu J-Y, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision, pp 2223–2232

  18. Ahn N, Jo SY, Kang S-J (2021) Eagnet: Elementwise attentive gating network-based single image de-raining with rain simplification. IEEE Transactions on Circuits and Systems for Video Technology

  19. Ye Y, Chang Y, Zhou H, Yan L (2021) Closing the loop: Joint rain generation and removal via disentangled image translation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2053–2062

  20. Zeng Y, Fu J, Chao H, Guo B (2019) Learning pyramid-context encoder network for high-quality image inpainting. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1486–1494

  21. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: 3rd International conference on learning representations (ICLR 2015). Computational and Biological Learning Society

  22. Li G, Xie Y, Lin L, Yu Y (2017) Instance-level salient object segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2386–2395

  23. Zhang H, Sindagi V, Patel VM (2019) Image de-raining using a conditional generative adversarial network. IEEE Trans Circuits Syst Video Technol 30(11):3943–3956

    Article  Google Scholar 

  24. Zhang H, Patel VM (2018) Density-aware single image de-raining using a multi-stream dense network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 695–704

  25. Fu X, Huang J, Zeng D, Yue H, Paisley J (2017) Removing rain from single images via a deep detail network. In: IEEE conference on computer vision and pattern recognition

  26. Li B, Ren W, Fu D, Tao D, Feng D, Zeng W, Wang Z (2018) Benchmarking single-image dehazing and beyond. IEEE Trans Image Process 28(1):492–505

    Article  MathSciNet  Google Scholar 

  27. Zhang Y, Ding L, Sharma G (2017) Hazerd: an outdoor scene dataset and benchmark for single image dehazing. In: 2017 IEEE international conference on image processing (ICIP), pp 3205–3209. IEEE

  28. Ancuti CO, Ancuti C, Timofte R, De Vleeschouwer C (2018) O-haze: a dehazing benchmark with real hazy and haze-free outdoor images. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 754–762

  29. Choi LK, You J, Bovik AC (2015) Referenceless prediction of perceptual fog density and perceptual image defogging. IEEE Trans Image Process 24(11):3888–3901

    Article  MathSciNet  Google Scholar 

  30. Wang C, Xing X, Wu Y, Su Z, Chen J (2020) Dcsfn: Deep cross-scale fusion network for single image rain removal. In: Proceedings of the 28th ACM international conference on multimedia, pp 1643–1651

  31. Zamir SW, Arora A, Khan S, Hayat M, Khan FS, Yang M-H, Shao L (2021) Multi-stage progressive image restoration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 14821–14831

  32. Wang T, Yang X, Xu K, Chen S, Zhang Q, Lau RW (2019) Spatial attentive single-image deraining with a high quality real rain dataset. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 12270–12279

  33. Fu X, Liang B, Huang Y, Ding X, Paisley J (2019) Lightweight pyramid networks for image deraining. IEEE Trans Neural Netw Learn Syst 31(6):1794–1807

    Article  Google Scholar 

  34. Qin X, Wang Z, Bai Y, Xie X, Jia H (2020) Ffa-net: Feature fusion attention network for single image dehazing. In: Proceedings of the AAAI conference on artificial intelligence, vol 34, pp 11908–11915

  35. Shao Y, Li L, Ren W, Gao C, Sang N (2020) Domain adaptation for image dehazing. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2808–2817

  36. Liu X, Ma Y, Shi Z, Chen J (2019) Griddehazenet: Attention-based multi-scale network for image dehazing. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 7314–7323

  37. Cai B, Xu X, Jia K, Qing C, Tao D (2016) Dehazenet: An end-to-end system for single image haze removal. IEEE Trans Image Process 25(11):5187–5198

    Article  MathSciNet  Google Scholar 

  38. Qu Y, Chen Y, Huang J, Xie Y (2019) Enhanced pix2pix dehazing network. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8160–8168

  39. Chen D, He M, Fan Q, Liao J, Zhang L, Hou D, Yuan L, Hua G (2019) Gated context aggregation network for image dehazing and deraining. In: 2019 IEEE winter conference on applications of computer vision (WACV), pp 1375–1383.IEEE

  40. Wang Z (2004) Image quality assessment : From error visibility to structural similarity. IEEE Trans Image Process

  41. Zhang R, Isola P, Efros AA, Shechtman E, Wang O (2018) The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 586–595

  42. Mittal A, Soundararajan R, Bovik AC (2012) Making a “completely blind’ image quality analyzer. IEEE Signal Process Lett 20(3):209–212

    Article  Google Scholar 

  43. Liu L, Liu B, Huang H, Bovik AC (2014) No-reference image quality assessment based on spatial and spectral entropies. Signal Process Image Commun 29(8):856–863

    Article  Google Scholar 

  44. Mittal A, Moorthy AK, Bovik AC (2012) No-reference image quality assessment in the spatial domain. IEEE Trans Image Process 21(12):4695–4708

    Article  MathSciNet  Google Scholar 

  45. Moorthy AK, Bovik AC (2010) A two-step framework for constructing blind image quality indices. IEEE Signal Process Lett 17(5):513–516

    Article  Google Scholar 

  46. Moorthy AK, Bovik AC (2011) Blind image quality assessment: From natural scene statistics to perceptual quality. IEEE Trans Image Process 20(12):3350–3364

    Article  MathSciNet  Google Scholar 

  47. Huynh-Thu Q, Ghanbari M (2008) Scope of validity of psnr in image/video quality assessment. Electron Lett 44(13):800–801

    Article  Google Scholar 

  48. Yang W, Tan RT, Feng J, Liu J, Guo Z, Yan S (2017) Deep joint rain detection and removal from a single image. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1357–1366

  49. Fu X, Huang J, Ding X, Liao Y, Paisley J (2017) Clearing the skies: A deep network architecture for single-image rain removal. IEEE Trans Image Process 26(6):2944–2956

    Article  MathSciNet  Google Scholar 

  50. Fan X, Cao P, Shi P, Chen X, Zhou X, Gong Q (2022) An underwater dam crack image segmentation method based on multi-level adversarial transfer learning. Neurocomputing 505:19–29

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (Grant numbers [52371373, 62001334]), Fund of National Engineering Research Center for Water Transport Safety [No. A202402] and Technology Innovation and Development Project of Supports Enterprise of Hubei Province (Grant numbers [2021BLB172]).

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Wuhan Institute of Technology, Wuhan, 430205, Hubei, China

    Wei Liu, Caiwang Zhang, Cheng Chen, Xiaoyu Huang & Minghui Li

  2. Hubei Key Laboratory of Intelligent Robot, Wuhan Institute of Technology, Wuhan, 430205, Hubei, China

    Wei Liu, Caiwang Zhang, Cheng Chen, Xiaoyu Huang & Minghui Li

Authors
  1. Wei Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Caiwang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cheng Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaoyu Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Minghui Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wei Liu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, W., Zhang, C., Chen, C. et al. Deep single image deraining using an asymmetric cyclic generative and adversarial framework. Appl Intell 54, 6776–6798 (2024). https://doi.org/10.1007/s10489-024-05494-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-024-05494-y

Keywords