Skip to main content
SpringerLink
Log in

Depth-guided asymmetric CycleGAN for rain synthesis and image deraining

  • 1190: Depth-Related Processing and Applications in Visual Systems
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Based on supervised learning, most of the existing single image deraining networks are trained on paired images including one clean image and one rain image. Since it is difficult to obtain a sufficient number of paired images, most of the rain images are manually synthesized from the clean ones. However, it costs huge time and effort, and requires professional experience to mimic the real rain images well. Moreover, the superior performance of these deraining networks trained on manually synthetic rain images is hard to be maintained when tested on real rain images. In this work, to obtain more realistic rain images for training supervised deraining networks, the depth-guided asymmetric CycleGAN (DA-CycleGAN) is proposed to translate clean images to their rainy counterparts automatically. Due to the cycle consistency strategy, DA-CycleGAN can also implement the single image deraining task unsupervised while synthesizing rain on clean images. Since rain streaks and rain mist vary with depth from the camera, DA-CycleGAN adopts depth information as an aid for rain synthesis and deraining. Furthermore, we design generators with different architectures for these two processes due to the information asymmetry in rain synthesis and deraining. Extensive experiments indicate that the DA-CycleGAN can synthesize more lifelike rain images and provide commensurate deraining performance compared with the state-of-the-art deraining methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. http://www.photoshopessentials.com/photo-effects/rain/

References

  1. Arbeláez P, Maire M, Fowlkes C, Malik J (2011) Contour detection and hierarchical image segmentation. IEEE Trans Pattern Anal Mach Intell 33(5):898–916

    Article  Google Scholar 

  2. Chen Y, Hsu C (2013) A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In: 2013 IEEE international conference on computer vision (ICCV), pp 1968–1975

  3. Choi Y, Choi M, Kim M, Ha J, Kim S, Choo J (2018) Stargan: unified generative adversarial networks for multi-domain image-to-image translation. In: 2018 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 8789–8797

  4. Deng J, Dong W, Socher R, Li L, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition (CVPR), pp 248–255

  5. Eigen D, Puhrsch C, Fergus R (2014) Depth map prediction from a single image using a multi-scale deep network. In: Advances in neural information processing systems, pp 2366–2374

  6. Fu X, Huang J, Ding X, Liao Y, Paisley J (2017) Clearing the skies: a deep network architecture for single-image rain removal. IEEE Trans Image Process 26(6):2944–2956

    Article  MathSciNet  Google Scholar 

  7. Fu X, Huang J, Zeng D, Huang Y, Ding X, Paisley J (2017) Removing rain from single images via a deep detail network. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 1715–1723

  8. Garg K, Nayar SK (2006) Photorealistic rendering of rain streaks. In: ACM Trans Graphics, vol 25, pp 996–1002

  9. Gerald S, Michal S (2004) UCID: an uncompressed color image database. In: Storage and retrieval methods and applications for multimedia 2004, vol 5307, pp 472–480

  10. Godard C, Aodha OM, Brostow GJ (2017) Unsupervised monocular depth estimation with left-right consistency. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 6602–6611

  11. Godard C, Aodha OM, Firman M, Brostow G (2019) Digging into self-supervised monocular depth estimation. In: 2019 IEEE/CVF international conference on computer vision (ICCV), pp 3827–3837

  12. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Proceedings of the 27th international conference on neural information processing systems (NIPS), pp 2672–2680

  13. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778

  14. Heusel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S (2017) Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: Neural information processing systems (NIPS)

  15. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(12):1735–80

    Article  Google Scholar 

  16. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications

  17. Hu X, Fu C, Zhu L, Heng P (2019) Depth-attentional features for single-image rain removal. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 8014–8023

  18. Huang X, Liu M, Belongie S, Kautz J (2018) Multimodal unsupervised image-to-image translation. In: European conference on computer vision (ECCV)

  19. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 2261–2269

  20. Huynh-Thu Q, Ghanbari M (2008) Scope of validity of psnr in image/video quality assessment. Electron Lett 44(13):800–801

    Article  Google Scholar 

  21. Iizuka S, Simo-Serra E, Ishikawa H (2017) Globally and locally consistent image completion. ACM Trans Graph 36(4):1–14

    Article  Google Scholar 

  22. Isola P, Zhu J, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 1125–1134

  23. Junbo JZ, Michaël M, Yann L (2016) Energy-based generative adversarial network. coRR

  24. Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: International conference on learning representations (ICLR)

  25. Li R, Cheong L, Tan RT (2019) Heavy rain image restoration: integrating physics model and conditional adversarial learning. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 1633–1642

  26. Li Y, Tan RT, Guo X, Lu J, Brown MS (2016) Rain streak removal using layer priors. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 2736–2744

  27. Li X, Wu J, Lin Z, Liu H, Zha H (2018) Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: European conference on computer vision (ECCV), pp 262–277

  28. Liu M, Breuel T, Kautz J (2017) Unsupervised image-to-image translation networks. coRR

  29. Liu F, Shen C, Lin G (2014) Deep convolutional neural fields for depth estimation from a single image. In: 2014 IEEE conference on computer vision and pattern recognition (CVPR), pp 5162–5170

  30. Luo Y, Xu Y, Ji H (2015) Removing rain from a single image via discriminative sparse coding. In: 2015 IEEE international conference on computer vision (ICCV), pp 3397–3405

  31. Mao X, Li Q, Xie H, Lau RY, Wang Z, Paul Smolley S (2017) Least squares generative adversarial networks. In: 2017 IEEE international conference on computer vision (ICCV), pp 2794–2802

  32. Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L et al (2019) Pytorch: an imperative style, high-performance deep learning library. In: Advances in neural information processing systems, pp 8026–8037

  33. Ren D, Zuo W, Hu Q, Zhu P, Meng D (2019) Progressive image deraining networks: a better and simpler baseline. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 3932–3941

  34. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention (MICCAI), pp 234–241

  35. Saxena A, Chung SH, Ng AY (2006) Learning depth from single monocular images. In: Advances in neural information processing systems, pp 1161–1168

  36. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: International conference on learning representations (ICLR)

  37. Tero K, Timo A, Samuli L, Jaakko L (2017) Progressive growing of gans for improved quality, stability, and variation. coRR

  38. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  39. Wang T, Yang X, Xu K, Chen S, Zhang Q, Lau RWH (2019) Spatial attentive single-image deraining with a high quality real rain dataset. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 12262–12271

  40. Wei Y, Zhang Z, Wang Y, Fan J, Yan S, Wang M (2019) Deraincyclegan: a simple unsupervised network for single image deraining and rainmaking

  41. Wei Y, Zhang Z, Zhang H, Hong R, Wang M (2019) A coarse-to-fine multi-stream hybrid deraining network for single image deraining. In: 2019 IEEE international conference on data mining (ICDM), pp 628–637

  42. Wofk D, Ma F, Yang T, Karaman S, Sze V (2019) Fastdepth: fast monocular depth estimation on embedded systems. In: ICRA, pp 6101–6108

  43. Yang W, Tan RT, Feng J, Liu J, Guo Z, Yan S (2017) Deep joint rain detection and removal from a single image. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 1685–1694

  44. Zhang H, Patel VM (2018) Density-aware single image de-raining using a multi-stream dense network. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 695–704

  45. Zhang H, Sindagi V, Patel VM (2020) Image de-raining using a conditional generative adversarial network. IEEE Trans Circuits Syst Video Technol 30(11):3943–3956

    Article  Google Scholar 

  46. Zhu J, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE international conference on computer vision (ICCV), pp 2242–2251

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (No. 62071500), Shenzhen Science and Technology Program (Grant No. GXWD20201231165807008, 2021A26).

Author information

Authors and Affiliations

  1. School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University, Shenzhen, 518107, Guangdong, China

    Yinhe Qi, Huanrong Zhang, Zhi Jin & Wanquan Liu

  2. Guangdong Provincial Key Laboratory of Fire Science and Technology, Guangzhou, 510006, China

    Zhi Jin

  3. Guangdong Provincial Key Laboratory of Robotics and Digital Intelligent Manufacturing Technology, Guangzhou, 510535, China

    Zhi Jin

Authors
  1. Yinhe Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Huanrong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhi Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wanquan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhi Jin.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qi, Y., Zhang, H., Jin, Z. et al. Depth-guided asymmetric CycleGAN for rain synthesis and image deraining. Multimed Tools Appl 81, 35935–35952 (2022). https://doi.org/10.1007/s11042-022-13342-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13342-9

Keywords

Search

Navigation