Skip to main content
SpringerLink
Log in

DerainGAN: Single image deraining using wasserstein GAN

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Rainy weather greatly affects the visibility of salient objects and scenes in the captured images and videos. The object/scene visibility varies with the type of raindrops, i.e. adherent rain droplets, streaks, rain, mist, etc. Moreover, they pose multifaceted challenges to detect and remove the raindrops to reconstruct the rain-free image for higher-level tasks like object detection, road segmentation etc. Recently, both Convolutional Neural Networks (CNN) and Generative Adversarial Network (GAN) based models have been designed to remove rain droplets from a single image by dealing with it as an image to image mapping problem. However, most of them fail to capture the complexities of the task, create blurry output, or are not time efficient. GANs are a prime candidate for solving this problem as they are extremely effective in learning image maps without harsh overfitting. In this paper, we design a simple yet effective ‘DerainGAN’ framework to achieve improved deraining performance over the existing state-of-the-art methods. The learning is based on a Wasserstein GAN and perceptual loss incorporated into the architecture. We empirically analyze the effect of different parameter choices to train the model for better optimization. We also identify the strengths and limitations of various components for single image deraining by performing multiple ablation studies on our model. The robustness of the proposed method is evaluated over two synthetic and one real-world rainy image datasets using Peak Signal to Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) values. The proposed DerainGAN significantly outperforms almost all state-of-the-art models in Rain100L and Rain700 datasets, both in semantic and visual appearance, achieving SSIM of 0.8201 and PSNR 24.15 in Rain700 and SSIM of 0.8701 and PSNR of 28.30 in Rain100L. This accounts for an average improvement of 10 percent in PSNR and 20 percent in SSIM over benchmarked methods. Moreover, the DerainGAN is one of the fastest methods in terms of time taken to process the image, giving it over 0.1 to 150 seconds of advantage in some cases.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Ali A, Zhu Y, Chen Q, Yu J, Cai H (2019) Leveraging spatio-temporal patterns for predicting citywide traffic crowd flows using deep hybrid neural networks. In: 2019 IEEE 25th International Conference on Parallel and Distributed Systems (ICPADS), IEEE, pp 125–132

  2. Ali A, Zhu Y, Zakarya M (2021) A data aggregation based approach to exploit dynamic spatio-temporal correlations for citywide crowd flows prediction in fog computing. Multimed Tools Appl 1–33

  3. Arbelaez P, Maire M, Fowlkes C, Malik J (2010) Contour detection and hierarchical image segmentation. IEEE transactions on pattern analysis and machine intelligence 33(5):898–916

    Article  Google Scholar 

  4. Arjovsky M, Chintala S, Bottou L (2017) Wasserstein generative adversarial networks. In: Precup D, Teh YW (eds) Proceedings of the 34th International Conference on Machine Learning, PMLR, International Convention Centre, Sydney, Australia. Proceedings of Machine Learning Research 70:214–223. http://proceedings.mlr.press/v70/arjovsky17a.html

  5. Awan N, Ali A, Khan F, Zakarya M, Alturki R, Kundi M, Alshehri MD, Haleem M (2021) Modeling dynamic spatio-temporal correlations for urban traffic flows prediction. IEEE Access 9:26502–26511

    Article  Google Scholar 

  6. Barnum PC, Narasimhan S, Kanade T (2010) Analysis of rain and snow in frequency space. Int J Comput Vis 86(2–3):256

    Article  Google Scholar 

  7. Bi X, Xing J (2020) Multi-scale weighted fusion attentive generative adversarial network for single image de-raining. IEEE Access 8:69838–69848

    Article  Google Scholar 

  8. Bossu J, Hautière N, Tarel JP (2011) Rain or snow detection in image sequences through use of a histogram of orientation of streaks. Int J Comput Vis 93(3):348–367

    Article  Google Scholar 

  9. Chen YL, Hsu CT (2013) A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In: Proceedings of the IEEE International Conference on Computer Vision 1968–1975

  10. Chui KT, Liu RW, Zhao M, De Pablos PO (2020) Predicting students performance with school and family tutoring using generative adversarial network-based deep support vector machine. IEEE Access 8:86745–86752. https://doi.org/10.1109/ACCESS.2020.2992869

  11. Deng LJ, Huang TZ, Zhao XL, Jiang TX (2018) A directional global sparse model for single image rain removal. Appl Math Model 59:662–679

    Article  MathSciNet  Google Scholar 

  12. Du Y, Xu J, Zhen X, Cheng MM, Shao L (2020) Conditional variational image deraining. IEEE Trans Image Process

  13. Eigen D, Krishnan D, Fergus R (2013) Restoring an image taken through a window covered with dirt or rain. In: Proceedings of the IEEE international conference on computer vision 633–640

  14. Fu X, Huang J, Ding X, Liao Y, Paisley J (2017) Clearing the skies: A deep network architecture for single-image rain removal. IEEE Trans Image Process 26(6):2944–2956

    Article  MathSciNet  Google Scholar 

  15. Fu X, Huang J, Zeng D, Huang Y, Ding X, Paisley J (2017b) Removing rain from single images via a deep detail network. In: Proc IEEE Conf Comput Vis Pattern Recognit 3855–3863

  16. Fu X, Liang B, Huang Y, Ding X, Paisley J (2019) Lightweight pyramid networks for image deraining. IEEE transactions on neural networks and learning systems

  17. Garg K, Nayar SK (2004) Detection and removal of rain from videos. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., IEEE, vol 1, pp I–I

  18. Garg K, Nayar SK (2005) When does a camera see rain? In: Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1, IEEE 2:1067–1074

  19. Garg K, Nayar SK (2006) Photorealistic rendering of rain streaks. ACM Trans Graph (TOG) 25(3):996–1002

    Article  Google Scholar 

  20. Gu S, Meng D, Zuo W, Zhang L (2017) Joint convolutional analysis and synthesis sparse representation for single image layer separation. In: Proceedings of the IEEE International Conference on Computer Vision 1708–1716

  21. Hang R, Zhou F, Liu Q, Ghamisi P (2020) Classification of hyperspectral images via multitask generative adversarial networks. IEEE Trans Geosci Remote Sens 1–13. https://doi.org/10.1109/TGRS.2020.3003341

  22. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  23. Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proc IEEE Conf Comput. Vis Pattern Recognit 1125–1134

  24. Jin X, Chen Z, Li W (2020) Ai-gan: Asynchronous interactive generative adversarial network for single image rain removal. Pattern Recogn 100:107143

  25. Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: European conference on computer vision, Springer 694–711

  26. Kang LW, Lin CW, Fu YH (2011) Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans Image Process 21(4):1742–1755

    Article  MathSciNet  Google Scholar 

  27. Kim JH, Lee C, Sim JY, Kim CS (2013) Single-image deraining using an adaptive nonlocal means filter. In: 2013 IEEE International Conference on Image Processing, IEEE 914–917

  28. Kim JH, Sim JY, Kim CS (2015) Video deraining and desnowing using temporal correlation and low-rank matrix completion. IEEE Trans Image Process 24(9):2658–2670

    Article  MathSciNet  Google Scholar 

  29. Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv: 14126980

  30. LeCun Y, Bengio Y et al (1995) (1995) Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks 3361(10):1995

    Google Scholar 

  31. Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proc IEEE Conf Comput Vis Pattern Recognit 4681–4690

  32. Li S, Araujo IB, Ren W, Wang Z, Tokuda EK, Junior RH, Cesar-Junior R, Zhang J, Guo X, Cao X (2019) Single image deraining: A comprehensive benchmark analysis. In: Proc IEEE Conf Comput Vis Pattern Recognit 3838–3847

  33. Li X, Wu J, Lin Z, Liu H, Zha H (2018) Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: Proceedings of the European Conference on Computer Vision (ECCV) 254–269

  34. Li Y, Tan RT, Guo X, Lu J, Brown MS (2016) Rain streak removal using layer priors. In: Proc IEEE Conf Comput Vis Pattern Recognit 2736–2744

  35. Luo Y, Xu Y, Ji H (2015) Removing rain from a single image via discriminative sparse coding. In: Proceedings of the IEEE International Conference on Computer Vision 3397–3405

  36. Martin D, Fowlkes C, Tal D, Malik J (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings Eighth IEEE International Conference on Computer Vision ICCV IEEE 2:416–423

  37. Mehra A, Narang P, Mandal M (2021) Theianet: Towards fast and inexpensive cnn design choices for image dehazing. J Vis Commun Image Represent 77:103137

  38. Mehta A, Sinha H, Narang P, Mandal M (2020) Hidegan: A hyperspectral-guided image dehazing gan. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops 212–213

  39. Mehta A, Sinha H, Mandal M, Narang P (2021) Domain-aware unsupervised hyperspectral reconstruction for aerial image dehazing. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision 413–422

  40. Qian R, Tan RT, Yang W, Su J, Liu J (2018) Attentive generative adversarial network for raindrop removal from a single image. In: Proc IEEE Conf Comput Vis Pattern Recognit 2482–2491

  41. Radford A, Metz L, Chintala S (2015) Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 151106434

  42. Ren D, Zuo W, Hu Q, Zhu P, Meng D (2019) Progressive image deraining networks: a better and simpler baseline. In: Proc IEEE Conf Comput Vis Pattern Recognit 3937–3946

  43. Ren Y, Li S, Nie M, Li C (2020) Single image de-raining via improved generative adversarial nets. Sensors 20(6):1591

    Article  Google Scholar 

  44. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. Springer 234–241

  45. Schaefer G, Stich M (2003) Ucid: An uncompressed color image database. In: Storage and Retrieval Methods and Applications for Multimedia 2004. International Society for Optics and Photonics 5307:472–480

  46. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 14091556

  47. Sinha H, Mehta A, Mandal M, Narang P (2021) Learning to enhance visual quality via hyperspectral domain mapping. Proceedings of the AAAI Conference on Artificial Intelligence 35:15895–15896

    Google Scholar 

  48. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958

    MathSciNet  MATH  Google Scholar 

  49. Wang C, Wu Y, Su Z, Chen J (2020a) Joint self-attention and scale-aggregation for self-calibrated deraining network. In: Proceedings of the 28th ACM International Conference on Multimedia 2517–2525

  50. Wang Y, Liu S, Xie D, Zeng B (2020b) Removing rain streaks by a linear model. IEEE Access 8:54802–54815

    Article  Google Scholar 

  51. Wang YT, Zhao XL, Jiang TX, Deng LJ, Chang Y, Huang TZ (2020c) Rain streaks removal for single image via kernel-guided convolutional neural network. IEEE Transactions on Neural Networks and Learning Systems

  52. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP et al (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  53. Xu B, Wang N, Chen T, Li M (2015) Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv: 150500853

  54. Yang W, Tan RT, Feng J, Liu J, Guo Z, Yan S (2017) Deep joint rain detection and removal from a single image. In: Proc IEEE Conf Comput Vis Pattern Recognit 1357–1366

  55. Yasarla R, Patel VM (2020) Confidence measure guided single image de-raining. IEEE Trans Image Process 29:4544–4555

    Article  Google Scholar 

  56. Zhang D, Khoreva A (2019) Progressive augmentation of gans. In: Adv Neural Inf Proces Syst 6249–6259

  57. Zhang H, Patel VM (2017) Convolutional sparse and low-rank coding-based rain streak removal. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE 1259–1267

  58. Zhang H, Patel VM (2018) Density-aware single image de-raining using a multi-stream dense network. In: Proc IEEE Conf Comput Vis Pattern Recognit 695–704

  59. Zhang H, Sindagi V, Patel VM (2019) Image de-raining using a conditional generative adversarial network. IEEE Trans Circuits Syst Video Technol

  60. Zhang Z, Liu W, Ma H, Liu X (2017) Going clear from misty rain in dark channel guided network. International Joint Conference on Artificial Intelligence - Workshop on AI for Internet of Things

  61. Zheng Q, Shi B, Jiang X, Duan LY, Kot AC (2019) Denoising adversarial networks for rain removal and reflection removal. In: 2019 IEEE International Conference on Image Processing (ICIP), IEEE 2766–2770

  62. Zheng X, Liao Y, Guo W, Fu X, Ding X (2013) Single-image-based rain and snow removal using multi-guided filter. In: International Conference on Neural Information Processing, Springer 258–265

  63. Zhu L, Fu CW, Lischinski D, Heng PA (2017) Joint bi-layer optimization for single-image rain streak removal. In: Proceedings of the IEEE International Conference on Computer Vision 2526–2534

Download references

Acknowledgements

This work is supported by BITS Additional Competitive Research Grant funding under Project Grant File no. PLN/AD/2018-19/5 for the Project titled “Disaster Monitoring from Aerial Imagery using Deep Learning”. The authors gratefully acknowledge the support of NVIDIA Corporation for the donation of the Titan Xp GPU used in this research, and the support of IBM for providing with online Power9 GPU server grant.

Author information

Authors and Affiliations

  1. Department of CSIS, BITS Pilani, Rajasthan, India

    Sahil Yadav, Aryan Mehra, Honnesh Rohmetra & Pratik Narang

  2. Department of ECE, MNIT, Jaipur, Rajasthan, India

    Rahul Ratnakumar

  3. Department of Electronics Engg., KNIT, Sultanpur, U.P., India

    Rahul Ratnakumar

Authors
  1. Sahil Yadav
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aryan Mehra
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Honnesh Rohmetra
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rahul Ratnakumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Pratik Narang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pratik Narang.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yadav, S., Mehra, A., Rohmetra, H. et al. DerainGAN: Single image deraining using wasserstein GAN. Multimed Tools Appl 80, 36491–36507 (2021). https://doi.org/10.1007/s11042-021-11442-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11442-6

Keywords

Search

Navigation