Skip to main content
SpringerLink
Log in

Single image rain streaks removal: a review and an exploration

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Recently, rain streaks removal from a single image has attracted much research attention to alleviate the degenerated performance of computer vision tasks implemented on rainy images. In this paper, we provide a thorough review for current single-image-based rain removal techniques, which can be mainly categorized into three classes: early filter-based, conventional prior-based, and recent deep learning-based approaches. Furthermore, inspired by the rationality of current deep learning-based methods and insightful characteristics underlying rain shapes, we build a specific coarse-to-fine deraining network architecture, which can finely deliver the rain structures and progressively removes rain streaks from the input image, accordingly. The superiority of the proposed network is substantiated by experiments implemented on synthetic and real rainy images both visually and quantitatively, as compared with comprehensive state-of-the-art methods along this line. Especially, it is verified that the proposed network possesses better generalization capability on real rainy images, implying its potential usefulness for this task.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

Notes

  1. https://sites.google.com/view/taixiangjiang/%E9%A6%96%E9%A1%B5/state-of-the-art-methods

  2. http://yu-li.github.io/

  3. https://sites.google.com/site/shuhanggu/home

  4. https://xueyangfu.github.io/projects/tip2017.html

  5. https://xueyangfu.github.io/projects/cvpr2017.html

  6. http://www.icst.pku.edu.cn/struct/Projects/joint_rain_removal.html

  7. https://github.com/XiaLiPKU/RESCAN

  8. https://github.com/csdwren/PReNet

  9. https://github.com/wwzjer/Semi-supervised-IRR

References

  1. Farenzena M, Bazzani L, Perina A, Murino V, Cristani M (2010) Person re-identification by symmetry-driven accumulation of local features. In Proc. of the IEEE Conf. on Comput. vision and pattern recognition

  2. Maji S, Berg AC, Malik J (2008) Classification using intersection kernel support vector machines is efficient. In Proc. of the IEEE Conf. on Comput. vision and pattern recognition, pp 1–8

  3. Junior OL, Delgado D, Gonalve V, Nunes U (2009) Trainable classifier-fusion schemes: an application to pedestrian detection. In Intelligent Transportation Syst., vol. 2

  4. Itti L, Koch C, Niebur E (1998) A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Pattern Anal Mach Intell 20(11):1254–1259

    Article  Google Scholar 

  5. Kang L, Lin C, Fu Y (2012) Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans Image Process 24(4):1742–1755

    Article  MathSciNet  MATH  Google Scholar 

  6. Li Y, Tan RT, Guo X, Lu J, Brown MS (2016) Rain streak removal using layer priors. IEEE Conf. on Comput. vision and pattern recognition

  7. Gu S, Meng D, Zuo W, Zhang L (2017) Joint convolutional analysis and synthesis sparse representation for single image layer separation. IEEE Int. Conf. on Comput. Vision, pp 1717–1725

  8. Luo Y, Xu Y, Ji H (2015) Removing rain from a single image via discriminative sparse coding. IEEE Int. Conf. on Comput. Vision, pp 3397–3405

  9. Wei W, Yi L, Xie Q, Zhao Q, Meng D, Xu Z (2017) Should We encode rain streaks in video as deterministic or stochastic? IEEE Int. Conf. on Comput. Vision, pp 2535–2544

  10. Kang L, Lin C, Lin C, Lin Y (2012) Self-learning-based rain streak removal for image/video. IEEE Int Symp Circuits Syst 57(1):1871–1874

    Google Scholar 

  11. Chen YL, Hsu CT (2013) A generalized low-rank appearance model for spatio-temporally correlated rain streaks. IEEE Int. Conf. on Comput. Vision, pp 1968–1975

  12. Ren D, Zuo W, Hu Q, Zhu P, Meng D (2019) Progressive image deraining networks: a better and simpler baseline. IEEE Conf. on Comput. Vision and Pattern Recognition

  13. He K, Sun J, Tang X (2010) Guided image filtering. European Conf. on Comput. vision, pp 1–14

    Google Scholar 

  14. Xu J, Zhao W, Liu P, Tang X (2012) Removing rain and snow in a single image using guided filter. IEEE Int. Conf. on Comput. Sci. and Automation Eng, pp 304–307

  15. Zheng X, Liao Y, Guo W, Fu X, Ding X (2013) Single-image-based rain and snow removal using multi-guided filter. Neural Inform. Process, pp. 258–265

    Chapter  Google Scholar 

  16. Chen D, Chen C, Kang L (2014) Visual depth guided color image rain streaks removal using sparse coding. IEEE Trans Circuits Syst Video Technol 24(24):1430–1455

    Article  Google Scholar 

  17. Ding X, Chen L, Zheng X, Huang Y, Zeng D (2016) Single image rain and snow removal via guided L0 smoothing filter. Multimed Tools Appl 75(5):2697–2712

    Article  Google Scholar 

  18. Kim JH, Lee C, Sim JY, Kim (2013) Single-image deraining using an adaptive nonlocal means filter. IEEE Int. Conf. on Image Process, pp 914–917

  19. Fu Y, Kang L, Lin C, Hsu CT (2013) Single-frame-based rain removal via image decomposition. IEEE Int. Conf. on Acoustics, pp 914–917

  20. Wang Y, Liu S, Chen C, Zeng B (2017) A hierarchical approach for rain or snow removing in a single color image. IEEE Trans Image Process 26(8):3936–3950

    Article  MathSciNet  MATH  Google Scholar 

  21. Wei W, Meng D, Zhao Q, Xu Z, Wu Y (2019) Semi-supervised transfer learning for image rain removal. In Proc. of the IEEE Conf. on Comput. vision and pattern recognition, pp 3877–3886

  22. Meng D, Zhao Q, Xu Z (2012) Improve robustness of sparse pca by l1-norm maximization. Pattern Recogn 45(1):487–497

    Article  MATH  Google Scholar 

  23. Zhu L, Fu C, Lischinski D, Heng P (2017) Joint bi-layer optimization for single-image rain streak removal. IEEE Int. Conf. on Comput. vision, pp 2545–2553

  24. Yang W, Tan RT, Feng J, Liu J, Guo Z, Yan S (2017) Deep joint rain detection and removal from a single image. IEEE Conf. on Comput. vision and pattern recognition, pp 1685–1694

  25. Mu P, Chen J, Liu R, Fan X, Luo Z (2018) Learning bilevel layer priors for single image rain streaks removal. IEEE Signal Process Lett 26(2):307–311

    Article  Google Scholar 

  26. Zhang K, Zuo W, Gu S, Zhang L (2017) Learning deep CNN denoiser prior for image restoration. In Proc. of the IEEE Conf. on computer vision and pattern recognition, pp 3929–3938

  27. Xie Q, Zhou M, Zhao Q, Meng D, Zuo W, Xu Z (2019) Multispectral and hyperspectral image fusion by MS/HS fusion net. IEEE Conf. on Comput. vision and pattern recognition

  28. Eigen D, Krishnan D, Fergus R (2013) Restoring an image taken through a window covered with dirt or rain. IEEE Int. Conf. on comput. vision, pp 633–640

  29. Qian R, Tan R, Yang W, Su J, Liu J (2018) Attentive generative adversarial network for raindrop removal from a single image. IEEE Conf. on Comput. vision and pattern recognition, pp 1–1

  30. Fu X, Huang J, Ding X (2017) Clearing the skies: a deep network architecture for single-image rain streaks removal. IEEE Trans Image Process 1(1):99

    MATH  Google Scholar 

  31. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In Proc. of the IEEE Conf. on Comput. vision and pattern recognition, pp 770–778

  32. Fu X, Huang J, Zeng D, Huang Y, Ding X, Paisle J (2017) Removing rain from single images via a deep detail network. IEEE Conf. on Comput. vision and pattern recognition, pp 1715–1723

  33. Zhang H, Sindagi V, Patel VM (2017) Image de-raining using a conditional generative adversarial network. arXiv:1701.05957

  34. Zhang H, Patel VM (2018) Density-aware single image de-raining using a multi-stream dense network. IEEE Conf. on Comput. vision and pattern recognition, pp 1–10

  35. Yu F, Koltun V (2016) Multi-scale context aggregation by dilated convolutions. In Int. Conf. on learning representation

  36. Li X, Wu J, Lin Z, Liu H, Zha H (2018) Recurrent squeezeand-excitation context aggregation net for single image deraining. In European Conf. on computer vision, pp 262–277

    Chapter  Google Scholar 

  37. Li G, He X, Zhang W, Chang H, Dong L, Lin L (2018) Non-locally enhanced encoder-decoder network for single image de-raining. In 2018 ACM Multimedia Conf. on Multimedia Conf., pp 1056–1064

  38. Li S, Araujo IB, Ren W, Wang Z, Tokuda EK, Junior RH et al (2019) Single image deraining: a comprehensive benchmark analysis. arXiv:1903.08558

  39. Wang T, Yang X, Xu K, Chen S, Zhang Q, Lau RWH (2019) Spatial attentive single-image deraining with a high quality real rain dataset. IEEE Conf. on Comput. vision and pattern recognition

  40. Wang Y, Zhao X, Jiang T, Deng L, Chang Y, Huang T (2018) Rain streak removal for single image via kernel guided cnn. arXiv:1808.08545

  41. Pan J, Liu S, Zhang J, Liu Y, Ren J, Li Zechao (2018) Learning dual convolutional neural networks for low-level vision. IEEE Conf. on Comput. vision and pattern recognition, pp 1–10

  42. Quan H, Ghanbari M (2008) Scope of validity of PSNR in image/video quality assessment. Electron Lett 44(13):800–801

    Article  Google Scholar 

  43. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  44. Li M, Xie Q, Zhao Q, Wei W, Gu S, Tao J, Meng D (2018) Video rain streak removal by multiscale convolutional sparse coding. IEEE Conf. on Comput. vision and pattern recognition, pp. 1-10

  45. Fan Z, Wu H, Fu X, Huang Y, Ding X (2018) Residual guide feature fusion network for single image deraining. In ACM Multimedia

  46. Fu X, Liang B, Huang Y, Ding X, Paisley J (2018) Lightweight pyramid networks for image deraining. arXiv:1805.06173

  47. Lin H, Li Y, Ding X, Zeng W, Huang Y, Paisley J (2019) Rain O’er Me: synthesizing real rain to derain with data distillation. arXiv:1904.04605

  48. Paszke A, Gross S, Chintala S, Chanan G, Yang E, De-Vito Z (2017) Automatic differentiation in pytorch

  49. Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980

  50. Martin D, Fowlkes C, Tal D, Malik J (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. IEEE Int Conf Comput Vis 2:416–423

    Google Scholar 

  51. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167

  52. Xu J, Zhao W, Liu P, Tang X (2012) An improved guidance image based method to remove rain and snow in a single image. Comput Inf Sci 5(3):49–55

    Google Scholar 

  53. Zhang H, Patel M (2017) Convolutional sparse and low-rank coding-based rain streak removal. IEEE Winter Conf. on applications of Comput. vision, pp 1259–1267

  54. Zhou W, Bovik AC (2009) Mean squared error: love it or leave it? a new look at signal fidelity measures. IEEE Signal Process Mag 26(1):98–117

    Article  Google Scholar 

  55. Zhao H, Gallo O, Frosio I, Kautz J (2017) Loss functions for image restoration with neural networks. IEEE Trans Comput Imaging 3(1):47–57

    Article  Google Scholar 

Download references

Acknowledgements

This research was supported by National Key R&D Program of China (2018YFB1004300) and China NSFC projects (61661166011, 11690011, 61603292, 61721002, U1811461). This work is also partially supported by MoE-CMCC “Artifical Intelligence” Project No. MCM20190701.

Author information

Authors and Affiliations

  1. Institute for Information and System Sciences and Ministry of Education Key Lab of Intelligent Networks and Network Security, Xian Jiaotong University, Xi’an, Shaanxi Province, 710049, People’s Republic of China

    Hong Wang, Qi Xie, Yichen Wu, Qian Zhao & Deyu Meng

  2. Faculty of Information Technology, Macau University of Science and Technology, Taipa, Macau, People’s Republic of China

    Deyu Meng

Authors
  1. Hong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qi Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yichen Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qian Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Deyu Meng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Deyu Meng.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, H., Xie, Q., Wu, Y. et al. Single image rain streaks removal: a review and an exploration. Int. J. Mach. Learn. & Cyber. 11, 853–872 (2020). https://doi.org/10.1007/s13042-020-01061-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-020-01061-2

Keywords

Search

Navigation