Abstract
Rain streaks can eclipse some information of an image taken during rainfall which can degrade the performance of a vision system. While existing rain removing methods can recover the semantic structure, they lack natural texture recovery. The aim of this work is to recover the hidden structure and texture under the rain streaks with fine details. We propose a novel generative adversarial network with two discriminators to remove rain called rain removal generative adversarial network, where a combination of reconstruction, feature and adversarial losses is used for low level, structural and natural recovery, respectively. We have found that exploiting low-level \({l_1}\) loss with high-level structural similarity loss as a reconstruction loss is quite effective in attaining visually plausible and consistent texture. Qualitative and quantitative evaluations on our synthetically created dataset and a benchmark dataset show substantial performance gain than state-of-the-art rain removing methods.
Similar content being viewed by others
References
Barnum PC, Narasimhan S, Kanade T (2010) Analysis of rain and snow in frequency space. Int J Comput Vis 86(2–3):256
Bossu J, Hautière N, Tarel JP (2011) Rain or snow detection in image sequences through use of a histogram of orientation of streaks. Int J Comput Vis 93(3):348–367
Chen LC, Papandreou G, Schroff F, Adam H (2017) Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587
Chen YL, Hsu CT (2013) A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In: Proceedings of the IEEE international conference on computer vision, pp 1968–1975
Choi Y, Choi M, Kim M, Ha JW, Kim S, Choo J (2018) Stargan: unified generative adversarial networks for multi-domain image-to-image translation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8789–8797
Eigen D, Krishnan D, Fergus R (2013) Restoring an image taken through a window covered with dirt or rain. In: Proceedings of the IEEE international conference on computer vision, pp 633–640
Fu X, Huang J, Ding X, Liao Y, Paisley J (2017) Clearing the skies: a deep network architecture for single-image rain removal. IEEE Tran Image Process 26(6):2944–2956
Fu X, Huang J, Zeng D, Huang Y, Ding X, Paisley J (2017) Removing rain from single images via a deep detail network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3855–3863
Garg K, Nayar SK (2004) Detection and removal of rain from videos. In: Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition, CVPR 2004, vol 1. IEEE, pp I–I
Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. Adv Neural Inf Process Syst 2:2672–2680
Gulrajani I, Ahmed F, Arjovsky M, Dumoulin V, Courville AC (2017) Improved training of wasserstein gans. Adv Neural Inf Process Syst 27:5767–5777
Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141
Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: IEEE CVPR, pp 5967–5976
Jetchev N, Bergmann U, Vollgraf R (2016) Texture synthesis with spatial generative adversarial networks. arXiv preprint arXiv:1611.08207
Kim JH, Lee C, Sim JY, Kim CS (2013) Single-image deraining using an adaptive nonlocal means filter. In: 2013 IEEE international conference on image processing. IEEE, pp 914–917
Kingma DP, Ba JL (2014) Adam: a method for stochastic optimization. In: Proceedings of the 3rd international conference on learning representations
Li S, Ren W, Zhang J, Yu J, Guo X (2019) Single image rain removal via a deep decomposition-composition network. Computer Vis Image Underst 186:8–57
Mao X, Li Q, Xie H, Lau RY, Wang Z, Smolley SP (2017) Least squares generative adversarial networks. In: 2017 IEEE international conference on computer vision (ICCV). IEEE, pp 2813–2821
Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 234–241
Rudin LI, Osher S, Fatemi E (1992) Nonlinear total variation based noise removal algorithms. Phys D Nonlinear Phenom 60(1–4):259–268
Santhaseelan V, Asari VK (2015) Utilizing local phase information to remove rain from video. Int J Comput Vis 112(1):71–89
Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: ICLR
Wang Y, Gong D, Yang J, Shi Q, Hengel Avd, Xie D, Zeng B (2019) An effective two-branch model-based deep network for single image deraining. arXiv preprint arXiv:1905.05404
Wang YT, Zhao XL, Jiang TX, Deng LJ, Chang Y, Huang TZ (2018) Rain streak removal for single image via kernel guided cnn. arXiv preprint arXiv:1808.08545
Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612
Yang W, Tan RT, Feng J, Liu J, Guo Z, Yan S (2017) Deep joint rain detection and removal from a single image. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1357–1366
Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS (2018) Generative image inpainting with contextual attention. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5505–5514
Zhang H, Sindagi V, Patel VM (2019) Image de-raining using a conditional generative adversarial network. In: IEEE transactions on circuits and systems for video technology
Zhang H, Xu T, Li H, Zhang S, Huang X, Wang X, Metaxas D (2017) Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In: IEEE international conference on computer vision (ICCV), pp 5907–5915
Zhou B, Lapedriza A, Khosla A, Oliva A, Torralba A (2017) Places: a 10 million image database for scene recognition. IEEE Trans Pattern Anal Mach Intell 40(6):1452–1464
Acknowledgements
This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIP) (No. 2017R1A2B4001801).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Javed, K., Hussain, G., Shaukat, F. et al. A neural network approach to remove rain using reconstruction and feature losses. Neural Comput & Applic 32, 13129–13138 (2020). https://doi.org/10.1007/s00521-019-04558-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00521-019-04558-2