Abstract
With rapid development of deep learning, a lot of computer vision tasks, such as object detection and semantic segmentation, have been applied to various Advanced Driver Assistance Systems. However, few computer vision solutions to estimate rainfall amount have been developed so far. So, we propose a rainfall amount estimation method based on deep learning and computer vision. The proposed method mainly consists of two steps. The first step is raindrop segmentation, and the second step is rainfall amount estimation. The raindrop segmentation is specifically based on three techniques: A relational ASPP to explore the correlation between raindrop features, a height attention module to consider the features that vary depending on raindrop locations, and a masking loss to further improve the performance of raindrop segmentation. Second, using the segmented raindrops, we present a rainfall amount estimation algorithm for auto-wiping. We experimentally achieved acceptable raindrop segmentation performance, i.e., mean IoU (mIoU) score of 70.6% that is much higher than other algorithms. This verifies that the proposed network is good at segmenting raindrops on a windshield. And, we demonstrate that the proposed rainfall amount estimation scheme provides sufficiently high accuracy of about 93%. In addition, we have built a rainy driving dataset for computer vision-based auto-wiping purpose and published it publicly on https://github.com/jjh930910/raindrop-segmentation.
Similar content being viewed by others
References
Ishikawa J (10 Aug. 2010) Raindrop detecting device and method of selecting wiping mode for vehicle. U.S. Patent No. 7,772,793
VanDam SA (14 Jul 1998) Windshield wiper rain sensor system. U.S. Patent No. 5,780,719
Cord A, Aubert D (2011) Towards rain detection through use of in-vehicle multipurpose cameras In: Proc IEEE IV Symp, Baden-Baden, Germany, pp 399–404
Vijay CS, Bhat R, Ragavan V (2018) Raindrop detection considering extremal regions and salient features In: Proceedings electronic imaging, pp 348-1-348-6
Qian R, Tan RT, Yang W, Su J, Liu J (2018) Attentive generative adversarial network for raindrop removal from a single image In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2482-2491
Otsu N (1979) A threshold selection method from gray-level histograms. IEEE Transactions Syst Man Cybern 9(1):62–66
Dhanachandra N, Manglem K, Chanu YJ (2015) Image segmentation using K-means clustering algorithm and subtractive clustering algorithm. Procedia Computer Sci 54:764–771
Najman L, Schmitt M (1994) Watershed of a continuous function. Signal Process 38(1):99–112
Kass M, Witkin A, Terzopoulos D (1988) Snakes: active contour models. Int J Computer V 1(4):321–331
Boykov Y, Veksler O, Zabih R (2001) Fast approximate energy minimization via graph cuts. IEEE Transactions Pattern Anal Mach Intell 23(11):1222–1239
Plath N, Toussaint M, Nakajima S (2009) Multi-class image segmentation using conditional random fields and global classification In: Proceedings of the 26th Annual International Conference on Machine Learning, pp 817–824
Minaee S, Boykov YY, Porikli F, Plaza AJ, Kehtarnavaz N, Terzopoulos D (2021) Image segmentation using deep learning: a survey IEEE Transactions Pattern Anal Mach Intell (Early Access)
Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation In: International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, pp 234–241
Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431-3440
Fan DP, Ji GP, Zhou T, Chen G, Fu H, Shen J, Shao L (2020) Pranet: parallel reverse attention network for polyp segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention Springer, Cham, pp 263–273
Jha D, Riegler MA, Johansen D, Halvorsen P, Johansen HD (2020) DoubleU-Net: a deep convolutional neural network for medical image segmentation In: arXiv preprint arXiv:2006.04868
Gu Z, Cheng J, Fu H, Zhou K, Hao H, Zhao Y, Liu J (2019) Ce-net: context encoder network for 2d medical image segmentation. Proc IEEE Transactions Med Imag 38(10):2281–2292
Paluru N et al (2021) Anam-Net: anamorphic depth embedding-based lightweight CNN for segmentation of anomalies in COVID-19 chest CT images. IEEE Transactions Neural Netw Learn Syst 32(3):932–946
Yeung M, Sala E, Schönlieb C-B, Rundo L (2021) Focus U-Net: a novel dual attention-gated CNN for polyp segmentation during colonoscopy. Computers Biol Med 137:104815
Yu F, Koltun V (2015) Multi-scale context aggregation by dilated convolutions. In: arXiv preprint arXiv:1511.07122
Chen LC, Zhu Y, Papandreou G, Schroff F, Adam H (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation In: Proceedings of the European conference on computer vision, pp 801-818
Liu W, Rabinovich A, Berg AC (2015) Parsenet: looking wider to see better. In: arXiv preprint arXiv:1506.04579
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, ... Polosukhin I (2017) Attention is all you need Adv Neural Information Process Syst pp 5998-6008
Meng Z, Gaur Y, Li J, Gong Y (2019) Character-aware attention-based end-to-end speech recognition In: Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop, pp 949-955
Fu J, Liu J, Tian H, Li Y, Bao Y, Fang Z, Lu H (2019) Dual attention network for scene segmentation In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3146-3154
Zhang F, Chen Y, Li Z, Hong Z, Liu J, Ma F, ... Ding E, (2019) Acfnet: attentional class feature network for semantic segmentation In: Proceedings of the IEEE International Conference on Computer Vision, pp 6798-6807
Strudel R, Garcia R, Laptev I, Schmid C (2021) Segmenter: transformer for semantic segmentation In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 7262–7272
Vaswani A et al. (2017) Attention is all you need Advances in neural information processing systems. In: Proceedings of the advances in neural information processing systems, pp 5998–6008
Dosovitskiy A et al. (2021) An Image is Worth 16x16 Words: transformers for image recognition at scale ICLR. In: Proceedings of the international conference on learning representations, pp 1–21
Choi S, Kim JT, Choo J (2020) Cars can’t fly up in the sky: improving urban-scene segmentation via height-driven attention networks In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
You S, Tan RT, Kawakami R, Ikeuchi K (2013) Adherent raindrop detection and removal in video In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1035–1042
Ito K, Noro K, Aoki T (2015) An adherent raindrop detection method using MSER In: 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), pp 105–109
Ishizuka J, Onoguchi K (2016) Detection of raindrop with various shapes on aWindshield ICPRAM, pp 475–483
Shi X, Chen Z, Wang H, Yeung DY, Wong WK, Woo WC (2015) Convolutional LSTM network: a machine learning approach for precipitation nowcasting. Proc Adv Neural Information Process Syst 28:802–810
Lin J, Dai L (2020) X-NET for single image raindrop removal. In: Proceeding of the IEEE International Conference on Image Processing, pp 1003-1007
Alletto S, Carlin C, Rigazio L, Ishii Y, Tsukizawa S (2019) Adherent raindrop removal with self-supervised attention maps and spatio-temporal generative adversarial networks In: Proceedings of the IEEE International Conference on Computer Vision Workshops
Welch G, Bishop G (1995) An introduction to the Kalman filter. In: Proceedings of the SIGGRAPH, Course 8, pp 127-132
Oktay O, Schlemper J, Folgoc LL, Lee M, Heinrich M, Misawa K, ... Glocker B (2018) Attention u-net: learning where to look for the pancreas In: arXiv preprint arXiv:1804.03999
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Lee, S.H., Jeon, J.H., Choi, D.Y. et al. Image based rainfall amount estimation for auto-wiping of vehicles. Neural Comput & Applic 34, 15543–15554 (2022). https://doi.org/10.1007/s00521-022-07269-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00521-022-07269-3