Abstract
Existing methods in removing raindrops from images have encountered a key challenge, i.e., removing raindrops of different sizes and shapes while recovering the lost details. These approaches are based on a pixel-wise regression process. However, they are lacking in maintaining the balance between raindrop removal and image inpainting. To resolve the problem, we propose a method based on a hierarchical supervision network (HSNet). This network is a fusion of dense network blocks and a U-Net network. Our proposed method works in two stages. In the first stage, the HSNet is used to extract raindrop features to make a residual between raindrop image samples, i.e., input data and output of HSNet to obtain an image without raindrops. In the second stage, image inpainting is combined with an attention mechanism through HSNet to recover the areas covered by raindrops in the input image. Experimental results show that our proposed method outperforms the existing methods proposed for image deraining.
Similar content being viewed by others
References
Lan, X., Ye, M., Zhang, S., et al.: Modality-correlation-aware sparse representation for RGB-infrared object tracking. Pattern Recognit. Lett. 130, 12–20 (2020)
Zhang, H., Patel, V. M. Density-aware single image de-raining using a multi-stream dense network. Computer Vision and Pattern Recognition, pp. 695–704 (2018)
Qian, R., Tan, R. T., Yang, W., Su, J., Liu, J.: Attentive generative adversarial network for raindrop removal from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2482–2491 (2018)
Luo, Y., Xu, Y., Ji, H.: Removing rain from a single image via discriminative sparse coding. In: International Conference on Computer Vision, pp. 3397–3405 (2015)
Li, Y., Tan, R. T., Guo, X., Lu, J., Brown, M. S.: Rain streak removal using layer priors. Computer Vision and Pattern Recognition, pp. 2736–2744 (2016)
Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., Paisley, J.: Removing rain from single images via a deep detail network. Computer Vision and Pattern Recognition, pp. 1715–1723 (2017)
Liu, H., Ye, H., Li, X., Shi, W., Liu, M., Sun, Q.: Self-refining deep symmetry enhanced network for rain removal. In: 2019 IEEE International Conference on Image Processing (ICIP), pp. 2786–2790 (2019)
Lin, C., Zhang, Q., Cao, Y.: Multi-scale contour detection model based on fixational eye movement mechanism. SIViP 14, 57–65 (2020)
ElTantawy, A., Shehata, M.S.: Local null space pursuit for real-time moving object detection in aerial surveillance. SIViP 14, 87–95 (2020)
Lan, X., Ye, M., Shao, R., et al.: Learning modality-consistency feature templates: a robust RGB-infrared tracking system. IEEE Trans. Ind. Electron. 66(12), 9887–9897 (2019)
Gao, X.: A post-processing scheme for the performance improvement of vehicle detection in wide-area aerial imagery. SIViP 14, 625–633 (2020)
Lan, X., Zhang, W., Zhang, S., et al.: Robust multi-modality anchor graph-based label prediction for RGB-infrared tracking. IEEE Trans. Ind. Inform. (2019)
Liu, D.S.M., Chen, Y.J.: Rain removal system for dynamic scene in diminished reality. SIViP 14, 945–953 (2020)
Kurihata, H., Takahashi, T., Ide, I., Mekada, Y., Murase, H., Tamatsu, Y., Miyahara, T.: Rainy weather recognition from in-vehicle camera images for driver assistance. In: IEEE Proceedings of the Intelligent Vehicles Symposium, pp. 205–210 (2005)
Roser, M., Geiger, A.: Video-based raindrop detection for improved image registration. In: 2009 IEEE 12th International Conference on Computer Vision Workshops, pp. 570–577 (2009)
You, S., Tan, R.T., Kawakami, R., Mukaigawa, Y., Ikeuchi, K.: Adherent raindrop modeling, detectionand removal in video. IEEE Trans. Pattern Anal. Mach. Intell. 38(9), 1721–1733 (2015)
Eigen, D., Krishnan, D., Fergus, R.: Restoring an image taken through a window covered with dirt or rain. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 633–640 (2013)
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Bengio, Y.: Generative adversarial nets. In: Advances in neural information processing systems, pp. 2672–2680 (2014)
Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-assisted Intervention, pp. 234–241 (2015)
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K. Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)
Krizhevsky, A., Sutskever, I., Hinton, G. E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)
Roser, M., Kurz, J., Geiger, A.: Realistic modeling of water droplets for monocular adherent raindrop recognition using Bezier curves. In: Asian Conference on Computer Vision, pp. 235–244 (2010)
Zhang, H., Sindagi, V., Patel, V.M.: Image de-raining using a conditional generative adversarial network. IEEE Transactions on Circuits and Systems for Video Technology (2019)
Yang, W., Tan, R. T., Feng, J., Liu, J., Guo, Z., Yan, S.: Deep joint rain detection and removal from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1357–1366 (2017)
Kingma, D., Ba, J.: Adam: A Method for Stochastic Optimization. Computer Science (2014)
Kang, L.W., Lin, C.W., Fu, Y.H.: Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans. Image Process. 21(4), 1742–1755 (2011)
Li, X., Wu, J., Lin, Z., Liu, H., Zha, H.: Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 254–269 (2018)
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant 61762014 and Grant 62106054, and in part by the Science and Technology Project of Guangxi under Grant 2018GXNSFAA281351.
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
Conflict of interest
All authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Xia, H., Lan, Y., Song, S. et al. Raindrop removal from a single image using a two-step generative adversarial network. SIViP 16, 677–684 (2022). https://doi.org/10.1007/s11760-021-02007-z
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11760-021-02007-z