Abstract
Deep convolutional neural networks (CNNs) have shown their advantages in the single image de-raining task. However, most existing CNNs-based methods utilize only local spatial information without considering long-range contextual information. In this paper, we propose a graph convolutional networks (GCNs)-based model to solve the above problem. We specifically design two graphs to extract representations from new dimensions. The first graph models the global spatial relationship between pixels in the feature, while the second graph models the interrelationship across the channels. By integrating conventional CNNs and our GCNs into a single framework, the proposed method is able to explore comprehensive feature representations from three aspects, i.e., local spatial patterns, global spatial coherence and channel correlation. To better exploit the explored rich feature representations, we further introduce a simple yet effective recurrent operations to perform the de-raining process in a successive manner. Benefiting from the rich information exploration and exploitation, our method achieves state-of-the-art results on both synthetic and real-world data sets.
Similar content being viewed by others
References
Abadi, M., Agarwal, A., & Barham, P., et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv: 1603.04467 (2016)
Arbelaez, P., Maire, M., Fowlkes, C., & Malik, J. (2010). Contour detection and hierarchical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(5), 898–916.
Barnum, P. C., Narasimhan, S., & Kanade, T. (2010). Analysis of rain and snow in frequency space. International Journal of Computer Vision, 86(2), 256–274.
Bossu, J., Hautière, N., & Tarel, J. P. (2011). Rain or snow detection in image sequences through use of a histogram of orientation of streaks. International Journal of Computer Vision, 93(3), 348–367.
Chang, Y., Yan, L., & Zhong, S. (2017). Transformed low-rank model for line pattern noise removal. In: ICCV
Chen, C., Xiong, Z., Tian, X., Zha, Z. J., & Wu, F. (2019). Real-world image denoising with deep boosting. IEEE Transactions on Pattern Analysis and Machine Intelligence,. https://doi.org/10.1109/TPAMI.2019.2921548.
Chen, J., & Chau, L. P. (2013). A rain pixel recovery algorithm for videos with highly dynamic scenes. IEEE Transactions on Image Processing, 23(3), 1097–1104.
Chen, J., Tan, C.H., Hou, J., Chau, L.P., & Li, H. (2018). Robust video content alignment and compensation for rain removal in a cnn framework. In: CVPR
Chen, L. C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2017). Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4), 834–848.
Chen, Y., Kalantidis, Y., Li, J., Yan, S., Feng, J.: A\(\hat{2}\)-nets: Double attention networks. In: NeurIPS (2018)
Chen, Y., & Pock, T. (2016). Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6), 1256–1272.
Chen, Y., Rohrbach, M., Yan, Z., Shuicheng, Y., Feng, J., & Kalantidis, Y. (2019). Graph-based global reasoning networks. In: CVPR
Chen, Y.L., & Hsu, C.T. (2013) A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In: ICCV
Dong, C., Deng, Y., Change Loy, C., & Tang, X. (2015). Compression artifacts reduction by a deep convolutional network. In: ICCV, pp. 576–584
Dong, C., Loy, C. C., He, K., & Tang, X. (2016). Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2), 295–307.
Eigen, D., Krishnan, D., & Fergus, R. (2013). Restoring an image taken through a window covered with dirt or rain. In: ICCV
Fan, Q., Chen, D., Yuan, L., Hua, G., Yu, N., & Chen, B. (2018). Decouple learning for parameterized image operators. In: ECCV, pp. 442–458
Fu, X., Huang, J., Ding, X., Liao, Y., & Paisley, J. (2017). Clearing the skies: A deep network architecture for single-image rain removal. IEEE Transactions on Image Processing, 26(6), 2944–2956.
Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., & Paisley, J. (2017). Removing rain from single images via a deep detail network. In: CVPR
Fu, X., Liang, B., Huang, Y., Ding, X., & Paisley, J. (2019). Lightweight pyramid networks for image deraining. IEEE Transactions on Neural Networks and Learning Systems,. https://doi.org/10.1109/TNNLS.2019.2926481.
Garg, K., & Nayar, S.K. (2004). Detection and removal of rain from videos. In: CVPR
Garg, K., & Nayar, S.K. (2005) When does a camera see rain? In: ICCV
Garg, K., & Nayar, S. K. (2007). Vision and rain. International Journal of Computer Vision, 75(1), 3–27.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014) Generative adversarial nets. In: NeurIPS, pp. 2672–2680
Gu, S., Meng, D., Zuo, W., & Zhang, L. (2017). Joint convolutional analysis and synthesis sparse representation for single image layer separation. In: ICCV
He, K., Sun, J., & Tang, X. (2013). Guided image filtering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(6), 1397–1409.
He, K., Zhang, X., Ren, S., & Sun, J. (2016) Deep residual learning for image recognition. In: CVPR
Hu, J., Shen, L., & Sun, G. (2018). Squeeze-and-excitation networks. In: CVPR
Hu, X., Fu, C.W., Zhu, L., & Heng, P.A. (2019). Depth-attentional features for single-image rain removal. In: CVPR
Huang, D. A., Kang, L. W., Wang, Y. C. F., & Lin, C. W. (2014). Self-learning based image decomposition with applications to single image denoising. IEEE Transactions on Multimedia, 16(1), 83–93.
Jiang, K., Wang, Z., Yi, P., Chen, C., Huang, B., Luo, Y., Ma, J., & Jiang, J. (2020). Multi-scale progressive fusion network for single image deraining. In: CVPR
Jiang, T.X., Huang, T.Z., Zhao, X.L., Deng, L.J., & Wang, Y. (2017). A novel tensor-based video rain streaks removal approach via utilizing discriminatively intrinsic priors. In: CVPR
Jiang, T. X., Huang, T. Z., Zhao, X. L., Deng, L. J., & Wang, Y. (2018). Fastderain: A novel video rain streak removal method using directional gradient priors. IEEE Transactions on Image Processing, 28(4), 2089–2102.
Johnson, J., Alahi, A., & Fei-Fei, L. (2016). Perceptual losses for real-time style transfer and super-resolution. In: ECCV, pp. 694–711
Johnson, J., Gupta, A., & Fei-Fei, L. (2018). Image generation from scene graphs. In: CVPR
Kang, L. W., Lin, C. W., & Fu, Y. H. (2012). Automatic single image-based rain streaks removal via image decomposition. IEEE Transactions on Image Processing, 21(4), 1742–1755.
Kim, J.H., Lee, C., Sim, J.Y., & Kim, C.S. (2013). Single-image deraining using an adaptive nonlocal means filter. In: IEEE ICIP
Kim, J. H., Sim, J. Y., & Kim, C. S. (2015). Video deraining and desnowing using temporal correlation and low-rank matrix completion. IEEE Transactions on Image Processing, 24(9), 2658–2670.
Kingma, D.P., & Ba, J. (2014). Adam: A method for stochastic optimization. In: ICLR
Kipf, T.N., & Welling, M. (2017). Semi-supervised classification with graph convolutional networks. In: ICLR
Krizhevsky, A., Sutskever, I., & Hinton, G.E. (2012). ImageNet classification with deep convolutional neural networks. In: NeurIPS
Li, G., He, X., Zhang, W., Chang, H., Dong, L., & Lin, L. (2018). Non-locally enhanced encoder-decoder network for single image de-raining. In: ACM MM
Li, J., Ma, A. J., & Yuen, P. C. (2018). Semi-supervised region metric learning for person re-identification. International Journal of Computer Vision, 126(8), 855–874.
Li, L., Pan, J., Lai, W. S., Gao, C., Sang, N., & Yang, M. H. (2019). Blind image deblurring via deep discriminative priors. International Journal of Computer Vision, 127(8), 1025–1043.
Li, M., Xie, Q., Zhao, Q., Wei, W., Gu, S., Tao, J., & Meng, D. (2018). Video rain streak removal by multiscale convolutional sparse coding. In: CVPR
Li, R., Cheong, L.F., & Tan, R.T. (2019). Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In: CVPR
Li, R., Tan, R.T., Cheong, L.F., Aviles-Rivero, A.I., Fan, Q., & Schonlieb, C.B. (2019). Rainflow: Optical flow under rain streaks and rain veiling effect. In: ICCV
Li, S., Araujo, I.B., Ren, W., Wang, Z., Tokuda, E.K., Junior, R.H., Cesar-Junior, R., Zhang, J., Guo, X., & Cao, X. (2019). Single image deraining: A comprehensive benchmark analysis. In: CVPR
Li, X., Wu, J., Lin, Z., Liu, H., & Zha, H. (2018). Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: ECCV
Li, Y., Ouyang, W., Zhou, B., Shi, J., Zhang, C., & Wang, X. (2018). Factorizable net: an efficient subgraph-based framework for scene graph generation. In: ECCV
Li, Y., Tan, R.T., Guo, X., Lu, J., & Brown, M.S. (2016). Rain streak removal using layer priors. In: CVPR
Lin, M., Chen, Q., & Yan, S. (2014). Network in network. In: ICLR
Liu, J., Yang, W., Yang, S., & Guo, Z. (2018). D3R-Net: Dynamic routing residue recurrent network for video rain removal. IEEE Transactions on Image Processing, 28(2), 699–712.
Liu, J., Yang, W., Yang, S., & Guo, Z. (2018). Erase or fill? deep joint recurrent rain removal and reconstruction in videos. In: CVPR
Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., et al. (2019). Deep learning for generic object detection: A survey. International Journal of Computer Vision,. https://doi.org/10.1007/s11263-019-01247-4.
Luo, Y., Xu, Y., & Ji, H. (2015). Removing rain from a single image via discriminative sparse coding. In: ICCV
Mordan, T., Thome, N., Henaff, G., & Cord, M. (2019). End-to-end learning of latent deformable part-based representations for object detection. International Journal of Computer Vision, 127(11–12), 1659–1679.
Narasimhan, S. G., & Nayar, S. K. (2002). Vision and the atmosphere. International Journal of Computer Vision, 48(3), 233–254.
Qi, X., Liao, R., Jia, J., Fidler, S., & Urtasun, R. (2017). 3d graph neural networks for rgbd semantic segmentation. In: ICCV
Qian, R., Tan, R.T., Yang, W., Su, J., & Liu, J. (2018). Attentive generative adversarial network for raindrop removal from a single image. In: CVPR
Ren, D., Zuo, W., Hu, Q., Zhu, P., & Meng, D. (2019). Progressive image deraining networks: A better and simpler baseline. In: CVPR
Ren, W., Pan, J., Zhang, H., Cao, X., & Yang, M. H. (2019). Single image dehazing via multi-scale convolutional neural networks with holistic edges. International Journal of Computer Vision,. https://doi.org/10.1007/s11263-019-01235-8.
Ren, W., Tian, J., Han, Z., Chan, A., & Tang, Y. (2017). Video desnowing and deraining based on matrix decomposition. In: ICCV
Romano, Y., & Elad, M. (2015). Boosting of image denoising algorithms. SIAM Journal on Imaging Sciences, 8(2), 1187–1219.
Sakaridis, C., Dai, D., & Van Gool, L. (2018). Semantic foggy scene understanding with synthetic data. International Journal of Computer Vision, 126(9), 973–992.
Santhaseelan, V., & Asari, V. K. (2015). Utilizing local phase information to remove rain from video. International Journal of Computer Vision, 112(1), 71–89.
Shao, Y., Li, L., Ren, W., Gao, C., & Sang, N. (2020). Domain adaptation for image dehazing. In: CVPR
Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A.A. (2017). Inception-v4, inception-resnet and the impact of residual connections on learning. In: AAAI
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. In: CVPR
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In: CVPR
Tai, Y., Yang, J., & Liu, X. (2017). Image super-resolution via deep recursive residual network. In: CVPR
Wang, C., Li, Z., Wu, J., Fan, H., Xiao, G., & Zhang, H. (2020). Deep residual haze network for image dehazing and deraining. IEEE Access, 8, 9488–9500.
Wang, G., Sun, C., & Sowmya, A. (2019). Erl-net: Entangled representation learning for single image de-raining. In: ICCV
Wang, H., Li, M., Wu, Y., Zhao, Q., & Meng, D. (2020). A survey on rain removal from video and single image. International Journal of Machine Learning and Cybernetics,. https://doi.org/10.1007/s13042-020-01061-2.
Wang, H., Xie, Q., Zhao, Q., & Meng, D. (2020). A model-driven deep neural network for single image rain removal. In: CVPR
Wang, H., Zhu, X., Gong, S., & Xiang, T. (2018). Person re-identification in identity regression space. International Journal of Computer Vision, 126(12), 1288–1310.
Wang, T., Yang, X., Xu, K., Chen, S., Zhang, Q., & Lau, R.W. (2019). Spatial attentive single-image deraining with a high quality real rain dataset. In: CVPR
Wang, X., Girshick, R., Gupta, A., & He, K. (2018). Non-local neural networks. In: CVPR
Wang, Y., Liu, S., Chen, C., & Zeng, B. (2017). A hierarchical approach for rain or snow removing in a single color image. IEEE Transactions on Image Processing, 26(8), 3936–3950.
Wei, W., Meng, D., Zhao, Q., Wu, C., & Xu, Z. (2019). Semi-supervised transfer learning for image rain removal. In: CVPR
Wei, W., Yi, L., Xie, Q., Zhao, Q., Meng, D., & Xu, Z. (2017). Should we encode rain streaks in video as deterministic or stochastic? In: ICCV
Wojna, Z., Ferrari, V., Guadarrama, S., Silberman, N., Chen, L. C., Fathi, A., et al. (2019). The devil is in the decoder: Classification, regression and gans. International Journal of Computer Vision, 127(11–12), 1694–1706.
Xingjian, S., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., & Woo, W.c. (2015). Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In: NeurIPS
Yang, W., Liu, J., Yang, S., & Guo, Z. (2019). Scale-free single image deraining via visibility-enhanced recurrent wavelet learning. IEEE Transactions on Image Processing, 28(6), 2948–2961.
Yang, W., Tan, R.T., Feng, J., Liu, J., Guo, Z., & Yan, S. (2017). Deep joint rain detection and removal from a single image. In: CVPR
Yang, W., Tan, R. T., Feng, J., Liu, J., Yan, S., & Guo, Z. (2019). Joint rain detection and removal from a single image with contextualized deep networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(6), 1377–1393.
Yang, W., Tan, R. T., Wang, S., Fang, Y., & Liu, J. (2020). Single image deraining: From model-based to data-driven and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence,. https://doi.org/10.1109/TPAMI.2020.2995190.
Yasarla, R., & Patel, V.M. (2019). Uncertainty guided multi-scale residual learning using a cycle spinning cnn for single image de-raining. In: CVPR
Yasarla, R., & Patel, V. M. (2020). Confidence measure guided single image de-raining. IEEE Transactions on Image Processing, 29, 4544–4555.
Yim, C., & Bovik, A. C. (2010). Quality assessment of deblocked images. IEEE Transactions on Image Processing, 20(1), 88–98.
Yu, F., & Koltun, V. (2016). Multi-scale context aggregation by dilated convolutions. In: ICLR
Zhang, H., & Patel, V. M. (2016). Sparse representation-based open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(8), 1690–1696.
Zhang, H., & Patel, V.M. (2017). Convolutional sparse and low-rank coding-based rain streak removal. In: IEEE Winter Conference on Applications of Computer Vision
Zhang, H., & Patel, V.M. (2018). Densely connected pyramid dehazing network. In: CVPR
Zhang, H., & Patel, V.M. (2018). Density-aware single image de-raining using a multi-stream dense network. In: CVPR
Zhang, H., Riggan, B. S., Hu, S., Short, N. J., & Patel, V. M. (2019). Synthesis of high-quality visible faces from polarimetric thermal faces using generative adversarial networks. International Journal of Computer Vision, 127(6–7), 845–862.
Zhang, H., Sindagi, V., & Patel, V. M. (2019). Image de-raining using a conditional generative adversarial network. IEEE Transactions on Circuits and Systems for Video Technology,. https://doi.org/10.1109/TCSVT.2019.2920407.
Zhang, K., Zuo, W., Chen, Y., Meng, D., & Zhang, L. (2017). Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing, 26(7), 3142–3155.
Zhang, T., Ghanem, B., Liu, S., & Ahuja, N. (2013). Robust visual tracking via structured multi-task sparse learning. International Journal of Computer Vision, 101(2), 367–383.
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2018). Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: CVPR
Zheng, X., Liao, Y., Guo, W., Fu, X., & Ding, X. (2013). Single-image-based rain and snow removal using multi-guided filter. In: International Conference on Neural Information Processing
Zhu, L., Fu, C.W., Lischinski, D., & Heng, P.A. (2017). Joint bi-layer optimization for single-image rain streak removal. In: ICCV
Acknowledgements
This work was supported by the National Key R&D Program of China under Grant 2020AAA0105702, the National Natural Science Foundation of China (NSFC) under Grants U19B2038, 61620106009 and 61901433, the USTC Research Funds of the Double First-Class Initiative under Grant YD2100002003.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Communicated by Vishal Patel.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Fu, X., Qi, Q., Zha, ZJ. et al. Successive Graph Convolutional Network for Image De-raining. Int J Comput Vis 129, 1691–1711 (2021). https://doi.org/10.1007/s11263-020-01428-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-020-01428-6