Abstract
Single image deraining refers to the process of restoring the clean background scene from a rainy image. Current approaches have resorted to deep learning techniques to remove rain from a single image by leveraging some prior information. However, due to the various appearances of rain streaks and accumulation, it is difficult to separate rain and background information in the embedding space, which results in inaccurate deraining. To address this issue, this paper proposes a selective wavelet attention learning method by learning a series of wavelet attention maps to guide the separation of rain and background information in both spatial and frequency domains. The key aspect of our method is utilizing wavelet transform to learn the content and structure of rainy features because the high-frequency features are more sensitive to rain degradations, whereas the low-frequency features preserve more of the background content. To begin with, we develop a selective wavelet attention encoder–decoder network to learn wavelet attention maps guiding the separation of rainy and background features at multiple scales. Meanwhile, we introduce wavelet pooling and unpooling to the encoder–decoder network, which shows superiority in learning increasingly abstract representations while preserving the background details. In addition, we propose latent alignment learning to supervise the background features as well as augment the training data to further improve the accuracy of deraining. Finally, we employ a hierarchical discriminator network based on selective wavelet attention to adversarially improve the visual fidelity of the generated results both globally and locally. Extensive experiments on synthetic and real datasets demonstrate that the proposed approach achieves more appealing results both quantitatively and qualitatively than the recent state-of-the-art methods.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Billings, S. A., & Wei, H. L. (2005). A new class of wavelet networks for nonlinear system identification. IEEE Transactions on Neural Networks, 16(4), 862–874.
Chen, X., Xu, C., Yang, X., & Tao, D. (2018). Attention-gan for object transfiguration in wild images. In European conference on computer vision, pp. 164–180.
De Silva, D., Vithanage, H., Fernando, K., & Piyatilake, I. (2020). Multi-path learnable wavelet neural network for image classification. In International conference on machine vision, vol. 11433, p. 1143310.
Deco, G., & Zihl, J. (2001). A neurodynamical model of visual attention: Feedback enhancement of spatial resolution in a hierarchical system. Journal of Computational Neuroscience, 10(3), 231–253.
Deng, X., Yang, R., Xu, M., & Dragotti, P. L. (2019). Wavelet domain style transfer for an effective perception-distortion tradeoff in single image super-resolution. In IEEE international conference on computer vision, pp. 3076–3085.
Eigen, D., Krishnan, D., & Fergus, R. (2013). Restoring an image taken through a window covered with dirt or rain. In IEEE international conference on computer vision, pp. 633–640.
Fan, Z., Wu, H., Fu, X., Huang, Y., & Ding, X. (2018). Residual-guide network for single image deraining. In ACM international conference on multimedia, pp. 1751–1759.
Fu, X., Huang, J., Ding, X., Liao, Y., & Paisley, J. (2017a). Clearing the skies: A deep network architecture for single-image rain removal. IEEE Transactions on Image Processing, 26(6), 2944–2956.
Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., & Paisley, J. (2017b). Removing rain from single images via a deep detail network. In IEEE conference on computer vision and pattern recognition, pp. 3855–3863.
Fu, X., Liang, B., Huang, Y., Ding, X., & Paisley, J. (2019). Lightweight pyramid networks for image deraining. IEEE Transactions on Neural Networks and Learning Systems.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial nets. In Advances in neural information processing systems, pp. 2672–2680.
Hu, J., Shen, L., & Sun, G. (2018). Squeeze-and-excitation networks. In IEEE conference on computer vision and pattern recognition, pp. 7132–7141.
Hu, X., Fu, C. W., Zhu, L., & Heng, P. A. (2019). Depth-attentional features for single-image rain removal. In IEEE conference on computer vision and pattern recognition, pp. 8022–8031.
Huang, H., He, R., Sun, Z., & Tan, T. (2017). Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution. In IEEE international conference on computer vision, pp. 1689–1697.
Huang, H., He, R., Sun, Z., & Tan, T. (2019). Wavelet domain generative adversarial network for multi-scale face hallucination. International Journal of Computer Vision, 127(6–7), 763–784.
Isola, P., Zhu, J. Y., Zhou, T., & Efros, A. A. (2017). Image-to-image translation with conditional adversarial networks. In IEEE conference on computer vision and pattern recognition, pp. 1125–1134.
Kang, L. W., Lin, C. W., & Fu, Y. H. (2011). Automatic single-image-based rain streaks removal via image decomposition. IEEE Transactions on Image Processing, 21(4), 1742–1755.
Kuen, J., Wang, Z., & Wang, G. (2016). Recurrent attentional networks for saliency detection. In IEEE conference on computer vision and pattern recognition, pp. 3668–3677.
Li, G., He, X., Zhang, W., Chang, H., Dong, L., & Lin, L. (2018a). Non-locally enhanced encoder-decoder network for single image de-raining. In ACM international conference on multimedia, pp. 1056–1064.
Li, R., Cheong, L. F., & Tan, R. T. (2017). Single image deraining using scale-aware multi-stage recurrent network. arXiv:1712.06830.
Li, R., Cheong, L. F., & Tan, R. T. (2019). Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In IEEE conference on computer vision and pattern recognition, pp. 1633–1642.
Li, X., Wu, J., Lin, Z., Liu, H., & Zha, H. (2018b). Recurrent squeeze-and-excitation context aggregation net for single image deraining. In European conference on computer vision, pp. 254–269.
Li, Y., Tan, R. T., Guo, X., Lu, J., & Brown, M. S. (2016). Rain streak removal using layer priors. In IEEE conference on computer vision and pattern recognition, pp. 2736–2744.
Lim, B., Son, S., Kim, H., Nah, S., & Mu Lee, K. (2017). Enhanced deep residual networks for single image super-resolution. In IEEE conference on computer vision and pattern recognition workshops, pp. 136–144.
Liu, P., Zhang, H., Zhang, K., Lin, L., & Zuo, W. (2018). Multi-level wavelet-cnn for image restoration. In IEEE conference on computer vision and pattern recognition workshops, pp. 773–782.
Luo, Y., Xu, Y., & Ji, H. (2015). Removing rain from a single image via discriminative sparse coding. In IEEE international conference on computer vision, pp. 3397–3405.
Mallat, S. (1996). Wavelets for a vision. Proceedings of the IEEE, 84(4), 604–614.
Mallat, S. G. (1989). A theory for multiresolution signal decomposition: The wavelet representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(7), 674–693.
Mao, X., Li, Q., Xie, H., Lau, R. Y., Wang, Z., & Smolley, S. P. (2017). Least squares generative adversarial networks. In IEEE international conference on computer vision, pp. 2813–2821.
Mirza, M., & Osindero, S. (2014). Conditional generative adversarial nets. arXiv:1411.1784.
Pan, J., Liu, S., Sun, D., Zhang, J., Liu, Y., Ren, J., Li, Z., Tang, J., Lu, H., Tai, Y. W., et al. (2018). Learning dual convolutional neural networks for low-level vision. In IEEE conference on computer vision and pattern recognition, pp. 3070–3079.
Provenzi, E., & Caselles, V. (2014). A wavelet perspective on variational perceptually-inspired color enhancement. International Journal of Computer Vision, 106(2), 153–171.
Qian, R., Tan, R. T., Yang, W., Su, J., & Liu, J. (2018). Attentive generative adversarial network for raindrop removal from a single image. In IEEE conference on computer vision and pattern recognition, pp. 2482–2491.
Ren, D., Zuo, W., Hu, Q., Zhu, P., & Meng, D. (2019). Progressive image deraining networks: A better and simpler baseline. In IEEE conference on computer vision and pattern recognition, pp. 3937–3946.
Rensink, R. A. (2000). The dynamic representation of scenes. Visual Cognition, 7(1–3), 17–42.
Szu, H. H., Telfer, B. A., & Kadambe, S. L. (1992). Neural network adaptive wavelets for signal representation and classification. Optical Engineering, 31(9), 1907–1917.
Treisman, A. (1982). Perceptual grouping and attention in visual search for features and for objects. Journal of Experimental Psychology: Human Perception and Performance, 8(2), 194.
Treisman, A. M., & Gelade, G. (1980). A feature-integration theory of attention. Cognitive Psychology, 12(1), 97–136.
Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., & Tang, X. (2017). Residual attention network for image classification. In IEEE conference on computer vision and pattern recognition, pp. 3156–3164.
Wang, G., Sun, C., & Sowmya, A. (2019a). Erl-net: Entangled representation learning for single image de-raining. In IEEE international conference on computer vision, pp. 5644–5652.
Wang, T., Yang, X., Xu, K., Chen, S., Zhang, Q., & Lau, R. W. (2019b). Spatial attentive single-image deraining with a high quality real rain dataset. In IEEE conference on computer vision and pattern recognition, pp. 12270–12279.
Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., & He, X. (2018). Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In IEEE conference on computer vision and pattern recognition, pp. 1316–1324.
Yang, J., Kannan, A., Batra, D., & Parikh, D. (2017a). Lr-gan: Layered recursive generative adversarial networks for image generation. arXiv:1703.01560.
Yang, W., Tan, R. T., Feng, J., Liu, J., Guo, Z., & Yan, S. (2017b). Deep joint rain detection and removal from a single image. In IEEE conference on computer vision and pattern recognition, pp. 1357–1366.
Yang, W., Liu, J., Yang, S., & Guo, Z. (2019a). Scale-free single image deraining via visibility-enhanced recurrent wavelet learning. IEEE Transactions on Image Processing, 28(6), 2948–2961.
Yang, W., Tan, R. T., Wang, S., Fang, Y., & Liu, J. (2019b). Single image deraining: From model-based to data-driven and beyond. arXiv:1912.07150.
Yasarla, R., & Patel, V. M. (2019). Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining. In IEEE conference on computer vision and pattern recognition, pp. 8405–8414.
Yoo, J., Uh, Y., Chun, S., Kang, B., & Ha, J. W. (2019). Photorealistic style transfer via wavelet transforms. In IEEE international conference on computer vision, pp. 9036–9045.
You, S., Tan, R. T., Kawakami, R., Mukaigawa, Y., & Ikeuchi, K. (2015). Adherent raindrop modeling, detectionand removal in video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(9), 1721–1733.
Yuan, B., Wang, C., Jiang, F., Long, M., Yu, P. S., & Liu, Y. (2019). Waveletfcnn: A deep time series classification model for wind turbine blade icing detection. arXiv:1902.05625.
Zagoruyko, S., & Komodakis, N. (2016). Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv:1612.03928.
Zhang, H., & Patel, V. M. (2018). Density-aware single image de-raining using a multi-stream dense network. In IEEE conference on computer vision and pattern recognition, pp. 695–704.
Zhang, H., Goodfellow, I., Metaxasm D., & Odena, A. (2018). Self-attention generative adversarial networks. arXiv:1805.08318.
Zhang, H., Sindagi, V., & Patel, V. M. (2019). Image de-raining using a conditional generative adversarial network. IEEE Transactions on Circuits and Systems for Video Technology.
Zhang, Q., & Benveniste, A. (1992). Wavelet networks. IEEE Transactions on Neural Networks, 3(6), 889–898.
Zhang, R. (2019). Making convolutional networks shift-invariant again. In International conference on machine learning, pp. 7324–7334.
Zhong, Z., Shen, T., Yang, Y., Linm Z., & Zhang, C. (2018). Joint sub-bands learning with clique structures for wavelet domain super-resolution. In Advances in neural information processing systems, pp. 165–175.
Zhu, L., Fu, C. W., Lischinski, D., & Heng, P. A. (2017). Joint bi-layer optimization for single-image rain streak removal. In IEEE international conference on computer vision, pp. 2526–2534.
Acknowledgements
This work is partially funded by National Natural Science Foundation of China (Grant No. 62006228, 61721004, U20A20223).
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by Dengxin Dai.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Huang, H., Yu, A., Chai, Z. et al. Selective Wavelet Attention Learning for Single Image Deraining. Int J Comput Vis 129, 1282–1300 (2021). https://doi.org/10.1007/s11263-020-01421-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-020-01421-z