Skip to main content
SpringerLink
Log in

Selective Wavelet Attention Learning for Single Image Deraining

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Single image deraining refers to the process of restoring the clean background scene from a rainy image. Current approaches have resorted to deep learning techniques to remove rain from a single image by leveraging some prior information. However, due to the various appearances of rain streaks and accumulation, it is difficult to separate rain and background information in the embedding space, which results in inaccurate deraining. To address this issue, this paper proposes a selective wavelet attention learning method by learning a series of wavelet attention maps to guide the separation of rain and background information in both spatial and frequency domains. The key aspect of our method is utilizing wavelet transform to learn the content and structure of rainy features because the high-frequency features are more sensitive to rain degradations, whereas the low-frequency features preserve more of the background content. To begin with, we develop a selective wavelet attention encoder–decoder network to learn wavelet attention maps guiding the separation of rainy and background features at multiple scales. Meanwhile, we introduce wavelet pooling and unpooling to the encoder–decoder network, which shows superiority in learning increasingly abstract representations while preserving the background details. In addition, we propose latent alignment learning to supervise the background features as well as augment the training data to further improve the accuracy of deraining. Finally, we employ a hierarchical discriminator network based on selective wavelet attention to adversarially improve the visual fidelity of the generated results both globally and locally. Extensive experiments on synthetic and real datasets demonstrate that the proposed approach achieves more appealing results both quantitatively and qualitatively than the recent state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  • Billings, S. A., & Wei, H. L. (2005). A new class of wavelet networks for nonlinear system identification. IEEE Transactions on Neural Networks, 16(4), 862–874.

    Article  Google Scholar 

  • Chen, X., Xu, C., Yang, X., & Tao, D. (2018). Attention-gan for object transfiguration in wild images. In European conference on computer vision, pp. 164–180.

  • De Silva, D., Vithanage, H., Fernando, K., & Piyatilake, I. (2020). Multi-path learnable wavelet neural network for image classification. In International conference on machine vision, vol. 11433, p. 1143310.

  • Deco, G., & Zihl, J. (2001). A neurodynamical model of visual attention: Feedback enhancement of spatial resolution in a hierarchical system. Journal of Computational Neuroscience, 10(3), 231–253.

    Article  Google Scholar 

  • Deng, X., Yang, R., Xu, M., & Dragotti, P. L. (2019). Wavelet domain style transfer for an effective perception-distortion tradeoff in single image super-resolution. In IEEE international conference on computer vision, pp. 3076–3085.

  • Eigen, D., Krishnan, D., & Fergus, R. (2013). Restoring an image taken through a window covered with dirt or rain. In IEEE international conference on computer vision, pp. 633–640.

  • Fan, Z., Wu, H., Fu, X., Huang, Y., & Ding, X. (2018). Residual-guide network for single image deraining. In ACM international conference on multimedia, pp. 1751–1759.

  • Fu, X., Huang, J., Ding, X., Liao, Y., & Paisley, J. (2017a). Clearing the skies: A deep network architecture for single-image rain removal. IEEE Transactions on Image Processing, 26(6), 2944–2956.

    Article  MathSciNet  Google Scholar 

  • Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., & Paisley, J. (2017b). Removing rain from single images via a deep detail network. In IEEE conference on computer vision and pattern recognition, pp. 3855–3863.

  • Fu, X., Liang, B., Huang, Y., Ding, X., & Paisley, J. (2019). Lightweight pyramid networks for image deraining. IEEE Transactions on Neural Networks and Learning Systems.

  • Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial nets. In Advances in neural information processing systems, pp. 2672–2680.

  • Hu, J., Shen, L., & Sun, G. (2018). Squeeze-and-excitation networks. In IEEE conference on computer vision and pattern recognition, pp. 7132–7141.

  • Hu, X., Fu, C. W., Zhu, L., & Heng, P. A. (2019). Depth-attentional features for single-image rain removal. In IEEE conference on computer vision and pattern recognition, pp. 8022–8031.

  • Huang, H., He, R., Sun, Z., & Tan, T. (2017). Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution. In IEEE international conference on computer vision, pp. 1689–1697.

  • Huang, H., He, R., Sun, Z., & Tan, T. (2019). Wavelet domain generative adversarial network for multi-scale face hallucination. International Journal of Computer Vision, 127(6–7), 763–784.

    Article  Google Scholar 

  • Isola, P., Zhu, J. Y., Zhou, T., & Efros, A. A. (2017). Image-to-image translation with conditional adversarial networks. In IEEE conference on computer vision and pattern recognition, pp. 1125–1134.

  • Kang, L. W., Lin, C. W., & Fu, Y. H. (2011). Automatic single-image-based rain streaks removal via image decomposition. IEEE Transactions on Image Processing, 21(4), 1742–1755.

    Article  MathSciNet  Google Scholar 

  • Kuen, J., Wang, Z., & Wang, G. (2016). Recurrent attentional networks for saliency detection. In IEEE conference on computer vision and pattern recognition, pp. 3668–3677.

  • Li, G., He, X., Zhang, W., Chang, H., Dong, L., & Lin, L. (2018a). Non-locally enhanced encoder-decoder network for single image de-raining. In ACM international conference on multimedia, pp. 1056–1064.

  • Li, R., Cheong, L. F., & Tan, R. T. (2017). Single image deraining using scale-aware multi-stage recurrent network. arXiv:1712.06830.

  • Li, R., Cheong, L. F., & Tan, R. T. (2019). Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In IEEE conference on computer vision and pattern recognition, pp. 1633–1642.

  • Li, X., Wu, J., Lin, Z., Liu, H., & Zha, H. (2018b). Recurrent squeeze-and-excitation context aggregation net for single image deraining. In European conference on computer vision, pp. 254–269.

  • Li, Y., Tan, R. T., Guo, X., Lu, J., & Brown, M. S. (2016). Rain streak removal using layer priors. In IEEE conference on computer vision and pattern recognition, pp. 2736–2744.

  • Lim, B., Son, S., Kim, H., Nah, S., & Mu Lee, K. (2017). Enhanced deep residual networks for single image super-resolution. In IEEE conference on computer vision and pattern recognition workshops, pp. 136–144.

  • Liu, P., Zhang, H., Zhang, K., Lin, L., & Zuo, W. (2018). Multi-level wavelet-cnn for image restoration. In IEEE conference on computer vision and pattern recognition workshops, pp. 773–782.

  • Luo, Y., Xu, Y., & Ji, H. (2015). Removing rain from a single image via discriminative sparse coding. In IEEE international conference on computer vision, pp. 3397–3405.

  • Mallat, S. (1996). Wavelets for a vision. Proceedings of the IEEE, 84(4), 604–614.

    Article  Google Scholar 

  • Mallat, S. G. (1989). A theory for multiresolution signal decomposition: The wavelet representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(7), 674–693.

    Article  Google Scholar 

  • Mao, X., Li, Q., Xie, H., Lau, R. Y., Wang, Z., & Smolley, S. P. (2017). Least squares generative adversarial networks. In IEEE international conference on computer vision, pp. 2813–2821.

  • Mirza, M., & Osindero, S. (2014). Conditional generative adversarial nets. arXiv:1411.1784.

  • Pan, J., Liu, S., Sun, D., Zhang, J., Liu, Y., Ren, J., Li, Z., Tang, J., Lu, H., Tai, Y. W., et al. (2018). Learning dual convolutional neural networks for low-level vision. In IEEE conference on computer vision and pattern recognition, pp. 3070–3079.

  • Provenzi, E., & Caselles, V. (2014). A wavelet perspective on variational perceptually-inspired color enhancement. International Journal of Computer Vision, 106(2), 153–171.

    Article  MathSciNet  Google Scholar 

  • Qian, R., Tan, R. T., Yang, W., Su, J., & Liu, J. (2018). Attentive generative adversarial network for raindrop removal from a single image. In IEEE conference on computer vision and pattern recognition, pp. 2482–2491.

  • Ren, D., Zuo, W., Hu, Q., Zhu, P., & Meng, D. (2019). Progressive image deraining networks: A better and simpler baseline. In IEEE conference on computer vision and pattern recognition, pp. 3937–3946.

  • Rensink, R. A. (2000). The dynamic representation of scenes. Visual Cognition, 7(1–3), 17–42.

    Article  Google Scholar 

  • Szu, H. H., Telfer, B. A., & Kadambe, S. L. (1992). Neural network adaptive wavelets for signal representation and classification. Optical Engineering, 31(9), 1907–1917.

    Article  Google Scholar 

  • Treisman, A. (1982). Perceptual grouping and attention in visual search for features and for objects. Journal of Experimental Psychology: Human Perception and Performance, 8(2), 194.

    Google Scholar 

  • Treisman, A. M., & Gelade, G. (1980). A feature-integration theory of attention. Cognitive Psychology, 12(1), 97–136.

    Article  Google Scholar 

  • Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., & Tang, X. (2017). Residual attention network for image classification. In IEEE conference on computer vision and pattern recognition, pp. 3156–3164.

  • Wang, G., Sun, C., & Sowmya, A. (2019a). Erl-net: Entangled representation learning for single image de-raining. In IEEE international conference on computer vision, pp. 5644–5652.

  • Wang, T., Yang, X., Xu, K., Chen, S., Zhang, Q., & Lau, R. W. (2019b). Spatial attentive single-image deraining with a high quality real rain dataset. In IEEE conference on computer vision and pattern recognition, pp. 12270–12279.

  • Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., & He, X. (2018). Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In IEEE conference on computer vision and pattern recognition, pp. 1316–1324.

  • Yang, J., Kannan, A., Batra, D., & Parikh, D. (2017a). Lr-gan: Layered recursive generative adversarial networks for image generation. arXiv:1703.01560.

  • Yang, W., Tan, R. T., Feng, J., Liu, J., Guo, Z., & Yan, S. (2017b). Deep joint rain detection and removal from a single image. In IEEE conference on computer vision and pattern recognition, pp. 1357–1366.

  • Yang, W., Liu, J., Yang, S., & Guo, Z. (2019a). Scale-free single image deraining via visibility-enhanced recurrent wavelet learning. IEEE Transactions on Image Processing, 28(6), 2948–2961.

    Article  MathSciNet  Google Scholar 

  • Yang, W., Tan, R. T., Wang, S., Fang, Y., & Liu, J. (2019b). Single image deraining: From model-based to data-driven and beyond. arXiv:1912.07150.

  • Yasarla, R., & Patel, V. M. (2019). Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining. In IEEE conference on computer vision and pattern recognition, pp. 8405–8414.

  • Yoo, J., Uh, Y., Chun, S., Kang, B., & Ha, J. W. (2019). Photorealistic style transfer via wavelet transforms. In IEEE international conference on computer vision, pp. 9036–9045.

  • You, S., Tan, R. T., Kawakami, R., Mukaigawa, Y., & Ikeuchi, K. (2015). Adherent raindrop modeling, detectionand removal in video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(9), 1721–1733.

    Article  Google Scholar 

  • Yuan, B., Wang, C., Jiang, F., Long, M., Yu, P. S., & Liu, Y. (2019). Waveletfcnn: A deep time series classification model for wind turbine blade icing detection. arXiv:1902.05625.

  • Zagoruyko, S., & Komodakis, N. (2016). Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv:1612.03928.

  • Zhang, H., & Patel, V. M. (2018). Density-aware single image de-raining using a multi-stream dense network. In IEEE conference on computer vision and pattern recognition, pp. 695–704.

  • Zhang, H., Goodfellow, I., Metaxasm D., & Odena, A. (2018). Self-attention generative adversarial networks. arXiv:1805.08318.

  • Zhang, H., Sindagi, V., & Patel, V. M. (2019). Image de-raining using a conditional generative adversarial network. IEEE Transactions on Circuits and Systems for Video Technology.

  • Zhang, Q., & Benveniste, A. (1992). Wavelet networks. IEEE Transactions on Neural Networks, 3(6), 889–898.

    Article  Google Scholar 

  • Zhang, R. (2019). Making convolutional networks shift-invariant again. In International conference on machine learning, pp. 7324–7334.

  • Zhong, Z., Shen, T., Yang, Y., Linm Z., & Zhang, C. (2018). Joint sub-bands learning with clique structures for wavelet domain super-resolution. In Advances in neural information processing systems, pp. 165–175.

  • Zhu, L., Fu, C. W., Lischinski, D., & Heng, P. A. (2017). Joint bi-layer optimization for single-image rain streak removal. In IEEE international conference on computer vision, pp. 2526–2534.

Download references

Acknowledgements

This work is partially funded by National Natural Science Foundation of China (Grant No. 62006228, 61721004, U20A20223).

Author information

Authors and Affiliations

  1. National Laboratory of Pattern Recognition, CASIA, Beijing, China

    Huaibo Huang, Aijing Yu, Ran He & Tieniu Tan

  2. Center for Excellence in Brain Science and Intelligence Technology, CAS, Beijing, China

    Ran He & Tieniu Tan

  3. Center for Research on Intelligent Perception and Computing, CASIA, Beijing, China

    Huaibo Huang, Aijing Yu, Ran He & Tieniu Tan

  4. School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China

    Huaibo Huang, Aijing Yu, Ran He & Tieniu Tan

  5. Vision Intelligence Center, AI Platform, Meituandianping Group, Beijing, China

    Zhenhua Chai

Authors
  1. Huaibo Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aijing Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhenhua Chai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ran He
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tieniu Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ran He.

Additional information

Communicated by Dengxin Dai.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, H., Yu, A., Chai, Z. et al. Selective Wavelet Attention Learning for Single Image Deraining. Int J Comput Vis 129, 1282–1300 (2021). https://doi.org/10.1007/s11263-020-01421-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-020-01421-z

Keywords

Search

Navigation