Skip to main content
SpringerLink
Log in

Twin-stage Unet-like network for single image deraining

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

The performance of visual processing is commonly constrained in extreme outside weather such as heavy rain. Rain streaks may substantially damage image optical quality and impact image processing in many scenarios. Thus, it has practical application value in researching the problem of single image rain removal. However, removing rain streaks from a single image is a challenging task. Although end-to-end learning approaches based on convolutional neural networks have lately made significant progress on this task, most existing methods still cannot perform deraining well. They fail to process the details of the background layer, resulting in the loss of certain information. To address this issue, we propose a single image deraining network named twin-stage Unet-like network (TUNet). Specifically, a reconstitution residual block (RRB) is presented as the basic structure of encoder–decoder to obtain more spatial contextual information for extracting rain components. Then, a residual sampling module (RSM) is introduced to perform downsampling and upsampling operations to preserve residual properties in the structure while obtaining deeper image features. Finally, the convolutional block attention module (CBAM) is adopted to fuse shallow and deep features of the same size in the model. Extensive experiments on five publicly synthetic datasets and a real-world dataset demonstrate that our proposed TUNet model outperforms the state-of-the-art deraining approaches. The average PSNR value of TUNet is 0.41 dB higher than the state-of-the-art method (OSAM-Net) on synthetic datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. He, K., Gkioxari, G., Dollár, P., et al.: Mask r-cnn. In: IEEE International Conference on Computer Vision (2017)

  2. Zhou, W.N., Zhou, Y., Zeng, X.Y.: An attention nested u-structure suitable for salient ship detection in complex maritime environment. IEICE Trans. Inf. Syst. 6, 1–7 (2022)

    Google Scholar 

  3. Chen, Y.C., Lin, Y.Y., Yang, M.H., et al.: Show, match and segment: joint weakly supervised learning of semantic matching and object co-segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 43(10), 3632–3647 (2020)

    Article  Google Scholar 

  4. Li, F., Tian, C., Zuo, W., et al.: Learning spatial-temporal regularized correlation filters for visual tracking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4904–4913 (2018)

  5. Chen, J., Tan, C.H., Hou, J., et al.: Robust video content alignment and compensation for rain removal in a CNN framework. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6286–6295 (2018)

  6. Bossu, J., Hautiere, N., Tarel, J.P.: Rain or snow detection in image sequences through use of a histogram of orientation of streaks. Int. J. Comput. Vis. 93(3), 348–367 (2011)

    Article  Google Scholar 

  7. He, K., Sun, J., Tang, X.: Single image haze removal using dark channel prior. IEEE Trans. Pattern Anal. Mach. Intell. 33(12), 2341–2353 (2010)

    PubMed  Google Scholar 

  8. Celik, T., Tjahjadi, T.: Contextual and variational contrast enhancement. IEEE Trans. Image Process. 20(12), 3431–3441 (2011)

    Article  MathSciNet  PubMed  ADS  Google Scholar 

  9. Kang, L.W., Lin, C.W., Fu, Y.H.: Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans. Image Process. 21(4), 1742–1755 (2011)

    Article  MathSciNet  PubMed  ADS  Google Scholar 

  10. Luo, Y., Xu, Y., Ji, H.: Removing rain from a single image via discriminative sparse coding. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3397–3405 (2015)

  11. Li, Y., Tan, R.T., Guo, X., et al.: Rain streak removal using layer priors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2736–2744 (2016)

  12. Fu, X., Huang, J., Zeng, D., et al.: Removing rain from single images via a deep detail network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3855–3863 (2017)

  13. Li, X., Wu, J., Lin, Z., et al.: Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 254–269 (2018)

  14. Ren, D., Zuo, W., Hu, Q., et al.: Progressive image deraining networks: a better and simpler baseline. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3937–3946 (2019)

  15. Jiang, K., Wang, Z., Yi, P., et al.: Multi-scale progressive fusion network for single image deraining. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

  16. Hu, J., Shen, L., Albanie, S., et al.: Gather-excite: exploiting feature context in convolutional neural networks. In: Advances in Neural Information Processing Systems, vol. 31 (2018)

  17. Fan, Z., Wu, H., Fu, X., et al.: Residual-guide network for single image deraining. In: Proceedings of the 26th ACM International Conference on Multimedia, pp. 1751–1759 (2018)

  18. Chen, P., Xie, Z., Fang, Y., et al.: Physical-layer network coding: An efficient technique for wireless communications. IEEE Netw. 34(2), 270–276 (2019)

    Article  Google Scholar 

  19. Ren, S., He, K., Girshick, R., et al.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, vol. 28 (2015)

  20. Zhou, W.N., Chen, K.: A lightweight hand gesture recognition in complex backgrounds. Displays 102226 (2022)

  21. Zamir, S.W., Arora, A., Khan, S., et al.: Multi-stage progressive image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14821–14831 (2021)

  22. Chen, L., Lu, X., Zhang, J., et al.: Hinet: half instance normalization network for image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 182–192 (2021)

  23. Yan, L., Zhao, M., Liu, S., et al.: Cascaded transformer u-net for image restoration. Signal Process. 206(108), 902 (2023)

    Google Scholar 

  24. Frants, V., Agaian, S., Panetta, K.: Qsam-net: rain streak removal by quaternion neural network with self-attention module. IEEE Trans. Multimedia (2023)

  25. Liu, Y., Zhang, R., Zhang, Y., et al.: Single image deraining via a recurrent multi-attention enhancement network. Signal Process. Image Commun. 113(116), 926 (2023)

    Google Scholar 

  26. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)

  27. Woo, S., Park, J., Lee, J.Y., et al.: Cbam: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)

  28. Chen, X., Huang, Y., Xu, L.: Multi-scale hourglass hierarchical fusion network for single image deraining. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 872–879 (2021)

  29. Guo, M.H., Xu, T.X., Liu, J.J., et al.: Attention mechanisms in computer vision: a survey. Comput. Vis. Media 1–38 (2022)

  30. Wang, C., Fan, W., Zhu, H., et al.: Single image deraining via nonlocal squeeze-and-excitation enhancing network. Appl. Intell. 50(9), 2932–2944 (2020)

    Article  Google Scholar 

  31. Almushyti, M., Li, F.W.: Distillation of human-object interaction contexts for action recognition. arXiv preprint arXiv:2112.09448 (2021)

  32. Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1501–1510 (2017)

  33. Kim, Y., Soh, J.W., Park, G.Y., et al.: Transfer learning from synthetic to real-noise denoising with adaptive instance normalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3482–3492 (2020)

  34. Zhang, R.: Making convolutional networks shift-invariant again. In: International Conference on Machine Learning, PMLR, pp. 7324–7334 (2019)

  35. Huynh-Thu, Q., Ghanbari, M.: Scope of validity of PSNR in image/video quality assessment. Electron. Lett. 44(13), 800–801 (2008)

    Article  ADS  Google Scholar 

  36. Yang, W., Tan, R.T., Feng, J., et al.: Deep joint rain detection and removal from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1357–1366 (2017)

  37. Zhang, H., Sindagi, V., Patel, V.M.: Image de-raining using a conditional generative adversarial network. IEEE Trans. Circuits Syst. Video Technol. 30(11), 3943–3956 (2019)

    Article  Google Scholar 

  38. Wei, W., Meng, D., Zhao, Q., et al.: Semi-supervised transfer learning for image rain removal. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3877–3886 (2019)

  39. Yang, W., Tan, R.T., Feng, J., et al.: Deep joint rain detection and removal from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1357–1366 (2017)

  40. Wang, T., Yang, X., Xu, K., et al.: Spatial attentive single-image deraining with a high quality real rain dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12,270–12,279 (2019)

  41. Wang, Z., Bovik, A.C., Sheikh, H.R., et al.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  PubMed  ADS  Google Scholar 

  42. Zhang, H., Patel, V.M.: Density-aware single image de-raining using a multi-stream dense network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 695–704 (2018)

  43. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

Download references

Funding

This work was supported by the National Natural Science Foundation of China (Grant No. 61404083) and State Key Laboratory of ASIC & System (2021KF010).

Author information

Authors and Affiliations

  1. College of Information Engineering, Shanghai Maritime University, Shanghai, China

    Weina Zhou & Xiu Wang

Authors
  1. Weina Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

W.Z. and X.W. contributed to the conception of the study; X.W. performed the experiment; W.Z. and X.W. contributed significantly to analysis and manuscript preparation; X.W. performed the data analyses and wrote the manuscript; W.Z. and X.W. helped perform the analysis with constructive discussions. All authors have read and agreed to the published version of the manuscript. All authors contributed to the study conception and design.

Corresponding author

Correspondence to Weina Zhou.

Ethics declarations

Conflict of interest

All authors declare that they have no conflict of interest.

Ethical approval

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, W., Wang, X. Twin-stage Unet-like network for single image deraining. SIViP 18, 1285–1293 (2024). https://doi.org/10.1007/s11760-023-02824-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-023-02824-4

Keywords

Search

Navigation