Skip to main content
SpringerLink
Log in

SRENet: Structure recovery ensemble network for single image deraining

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Single-image deraining remains a formidable challenge. Its objective is not only to eliminate rain streaks from the target image but also to restore its spatial details and high-level contextual structure. Recently, numerous CNN-based methods have been introduced for this purpose. While these methods can adeptly remove rain streaks, they often face difficulties in restoring high-quality, rain-free images while preserving clear and precise structures—especially when those structures resemble rain streaks. To extract a diverse array of feature information for image structure restoration, we introduce a network called the Structure Recovery Ensemble Network (SRENet). This network comprises three learners, each designed with three parallel subnetworks. This arrangement allows for the acquisition of a broader range of structural features by transitioning from deep vertical networks to horizontal parallelization across multiple subnetworks. To extract structural features without interference from rain streak information, we employ an independent loss strategy in which each learner is trained to address different specific challenges in the image deraining process, such as rain removal and structure restoration. Additionally, we design a guided fusion module to seamlessly integrate features from the various learners and subnetworks. Comprehensive experiments on benchmark datasets confirm that our proposed method sets new state-of-the-art standards. On the synthetic datasets, SRENet achieved average improvements of 0.85% in SSIM and 0.81% in PSNR compared to mainstream benchmarks. On real-world rainy images, SRENet demonstrates improvements of 6.71% in NIQE and 11.4% in BRISQUE.

Graphical Abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availability

Data will be made available on request.

References

  1. Li S, Ren W, Wang F, Araujo IB, Tokuda EK, Junior RH, Cesar RM, Wang Z, Cao X (2021) A Comprehensive Benchmark Analysis of Single Image Deraining: Current Challenges and Future Perspectives. Int J Comput Vis 129:1301–1322. https://doi.org/10.1007/s11263-020-01416-w

    Article  Google Scholar 

  2. Zhang Z, Wei Y, Zhang H, Yang Y, Yan S, Wang M (2023) Data-Driven single image deraining: A Comprehensive review and new perspectives. Pattern Recognit 143:109740. https://doi.org/10.1016/j.patcog.2023.109740

  3. Jiang Y, Zhu B, Zhao X, Deng W (2023) Pixel-wise content attention learning for single-image deraining of autonomous vehicles. Expert Syst Appl 224:119990

    Article  Google Scholar 

  4. Hsu W-Y, Chang W-C (2023) Recurrent wavelet structure-preserving residual network for single image deraining. Pattern Recogn 137:109294

    Article  Google Scholar 

  5. Luo Y, Huang Q, Ling J, Lin K, Zhou T (2023) Local and global knowledge distillation with direction-enhanced contrastive learning for single-image deraining. Knowl-Based Syst 268:110480

    Article  Google Scholar 

  6. Wang Q, Sun G, Dong J, Zhang Y (2022) Pfdn: Pyramid feature decoupling network for single image deraining. IEEE Trans Image Process 31:7091–7101

    Article  Google Scholar 

  7. Yang W, Tan RT, Wang S, Fang Y, Liu J (2020) Single image deraining: From model-based to data-driven and beyond. IEEE Trans Pattern Anal Mach Intell 43:4059–4077

    Article  Google Scholar 

  8. Zamir SW, Arora A, Khan S, Hayat M, Khan FS, Yang M-H (2022) Restormer: Efficient transformer for high-resolution image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 5728–5739

  9. Zhang Y-N, Shen L, Lai Z (2023) Adaptive weighted rain streaks model-driven deep network for single image deraining. Expert Syst Appl 222:119807

    Article  Google Scholar 

  10. Liu B, Fang S (2023) Multi-level wavelet network based on CNN-Transformer hybrid attention for single image deraining. Neural Comput & Applic 35:22387–22404. https://doi.org/10.1007/s00521-023-08899-x

    Article  Google Scholar 

  11. Yang Y, Lu H (2019) Single image deraining using a recurrent multi-scale aggregation and enhancement network. In: 2019 IEEE International Conference on Multimedia and Expo (ICME), pp 1378–1383. https://doi.org/10.1109/ICME.2019.00239

  12. Yi Q, Li J, Dai Q, Fang F, Zhang G, Zeng T (2021) Structure-preserving deraining with residue channel prior guidance. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp 4238–4247

  13. Deng S, Wei M, Wang J, Feng Y, Liang L, Xie H, Wang FL, Wang M (2020) Detail-recovery image deraining via context aggregation networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

  14. Carino-Escobar RI, Alonso-Silverio GA, Alarcón-Paredes A, Cantillo-Negrete J (2023) Feature-ranked self-growing forest: a tree ensemble based on structure diversity for classification and regression. Neural Comput & Applic 35:9285–9298. https://doi.org/10.1007/s00521-023-08202-y

    Article  Google Scholar 

  15. Qasem A, Sheikh Abdullah SNH, Sahran S, Albashish D, Goudarzi S, Arasaratnam S (2022) An improved ensemble pruning for mammogram classification using modified Bees algorithm. Neural Comput & Applic 34:10093–10116. https://doi.org/10.1007/s00521-022-06995-y

    Article  Google Scholar 

  16. Zhou K, Yang Y, Qiao Y, Xiang T (2021) Domain adaptive ensemble learning. IEEE Trans Image Process 30:8008–8018

    Article  Google Scholar 

  17. Lyu Q, Shan H, Wang G (2020) MRI super-resolution with ensemble learning and complementary priors. IEEE Transactions on Computational Imaging 6:615–624

    Article  Google Scholar 

  18. Albaba BM, Ozer S (2021) SyNet: An ensemble network for object detection in UAV images. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp 10227–10234. https://doi.org/10.1109/ICPR48806.2021.9412847

  19. Paul A, Basu A, Mahmud M, Kaiser MS, Sarkar R (2023) Inverted bell-curve-based ensemble of deep learning models for detection of COVID-19 from chest X-rays. Neural Comput & Applic 35:16113–16127. https://doi.org/10.1007/s00521-021-06737-6

    Article  Google Scholar 

  20. Dua M, Shakshi SR, Raj S, Jangra A (2021) Deep CNN models-based ensemble approach to driver drowsiness detection. Neural Comput & Applic 33:3155–3168. https://doi.org/10.1007/s00521-020-05209-7

    Article  Google Scholar 

  21. Ganaie MA, Hu M, Malik AK, Tanveer M, Suganthan PN (2022) Ensemble deep learning: A review. Eng Appl Artif Intell 115:105151

    Article  Google Scholar 

  22. Zeiler MD, Fergus R (2014) Visualizing and Understanding Convolutional Networks. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T (eds) Computer Vision – ECCV 2014. Springer International Publishing, Cham, pp 818–833

    Chapter  Google Scholar 

  23. Yang W, Tan RT, Feng J, Liu J, Guo Z, Yan S (2017) Deep joint rain detection and removal from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 1357–1366

  24. Yang Y, Lu H (2021) A fast and efficient network for single image deraining. In: ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp 2030–2034. https://doi.org/10.1109/ICASSP39728.2021.9414978

  25. Jiang K, Wang Z, Yi P, Chen C, Huang B, Luo Y, Ma J, Jiang J (2020) Multi-scale progressive fusion network for single image deraining. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 8346–8355

  26. Yasarla R, Patel VM (2019) Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 8405–8414

  27. Yamamichi K, Han X-H (2021) Lightweight Multi-Scale context aggregation deraining network with Artifact-Attenuating pooling and activation functions. IEEE Access 9:146948–146958

    Article  Google Scholar 

  28. Fu X, Liang B, Huang Y, Ding X, Paisley J (2019) Lightweight pyramid networks for image deraining. IEEE transactions on neural networks and learning systems 31:1794–1807

    Article  Google Scholar 

  29. Yasarla R, Patel VM (2020) Confidence measure guided single image de-raining. IEEE Trans Image Process 29:4544–4555

    Article  Google Scholar 

  30. Zamir SW, Arora A, Khan S, Hayat M, Khan FS, Yang M-H, Shao L (2021) Multi-stage progressive image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 14821–14831

  31. Zhang H, Patel VM (2018) Density-aware single image de-raining using a multi-stream dense network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 695–704

  32. Wang Y, Song Y, Ma C, Zeng B (2020) Rethinking Image Deraining via Rain Streaks and Vapors. In: Vedaldi A, Bischof H, Brox T, Frahm J-M (eds) Computer Vision – ECCV 2020. Springer International Publishing, Cham, pp 367–382

    Chapter  Google Scholar 

  33. Zhang Y, Guo J, Li J, Zhang J (2023) Single-image Deraining via a channel memory network. Appl Intell 53:1009–1020. https://doi.org/10.1007/s10489-022-03441-3

    Article  Google Scholar 

  34. Han K, Wang Y, Chen H, Chen X, Guo J, Liu Z, Tang Y, Xiao A, Xu C, Xu Y (2022) A survey on vision transformer. IEEE Trans Pattern Anal Mach Intell 45:87–110

    Article  Google Scholar 

  35. Zuo S, Xiao Y, Chang X, Wang X (2022) Vision transformers for dense prediction: A survey. Knowl-Based Syst 253:109552

    Article  Google Scholar 

  36. Meng X, Wang N, Shao F, Li S (2022) Vision transformer for pansharpening. IEEE Trans Geosci Remote Sens 60:1–11

    Google Scholar 

  37. Huang J, Tang Z, He X, Zhou J, Zhou D, Chen CY-C (2024) Progressive network based on detail scaling and texture extraction: A more general framework for image deraining. Neurocomputing 568:127066. https://doi.org/10.1016/j.neucom.2023.127066

  38. Liang Y, Anwar S, Liu Y (2022) DRT: A lightweight single image deraining recursive transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp 589–598

  39. Yang W, Tan RT, Feng J, Guo Z, Yan S, Liu J (2019) Joint rain detection and removal from a single image with contextualized deep networks. IEEE Trans Pattern Anal Mach Intell 42:1377–1393

    Article  Google Scholar 

  40. Wang T, Yang X, Xu K, Chen S, Zhang Q, Lau RW (2019) Spatial attentive single-image deraining with a high quality real rain dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 12270–12279

  41. Wang H, Xie Q, Zhao Q, Meng D (2020) A model-driven deep neural network for single image rain removal. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 3103–3112

  42. Sujit S, Deivalakshmi S, Ko S-B (2022) Factorized multi-scale multi-resolution residual network for single image deraining. Appl Intell 52:7582–7598. https://doi.org/10.1007/s10489-021-02772-x

    Article  Google Scholar 

  43. Yang H, Zhou D, Li M, Zhao Q (2023) A two-stage network with wavelet transformation for single-image deraining. Vis Comput 39:3887–3903. https://doi.org/10.1007/s00371-022-02533-y

    Article  Google Scholar 

  44. Peng L, Jiang A, Wei H, Liu B, Wang M (2021) Ensemble single image deraining network via progressive structural boosting constraints. Signal Processing: Image Communication 99:116460

    Google Scholar 

  45. Wang C, Fan W, Zhu H, Su Z (2020) Single image deraining via nonlocal squeeze-and-excitation enhancing network. Appl Intell 50:2932–2944. https://doi.org/10.1007/s10489-020-01693-5

    Article  Google Scholar 

  46. Yang H, Zhou D, Cao J, Zhao Q (2022) DPNet: Detail-preserving image deraining via learning frequency domain knowledge. Digital Signal Processing 130:103740

    Article  Google Scholar 

  47. Gao F, Mu X, Ouyang C, Yang K, Ji S, Guo J, Wei H, Wang N, Ma L, Yang B (2022) MLTDNet: an efficient multi-level transformer network for single image deraining. Neural Comput & Applic 34:14013–14027. https://doi.org/10.1007/s00521-022-07226-0

    Article  Google Scholar 

  48. Li J, Liu Z (2019) Ensemble dictionary learning for single image deblurring via low-rank regularization. Sensors 19:1143

    Article  Google Scholar 

  49. Chen M, Quan Y, Xu Y, Ji H (2022) Self-supervised blind image deconvolution via deep generative ensemble learning. IEEE Trans Circuits Syst Video Technol 33:634–647

    Article  Google Scholar 

  50. Aetesam H, Maji SK (2021) Noise dependent training for deep parallel ensemble denoising in magnetic resonance images. Biomed Signal Process Control 66:102405

    Article  Google Scholar 

  51. Chen X, Chen H, Yang Y, Wu H, Zhang W, Zhao J, Xiong Y (2021) Traffic flow prediction by an ensemble framework with data denoising and deep learning model. Physica A 565:125574

    Article  Google Scholar 

  52. Yang X, Xu Y, Quan Y, Ji H (2020) Image denoising via sequential ensemble learning. IEEE Trans Image Process 29:5038–5049

    Article  Google Scholar 

  53. Sahoo DK, Das A, Mohanty MN, Mishra S (2022) Brain tumor detection using inpainting and deep ensemble model. J Inf Optim Sci 43:1925–1933. https://doi.org/10.1080/02522667.2022.2091094

    Article  Google Scholar 

  54. Shin Y-G, Sagong M-C, Yeo Y-J, Kim S-W, Ko S-J (2020) Pepsi++: Fast and lightweight network for image inpainting. IEEE transactions on neural networks and learning systems 32:252–265

    Article  Google Scholar 

  55. Gupta C, Kuchibhotla AK, Ramdas A (2022) Nested conformal prediction and quantile out-of-bag ensemble methods. Pattern Recogn 127:108496

    Article  Google Scholar 

  56. Yang Y, Ran W, Lu H (2020) Rddan: A residual dense dilated aggregated network for single image deraining. In: 2020 IEEE International Conference on Multimedia and Expo (ICME), pp 1–6. https://doi.org/10.1109/ICME46284.2020.9102945

  57. Fu X, Huang J, Zeng D, Huang Y, Ding X, Paisley J (2017) Removing rain from single images via a deep detail network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 3855–3863

  58. Wang X, Zhang R, Kong T, Li L, Shen C (2020) Solov2: Dynamic and fast instance segmentation. Adv Neural Inf Process Syst 33:17721–17732

    Google Scholar 

Download references

Acknowledgements

This work was partially supported by the National Natural Science Foundation of China (No.42306226 and 61973208) and National Science Fund for Distinguished Young Scholars (No.62225308). This work was supported by the Natural Science Foundation of Shanghai (No.23ZR1422800).This work was supported in part by the Shanghai Municipal Natural Science Foundation under Grant 21ZR1423300.

Author information

Authors and Affiliations

  1. School of Mechatronic Engineering and Automation, Shanghai University, Shanghai, 200444, China

    Dan Zhang, Yingbing Xu, Xiaowei Li, Xiangyu Zhang & Yan Peng

  2. School of Computer Engineering and Science, Shanghai University, Shanghai, 200444, China

    Liyan Ma

  3. Institute of Artificial Intelligence, Collaborative Innovation Center for the Marine Artificial Intelligence, Shanghai University, Shanghai, 200444, China

    Yaoran Chen

Authors
  1. Dan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yingbing Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Liyan Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaowei Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiangyu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yan Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yaoran Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Dan Zhang: Methodology, Software, Reviewing, Supervision.

Yingbing Xu: Writing, Methodology, Software.

Liyan Ma: Reviewing, Supervision.

Xiaowei Li: Reviewing, Supervision.

Xiangyu Zhang: Reviewing.

Yan Peng: Conceptualization, Supervision.

Yaoran Chen: Writing, Reviewing.

Corresponding author

Correspondence to Yaoran Chen.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest.

Ethical and informed consent

As this study involved a secondary analysis of publicly available data, ethical approval was not sought. All participants provided written informed consent. They were informed about the study’s purpose, potential risks and benefits, confidentiality measures, and their right to withdraw at any time without any consequences.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, D., Xu, Y., Ma, L. et al. SRENet: Structure recovery ensemble network for single image deraining. Appl Intell 54, 4425–4442 (2024). https://doi.org/10.1007/s10489-024-05382-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-024-05382-5

Keywords

Search

Navigation