Skip to main content
SpringerLink
Log in

Joint feedback and recurrent deraining network with ensemble learning

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Rainy images typically contain heterogeneous rain distributions; however, many existing methods perform well in simple homogeneous rain and fail to handle complex heterogeneous rain effectively. In this paper, we try to solve this problem by fully exploiting the complementary contextual information in the manner of a Joint Feedback and Recurrent deraining scheme with Ensemble Learning (JFREL). First, the proposed JFREL is built on a recurrent multistage architecture, and the output of each stage is fused automatically via ensemble learning. Second, the feedback mechanism is utilized to refine information from inter- and intra-stages. Third, at each stage the residual dilated aggregation attention module is recursively adopted to adequately characterize complementary high-level contextual information in multiple receptive fields and adaptively aggregate beneficial details to achieve feature compensation. Extensive experiments demonstrate that the proposed JFREL can achieve a competitive performance over the state-of-the-art methods on both synthetic and real-world datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Ahn, N., Jo, S.Y., Kang, S.J.: Eagnet: elementwise attentive gating network-based single image de-raining with rain simplification. IEEE Trans. Circuits Syst. Video Technol. 32(2), 608–620 (2022). https://doi.org/10.1109/TCSVT.2021.3068985

    Article  Google Scholar 

  2. Cai, L., Li, S.Y., Ren, D., Wang, P.: Dual recursive network for fast image deraining. In: 2019 IEEE International Conference on Image Processing (ICIP), pp. 2756–2760. IEEE (2019)

  3. Chen, Y.L., Hsu, C.T.: A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1968–1975 (2013)

  4. Deng, L.J., Huang, T.Z., Zhao, X.L., Jiang, T.X.: A directional global sparse model for single image rain removal. Appl. Math. Model. 59 (2018)

  5. Deng, S., Wei, M., Wang, J., Feng, Y., Liang, L., Xie, H., Wang, F.L., Wang, M.: Detail-recovery image deraining via context aggregation networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14560–14569 (2020)

  6. Ding, J., Guo, H., Zhou, H., Yu, J., He, X., Jiang, B.: Distributed feedback network for single-image deraining. Inf. Sci. 572, 611–626 (2021)

    Article  MathSciNet  Google Scholar 

  7. Fu, X., Huang, J., Ding, X., Liao, Y., Paisley, J.: Clearing the skies: a deep network architecture for single-image rain removal. IEEE Trans. Image Process. 26(6), 2944–2956 (2017)

    Article  MathSciNet  Google Scholar 

  8. Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., Paisley, J.: Removing rain from single images via a deep detail network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3855–3863 (2017)

  9. Fu, X., Liang, B., Yue, H., Ding, X., Paisley, J.: Lightweight pyramid networks for image deraining. IEEE Trans. Neural Netw. Learn. Syst. (2018)

  10. Fu, Y.H., Kang, L.W., Lin, C.W., Hsu, C.T.: Single-frame-based rain removal via image decomposition. In: IEEE International Conference on Acoustics (2014)

  11. Gu, S., Meng, D., Zuo, W., Zhang, L.: Joint convolutional analysis and synthesis sparse representation for single image layer separation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1708–1716 (2017)

  12. Guo, Z., Hou, M., Sima, M., Feng, Z.: Derainattentiongan: unsupervised single-image deraining using attention-guided generative adversarial networks. SIViP 16(1), 185–192 (2022)

    Article  Google Scholar 

  13. Hu, X., Fu, C.W., Zhu, L., Heng, P.A.: Depth-attentional features for single-image rain removal. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

  14. Huynh-Thu, Q., Ghanbari, M.: Scope of validity of psnr in image/video quality assessment. Electron. Lett. 44(13), 800–801 (2008)

    Article  Google Scholar 

  15. Jiang, K., Wang, Z., Yi, P., Chen, C., Huang, B., Luo, Y., Ma, J., Jiang, J.: Multi-scale progressive fusion network for single image deraining. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8346–8355 (2020)

  16. Jin, X., Chen, Z., Lin, J., Chen, Z., Zhou, W.: Unsupervised single image deraining with self-supervised constraints. In: 2019 IEEE International Conference on Image Processing (ICIP), pp. 2761–2765 (2019). https://doi.org/10.1109/ICIP.2019.8803238

  17. Kang, L.W., Lin, C.W., Fu, Y.H.: Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans. Image Process. 21(4), 1742–1755 (2011)

    Article  MathSciNet  Google Scholar 

  18. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv:1412.6980 (2014)

  19. Li, R., Cheong, L.F., Tan, R.T.: Single image deraining using scale-aware multi-stage recurrent network. arXiv:1712.06830 (2017)

  20. Li, X., Wu, J., Lin, Z., Liu, H., Zha, H.: Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 254–269 (2018)

  21. Li, Y., Tan, R.T., Guo, X., Lu, J., Brown, M.S.: Rain streak removal using layer priors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2736–2744 (2016)

  22. Luo, Y., Xu, Y., Ji, H.: Removing rain from a single image via discriminative sparse coding. In: 2015 IEEE International Conference on Computer Vision (ICCV) (2015)

  23. Qin, X., Wang, Z., Bai, Y., Xie, X., Jia, H.: Ffa-net: Feature fusion attention network for single image dehazing. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 11908–11915 (2020)

  24. Ren, D., Shang, W., Zhu, P., Hu, Q., Meng, D., Zuo, W.: Single image deraining using bilateral recurrent network. IEEE Trans. Image Process. 29, 6852–6863 (2020)

    Article  Google Scholar 

  25. Ren, D., Zuo, W., Hu, Q., Zhu, P., Meng, D.: Progressive image deraining networks: A better and simpler baseline. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3937–3946 (2019)

  26. Takikawa, T., Acuna, D., Jampani, V., Fidler, S.: Gated-scnn: Gated shape cnns for semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5229–5238 (2019)

  27. Wang, C., Zhu, H., Fan, W., Wu, X.M., Chen, J.: Single image rain removal using recurrent scale-guide networks. Neurocomputing 467, 242–255 (2022)

    Article  Google Scholar 

  28. Wang, T., Yang, X., Xu, K., Chen, S., Zhang, Q., Lau, R.W.: Spatial attentive single-image deraining with a high quality real rain dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12270–12279 (2019)

  29. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  30. Wei, W., Meng, D., Zhao, Q., Xu, Z., Wu, Y.: Semi-supervised transfer learning for image rain removal. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

  31. Yang, W., Tan, R.T., Feng, J., Liu, J., Guo, Z., Yan, S.: Deep joint rain detection and removal from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1357–1366 (2017)

  32. Yang, Y., Lu, H.: Single image deraining using a recurrent multi-scale aggregation and enhancement network. In: 2019 IEEE International Conference on Multimedia and Expo (ICME) (2019)

  33. Zhang, H., Patel, V.M.: Density-aware single image de-raining using a multi-stream dense network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 695–704 (2018)

  34. Zhu, L., Fu, C.W., Lischinski, D., Heng, P.A.: Joint bi-layer optimization for single-image rain streak removal. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2526–2534 (2017)

Download references

Acknowledgements

We are sincerely grateful to anonymous reviewers for their insightful comments that improve the quality of this paper. This work was supported by the Project of Guangzhou Science and Technology (Nos. 202102020591, 202007010004 and 202007040005).

Author information

Authors and Affiliations

  1. School of Computer Science, Guangdong University of Technology, Guangzhou, China

    Yu Luo, Menghua Wu, Qingdong Huang, Jian Zhu & Jie Ling

  2. Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China

    Bin Sheng

Authors
  1. Yu Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Menghua Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qingdong Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jian Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jie Ling
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Bin Sheng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Jian Zhu or Jie Ling.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Luo, Y., Wu, M., Huang, Q. et al. Joint feedback and recurrent deraining network with ensemble learning. Vis Comput 38, 3109–3119 (2022). https://doi.org/10.1007/s00371-022-02567-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02567-2

Keywords

Search

Navigation