Skip to main content
SpringerLink
Log in

LightweightDeRain: learning a lightweight multi-scale high-order feedback network for single image de-raining

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

In recent years, deep convolutional neural networks have gained significant performance in single image de-raining. However, this progress is contributing to their complicated model design. These complicated models generally contain a huge number of parameters, resulting in high memory footprints and low efficiency. To handle this issue, we propose a novel Lightweight Multi-scale High-order Feedback Network (LMHFNet) for single image de-raining. First, we regard the de-raining problem as a multi-stage task and combine a high-order feedback mechanism with global residual learning to assist the network training. This combination brings obvious performance improvement and avoids increasing additional parameters. Then, we design a novel Lightweight Multi-scale (LM) block as the core component of our network by utilizing the depthwise separable convolution. Next, we propose a novel Lightweight Multi-scale ConvLSTM (LM-ConvLSTM) module to integrate the deep features generated by the feedback mechanism. Last, we discuss the influence of different factors (i.e., loss function and network input/output) to tap the maximum potential of our lightweight network. Our LMHFNet could achieve competitive performance compared with the latest state-of-the-art methods (i.e., RCDNet and DRDNet), and bring a 28- or 46- times compression at the same time. The extensive experiments demonstrate the effectiveness and efficiency of our model in both quantitative assessments and visual quality.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Ruan W, Liu W, Bao Q, Chen J, Cheng, Y, Mei T (2019) Poinet: pose-guided ovonic insight network for multi-person pose tracking. In: Proceedings of the 27th ACM International Conference on Multimedia, pp 284–292

  2. Ruan W, Chen J, Yi W, Wang J, Liang C, Ruimin H, Jiang J (2018) Multi-correlation filters with triangle-structure constraints for object tracking. IEEE Trans Multimed 21(5):1122–1134

    Article  Google Scholar 

  3. Lin G, Milan A, Shen C, Reid I (2017) Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1925–1934

  4. Chen L-C, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2017) Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFS. IEEE Trans Pattern Anal Mach Intell, 40(4):834–848

  5. Redmon J, Farhadi A (2018) Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767

  6. Chen Y-L, Hsu C-T (2013) A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In: Proceedings of the IEEE International Conference on Computer Vision, pp 1968–1975

  7. Luo Y, Xu Y, Ji H (2015) Removing rain from a single image via discriminative sparse coding. In: Proceedings of the IEEE international conference on computer vision, pp 3397–3405

  8. Li Y, Tan RT, Guo X, Lu J, Brown MS (2016) Rain streak removal using layer priors. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2736–2744

  9. Wang H, Xie Q, Zhao Q, Meng D (2020) A model-driven deep neural network for single image rain removal. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3103–3112

  10. Xueyang F, Huang J, Ding X, Liao Y, Paisley J (2017) Clearing the skies: a deep network architecture for single-image rain removal. IEEE Trans Image Process 26(6):2944–2956

    Article  MathSciNet  Google Scholar 

  11. Fan Z, Wu H, Fu X, Huang Y, Ding X (2018) Residual-guide network for single image deraining. In Proceedings of the 26th ACM international conference on Multimedia, pp 1751–1759

  12. Wang T, Yang X, Xu K, Chen S, Zhang Q, Lau RWH (2019) Spatial attentive single-image deraining with a high quality real rain dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 12270–12279

  13. Wei Y, Zhang Z, Zhang H, Hong R, Wang M (2019) A coarse-to-fine multi-stream hybrid deraining network for single image deraining. In: 2019 IEEE International Conference on Data Mining (ICDM), pp 628–637. IEEE

  14. Yang W, Tan RT, Feng J, Guo Z, Yan S, Liu J (2019) Joint rain detection and removal from a single image with contextualized deep networks. IEEE Trans Pattern Anal Mach Intell 42(6):1377–1393

    Article  Google Scholar 

  15. Wei W, Meng D, Zhao Q, Xu Z, Wu Y (2019) Semi-supervised transfer learning for image rain removal. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 77–3886

  16. Zhang H, Patel VM (2018) Density-aware single image de-raining using a multi-stream dense network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 695–704

  17. Xueyang F, Liang B, Huang Y, Ding X, Paisley J (2019) Lightweight pyramid networks for image deraining. IEEE Trans Neural Netw Learn Syst 31(6):1794–1807

    Google Scholar 

  18. Deng S, Wei M, Wang J, Feng Y, Liang L, Xie H, Wang FL, Wang M (2020) Detail-recovery image deraining via context aggregation networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 14560–14569

  19. Ren D, Zuo W, Hu Q, Zhu P, Meng D (2019) Progressive image deraining networks: A better and simpler baseline. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3937–3946

  20. Wei Y, Zhang Z, Wang Y, Zhang H, Zhao M, Xu M, Wang M (2021) Semi-deraingan: A new semi-supervised single image deraining. In: 2021 IEEE International Conference on Multimedia and Expo (ICME), pp 1–6. IEEE

  21. Wei Y, Zhang Z, Wang Y, Mingliang X, Yang Y, Yan S, Wang M (2021) Deraincyclegan: Rain attentive cyclegan for single image deraining and rainmaking. IEEE Trans Image Process 30:4788–4801

    Article  Google Scholar 

  22. Zhao Z, Yanyan W, Haijun Z, Yi Y, Shuicheng Y, Wang M (2021) A comprehensive review and new perspectives, Data-driven single image deraining

  23. Guo Q, Sun J, Juefei-Xu F, Ma L, Xie X, Feng W, Liu Y (2020) Efficientderain: Learning pixel-wise dilation filtering for high-efficiency single-image deraining. arXiv preprint arXiv:2009.09238

  24. Yang W, Tan, RT Feng J, Liu J, Guo Z, Yan S (2017) Deep joint rain detection and removal from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1357–1366

  25. Li X, Wu J, Lin Z, Liu H, Zha H (2018) Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 254–269

  26. Newell A, Yang K, Deng J (2016) Stacked hourglass networks for human pose estimation. In: European conference on computer vision, pp 483–499. Springer

  27. Li W, Wang Z, Yin B, Peng Q, Du Y, Xiao T, Yu G, Lu H, Wei Y, Sun J (2019) Rethinking on multi-stage networks for human pose estimation. arXiv preprint arXiv:1901.00148

  28. Chen Y, Wang Z, Peng Y, Zhang Z, Yu G, Sun J (2018) Cascaded pyramid network for multi-person pose estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7103–7112

  29. Farha YA, Gall J (2019) Ms-tcn: Multi-stage temporal convolutional network for action segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3575–3584

  30. Li S-J, AbuFarha Y, Liu Y, Cheng M-M, Gall J (2020) Ms-tcn++: Multi-stage temporal convolutional network for action segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence

  31. Ghosh P, Yao Y, Davis L, Divakaran A (2020) Stacked spatio-temporal graph convolutional networks for action segmentation. InL Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp 576–585

  32. Cheng B, Chen L-C, Wei Y, Zhu Y, Huang Z, Xiong J, Huang TS, Hwu W-M, Shi H (2019) Spgnet: Semantic prediction guidance for scene parsing. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 5218–5228

  33. Li Z, Yang J, Liu Z, Yang X, Jeon G, Wu W (2019) Feedback network for image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3867–3876

  34. Han W, Chang S, Liu D, Yu M, Witbrock M, Huang TS (2018) Image super-resolution via dual-state recurrent networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition pp 1654–1663

  35. Lim B, Son S, Kim H, Nah S, Lee KM (2017) Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 136–144

  36. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  37. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H(2017) Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861

  38. Chollet F (2017) Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1251–1258

  39. Shi X, Chen Z, Wang H, Yeung DY, Wong WK, Woo WC (2015) Convolutional lSTM network: a machine learning approach for precipitation nowcasting. Adv Neural Inf Process Syst 2015:802–810

    Google Scholar 

  40. Xu J, Zhao W, Liu P, Tang X (2012) Removing rain and snow in a single image using guided filter. In: 2012 IEEE International Conference on Computer Science and Automation Engineering (CSAE), vol 2, p 304–307. IEEE

  41. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  42. Lin T-Y, Goyal P, Girshick R, He K, Dollár P (2017) Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision, pp 2980–2988

  43. Chen L-C, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2014) Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint arXiv:1412.7062

  44. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440

  45. Xu L, Zhang J, Cheng X, Zhang F, Wei X, Ren J (2021) Efficient deep image denoising via class specific convolution. arXiv preprint arXiv:2103.01624

  46. Zhang Yu, Wang X, Bi X, Tao D (2018) A light dual-task neural network for haze removal. IEEE Signal Process Lett 25(8):1231–1235

    Article  Google Scholar 

  47. Kong X, Zhao H, Qiao Y, Dong C (2021) Classsr: A general framework to accelerate super-resolution networks by data characteristic. arXiv preprint arXiv:2103.04039

  48. Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K (2016) Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv:1602.07360

  49. Han K, Wang Y, Tian Q, Guo J, Xu C, Xu C(2020) Ghostnet: More features from cheap operations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 1580–1589

  50. Zhang X, Zhou X, Lin M, Sun J (2018) Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6848–6856

  51. Howard A, Sandler M, Chu G, Chen L-C, Chen B, Tan M, Wang W, Zhu Y, Pang R, Vasudevan V et al. (2019) Searching for mobilenetv3. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 1314–1324

  52. Ma N, Zhang X, Zheng H-T, Sun J (2018) Shufflenet v2: Practical guidelines for efficient cnn architecture design. In: Proceedings of the European conference on computer vision (ECCV), pp 116–131

  53. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4510-4520

  54. Singh P, Verma VK, Rai P, Namboodiri VP (2019) Hetconv: Beyond homogeneous convolution kernels for deep cnns. Int J Comput Vis, pp 1–21

  55. Zhou S, Wang J, Zhang S, Liang Y, Gong Y (2016) Active contour model based on local and global intensity information for medical image segmentation. Neurocomputing 186:107–118

    Article  Google Scholar 

  56. Tan M, Le QV (2019) Mixconv: Mixed depthwise convolutional kernels. arXiv preprint arXiv:1907.09595

  57. He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision, pp 1026–1034

  58. He Zhang (2019) Sindagi V, Patel VM. Image de-raining using a conditional generative adversarial network. Ieee Transactions on Circuits and Systems for Video Technology, 30(11):3943–3956

  59. Zhang Yu, Lyu J, Bi X (2021) A dual-task dual-domain model for blind MRI reconstruction. Comput Med Imag Graph 89:101862

    Article  Google Scholar 

  60. Huynh-Thu Q, Ghanbari M (2008) Scope of validity of psnr in image/video quality assessment. Electr Lett 44(13):800–801

    Article  Google Scholar 

  61. Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

  62. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, Yoshua Bengio (2014) Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078

Download references

Acknowledgements

This study was supported by the National Natural Science Foundation of China (Grant No.51779050) and the National Social Science Foundation of China (Grant No.20&ZD279).

Author information

Author notes

  1. Chen Zheng

    Present address: The College of Information and Communication Engineering, Harbin Engineering University, Harbin, 150001, China

Authors and Affiliations

  1. The College of Information and Communication Engineering, Harbin Engineering University, Harbin, 150001, China

    Zhang Yu & Yue Jianyu

  2. The School of Information Engineering, Minzu University of China, Beijing, 100081, China

    Bi Xiaojun

  3. HeFei Innovation Research Institute, BeiHang University, Anhui Province, 230000, China

    Wang Haibo

Authors
  1. Chen Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bi Xiaojun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhang Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yue Jianyu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wang Haibo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bi Xiaojun.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, Z., Bi, X., Zhang, Y. et al. LightweightDeRain: learning a lightweight multi-scale high-order feedback network for single image de-raining. Neural Comput & Applic 34, 5431–5448 (2022). https://doi.org/10.1007/s00521-021-06700-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06700-5

Keywords

Search

Navigation