Skip to main content
SpringerLink
Log in

MSL-MNN: image deraining based on multi-scale lightweight memristive neural network

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Images and videos captured in rainy conditions show visual degradation, resulting in poor performance for outdoor computer vision systems. Traditional deraining methods fail to work in moderate and torrential rain removal, while the data-driven learning-based methods are hindered by removing rain streaks in an efficient manner, and lack consideration applied to edge devices with limited resources. So, this paper proposes a software-hardware co-designed image deraining system named MSL-MNN based on the multi-scale lightweight memristive neural network, which combines deep learning and the nanoscale device memristor. Specifically, a novel feature extraction block ReFiSE based on the lightweight residual mode and squeeze-and-excitation operation is designed. Benefiting from this, MSL-MNN is endowed with a lightweight network framework. Then, the multi-scale skip connection design is introduced to encourage the decoder to use low-level semantic information. Besides, a hardware implementation scheme of MSL-MNN is presented by leveraging the memristor crossbar array, which organically combines the deep learning algorithm with the neuro-inspired computing chip to speed the real-time processing of image data. Finally, a series of experimental results prove the effectiveness and superiority of MSL-MNN in rain removal and the recovery of image details. This study is expected to provide an end-to-end algorithm design and hardware acceleration solution for intelligent image processing besides image deraining.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Garg K, Nayar SK (2004) Detection and removal of rain from videos. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. pp 528–535

  2. Varun S, Vijayan AK (2015) Utilizing local phase information to remove rain from video. Int J Comput Vision 112:71–89. https://doi.org/10.1007/s11263-014-0759-8

    Article  Google Scholar 

  3. Chen Y, Hsu C (2013) A Generalized Low-Rank Appearance Model for Spatio-temporally Correlated Rain Streaks. In: 2013 IEEE International Conference on Computer Vision. pp 1968–1975

  4. Luo Y, Xu Y, Ji H (2015) Removing Rain from a Single Image via Discriminative Sparse Coding. In: 2015 IEEE International Conference on Computer Vision. pp 3397–3405

  5. Li Y, Tan RT, Guo X, Lu J, Brown MS (2016) Rain Streak Removal Using Layer Priors. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition. pp 2736–2744

  6. Eigen D, Krishnan D, Fergus R (2013) Restoring an Image Taken through a Window Covered with Dirt or Rain. In: 2013 IEEE International Conference on Computer Vision. pp 633–640

  7. Li X, Wu J, Lin Z, Liu H, Zha H (2018) Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining. In: Proceedings of the European Conference on Computer Vision. pp 262–277

  8. Deng S, Wei M, Wang J, Feng Y, Liang L, Xie H, Wang FL, Wang M (2020) Detail-recovery Image Deraining via Context Aggregation Networks. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp 14548–14557

  9. Wang H, Xie Q, Zhao Q, Meng D (2020) A Model-Driven Deep Neural Network for Single Image Rain Removal. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp 3100–3109

  10. Wang Z, Li J, Song G (2020) DTDN: Dual-task De-raining Network. In: Proceedings of the 27th ACM International Conference on Multimedia. pp 1833–1841

  11. Guo Q, Sun J, Juefei-Xu F, Ma L, Xie X, Feng W, Liu Y (2021) EfficientDeRain: Learning Pixel-wise Dilation Filtering for High-Efficiency Single-Image Deraining. In: Proceedings of the AAAI Conference on Artificial Intelligence

  12. Chua L (1971) Memristor-the missing circuit element. IEEE Trans Circuit Theory 18:507–519. https://doi.org/10.1109/TCT.1971.1083337

    Article  Google Scholar 

  13. Williams RS (2008) How we found the missing memristor. IEEE Spectr 45:28–35. https://doi.org/10.1109/MSPEC.2008.4687366

    Article  Google Scholar 

  14. Strukov DB, Snider GS, Stewart DR, Williams RS (2008) The missing memristor found. Nature 453:80–83. https://doi.org/10.1038/nature06932

    Article  Google Scholar 

  15. Hu X, Feng G, Duan S, Liu L (2017) A memristive multilayer cellular neural network with applications to image processing. IEEE Trans Neural Netw Learning Syst 28:1889–1901. https://doi.org/10.1109/TNNLS.2016.2552640

    Article  MathSciNet  Google Scholar 

  16. Ji X, Hu X, Zhou Y, Dong Z, Duan S (2019) Adaptive sparse coding based on memristive neural network with applications. Cogn Neurodyn 13:475–488. https://doi.org/10.1007/s11571-019-09537-w

    Article  Google Scholar 

  17. Zheng Y, Hu X, Zhou Y, Luo L, Duan S (2020) Biomimetic adaptive memristive cellular neural network for image enhancement. Sci Sinica Inf 50:1850–1866

    Article  Google Scholar 

  18. Cai F, Correll JM, Lee SH, Lim Y, Bothra V, Zhang Z, Flynn MP, Lu WD (2019) A fully integrated reprogrammable memristor–CMOS system for efficient multiply–accumulate operations. Nat Electron 2:290–299. https://doi.org/10.1038/s41928-019-0270-x

    Article  Google Scholar 

  19. Yao P, Wu H, Gao B, Tang J, Zhang Q, Zhang W, Yang JJ, Qian H (2020) Fully hardware-implemented memristor convolutional neural network. Nature 577:641–646. https://doi.org/10.1038/s41586-020-1942-4

    Article  Google Scholar 

  20. Liu Z, Tang J, Gao B, Yao P, Li X, Liu D, Zhou Y, Qian H, Hong B, Wu H (2020) Neural signal analysis with memristor arrays towards high-efficiency brain–machine interfaces. Nat Commun 11:4234. https://doi.org/10.1038/s41467-020-18105-4

    Article  Google Scholar 

  21. Ascoli A, Corinto F, Senger V, Tetzlaff R (2013) Memristor Model Comparison. IEEE Circuits Syst Mag 13:89–105. https://doi.org/10.1109/MCAS.2013.2256272

    Article  Google Scholar 

  22. Hu X, Wang W, Sun B, Wang Y, Li J, Zhou G (2021) Refining the negative differential resistance effect in a TiOx-based memristor. J Phys Chem Lett. https://doi.org/10.1021/acs.jpclett.1c01420

    Article  Google Scholar 

  23. Mladenov V (2021) A unified and open LTSPICE memristor model library. Electronics 10:1594. https://doi.org/10.3390/electronics10131594

    Article  Google Scholar 

  24. Kvatinsky S, Ramadan M, Friedman EG, Kolodny A (2015) VTEAM: a general model for voltage-controlled memristors. IEEE Trans Circuits Syst II Express Briefs 62:786–790. https://doi.org/10.1109/TCSII.2015.2433536

    Article  Google Scholar 

  25. Sheu S-S, Chiang P-C, Lin W-P, Lee H-Y, Chen P-S, Chen Y-S, Wu T-Y, Chen FT, Su K-L, Kao M-J, Cheng K-H, Tsai M-J (2009) A 5ns fast write multi-level non-volatile 1 K bits RRAM memory with advance write scheme. In: 2009 Symposium on VLSI Circuits. pp 82–83

  26. Iandola F, Han S, Moskewicz M, Ashraf K, Dally W, Keutzer K (2016) SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size

  27. Hu J, Shen L, Albanie S, Sun G, Wu E (2020) Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell 42:2011–2023. https://doi.org/10.1109/TPAMI.2019.2913372

    Article  Google Scholar 

  28. Liang Z, Feng Y, Guo Y, Liu H, Qiao L, Chen W, Zhou L, Zhang J (2018) Learning for Disparity Estimation Through Feature Constancy. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp 2811–2820

  29. Zhou Z, Siddiquee MMR, Tajbakhsh N, Liang J (2020) UNet++: redesigning skip connections to exploit multiscale features in image segmentation. IEEE Trans Med Imaging 39:1856–1867. https://doi.org/10.1109/TMI.2019.2959609

    Article  Google Scholar 

  30. Chen J, Wu Y, Yang Y, Wen S, Shi K, Bermak A, Huang T (2021) An efficient memristor-based circuit implementation of squeeze-and-excitation fully convolutional neural networks. IEEE Trans Neural Netw Learning Syst. https://doi.org/10.1109/TNNLS.2020.3044047

    Article  Google Scholar 

  31. Yakopcic C, Alom MZ, Taha TM (2016) Memristor crossbar deep network implementation based on a Convolutional neural network. In: 2016 International Joint Conference on Neural Networks (IJCNN). pp 963–970

  32. Yang W, Tan RT, Feng J, Liu J, Guo Z, Yan S (2017) Deep Joint Rain Detection and Removal from a Single Image. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition. pp 1685–1694

  33. Zhang H, Patel VM (2018) Density-Aware Single Image De-raining Using a Multi-stream Dense Network. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp 695–704

  34. Fu X, Huang J, Zeng D, Huang Y, Ding X, Paisley J (2017) Removing Rain from Single Images via a Deep Detail Network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition. pp 1715–1723

  35. Fu X, Liang B, Huang Y, Ding X, Paisley J (2020) Lightweight pyramid networks for image deraining. IEEE Trans Neural Netw Learning Syst 31:1794–1807. https://doi.org/10.1109/TNNLS.2019.2926481

    Article  Google Scholar 

  36. Ren D, Zuo W, Hu Q, Zhu P, Meng D (2019) Progressive Image Deraining Networks: A Better and Simpler Baseline. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp 3932–3941

  37. Yang W, Tan R, Wang S, Fang Y, Liu J (2019) Single image deraining: from model-based to data-driven and beyond. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2020.2995190

    Article  Google Scholar 

  38. Huynh-Thu Q, Ghanbari M (2008) Scope of validity of PSNR in image/video quality assessment. Electron Lett 44:800–801. https://doi.org/10.1049/el:20080522

    Article  Google Scholar 

  39. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13:600–612. https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (Grant Nos. 61976246, U20A20227), National Key R&D Program of China (Grant Nos. 2018YFB1306600, 2018YFB1306604), Natural Science Foundation of Chongqing (Grant No. cstc2020jcyj-msxmX0385).

Author information

Authors and Affiliations

  1. College of Artificial Intelligence, Southwest University, Chongqing, 400715, China

    Lin Zhang, Xiaofang Hu, Fan Sun & Shukai Duan

  2. College of Electronics and Information Engineering, Southwest University, Chongqing, 400715, China

    Yue Zhou

  3. Brain-Inspired Computing & Intelligent Control of Chongqing Key Lab, Chongqing, 400715, China

    Yue Zhou, Xiaofang Hu & Shukai Duan

Authors
  1. Lin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yue Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaofang Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fan Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shukai Duan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaofang Hu.

Ethics declarations

Conflict of interest

The authors declare that they have no potential conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, L., Zhou, Y., Hu, X. et al. MSL-MNN: image deraining based on multi-scale lightweight memristive neural network. Neural Comput & Applic 34, 7299–7309 (2022). https://doi.org/10.1007/s00521-021-06835-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06835-5

Keywords

Search

Navigation