skip to main content
10.1145/3379299.3379305acmotherconferencesArticle/Chapter ViewAbstractPublication PagesdmipConference Proceedingsconference-collections
research-article

A Lightweight Channel-spatial Attention Network for Real-time Image De-raining

Published: 20 March 2020 Publication History

Abstract

Image de-raining aims to eliminate rain streaks captured by outdoor equipment such as video surveillance, remote sensor and automatic pilot. Recently, a de-raining method called non-locally enhanced encoder-decoder network (NLEDN) has achieved reliability performance. Nevertheless, it is very time consuming (2.2571s per image) and takes up memory so that it cannot be applied to mobile devices to process image in real-time. To solve this problem, we design a lightweight channel-spatial attention network that is 55 times faster (41ms per image) and memory saving. The most advanced performances are achieved in most de-raining data sets. More specifically, we design a channel-spatial attention dense block (CSADB). The channel attention operation will be carried out together with the spatial attention. Our experiments demonstrate that the network can learn more effective features by this way. In order to make our proposed method more lightweight, the depthwise convolutions are adapted in each block to reduce parameters. We conduct experiments on four public synthetic datasets to demonstrate the effectiveness of our proposed method, which achieve excellent performance. And the real-world de-raining results are also tacked into comparison. Moreover, an additional experiment demonstrates that our method also works well on face hallucination task. The relevant code and trained models will be available in GitHub soon.

References

[1]
Yilei Chen and Chiou Ting Hsu. 2013. A Generalized Low-Rank Appearance Model for Spatio-temporally Correlated Rain Streaks. In IEEE International Conference on Computer Vision. 1968--1975.
[2]
Li-Wei Kang, Chia-Wen Lin and Yu-Hsiang Fu. 2012. Automatic single-image-based rain streaks removal via image decomposition. IEEE Transactions on Image Processing, vol. 21. 1742--1755.
[3]
Yu Li, Robby T. Tan, Xiaojie Guo, Jiangbo Lu and Michael S. Brown. 2016. Rain streak removal using layer priors. IEEE Conference on Computer Vision and Pattern Recognition. 2736--2744.
[4]
Wenhan Yang, Robby T. Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. 2017. Deep Joint Rain Detection and Removal from a Single Image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
[5]
He Zhang and Vishal M. Patel. 2018. Density-aware Single Image De-raining using a Multi-stream Dense Network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
[6]
Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley. 2017. Removing Rain from Single Images via a Deep Detail Network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1715--1723.
[7]
Guanbin Li, Xiang He, Wei Zhang, Huiyou Chang, Le Dong and Liang Lin. 2018. Non-locally Enhanced Encoder-Decoder Network for Single Image De-raining. In ACM International Conference on Multimedia.
[8]
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. 2018. Non-local Neural Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
[9]
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto and Hartwig Adam. 2017. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv:Computer Vision and Pattern Recognition.
[10]
Jie Hu, Li Shen, Samuel Albanie, Gang Sun and Enhua Wu. 2017. Squeeze-and-Excitation Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence.
[11]
Quan Huynh-Thu and Mohammed Ghanbari. 2008. Scope of validity of PSNR in image/video quality assessment. Electronices Letters 44, 13 (2008), 800--8001.
[12]
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4(2004), 600--612.
[13]
Tai-Xiang Jiang, Ting-Zhu Huang, Xi-Le Zhao, Liang-Jian Deng, Yao Wang. 2018. FastDeRain: A Novel Video Rain Streak Removal Method Using Directional Gradient Priors. IEEE Transactions on Image Processing. 1--1.
[14]
Michael Blot, Matthieu Cord and Nicolas Thome. 2016. Maxmin convolutional neural networks for image classification. In IEEE International Conference on Image Processing (ICIP).
[15]
Bing Xu, Naiyan Wang, Tianqi Chen and Mu Li. 2015. Empirical Evaluation of Rectified Activations in Convolutional Network. Computer Science.
[16]
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. 2001. A database of human segmented natural images and its application to evaluating semantation algorithms and meaduring ecological statistics. In Proceedings of the IEEE Internatioanl Conference on Computer Vision, vol.2. 416--423.
[17]
Chih-Yuan Yang, Sifei Liu and Ming-Hsuan Yang. 2013. Structured Face Hallucination. IEEE Conference on Computer Vision and Pattern Recognition. 1099--1106.
[18]
Erjin Zhou, Haoqiang Fan, Zhimin Cao, Yuning Jiang and Qi Yin. 2015. Learning face hallucination in the wild. In AAAI Conference on Artificial Intelligence. 3871--3877.
[19]
Xiang Ma, Junping Zhang and Chun Qi. 2010. Hallucinating face by position-patch. Pattern Recognition. 2224--2236.
[20]
Chao Dong, Chen Change Loy, Kaiming He and Xiaoou Tang. 2014. Learning a Deep Convolutional Network for Image Super-Resolution. In European Conference on Computer Vision. 184--199.
[21]
Jiwon Kim, Jung Kwon Lee and Kyoung Mu Lee. 2016. Accurate Image Super-Resolution Using Very Deep Convolutional Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1646--1654.
[22]
Tuzel Oncel, Taguchi Yuichi and Hershey John R. 2016. Global-Local Face Upsampling Network. arXiv:Computer Vision and Pattern Recognition.
[23]
Qingxing Cao, Liang Lin, Yukai Shi, Xiaodan Liang and Guanbin Li. 2017. Attention-Aware Face Hallucination via Deep Reinforcement Learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1656--1664.
[24]
Huang G B, Mattar M and Berg T. 2008. Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. Workshop on faces in'Real-Life'Images: detection, alignment, and recognition.

Cited By

View all
  • (2024)An all-in-one lightweight image restoration network based on polarized attention mechanism and efficient feature extractionInternational Conference on Mechatronic Engineering and Artificial Intelligence (MEAI 2023)10.1117/12.3025634(159)Online publication date: 28-Feb-2024
  • (2024)Lightweight Image Deraining Network Based on Dilated Depthwise Separable Convolution and Enhanced Channel Attention2024 2nd International Conference on Signal Processing and Intelligent Computing (SPIC)10.1109/SPIC62469.2024.10691394(1016-1021)Online publication date: 20-Sep-2024
  • (2021)RoDeRain: Rotational Video Derain via Nonconvex and Nonsmooth OptimizationMobile Networks and Applications10.1007/s11036-020-01721-126:1(57-66)Online publication date: 19-Mar-2021

Index Terms

  1. A Lightweight Channel-spatial Attention Network for Real-time Image De-raining

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Other conferences
      DMIP '19: Proceedings of the 2019 2nd International Conference on Digital Medicine and Image Processing
      November 2019
      59 pages
      ISBN:9781450376983
      DOI:10.1145/3379299
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      In-Cooperation

      • East China Normal University
      • University of Tsukuba: University of Tsukuba

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 20 March 2020

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. Image de-raining
      2. Mobile net
      3. Non-local mean calculation
      4. Squeeze-and-excitation attention

      Qualifiers

      • Research-article
      • Research
      • Refereed limited

      Conference

      DMIP '19

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)6
      • Downloads (Last 6 weeks)0
      Reflects downloads up to 14 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)An all-in-one lightweight image restoration network based on polarized attention mechanism and efficient feature extractionInternational Conference on Mechatronic Engineering and Artificial Intelligence (MEAI 2023)10.1117/12.3025634(159)Online publication date: 28-Feb-2024
      • (2024)Lightweight Image Deraining Network Based on Dilated Depthwise Separable Convolution and Enhanced Channel Attention2024 2nd International Conference on Signal Processing and Intelligent Computing (SPIC)10.1109/SPIC62469.2024.10691394(1016-1021)Online publication date: 20-Sep-2024
      • (2021)RoDeRain: Rotational Video Derain via Nonconvex and Nonsmooth OptimizationMobile Networks and Applications10.1007/s11036-020-01721-126:1(57-66)Online publication date: 19-Mar-2021

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media