research-article

Single Image Deraining via Recurrent Hierarchy Enhancement Network

Authors:

Hong LuAuthors Info & Claims

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

Pages 1814 - 1822

https://doi.org/10.1145/3343031.3351149

Published: 15 October 2019 Publication History

Abstract

Single image deraining is an important problem in many computer vision tasks since rain streaks can severely hamper and degrade the visibility of images. In this paper, we propose a novel network named Recurrent Hierarchy Enhancement Network (ReHEN) to remove rain streaks from rainy images stage by stage. Unlike previous deep convolutional network methods, we adopt a Hierarchy Enhancement Unit (HEU) to fully extract local hierarchical features and generate effective features. Then a Recurrent Enhancement Unit (REU) is added to keep the useful information from HEU and benefit the rain removal in the later stages. To focus on different scales, shapes, and densities of rain streaks adaptively, Squeeze-and-Excitation (SE) block is applied in both HEU and REU to assign different scale factors to high-level features. Experiments on five synthetic datasets and a real-world rainy image set show that the proposed method outperforms the state-of-the-art methods considerably. The source code is available at https://github.com/nnUyi/ReHEN.

References

[1]

Mart'in Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et almbox. 2016. Tensorflow: a system for large-scale machine learning. In OSDI, Vol. 16. 265--283.

Digital Library

[2]

Peter C Barnum, Srinivasa Narasimhan, and Takeo Kanade. 2010. Analysis of rain and snow in frequency space. International Journal of Computer Vision, Vol. 86, 2--3 (2010), 256.

Digital Library

[3]

Jie Chen, Cheen-Hau Tan, Junhui Hou, Lap-Pui Chau, and He Li. 2018. Robust Video Content Alignment and Compensation for Rain Removal in a CNN Framework. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 6286--6295.

[4]

Yi-Lei Chen and Chiou-Ting Hsu. 2013. A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In Proceedings of the IEEE International Conference on Computer Vision. 1968--1975.

Digital Library

[5]

Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014).

[6]

Liang-Jian Deng, Ting-Zhu Huang, Xi-Le Zhao, and Tai-Xiang Jiang. 2018. A directional global sparse model for single image rain removal. Applied Mathematical Modelling, Vol. 59 (2018), 662--679.

[7]

David Eigen, Dilip Krishnan, and Rob Fergus. 2013. Restoring an image taken through a window covered with dirt or rain. In Proceedings of the IEEE International Conference on Computer Vision. 633--640.

Digital Library

[8]

Zhiwen Fan, Huafeng Wu, Xueyang Fu, Yue Huang, and Xinghao Ding. 2018. Residual-Guide Network for Single Image Deraining. In Proceedings of the 26th ACM International Conference on Multimedia. 1751--1759.

Digital Library

[9]

Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley. 2017. Removing rain from single images via a deep detail network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1715--1723.

[10]

Kshitiz Garg and Shree K Nayar. 2004. Detection and removal of rain from videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 528--535.

[11]

Shuhang Gu, Deyu Meng, Wangmeng Zuo, and Lei Zhang. 2017. Joint convolutional analysis and synthesis sparse representation for single image layer separation. In Proceedings of the IEEE International Conference on Computer Vision. 1717--1725.

[12]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770--778.

[13]

Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-Excitation Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7132--7141.

[14]

Quan Huynh-Thu and Mohammed Ghanbari. 2008. Scope of validity of PSNR in image/video quality assessment. Electronics letters, Vol. 44, 13 (2008), 800--801.

[15]

Tai-Xiang Jiang, Ting-Zhu Huang, Xi-Le Zhao, Liang-Jian Deng, and Yao Wang. 2018. FastDeRain: A Novel Video Rain Streak Removal Method Using Directional Gradient Priors. arXiv preprint arXiv:1803.07487 (2018).

[16]

Li-Wei Kang, Chia-Wen Lin, and Yu-Hsiang Fu. 2012. Automatic single-image-based rain streaks removal via image decomposition. IEEE Transactions on Image Processing, Vol. 21, 4 (2012), 1742--1755.

Digital Library

[17]

Jin-Hwan Kim, Jae-Young Sim, and Chang-Su Kim. 2015. Video deraining and desnowing using temporal correlation and low-rank matrix completion. IEEE Transactions on Image Processing, Vol. 24, 9 (2015), 2658--2670.

Digital Library

[18]

Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et almbox. 2017. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 105--114.

[19]

Minghan Li, Qi Xie, Qian Zhao, Wei Wei, Shuhang Gu, Jing Tao, and Deyu Meng. 2018b. Video Rain Streak Removal By Multiscale Convolutional Sparse Coding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 6644--6653.

[20]

Ruoteng Li, Loong-Fah Cheong, and Robby T Tan. 2017. Single Image Deraining using Scale-Aware Multi-Stage Recurrent Network. arXiv preprint arXiv:1712.06830 (2017).

[21]

Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, and Hongbin Zha. 2018a. Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining. In The European Conference on Computer Vision (ECCV). 262--277.

[22]

Yu Li, Robby T Tan, Xiaojie Guo, Jiangbo Lu, and Michael S Brown. 2016. Rain streak removal using layer priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2736--2744.

[23]

Jiaying Liu, Wenhan Yang, Shuai Yang, and Zongming Guo. 2018. Erase or Fill? Deep Joint Recurrent Rain Removal and Reconstruction in Videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 3233--3242.

[24]

Yu Luo, Yong Xu, and Hui Ji. 2015. Removing rain from a single image via discriminative sparse coding. In Proceedings of the IEEE International Conference on Computer Vision . 3397--3405.

Digital Library

[25]

Danilo P Mandic and Jonathon Chambers. 2001. Recurrent neural networks for prediction: learning algorithms, architectures and stability .John Wiley & Sons, Inc.

[26]

Jinshan Pan, Sifei Liu, Deqing Sun, Jiawei Zhang, Yang Liu, Jimmy Ren, Zechao Li, Jinhui Tang, Huchuan Lu, Yu-Wing Tai, et almbox. 2018. Learning Dual Convolutional Neural Networks for Low-Level Vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 3070--3079.

[27]

Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. 2016. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2536--2544.

[28]

Weihong Ren, Jiandong Tian, Zhi Han, Antoni Chan, and Yandong Tang. 2017. Video desnowing and deraining based on matrix decomposition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 4210--4219.

[29]

Varun Santhaseelan and Vijayan K Asari. 2015. Utilizing local phase information to remove rain from video. International Journal of Computer Vision, Vol. 112, 1 (2015), 71--89.

Digital Library

[30]

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, Vol. 13, 4 (2004), 600--612.

Digital Library

[31]

Wei Wei, Lixuan Yi, Qi Xie, Qian Zhao, Deyu Meng, and Zongben Xu. 2017. Should we encode rain streaks in video as deterministic or stochastic. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 2516--2525.

[32]

Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. 2016. Joint rain detection and removal via iterative region dependent multi-task learning. CoRR, abs/1609.07769, Vol. 2 (2016), 3.

[33]

Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. 2017. Deep joint rain detection and removal from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1357--1366.

[34]

Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329 (2014).

[35]

He Zhang and Vishal M. Patel. 2018. Density-Aware Single Image De-Raining Using a Multi-Stream Dense Network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 695--704.

[36]

He Zhang, Vishwanath Sindagi, and Vishal M Patel. 2017. Image de-raining using a conditional generative adversarial network. arXiv preprint arXiv:1701.05957 (2017).

[37]

Lei Zhu, Chi-Wing Fu, Dani Lischinski, and Pheng-Ann Heng. 2017. Joint bilayer optimization for single-image rain streak removal. In Proceedings of the IEEE International Conference on Computer Vision. 2526--2534.

Cited By

Gumma SChintakindi B(2025)FoNet: Focused Network for Single Image DerainingCircuits, Systems, and Signal Processing10.1007/s00034-025-03009-9Online publication date: 7-Feb-2025
https://doi.org/10.1007/s00034-025-03009-9
Ran WMa PHe ZLu HCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Rainmer: Learning Multi-view Representations for Comprehensive Image Deraining and BeyondProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681342(2766-2775)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681342
Zhang WXiao HXie DZhou YDuan SHu X(2024)A Global Self-Attention Memristive Neural Network for Image RestorationIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2024.33694478:3(2613-2624)Online publication date: Jun-2024
https://doi.org/10.1109/TETCI.2024.3369447
Show More Cited By

Index Terms

Single Image Deraining via Recurrent Hierarchy Enhancement Network
1. Computing methodologies
  1. Computer graphics
    1. Image manipulation
      1. Image processing

Recommendations

Recurrent wavelet structure-preserving residual network for single image deraining
Highlights
- A novel Recurrent Wavelet Structure-preserving Residual Network (RWSRNet) is proposed to pay attention to both the low-frequency and high-frequency parts of ...
Abstract
The combination of deep learning and image prior has been widely used in single image deraining since 2017. Recent studies have demonstrated an excellent deraining effect on the high-frequency part of rain images, but less attention ...
Single image deraining via a recurrent multi-attention enhancement network
Abstract
Single image deraining has attracted considerable attention because rain streaks can severely degrade image quality and affect performance in computer vision tasks. While recurrent networks have achieved promising results, they still have ...
Highlights
- Propose multi-attention enhancement deraining network to recover rainy images.
- Integrate multi-attentions to enhance effective features.
- The proposed network achieves superior performance on public rain datasets.
A Low-Rank Appearance Recurrent Network for Single Image Rain Removal
Advanced Intelligent Computing Technology and Applications
Abstract
Deep learning (DL) methods have achieved state-of-the-art performance for removing single image rain streaks. However, most existing DL models directly learn the residual between rainy images and clear images, ignoring the physical characteristics ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

October 2019

2794 pages

ISBN:9781450368896

DOI:10.1145/3343031

General Chairs:
Laurent Amsaleg
CNRS-IRISA, France
,
Benoit Huet
EURECOM, France
,
Martha Larson
Radboud University and TU Delft (Netherlands)
,
Program Chairs:
Guillaume Gravier
CNRS-IRISA, France
,
Hayley Hung
Delft University of Technology Netherlands
,
Chong-Wah Ngo
City University of Hong Kong Hong Kong
,
Wei Tsang Ooi
National University of Singapore Singapore

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM '19

Sponsor:

SIGMM

MM '19: The 27th ACM International Conference on Multimedia

October 21 - 25, 2019

Nice, France

Acceptance Rates

MM '19 Paper Acceptance Rate 252 of 936 submissions, 27%;

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

51
Total Citations
View Citations
611
Total Downloads

Downloads (Last 12 months)19
Downloads (Last 6 weeks)0

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gumma SChintakindi B(2025)FoNet: Focused Network for Single Image DerainingCircuits, Systems, and Signal Processing10.1007/s00034-025-03009-9Online publication date: 7-Feb-2025
https://doi.org/10.1007/s00034-025-03009-9
Ran WMa PHe ZLu HCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Rainmer: Learning Multi-view Representations for Comprehensive Image Deraining and BeyondProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681342(2766-2775)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681342
Zhang WXiao HXie DZhou YDuan SHu X(2024)A Global Self-Attention Memristive Neural Network for Image RestorationIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2024.33694478:3(2613-2624)Online publication date: Jun-2024
https://doi.org/10.1109/TETCI.2024.3369447
Xie DXiao HHu XZhou YZhou GDuan S(2024)Neuromorphic Vision Restoration Network for Advanced Driver Assistance SystemIEEE Transactions on Consumer Electronics10.1109/TCE.2024.336772870:1(3658-3668)Online publication date: Feb-2024
https://doi.org/10.1109/TCE.2024.3367728
Babar KYaseen MAl-Shamayleh AImran MAl-Ghushami AAkhunzada A(2024)LPN-IDD: A Lightweight Pyramid Network for Image Deraining and DetectionIEEE Access10.1109/ACCESS.2024.337148412(37103-37119)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3371484
Shen YZhang YLi WQin CHuang Y(2024)Single image deraining via wide rectangular regional blocks and dual attention complementary enhancement networkScientific Reports10.1038/s41598-024-70329-214:1Online publication date: 21-Aug-2024
https://doi.org/10.1038/s41598-024-70329-2
Ali ASarkar RChaudhuri S(2024)Wavelet-based Auto-Encoder for simultaneous haze and rain removal from imagesPattern Recognition10.1016/j.patcog.2024.110370150:COnline publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1016/j.patcog.2024.110370
Wen YGao TZhang JZhang KChen T(2024)From heavy rain removal to detail restoration: A faster and better networkPattern Recognition10.1016/j.patcog.2023.110205148(110205)Online publication date: Apr-2024
https://doi.org/10.1016/j.patcog.2023.110205
Xie DXiao HZhou YDuan SHu X(2024)MWA-MNNExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.122427240:COnline publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.122427
Xu YHan CLv SWang ZWang M(2024)Internal and external transmission encoder–decoder network for single-image derainingThe Visual Computer10.1007/s00371-024-03261-140:12(8653-8663)Online publication date: 20-Mar-2024
https://doi.org/10.1007/s00371-024-03261-1
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten