skip to main content
10.1145/3343031.3351149acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Single Image Deraining via Recurrent Hierarchy Enhancement Network

Published: 15 October 2019 Publication History

Abstract

Single image deraining is an important problem in many computer vision tasks since rain streaks can severely hamper and degrade the visibility of images. In this paper, we propose a novel network named Recurrent Hierarchy Enhancement Network (ReHEN) to remove rain streaks from rainy images stage by stage. Unlike previous deep convolutional network methods, we adopt a Hierarchy Enhancement Unit (HEU) to fully extract local hierarchical features and generate effective features. Then a Recurrent Enhancement Unit (REU) is added to keep the useful information from HEU and benefit the rain removal in the later stages. To focus on different scales, shapes, and densities of rain streaks adaptively, Squeeze-and-Excitation (SE) block is applied in both HEU and REU to assign different scale factors to high-level features. Experiments on five synthetic datasets and a real-world rainy image set show that the proposed method outperforms the state-of-the-art methods considerably. The source code is available at https://github.com/nnUyi/ReHEN.

References

[1]
Mart'in Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et almbox. 2016. Tensorflow: a system for large-scale machine learning. In OSDI, Vol. 16. 265--283.
[2]
Peter C Barnum, Srinivasa Narasimhan, and Takeo Kanade. 2010. Analysis of rain and snow in frequency space. International Journal of Computer Vision, Vol. 86, 2--3 (2010), 256.
[3]
Jie Chen, Cheen-Hau Tan, Junhui Hou, Lap-Pui Chau, and He Li. 2018. Robust Video Content Alignment and Compensation for Rain Removal in a CNN Framework. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 6286--6295.
[4]
Yi-Lei Chen and Chiou-Ting Hsu. 2013. A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In Proceedings of the IEEE International Conference on Computer Vision. 1968--1975.
[5]
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014).
[6]
Liang-Jian Deng, Ting-Zhu Huang, Xi-Le Zhao, and Tai-Xiang Jiang. 2018. A directional global sparse model for single image rain removal. Applied Mathematical Modelling, Vol. 59 (2018), 662--679.
[7]
David Eigen, Dilip Krishnan, and Rob Fergus. 2013. Restoring an image taken through a window covered with dirt or rain. In Proceedings of the IEEE International Conference on Computer Vision. 633--640.
[8]
Zhiwen Fan, Huafeng Wu, Xueyang Fu, Yue Huang, and Xinghao Ding. 2018. Residual-Guide Network for Single Image Deraining. In Proceedings of the 26th ACM International Conference on Multimedia. 1751--1759.
[9]
Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley. 2017. Removing rain from single images via a deep detail network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1715--1723.
[10]
Kshitiz Garg and Shree K Nayar. 2004. Detection and removal of rain from videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 528--535.
[11]
Shuhang Gu, Deyu Meng, Wangmeng Zuo, and Lei Zhang. 2017. Joint convolutional analysis and synthesis sparse representation for single image layer separation. In Proceedings of the IEEE International Conference on Computer Vision. 1717--1725.
[12]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770--778.
[13]
Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-Excitation Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7132--7141.
[14]
Quan Huynh-Thu and Mohammed Ghanbari. 2008. Scope of validity of PSNR in image/video quality assessment. Electronics letters, Vol. 44, 13 (2008), 800--801.
[15]
Tai-Xiang Jiang, Ting-Zhu Huang, Xi-Le Zhao, Liang-Jian Deng, and Yao Wang. 2018. FastDeRain: A Novel Video Rain Streak Removal Method Using Directional Gradient Priors. arXiv preprint arXiv:1803.07487 (2018).
[16]
Li-Wei Kang, Chia-Wen Lin, and Yu-Hsiang Fu. 2012. Automatic single-image-based rain streaks removal via image decomposition. IEEE Transactions on Image Processing, Vol. 21, 4 (2012), 1742--1755.
[17]
Jin-Hwan Kim, Jae-Young Sim, and Chang-Su Kim. 2015. Video deraining and desnowing using temporal correlation and low-rank matrix completion. IEEE Transactions on Image Processing, Vol. 24, 9 (2015), 2658--2670.
[18]
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et almbox. 2017. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 105--114.
[19]
Minghan Li, Qi Xie, Qian Zhao, Wei Wei, Shuhang Gu, Jing Tao, and Deyu Meng. 2018b. Video Rain Streak Removal By Multiscale Convolutional Sparse Coding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 6644--6653.
[20]
Ruoteng Li, Loong-Fah Cheong, and Robby T Tan. 2017. Single Image Deraining using Scale-Aware Multi-Stage Recurrent Network. arXiv preprint arXiv:1712.06830 (2017).
[21]
Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, and Hongbin Zha. 2018a. Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining. In The European Conference on Computer Vision (ECCV). 262--277.
[22]
Yu Li, Robby T Tan, Xiaojie Guo, Jiangbo Lu, and Michael S Brown. 2016. Rain streak removal using layer priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2736--2744.
[23]
Jiaying Liu, Wenhan Yang, Shuai Yang, and Zongming Guo. 2018. Erase or Fill? Deep Joint Recurrent Rain Removal and Reconstruction in Videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 3233--3242.
[24]
Yu Luo, Yong Xu, and Hui Ji. 2015. Removing rain from a single image via discriminative sparse coding. In Proceedings of the IEEE International Conference on Computer Vision . 3397--3405.
[25]
Danilo P Mandic and Jonathon Chambers. 2001. Recurrent neural networks for prediction: learning algorithms, architectures and stability .John Wiley & Sons, Inc.
[26]
Jinshan Pan, Sifei Liu, Deqing Sun, Jiawei Zhang, Yang Liu, Jimmy Ren, Zechao Li, Jinhui Tang, Huchuan Lu, Yu-Wing Tai, et almbox. 2018. Learning Dual Convolutional Neural Networks for Low-Level Vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 3070--3079.
[27]
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. 2016. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2536--2544.
[28]
Weihong Ren, Jiandong Tian, Zhi Han, Antoni Chan, and Yandong Tang. 2017. Video desnowing and deraining based on matrix decomposition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 4210--4219.
[29]
Varun Santhaseelan and Vijayan K Asari. 2015. Utilizing local phase information to remove rain from video. International Journal of Computer Vision, Vol. 112, 1 (2015), 71--89.
[30]
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, Vol. 13, 4 (2004), 600--612.
[31]
Wei Wei, Lixuan Yi, Qi Xie, Qian Zhao, Deyu Meng, and Zongben Xu. 2017. Should we encode rain streaks in video as deterministic or stochastic. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 2516--2525.
[32]
Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. 2016. Joint rain detection and removal via iterative region dependent multi-task learning. CoRR, abs/1609.07769, Vol. 2 (2016), 3.
[33]
Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. 2017. Deep joint rain detection and removal from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1357--1366.
[34]
Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329 (2014).
[35]
He Zhang and Vishal M. Patel. 2018. Density-Aware Single Image De-Raining Using a Multi-Stream Dense Network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 695--704.
[36]
He Zhang, Vishwanath Sindagi, and Vishal M Patel. 2017. Image de-raining using a conditional generative adversarial network. arXiv preprint arXiv:1701.05957 (2017).
[37]
Lei Zhu, Chi-Wing Fu, Dani Lischinski, and Pheng-Ann Heng. 2017. Joint bilayer optimization for single-image rain streak removal. In Proceedings of the IEEE International Conference on Computer Vision. 2526--2534.

Cited By

View all
  • (2025)FoNet: Focused Network for Single Image DerainingCircuits, Systems, and Signal Processing10.1007/s00034-025-03009-9Online publication date: 7-Feb-2025
  • (2024)Rainmer: Learning Multi-view Representations for Comprehensive Image Deraining and BeyondProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681342(2766-2775)Online publication date: 28-Oct-2024
  • (2024)A Global Self-Attention Memristive Neural Network for Image RestorationIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2024.33694478:3(2613-2624)Online publication date: Jun-2024
  • Show More Cited By

Index Terms

  1. Single Image Deraining via Recurrent Hierarchy Enhancement Network

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MM '19: Proceedings of the 27th ACM International Conference on Multimedia
    October 2019
    2794 pages
    ISBN:9781450368896
    DOI:10.1145/3343031
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 15 October 2019

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. hierarchy features
    2. low-level vision
    3. recurrent network
    4. single image deraining

    Qualifiers

    • Research-article

    Conference

    MM '19
    Sponsor:

    Acceptance Rates

    MM '19 Paper Acceptance Rate 252 of 936 submissions, 27%;
    Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)19
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 28 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)FoNet: Focused Network for Single Image DerainingCircuits, Systems, and Signal Processing10.1007/s00034-025-03009-9Online publication date: 7-Feb-2025
    • (2024)Rainmer: Learning Multi-view Representations for Comprehensive Image Deraining and BeyondProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681342(2766-2775)Online publication date: 28-Oct-2024
    • (2024)A Global Self-Attention Memristive Neural Network for Image RestorationIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2024.33694478:3(2613-2624)Online publication date: Jun-2024
    • (2024)Neuromorphic Vision Restoration Network for Advanced Driver Assistance SystemIEEE Transactions on Consumer Electronics10.1109/TCE.2024.336772870:1(3658-3668)Online publication date: Feb-2024
    • (2024)LPN-IDD: A Lightweight Pyramid Network for Image Deraining and DetectionIEEE Access10.1109/ACCESS.2024.337148412(37103-37119)Online publication date: 2024
    • (2024)Single image deraining via wide rectangular regional blocks and dual attention complementary enhancement networkScientific Reports10.1038/s41598-024-70329-214:1Online publication date: 21-Aug-2024
    • (2024)Wavelet-based Auto-Encoder for simultaneous haze and rain removal from imagesPattern Recognition10.1016/j.patcog.2024.110370150:COnline publication date: 1-Jun-2024
    • (2024)From heavy rain removal to detail restoration: A faster and better networkPattern Recognition10.1016/j.patcog.2023.110205148(110205)Online publication date: Apr-2024
    • (2024)MWA-MNNExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.122427240:COnline publication date: 15-Apr-2024
    • (2024)Internal and external transmission encoder–decoder network for single-image derainingThe Visual Computer10.1007/s00371-024-03261-140:12(8653-8663)Online publication date: 20-Mar-2024
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media