skip to main content
10.1145/3474085.3475632acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Disentangled Representation Learning and Enhancement Network for Single Image De-Raining

Published: 17 October 2021 Publication History

Abstract

In this paper, we present a disentangled representation learning and enhancement network (DRLE-Net) to address the challenging single image de-raining problems, i.e., raindrop and rain streak removal. Specifically, the DRLE-Net is formulated as a multi-task learning framework, and an elegant knowledge transfer strategy is designed to train the encoder of DRLE-Net to embed a rainy image into two separated latent spaces representing the task (clean image reconstruction in this paper) relevant and irrelevant variations respectively, such that only the essential task-relevant factors will be used by the decoder of DRLE-Net to generate high-quality de-raining results. Furthermore, visual attention information is modeled and fed into the disentangled representation learning network to enhance the task-relevant factor learning. To facilitate the optimization of the hierarchical network, a new adversarial loss formulation is proposed and used together with the reconstruction loss to train the proposed DRLE-Net. Extensive experiments are carried out for removing raindrops or rainstreaks from both synthetic and real rainy images, and DRLE-Net is demonstrated to produce significantly better results than state-of-the-art models.

References

[1]
Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, and Gang Hua. 2018. Towards Open-Set Identity Preserving Face Synthesis. In CVPR. 6713--6722.
[2]
Jiezhang Cao, Langyuan Mo, Yifan Zhang, Kui Jia, Chunhua Shen, and Mingkui Tan. 2019. Multi-Marginal Wasserstein GAN. In NIPS. 1776--1786.
[3]
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. 2016. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. In NIPS. 2180--2188.
[4]
David Eigen, Dilip Krishnan, and Rob Fergus. 2013. Restoring an Image Taken Through a Window Covered with Dirt or Rain. In ICCV. 633--640.
[5]
Xueyang Fu, Jiabin Huang, Xinghao Ding, Yinghao Liao, and John Paisley. 2017a. Clearing the Skies: A Deep Network Architecture for Single-Image Rain Removal. IEEE T-IP, Vol. 26, 6 (2017), 2944--2956.
[6]
Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley. 2017b. Removing Rain from Single Images via a Deep Detail Network. In CVPR. 3855--3863.
[7]
Abel Gonzalez-Garcia, Joost Van De Weijer, and Yoshua Bengio. 2018. Image-to-Image Translation for Cross-Domain Disentanglement. In NIPS. 1287--1298.
[8]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In NIPS. 2672--2680.
[9]
Adam Harley, Shih-En Wei, Jason Saragih, and Katerina Fragkiadaki. 2019. Image Disentanglement and Uncooperative Re-Entanglement for High-Fidelity Image-to-Image Translation. In ICCVW. 3324--3332.
[10]
Kaiming He, Jian Sun, and Xiaoou Tang. 2011. Single Image Haze Removal Using Dark Channel Prior. IEEE T-PAMI, Vol. 33, 12 (2011), 2341--2353.
[11]
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely Connected Convolutional Networks. In CVPR. 4700--4708.
[12]
Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. 2018. Multimodal Unsupervised Image-to-Image Translation. In ECCV. 172--189.
[13]
Nabil Ibtehaz and M Sohel Rahman. 2020. MultiResUNet: Rethinking the U-Net Architecture for Multimodal Biomedical Image Segmentation. Neural Networks, Vol. 121 (2020), 74--87.
[14]
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. In CVPR. 1125--1134.
[15]
Diederik P Kingma and Max Welling. 2013. Auto-Encoding Variational Bayes. arXiv preprint arXiv:1312.6114.
[16]
Tejas D Kulkarni, Will Whitney, Pushmeet Kohli, and Joshua B Tenenbaum. 2015. Deep Convolutional Inverse Graphics Network. In NIPS. 2539--2547.
[17]
Hsin-Ying Lee, Hung-Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. 2018. Diverse Image-to-Image Translation via Disentangled Representations. In ECCV. 35--51.
[18]
Guanbin Li, Xiang He, Wei Zhang, Huiyou Chang, Le Dong, and Liang Lin. 2018a. Non-Locally Enhanced Encoder-Decoder Network for Single Image De-Raining. In ACM Multimedia. 1056--1064.
[19]
Ruoteng Li, Loong-Fah Cheong, and Robby T Tan. 2019 b. Heavy Rain Image Restoration: Integrating Physics Model and Conditional Adversarial Learning. In CVPR. 1633--1642.
[20]
Siyuan Li, Iago Breno Araujo, Wenqi Ren, Zhangyang Wang, Eric K Tokuda, Roberto Hirata Junior, Roberto Cesar-Junior, Jiawan Zhang, Xiaojie Guo, and Xiaochun Cao. 2019 a. Single Image Deraining: A Comprehensive Benchmark Analysis. In CVPR. 3838--3847.
[21]
Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, and Hongbin Zha. 2018b. Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining. In ECCV. 254--269.
[22]
Yu Li, Robby T Tan, Xiaojie Guo, Jiangbo Lu, and Michael S Brown. 2016. Rain Streak Removal using Layer Priors. In CVPR. 2736--2744.
[23]
Boyu Lu, Jun-Cheng Chen, and Rama Chellappa. 2019. Unsupervised Domain-Specific Deblurring via Disentangled Representations. In CVPR. 10225--10234.
[24]
Yu Luo, Yong Xu, and Hui Ji. 2015. Removing Rain from a Single Image via Discriminative Sparse Coding. In ICCV. 3397--3405.
[25]
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. 2017. Least Squares Generative Adversarial Networks. In ICCV. 2794--2802.
[26]
Xiao-Jiao Mao, Chunhua Shen, and Yu-Bin Yang. 2016. Image Restoration Using Convolutional Auto-Encoders with Symmetric Skip Connections. In NIPS. 168--176.
[27]
Rui Qian, Robby T Tan, Wenhan Yang, Jiajun Su, and Jiaying Liu. 2018. Attentive Generative Adversarial Network for Raindrop Removal from A Single Image. In CVPR. 2482--2491.
[28]
Scott Reed, Kihyuk Sohn, Yuting Zhang, and Honglak Lee. 2014. Learning to Disentangle Factors of Variation with Manifold Interaction. In ICML. 1431--1439.
[29]
Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng. 2019 a. Progressive Image Deraining Networks: a Better and Simpler Baseline. In CVPR. 3937--3946.
[30]
Dongwei Ren, Wangmeng Zuo, David Zhang, Lei Zhang, and Ming-Hsuan Yang. 2019 b. Simultaneous Fidelity and Regularization Learning for Image Restoration. IEEE T-PAMI, Vol. 43, 1 (2019), 284--299.
[31]
Weihong Ren, Jiandong Tian, Zhi Han, Antoni Chan, and Yandong Tang. 2017. Video Desnowing and Deraining Based on Matrix Decomposition. In CVPR. 4210--4219.
[32]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. In MICCAI. 234--241.
[33]
Zhixin Shu, Mihir Sahasrabudhe, Riza Alp Guler, Dimitris Samaras, Nikos Paragios, and Iasonas Kokkinos. 2018. Deforming Autoencoders: Unsupervised Disentangling of Shape and Appearance. In ECCV. 650--665.
[34]
Zhixin Shu, Ersin Yumer, Sunil Hadap, Kalyan Sunkavalli, Eli Shechtman, and Dimitris Samaras. 2017. Neural Face Editing with Intrinsic Image Disentangling. In CVPR. 5541--5550.
[35]
Guoqing Wang, Changming Sun, and Arcot Sowmya. 2019. ERL-Net: Entangled Representation Learning for Single Image De-Raining. In ICCV. 5644--5652.
[36]
Guoqing Wang, Changming Sun, and Arcot Sowmya. 2020. Cascaded Attention Guidance Network for Single Rainy Image Restoration. IEEE T-IP, Vol. 29 (2020), 9190--9203.
[37]
Guoqing Wang, Changming Sun, and Arcot Sowmya. 2021 a. Attentive Feature Refinement Network for Single Rainy Image Restoration. IEEE T-IP, Vol. 30 (2021), 3734--3747.
[38]
Guoqing Wang, Changming Sun, and Arcot Sowmya. 2021 b. Context-Enhanced Representation Learning for Single Image Deraining. IJCV, Vol. 129, 5 (2021), 1650--1674.
[39]
Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. 2017. Deep Joint Rain Detection and Removal From a Single Image. In CVPR. 1357--1366.
[40]
Rajeev Yasarla and Vishal M Patel. 2019. Uncertainty Guided Multi-Scale Residual Learning-using a Cycle Spinning CNN for Single Image De-Raining. In CVPR. 8405--8414.
[41]
He Zhang and Vishal M Patel. 2018. Density-Aware Single Image De-Raining Using a Multi-Stream Dense Network. In CVPR. 695--704.
[42]
Zhifei Zhang, Yang Song, and Hairong Qi. 2018. Decoupled Learning for Conditional Adversarial Networks. In WACV. 700--708.
[43]
Ziqiang Zheng, Zhibin Yu, Haiyong Zheng, Yang Yang, and Heng Tao Shen. 2021. One-Shot Image-to-Image Translation via Part-Global Learning with a Multi-Adversarial Framework. IEEE T-MM (2021).
[44]
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017. Unpaired Image-to-Image Translation Using cycle-Consistent Adversarial Networks. In ICCV. 2223--2232.

Cited By

View all
  • (2024)Complex Relation Embedding for Scene Graph GenerationIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2022.322687135:6(8321-8335)Online publication date: Jun-2024
  • (2024)Semantics Disentangling for Cross-Modal RetrievalIEEE Transactions on Image Processing10.1109/TIP.2024.337411133(2226-2237)Online publication date: 2024
  • (2024)Unsupervised Adaptation Learning for Real Multiplatform Hyperspectral Image DenoisingIEEE Transactions on Cybernetics10.1109/TCYB.2024.341227054:10(5781-5794)Online publication date: Oct-2024

Index Terms

  1. Disentangled Representation Learning and Enhancement Network for Single Image De-Raining

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MM '21: Proceedings of the 29th ACM International Conference on Multimedia
    October 2021
    5796 pages
    ISBN:9781450386517
    DOI:10.1145/3474085
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 17 October 2021

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. attention guided representation enhancement
    2. factor disentanglement
    3. knowledge transfer
    4. raindrop and rainstreak removal
    5. regularized adversarial loss.

    Qualifiers

    • Research-article

    Funding Sources

    • the Fundamental Research Funds for the Central Universities
    • the National Natural Science Foundation of China
    • the Sichuan Science and Technology Program, China

    Conference

    MM '21
    Sponsor:
    MM '21: ACM Multimedia Conference
    October 20 - 24, 2021
    Virtual Event, China

    Acceptance Rates

    Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)44
    • Downloads (Last 6 weeks)4
    Reflects downloads up to 17 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Complex Relation Embedding for Scene Graph GenerationIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2022.322687135:6(8321-8335)Online publication date: Jun-2024
    • (2024)Semantics Disentangling for Cross-Modal RetrievalIEEE Transactions on Image Processing10.1109/TIP.2024.337411133(2226-2237)Online publication date: 2024
    • (2024)Unsupervised Adaptation Learning for Real Multiplatform Hyperspectral Image DenoisingIEEE Transactions on Cybernetics10.1109/TCYB.2024.341227054:10(5781-5794)Online publication date: Oct-2024
    • (2024)Unsupervised face image deblurring via disentangled representation learningPattern Recognition Letters10.1016/j.patrec.2024.04.020183(9-16)Online publication date: Jul-2024

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media