skip to main content
10.1145/3664647.3681342acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Rainmer: Learning Multi-view Representations for Comprehensive Image Deraining and Beyond

Published: 28 October 2024 Publication History

Abstract

We address image deraining under complex backgrounds, diverse rain scenarios, and varying illumination conditions, representing a highly practical and challenging problem. Our approach utilizes synthetic, real-world, and nighttime datasets, wherein rich backgrounds, multiple degradation types, and diverse illumination conditions coexist. The primary challenge in training models on these datasets arises from the discrepancies among them, potentially leading to conflicts or competition during the training period. To address this issue, we first align the distribution of synthetic, real-world and nighttime datasets. Then we propose a novel contrastive learning strategy to extract multi-view (multiple) representations that effectively capture image details, degradations, and illuminations, thereby facilitating training across all datasets. Regarding multiple representations as profitable prompts for deraining, we devise a prompting strategy to integrate them into the decoding process. This contributes to a potent deraining model, dubbed Rainmer. Additionally, a spatial-channel interaction module is introduced to fully exploit cues when extracting multi-view representations. Extensive experiments on synthetic, real-world, and nighttime datasets demonstrate that Rainmer outperforms current representative methods. Moreover, Rainmer achieves superior performance on the All-in-One image restoration dataset, underscoring its effectiveness. Furthermore, quantitative results reveal that Rainmer significantly improves object detection performance on both daytime and nighttime rainy datasets. These observations substantiate the potential of Rainmer for practical applications.

References

[1]
Mina Alibeigi, William Ljungbergh, Adam Tonderski, Georg Hess, Adam Lilja, Carl Lindström, Daria Motorniuk, Junsheng Fu, Jenny Widahl, and Christoffer Petersson. 2023. Zenseact open dataset: A large-scale and diverse multimodal dataset for autonomous driving. In Proceedings of the IEEE International Conference on Computer Vision. 20178--20188.
[2]
Yunhao Ba, Howard Zhang, Ethan Yang, Akira Suzuki, Arnold Pfahnl, Chethan Chinder Chandrappa, Celso M de Melo, Suya You, Stefano Soatto, Alex Wong, et al. 2022. Not just streaks: Towards ground truth for single image deraining. In Proceedings of the European Conference on Computer Vision. Springer, 723--740.
[3]
Joao Cartucho, Rodrigo Ventura, and Manuela Veloso. 2018. Robust object recognition through symbiotic deep learning in mobile robots. In International Conference on Intelligent Robots and Systems. IEEE, 2336--2341.
[4]
Xiang Chen, Hao Li, Mingqiang Li, and Jinshan Pan. 2023. Learning A Sparse Transformer Network for Effective Image Deraining. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5896--5905.
[5]
Xiang Chen, Jinshan Pan, Kui Jiang, Yufeng Li, Yufeng Huang, Caihua Kong, Longgang Dai, and Zhentao Fan. 2022. Unpaired deep image deraining using dual contrastive learning. In IEEE Conference on Computer Vision and Pattern Recognition. 2017--2026.
[6]
Xiang Chen, Jinshan Pan, Jiyang Lu, Zhentao Fan, and Hao Li. 2023. Hybrid cnn-transformer feature fusion for single image deraining. In AAAI Conference on Artificial Intelligence, Vol. 37. 378--386.
[7]
Imre Csiszár. 1975. I-divergence geometry of probability distributions and minimization problems. The Annals of Probability (1975), 146--158.
[8]
Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley. 2017. Removing rain from single images via a deep detail network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3855--3863.
[9]
Xueyang Fu, Borong Liang, Yue Huang, Xinghao Ding, and John Paisley. 2019. Lightweight pyramid networks for image deraining. IEEE Transactions on Neural Networks and Learning Systems, Vol. 31, 6 (2019), 1794--1807.
[10]
Shuhang Gu, Deyu Meng, Wangmeng Zuo, and Lei Zhang. 2017. Joint convolutional analysis and synthesis sparse representation for single image layer separation. In Proceedings of the IEEE International Conference on Computer Vision. 1708--1716.
[11]
Qing Guo, Jingyang Sun, Felix Juefei-Xu, Lei Ma, Xiaofei Xie, Wei Feng, Yang Liu, and Jianjun Zhao. 2021. Efficientderain: Learning pixel-wise dilation filtering for high-efficiency single-image deraining. In AAAI Conference on Artificial Intelligence, Vol. 35. 1487--1495.
[12]
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9729--9738.
[13]
Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-excitation networks. In IEEE Conference on Computer Vision and Pattern Recognition. 7132--7141.
[14]
Huaibo Huang, Aijing Yu, and Ran He. 2021. Memory oriented transfer learning for semi-supervised image deraining. In IEEE Conference on Computer Vision and Pattern Recognition. 7732--7741.
[15]
Quan Huynh-Thu and Mohammed Ghanbari. 2008. Scope of validity of PSNR in image/video quality assessment. Electronics Letters, Vol. 44, 13 (2008), 800--801.
[16]
Kui Jiang, Zhongyuan Wang, Peng Yi, Chen Chen, Baojin Huang, Yimin Luo, Jiayi Ma, and Junjun Jiang. 2020. Multi-scale progressive fusion network for single image deraining. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8346--8355.
[17]
Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. 2018. Fast and accurate image super-resolution with deep laplacian pyramid networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 41, 11 (2018), 2599--2613.
[18]
Mohit Lamba and Kaushik Mitra. 2021. Restoring extremely dark images in real time. In IEEE Conference on Computer Vision and Pattern Recognition. 3487--3497.
[19]
Boyun Li, Xiao Liu, Peng Hu, Zhongqin Wu, Jiancheng Lv, and Xi Peng. 2022. All-in-one image restoration for unknown corruption. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 17452--17462.
[20]
Ruoteng Li, Loong-Fah Cheong, and Robby T Tan. 2019. Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1633--1642.
[21]
Ruoteng Li, Robby T Tan, and Loong-Fah Cheong. 2020. All in one bad weather removal using architectural search. In IEEE Conference on Computer Vision and Pattern Recognition. 3175--3185.
[22]
Siyuan Li, Iago Breno Araujo, Wenqi Ren, Zhangyang Wang, Eric K Tokuda, Roberto Hirata Junior, Roberto Cesar-Junior, Jiawan Zhang, Xiaojie Guo, and Xiaochun Cao. 2019. Single image deraining: A comprehensive benchmark analysis. In IEEE Conference on Computer Vision and Pattern Recognition. 3838--3847.
[23]
Yu Li, Robby T Tan, Xiaojie Guo, Jiangbo Lu, and Michael S Brown. 2016. Rain streak removal using layer priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2736--2744.
[24]
Yun-Fu Liu, Da-Wei Jaw, Shih-Chia Huang, and Jenq-Neng Hwang. 2018. Desnownet: Context-aware deep network for snow removal. IEEE Transactions on Image Processing, Vol. 27, 6 (2018), 3064--3073.
[25]
Ilya Loshchilov and Frank Hutter. 2016. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations.
[26]
Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Conference on Learning Representations.
[27]
Yu Luo, Yong Xu, and Hui Ji. 2015. Removing rain from a single image via discriminative sparse coding. In IEEE International Conference on Computer Vision. 3397--3405.
[28]
Chengqi Lyu, Wenwei Zhang, Haian Huang, Yue Zhou, Yudong Wang, Yanyi Liu, Shilong Zhang, and Kai Chen. 2022. Rtmdet: An empirical study of designing real-time object detectors. arXiv preprint arXiv:2212.07784 (2022).
[29]
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the IEEE International Conference on Computer Vision, Vol. 2. IEEE, 416--423.
[30]
Chong Mou, Qian Wang, and Jian Zhang. 2022. Deep generalized unfolding networks for image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 17399--17410.
[31]
Ozan Özdenizci and Robert Legenstein. 2023. Restoring vision in adverse weather conditions with patch-based denoising diffusion models. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).
[32]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In Annual Conference on Neural Information Processing Systems, Vol. 32.
[33]
Vaishnav Potlapalli, Syed Waqas Zamir, Salman Khan, and Fahad Shahbaz Khan. 2023. Promptir: Prompting for all-in-one blind image restoration. Advances in Neural Information Processing Systems, Vol. 36 (2023).
[34]
Rui Qian, Robby T Tan, Wenhan Yang, Jiajun Su, and Jiaying Liu. 2018. Attentive generative adversarial network for raindrop removal from a single image. In IEEE Conference on Computer Vision and Pattern Recognition. 2482--2491.
[35]
Wu Ran, Peirong Ma, Zhiquan He, Hao Ren, and Hong Lu. 2024. Harnessing joint rain-/detail-aware representations to eliminate intricate rains. In International Conference on Learning Representations.
[36]
Dongwei Ren, Wei Shang, Pengfei Zhu, Qinghua Hu, Deyu Meng, and Wangmeng Zuo. 2020. Single image deraining using bilateral recurrent network. IEEE Transactions on Image Processing, Vol. 29 (2020), 6852--6863.
[37]
Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng. 2019. Progressive image deraining networks: A better and simpler baseline. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3937--3946.
[38]
Gerald Schaefer and Michal Stich. 2003. UCID: An uncompressed color image database. In Storage and retrieval methods and applications for multimedia 2004, Vol. 5307. SPIE, 472--480.
[39]
Jeya Maria Jose Valanarasu, Rajeev Yasarla, and Vishal M Patel. 2022. Transweather: Transformer-based restoration of images degraded by adverse weather conditions. In IEEE Conference on Computer Vision and Pattern Recognition. 2353--2363.
[40]
Cong Wang, Jinshan Pan, Wei Wang, Jiangxin Dong, Mengzhu Wang, Yakun Ju, and Junyang Chen. 2023. PromptRestorer: A Prompting Image Restoration Method with Degradation Perception. Advances in Neural Information Processing Systems, Vol. 36 (2023).
[41]
Cong Wang, Yutong Wu, Zhixun Su, and Junyang Chen. 2020. Joint self-attention and scale-aggregation for self-calibrated deraining network. In ACM International Conference on Multimedia. 2517--2525.
[42]
Cong Wang, Xiaoying Xing, Yutong Wu, Zhixun Su, and Junyang Chen. 2020. Dcsfn: Deep cross-scale fusion network for single image rain removal. In ACM International Conference on Multimedia. 1643--1651.
[43]
Hong Wang, Qi Xie, Qian Zhao, and Deyu Meng. 2020. A model-driven deep neural network for single image rain removal. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3103--3112.
[44]
Longguang Wang, Yingqian Wang, Xiaoyu Dong, Qingyu Xu, Jungang Yang, Wei An, and Yulan Guo. 2021. Unsupervised degradation representation learning for blind super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition. 10581--10590.
[45]
Tianyu Wang, Xin Yang, Ke Xu, Shaozhe Chen, Qiang Zhang, and Rynson WH Lau. 2019. Spatial attentive single-image deraining with a high quality real rain dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 12270--12279.
[46]
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, Vol. 13, 4 (2004), 600--612.
[47]
Wei Wei, Deyu Meng, Qian Zhao, Zongben Xu, and Ying Wu. 2019. Semi-supervised transfer learning for image rain removal. In IEEE Conference on Computer Vision and Pattern Recognition. 3877--3886.
[48]
Gang Wu, Junjun Jiang, and Xianming Liu. 2023. A Practical Contrastive Learning Framework for Single-Image Super-Resolution. IEEE Transactions on Neural Networks and Learning Systems (2023).
[49]
Haiyan Wu, Yanyun Qu, Shaohui Lin, Jian Zhou, Ruizhi Qiao, Zhizhong Zhang, Yuan Xie, and Lizhuang Ma. 2021. Contrastive learning for compact single image dehazing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10551--10560.
[50]
Jie Xiao, Xueyang Fu, Aiping Liu, Feng Wu, and Zheng-Jun Zha. 2022. Image de-raining transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).
[51]
Wenhan Yang, Robby T Tan, Jiashi Feng, Zongming Guo, Shuicheng Yan, and Jiaying Liu. 2019. Joint rain detection and removal from a single image with contextualized deep networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 42, 6 (2019), 1377--1393.
[52]
Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. 2017. Deep joint rain detection and removal from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1357--1366.
[53]
Youzhao Yang and Hong Lu. 2019. Single image deraining using a recurrent multi-scale aggregation and enhancement network. In IEEE International Conference on Multimedia and Expo. IEEE, 1378--1383.
[54]
Youzhao Yang and Hong Lu. 2019. Single image deraining via recurrent hierarchy enhancement network. In Proceedings of the ACM International Conference on Multimedia. 1814--1822.
[55]
Rajeev Yasarla, Vishwanath A Sindagi, and Vishal M Patel. 2020. Syn2real transfer learning for image deraining using gaussian processes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2726--2736.
[56]
Yuntong Ye, Changfeng Yu, Yi Chang, Lin Zhu, Xi-le Zhao, Luxin Yan, and Yonghong Tian. 2022. Unsupervised deraining: Where contrastive learning meets self-similarity. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5821--5830.
[57]
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. 2022. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5728--5739.
[58]
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. 2021. Multi-stage progressive image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 14821--14831.
[59]
Jingfeng Zang, Guibin Ren, Jianning Dong, Yan Piao, and Seio Jim. 2019. Removal of rain video based on temporal intensity and chromatic constraint of raindrops. Evolutionary Intelligence, Vol. 12, 3 (2019), 349--355.
[60]
Fan Zhang, Shaodi You, Yu Li, and Ying Fu. 2022. GTAV-NightRain: Photometric Realistic Large-scale Dataset for Night-time Rain Streak Removal. arXiv preprint arXiv:2210.04708 (2022).
[61]
Fan Zhang, Shaodi You, Yu Li, and Ying Fu. 2023. Learning Rain Location Prior for Nighttime Deraining. In Proceedings of the IEEE International Conference on Computer Vision. 13148--13157.
[62]
Howard Zhang, Yunhao Ba, Ethan Yang, Varan Mehra, Blake Gella, Akira Suzuki, Arnold Pfahnl, Chethan Chinder Chandrappa, Alex Wong, and Achuta Kadambi. 2023. Weatherstream: Light transport automation of single image deweathering. In IEEE Conference on Computer Vision and Pattern Recognition. 13499--13509.
[63]
He Zhang and Vishal M Patel. 2018. Density-aware single image de-raining using a multi-stream dense network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 695--704.
[64]
He Zhang, Vishwanath Sindagi, and Vishal M Patel. 2019. Image de-raining using a conditional generative adversarial network. IEEE Transactions on Circuits and Systems for Video Technology, Vol. 30, 11 (2019), 3943--3956.
[65]
Xiaopeng Zhang, Hao Li, Yingyi Qi, Wee Kheng Leow, and Teck Khim Ng. 2006. Rain Removal in Video by Combining Temporal and Chromatic Properties. In IEEE International Conference on Multimedia and Expo. 461--464. https://doi.org/10.1109/ICME.2006.262572
[66]
Hongyuan Zhu, Xi Peng, Joey Tianyi Zhou, Songfan Yang, Vijay Chanderasekh, Liyuan Li, and Joo-Hwee Lim. 2019. Singe image rain removal with unpaired information: A differentiable programming perspective. In AAAI Conference on Artificial Intelligence. 9332--9339.
[67]
Yurui Zhu, Tianyu Wang, Xueyang Fu, Xuanyu Yang, Xin Guo, Jifeng Dai, Yu Qiao, and Xiaowei Hu. 2023. Learning weather-general and weather-specific features for image restoration under multiple adverse weather conditions. In IEEE Conference on Computer Vision and Pattern Recognition. 21747--21758.

Index Terms

  1. Rainmer: Learning Multi-view Representations for Comprehensive Image Deraining and Beyond

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MM '24: Proceedings of the 32nd ACM International Conference on Multimedia
    October 2024
    11719 pages
    ISBN:9798400706868
    DOI:10.1145/3664647
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 28 October 2024

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. image deraining
    2. multi-view representations
    3. prompting deraining

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    MM '24
    Sponsor:
    MM '24: The 32nd ACM International Conference on Multimedia
    October 28 - November 1, 2024
    Melbourne VIC, Australia

    Acceptance Rates

    MM '24 Paper Acceptance Rate 1,150 of 4,385 submissions, 26%;
    Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 131
      Total Downloads
    • Downloads (Last 12 months)131
    • Downloads (Last 6 weeks)73
    Reflects downloads up to 28 Feb 2025

    Other Metrics

    Citations

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media