skip to main content
10.1145/3581783.3612271acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Personalized Single Image Reflection Removal Network through Adaptive Cascade Refinement

Published: 27 October 2023 Publication History

Abstract

In this paper, we aim to restore a reflection-free image from a single reflection-contaminated image captured through the glass. Many deep-learning-based methods attempt to solve the challenging problem by utilizing a uniform model obtained from training data for all test images. Hence, the distinctive characteristics of the test images are not considered. Besides, several methods use a cascade structure in image restoration to refine the results. But they blindly cascade modules with the same weights, improving the model's performance only to a certain extent. To address these problems, we propose a personalized single-image reflection removal network through adaptive cascade refinement (PNACR) based on meta-learning and self-supervised learning. While meta-learning can rapidly adapt to a new task with a few samples, PNACR can remove reflections of a new image with its distinctive characteristics learned by self-supervised learning. Furthermore, the proposed adaptive cascade model can adjust the weights of the model at the next iteration according to the output of the model at the current iteration, significantly improving the model's performance. Hence, the proposed model can learn information from both external training data and the new input image to provide a personalized reflection removal model for each new input image. Extensive comparison and ablation experiments on publicly available datasets demonstrate the validity of the proposed method in quantitative evaluation metrics and qualitative visualization.

Supplemental Material

MP4 File
This presentation video describes the introduction, method, experiments, and conclusion sections of the paper named Personalized Single Image Reflection Removal Network through Adaptive Cascade Refinement, which can help you quickly understand the paper's content.

References

[1]
Nikolaos Arvanitopoulos, Radhakrishna Achanta, and Sabine Susstrunk. 2017. Single image reflection suppression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4498--4506.
[2]
Zhixiang Chi, Yang Wang, Yuanhao Yu, and Jin Tang. 2021. Test-time fast adaptation for dynamic scene deblurring via meta-auxiliary learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9137--9146.
[3]
Zheng Dong, Ke Xu, Yin Yang, Hujun Bao, Weiwei Xu, and Rynson WH Lau. 2021. Location-aware single image reflection removal. In Proceedings of the IEEE International Conference on Computer Vision. 5017--5026.
[4]
Qingnan Fan, Jiaolong Yang, Gang Hua, Baoquan Chen, and David Wipf. 2017. A generic deep architecture for single image reflection removal and image smoothing. In Proceedings of the IEEE International Conference on Computer Vision. 3238--3247.
[5]
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning. 1126--1135.
[6]
Kun Gai, Zhenwei Shi, and Changshui Zhang. 2011. Blind separation of superimposed moving images using image statistics. IEEE Transactions on Pattern Analysis and Machine Intelligence (2011), 19--32.
[7]
Byeong-Ju Han and Jae-Young Sim. 2017. Reflection removal using low-rank matrix completion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5438--5446.
[8]
Byeong-Ju Han and Jae-Young Sim. 2018. Glass reflection removal using co-saliency-based image alignment and low-rank matrix completion in gradient domain. IEEE Transactions on Image Processing (2018), 4873--4888.
[9]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770--778.
[10]
Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7132--7141.
[11]
Qiming Hu and Xiaojie Guo. 2021. Trash or treasure? an interactive dual-stream strategy for single image reflection separation. Advances in Neural Information Processing Systems (2021), 24683--24694.
[12]
Jun-Jie Huang, Tianrui Liu, Zhixiong Yang, Shaojing Fu, Wentao Zhao, and Pier Luigi Dragotti. 2022. DURRNet: Deep Unfolded Single Image Reflection Removal Network. arXiv preprint arXiv:2203.06306 (2022).
[13]
Yan Huang, Yuhui Quan, Yong Xu, Ruotao Xu, and Hui Ji. 2019. Removing reflection from a single image with ghosting effect. IEEE Transactions on Computational Imaging (2019), 34--45.
[14]
Alexia Jolicoeur-Martineau. 2018. The relativistic discriminator: a key element missing from standard GAN. arXiv preprint arXiv:1807.00734 (2018).
[15]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[16]
Anat Levin and Yair Weiss. 2007. User assisted separation of reflections from a single image using a sparsity prior. IEEE Transactions on Pattern Analysis and Machine Intelligence (2007), 1647--1654.
[17]
Chao Li, Yixiao Yang, Kun He, Stephen Lin, and John E Hopcroft. 2020. Single image reflection removal through cascaded refinement. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3565--3574.
[18]
Yu Li and Michael S Brown. 2013. Exploiting reflection change for automatic reflection removal. In Proceedings of the IEEE International Conference on Computer Vision. 2432--2439.
[19]
Yu Li and Michael S Brown. 2014. Single image layer separation using relative smoothness. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2752--2759.
[20]
Yu Li, Ming Liu, Yaling Yi, Qince Li, Dongwei Ren, and Wangmeng Zuo. 2023. Two-stage single image reflection removal with reflection-aware guidance. Applied Intelligence (2023), 1--16.
[21]
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 136--144.
[22]
Huan Liu, Zijun Wu, Liangyan Li, Sadaf Salehkalaibar, Jun Chen, and Keyan Wang. 2022b. Towards multi-domain single image dehazing via test-time training. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5831--5840.
[23]
Ming Liu, Jianan Pan, Zifei Yan, Wangmeng Zuo, and Lei Zhang. 2022a. Adaptive Network Combination for Single-Image Reflection Removal: A Domain Generalization Perspective. arXiv preprint arXiv:2204.01505 (2022).
[24]
Yunfei Liu, Yu Li, Shaodi You, and Feng Lu. 2019. Semantic Guided Single Image Reflection Removal. ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 18 (2019), 1--23.
[25]
Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. 2017. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3883--3891.
[26]
Seobin Park, Jinsu Yoo, Donghyeon Cho, Jiwon Kim, and Tae Hyun Kim. 2020. Fast adaptation to super-resolution networks via meta-learning. In Proceedings of the European Conference on Computer Vision. 754--769.
[27]
BH Prasad, Lokesh R Boregowda, Kaushik Mitra, Sanjoy Chowdhury, et al. 2021. V-desirr: Very fast deep embedded single image reflection removal. In Proceedings of the IEEE International Conference on Computer Vision. 2390--2399.
[28]
Dongwei Ren, Wei Shang, Pengfei Zhu, Qinghua Hu, Deyu Meng, and Wangmeng Zuo. 2020. Single image deraining using bilateral recurrent network. IEEE Transactions on Image Processing (2020), 6852--6863.
[29]
Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng. 2019. Progressive image deraining networks: A better and simpler baseline. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3937--3946.
[30]
YiChang Shih, Dilip Krishnan, Fredo Durand, and William T Freeman. 2015. Reflection removal using ghosting cues. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3193--3201.
[31]
Christian Simon and In Kyu Park. 2015. Reflection removal for in-vehicle black box videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4231--4239.
[32]
Jae Woong Soh, Sunwoo Cho, and Nam Ik Cho. 2020. Meta-transfer learning for zero-shot super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3516--3525.
[33]
Chao Sun, Shuaicheng Liu, Taotao Yang, Bing Zeng, Zhengning Wang, and Guanghui Liu. 2016. Automatic reflection removal using gradient intensity and motion cues. In Proceedings of the 24th ACM International Conference on Multimedia. 466--470.
[34]
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi. 2017. Inception-v4, inception-resnet and the impact of residual connections on learning. In Proceedings of the Association for the Advancement of Artificial Intelligence Conference on Artificial Intelligence.
[35]
Richard Szeliski, Shai Avidan, and Padmanabhan Anandan. 2000. Layer extraction from multiple images containing reflections and transparency. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition. 246--253.
[36]
Renjie Wan, Boxin Shi, Ling-Yu Duan, Ah-Hwee Tan, Wen Gao, and Alex C Kot. 2018. Region-aware reflection removal with unified content and gradient priors. IEEE Transactions on Image Processing (2018), 2927--2941.
[37]
Renjie Wan, Boxin Shi, Ling-Yu Duan, Ah-Hwee Tan, and Alex C Kot. 2017. Benchmarking single-image reflection removal algorithms. In Proceedings of the IEEE International Conference on Computer Vision. 3922--3930.
[38]
Renjie Wan, Boxin Shi, Haoliang Li, Ling-Yu Duan, Ah-Hwee Tan, and Alex C Kot. 2019. CoRRN: Cooperative reflection removal network. IEEE Transactions on Pattern Analysis and Machine Intelligence (2019), 2969--2982.
[39]
Kaixuan Wei, Jiaolong Yang, Ying Fu, David Wipf, and Hua Huang. 2019. Single image reflection removal exploiting misaligned training data and network enhancements. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8178--8187.
[40]
Qiang Wen, Yinjie Tan, Jing Qin, Wenxi Liu, Guoqiang Han, and Shengfeng He. 2019. Single image reflection removal beyond linearity. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3771--3779.
[41]
Tianfan Xue, Michael Rubinstein, Ce Liu, and William T Freeman. 2015. A computational approach for obstruction-free photography. ACM Transactions on Graphics (2015), 1--11.
[42]
Jie Yang, Dong Gong, Lingqiao Liu, and Qinfeng Shi. 2018. Seeing deeply and bidirectionally: A deep learning approach for single image reflection removal. In Proceedings of the European Conference on Computer Vision. 654--669.
[43]
Yang Yang, Wenye Ma, Yin Zheng, Jian-Feng Cai, and Weiyu Xu. 2019. Fast single image reflection suppression via convex optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8141--8149.
[44]
Xuaner Zhang, Ren Ng, and Qifeng Chen. 2018a. Single image reflection separation with perceptual losses. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4786--4794.
[45]
Xinxin Zhang, Kaixin Xing, Qifang Liu, Da Chen, and Yilong Yin. 2023. Single Image Reflection Removal Based on Dark Channel Sparsity Prior. IEEE Transactions on Circuits and Systems for Video Technology (2023).
[46]
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. 2018b. Residual dense network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2472--2481.
[47]
Ya-Nan Zhang, Linlin Shen, and Qiufu Li. 2022. Content and Gradient Model-driven Deep Network for Single Image Reflection Removal. In Proceedings of the 30th ACM International Conference on Multimedia. 6802--6812.
[48]
Qian Zheng, Boxin Shi, Jinnan Chen, Xudong Jiang, Ling-Yu Duan, and Alex C Kot. 2021. Single image reflection removal with absorption effect. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 13395--13404.
[49]
Yurui Zhu, Xueyang Fu, Zheyu Zhang, Aiping Liu, Zhiwei Xiong, and Zheng-Jun Zha. 2023. Hue Guidance Network for Single Image Reflection Removal. IEEE Transactions on Neural Networks and Learning Systems (2023).
[50]
Zhengxia Zou, Sen Lei, Tianyang Shi, Zhenwei Shi, and Jieping Ye. 2020. Deep adversarial decomposition: A unified framework for separating superimposed images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 12806--12816.

Cited By

View all
  • (2024)Reflection Removal via Recurrent Learning Guided by Physics Prior and Focal Perceptual LossIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.340393234:10(10152-10165)Online publication date: Oct-2024
  • (2024)Spatio-Temporal Multi-Image Reflection RemovalIEEE Signal Processing Letters10.1109/LSP.2024.345600631(2345-2349)Online publication date: 2024
  • (2024)Revisiting Single Image Reflection Removal in the Wild2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.02406(25468-25478)Online publication date: 16-Jun-2024

Index Terms

  1. Personalized Single Image Reflection Removal Network through Adaptive Cascade Refinement

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MM '23: Proceedings of the 31st ACM International Conference on Multimedia
    October 2023
    9913 pages
    ISBN:9798400701085
    DOI:10.1145/3581783
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 27 October 2023

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. adaptive cascade refinement
    2. meta-learning
    3. reflection removal
    4. self-supervised learning

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    MM '23
    Sponsor:
    MM '23: The 31st ACM International Conference on Multimedia
    October 29 - November 3, 2023
    Ottawa ON, Canada

    Acceptance Rates

    Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)125
    • Downloads (Last 6 weeks)9
    Reflects downloads up to 17 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Reflection Removal via Recurrent Learning Guided by Physics Prior and Focal Perceptual LossIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.340393234:10(10152-10165)Online publication date: Oct-2024
    • (2024)Spatio-Temporal Multi-Image Reflection RemovalIEEE Signal Processing Letters10.1109/LSP.2024.345600631(2345-2349)Online publication date: 2024
    • (2024)Revisiting Single Image Reflection Removal in the Wild2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.02406(25468-25478)Online publication date: 16-Jun-2024

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media