R2SFD: Improving Single Image Reflection Removal using Semantic Feature Dictionary
Pages 10277 - 10286
Abstract
Single image reflection removal is a severely ill-posed problem and it is very hard to separate the desirable transmission and undesirable reflection layers. Most of the existing single image reflection removal methods try to recover the transmission layer by exploiting cues that are extracted only from the given input image. However, there is abundant unutilized information in the form of millions of reflection free images available publicly. Even though this information is easily available, utilizing the same for effectively removing reflections is non-trivial. In this paper, we propose a novel method, termed R^2SFD, for improving single image reflection removal using a Semantic Feature Dictionary (SFD) constructed from a database of reflection-free images. The SFD is constructed using a novel Reflection Aware Feature Extractor (RAFENet) that extracts features invariant to the presence of reflections. The SFD and the input image are then passed to another novel network termed SFDNet. This network first extracts RAFENet features from the reflection-corrupted input image, searches for similar features in the SFD, and transfers the semantic content to generate the final output. To further improve reflection removal, we also introduce a Large Scale Reflection Removal (LSRR) dataset consisting of 2650 image pairs comprising of a variety of real world reflection scenarios. The proposed method achieves superior results both qualitatively and quantitatively compared to the state of the art single image reflection removal methods on real public datasets as well as our LSRR dataset. We will release the dataset at https://github.com/ee19d005/r2sfd.
References
[1]
Amit Agrawal, Ramesh Raskar, Shree K Nayar, and Yuanzhen Li. 2005. Removing photography artifacts using gradient projection and flash-exposure sampling. In ACM SIGGRAPH 2005 Papers. 828--835.
[2]
Nikolaos Arvanitopoulos, Radhakrishna Achanta, and Sabine Susstrunk. 2017. Single image reflection suppression. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4498--4506.
[3]
Zhikai Chen, Fuchen Long, Zhaofan Qiu, Juyong Zhang, Zheng-Jun Zha, Ting Yao, and Jiebo Luo. 2024. A Closer Look at the Reflection Formulation in Single Image Reflection Removal. IEEE Transactions on Image Processing (2024).
[4]
Zheng Dong, Ke Xu, Yin Yang, Hujun Bao, Weiwei Xu, and Rynson WH Lau. 2021. Location-aware single image reflection removal. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 5017--5026.
[5]
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. 2010. The pascal visual object classes (voc) challenge. International journal of computer vision, Vol. 88, 2 (2010), 303--338.
[6]
Qingnan Fan, Jiaolong Yang, Gang Hua, Baoquan Chen, and David Wipf. 2017. A generic deep architecture for single image reflection removal and image smoothing. In Proceedings of the IEEE International Conference on Computer Vision. 3238--3247.
[7]
Xiaojie Guo, Xiaochun Cao, and Yi Ma. 2014. Robust separation of reflection from multiple images. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2187--2194.
[8]
Yuchen Hong, Qian Zheng, Lingran Zhao, Xudong Jiang, Alex C Kot, and Boxin Shi. 2021. Panoramic image reflection removal. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7762--7771.
[9]
Qiming Hu and Xiaojie Guo. 2021. Trash or treasure? an interactive dual-stream strategy for single image reflection separation. Advances in Neural Information Processing Systems, Vol. 34 (2021), 24683--24694.
[10]
Yan Huang, Yuhui Quan, Yong Xu, Ruotao Xu, and Hui Ji. 2019. Removing reflection from a single image with ghosting effect. IEEE Transactions on Computational Imaging, Vol. 6 (2019), 34--45.
[11]
Yuming Jiang, Kelvin CK Chan, Xintao Wang, Chen Change Loy, and Ziwei Liu. 2021. Robust reference-based super-resolution via C2-matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2103--2112.
[12]
Meiguang Jin, Sabine Süsstrunk, and Paolo Favaro. 2018. Learning to see through reflections. In 2018 IEEE International Conference on Computational Photography (ICCP). IEEE, 1--12.
[13]
Soomin Kim, Yuchi Huo, and Sung-Eui Yoon. 2020. Single image reflection removal with physically-based training images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5164--5173.
[14]
Suhong Kim, Hamed RahmaniKhezri, Seyed Mohammad Nourbakhsh, and Mohamed Hefeeda. 2020. Unsupervised single-image reflection separation using perceptual deep image priors. arXiv preprint arXiv:2009.00702 (2020).
[15]
Green Rosh KS, Nikhil Krishnan, BH Pawan Prasad, and Sachin Lomte. 2022. Content preserving scale space network for fast image restoration from noisy-blurry pairs. In ICASSP 2022--2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2554--2558.
[16]
Chenyang Lei and Qifeng Chen. 2021. Robust reflection removal with reflection-free flash-only cues. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14811--14820.
[17]
Chenyang Lei, Xuhua Huang, Chenyang Qi, Yankun Zhao, Wenxiu Sun, Qiong Yan, and Qifeng Chen. 2022. A categorized reflection removal dataset with diverse real-world scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3040--3048.
[18]
Chenyang Lei, Xuhua Huang, Mengdi Zhang, Qiong Yan, Wenxiu Sun, and Qifeng Chen. 2020. Polarized reflection removal with perfect alignment in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1750--1758.
[19]
Anat Levin and Yair Weiss. 2007. User assisted separation of reflections from a single image using a sparsity prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 29, 9 (2007), 1647--1654.
[20]
Anat Levin, Assaf Zomet, and Yair Weiss. 2002. Learning to perceive transparency from the statistics of natural scenes. Advances in Neural Information Processing Systems, Vol. 15 (2002).
[21]
Chao Li, Yixiao Yang, Kun He, Stephen Lin, and John E Hopcroft. 2020. Single image reflection removal through cascaded refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3565--3574.
[22]
Rui Li, Simeng Qiu, Guangming Zang, and Wolfgang Heidrich. 2020. Reflection separation via multi-bounce polarization state tracing. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XIII 16. Springer, 781--796.
[23]
Yu Li and Michael S Brown. 2013. Exploiting reflection change for automatic reflection removal. In Proceedings of the IEEE international conference on computer vision. 2432--2439.
[24]
Yu Li and Michael S Brown. 2014. Single image layer separation using relative smoothness. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2752--2759.
[25]
Yu Li, Ming Liu, Yaling Yi, Qince Li, Dongwei Ren, and Wangmeng Zuo. 2023. Two-stage single image reflection removal with reflection-aware guidance. Applied Intelligence, Vol. 53, 16 (2023), 19433--19448.
[26]
Yunfei Liu, Yu Li, Shaodi You, and Feng Lu. 2022. Semantic guided single image reflection removal. ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 18, 3s (2022), 1--23.
[27]
Yu-Lun Liu, Wei-Sheng Lai, Ming-Hsuan Yang, Yung-Yu Chuang, and Jia-Bin Huang. 2020. Learning to see through obstructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14215--14224.
[28]
Liying Lu, Wenbo Li, Xin Tao, Jiangbo Lu, and Jiaya Jia. 2021. Masa-sr: Matching acceleration and spatial adaptation for reference-based image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6368--6377.
[29]
Daiqian Ma, Renjie Wan, Boxin Shi, Alex C Kot, and Ling-Yu Duan. 2019. Learning to jointly generate and separate reflections. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2444--2452.
[30]
Roey Mechrez, Itamar Talmi, and Lihi Zelnik-Manor. 2018. The contextual loss for image transformation with non-aligned data. In Proceedings of the European conference on computer vision (ECCV). 768--783.
[31]
Ajay Nandoriya, Mohamed Elgharib, Changil Kim, Mohamed Hefeeda, and Wojciech Matusik. 2017. Video reflection removal through spatio-temporal optimization. In Proceedings of the IEEE International Conference on Computer Vision. 2411--2419.
[32]
Marco Pesavento, Marco Volino, and Adrian Hilton. 2021. Attention-based Multi-Reference Learning for Image Super-Resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 14697--14706.
[33]
BH Prasad, Green Rosh, Lokesh Boregowda, and Kaushik Mitra. 2023. Burst Reflection Removal Using Reflection Motion Aggregation Cues. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 239--248.
[34]
B H Pawan Prasad, Green Rosh K S, Lokesh R. Boregowda, Kaushik Mitra, and Sanjoy Chowdhury. 2021. V-DESIRR: Very Fast Deep Embedded Single Image Reflection Removal. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2390--2399.
[35]
BH Pawan Prasad, Green Rosh KS, RB Lokesh, and Kaushik Mitra. 2022. Reference Guided Reflection Removal Using Deep Visual Attribute Cues. In 2022 IEEE International Conference on Image Processing (ICIP). IEEE, 1146--1150.
[36]
Abhijith Punnappurath and Michael S Brown. 2019. Reflection removal using a dual-pixel sensor. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1556--1565.
[37]
Green Rosh, BH Pawan Prasad, Lokesh R Boregowda, and Kaushik Mitra. 2023. Deep Unsupervised Reflection Removal Using Diffusion Models. In 2023 IEEE International Conference on Image Processing (ICIP). IEEE, 2045--2049.
[38]
Green Rosh K S, Sachin Lomte, Nikhil Krishnan, and B H Pawan Prasad. 2021. Attention Guided Complementary Feature Integration for Latent Image Recovery from Noisy/Blurry Pairs. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3490035.3490302
[39]
Tushar Sandhan and Jin Young Choi. 2017. Anti-glare: Tightly constrained optimization for eyeglass reflection removal. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1241--1250.
[40]
Yoav Y Schechner, Nahum Kiryati, and Ronen Basri. 2000. Separation of transparent layers using focus. International Journal of Computer Vision, Vol. 39 (2000), 25--39.
[41]
YiChang Shih, Dilip Krishnan, Fredo Durand, and William T Freeman. 2015. Reflection removal using ghosting cues. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3193--3201.
[42]
YiChang Shih, Dilip Krishnan, Fredo Durand, and William T Freeman. 2015. Reflection removal using ghosting cues. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3193--3201.
[43]
Christian Simon and In Kyu Park. 2015. Reflection removal for in-vehicle black box videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4231--4239.
[44]
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
[45]
Renjie Wan, Boxin Shi, Ling-Yu Duan, Ah-Hwee Tan, Wen Gao, and Alex C Kot. 2018. Region-aware reflection removal with unified content and gradient priors. IEEE Transactions on Image Processing, Vol. 27, 6 (2018), 2927--2941.
[46]
Renjie Wan, Boxin Shi, Ling-Yu Duan, Ah-Hwee Tan, and Alex C. Kot. 2017. Benchmarking Single-Image Reflection Removal Algorithms. In International Conference on Computer Vision (ICCV).
[47]
Renjie Wan, Boxin Shi, Tan Ah Hwee, and Alex C Kot. 2016. Depth of field guided reflection removal. In 2016 IEEE International Conference on Image Processing (ICIP). IEEE, 21--25.
[48]
Renjie Wan, Boxin Shi, Haoliang Li, Ling-Yu Duan, Ah-Hwee Tan, and Alex C Kot. 2019. CoRRN: Cooperative reflection removal network. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 42, 12 (2019), 2969--2982.
[49]
Renjie Wan, Boxin Shi, Haoliang Li, Yucheng Hong, Ling-Yu Duan, and Alex C. Kot. 2022. Benchmarking Single-Image Reflection Removal Algorithms. In IEEE Transacations on Pattern Analysis and Machine Intelligence.
[50]
Renjie Wan, Boxin Shi, Ah-Hwee Tan, and Alex C Kot. 2017. Sparsity based reflection removal using external patch search. In 2017 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 1500--1505.
[51]
Kaixuan Wei, Jiaolong Yang, Ying Fu, David Wipf, and Hua Huang. 2019. Single image reflection removal exploiting misaligned training data and network enhancements. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8178--8187.
[52]
Qiang Wen, Yinjie Tan, Jing Qin, Wenxi Liu, Guoqiang Han, and Shengfeng He. 2019. Single image reflection removal beyond linearity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3771--3779.
[53]
Sijia Wen, Yinqiang Zheng, and Feng Lu. 2021. Polarization guided specular reflection separation. IEEE Transactions on Image Processing, Vol. 30 (2021), 7280--7291.
[54]
Qing Yan, Yi Xu, and Xiaokang Yang. 2013. Separation of weak reflection from a single superimposed image using gradient profile sharpness. In 2013 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, 937--940.
[55]
Xu Yan, Weibing Zhao, Kun Yuan, Ruimao Zhang, Zhen Li, and Shuguang Cui. 2020. Towards content-independent multi-reference super-resolution: Adaptive pattern matching and feature aggregation. In European Conference on Computer Vision. Springer, 52--68.
[56]
Jie Yang, Dong Gong, Lingqiao Liu, and Qinfeng Shi. 2018. Seeing deeply and bidirectionally: A deep learning approach for single image reflection removal. In Proceedings of the european conference on computer vision (ECCV). 654--669.
[57]
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, Vol. 2 (2014), 67--78.
[58]
Huaidong Zhang, Xuemiao Xu, Hai He, Shengfeng He, Guoqiang Han, Jing Qin, and Dapeng Wu. 2019. Fast user-guided single image reflection removal via edge-aware cascaded networks. IEEE Transactions on Multimedia, Vol. 22, 8 (2019), 2012--2023.
[59]
Xuaner Zhang, Ren Ng, and Qifeng Chen. 2018. Single image reflection separation with perceptual losses. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4786--4794.
[60]
Zhifei Zhang, Zhaowen Wang, Zhe Lin, and Hairong Qi. 2019. Image super-resolution by neural texture transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7982--7991.
[61]
Qian Zheng, Boxin Shi, Jinnan Chen, Xudong Jiang, Ling-Yu Duan, and Alex C Kot. 2021. Single image reflection removal with absorption effect. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13395--13404.
Index Terms
- R2SFD: Improving Single Image Reflection Removal using Semantic Feature Dictionary
Recommendations
Single Image Reflection Removal with Diffusion Model
ICCPR '23: Proceedings of the 2023 12th International Conference on Computing and Pattern RecognitionThe removal of undesirable reflections from workpiece surface images captured under industrial conditions is a challenging task in image enhancement for the industry. While existing reflection removal methods have shown promising results in natural ...
Semantic Guided Single Image Reflection Removal
Reflection is common when we see through a glass window, which not only is a visual disturbance but also influences the performance of computer vision algorithms. Removing the reflection from a single image, however, is highly ill-posed since the color at ...
A Model-Guided Unfolding Network for Single Image Reflection Removal
MMAsia '21: Proceedings of the 3rd ACM International Conference on Multimedia in AsiaRemoving undesirable reflections from a single image captured through a glass surface is of broad application to various image processing and computer vision tasks, but it is an ill-posed and challenging problem. Existing traditional single image ...
Comments
Information & Contributors
Information
Published In
October 2024
11719 pages
ISBN:9798400706868
DOI:10.1145/3664647
- General Chairs:
- Jianfei Cai,
- Mohan Kankanhalli,
- Balakrishnan Prabhakaran,
- Susanne Boll,
- Program Chairs:
- Ramanathan Subramanian,
- Liang Zheng,
- Vivek K. Singh,
- Pablo Cesar,
- Lexing Xie,
- Dong Xu
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 28 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
MM '24
Sponsor:
MM '24: The 32nd ACM International Conference on Multimedia
October 28 - November 1, 2024
Melbourne VIC, Australia
Acceptance Rates
MM '24 Paper Acceptance Rate 1,150 of 4,385 submissions, 26%;
Overall Acceptance Rate 2,145 of 8,556 submissions, 25%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 189Total Downloads
- Downloads (Last 12 months)189
- Downloads (Last 6 weeks)132
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in