skip to main content
10.1145/3664647.3681434acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

HighlightRemover: Spatially Valid Pixel Learning for Image Specular Highlight Removal

Published: 28 October 2024 Publication History

Abstract

Recently, learning-based methods have made significant progress for image specular highlight removal. However, many of these approaches treat all the image pixels uniformly, overlooking the negative impact of invalid pixels on feature reconstruction. This oversight often leads to undesirable outcomes, such as color distortion or residual highlights. In this paper, we propose a novel image specular highlight removal network called HighlightRNet, which utilizes valid pixels as references to reconstruct the highlight-free image. To achieve this, we introduce a context-aware fusion block (CFBlock) that aggregates information in four directions, effectively capturing global contextual information. Additionally, we introduce a location-aware feature transformation module (LFTModule) to adaptively learn the valid pixels for feature reconstruction, thereby avoiding information errors caused by invalid pixels. With these modules, our method can produce high-quality highlight-free results without color distortion and highlight residual. Furthermore, we develop a multiple light image-capturing system to construct a large-scale highlight dataset called NSH, which exhibits minimal misalignment in image pairs and minimal brightness variation in non-highlight regions. Experimental results on various datasets demonstrate the superiority of our method over state-of-the-art methods, both qualitatively and quantitatively.

Supplemental Material

MP4 File - HighlightRemover: Spatially Valid Pixel Learning for Image Specular Highlight Removal
In this paper, we propose a network called HighlightRNet to remove specular highlights in the image and restore a clear image. By utilizing valid pixels in the image, our HighlightRNet can recover a high-quality highlight removal results without color distortion and highlight residual. Specifically, we introduce a context-aware fusion block (CFBlock) in the bottleneck module to capture global contextual information. Additionally, we introduce a location-aware feature transformation module (LFTModule) to adaptively learn the valid pixels for feature reconstruction and reconstruct image features. With these modules, our method can produce high-quality highlight-free results without color distortion and highlight residual. Furthermore, we construct a large-scale highlight dataset called NSH, which exhibits minimal misalignment in image pairs and minimal brightness variation in non-highlight regions.

References

[1]
Yasuhiro Akashi and Takayuki Okatani. 2014. Separation of reflection components by sparse non-negative matrix factorization. In Asian Conference on Computer Vision. Springer, 611--625.
[2]
Max Born and Emil Wolf. 2013. Principles of optics: electromagnetic theory of propagation, interference and diffraction of light. Elsevier.
[3]
Daniele Di Mauro, Antonino Furnari, Giuseppe Patanè, Sebastiano Battiato, and Giovanni Maria Farinella. 2020. SceneAdapt: Scene-based domain adaptation for semantic segmentation using adversarial learning. Pattern Recognition Letters, Vol. 136 (2020), 175--182.
[4]
Gang Fu, Qing Zhang, Qifeng Lin, Lei Zhu, and Chunxia Xiao. 2020. Learning to Detect Specular Highlights from Real-world Images. In ACM International Conference on Multimedia.
[5]
Gang Fu, Qing Zhang, and Chunxia Xiao. 2024. Towards High-Resolution Specular Highlight Detection. International Journal of Computer Vision, Vol. 132, 1 (2024), 95--117.
[6]
Gang Fu, Qing Zhang, Lei Zhu, Ping Li, and Chunxia Xiao. 2021. A multi-task network for joint specular highlight detection and removal. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7752--7761.
[7]
Gang Fu, Qing Zhang, Lei Zhu, Chunxia Xiao, and Ping Li. 2023. Towards High-Quality Specular Highlight Removal by Leveraging Large-Scale Synthetic Data. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 12857--12865.
[8]
Junyu Gao, Tianzhu Zhang, and Changsheng Xu. 2019. Graph convolutional tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4649--4659.
[9]
Alex Graves, Navdeep Jaitly, and Abdel-rahman Mohamed. 2013. Hybrid speech recognition with deep bidirectional LSTM. In 2013 IEEE workshop on automatic speech recognition and understanding. IEEE, 273--278.
[10]
Xiaojie Guo, Xiaochun Cao, and Yi Ma. 2014. Robust separation of reflection from multiple images. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2187--2194.
[11]
Guangwei Hu, Yuanfeng Zheng, Haoran Yan, Guang Hua, and Yuchen Yan. 2022. Mask-guided cycle-GAN for specular highlight removal. Pattern Recognition Letters, Vol. 161 (2022), 108--114.
[12]
Zhaoyangfan Huang, Kun Hu, and Xingjun Wang. 2022. M2-Net: multi-stages specular highlight detection and removal in multi-scenes. arXiv preprint arXiv:2207.09965 (2022).
[13]
Alexia Jolicoeur-Martineau. 2018. The relativistic discriminator: a key element missing from standard GAN. arXiv preprint arXiv:1807.00734 (2018).
[14]
Hyeongwoo Kim, Hailin Jin, Sunil Hadap, and Inso Kweon. 2013. Specular reflection separation using dark channel prior. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1460--1467.
[15]
Seung-Wook Kim, Hyong-Keun Kook, Jee-Young Sun, Mun-Cheon Kang, and Sung-Jea Ko. 2018. Parallel feature pyramid network for object detection. In Proceedings of the European Conference on Computer Vision (ECCV). 234--250.
[16]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[17]
Gudrun J Klinker, Steven A Shafer, and Takeo Kanade. 1988. The measurement of highlights in color images. International Journal of Computer Vision, Vol. 2, 1 (1988), 7--32.
[18]
John Lin, Mohamed El Amine Seddik, Mohamed Tamaazousti, Youssef Tamaazousti, and Adrien Bartoli. 2019. Deep multi-class adversarial specularity removal. In Scandinavian Conference on Image Analysis. Springer, 3--15.
[19]
Stephen Lin, Yuanzhen Li, Sing Bing Kang, Xin Tong, and Heung-Yeung Shum. 2002. Diffuse-specular separation and depth recovery from image sequences. In European conference on computer vision. Springer, 210--224.
[20]
Siraj Muhammad, Matthew N Dailey, Muhammad Farooq, Muhammad F Majeed, and Mongkol Ekpanyapong. 2020. Spec-Net and Spec-CGAN: Deep learning models for specularity removal from faces. Image and Vision Computing, Vol. 93 (2020), 103823.
[21]
Shree K Nayar, Xi-Sheng Fang, and Terrance Boult. 1997. Separation of reflection components using color and polarization. International Journal of Computer Vision, Vol. 21, 3 (1997), 163--186.
[22]
S Shafer. 1992. Using Color to Separate Reflection Components. Jones and Bartlett Publishers, Inc. (1992).
[23]
Steven A Shafer. 1985. Using color to separate reflection components. Color Research & Application, Vol. 10, 4 (1985), 210--218.
[24]
Hui-Liang Shen, Hong-Gang Zhang, Si-Jie Shao, and John H Xin. 2008. Chromaticity-based separation of reflection components in a single image. Pattern Recognition, Vol. 41, 8 (2008), 2461--2469.
[25]
Hui-Liang Shen and Zhi-Huan Zheng. 2013. Real-time highlight removal using intensity ratio. Applied optics, Vol. 52, 19 (2013), 4483--4493.
[26]
Jinli Suo, Dongsheng An, Xiangyang Ji, Haoqian Wang, and Qionghai Dai. 2016. Fast and high quality highlight removal from a single image. IEEE Transactions on Image Processing, Vol. 25, 11 (2016), 5441--5454.
[27]
Laurent Valentin Jospin, Gilles Baechler, and Adam Scholefield. 2018. Embedded polarizing filters to separate diffuse and specular reflection. arXiv e-prints (2018), arXiv--1811.
[28]
Xing Wei, Xiaobin Xu, Jiawei Zhang, and Yihong Gong. 2018. Specular highlight reduction with known surface geometry. Computer Vision and Image Understanding, Vol. 168 (2018), 132--144.
[29]
Zhongqi Wu, Chuanqing Zhuang, Jian Shi, Jianwei Guo, Jun Xiao, Xiaopeng Zhang, and Dong-Ming Yan. 2021. Single-image specular highlight removal via real-world dataset construction. IEEE Transactions on Multimedia, Vol. 24 (2021), 3782--3793.
[30]
Takahisa Yamamoto and Atsushi Nakazawa. 2019. General improvement method of specular component separation using high-emphasis filter and similarity function. ITE Transactions on Media Technology and Applications, Vol. 7, 2 (2019), 92--102.
[31]
Jianwei Yang, Lixing Liu, and Stan Li. 2013. Separating specular and diffuse reflection components in the HSI color space. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 891--898.
[32]
Qingxiong Yang, Jinhui Tang, and Narendra Ahuja. 2014. Efficient and robust specular highlight removal. IEEE transactions on pattern analysis and machine intelligence, Vol. 37, 6 (2014), 1304--1311.
[33]
Qingxiong Yang, Shengnan Wang, and Narendra Ahuja. 2010. Real-time specular highlight removal using bilateral filtering. In European conference on computer vision. Springer, 87--100.
[34]
Ling Zhang, Chengjiang Long, Xiaolong Zhang, and Chunxia Xiao. 2023. Exploiting Residual and Illumination with GANs for Shadow Detection and Shadow Removal. ACM transactions on multimedia computing communications and applications, Vol. 19, 3 (2023), 120.1--120.22.
[35]
Wuming Zhang, Xi Zhao, Jean-Marie Morvan, and Liming Chen. 2018. Improving shadow suppression for illumination robust face recognition. IEEE transactions on pattern analysis and machine intelligence, Vol. 41, 3 (2018), 611--624.

Index Terms

  1. HighlightRemover: Spatially Valid Pixel Learning for Image Specular Highlight Removal
              Index terms have been assigned to the content through auto-classification.

              Recommendations

              Comments

              Information & Contributors

              Information

              Published In

              cover image ACM Conferences
              MM '24: Proceedings of the 32nd ACM International Conference on Multimedia
              October 2024
              11719 pages
              ISBN:9798400706868
              DOI:10.1145/3664647
              Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

              Sponsors

              Publisher

              Association for Computing Machinery

              New York, NY, United States

              Publication History

              Published: 28 October 2024

              Permissions

              Request permissions for this article.

              Check for updates

              Author Tags

              1. contextual information
              2. image specular highlight removal
              3. valid pixels

              Qualifiers

              • Research-article

              Conference

              MM '24
              Sponsor:
              MM '24: The 32nd ACM International Conference on Multimedia
              October 28 - November 1, 2024
              Melbourne VIC, Australia

              Acceptance Rates

              MM '24 Paper Acceptance Rate 1,150 of 4,385 submissions, 26%;
              Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

              Contributors

              Other Metrics

              Bibliometrics & Citations

              Bibliometrics

              Article Metrics

              • 0
                Total Citations
              • 63
                Total Downloads
              • Downloads (Last 12 months)63
              • Downloads (Last 6 weeks)17
              Reflects downloads up to 13 Feb 2025

              Other Metrics

              Citations

              View Options

              Login options

              View options

              PDF

              View or Download as a PDF file.

              PDF

              eReader

              View online with eReader.

              eReader

              Figures

              Tables

              Media

              Share

              Share

              Share this Publication link

              Share on social media