Skip to main content
Springer Nature Link
Log in

A Mutual Enhancement Framework for Specular Highlight Detection and Removal

  • Conference paper
  • First Online:
Pattern Recognition and Computer Vision (PRCV 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14435))

Included in the following conference series:

Abstract

Specular highlights, generated by direct light reflection from surfaces, can significantly reduce the image quality and impair various computer vision applications. Recently, the existing approaches for jointly detecting and removing specular highlights attempted to use the detection as guidance for highlight removal. However, they ignored that this kind of unidirectional enhancement was susceptible to detection tasks. To achieve mutual enhancement, we assume that discriminative features would benefit the highlight detection task which needs to distinguish between highlight areas and highlight-free areas, while coherent features would facilitate the learning of highlight removal since it requires converting highlight areas to highlight-free areas. Specifically, we propose a mutual enhancement framework (MEF-SHDR) that addresses both specular highlight detection and removal in a unified manner. The proposed framework designs a Feature Decomposition and Aggregation Module (FDAM) that separates highlight and highlight-free features explicitly and aggregates them for improved detection and removal performance. Comprehensive experiments are implemented on five widely used datasets, i.e., SHIQ, LIME, SD1, SD2, and RD, demonstrating the superiority of the proposed approach over previous state-of-the-art methods, as well as the effectiveness of jointly detecting and removing highlights. Code is available at https://github.com/drafly/MEF-SHDR.

Supported by the Institute of Artificial Intelligence, Hefei Comprehensive National Science Center Project under Grant (21KT008), the University Synergy Innovation Program of Anhui Province (GXXT-2022-052), and National Natural Science Foundation of China (No. 62202015).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Akashi, Y., Okatani, T.: Separation of reflection components by sparse non-negative matrix factorization. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9007, pp. 611–625. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16814-2_40

    Chapter  Google Scholar 

  2. Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 898–916 (2010)

    Article  Google Scholar 

  3. Fu, G., Zhang, Q., Lin, Q., Zhu, L., Xiao, C.: Learning to detect specular highlights from real-world images. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 1873–1881 (2020)

    Google Scholar 

  4. Fu, G., Zhang, Q., Zhu, L., Li, P., Xiao, C.: A multi-task network for joint specular highlight detection and removal. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7752–7761 (2021)

    Google Scholar 

  5. Gao, J., Zhang, T., Xu, C.: Graph convolutional tracking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4649–4659 (2019)

    Google Scholar 

  6. Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2414–2423 (2016)

    Google Scholar 

  7. Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems 27 (2014)

    Google Scholar 

  8. Guo, J., Zhou, Z., Wang, L.: Single image highlight removal with a sparse and low-rank reflection model. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11208, pp. 282–298. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01225-0_17

    Chapter  Google Scholar 

  9. Hou, S., Wang, C., Quan, W., Jiang, J., Yan, D.-M.: Text-aware single image specular highlight removal. In: Ma, H., et al. (eds.) PRCV 2021. LNCS, vol. 13022, pp. 115–127. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-88013-2_10

    Chapter  Google Scholar 

  10. Hu, G., Zheng, Y., Yan, H., Hua, G., Yan, Y.: Mask-guided cycle-GAN for specular highlight removal. Pattern Recogn. Lett. 161, 108–114 (2022)

    Article  Google Scholar 

  11. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7132–7141 (2018)

    Google Scholar 

  12. Hu, X., Zhu, L., Fu, C.W., Qin, J., Heng, P.A.: Direction-aware spatial context features for shadow detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7454–7462 (2018)

    Google Scholar 

  13. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43

    Chapter  Google Scholar 

  14. Kim, S.-W., Kook, H.-K., Sun, J.-Y., Kang, M.-C., Ko, S.-J.: Parallel feature pyramid network for object detection. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11209, pp. 239–256. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01228-1_15

    Chapter  Google Scholar 

  15. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  16. Li, R., Pan, J., Si, Y., Yan, B., Hu, Y., Qin, H.: Specular reflections removal for endoscopic image sequences with adaptive-RPCA decomposition. IEEE Trans. Med. Imaging 39(2), 328–340 (2019)

    Article  Google Scholar 

  17. Lin, J., El Amine Seddik, M., Tamaazousti, M., Tamaazousti, Y., Bartoli, A.: Deep multi-class adversarial specularity removal. In: Felsberg, M., Forssén, P.-E., Sintorn, I.-M., Unger, J. (eds.) SCIA 2019. LNCS, vol. 11482, pp. 3–15. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20205-7_1

    Chapter  Google Scholar 

  18. Meka, A., et al.: LIME: live intrinsic material estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6315–6324 (2018)

    Google Scholar 

  19. Muhammad, S., Dailey, M.N., Farooq, M., Majeed, M.F., Ekpanyapong, M.: Spec-net and spec-CGAN: deep learning models for specularity removal from faces. Image Vis. Comput. 93, 103823 (2020)

    Article  Google Scholar 

  20. Shafer, S.A.: Using color to separate reflection components. Color. Res. Appl. 10(4), 210–218 (1985)

    Article  Google Scholar 

  21. Shi, J., Dong, Y., Su, H., Yu, S.X.: Learning non-lambertian object intrinsics across shapenet categories. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1685–1694 (2017)

    Google Scholar 

  22. Tan, M., Le, Q.: EfficientNet: rethinking model scaling for convolutional neural networks. In: Proceedings of the 36th International Conference on Machine Learning, pp. 6105–6114. PMLR (2019)

    Google Scholar 

  23. Wang, C., Wu, Z., Guo, J., Zhang, X.: Contour-constrained specular highlight detection from real-world images. In: Proceedings of the 18th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry, pp. 1–4 (2022)

    Google Scholar 

  24. Wu, Z., Guo, J., Zhuang, C., Xiao, J., Yan, D.M., Zhang, X.: Joint specular highlight detection and removal in single images via unet-transformer. Comput. Vis.Media 9(1), 141–154 (2023)

    Article  Google Scholar 

  25. Wu, Z.: Single-image specular highlight removal via real-world dataset construction. IEEE Trans. Multimedia 24, 3782–3793 (2021)

    Article  Google Scholar 

  26. Xu, J., Liu, S., Chen, G., Liu, Q.: Highlight detection and removal method based on bifurcated-CNN. In: Liu, H., et al. (eds.) Intelligent Robotics and Applications, pp. 307–318. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-13841-6_29

  27. Yamamoto, T., Kitajima, T., Kawauchi, R.: Efficient improvement method for separation of reflection components based on an energy function. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 4222–4226. IEEE (2017)

    Google Scholar 

  28. Yang, Q., Tang, J., Ahuja, N.: Efficient and robust specular highlight removal. IEEE Trans. Pattern Anal. Mach. Intell. 37(6), 1304–1311 (2014)

    Article  Google Scholar 

  29. Yi, R., Tan, P., Lin, S.: Leveraging multi-view image sets for unsupervised intrinsic image decomposition and highlight separation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 12685–12692 (2020)

    Google Scholar 

  30. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Free-form image inpainting with gated convolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4471–4480 (2019)

    Google Scholar 

  31. Zhang, L., Long, C., Zhang, X., Xiao, C.: RIS-GAN: explore residual and illumination with generative adversarial networks for shadow removal. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 12829–12836 (2020)

    Google Scholar 

  32. Zhang, L., Long, C., Zhang, X., Xiao, C.: Exploiting residual and illumination with gans for shadow detection and shadow removal. ACM Trans. Multimedia Comput. Commun. Appl. 19, 1–22 (2022)

    Google Scholar 

  33. Zhang, W., Zhao, X., Morvan, J.M., Chen, L.: Improving shadow suppression for illumination robust face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 41(3), 611–624 (2018)

    Article  Google Scholar 

  34. Zhu, T., Xia, S., Bian, Z., Lu, C.: Highlight removal in facial images. In: Peng, Y., et al. (eds.) PRCV 2020. LNCS, vol. 12305, pp. 422–433. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60633-6_35

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. AHU-IAI AI Joint Laboratory, Anhui University, Hefei, China

    Ge Huang

  2. Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, China

    Ge Huang

  3. School of Automation, Northwestern Polytechnical University, Xi’an, China

    Jieru Yao & Peiliang Huang

  4. School of Computer Science and Engineering, Beijing Technology and Business University, Beijing, China

    Longfei Han

  5. School of Information Science and Technology, University of Science and Technology of China, Hefei, China

    Longfei Han

Authors
  1. Ge Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jieru Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Peiliang Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Longfei Han
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Nanjing University of Information Science and Technology, Nanjing, China

    Qingshan Liu

  2. Xiamen University, Xiamen, China

    Hanzi Wang

  3. Beijing University of Posts and Telecommunications, Beijing, China

    Zhanyu Ma

  4. Sun Yat-sen University, Guangzhou, China

    Weishi Zheng

  5. Peking University, Beijing, China

    Hongbin Zha

  6. Chinese Academy of Sciences, Beijing, China

    Xilin Chen

  7. Chinese Academy of Sciences, Beijing, China

    Liang Wang

  8. Xiamen University, Xiamen, China

    Rongrong Ji

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Huang, G., Yao, J., Huang, P., Han, L. (2024). A Mutual Enhancement Framework for Specular Highlight Detection and Removal. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14435. Springer, Singapore. https://doi.org/10.1007/978-981-99-8552-4_36

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-8552-4_36

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-8551-7

  • Online ISBN: 978-981-99-8552-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics