Skip to main content
SpringerLink
Log in

Towards High-Resolution Specular Highlight Detection

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Specular highlight detection is an essential task with various applications in computer vision. This paper aims to detect specular highlights in single high-resolution images using deep learning while avoiding excessive GPU memory consumption. To achieve this, we present a high-resolution specular highlight detection dataset with manual annotations of specular highlights. Given our dataset, we propose a patch-level bidirectional refinement network for high-resolution specular highlight detection. The main idea is to utilize both the pathway from small-scale patch to large-scale patch and its reverse pathway to progressively refine the detection results of adjacent-scale specular highlight patches. Moreover, based on our detection network, we propose a modified inpainting framework for specular highlight removal as an application. Lastly, we provide ten potential research directions for specular highlight detection, inspiring researchers for further study.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

Notes

  1. https://www.flickr.com.

  2. https://www.pinterest.com.

  3. https://images.google.com.

  4. https://www.bing.com/images.

  5. We use the same term specular residual as in (Shi et al., 2017) to represent the remaining component obtained by subtracting the diffuse reflection component from an observed image.

References

  • Akashi, Y., & Okatani, T. (2015). Separation of reflection components by sparse non-negative matrix factorization. Computer Vision and Image Understanding, 100(146), 77–85.

    Google Scholar 

  • Angelopoulou, E. (2007) Specular highlight detection based on the fresnel reflection coefficient, In Proceedings of the IEEE International Conference on Computer Vision (pp. 1–8).

  • Bajcsy, R., Lee, S. W., & Leonardis, A. (1990). Color image segmentation with detection of highlights and local illumination induced by inter-reflections. In Proceedings of the IEEE International Conference on Pattern Recognition (pp. 785–790).

  • Barron, J. T. & Tsai, Y.-T. (2017) Fast Fourier color constancy. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 886–894)

  • Bell, S., Bala, K., & Snavely, N. (2014). Intrinsic images in the wild. Transactions on Graphics, 33(4), 159.

    Article  Google Scholar 

  • Borji, A., Cheng, M.-M., Jiang, H., & Li, J. (2015). Salient object detection: A benchmark. IEEE Transactions on Image Processing, 24(12), 5706–5722.

    Article  MathSciNet  Google Scholar 

  • Brelstaff, G., & Blake, A. (1988). Detecting specular reflections using Lambertian constraints. In Proceedings of the IEEE International Conference on Computer Vision (pp. 297–302).

  • Chen, W., Jiang, Z., Wang, Z., Cui, K., & Qian, X. (2019). Collaborative global-local networks for memory-efficient segmentation of ultra-high resolution images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 8924–8933).

  • Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F. & Adam, H. (2018a). Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European Conference on Computer Vision (pp. 801–818).

  • Chen, L., Zhu, Y., Papandreou, G., Schroff, F. & Adam, H. (2018b). Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European Conference on Computer Vision (pp. 833–851).

  • Cheng, H. K., Chung, J., Tai, Y.-W., & Tang, C.-K. (2020). CascadePSP: Toward class-agnostic and very high-resolution segmentation via global and local refinement. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 8890–8899).

  • Cheng, M.-M., Mitra, N. J., Huang, X., Torr, P. H., & Hu, S.-M. (2014). Global contrast based salient region detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(3), 569–582.

    Article  Google Scholar 

  • Croitoru, I., Bogolin, S.-V., & Leordeanu, M. (2019). Unsupervised learning of foreground object segmentation. International Journal of Computer Vision, 127, 1279–1302.

    Article  Google Scholar 

  • Fan, D.-P., Cheng, M.-M., Liu, Y., Li, T., & Borji, A. (2017). Structure-measure: A new way to evaluate foreground maps, In Proceedings of the IEEE International Conference on Computer Vision (pp. 4548–4557).

  • Fan, D.-P., et al. (2018). Enhanced-alignment measure for binary foreground map evaluation. arXiv preprint arXiv:1805.10421

  • Fu, G., Zhang, Q., Lin, Q., Zhu, L., & Xiao, C. (2020). Learning to detect spec1ular highlights from real-world images. In Proceedings of the ACM International Conference on Multimedia (pp. 1873–1881).

  • Fu, G., Zhang, Q., Zhu, L., Li, P., & Xiao, C. (2021). A multi-task network for joint specular highlight detection and removal. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 7752–7761).

  • Grosse, R., Johnson, M. K., Adelson, E. H., & Freeman, W. T. (2009). Ground truth dataset and baseline evaluations for intrinsic image algorithms. In Proceedings of the IEEE International Conference on Computer Vision (pp. 2335–2342).

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 770–778).

  • Hou, Q., Cheng, M.-M., Hu, X., Borji, A., Tu, Z., & Torr, P. (2019). Deeply supervised salient object detection with short connections. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(4), 815–828.

    Article  Google Scholar 

  • Hoyer, P. O. (2004). Non-negative matrix factorization with sparseness constraints. Journal of Machine Learning Research, 5(9), 1457–1469.

    MathSciNet  Google Scholar 

  • Hu, J., Shen, L., & Sun, G. (2018). Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 7132–7141).

  • Hu, X., Zhu, L., Fu, C.-W., Qin, J., & Heng, P.-A. (2018). Direction-aware spatial context features for shadow detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 7454–7462).

  • Hu, X., Fu, C.-W., Zhu, L., Qin, J., & Heng, P.-A. (2019). Direction-aware spatial context features for shadow detection and removal. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(11), 2795–2808.

    Article  Google Scholar 

  • Huynh, C., Tran, A. T., Luu, K., & Hoai, M. (2021). Progressive semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 16755–16764).

  • Kim, H., Jin, H., Hadap, S., & Kweon, I. (2013). Specular reflection separation using dark channel prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1460–1467).

  • Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

  • Lang, C., Feng, J., Feng, S., Wang, J., & Yan, S. (2016). Dual low-rank pursuit: Learning salient features for saliency detection. IEEE Transactions on Neural Networks and Learning Systems, 27(6), 1190–1200.

    Article  MathSciNet  Google Scholar 

  • Li, Z., & Snavely, N. (2018). Learning intrinsic image decomposition from watching the world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 9039–9048).

  • Li, R., et al. (2019). Specular reflections removal for endoscopic image sequences with adaptive-rpca decomposition. IEEE Transactions on Medical Imaging, 39(2), 328–340.

    Article  Google Scholar 

  • Lin, T.-Y., et al. (2017). Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2117–2125).

  • Lin, P., et al. (2020). Graph-guided architecture search for real-time semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 4203–4212).

  • Liu, Y., Li, Y., You, S., & Lu, F. (2020). Unsupervised learning for intrinsic image decomposition from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3248–3257).

  • Liu, G., et al. (2018). Image inpainting for irregular holes using partial convolutions. In Proceedings of the European Conference on Computer Vision (pp. 85–100).

  • Murmann, L., Gharbi, M., Aittala, M., & Durand, F. (2019). A dataset of multi-illumination images in the wild. In Proceedings of the IEEE International Conference on Computer Vision (pp. 4080–4089).

  • Netz, A., & Osadchy, M. (2012). Recognition using specular highlights. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(3), 639–652.

    Article  Google Scholar 

  • Osadchy, M., Jacobs, D. W., & Ramamoorthi, R. (2003). Using specularities for recognition. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1512–1519).

  • Pan, X., Shi, J., Luo, P., Wang, X., & Tang, X. (2018). Spatial as deep: Spatial CNN for traffic scene understanding. In Proceedings of the AAAI Conference on Artificial Intelligence (pp. 7276–7283).

  • Park, J. B., & Kak, A. C. (2003). A truncated least squares approach to the detection of specular highlights in color images. In Proceedings of the IEEE International Conference on Robotics and Automation (pp. 1397–1403).

  • Phong, B. T. (1975). Illumination for computer generated pictures. Communications of the ACM, 18(6), 311–317.

    Article  Google Scholar 

  • Qin, X., et al. (2019). BASNet: Boundary-aware salient object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 7479–7489).

  • Shafer, S. A. (1985). Using color to separate reflection components. Color Research and Application, 10(4), 210–218.

    Article  Google Scholar 

  • Shen, L., Tan, P., & Lin, S. (2008). Intrinsic image decomposition with non-local texture cues. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1–7).

  • Shen, J., Yang, X., Jia, Y. & Li, X., (2011). Intrinsic images using optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3481–3487).

  • Shen, H.-L., & Zheng, Z.-H. (2013). Real-time highlight removal using intensity ratio. Applied Optics, 52(19), 4483–4493.

    Article  Google Scholar 

  • Shi, J., Dong, Y., Su, H., & Yu, S. X. (2017). Learning non-Lambertian object intrinsics across shapenet categories. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1685–1290).

  • Sun, Z., Cao, S., Yang, Y., & Kitani, K. (2020). Rethinking Transformer-based set prediction for object detection. arXiv preprint arXiv:2011.10881

  • Tan, R. T., & Ikeuchi, K. (2005). Separating reflection components of textured surfaces using a single image. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(2), 178–193.

    Article  Google Scholar 

  • Tian, Q., & Clark, J. J. (2013). Real-time specularity detection using unnormalized wiener entropy. In Proceedings of the IEEE International Conference on Computer and Robot Vision (pp. 356–363).

  • Van Gansbeke, W., Vandenhende, S., Georgoulis, S., & Van Gool, L. (2021). Unsupervised semantic segmentation by contrasting object mask proposals. In Proceedings of the IEEE International Conference on Computer Vision (pp. 10052–10062).

  • Wang, T., Hu, X., Wang, Q., Heng, P.-A., & Fu, C.-W. (2020). Instance shadow detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1880–1889).

  • Wang, Y., Zhang, W., Wang, L., Liu, T., & Lu, H. (2022). Multi-source uncertainty mining for deep unsupervised saliency detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 11727–11736).

  • Wang, R., et al. (2019). Underexposed photo enhancement using deep illumination estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 6849–6857).

  • Wu, T., et al. (2020). Patch proposal network for fast semantic segmentation of high-resolution images. In Proceedings of the AAAI Conference on Artificial Intelligence (pp. 12402–12409).

  • Wu, Z., et al. (2021). Single-image specular highlight removal via real-world dataset construction. IEEE Transactions on Multimedia, 24, 3782–3793.

    Article  Google Scholar 

  • Xie, C., et al. (2022). Pyramid grafting network for one-stage high resolution saliency detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 11717–11726).

  • Yang, Y., & Soatto, S. (2020). FDA: Fourier domain adaptation for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 4085–4095).

  • Yang, Q., Tang, J., & Ahuja, N. (2015). Efficient and robust specular highlight removal. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(6), 1304–1311.

    Article  Google Scholar 

  • Zeng, Y., Zhang, P., Zhang, J., Lin, Z., & Lu, H. (2019). Towards high-resolution salient object detection. In Proceedings of the IEEE International Conference on Computer Vision (pp. 7234–7243).

  • Zhang, P., Liu, W., Zeng, Y., Lei, Y., & Lu, H. (2021). Looking for the detail and context devils: High-resolution salient object detection. IEEE Transactions on Image Processing, 30, 3204-3216.

    Article  Google Scholar 

  • Zhang, L., Yan, Q., Liu, Z., Zou, H., & Xiao, C. (2017). Illumination decomposition for photograph with multiple light sources. IEEE Transactions on Image Processing, 26(9), 4114–4127.

  • Zhang, W., Zhao, X., Morvan, J.-M., & Chen, L. (2018). Improving shadow suppression for illumination robust face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(3), 611–624.

    Article  Google Scholar 

  • Zheng, Q., Qiao, X., Cao, Y., & Lau, R. W. (2019). Distraction-aware shadow detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 5167–5176).

  • Zhou, P., Price, B., Cohen, S., Wilensky, G., & Davis, L. S. (2020). DeepStrip: High-resolution boundary refinement. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 10558–1350).

  • Zhou, H., Xie, X., Lai, J.-H., Chen, Z., & Yang, L. (2020). Interactive two-stream decoder for accurate and fast saliency detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 9141–9150).

  • Zhu, L., et al. (2018). Bidirectional feature pyramid network with recurrent attention residual modules for shadow detection. In Proceedings of the European Conference on Computer Vision (pp. 122–137).

Download references

Acknowledgements

This work is partially supported by the National Natural Science Foundation of China under Grant (No. 61972298), CAAI-Huawei MindSpore Open Fund, and the Research Program for Young and Middle-Aged Teachers of Fujian Province under Grant (No. JAT210036).

Author information

Authors and Affiliations

  1. School of Computer Science, Wuhan University, Wuhan, 430072, Hubei, China

    Gang Fu, Yihao Wang, Siyuan Fan & Chunxia Xiao

  2. School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, 510006, Guangdong, China

    Qing Zhang

  3. ROAS Thrust, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, 511400, Guangdong, China

    Lei Zhu

  4. Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China

    Lei Zhu

  5. College of Computer and Data Science, Fuzhou University, Fuzhou, 350116, Fujian, China

    Qifeng Lin

Authors
  1. Gang Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qing Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lei Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qifeng Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yihao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Siyuan Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Chunxia Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chunxia Xiao.

Additional information

Communicated by Shaodi You.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fu, G., Zhang, Q., Zhu, L. et al. Towards High-Resolution Specular Highlight Detection. Int J Comput Vis 132, 95–117 (2024). https://doi.org/10.1007/s11263-023-01845-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-023-01845-3

Keywords

Search

Navigation