Skip to main content
SpringerLink
Log in

Specular highlight removal using a divide-and-conquer multi-resolution deep network

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The highlights in the digital images, which are from specular reflections on the surface of the object, are inevitable in the real world. It leads to erroneous results in many image processing algorithms. A specular highlight removal model based on a divide-and-conquer multi-resolution deep network is proposed according to the relation between the image patch and its corresponding maximum diffuse chromaticity. We use a Laplacian pyramid to decompose the image patch into two levels, including the same-sized patch with high- frequency components and the low-resolution blurred patch. The former exploits image textures and preserves local structures, and the latter preserves local intensity. Then we design different sub-networks to extract features for the two levels in a divide-and-conquer way and then fuse the features to achieve each pixel’s maximum diffuse chromaticity. Unlike the state-of-the-art methods where the model is trained in an image space, the proposed model for highlight removal based on the dichromatic reflection model is trained in a patch space. Experimental results demonstrate the proposed model is superior or competitive to the existing methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data Availability

The data supporting the findings of this study are available from the corresponding author on request.

Notes

  1. We thank the author of [37] for providing source codes for literatures. https://github.com/vitorsr/SIHR

References

  1. Abadi M, Agarwal A, Barham P et al (2016) Tensorflow: Largescale machine learning on heterogeneous distributed systems. https://arxiv.org/abs/1603.04467. Accessed 28 May 2021

  2. Akashi Y, Okatani T (2016) Separation of reflection components by sparse nonnegative matrix factorization. Comput Vis Image Underst 146:77–85

    Article  Google Scholar 

  3. Bajcsy S W, Lee R, Leonardis A (1996) Detection of diffuse and specular interface reflections and inter-reflections by color image segmentation. Int J Comput Vis 17:241–272

    Article  Google Scholar 

  4. Chen F, Ma J (2009) An empirical identification method of gaussian blur parameter for image deblurring. IEEE Trans Signal Process 57:2467–2478

    Article  MathSciNet  MATH  Google Scholar 

  5. Criminisi A, Kang S B, Swaminathan R, Szeliski R, Anandan P (2005) Extracting layers and analyzing their specular properties using epipolarplane-image analysis. Comput Vis Image Underst 97:51–85

    Article  Google Scholar 

  6. Ephraim Y, Malah D (1984) Speech enhancement using a minimum mean square error log-spectral amplitude estimator. IEEE Trans Acoust Speech Signal Process 32:1109–1121

    Article  Google Scholar 

  7. Fu X, Huang J, Ding X, Liao Y, Paisley J (2017) Clearing the skies: a deep network architecture for single-image rain removal. IEEE Trans Image Process 26:2944–2956

    Article  MathSciNet  MATH  Google Scholar 

  8. Fu X, Liang B, Huang Y, Ding X, Paisley J (2020) Lightweight pyramid networks for image deraining. IEEE Trans Neural Netw Learn Syst 31:1794–1807

    Article  Google Scholar 

  9. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proc. 2016 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, pp 770–778

  10. Huang D A, Kang L W, Wang Y C F, Lin C W (2013) Self-learning based image decomposition with applications to single image denoising. IEEE Trans Multimedia 16:83–93

    Article  Google Scholar 

  11. Karpathy A, Toderici G, Shetty S, Leung T, Sukthankar R, Fei-Fei L (2014) Large-scale video classification with convolutional neural networks. In: Proc. 2014 IEEE Conference on computer vision and pattern recognition. IEEE, Los Alamitos, pp 1725–1732

  12. Keerthi S, Lin C (2003) Asymptotic behaviors of support vector machines with gaussian kernel. Neural Comput 15:1667–1667

    Article  MATH  Google Scholar 

  13. Kingma D P, Ba J (2014) Adam: a method for stochastic optimization. https://arxiv.org/pdf/1412.6980.pdf. Accessed 28 May 2021

  14. Klinker G J, Shafer S A, Kanade T (1990) The measurement of highlights in color images. Int J Comput Vis 2:7–32

    Article  Google Scholar 

  15. Krizhevsky A, Sutskever I, Hinton G E (2012) Imagenet classification with deep convolutional neural networks. In: Proc. International conference on neural information processing systems. ACM, New York, pp 1097–1105

  16. Lecun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436–444

    Article  Google Scholar 

  17. Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. In: Proc. of the IEEE, p 2278–2324

  18. Lee S W, Bajcsy R (1992) Detection of specularity using color and multiple views. In: Proc. european conference on computer vision. Springer, New York, pp 99–114

  19. Lin S, Li Y Z, Kang S B, Xin T, Shum H Y (2002) Diffuse-specular separation and depth recovery from image sequences. In: Proc. 7th European conference on computer vision. Springer, New York, pp 210–224

  20. Lin S, Shum H (2001) Separation of diffuse and specular reflection in color images. In: Proc. 2001 IEEE computer society conference on computer vision and pattern recognition. IEEE, Los Alamitos, pp 341–346

  21. Mallick S P, Zickler T, Kriegman D J (2005) Beyond lambert: Reconstructing specular surfaces using color. In: Proc. 2005 IEEE Computer society conference on computer vision and pattern recognition. IEEE, Los Alamitos, pp 210–224

  22. Montes A, Salvador A, Pascual S, Giro-i Nieto X (2016) Temporal activity detection in untrimmed videos with recurrent neural networks. In: Proc. 1st NIPS workshop on large scale computer vision systems. NIPS, La Jolla, California, pp 1–5

  23. Nayar S K, Fang X S, Boult T (1997) Separation of reflection components using color and polarization. Int J Comput Vis 21:163–186

    Article  Google Scholar 

  24. Parkkinen J P S, Hallikainen J, Jaaskelainen T (1989) Characteristic spectra of munsell colors. J Opt Soc Am A 6:318–322

    Article  Google Scholar 

  25. Pascanu R, Mikolov T, Bengio Y (2012) On the difficulty of training recurrent neural networks. https://arxiv.org/pdf/1211.5063.pdf. Accessed 28 May 2021

  26. Sato Y, Ikeuchi K (1994) Temporal-color space analysis of reflection. J Opt Soc Am A 11:2990–3002

    Article  Google Scholar 

  27. Schluns K, Koschan A (2000) Global and local highlight analysis in color images. In: Proc.1st.Int.Conf. Color Graphics Image Processing. Wiley, Hoboken, pp 300–304

  28. Shafer S A (1985) Using color to separate reflection components. Color Res Appl 10:210–218

    Article  Google Scholar 

  29. Shen H L, Zhang Z H (2013) Real-time highlight removal using intensity ratio. Appl Opt 52:4483–4493

    Article  Google Scholar 

  30. Shen H L, Zhang H G, Shao S J (2008) Chromaticity-based separation of reflection components in a single image. Pattern Recogn 41:2461–2469

    Article  MATH  Google Scholar 

  31. Suo J, An D, Ji X, Wang H, Dai Q (2016) Fast and high quality highlight removal from a single image. IEEE Trans Image Process 25:5441–5454

    Article  MathSciNet  MATH  Google Scholar 

  32. Tan R T, Ikeuchi K (2005) Separating reflection components of textured surfaces using a single image. IEEE Trans Pattern Anal Mach Intell 27:178–193

    Article  Google Scholar 

  33. Tan P, Quan L, Lin S (2006) Separation of highlight reflections on textured surfaces. In: Proc. 2006 IEEE computer society conference on computer vision and pattern recognition. IEEE, Los Alamitos, pp 1855–1860

  34. Vinod N, Hinton G E (2010) Rectified linear units improve restricted boltzmann machines. In: Proc. international conference on machine learning. ACM, New York, pp 807–814

  35. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13:600–612

    Article  Google Scholar 

  36. Wolff L B, Boult T E (1991) Constraining object features using a polarization reflectance model. IEEE Trans Pattern Anal Mach Intell 13:635–657

    Article  Google Scholar 

  37. Yamamoto T, Nakazawa A (2019) General improvement method of specular component separation using high-emphasis filter and similarity function. ITE Trans Media Technol Appl 7:92–102

    Google Scholar 

  38. Yang Q X, Tan K H, Ahuja N (2009) Real-time o (1) bilateral filtering. In: Proc. 2009 IEEE computer society conference on computer vision and pattern recognition. IEEE, Los Alamitos, pp 557–564

  39. Yang Q X, Wang S N, Ahuja N (2010) Real-time specular highlight removal using bilateral filtering. In: Proc. 2010 european conference on computer vision. Springer, New York, pp 87–100

  40. Yang Q X, Tang J, Ahuja N (2015) Efficient and robust specular highlight removal. IEEE Trans Pattern Anal Mach Intell 37:1304–1311

    Article  Google Scholar 

  41. Yoon K, Choi Y, Kweon I S (2006) Fast separation of reflection components using a specularity-invariant image representation. In: Proc. 2006 International conference on image processing. IEEE, Los Alamitos, pp 973–976

Download references

Author information

Authors and Affiliations

  1. College of Communication Engineering, Hangzhou Dianzi University, Hangzhou, 310018, Zhejiang, China

    Huahua Chen, Lingjie Luo, Chunsheng Guo, Na Ying & Xueyi Ye

Authors
  1. Huahua Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lingjie Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chunsheng Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Na Ying
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xueyi Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Huahua Chen.

Ethics declarations

Competing Interests

The authors declare that they have no competing interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, H., Luo, L., Guo, C. et al. Specular highlight removal using a divide-and-conquer multi-resolution deep network. Multimed Tools Appl 82, 36885–36907 (2023). https://doi.org/10.1007/s11042-023-14591-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14591-y

Keywords

Search

Navigation