Skip to main content
SpringerLink
Log in

Csf: global–local shading orders for intrinsic image decomposition

  • Research
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Intrinsic image decomposition faces the long-standing challenge from the coupling of the components of the image-the surface albedo, direct illumination, and ambient illumination in the observed image. Without knowing the absolute values of the image components, we propose inferring shading by ordering pixels by relative brightness. The pairwise shading orders are estimated in two ways: brightness order and low-order fittings of the local shading field. The brightness order is a nonlocal metric that can be used to compare any two pixels, including those with different reflectances and shadings. Low-order fittings are used for pixel pairs within local regions of smooth shading. They can capture the global order structure and local variations in the shading when used together. To integrate the pairwise orders into a globally consistent order, we propose a Consistency-aware Selective Fusion method. The iterative selection process solves the inconsistencies between pairwise orders obtained using different estimation methods. To avoid polluting the global order, inconsistent or unreliable pairwise orders will be automatically excluded from the fusion. Experimental results show that the proposed model effectively recovers the shading, including deep shadows, on the MIT Intrinsic Image dataset. Moreover, our model works well on natural images from the IIW, UIUC Shadow, and NYU-depth datasets, where the colors of direct lights and ambient lights are quite different.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Algorithm 1
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Notes

  1. The method SIRFS is evaluated on the images of cup2, deer, frog2, paper2, raccoon, sun, teabag1, and turtle, whereas the other images are used for training. The results of Bell et al. [21] are obtained through relaxing the constraints on the absolute values of shading and removing the intensity from the features for clustering the reflectance. Otherwise, the deep shadows will be mistaken for black and clustered into individual categories.

  2. For this method, the shading is replaced by a shadow map, in which black pixels indicate shadows and gray ones stand for penumbra.

  3. We combine the direct irradiance, the indirect irradiance, and the illumination into shading if needed. The temporal constraints of [13] are removed for dealing with single images.

References

  1. Barrow, H.G., Tenenbaum, J.M.: Recovering intrinsic scene characteristics from images. In: Computer Vision Systems. Academic Press, Waltham (1978)

    Google Scholar 

  2. Zhang, R., Tsai, P.-S., Cryer, J.E., Shah, M.: Shape from shading: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 21(8), 690–706 (1999)

    Article  Google Scholar 

  3. Yue, H., Yang, J., Sun, X., Wu, F., Hou, C.: Contrast enhancement based on intrinsic image decomposition. IEEE Trans. Image Process. 26(8), 3981–3994 (2017)

    Article  MathSciNet  Google Scholar 

  4. Sengupta, S., Kanazawa, A., Castillo, C.D., Jacobs, D.W.: Sfsnet: Learning shape, reflectance and illuminance of facesin the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6296–6305 (2018)

  5. Gao, Y., Hu, H.-M., Li, B., Guo, Q.: Naturalness preserved nonuniform illumination estimation for image enhancement based on retinex. IEEE Trans. Multimed. 20(2), 335–344 (2017)

    Article  Google Scholar 

  6. Hao, S., Han, X., Guo, Y., Xu, X., Wang, M.: Low-light image enhancement with semi-decoupled decomposition. IEEE Trans. Multimed. 22(12), 3025–3038 (2020)

    Article  Google Scholar 

  7. Yu, Y., Smith, W.A.: Inverserendernet: Learning single image inverse rendering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3155–3164 (2019)

  8. Zhu, Y., Tang, J., Li, S., Shi, B.: Derendernet: Intrinsic image decomposition of urban scenes with shape-(in) dependent shading rendering. In: IEEE International Conference on Computational Photography, pp. 1–11 (2021)

  9. Luo, J., Huang, Z., Li, Y., Zhou, X., Zhang, G., Bao, H.: Niid-net: adapting surface normal knowledge for intrinsic image decomposition in indoor scenes. IEEE Trans. Vis. Comput. Graph. 26(12), 3434–3445 (2020)

    Article  Google Scholar 

  10. Sereno, M., Wang, X., Besançon, L., Mcguffin, M.J., Isenberg, T.: Collaborative work in augmented reality: a survey. IEEE Trans. Vis. Comput. Graph. 28(6), 2530–2549 (2020)

    Google Scholar 

  11. Shen, L., Ping, T., Lin, S.: Intrinsic image decomposition with non-local texture cues. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2008)

  12. Gehler, P., Rother, C., Kiefel, M., Zhang, L., Schölkopf, B.: Recovering intrinsic images with a global sparsity prior on reflectance. In: Advances in Neural Information Processing Systems, pp. 765–773 (2011)

  13. Lee, K., Zhao, Q., Tong, X., Gong, M., Izadi, S., Lee, S., Tan, P., Lin, S.: Estimation of intrinsic image sequences from image+depth video. In: European Conference on Computer Vision (2012)

  14. Chen, Q., Koltun, V.: A simple model for intrinsic image decomposition with depth cues. In: IEEE International Conference on Computer Vision (2013)

  15. Shen, L., Yeo, C., Hua, B.-S.: Intrinsic images decomposition using a sparse representation of reflectance. IEEE Transactions on Pattern Analysis and Machine Intelligence 35(12) (2013)

  16. Barron, J.T., Malik, J.: Color constancy, intrinsic images, and shape estimation. In: European Conference on Computer Vision (2012)

  17. Bi, S., Han, X., Yu, Y.: An \(l_1\) image transform for edge-preserving smoothing and scene-level intrinsic decomposition. ACM Trans. Graph. 34(4), 1–12 (2015)

    Article  Google Scholar 

  18. Garces, E., Munoz, A., Lopez-Moreno, J., Gutierrez, D.: Intrinsic images by clustering. Comput. Graph. Forum 31(4) (2012)

  19. Nie, X., Feng, W., Wan, L., Dai, H., Pun, C.-M.: Intrinsic image decomposition by hierarchical l0 sparsity. In: IEEE International Conference on Multimedia and Expo (2014)

  20. Wang, Y., Li, K., Yang, J., Ye, X.: Intrinsic decomposition from a single rgb-d image with sparse and non-local priors. In: Proceedings of IEEE International Conference on Multimedia and Expo, IEEE pp. 1201–1206 (2017)

  21. Bell, S., Bala, K., Snavely, N.: Intrinsic images in the wild. ACM Trans. Graph. 33(4), 1–2 (2014)

    Article  Google Scholar 

  22. Serra, M., Penacchio, O., Benavente, R., Vanrell, M.: Names and shades of color for intrinsic image estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2012)

  23. Chang, J., Cabezas, R., Fisher III, J.W.: Bayesian nonparametric intrinsic image decomposition. In: European Conference on Computer Vision (2014)

  24. Maxwell, B., Friedhoff, R., Smith, C.: A bi-illuminant dichromatic reflection model for understanding images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2008)

  25. Sheng, B., Li, P., Jin, Y., Tan, P., Lee, T.-Y.: Intrinsic image decomposition with step and drift shading separation. IEEE Trans. Vis. Comput. Graph. 26(2), 1332–1346 (2018)

    Article  Google Scholar 

  26. Land, E.H., McCann, J.J.: Lightness and retinex theory. Josa 61(1), 1–11 (1971)

    Article  Google Scholar 

  27. Tappen, M.F., Adelson, E.H., Freeman, W.T.: Estimating intrinsic component images using non-linear regression. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2006)

  28. Liu, Y., Yuan, Z., Zheng, N.: Intrinsic image decomposition from pair-wise shading ordering. In: Asian Conference on Computer Vision (2014)

  29. Zhao, Q., Tan, P., Dai, Q., Shen, L., Wu, E., Lin, S.: A closed-form solution to retinex with nonlocal texture constraints. IEEE Trans. Pattern Anal. Mach. Intell. 34(7), 1437–1444 (2012)

    Article  Google Scholar 

  30. Bell, M., Freeman, W.T.: Learning local evidence for shading and reflectance. In: IEEE International Conference on Computer Vision (2001)

  31. Tappen, M.F., Freeman, W.T., Adelson, E.H.: Recovering intrinsic images from a single image. IEEE Trans. Pattern Anal. Mach. Intell. 27(9), 1459–1472 (2005)

    Article  Google Scholar 

  32. Jiang, X., Schofield, A., Wyatt, J.: Correlation-based intrinsic image extraction from a single image. In: European Conference on Computer Vision (2010)

  33. Weiss, Y.: Deriving intrinsic images from image sequences. In: IEEE International Conference on Computer Vision (2001)

  34. Hauagge, D., Wehrwein, S., Bala, K., Snavely, N.: Photometric ambient occlusion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2013)

  35. Li, Y., Brown, M.S.: Single image layer separation using relative smoothness. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2014)

  36. Agrawal, A., Raskar, R., Nayar, S.K., Li, Y.: Removing photography artifacts using gradient projection and flash-exposure sampling. ACM Trans. Graph. 24(3), 828–835 (2005)

    Article  Google Scholar 

  37. Yu, S.X.: Angular embedding: From jarring intensity differences to perceived luminance. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2009)

  38. Maire, M.: Simultaneous segmentation and figure/ground organization using angular embedding. In: European Conference on Computer Vision (2010)

  39. Yu, S.X.: Angular embedding: a robust quadratic criterion. IEEE Trans. Pattern Anal. Mach. Intell. 34, 158–173 (2012)

    Article  Google Scholar 

  40. Singer, A.: Angular synchronization by eigenvectors and semidefinite programming. Appl. Comput. Harmon. Anal. 30(1), 20–36 (2011)

    Article  MathSciNet  Google Scholar 

  41. Beigpour, S., Weijer, J.: Object recoloring based on intrinsic image estimation. In: IEEE International Conference on Computer Vision (2011)

  42. Liao, Z., Rock, J., Wang, Y., Forsyth, D.: Non-parametric filtering for geometric detail extraction and material representation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2013)

  43. Carroll, R., Ramamoorthi, R., Agrawala, M.: Illumination decomposition for material recoloring with consistent interreflections. ACM Trans. Graph. 30(4), 43 (2011)

    Article  Google Scholar 

  44. Serra, M., Penacchio, O., Benavente, R., Vanrell, M., Samaras, D.: The photometry of intrinsic images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2014)

  45. Jeon, J., Cho, S., Tong, X., Lee, S.: Intrinsic image decomposition using structure-texture separation and surface normals. In: European Conference on Computer Vision (2014)

  46. Barron, J.T., Malik, J.: Intrinsic scene properties from a single rgb-d image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2013)

  47. Laffont, P.-Y., Bousseau, A., Drettakis, G.: Rich intrinsic image decomposition of outdoor scenes from multiple views. IEEE Trans. Vis. Comput. Graph. 19(2), 210–224 (2012)

    Article  Google Scholar 

  48. Kong, N., Gehler, P., Black, M.: Intrinsic video. In: European Conference on Computer Vision (2014)

  49. Ye, G., Garces, E., Liu, Y., Dai, Q., Gutierrez, D.: Intrinsic video and applications. ACM Trans. Graph. 33(4), 1–11 (2014)

    Article  Google Scholar 

  50. Bonneel, N., Sunkavalli, K., Tompkin, J., Sun, D., Paris, S., Pfister, H.: Interactive intrinsic video editing. ACM Trans. Graph. 33, 1–10 (2014)

    Article  Google Scholar 

  51. Fan, Q., Yang, J., Hua, G., Chen, B., Wipf, D.: Revisiting deep intrinsic image decompositions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8944–8952 (2018)

  52. Zhou, H., Yu, X., Jacobs, D.W.: Glosh: Global-local spherical harmonics for intrinsic image decomposition. In: Proceedings of IEEE/CVF International Conference on Computer Vision, pp. 7820–7829 (2019)

  53. Liu, Y., Li, Y., You, S., Lu, F.: Unsupervised learning for intrinsic image decomposition from a single image. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3248–3257 (2020)

  54. Ma, Y., Jiang, X., Xia, Z., Gabbouj, M., Feng, X.: Casqnet: intrinsic image decomposition based on cascaded quotient network. IEEE Trans. Circuits Syst. Video Technol. 31(7), 2661–2674 (2020)

  55. Baslamisli, A.S., Liu, Y., Karaoglu, S., Gevers, T.: Physics-based shading reconstruction for intrinsic image decomposition. Comput. Vis. Image Underst. 205, 103183 (2021)

    Article  Google Scholar 

  56. Weligampola, H., Jayatilaka, G., Sritharan, S., Ekanayake, P., Ragel, R., Herath, V., Godaliyadda, R.: An optical physics inspired cnn approach for intrinsic image decomposition. In: Proceedings of IEEE International Conference on Image Processing, pp. 1864–1868 (2021)

  57. Guo, R., Dai, Q., Hoiem, D.: Paired regions for shadow detection and removal. IEEE Trans. Pattern Anal. Mach. Intell. 35(12), 2956–2967 (2012)

    Article  Google Scholar 

  58. Grosse, R., Johnson, M.K., Adelson, E.H., Freeman, W.T.: Ground-truth dataset and baseline evaluations for intrinsic image algorithms. In: Proceedings of the IEEE International Conference on Computer Vision (2009)

  59. Finlayson, G., Drew, M., Lu, C.: Entropy minimization for shadow removal. Int. J. Comput. Vis. 85(1), 35–57 (2009)

    Article  Google Scholar 

  60. Omer, I., Werman, M.: Color lines: image specific color representation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2004)

  61. Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 888–905 (2000)

    Article  Google Scholar 

  62. Zou, H., Hastie, T.: Regularization and variable selection via the elastic net. J. Roy. Stat. Soc. B 67, 301–320 (2005)

    Article  MathSciNet  Google Scholar 

  63. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)

    Article  Google Scholar 

  64. Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from rgbd images. In: European Conference on Computer Vision (2012)

  65. Shen, J., Yang, X., Jia, Y., Li, X.: Intrinsic images using optimization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2011)

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (No. 61876112, No. 61976170) and Beijing Natural Science Foundation (No. L201022).

Author information

Author notes

  1. Handan Zhang and Yuanliu Liu have contributed equally to this work.

Authors and Affiliations

  1. College of Information Engineering, Capital Normal University, Beijing, 10048, China

    Handan Zhang & Tie Liu

  2. College of Artificial Intelligence, Xi’an Jiaotong University, Shaanxi, 710049, China

    Yuanliu Liu & Zejian Yuan

Authors
  1. Handan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tie Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuanliu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zejian Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Tie Liu or Zejian Yuan.

Ethics declarations

Conflict of interest

The authors declare they have no financial interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, H., Liu, T., Liu, Y. et al. Csf: global–local shading orders for intrinsic image decomposition. Machine Vision and Applications 35, 4 (2024). https://doi.org/10.1007/s00138-023-01485-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-023-01485-0

Keywords

Search

Navigation