Skip to main content
SpringerLink
Log in

Reflectance edge guided networks for detail-preserving intrinsic image decomposition

  • Research Paper
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

Deep learning-based intrinsic image decomposition methods rely heavily on large-scale training data. However, current real-world datasets only contain sparse annotations, leading to textureless reflectance estimation. Although densely-labeled synthetic datasets are available, the large bias between these two categories easily incurs noticeable artifacts (e.g., shading residuals) on reflectance. To address this issue, we introduce reflectance edges that are predicted by a neural network trained on synthetic data with full supervision. Once trained, this network is able to capture high-frequency details of reflectance while greatly reducing the bias stemming from the discrepancy between different data distributions. We design another neural network to remove shading as much as possible from the input image. As this network is trained solely on real-world datasets, little bias will be introduced but the predicted reflectance will be overly smooth due to limited annotations. To recover texture details of the reflectance while still suppressing bias, we leverage a third neural network to progressively fuse feature maps from both reflectance edge maps and coarse-grained reflectance maps. The well-designed fusion strategy makes the best use of features extracted from the real-world data and helps to generate texture-rich reflectance with fewer artifacts. Extensive experiments on multiple benchmark datasets demonstrate the superiority of the proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Wu C, Zollhöfer M, Nießner M, et al. Real-time shading-based refinement for consumer depth cameras. ACM Trans Graph, 2014, 33: 1–10

    Google Scholar 

  2. Zollhöfer M, Dai A, Innmann M, et al. Shading-based refinement on volumetric signed distance functions. ACM Trans Graph, 2015, 34: 1–14

    Article  Google Scholar 

  3. Shen J, Yan X, Chen L, et al. Re-texturing by intrinsic video. Inf Sci, 2014, 281: 726–735

    Article  MathSciNet  Google Scholar 

  4. Meka A, Fox G, Zollhofer M, et al. Live user-guided intrinsic video for static scenes. IEEE Trans Visual Comput Graph, 2017, 23: 2447–2454

    Article  Google Scholar 

  5. Tan J, Lien J M, Gingold Y. Decomposing images into layers via RGB-space geometry. ACM Trans Graph, 2017, 36: 1–14

    Article  Google Scholar 

  6. Wang Y L, Liu Y F, Xu K. An improved geometric approach for palette-based image decomposition and recoloring. Comput Graph Forum, 2019, 38: 11–22

    Article  Google Scholar 

  7. Cui M Y, Zhu Z, Yang Y, et al. Towards natural object-based image recoloring. Comp Visual Media, 2022, 8: 317–328

    Article  Google Scholar 

  8. Garces E, Munoz A, Lopez-Moreno J, et al. Intrinsic images by clustering. Comput Graph Forum, 2012, 31: 1415–1424

    Article  Google Scholar 

  9. Nestmeyer T, Lalonde J F, Matthews I, et al. Learning physics-guided face relighting under directional light. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 5124–5133

  10. Li C, Zhou K, Lin S. Simulating makeup through physics-based manipulation of intrinsic image layers. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015. 4621–4629

  11. Shu Z, Yumer E, Hadap S, et al. Neural face editing with intrinsic image disentangling. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. 5541–5550

  12. Narihira T, Maire M, Yu S X. Direct intrinsics: learning albedo-shading decomposition by convolutional regression. In: Proceedings of the IEEE International Conference on Computer Vision, 2015. 2992–2992

  13. Zhou T, Krahenbuhl P, Efros A A. Learning data-driven reflectance priors for intrinsic image decomposition. In: Proceedings of the IEEE International Conference on Computer Vision, 2015. 3469–3477

  14. Zoran D, Isola P, Krishnan D, et al. Learning ordinal relationships for mid-level vision. In: Proceedings of the IEEE International Conference on Computer Vision, 2015. 388–396

  15. Fan Q, Yang J, Hua G, et al. Revisiting deep intrinsic image decompositions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018. 8944–8952

  16. Li Z, Snavely N. CGIntrinsics: better intrinsic image decomposition through physically-based rendering. In: Proceedings of European Conference on Computer Vision (ECCV), 2018

  17. Zhou H, Yu X, Jacobs D W. GLoSH: global-local spherical harmonics for intrinsic image decomposition. In: Proceedings of the IEEE International Conference on Computer Vision, 2019. 7820–7829

  18. Sengupta S, Gu J, Kim K, et al. Neural inverse rendering of an indoor scene from a single image. In: Proceedings of the IEEE International Conference on Computer Vision, 2019. 8598–8607

  19. Luo J, Huang Z, Li Y, et al. NIID-Net: adapting surface normal knowledge for intrinsic image decomposition in indoor scenes. IEEE Trans Visual Comput Graph, 2020, 26: 3434–3445

    Article  Google Scholar 

  20. Lettry L, Vanhoey K, van Gool L. Unsupervised deep single-image intrinsic decomposition using illumination-varying image sequences. Comput Graph Forum, 2018, 37: 409–419

    Article  Google Scholar 

  21. Grosse R, Johnson M K, Adelson E H, et al. Ground truth dataset and baseline evaluations for intrinsic image algorithms. In: Proceedings of IEEE 12th International Conference on Computer Vision, 2009. 2335–2342

  22. Tappen M F, Freeman W T, Adelson E H. Recovering intrinsic images from a single image. IEEE Trans Pattern Anal Machine Intell, 2005, 27: 1459–1472

    Article  Google Scholar 

  23. Shen L, Yeo C. Intrinsic images decomposition using a local and global sparse representation of reflectance. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2011. 697–704

  24. Bell S, Bala K, Snavely N. Intrinsic images in the wild. ACM Trans Graph, 2014, 33: 1–12

    Article  Google Scholar 

  25. Barron J T, Malik J. Shape, illumination, and reflectance from shading. IEEE Trans Pattern Anal Mach Intell, 2015, 37: 1670–1687

    Article  Google Scholar 

  26. Land E H, McCann J J. Lightness and retinex theory. J Opt Soc Am, 1971, 61: 1–11

    Article  Google Scholar 

  27. Horn B K P. Determining lightness from an image. Comput Graph Image Process, 1974, 3: 277–299

    Article  MathSciNet  Google Scholar 

  28. Blake A. Boundary conditions for lightness computation in Mondrian World. Comput Vision Graph Image Process, 1985, 32: 314–327

    Article  Google Scholar 

  29. Funt B V, Drew M S, Brockington M. Recovering shading from color images. In: Proceedings of European Conference on Computer Vision. Berlin: Springer, 1992. 124–132

    Google Scholar 

  30. Omer I, Werman M. Color lines: image specific color representation. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004

  31. Rother C, Kiefel M, Zhang L, et al. Recovering intrinsic images with a global sparsity prior on reflectance. In: Proceedings of Advances in Neural Information Processing Systems, 2011. 765–773

  32. Shen L, Tan P, Lin S. Intrinsic image decomposition with non-local texture cues. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2008. 1–7

  33. Zhao Q, Tan P, Dai Q, et al. A closed-form solution to retinex with nonlocal texture constraints. IEEE Trans Pattern Anal Mach Intell, 2012, 34: 1437–1444

    Article  Google Scholar 

  34. Barron J T, Malik J. Intrinsic scene properties from a single RGB-D image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2013. 17–24

  35. Chen Q, Koltun V. A simple model for intrinsic image decomposition with depth cues. In: Proceedings of the IEEE International Conference on Computer Vision, 2013. 241–248

  36. Li Y, Brown M S. Single image layer separation using relative smoothness. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014. 2752–2759

  37. Bi S, Han X, Yu Y. An L1 image transform for edge-preserving smoothing and scene-level intrinsic decomposition. ACM Trans Graph, 2015, 34: 1–12

    Article  Google Scholar 

  38. Sheng B, Li P, Jin Y, et al. Intrinsic image decomposition with step and drift shading separation. IEEE Trans Visual Comput Graph, 2020, 26: 1332–1346

    Article  Google Scholar 

  39. Laffont P Y, Bousseau A, Drettakis G. Rich intrinsic image decomposition of outdoor scenes from multiple views. IEEE Trans Visual Comput Graph, 2013, 19: 210–224

    Article  Google Scholar 

  40. Laffont P Y, Bousseau A, Paris S, et al. Coherent intrinsic images from photo collections. ACM Trans Graph, 2012, 31: 1–11

    Article  Google Scholar 

  41. Nestmeyer T, Gehler P V. Reflectance adaptive filtering improves intrinsic image estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. 6789–6798

  42. Shi J, Dong Y, Su H, et al. Learning non-lambertian object intrinsics across shapenet categories. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. 1685–1694

  43. Cheng L, Zhang C, Liao Z. Intrinsic image transformation via scale space decomposition. In: Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018

  44. Baslamisli A S, Le H A, Gevers T. CNN based learning using reflection and retinex models for intrinsic image decomposition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018. 6674–6683

  45. Kovacs B, Bell S, Snavely N, et al. Shading annotations in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. 6998–7007

  46. Butler D J, Wulff J, Stanley G B, et al. A naturalistic open source movie for optical flow evaluation. In: Proceedings of European Conference on Computer Vision. Berlin: Springer, 2012. 611–625

    Google Scholar 

  47. Chang A X, Funkhouser T, Guibas L, et al. ShapeNet: an information-rich 3D model repository. 2015. ArXiv:1512.03012

  48. Liu Y, Li Y, You S, et al. Unsupervised learning for intrinsic image decomposition from a single image. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020

  49. Li Z, Snavely N. Learning intrinsic image decomposition from watching the world. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018. 9039–9048

  50. Ma W C, Chu H, Zhou B, et al. Single image intrinsic decomposition without a single intrinsic image. In: Proceedings of the European Conference on Computer Vision (ECCV), 2018. 201–217

  51. Janner M, Wu J, Kulkarni T D, et al. Self-supervised intrinsic image decomposition. In: Proceedings of Advances in Neural Information Processing Systems, 2017. 5936–5946

  52. Baslamisli A S, Groenestege T T, Das P, et al. Joint learning of intrinsic images and semantic segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV), 2018

  53. Kim S, Park K, Sohn K, et al. Unified depth prediction and intrinsic image decomposition from a single image via joint convolutional neural fields. In: Proceedings of European Conference on Computer Vision. Berlin: Springer, 2016. 143–159

    Google Scholar 

  54. Gastal E S, Oliveira M M. Domain transform for edge-aware image and video processing. In: Proceedings of ACM SIGGRAPH 2011, 2011. 1–12

  55. Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation. In: Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention, 2015. 234–241

  56. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. 2014. ArXiv:1409.1556

  57. Narihira T, Maire M, Yu S X. Learning lightness from human judgement on relative reflectance. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015. 2965–2973

  58. Zhang Y, Song S, Yumer E, et al. Physically-based rendering for indoor scene understanding using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. 5287–5295

  59. Wang J, Li X, Yang J. Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018. 1788–1797

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (Grant Nos. 61972194, 62032011) and Natural Science Foundation of Jiangsu Province (Grant No. BK20211147).

Author information

Authors and Affiliations

  1. State Key Lab for Novel Software Technology, Nanjing University, Nanjing, 210023, China

    Quewei Li, Jie Guo, Zhengyi Wu, Yang Fei & Yanwen Guo

Authors
  1. Quewei Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jie Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhengyi Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yang Fei
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yanwen Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Jie Guo or Yanwen Guo.

Additional information

Supporting information

Appendixes A–D. The supporting information is available online at info.scichina.com and link.springer.com. The supporting materials are published as submitted, without typesetting or editing. The responsibility for scientific accuracy and content remains entirely with the authors.

Supplementary File

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Q., Guo, J., Wu, Z. et al. Reflectance edge guided networks for detail-preserving intrinsic image decomposition. Sci. China Inf. Sci. 66, 122105 (2023). https://doi.org/10.1007/s11432-021-3481-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-021-3481-3

Keywords

Search

Navigation