skip to main content
research-article

Learning efficient illumination multiplexing for joint capture of reflectance and shape

Published: 08 November 2019 Publication History

Abstract

We propose a novel framework that automatically learns the lighting patterns for efficient, joint acquisition of unknown reflectance and shape. The core of our framework is a deep neural network, with a shared linear encoder that directly corresponds to the lighting patterns used in physical acquisition, as well as non-linear decoders that output per-pixel normal and diffuse / specular information from photographs. We exploit the diffuse and normal information from multiple views to reconstruct a detailed 3D shape, and then fit BRDF parameters to the diffuse / specular information, producing texture maps as reflectance results. We demonstrate the effectiveness of the framework with physical objects that vary considerably in reflectance and shape, acquired with as few as 16 ~ 32 lighting patterns that correspond to 7 ~ 15 seconds of per-view acquisition time. Our framework is useful for optimizing the efficiency in both novel and existing setups, as it can automatically adapt to various factors, including the geometry / the lighting layout of the device and the properties of appearance.

Supplementary Material

MP4 File (a165-kang.mp4)

References

[1]
Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2013. Practical SVBRDF Capture in the Frequency Domain. ACM Trans. Graph. 32, 4, Article 110 (July 2013), 12 pages.
[2]
Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2015. Two-shot SVBRDF Capture for Stationary Materials. ACM Trans. Graph. 34, 4, Article 110 (July 2015), 13 pages.
[3]
Sai Bi, Nima Khademi Kalantari, and Ravi Ramamoorthi. 2017. Patch-based Optimization for Image-based Texture Mapping. ACM Trans. Graph. 36, 4, Article 106 (July 2017), 11 pages.
[4]
Guojun Chen, Yue Dong, Pieter Peers, Jiawan Zhang, and Xin Tong. 2014. Reflectance Scanning: Estimating Shading Frame and BRDF with Generalized Linear Light Sources. ACM Trans. Graph. 33, 4, Article 117 (July 2014), 11 pages.
[5]
Kristin J. Dana, Bram van Ginneken, Shree K. Nayar, and Jan J. Koenderink. 1999. Reflectance and Texture of Real-world Surfaces. ACM Trans. Graph. 18, 1 (Jan. 1999), 1--34.
[6]
Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, and Adrien Bousseau. 2018. Single-image SVBRDF Capture with a Rendering-aware Deep Network. ACM Trans. Graph. 37, 4, Article 128 (July 2018), 15 pages.
[7]
Yue Dong. 2019. Deep appearance modeling: A survey. Visual Informatics (2019).
[8]
Yue Dong, Jiaping Wang, Xin Tong, John Snyder, Yanxiang Lan, Moshe Ben-Ezra, and Baining Guo. 2010. Manifold Bootstrapping for SVBRDF Capture. ACM Trans. Graph. 29, 4, Article 98 (July 2010), 10 pages.
[9]
Mark Fiala. 2005. ARTag, a fiducial marker system using digital techniques. In CVPR.
[10]
Andrew Gardner, Chris Tchou, Tim Hawkins, and Paul Debevec. 2003. Linear light source reflectometry. ACM Trans. Graph. 22, 3 (2003), 749--758.
[11]
Abhijeet Ghosh, Tongbo Chen, Pieter Peers, Cyrus A. Wilson, and Paul Debevec. 2009. Estimating Specular Roughness and Anisotropy from Second Order Spherical Gradient Illumination. Computer Graphics Forum 28, 4 (2009), 1161--1170.
[12]
Darya Guarnera, Giuseppe C. Guarnera, Abhijeet Ghosh, Cornelia Denk, and Mashhuda Glencross. 2016. BRDF Representation and Acquisition. Computer Graphics Forum 35, 2 (2016), 625--650.
[13]
Michael Holroyd, Jason Lawrence, and Todd Zickler. 2010. A Coaxial Optical Scanner for Synchronous Acquisition of 3D Geometry and Surface Reflectance. ACM Trans. Graph. 29, 4, Article 99 (July 2010), 12 pages.
[14]
Satoshi Ikehata. 2018. CNN-PS: CNN-based photometric stereo for general non-convex surfaces. In ECCV.
[15]
James T. Kajiya. 1986. The Rendering Equation (SIGGRAPH '86). 143--150.
[16]
Kaizhang Kang, Zimin Chen, Jiaping Wang, Kun Zhou, and Hongzhi Wu. 2018. Efficient Reflectance Capture Using an Autoencoder. ACM Trans. Graph. 37, 4, Article 127 (July 2018), 10 pages.
[17]
Michael Kazhdan and Hugues Hoppe. 2013. Screened Poisson Surface Reconstruction. ACM Trans. Graph. 32, 3, Article 29 (July 2013), 13 pages.
[18]
Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry. 2017. End-To-End Learning of Geometry and Context for Deep Stereo Regression. In ICCV.
[19]
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR.
[20]
Ares Lagae, Sylvain Lefebvre, George Drettakis, and Philip Dutré. 2009. Procedural Noise Using Sparse Gabor Convolution. ACM Trans. Graph. 28, 3, Article 54 (July 2009), 10 pages.
[21]
Jason Lawrence, Aner Ben-Artzi, Christopher DeCoro, Wojciech Matusik, Hanspeter Pfister, Ravi Ramamoorthi, and Szymon Rusinkiewicz. 2006. Inverse Shade Trees for Non-parametric Material Representation and Editing. ACM Trans. Graph. 25, 3 (July 2006), 735--745.
[22]
Hendrik P. A. Lensch, Jan Kautz, Michael Goesele, Wolfgang Heidrich, and Hans-Peter Seidel. 2003. Image-based Reconstruction of Spatial Appearance and Geometric Detail. ACM Trans. Graph. 22, 2 (April 2003), 234--257.
[23]
Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2017. Modeling Surface Appearance from a Single Photograph Using Self-augmented Convolutional Neural Networks. ACM Trans. Graph. 36, 4, Article 45 (July 2017), 11 pages.
[24]
Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. 2018. Learning to Reconstruct Shape and Spatially-varying Reflectance from a Single Image. ACM Trans. Graph. 37, 6, Article 269 (Dec. 2018), 11 pages.
[25]
Stephen McAuley, Stephen Hill, Naty Hoffman, Yoshiharu Gotanda, Brian Smits, Brent Burley, and Adam Martinez. 2012. Practical Physically-based Shading in Film and Game Production. In ACM SIGGRAPH 2012 Courses. Article 10, 7 pages.
[26]
José Luis Morales and Jorge Nocedal. 2011. Remark on "Algorithm 778: L-BFGS-B: Fortran Subroutines for Large-scale Bound Constrained Optimization". ACM Trans. Math. Softw. 38, 1, Article 7 (Dec. 2011), 4 pages.
[27]
Giljoo Nam, Joo Ho Lee, Diego Gutierrez, and Min H Kim. 2018. Practical SVBRDF acquisition of 3D objects with unstructured flash photography. In SIGGRAPH Asia Technical Papers. 267.
[28]
Diego Nehab, Szymon Rusinkiewicz, James Davis, and Ravi Ramamoorthi. 2005. Efficiently Combining Positions and Normals for Precise 3D Geometry. ACM Trans. Graph. 24, 3 (July 2005), 536--543.
[29]
Jannik Boll Nielsen, Henrik Wann Jensen, and Ravi Ramamoorthi. 2015. On Optimal, Minimal BRDF Sampling for Reflectance Acquisition. ACM Trans. Graph. 34, 6, Article 186 (Oct. 2015), 11 pages.
[30]
Daniel Scharstein and Richard Szeliski. 2003. High-accuracy stereo depth maps using structured light. In CVPR.
[31]
Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm. 2016. Pixelwise View Selection for Unstructured Multi-View Stereo. In ECCV.
[32]
Borom Tunwattanapong, Graham Fyffe, Paul Graham, Jay Busch, Xueming Yu, Abhijeet Ghosh, and Paul Debevec. 2013. Acquiring Reflectance and Shape from Continuous Spherical Harmonic Illumination. ACM Trans. Graph. 32, 4, Article 109 (July 2013), 12 pages.
[33]
Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance. 2007. Microfacet Models for Refraction through Rough Surfaces. In Rendering Techniques (Proc. EGWR).
[34]
Michael Weinmann and Reinhard Klein. 2015. Advances in Geometry and Reflectance Acquisition. In SIGGRAPH Asia Courses. Article 1, 71 pages.
[35]
Tim Weyrich, Jason Lawrence, Hendrik P. A. Lensch, Szymon Rusinkiewicz, and Todd Zickler. 2009. Principles of Appearance Acquisition and Representation. Found. Trends. Comput. Graph. Vis. 4, 2 (2009), 75--191.
[36]
Robert J Woodham. 1980. Photometric method for determining surface orientation from multiple images. Optical engineering 19, 1 (1980), 191139.
[37]
Hongzhi Wu, Zhaotian Wang, and Kun Zhou. 2016. Simultaneous Localization and Appearance Estimation with a Consumer RGB-D Camera. IEEE TVCG 22, 8 (Aug 2016), 2012--2023.
[38]
Shihao Wu, Hui Huang, Tiziano Portenier, Matan Sela, Daniel Cohen-Or, Ron Kimmel, and Matthias Zwicker. 2018. Specular-to-Diffuse Translation for Multi-View Reconstruction. In ECCV.
[39]
Rui Xia, Yue Dong, Pieter Peers, and Xin Tong. 2016. Recovering Shape and Spatially-varying Surface Reflectance Under Unknown Illumination. ACM Trans. Graph. 35, 6, Article 187 (Nov. 2016), 12 pages.
[40]
Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. 2018. MVSNet: Depth Inference for Unstructured Multi-view Stereo. In ECCV.
[41]
Kun Zhou, John Synder, Baining Guo, and Heung-Yeung Shum. 2004. Iso-charts: Stretch-driven Mesh Parameterization Using Spectral Analysis. In SGP.
[42]
Zhenglong Zhou, Zhe Wu, and Ping Tan. 2013. Multi-view photometric stereo with spatially varying isotropic materials. In CVPR.

Cited By

View all
  • (2024)DTDMat: A Comprehensive SVBRDF Dataset with Detailed Text DescriptionsProceedings of the 19th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry10.1145/3703619.3706053(1-15)Online publication date: 1-Dec-2024
  • (2024)NFPLight: Deep SVBRDF Estimation via the Combination of Near and Far Field Point LightingACM Transactions on Graphics10.1145/368797843:6(1-11)Online publication date: 19-Dec-2024
  • (2024)Evaluating Visual Perception of Object Motion in Dynamic EnvironmentsACM Transactions on Graphics10.1145/368791243:6(1-12)Online publication date: 19-Dec-2024
  • Show More Cited By

Index Terms

  1. Learning efficient illumination multiplexing for joint capture of reflectance and shape

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Transactions on Graphics
      ACM Transactions on Graphics  Volume 38, Issue 6
      December 2019
      1292 pages
      ISSN:0730-0301
      EISSN:1557-7368
      DOI:10.1145/3355089
      Issue’s Table of Contents
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 08 November 2019
      Published in TOG Volume 38, Issue 6

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. SVBRDF
      2. lighting patterns
      3. multi-view stereo
      4. optimal sampling

      Qualifiers

      • Research-article

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)49
      • Downloads (Last 6 weeks)10
      Reflects downloads up to 15 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)DTDMat: A Comprehensive SVBRDF Dataset with Detailed Text DescriptionsProceedings of the 19th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry10.1145/3703619.3706053(1-15)Online publication date: 1-Dec-2024
      • (2024)NFPLight: Deep SVBRDF Estimation via the Combination of Near and Far Field Point LightingACM Transactions on Graphics10.1145/368797843:6(1-11)Online publication date: 19-Dec-2024
      • (2024)Evaluating Visual Perception of Object Motion in Dynamic EnvironmentsACM Transactions on Graphics10.1145/368791243:6(1-12)Online publication date: 19-Dec-2024
      • (2024)Deep SVBRDF Acquisition and Modelling: A SurveyComputer Graphics Forum10.1111/cgf.1519943:6Online publication date: 16-Sep-2024
      • (2024)FROST-BRDF: A Fast and Robust Optimal Sampling Technique for BRDF AcquisitionIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.335520030:7(4390-4402)Online publication date: Jul-2024
      • (2024)Efficient Reflectance Capture With a Deep Gated Mixture-of-ExpertsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.326187230:7(4246-4256)Online publication date: Jul-2024
      • (2024)High-Fidelity Specular SVBRDF Acquisition From Flash PhotographsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.323527730:4(1885-1896)Online publication date: Apr-2024
      • (2024)Differentiable Display Photometric Stereo2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.01124(11831-11840)Online publication date: 16-Jun-2024
      • (2024)Delving into high-quality SVBRDF acquisition: A new setup and methodComputational Visual Media10.1007/s41095-023-0352-610:3(523-541)Online publication date: 9-Feb-2024
      • (2024)Hypernetworks for Generalizable BRDF RepresentationComputer Vision – ECCV 202410.1007/978-3-031-73116-7_5(73-89)Online publication date: 29-Sep-2024
      • Show More Cited By

      View Options

      Login options

      Full Access

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media