skip to main content
research-article

Shading-based refinement on volumetric signed distance functions

Published: 27 July 2015 Publication History

Abstract

We present a novel method to obtain fine-scale detail in 3D reconstructions generated with low-budget RGB-D cameras or other commodity scanning devices. As the depth data of these sensors is noisy, truncated signed distance fields are typically used to regularize out the noise, which unfortunately leads to over-smoothed results. In our approach, we leverage RGB data to refine these reconstructions through shading cues, as color input is typically of much higher resolution than the depth data. As a result, we obtain reconstructions with high geometric detail, far beyond the depth resolution of the camera itself. Our core contribution is shading-based refinement directly on the implicit surface representation, which is generated from globally-aligned RGB-D images. We formulate the inverse shading problem on the volumetric distance field, and present a novel objective function which jointly optimizes for fine-scale surface geometry and spatially-varying surface reflectance. In order to enable the efficient reconstruction of sub-millimeter detail, we store and process our surface using a sparse voxel hashing scheme which we augment by introducing a grid hierarchy. A tailored GPU-based Gauss-Newton solver enables us to refine large shape models to previously unseen resolution within only a few seconds.

Supplementary Material

ZIP File (a96-zollhofer.zip)
Supplemental files
MP4 File (a96.mp4)

References

[1]
Agarwal, S., Furukawa, Y., Snavely, N., Simon, I., Curless, B., Seitz, S. M., and Szeliski, R. 2011. Building rome in a day. Communications of the ACM 54, 10, 105--112.
[2]
AgiSoft, L. 2014. Agisoft photoscan. Professional Edition.
[3]
Beeler, T., Bradley, D., Zimmer, H., and Gross, M. 2012. Improved reconstruction of deforming surfaces by cancelling ambient occlusion. In Proc. ECCV, 30--43.
[4]
Bylow, E., Sturm, J., Kerl, C., Kahl, F., and Cremers, D. 2013. Real-time camera tracking and 3d reconstruction using signed distance functions. In Robotics: Science and Systems (RSS) Conference 2013, vol. 9.
[5]
Carr, J. C., Beatson, R. K., Cherrie, J. B., Mitchell, T. J., Fright, W. R., McCallum, B. C., and Evans, T. R. 2001. Reconstruction and representation of 3d objects with radial basis functions. In Proc. SIGGRAPH, ACM, 67--76.
[6]
Chan, D., Buisman, H., Theobalt, C., Thrun, S., et al. 2008. A noise-aware filter for real-time depth upsampling. In Workshop on Multi-camera and Multi-modal Sensor Fusion Algorithms and Applications-M2SFA2 2008.
[7]
Chen, Q., and Koltun, V. 2013. A simple model for intrinsic image decomposition with depth cues. In The IEEE International Conference on Computer Vision (ICCV).
[8]
Chen, J., Bautembach, D., and Izadi, S. 2013. Scalable real-time volumetric surface reconstruction. ACM TOG 32, 4, 113.
[9]
Cui, Y., Schuon, S., Chan, D., Thrun, S., and Theobalt, C. 2010. 3d shape scanning with a time-of-flight camera. In Proc. CVPR, 1173--1180.
[10]
Curless, B., and Levoy, M. 1996. A volumetric method for building complex models from range images. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, ACM, 303--312.
[11]
Debevec, P. 2012. The light stages and their applications to photoreal digital actors. In SIGGRAPH Asia Technical Briefs.
[12]
Diebel, J., and Thrun, S. 2006. An application of Markov Random Fields to range sensing. 291--298.
[13]
Dolson, J., Baek, J., Plagemann, C., and Thrun, S. 2010. Upsampling range data in dynamic environments. In Proc. CVPR, IEEE, 1141--1148.
[14]
Fuhrmann, S., and Goesele, M. 2014. Floating scale surface reconstruction. ACM TOG 33, 4, 46.
[15]
Ghosh, A., Fyffe, G., Tunwattanapong, B., Busch, J., Yu, X., and Debevec, P. 2011. Multiview face capture using polarized spherical gradient illumination. ACM TOG 30.
[16]
Goldluecke, B., Aubry, M., Kolev, K., and Cremers, D. 2014. A super-resolution framework for high-accuracy multiview reconstruction. ijcv 106, 2 (jan), 172--191.
[17]
Haber, T., Fuchs, C., Bekaer, P., Seidel, H.-P., Goesele, M., and Lensch, H. 2009. Relighting objects from image collections. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, 627--634.
[18]
Han, Y., Lee, J.-Y., and Kweon, I. S. 2013. High quality shape from a single rgb-d image under uncalibrated natural illumination. In Proc. ICCV.
[19]
Hasinoff, S., Levin, A., Goode, P., and Freeman, W. 2011. Diffuse reflectance imaging with astronomical applications. In Proc. ICCV, 185--192.
[20]
Henry, P., Krainin, M., Herbst, E., Ren, X., and Fox, D. 2012. RGB-D mapping: Using Kinect-style depth cameras for dense 3D modeling of indoor environments. Int. J. Robotics Research 31 (apr), 647--663.
[21]
Hernández, C., Vogiatzis, G., and Cipolla, R. 2008. Multiview photometric stereo. IEEE PAMI 30, 3, 548--554.
[22]
Hoppe, H., DeRose, T., Duchamp, T., McDonald, J., and Stuetzle, W. 1992. Surface reconstruction from unorganized points. In Proceedings of the 19th Annual Conference on Computer Graphics and Interactive Techniques, ACM, New York, NY, USA, SIGGRAPH '92, 71--78.
[23]
Horn, B. K. 1975. Obtaining shape from shading information. The psychology of computer vision, 115--155.
[24]
Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A., et al. 2011. Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In Proc. UIST, ACM, 559--568.
[25]
Kazhdan, M., Bolitho, M., and Hoppe, H. 2006. Poisson surface reconstruction. In Proc. SGP.
[26]
Kehl, W., Navab, N., and Ilic, S. 2014. Coloured signed distance fields for full 3d object reconstruction. In Proc. BMVC.
[27]
Keller, M., Lefloch, D., Lambers, M., Izadi, S., Weyrich, T., and Kolb, A. 2013. Real-time 3d reconstruction in dynamic scenes using point-based fusion. In Proc. 3DV, 1--8.
[28]
Kopf, J., Cohen, M. F., Lischinski, D., and Uyttendaele, M. 2007. Joint bilateral upsampling. ACM TOG 26, 3.
[29]
Lee, K. J., Zhao, Q., Tong, X., Gong, M., Izadi, S., Lee, S. U., Tan, P., and Lin, S. 2012. Estimation of intrinsic image sequences from image+depth video. In Proc. ECCV, 327--340.
[30]
Levoy, M., Pulli, K., Curless, B., Rusinkiewicz, S., Koller, D., Pereira, L., Ginzton, M., Anderson, S., Davis, J., Ginsberg, J., Shade, J., and Fulk, D. 2000. The digital michelangelo project: 3d scanning of large statues. In Proc. SIGGRAPH, 131--144.
[31]
Lindner, M., Kolb, A., and Hartmann, K. 2007. Data-fusion of PMD-based distance-information and high-resolution RGB-images. In Proc. ISSCS, 121--124.
[32]
Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. IJCV 60, 2, 91--110.
[33]
Mulligan, J., and Brolly, X. 2004. Surface determination by photometric ranging. In Proc. CVPR Workshops.
[34]
Nair, R., Ruhl, K., Lenzen, F., Meister, S., Schäfer, H., Garbe, C. S., Eisemann, M., Magnor, M., and Kondermann, D. 2013. A survey on time-of-flight stereo fusion. In Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications. Springer, 105--127.
[35]
Nehab, D., Rusinkiewicz, S., Davis, J., and Ramamoorthi, R. 2005. Efficiently combining positions and normals for precise 3D geometry. Proc. SIGGRAPH 24, 3.
[36]
Newcombe, R. A., and Davison, A. J. 2010. Live dense reconstruction with a single moving camera. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, IEEE, 1498--1505.
[37]
Newcombe, R. A., Davison, A. J., Izadi, S., Kohli, P., Hilliges, O., Shotton, J., Molyneaux, D., Hodges, S., Kim, D., and Fitzgibbon, A. 2011. Kinectfusion: Real-time dense surface mapping and tracking. In Proc. ISMAR, IEEE, 127--136.
[38]
Niessner, M., Zollhöfer, M., Izadi, S., and Stamminger, M. 2013. Real-time 3d reconstruction at scale using voxel hashing. ACM TOG 32, 6, 169.
[39]
Park, J., Kim, H., Tai, Y.-W., Brown, M. S., and Kweon, I.-S. 2011. High quality depth map upsampling for 3d-tof cameras. In Proc. ICCV, IEEE, 1623--1630.
[40]
Pradeep, V., Rhemann, C., Izadi, S., Zach, C., Bleyer, M., and Bathiche, S. 2013. Monofusion: Real-time 3d reconstruction of small scenes with a single web camera. In Proc. ISMAR.
[41]
Prados, E., and Faugeras, O. 2005. Shape from shading: a well-posed problem? In Proc. CVPR.
[42]
Ramamoorthi, R., and Hanrahan, P. 2001. A signal-processing framework for inverse rendering. In Proc. SIGGRAPH, ACM, 117--128.
[43]
Richardt, C., Stoll, C., Dodgson, N. A., Seidel, H.-P., and Theobalt, C. 2012. Coherent spatiotemporal filtering, upsampling and rendering of RGBZ videos. CGF (Proceedings of Eurographics) 31, 2 (May).
[44]
Roth, H., and Vona, M. 2012. Moving volume kinectfusion. In BMVC, 1--11.
[45]
Rusinkiewicz, S., Hall-Holt, O., and Levoy, M. 2002. Real-time 3D model acquisition. ACM TOG 21, 3, 438--446.
[46]
Scharstein, D., Hirschmüller, H., Kitajima, Y., Krathwohl, G., Nešić, N., Wang, X., and Westling, P. 2014. High-resolution stereo datasets with subpixel-accurate ground truth. In Pattern Recognition. Springer, 31--42.
[47]
Schuon, S., Theobalt, C., Davis, J., and Thrun, S. 2009. Lidarboost: Depth superresolution for tof 3d shape scanning. In Proc. CVPR, IEEE, 343--350.
[48]
Seitz, S., Curless, B., Diebel, J., Scharstein, D., and Szeliski, R. 2006. A comparison and evaluation of multi-view stereo reconstruction algorithms. In Proc. CVPR, vol. 1, 519--528.
[49]
Snavely, N., Seitz, S. M., and Szeliski, R. 2006. Photo tourism: exploring photo collections in 3d. ACM TOG 25, 3, 835--846.
[50]
Triggs, B., Mclauchlan, P. F., Hartley, R. I., and Fitzgibbon, A. W. 2000. Bundle adjustment--a modern synthesis. In Vision algorithms: theory and practice. Springer, 298--372.
[51]
Weber, D., Bender, J., Schnoes, M., Stork, A., and Fellner, D. 2013. Efficient gpu data structures and methods to solve sparse linear systems in dynamics applications. In CGF, vol. 32, Wiley Online Library, 16--26.
[52]
Weise, T., Wismer, T., Leibe, B., and Van Gool, L. 2009. In-hand scanning with online loop closure. In ICCV Workshops, 1630--1637.
[53]
Whelan, T., Kaess, M., Fallon, M., Johannsson, H., Leonard, J., and McDonald, J. 2012. Kintinuous: Spatially extended kinectfusion.
[54]
Wu, C., Varanasi, K., Liu, Y., Seidel, H.-P., and Theobalt, C. 2011. Shading-based dynamic shape refinement from multi-view video under general illumination. In Proc. ICCV, IEEE, 1108--1115.
[55]
Wu, C., Stoll, C., Valgaerts, L., and Theobalt, C. 2013. On-set performance capture of multiple actors with a stereo camera. ACM TOG (Proc. SIGGRAPh Asia) 32, 6, 161.
[56]
Wu, C., Zollhöfer, M., Niessner, M., Stamminger, M., Izadi, S., and Theobalt, C. 2014. Real-time shading-based refinement for consumer depth cameras. ACM TOG (Proc. SIGGRAPH Asia) 33, 6, 200:1--200:10.
[57]
Yu, L.-F., Yeung, S.-K., Tai, Y.-W., and Lin, S. 2013. Shading-based shape refinement of rgb-d images. In Proc. CVPR.
[58]
Zhang, Z., Tsa, P.-S., Cryer, J. E., and Shah, M. 1999. Shape from shading: A survey. IEEE PAMI 21, 8, 690--706.
[59]
Zhou, Q.-Y., and Koltun, V. 2014. Color map optimization for 3d reconstruction with consumer depth cameras. ACM Transactions on Graphics (TOG) 33, 4, 155.
[60]
Zollhöfer, M., Niessner, M., Izadi, S., Rehmann, C., Zach, C., Fisher, M., Wu, C., Fitzgibbon, A., Loop, C., Theobalt, C., et al. 2014. Real-time non-rigid reconstruction using an rgb-d camera. ACM TOG (Proc. SIGGRAPH) 4.

Cited By

View all
  • (2024)3D Reconstruction with Fast Dipole SumsACM Transactions on Graphics10.1145/368791443:6(1-19)Online publication date: 19-Nov-2024
  • (2024)Generative Portrait Shadow RemovalACM Transactions on Graphics10.1145/368790343:6(1-13)Online publication date: 19-Dec-2024
  • (2024)IntrinsicDiffusion: Joint Intrinsic Layers from Latent Diffusion ModelsACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657472(1-11)Online publication date: 13-Jul-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics
ACM Transactions on Graphics  Volume 34, Issue 4
August 2015
1307 pages
ISSN:0730-0301
EISSN:1557-7368
DOI:10.1145/2809654
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 July 2015
Published in TOG Volume 34, Issue 4

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. 3D reconstruction
  2. shading-based refinement

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)38
  • Downloads (Last 6 weeks)2
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)3D Reconstruction with Fast Dipole SumsACM Transactions on Graphics10.1145/368791443:6(1-19)Online publication date: 19-Nov-2024
  • (2024)Generative Portrait Shadow RemovalACM Transactions on Graphics10.1145/368790343:6(1-13)Online publication date: 19-Dec-2024
  • (2024)IntrinsicDiffusion: Joint Intrinsic Layers from Latent Diffusion ModelsACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657472(1-11)Online publication date: 13-Jul-2024
  • (2024)A Neural Height-Map Approach for the Binocular Photometric Stereo Problem2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00159(1557-1566)Online publication date: 3-Jan-2024
  • (2024)CRefNet: Learning Consistent Reflectance Estimation With a Decoder-Sharing TransformerIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.333787030:9(6407-6420)Online publication date: Sep-2024
  • (2024)FastHuman: Reconstructing High-Quality Clothed Human in Minutes2024 International Conference on 3D Vision (3DV)10.1109/3DV62453.2024.00054(280-290)Online publication date: 18-Mar-2024
  • (2024)In-process 4D reconstruction in robotic additive manufacturingRobotics and Computer-Integrated Manufacturing10.1016/j.rcim.2024.10278489(102784)Online publication date: Oct-2024
  • (2023)DiffCompleteProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3669440(75951-75966)Online publication date: 10-Dec-2023
  • (2023)High-Quality RGB-D Reconstruction via Multi-View Uncalibrated Photometric Stereo and Gradient-SDF2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV56688.2023.00312(3105-3114)Online publication date: Jan-2023
  • (2023)Adaptive Joint Optimization for 3D Reconstruction With Differentiable RenderingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.314824529:6(3039-3051)Online publication date: 1-Jun-2023
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media