Skip to main content
SpringerLink
Log in

Shape and Albedo Recovery by Your Phone using Stereoscopic Flash and No-Flash Photography

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Recovering shape and albedo for the immense number of existing cultural heritage artifacts is challenging. Accurate 3D reconstruction systems are typically expensive and thus inaccessible to many and cheaper off-the-shelf 3D sensors often generate results of unsatisfactory quality. This paper presents a high-fidelity shape and albedo recovery method that only requires a stereo camera and a flashlight, a typical camera setup equipped in many off-the-shelf smartphones. The stereo camera allows us to infer rough shape from a pair of no-flash images, and a flash image is further captured for shape refinement based on our flash/no-flash image formation model. We verify the effectiveness of our method on real-world artifacts in indoor and outdoor conditions using smartphones with different camera/flashlight configurations. Comparison results demonstrate that our stereoscopic flash and no-flash photography benefits the high-fidelity shape and albedo recovery on a smartphone. Using our method, people can immediately turn their phones into high-fidelity 3D scanners, facilitating the digitization of cultural heritage artifacts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. The flashlight can cause shadows because in practice its location is non-identical to the camera’s optic center.

  2. “The Getty Caligula” by CosmoWenman / CC BY 4.0. https://sketchfab.com/3d-models/the-getty-caligula-6bd927a5779d479e83303635c79f81ac, last accessed on April 1, 2021

  3. Mitsuba Renderer. https://www.mitsuba-renderer.org/index_old.html, last accessed on April 1, 2021

  4. High-Resolution Light Probe Image Gallery. http://vgl.ict.usc.edu/Data/HighResProbes/, last accessed on April 1, 2021

  5. https://github.com/neycyanshi/DDRNet, last accessed on April 1, 2021

  6. AVDetphData. https://developer.apple.com/documentation/avfoundation/avdepthdata, last accessed on April 1, 2021.

  7. DepthSRfromShading. https://github.com/BjoernHaefner/DepthSRfromShading;

    Two-shot-BRDF-shape.  https://github.com/NVlabs/two-shot-brdf-shape, last accessed on April 1, 2021.

References

  • Aittala, M., Aila, T., & Lehtinen, J. (2016). Reflectance modeling by neural texture synthesis. ACM Transactions on Graphics (Proceedings of the ACM SIGGRAPH), 35(4), 1–13.

    Article  Google Scholar 

  • Aittala, M., Weyrich, T., & Lehtinen, J. (2015). Two-shot SVBRDF capture for stationary materials. ACM Transactions on Graphics (Proceedings of the ACM SIGGRAPH), 34(4), 110–1.

    MATH  Google Scholar 

  • Basri, R., & Jacobs, D. W. (2003). Lambertian reflectance and linear subspaces. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 25(2), 218–233.

    Article  Google Scholar 

  • Basri, R., Jacobs, D., & Kemelmacher, I. (2007). Photometric stereo with general, unknown lighting. International Journal of Computer Vision (IJCV), 72(3), 239–257.

    Article  Google Scholar 

  • Boss, M., Jampani, V., Kim, K., Lensch, H., & Kautz, J. (2020). Two-shot spatially-varying BRDF and shape estimation. In Proceedings of the computer vision and pattern recognition (CVPR).

  • Brandt, A. (1977). Multi-level adaptive solutions to boundary-value problems. Mathematics of Computation, 31(138), 333–390.

    Article  MathSciNet  MATH  Google Scholar 

  • Cao, X., Shi, B., Okura, F., & Matsushita, Y. (2021). Normal integration via inverse plane fitting with minimum point-to-plane distance. In Proceedings of the computer vision and pattern recognition (CVPR).

  • Cao, X., Waechter, M., Shi, B., Gao, Y., Zheng, B., & Matsushita, Y. (2020). Stereoscopic flash and no-flash photography for shape and albedo recovery. In Proceedings of the of computer vision and pattern recognition (CVPR).

  • Choe, G., Park, J., Tai, Y.W., & So Kweon, I. (2014). Exploiting shading cues in Kinect IR images for geometry refinement. In Proceedings of the computer vision and pattern recognition (CVPR).

  • Cook, R. L., & Torrance, K. E. (1982). A reflectance model for computer graphics. ACM Transactions on Graphics (ToG), 1(1), 7–24.

    Article  Google Scholar 

  • Deschaintre, V., Aittala, M., Durand, F., Drettakis, G., & Bousseau, A. (2018). Single-image SVBRDF capture with a rendering-aware deep network. ACM Transactions on Graphics (Proceedings of the ACM SIGGRAPH), 37(4), 1–15.

    Article  Google Scholar 

  • Eisemann, E., & Durand, F. (2004). Flash photography enhancement via intrinsic relighting. ACM Transactions on Graphics (Proceedings of the ACM SIGGRAPH)., 23(3), 673–678.

    Article  Google Scholar 

  • Feris, R., Raskar, R., Chen, L., Tan, K.H., & Turk, M. (2005). Discontinuity preserving stereo with small baseline multi-flash illumination. In Proceedings of the international conference on computer vision (ICCV).

  • Gallardo, M., Collins, T., & Bartoli, A. (2017). Dense non-rigid structure-from-motion and shading with unknown albedos. In Proceedings of the international conference on computer vision (ICCV), (pp. 3884–3892).

  • Häfner, B., Quéau, Y., Möllenhoff, T., & Cremers, D. (2018). Fight ill-posedness with ill-posedness: Single-shot variational depth super-resolution from shading. In Proceedings of computer vision and pattern recognition (CVPR).

  • Häfner, B., Peng, S., Verma, A., Quéau, Y., & Cremers, D. (2019). Photometric depth super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 42(10), 2453–2464.

    Article  Google Scholar 

  • Han, Y., Lee, J.Y., & So Kweon, I. (2013). High quality shape from a single RGB-D image under uncalibrated natural illumination. In Proceedings of the international conference on computer vision (ICCV).

  • Haque, S.M., Chatterjee, A., & Madhav Govindu, V. (2014). High quality photometric reconstruction using a depth camera. In Proceedings of computer vision and pattern recognition (CVPR).

  • He, S., & Lau, R.W. (2014). Saliency detection with flash and no-flash image pairs. In Proceedings of the european conference on computer vision (ECCV).

  • Hoppe, H., DeRose, T., Duchamp, T., McDonald, J., & Stuetzle, W. (1992). Surface reconstruction from unorganized points. In Proceedings of the ACM SIGGRAPH.

  • Ikeuchi, K. (1987). Determining a depth map using a dual photometric stereo. The International Journal of Robotics Research, 6(1), 15–31.

    Article  Google Scholar 

  • Johnson, M.K., & Adelson, E.H. (2011). Shape estimation in natural illumination. In Proceedings of Computer Vision and Pattern Recognition (CVPR).

  • Klowsky, R., Kuijper, A., & Goesele, M. (2012). Modulation transfer function of patch-based stereo systems. In Proceedings of computer vision and pattern recognition (CVPR).

  • Li, Z., Sunkavalli, K., & Chandraker, M. (2018a). Materials for masses: SVBRDF acquisition with a single mobile phone image. In Proceedings of the european conference on computer vision (ECCV), (pp 72–87).

  • Liu, D. C., & Nocedal, J. (1989). On the limited memory BFGS method for large scale optimization. Mathematical Programming, 45(1–3), 503–528.

    Article  MathSciNet  MATH  Google Scholar 

  • Li, Z., Xu, Z., Ramamoorthi, R., Sunkavalli, K., & Chandraker, M. (2018). Learning to reconstruct shape and spatially-varying reflectance from a single image. ACM Transactions on Graphics (Proceedings of the ACM SIGGRAPH), 37(6), 1–11.

    Google Scholar 

  • Maier, R., Kim, K., Cremers, D., Kautz, J., & Nießner, M. (2017). Intrinsic3D: High-quality 3D reconstruction by joint appearance and geometry optimization with spatially-varying lighting. In Proceedings of the international conference on computer vision (ICCV).

  • Maurer, D., Ju, Y. C., Breuß, M., & Bruhn, A. (2018). Combining shape from shading and stereo: A joint variational method for estimating depth, illumination and albedo. International Journal of Computer Vision (IJCV), 126(12), 1342–1366.

    Article  Google Scholar 

  • Or-El, R., Rosman, G., Wetzler, A., Kimmel, R., & Bruckstein, A.M. (2015). RGBD-Fusion: Real-time high precision depth recovery. In Proceedings of computer vision and pattern recognition (CVPR).

  • Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., & Toyama, K. (2004). Digital photography with flash and no-flash image pairs. ACM Transactions on Graphics (Proc of the ACM SIGGRAPH).

  • Quéau, Y., Mélou, J., Castan, F., Cremers, D., & Durou, J.D. (2017). A variational approach to shape-from-shading under natural illumination. In International workshop on energy minimization methods in computer vision and pattern recognition.

  • Quéau, Y., Durou, J. D., & Aujol, J. F. (2018). Normal integration: A survey. Journal of Mathematical Imaging and Vision, 60(4), 576–593.

    Article  MathSciNet  MATH  Google Scholar 

  • Ramamoorthi, R., & Hanrahan, P. (2001). A signal-processing framework for inverse rendering. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques (pp. 117–128).

  • Sun, J., Kang, S.B., Xu, Z.B., Tang, X., & Shum, H.Y. (2007). Flash cut: Foreground extraction with flash and no-flash image pairs. In Proceedings of computer vision and pattern recognition (CVPR).

  • Sun, J., Li, Y., Kang, S.B., & Shum, H.Y. (2006). Flash matting. ACM Transactions on Graphics (Proc of the ACM SIGGRAPH).

  • Wu, C., Varanasi, K., Liu, Y., Seidel, H.P., & Theobalt, C. (2011a). Shading-based dynamic shape refinement from multi-view video under general illumination. In Proceedings of the international conference on computer vision (ICCV).

  • Wu, C., Wilburn, B., Matsushita, Y., & Theobalt, C. (2011b). High-quality shape from multi-view stereo and shading under general illumination. In Proceedings of computer vision and pattern recognition (CVPR).

  • Wu, C., Zollhöfer, M., Nießner, M., Stamminger, M., Izadi, S., & Theobalt, C. (2014). Real-time shading-based refinement for consumer depth cameras. ACM Transactions on Graphics (Proc of the ACM SIGGRAPH), 33(6), 200.

    Google Scholar 

  • Yan, S., Wu, C., Wang, L., Xu, F., An, L., Guo, K., & Liu, Y. (2018). DDRNet: Depth map denoising and refinement for consumer depth cameras using cascaded cnns. In Proceedings of the european conference on computer vision (ECCV).

  • Yu, L.F., Yeung, S.K., Tai, Y.W., & Lin, S. (2013). Shading-based shape refinement of RGB-D images. In Proceedings of computer vision and pattern recognition (CVPR).

  • Zhang, Q., Ye, M., Yang, R., Matsushita, Y., Wilburn, B., & Yu, H. (2012). Edge-preserving photometric stereo via depth fusion. In Proceedings of computer vision and pattern recognition (CVPR).

  • Zhou, C., Troccoli, A., & Pulli, K. (2012). Robust stereo with flash and no-flash image pairs. In Proceedings of computer vision and pattern recognition (CVPR).

Download references

Acknowledgements

This work was supported by JSPS KAKENHI Grant Number JP19H01123, JSPS postdoctoral fellowship (JP17F17350), and National Natural Science Foundation of China under Grant Number 62136001 and 61872012.

Author information

Authors and Affiliations

  1. Graduate School of Information Science and Technology, Osaka University, Osaka, Japan

    Xu Cao, Michael Waechter, Fumio Okura & Yasuyuki Matsushita

  2. Department of Computer Science and Technology, Peking University, Beijing, China

    Boxin Shi

  3. Peng Cheng Laboratory, Shenzhen, China

    Boxin Shi

  4. Huawei Technologies Co., Ltd., Shenzhen, China

    Ye Gao & Bo Zheng

Authors
  1. Xu Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael Waechter
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Boxin Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ye Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bo Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Fumio Okura
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yasuyuki Matsushita
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xu Cao.

Additional information

Communicated by Katsushi Ikeuchi.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cao, X., Waechter, M., Shi, B. et al. Shape and Albedo Recovery by Your Phone using Stereoscopic Flash and No-Flash Photography. Int J Comput Vis 130, 1403–1415 (2022). https://doi.org/10.1007/s11263-022-01597-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-022-01597-6

Keywords

Search

Navigation