Abstract
This paper presents a near-light photometric stereo method for spatially varying reflectances. Recent studies in photometric stereo proposed learning-based approaches to handle diverse real-world reflectances and achieve high accuracy compared to conventional methods. However, they assume distant (i.e., parallel) lights, which can in practical settings only be approximately realized, and they fail in near-light conditions. Near-light photometric stereo methods address near-light conditions but previous works are limited to over-simplified reflectances, such as Lambertian reflectance. The proposed method takes a hybrid approach of distant- and near-light models, where the surface normal of a small area (corresponding to a pixel) is computed locally with a distant light assumption, and the reconstruction error is assessed based on a near-light image formation model. This paper is the first work to solve unknown, spatially varying, diverse reflectances in near-light photometric stereo.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
\(\max (0, x)\) is differentiable everywhere except at \(x=0\), which is in practice not a problem with numerical differentiation as in many other works.
- 2.
PyTorch v1.1.0: http://pytorch.org.
- 3.
Mitsuba v0.5.0: http://mitsuba-renderer.org.
References
Ackermann, J., Fuhrmann, S., Goesele, M.: Geometric point light source calibration. In: Vision, Modeling, and Visualization, pp. 161–168 (2013)
Ahmad, J., Sun, J., Smith, L., Smith, M.: An improved photometric stereo through distance estimation and light vector optimization from diffused maxima region. Pattern Recogn. Lett. 50, 15–22 (2014)
Blinn, J.F.: Models of light reflection for computer synthesized pictures. In: SIGGRAPH (1977)
Bony, A., Bringier, B., Khoudeir, M.: Tridimensional reconstruction by photometric stereo with near spot light sources. In: European Signal Processing Conference (2013)
Burley, B.: Physically-based shading at Disney. In: SIGGRAPH 2012 Course Notes (2012)
Chen, G., Han, K., Wong, K.Y.K.: PS-FCN: a flexible learning framework for photometric stereo. In: European Conference on Computer Vision (ECCV) (2018)
Chen, L., Zheng, Y., Shi, B., Subpa-Asa, A., Sato, I.: A microfacet-based reflectance model for photometric stereo with highly specular surfaces. In: International Conference on Computer Vision (ICCV) (2017)
Collins, T., Bartoli, A.: 3D reconstruction in laparoscopy with close-range photometric stereo. In: Ayache, N., Delingette, H., Golland, P., Mori, K. (eds.) MICCAI 2012. LNCS, vol. 7511, pp. 634–642. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33418-4_78
Georghiades, A.S.: Incorporating the Torrance and Sparrow model of reflectance in uncalibrated photometric stereo. In: International Conference on Computer Vision (ICCV) (2003)
Huang, X., Walton, M., Bearman, G., Cossairt, O.: Near light correction for image relighting and 3D shape recovery. In: 2015 Digital Heritage (2015)
Ikehata, S.: CNN-PS: CNN-based photometric stereo for general non-convex surfaces. In: European Conference on Computer Vision (ECCV) (2018)
Iwahori, Y., Sugie, H., Ishii, N.: Reconstructing shape from shading images under point light source illumination. In: International Conference on Pattern Recognition (ICPR) (1990)
Johnson, M.K., Adelson, E.H.: Shape estimation in natural illumination. In: Computer Vision and Pattern Recognition (CVPR) (2011)
Kingma, D., Ba, J.: Adam: a method for stochastic optimization (2014)
Ma, L., Liu, J., Pei, X., Hu, Y., Sun, F.: Calibration of position and orientation for point light source synchronously with single image in photometric stereo. Opt. Express 27(4), 4024–4033 (2019)
Matusik, W., Pfister, H., Brand, M., McMillan, L.: A data-driven reflectance model. Trans. Graph. (TOG) 22(3), 759–769 (2003)
Mecca, R., Quéau, Y.: Unifying diffuse and specular reflections for the photometric stereo problem. In: Winter Conference on Applications of Computer Vision (WACV) (2016)
Mecca, R., Rodolà, E., Cremers, D.: Realistic photometric stereo using partial differential irradiance equation ratios. Comput. Graph. 51, 8–16 (2015)
Mecca, R., Wetzler, A., Bruckstein, A.M., Kimmel, R.: Near field photometric stereo with point light sources. SIAM J. Imaging Sci. 7(4), 2732–2770 (2014)
Nie, Y., Song, Z.: A novel photometric stereo method with nonisotropic point light sources. In: International Conference on Pattern Recognition (ICPR), pp. 1737–1742. IEEE (2016)
Park, J., Sinha, S.N., Matsushita, Y., Tai, Y., Kweon, I.: Calibrating a non-isotropic near point light source using a plane. In: Computer Vision and Pattern Recognition (CVPR), pp. 2267–2274 (2014)
Quéau, Y., Durix, B., Wu, T., Cremers, D., Lauze, F., Durou, J.D.: LED-based photometric stereo: modeling, calibration and numerical solution. J. Math. Imaging Vis. 60(3), 313–340 (2018). https://doi.org/10.1007/s10851-017-0761-1
Quéau, Y., Durou, J.D., Aujol, J.F.: Variational methods for normal integration. J. Math. Imaging Vis. 60(4), 609–632 (2018). https://doi.org/10.1007/s10851-017-0777-6
Quéau, Y., Wu, T., Lauze, F., Durou, J.D., Cremers, D.: A non-convex variational approach to photometric stereo under inaccurate lighting. In: Computer Vision and Pattern Recognition (CVPR) (2017)
Rodolà, E., Albarelli, A., Bergamasco, F., Torsello, A.: A scale independent selection process for 3D object recognition in cluttered scenes. Int. J. Comput. Vis. (IJCV) 102(1–3), 129–145 (2013). https://doi.org/10.1007/s11263-012-0568-x
Santo, H., Samejima, M., Sugano, Y., Shi, B., Matsushita, Y.: Deep photometric stereo network. In: ICCV Workshop on Physics Based Vision meets Deep Learning (PBDL) (2017)
Santo, H., Waechter, M., Lin, W.Y., Sugano, Y., Matsushita, Y.: Light structure from pin motion: geometric point light source calibration. Int. J. Comput. Vis. (IJCV) 128(7), 1889–1912 (2020). https://doi.org/10.1007/s11263-020-01312-3
Shi, B., Mo, Z., Wu, Z., Duan, D., Yeung, S.K., Tan, P.: A benchmark dataset and evaluation for non-Lambertian and uncalibrated photometric stereo. Trans. Pattern Anal. Machi. Intell. (PAMI) 41(2), 271–284 (2019)
Shi, B., Tan, P., Matsushita, Y., Ikeuchi, K.: Bi-polynomial modeling of low-frequency reflectances. Trans. Pattern Anal. Machi. Intell. (PAMI) 36(6), 1078–1091 (2014)
Silver, W.M.: Determining shape and reflectance using multiple images. Master’s Thesis, Massachusetts Institute of Technology (1980)
Taniai, T., Maehara, T.: Neural inverse rendering for general reflectance photometric stereo. In: International Conference on Machine Learning (ICML) (2018)
Torrance, K.E., Sparrow, E.M.: Theory for off-specular reflection from roughened surfaces. JOSA 57(9), 1105–1114 (1967)
Turk, G., Levoy, M.: Zippered polygon meshes from range images. In: SIGGRAPH. ACM (1994)
Woodham, R.J.: Photometric method for determining surface orientation from multiple images. Opt. Eng. 19(1), 139–144 (1980)
Acknowledgment
This work was supported by JSPS KAKENHI Grant Number JP19H01123. Hiroaki Santo and Michael Waechter are grateful for support through a JSPS Research Fellowship for Young Scientists (JP19J10326) and JSPS Postdoctoral Fellowship (JP17F17350), respectively.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Santo, H., Waechter, M., Matsushita, Y. (2020). Deep Near-Light Photometric Stereo for Spatially Varying Reflectances. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12353. Springer, Cham. https://doi.org/10.1007/978-3-030-58598-3_9
Download citation
DOI: https://doi.org/10.1007/978-3-030-58598-3_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58597-6
Online ISBN: 978-3-030-58598-3
eBook Packages: Computer ScienceComputer Science (R0)