Skip to main content
SpringerLink
Log in

An MRF Optimisation Framework for Full 3D Reconstruction of Scenes with Complex Reflectance

  • Conference paper
  • First Online:
Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2019)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1182))

Abstract

The ability to digitise real objects is fundamental in applications such as film post-production, cultural heritage preservation and video game development. While many existing modelling techniques achieve impressive results, they are often reliant on assumptions such as prior knowledge of the scene’s surface reflectance. This considerably restricts the range of scenes that can be reconstructed, as these assumptions are often violated in practice. One technique that allows surface reconstruction regardless of the scene’s reflectance model is Helmholtz Stereopsis (HS). However, to date, research on HS has mostly been limited to 2.5D scene reconstruction. In this paper, a framework is introduced to perform full 3D HS using Markov Random Field (MRF) optimisation for the first time. The paper introduces two complementary techniques. The first approach computes multiple 2.5D reconstructions from a small number of viewpoints and fuses these together to obtain a complete model, while the second approach directly reasons in the 3D domain by performing a volumetric MRF optimisation. Both approaches are based on optimising an energy function combining an HS confidence measure and normal consistency across the reconstructed surface. The two approaches are evaluated on both synthetic and real scenes, measuring the accuracy and completeness obtained. Further, the effect of noise on modelling accuracy is experimentally evaluated on the synthetic dataset. Both techniques achieve sub-millimetre accuracy and exhibit robustness to noise. In particular, the method based on full 3D optimisation is shown to significantly outperform the other approach.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Addari, G., Guillemaut, J.Y.: An MRF optimisation framework for full 3D Helmholtz stereopsis. In: 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, pp. 725–736, January 2019. https://doi.org/10.5220/0007407307250736

  2. Andres, B., Kappes, J.H., Beier, T., Köthe, U., Hamprecht, F.A.: The lazy flipper: efficient depth-limited exhaustive search in discrete graphical models. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, Part VII, vol. 7578, pp. 154–166. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33786-4_12

    Chapter  Google Scholar 

  3. Besag, J.: On the statistical analysis of dirty pictures. J. Roy. Stat. Soc. B 48(3), 48–259 (1986)

    MathSciNet  MATH  Google Scholar 

  4. Chandraker, M., Bai, J., Ramamoorthi, R.: On differential photometric reconstruction for unknown, isotropic brdfs. IEEE Trans. Pattern Anal. Mach. Intell. 35(12), 2941–2955 (2013). https://doi.org/10.1109/TPAMI.2012.217

    Article  Google Scholar 

  5. Delaunoy, A., Prados, E., Belhumeur, P.N.: Towards full 3D Helmholtz stereovision algorithms. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010. LNCS, Part I, vol. 6492, pp. 39–52. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-19315-6_4

    Chapter  Google Scholar 

  6. Goldman, D.B., Curless, B., Hertzmann, A., Seitz, S.M.: Shape and spatially-varying BRDFs from photometric stereo. IEEE Trans. Pattern Anal. Mach. Intell. 32(6), 1060–1071 (2010). https://doi.org/10.1109/TPAMI.2009.102. http://dx.doi.org/10.1109/TPAMI.2009.102

    Article  Google Scholar 

  7. Guillemaut, J., Drbohlav, O., Illingworth, J., Sára, R.: A maximum likelihood surface normal estimation algorithm for Helmholtz stereopsis. In: VISAPP 2008: Proceedings of the Third International Conference on Computer Vision Theory and Applications 2008, vol. 2, pp. 352–359 (2008)

    Google Scholar 

  8. Guillemaut, J.Y., Drbohlav, O., Sára, R., Illingworth, J.: Helmholtz stereopsis on rough and strongly textured surfaces. In: 3DPVT, pp. 10–17. IEEE Computer Society (2004)

    Google Scholar 

  9. Han, T., Shen, H.: Photometric stereo for general BRDFs via reflection sparsity modeling. IEEE Trans. Image Process. 24(12), 4888–4903 (2015). https://doi.org/10.1109/TIP.2015.2471081

    Article  MathSciNet  MATH  Google Scholar 

  10. Ishikawa, H.: Higher-order clique reduction without auxiliary variables. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1362–1369, June 2014. https://doi.org/10.1109/CVPR.2014.177

  11. Janko, Z., Drbohlav, O., Sara, R.: Radiometric calibration of a Helmholtz stereo rig. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. I-166, January 2004

    Google Scholar 

  12. Kazhdan, M., Bolitho, M., Hoppe, H.: Poisson surface reconstruction. In: Proceedings of the Fourth Eurographics Symposium on Geometry Processing, SGP 2006, pp. 61–70. Eurographics Association (2006). http://dl.acm.org/citation.cfm?id=1281957.1281965

  13. Kolmogorov, V.: Convergent tree-reweighted message passing for energy minimization. IEEE Trans. Pattern Anal. Mach. Intell. 28(10), 1568–1583 (2006). https://doi.org/10.1109/TPAMI.2006.200. http://dx.doi.org/10.1109/TPAMI.2006.200

    Article  Google Scholar 

  14. Krishnamurthy, V., Levoy, M.: Fitting smooth surfaces to dense polygon meshes. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1996, pp. 313–324 (1996). https://doi.org/10.1145/237170.237270, http://doi.acm.org/10.1145/237170.237270

  15. Laurentini, A.: The visual hull concept for silhouette-based image understanding. IEEE Trans. Pattern Anal. Mach. Intell. 16(2), 150–162 (1994). https://doi.org/10.1109/34.273735. http://dx.doi.org/10.1109/34.273735

    Article  Google Scholar 

  16. Lewis, R.R.: Making shaders more physically plausible. In: Fourth Eurographics Workshop on Rendering, pp. 47–62 (1994)

    Article  Google Scholar 

  17. Liang, C., Wong, K.Y.K.: 3D reconstruction using silhouettes from unordered viewpoints. Image Vision Comput. 28(4), 579–589 (2010). https://doi.org/10.1016/j.imavis.2009.09.012. http://dx.doi.org/10.1016/j.imavis.2009.09.012

    Article  Google Scholar 

  18. Lombardi, S., Nishino, K.: Radiometric scene decomposition: scene reflectance, illumination, and geometry from RGB-D images. CoRR abs/1604.01354 (2016). http://arxiv.org/abs/1604.01354

  19. Magda, S., Kriegman, D.J., Zickler, T.E., Belhumeur, P.N.: Beyond lambert: reconstructing surfaces with arbitrary brdfs. In: ICCV (2001)

    Google Scholar 

  20. Nasrin, R., Jabbar, S.: Efficient 3D visual hull reconstruction based on marching cube algorithm. In: 2015 International Conference on Innovations in Information, Embedded and Communication Systems (ICIIECS), pp. 1–6, March 2015. https://doi.org/10.1109/ICIIECS.2015.7193189

  21. Nishino, K.: Directional statistics BRDF model. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 476–483, September 2009. https://doi.org/10.1109/ICCV.2009.5459255

  22. Oxholm, G., Nishino, K.: Shape and reflectance estimation in the wild. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 376–389 (2016). https://doi.org/10.1109/TPAMI.2015.2450734

    Article  Google Scholar 

  23. Roubtsova, N., Guillemaut, J.: Bayesian Helmholtz stereopsis with integrability prior. IEEE Trans. Pattern Anal. Mach. Intell. 40(9), 2265–2272 (2018). https://doi.org/10.1109/TPAMI.2017.2749373

    Article  Google Scholar 

  24. Roubtsova, N., Guillemaut, J.Y.: Colour Helmholtz stereopsis for reconstruction of dynamic scenes with arbitrary unknown reflectance. Int. J. Comput. Vision 124(1), 18–48 (2017). https://doi.org/10.1007/s11263-016-0951-0

    Article  MathSciNet  Google Scholar 

  25. Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2006, vol. 1, pp. 519–528. IEEE Computer Society, Washington, DC, USA (2006). https://doi.org/10.1109/CVPR.2006.19, http://dx.doi.org/10.1109/CVPR.2006.19

  26. Szeliski, R., et al.: A comparative study of energy minimization methods for markov random fields with smoothness-based priors. IEEE Trans. Pattern Anal. Mach. Intell. 30(6), 1068–1080 (2008)

    Article  Google Scholar 

  27. Tu, P., Mendonca, P.R.S.: Surface reconstruction via Helmholtz reciprocity with a single image pair. In: 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings, vol. 1, pp. I-541–I-547, June 2003. https://doi.org/10.1109/CVPR.2003.1211401

  28. Turk, G., Levoy, M.: Zippered polygon meshes from range images. In: Proceedings of the 21st Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1994, pp. 311–318. ACM, New York (1994). https://doi.org/10.1145/192161.192241, http://doi.acm.org/10.1145/192161.192241

  29. Vogiatzis, G., Hernandez, C., Cipolla, R.: Reconstruction in the round using photometric normals and silhouettes. In: Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2006, vol. 2, pp. 1847–1854. IEEE Computer Society, Washington, DC, USA (2006). https://doi.org/10.1109/CVPR.2006.245, http://dx.doi.org/10.1109/CVPR.2006.245

  30. Von Helmholtz, H., Southall, J.P.: Helmholtz’s treatise on physiological optics, vol. 1. Optical Society of America, New York (1924). Translation

    Google Scholar 

  31. Weinmann, M., Ruiters, R., Osep, A., Schwartz, C., Klein, R.: Fusing structured light consistency and Helmholtz normals for 3D reconstruction. In: British Machine Vision Conference, September 2012. accepted for publication

    Google Scholar 

  32. Woodham, R.J.: Photometric method for determining surface orientation from multiple images. Opt. Eng. 19(1), 191139 (1980). https://doi.org/10.1117/12.7972479. http://dx.doi.org/10.1117/12.7972479

    Article  Google Scholar 

  33. Zickler, T.E., Ho, J., Kriegman, D.J., Ponce, J., Belhumeur, P.N.: Binocular Helmholtz stereopsis. In: Proceedings Ninth IEEE International Conference on Computer Vision, vol. 2, pp. 1411–1417, October 2003. https://doi.org/10.1109/ICCV.2003.1238655

  34. Zickler, T.: Reciprocal image features for uncalibrated Helmholtz stereopsis. In: IEEE Computer Vision and Pattern Recognitiion, pp. II: 1801–1808 (2006). http://www.cs.virginia.edu/~mjh7v/bib/Zickler06.pdf

  35. Zickler, T.E., Belhumeur, P.N., Kriegman, D.J.: Helmholtz stereopsis: exploiting reciprocity for surface reconstruction. Int. J. Comput. Vision 49(2), 215–227 (2002). https://doi.org/10.1023/A:1020149707513. http://dx.doi.org/10.1023/A:1020149707513

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, Guildford, GU2 7XH, United Kingdom

    Gianmarco Addari & Jean-Yves Guillemaut

Authors
  1. Gianmarco Addari
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jean-Yves Guillemaut
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gianmarco Addari .

Editor information

Editors and Affiliations

  1. University of Lisbon, Lisbon, Portugal

    Ana Paula Cláudio

  2. University of Rennes 1, Rennes, France

    Kadi Bouatouch

  3. University of Genoa, Genoa, Italy

    Manuela Chessa

  4. Mines ParisTech, Paris, France

    Alexis Paljic

  5. Linnaeus University, Växjö, Sweden

    Andreas Kerren

  6. French Civil Aviation University (ENAC), Toulouse, France

    Christophe Hurter

  7. University Jean Monnet, Saint-Etienne, France

    Alain Tremeau

  8. University of Catania, Catania, Italy

    Giovanni Maria Farinella

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Addari, G., Guillemaut, JY. (2020). An MRF Optimisation Framework for Full 3D Reconstruction of Scenes with Complex Reflectance. In: Cláudio, A., et al. Computer Vision, Imaging and Computer Graphics Theory and Applications. VISIGRAPP 2019. Communications in Computer and Information Science, vol 1182. Springer, Cham. https://doi.org/10.1007/978-3-030-41590-7_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-41590-7_19

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-41589-1

  • Online ISBN: 978-3-030-41590-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation