Skip to main content
Springer Nature Link
Log in

Capture, Reconstruction, and Representation of the Visual Real World for Virtual Reality

  • Chapter
  • First Online:
Real VR – Immersive Digital Reality

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 11900))

Abstract

We provide an overview of the concerns, current practice, and limitations for capturing, reconstructing, and representing the real world visually within virtual reality. Given that our goals are to capture, transmit, and depict complex real-world phenomena to humans, these challenges cover the opto-electro-mechanical, computational, informational, and perceptual fields. Practically producing a system for real-world VR capture requires navigating a complex design space and pushing the state of the art in each of these areas. As such, we outline several promising directions for future work to improve the quality and flexibility of real-world VR capture systems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Aggarwal, R., Vohra, A., Namboodiri, A.M.: Panoramic stereo videos with a single camera. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3755–3763, June 2016. https://doi.org/10.1109/CVPR.2016.408

  2. Aliev, K.A., Ulyanov, D., Lempitsky, V.: Neural point-based graphics (2019). arXiv:1906.08240

  3. Anderson, R., et al.: Jump: virtual reality video. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 35(6), 198:1–198:13 (2016). https://doi.org/10.1145/2980179.2980257

    Article  Google Scholar 

  4. Bau, D., et al.: Seeing what a GAN cannot generate. In: Proceedings of the International Conference on Computer Vision (ICCV) (2019)

    Google Scholar 

  5. Bertel, T., Campbell, N.D.F., Richardt, C.: MegaParallax: casual 360\(^\circ \) panoramas with motion parallax. IEEE Trans. Visual Comput. Graphics 25(5), 1828–1835 (2019). https://doi.org/10.1109/TVCG.2019.2898799

    Article  Google Scholar 

  6. Brown, M., Lowe, D.G.: Automatic panoramic image stitching using invariant features. Int. J. Comput. Vis. 74(1) (2007). https://doi.org/10.1007/s11263-006-0002-3

  7. Buehler, C., Bosse, M., McMillan, L., Gortler, S., Cohen, M.: Unstructured lumigraph rendering. In: Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pp. 425–432 (2001). https://doi.org/10.1145/383259.383309

  8. Bussone, W.: Linear and angular head accelerations in daily life. Ph.D. thesis, Virginia Tech (2005)

    Google Scholar 

  9. Cabral, B.: VR capture: designing and building an open source 3D-360 video camera. In: SIGGRAPH Asia Keynote, December 2016

    Google Scholar 

  10. Chai, J.X., Tong, X., Chan, S.C., Shum, H.Y.: Plenoptic sampling. In: Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pp. 307–318 (2000). https://doi.org/10.1145/344779.344932

  11. Chaurasia, G., Duchêne, S., Sorkine-Hornung, O., Drettakis, G.: Depth synthesis and local warps for plausible image-based navigation. ACM Trans. Graph. 32(3), 30:1–30:12 (2013). https://doi.org/10.1145/2487228.2487238

    Article  Google Scholar 

  12. Chaurasia, G., Sorkine-Hornung, O., Drettakis, G.: Silhouette-aware warping for image-based rendering. Comput. Graph. Forum (Proc. Eurographics Symp. Rendering) 30(4), 1223–1232 (2011). https://doi.org/10.1111/j.1467-8659.2011.01981.x

    Article  Google Scholar 

  13. Chen, S.E., Williams, L.: View interpolation for image synthesis. In: Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pp. 279–288 (1993). https://doi.org/10.1145/166117.166153

  14. Cohen, T.S., Welling, M.: Transformation properties of learned visual representations. In: Proceedings of the International Conference on Learning Representations (ICLR) (2015)

    Google Scholar 

  15. Collet, A., et al.: High-quality streamable free-viewpoint video. ACM Trans. Graph. (Proc. SIGGRAPH) 34(4), 69:1–69:13 (2015). https://doi.org/10.1145/2766945

    Article  Google Scholar 

  16. Curless, B., Seitz, S., Bouguet, J.Y., Debevec, P., Levoy, M., Nayar, S.K.: 3D photography. In: SIGGRAPH Courses (2000). http://www.cs.cmu.edu/~seitz/course/3DPhoto.html

  17. Dai, A., Nießner, M., Zollhöfer, M., Izadi, S., Theobalt, C.: BundleFusion: real-time globally consistent 3D reconstruction using on-the-fly surface reintegration. ACM Trans. Graph. 36(3), 24:1–24:18 (2017). https://doi.org/10.1145/3054739

    Article  Google Scholar 

  18. Davis, A., Levoy, M., Durand, F.: Unstructured light fields. Comput. Graph. Forum (Proc. Eurographics) 31(2), 305–314 (2012). https://doi.org/10.1111/j.1467-8659.2012.03009.x

    Article  Google Scholar 

  19. Debevec, P.: The light stages and their applications to photoreal digital actors. In: SIGGRAPH Asia Technical Briefs (2012)

    Google Scholar 

  20. Debevec, P., Bregler, C., Cohen, M.F., McMillan, L., Sillion, F., Szeliski, R.: Image-based modeling, rendering, and lighting. In: SIGGRAPH Courses (2000). https://www.pauldebevec.com/IBMR99/

  21. Debevec, P.E., Taylor, C.J., Malik, J.: Modeling and rendering architecture from photographs: a hybrid geometry- and image-based approach. In: Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pp. 11–20, August 1996. https://doi.org/10.1145/237170.237191

  22. Dosovitskiy, A., Springenberg, J.T., Tatarchenko, M., Brox, T.: Learning to generate chairs, tables and cars with convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 692–705 (2017). https://doi.org/10.1109/TPAMI.2016.2567384

    Article  Google Scholar 

  23. DXOMARK: RED Helium 8K DxOMark sensor score: 108—a new all-time-high score! https://www.dxomark.com/red-helium-8k-dxomark-sensor-score-108-a-new-all-time-high-score2/. Accessed 30 Oct 2019

  24. Eslami, S.M.A., et al.: Neural scene representation and rendering. Science 360(6394), 1204–1210 (2018). https://doi.org/10.1126/science.aar6170

    Article  Google Scholar 

  25. Flynn, J., et al.: DeepView: view synthesis with learned gradient descent. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2367–2376, June 2019

    Google Scholar 

  26. Flynn, J., Neulander, I., Philbin, J., Snavely, N.: DeepStereo: learning to predict new views from the world’s imagery. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5515–5524, June 2016. https://doi.org/10.1109/CVPR.2016.595

  27. Fuhrmann, S., Langguth, F., Goesele, M.: MVE: a multi-view reconstruction environment. In: Proceedings of the Eurographics Workshop on Graphics and Cultural Heritage, pp. 11–18 (2014). https://doi.org/10.2312/gch.20141299

  28. Galliani, S., Lasinger, K., Schindler, K.: Massively parallel multiview stereopsis by surface normal diffusion. In: Proceedings of the International Conference on Computer Vision (ICCV), pp. 873–881, December 2015. https://doi.org/10.1109/ICCV.2015.106

  29. Garon, M., Sunkavalli, K., Hadap, S., Carr, N., Lalonde, J.F.: Fast spatially-varying indoor lighting estimation. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  30. Gortler, S.J., Grzeszczuk, R., Szeliski, R., Cohen, M.F.: The lumigraph. In: Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pp. 43–54, August 1996. https://doi.org/10.1145/237170.237200

  31. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press (2004). https://doi.org/10.1017/CBO9780511811685

  32. Hedman, P., Alsisan, S., Szeliski, R., Kopf, J.: Casual 3D photography. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 36(6), 234:1–234:15 (2017). https://doi.org/10.1145/3130800.3130828

    Article  Google Scholar 

  33. Hedman, P., Kopf, J.: Instant 3D photography. ACM Trans. Graph. (Proc. SIGGRAPH) 37(4), 101:1–101:12 (2018). https://doi.org/10.1145/3197517.3201384

    Article  Google Scholar 

  34. Hedman, P., Philip, J., Price, T., Frahm, J.M., Drettakis, G.: Deep blending for free-viewpoint image-based rendering. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 37(6), 257:1–257:15 (2018). https://doi.org/10.1145/3272127.3275084

    Article  Google Scholar 

  35. Hedman, P., Ritschel, T., Drettakis, G., Brostow, G.: Scalable inside-out image-based rendering. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 35(6), 231:1–231:11 (2016). https://doi.org/10.1145/2980179.2982420

    Article  Google Scholar 

  36. Huang, J., Chen, Z., Ceylan, D., Jin, H.: 6-DOF VR videos with a single 360-camera. In: Proceedings of IEEE Virtual Reality (VR), pp. 37–44, March 2017. https://doi.org/10.1109/VR.2017.7892229

  37. Huang, P.H., Matzen, K., Kopf, J., Ahuja, N., Huang, J.B.: DeepMVS: learning multi-view stereopsis. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

    Google Scholar 

  38. Hukkelås, H., Mester, R., Lindseth, F.: DeepPrivacy: a generative adversarial network for face anonymization. In: Bebis, G., et al. (eds.) ISVC 2019. LNCS, vol. 11844, pp. 565–578. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-33720-9_44

    Chapter  Google Scholar 

  39. Ishiguro, H., Yamamoto, M., Tsuji, S.: Omni-directional stereo. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 257–262 (1992). https://doi.org/10.1109/34.121792

    Article  Google Scholar 

  40. Jahanian, A., Chai, L., Isola, P.: On the “steerability” of generative adversarial networks (2019). arXiv:1907.07171

  41. Jancosek, M., Pajdla, T.: Multi-view reconstruction preserving weakly-supported surfaces. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3121–3128, June 2011. https://doi.org/10.1109/CVPR.2011.5995693

  42. Ji, D., Kwon, J., McFarland, M., Savarese, S.: Deep view morphing. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7092–7100, July 2017. https://doi.org/10.1109/CVPR.2017.750

  43. Kalantari, N.K., Wang, T.C., Ramamoorthi, R.: Learning-based view synthesis for light field cameras. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 35(6), 193:1–193:10 (2016). https://doi.org/10.1145/2980179.2980251

    Article  Google Scholar 

  44. Keysers, C., Xiao, D.K., Földiák, P., Perrett, D.I.: The speed of sight. J. Cogn. Neurosci. 13(1), 90–101 (2001). https://doi.org/10.1162/089892901564199

    Article  Google Scholar 

  45. Kim, C., Zimmer, H., Pritch, Y., Sorkine-Hornung, A., Gross, M.: Scene reconstruction from high spatio-angular resolution light fields. ACM Trans. Graph. (Proc. SIGGRAPH) 32(4), 73:1–73:12 (2013). https://doi.org/10.1145/2461912.2461926

    Article  MATH  Google Scholar 

  46. Kim, H., et al.: Deep video portraits. ACM Trans. Graph. (Proc. SIGGRAPH) 37(4), 163:1–163:14 (2018). https://doi.org/10.1145/3197517.3201283

    Article  Google Scholar 

  47. Kirillov, A., He, K., Girshick, R., Rother, C., Dollár, P.: Panoptic segmentation. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  48. Konrad, R., Dansereau, D.G., Masood, A., Wetzstein, G.: SpinVR: towards live-streaming 3D virtual reality video. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 36(6), 209:1–209:12 (2017). https://doi.org/10.1145/3130800.3130836

    Article  Google Scholar 

  49. Kopf, J., et al.: Practical 3D photography. In: Proceedings of CVPR Workshops (2019)

    Google Scholar 

  50. Koulieris, G.A., Akşit, K., Stengel, M., Mantiuk, R.K., Mania, K., Richardt, C.: Near-eye display and tracking technologies for virtual and augmented reality. Comput. Graph. Forum 38(2), 493–519 (2019). https://doi.org/10.1111/cgf.13654

    Article  Google Scholar 

  51. Kulkarni, T.D., Whitney, W., Kohli, P., Tenenbaum, J.B.: Deep convolutional inverse graphics network. In: Advances in Neural Information Processing Systems (NIPS) (2015)

    Google Scholar 

  52. Lanman, D., Wetzstein, G., Hirsch, M., Heidrich, W., Raskar, R.: Polarization fields: dynamic light field display using multi-layer LCDs. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 30(6), 186:1–186:10 (2011). https://doi.org/10.1145/2070781.2024220

    Article  Google Scholar 

  53. Lee, J., Kim, B., Kim, K., Kim, Y., Noh, J.: Rich360: optimized spherical representation from structured panoramic camera arrays. ACM Trans. Graph. (Proc. SIGGRAPH) 35(4), 63:1–63:11 (2016). https://doi.org/10.1145/2897824.2925983

    Article  Google Scholar 

  54. LeGendre, C., et al.: DeepLight: learning illumination for unconstrained mobile mixed reality. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  55. Levoy, M., Hanrahan, P.: Light field rendering. In: Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pp. 31–42, August 1996. https://doi.org/10.1145/237170.237199

  56. Lipski, C., Linz, C., Berger, K., Sellent, A., Magnor, M.: Virtual video camera: image-based viewpoint navigation through space and time. Comput. Graph. Forum 29(8), 2555–2568 (2010). https://doi.org/10.1111/j.1467-8659.2010.01824.x

    Article  Google Scholar 

  57. Lombardi, S., Simon, T., Saragih, J., Schwartz, G., Lehrmann, A., Sheikh, Y.: Neural volumes: learning dynamic renderable volumes from images. ACM Trans. Graph. (Proc. SIGGRAPH) 38(4), 65:1–65:14 (2019). https://doi.org/10.1145/3306346.3323020

    Article  Google Scholar 

  58. Luo, B., Xu, F., Richardt, C., Yong, J.H.: Parallax360: stereoscopic 360\(^\circ \) scene representation for head-motion parallax. IEEE Trans. Vis. Comput. Graph. 24(4), 1545–1553 (2018). https://doi.org/10.1109/TVCG.2018.2794071

    Article  Google Scholar 

  59. Magnor, M., Grau, O., Sorkine-Hornung, O., Theobalt, C. (eds.): Digital Representations of the Real World: How to Capture, Model, and Render Visual Reality. A K Peters/CRC Press, New York (2015)

    MATH  Google Scholar 

  60. Martin-Brualla, R., et al.: LookinGood: enhancing performance capture with real-time neural re-rendering. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 37(6), 255:1–255:14 (2018). https://doi.org/10.1145/3272127.3275099

    Article  Google Scholar 

  61. Meka, A., et al.: Deep reflectance fields: high-quality facial reflectance field inference from color gradient illumination. ACM Trans. Graph. (Proc. SIGGRAPH) 38(4), 77:1–77:12 (2019). https://doi.org/10.1145/3306346.3323027

    Article  Google Scholar 

  62. Meshry, M., et al.: Neural rerendering in the wild. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  63. Mildenhall, B., et al.: Local light field fusion: practical view synthesis with prescriptive sampling guidelines. ACM Trans. Graph. (Proc. SIGGRAPH) 38(4), 29:1–29:14 (2019). https://doi.org/10.1145/3306346.3322980

    Article  Google Scholar 

  64. Mori, M.: The uncanny valley. Energy 7(4), 33–35 (1970). (in Japanese)

    Google Scholar 

  65. Moulon, P., Monasse, P., Marlet, R.: Adaptive structure from motion with a Contrario model estimation. In: Lee, K.M., Matsushita, Y., Rehg, J.M., Hu, Z. (eds.) ACCV 2012. LNCS, vol. 7727, pp. 257–270. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-37447-0_20

    Chapter  Google Scholar 

  66. Mustafa, A., Volino, M., Guillemaut, J.Y., Hilton, A.: 4D temporally coherent light-field video. In: Proceedings of International Conference on 3D Vision (3DV) (2017)

    Google Scholar 

  67. Mustafa, A., Volino, M., Kim, H., Guillemaut, J.Y., Hilton, A.: Temporally coherent general dynamic scene reconstruction (2019). arXiv:1907.08195

  68. Nam, G., Lee, J.H., Gutierrez, D., Kim, M.H.: Practical SVBRDF acquisition of 3D objects with unstructured flash photography. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 37(6), 267:1–267:12 (2018). https://doi.org/10.1145/3272127.3275017

    Article  Google Scholar 

  69. Newcombe, R.A., et al.: KinectFusion: real-time dense surface mapping and tracking. In: Proceedings of the International Symposium on Mixed and Augmented Reality (ISMAR), pp. 127–136, October 2011. https://doi.org/10.1109/ISMAR.2011.6092378

  70. Nguyen-Phuoc, T., Li, C., Theis, L., Richardt, C., Yang, Y.L.: HoloGAN: unsupervised learning of 3D representations from natural images. In: Proceedings of the International Conference on Computer Vision (ICCV) (2019)

    Google Scholar 

  71. Nießner, M., Zollhöfer, M., Izadi, S., Stamminger, M.: Real-time 3D reconstruction at scale using voxel hashing. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 32(6), 169:1–169:11 (2013). https://doi.org/10.1145/2508363.2508374

    Article  Google Scholar 

  72. Niklaus, S., Mai, L., Yang, J., Liu, F.: 3D Ken Burns effect from a single image. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38(6), 184:1–184:15 (2019). https://doi.org/10.1145/3355089.3356528

    Article  Google Scholar 

  73. Oculus: From the lab to the living room: the story behind Facebook’s Oculus Insight technology and a new era of consumer VR. https://tech.fb.com/the-story-behind-oculus-insight-technology/. Accessed 30 Oct 2019

  74. Olszewski, K., Tulyakov, S., Woodford, O., Li, H., Luo, L.: Transformable bottleneck networks. In: Proceedings of the International Conference on Computer Vision (ICCV) (2019)

    Google Scholar 

  75. Overbeck, R.S., Erickson, D., Evangelakos, D., Pharr, M., Debevec, P.: A system for acquiring, compressing, and rendering panoramic light field stills for virtual reality. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 37(6), 197:1–197:15 (2018). https://doi.org/10.1145/3272127.3275031

    Article  Google Scholar 

  76. Park, E., Yang, J., Yumer, E., Ceylan, D., Berg, A.C.: Transformation-grounded image generation network for novel 3D view synthesis. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 702–711, July 2017. https://doi.org/10.1109/CVPR.2017.82

  77. Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: DeepSDF: learning continuous signed distance functions for shape representation. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  78. Parra Pozo, A., et al.: An integrated 6DoF video camera and system design. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38(6), 216:1–216:16 (2019). https://doi.org/10.1145/3355089.3356555. https://github.com/facebook/facebook360dep

    Article  Google Scholar 

  79. Peleg, S., Ben-Ezra, M., Pritch, Y.: Omnistereo: panoramic stereo imaging. IEEE Trans. Pattern Anal. Mach. Intell. 23(3), 279–290 (2001). https://doi.org/10.1109/34.910880

    Article  MATH  Google Scholar 

  80. Penner, E., Zhang, L.: Soft 3D reconstruction for view synthesis. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 36(6), 235:1–235:11 (2017). https://doi.org/10.1145/3130800.3130855

    Article  Google Scholar 

  81. Perazzi, F., et al.: Panoramic video from unstructured camera arrays. Comput. Graph. Forum (Proc. Eurographics) 34(2), 57–68 (2015). https://doi.org/10.1111/cgf.12541

    Article  Google Scholar 

  82. Prada, F., Kazhdan, M., Chuang, M., Collet, A., Hoppe, H.: Spatiotemporal atlas parameterization for evolving meshes. ACM Trans. Graph. (Proc. SIGGRAPH) 36(4), 58:1–58:12 (2017). https://doi.org/10.1145/3072959.3073679

    Article  Google Scholar 

  83. Qi, M., Li, W., Yang, Z., Wang, Y., Luo, J.: Attentive relational networks for mapping images to scene graphs. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  84. Rhodin, H., Salzmann, M., Fua, P.: Unsupervised geometry-aware representation for 3D human pose estimation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11214, pp. 765–782. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01249-6_46

    Chapter  Google Scholar 

  85. Richardt, C., Hedman, P., Overbeck, R.S., Cabral, B., Konrad, R., Sullivan, S.: Capture4VR: from VR photography to VR video. In: SIGGRAPH Courses (2019). https://doi.org/10.1145/3305366.3328028

  86. Richardt, C., Pritch, Y., Zimmer, H., Sorkine-Hornung, A.: Megastereo: constructing high-resolution stereo panoramas. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1256–1263, June 2013. https://doi.org/10.1109/CVPR.2013.166

  87. Schroers, C., Bazin, J.C., Sorkine-Hornung, A.: An omnistereoscopic video pipeline for capture and display of real-world VR. ACM Trans. Graph. 37(3), 37:1–37:13 (2018). https://doi.org/10.1145/3225150

    Article  Google Scholar 

  88. Schönberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4104–4113 (2016). https://doi.org/10.1109/CVPR.2016.445

  89. Schönberger, J.L., Zheng, E., Frahm, J.-M., Pollefeys, M.: Pixelwise view selection for unstructured multi-view stereo. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 501–518. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_31

    Chapter  Google Scholar 

  90. Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 519–528 (2006). https://doi.org/10.1109/CVPR.2006.19

  91. Serrano, A., et al.: Motion parallax for 360\(^\circ \) RGBD video. IEEE Trans. Vis. Comput. Graph. 25(5), 1817–1827 (2019). https://doi.org/10.1109/TVCG.2019.2898757

    Article  Google Scholar 

  92. Shum, H., Kang, S.B.: Review of image-based rendering techniques. In: Proceedings of the SPIE Visual Communications and Image Processing, vol. 4067 (2000). https://doi.org/10.1117/12.386541

  93. Shum, H.Y., Chan, S.C., Kang, S.B.: Image-Based Rendering. Springer, Boston (2007). https://doi.org/10.1007/978-0-387-32668-9

    Book  MATH  Google Scholar 

  94. Sitzmann, V., Thies, J., Heide, F., Nießner, M., Wetzstein, G., Zollhöfer, M.: DeepVoxels: learning persistent 3D feature embeddings. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2437–2446 (2019)

    Google Scholar 

  95. Sitzmann, V., Zollhöfer, M., Wetzstein, G.: Scene representation networks: continuous 3D-structure-aware neural scene representations. In: Advances in Neural Information Processing Systems (NeurIPS) (2019)

    Google Scholar 

  96. Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: exploring photo collections in 3D. ACM Trans. Graph. (Proc. SIGGRAPH) 25(3), 835–846 (2006). https://doi.org/10.1145/1141911.1141964

    Article  Google Scholar 

  97. Speciale, P., Schönberger, J.L., Kang, S.B., Sinha, S.N., Pollefeys, M.: Privacy preserving image-based localization. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  98. Srinivasan, P.P., Tucker, R., Barron, J.T., Ramamoorthi, R., Ng, R., Snavely, N.: Pushing the boundaries of view extrapolation with multiplane images. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  99. Sweeney, C.: Theia multiview geometry library (2016). http://theia-sfm.org

  100. Sweeney, C., Holynski, A., Curless, B., Seitz, S.M.: Structure from motion for panorama-style videos (2019). arXiv:1906.03539

  101. Szeliski, R.: Image alignment and stitching: a tutorial. Found. Trends Comput. Graph. Vis. 2(1), 1–104 (2006). https://doi.org/10.1561/0600000009

    Article  MathSciNet  MATH  Google Scholar 

  102. Szeliski, R.: Computer Vision: Algorithms and Applications. Springer, London (2010). https://doi.org/10.1007/978-1-84882-935-0. http://szeliski.org/Book/

    Book  MATH  Google Scholar 

  103. Tarko, J., Tompkin, J., Richardt, C.: Real-time virtual object insertion for moving 360\(^\circ \) videos. In: Proceedings of the International Conference on Virtual-Reality Continuum and its Applications in Industry (VRCAI) (2019)

    Google Scholar 

  104. Tatarchenko, M., Dosovitskiy, A., Brox, T.: Multi-view 3D models from single images with a convolutional network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 322–337. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46478-7_20

    Chapter  Google Scholar 

  105. Thies, J., Zollhöfer, M., Nießner, M.: Deferred neural rendering: image synthesis using neural textures. ACM Trans. Graph. (Proc. SIGGRAPH) 38(4), 66:1–66:12 (2019). https://doi.org/10.1145/3306346.3323035

    Article  Google Scholar 

  106. Tulsiani, S., Tucker, R., Snavely, N.: Layer-structured 3D scene inference via view synthesis. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 311–327. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_19

    Chapter  Google Scholar 

  107. Tung, H.Y.F., Cheng, R., Fragkiadaki, K.: Learning spatial common sense with geometry-aware recurrent networks. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2595–2603 (2019)

    Google Scholar 

  108. Valve: Index headset. www.valvesoftware.com/en/index/headset. Accessed 30 Oct 2019

  109. Ventura, J.: Structure from motion on a sphere. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 53–68. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_4

    Chapter  Google Scholar 

  110. Wei, S.E., et al.: VR facial animation via multiview image translation. ACM Trans. Graph. (Proc. SIGGRAPH) 38(4), 67:1–67:16 (2019). https://doi.org/10.1145/3306346.3323030

    Article  Google Scholar 

  111. Weissig, C., Schreer, O., Eisert, P., Kauff, P.: The ultimate immersive experience: panoramic 3D video acquisition. In: Schoeffmann, K., Merialdo, B., Hauptmann, A.G., Ngo, C.-W., Andreopoulos, Y., Breiteneder, C. (eds.) MMM 2012. LNCS, vol. 7131, pp. 671–681. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-27355-1_72

    Chapter  Google Scholar 

  112. Wetzstein, G., Lanman, D., Heidrich, W., Raskar, R.: Layered 3D: tomographic image synthesis for attenuation-based light field and high dynamic range displays. ACM Trans. Graph. (Proc. SIGGRAPH) 30(4), 95:1–95:12 (2011). https://doi.org/10.1145/2010324.1964990

    Article  Google Scholar 

  113. Wetzstein, G., Lanman, D., Hirsch, M., Raskar, R.: Tensor displays: compressive light field synthesis using multilayer displays with directional backlighting. ACM Trans. Graph. (Proc. SIGGRAPH) 31(4), 80:1–80:11 (2012). https://doi.org/10.1145/2185520.2185576

    Article  Google Scholar 

  114. Whelan, T., Salas-Moreno, R.F., Glocker, B., Davison, A.J., Leutenegger, S.: ElasticFusion: real-time dense SLAM and light source estimation. Int. J. Robot. Res. 35(14), 1697–1716 (2016). https://doi.org/10.1177/0278364916669237

    Article  Google Scholar 

  115. Wood, D.N., et al.: Surface light fields for 3D photography. In: Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pp. 287–296 (2000). https://doi.org/10.1145/344779.344925

  116. Worrall, D.E., Garbin, S.J., Turmukhambetov, D., Brostow, G.J.: Interpretable transformations with encoder-decoder networks. In: Proceedings of the International Conference on Computer Vision (ICCV), pp. 5737–5746 (2017). https://doi.org/10.1109/ICCV.2017.611

  117. Wu, C.: VisualSFM: a visual structure from motion system (2011). http://ccwu.me/vsfm/

  118. Yang, J., Reed, S.E., Yang, M.H., Lee, H.: Weakly-supervised disentangling with recurrent transformations for 3D view synthesis. In: Advances in Neural Information Processing Systems (NIPS), pp. 1099–1107 (2015)

    Google Scholar 

  119. Yifan, W., Serena, F., Wu, S., Öztireli, C., Sorkine-Hornung, O.: Differentiable surface splatting for point-based geometry processing. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38(6) (2019). https://doi.org/10.1145/3355089.3356513

  120. Yücer, K., Sorkine-Hornung, A., Wang, O., Sorkine-Hornung, O.: Efficient 3D object segmentation from densely sampled light fields with applications to 3D reconstruction. ACM Trans. Graph. 35(3), 22:1–22:15 (2016). https://doi.org/10.1145/2876504

    Article  Google Scholar 

  121. Zaragoza, J., Chin, T.J., Tran, Q.H., Brown, M.S., Suter, D.: As-projective-as-possible image stitching with moving DLT. IEEE Trans. Pattern Anal. Mach. Intell. 36(7), 1285–1298 (2014). https://doi.org/10.1109/TPAMI.2013.247

    Article  Google Scholar 

  122. Zhang, F., Liu, F.: Parallax-tolerant image stitching. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3262–3269, June 2014. https://doi.org/10.1109/CVPR.2014.423

  123. Zheng, K.C., Kang, S.B., Cohen, M.F., Szeliski, R.: Layered depth panoramas. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR) (2007). https://doi.org/10.1109/CVPR.2007.383295

  124. Zhou, T., Tucker, R., Flynn, J., Fyffe, G., Snavely, N.: Stereo magnification: learning view synthesis using multiplane images. ACM Trans. Graph. (Proc. SIGGRAPH) 37(4), 65:1–65:12 (2018). https://doi.org/10.1145/3197517.3201323

    Article  Google Scholar 

  125. Zhou, T., Tulsiani, S., Sun, W., Malik, J., Efros, A.A.: View synthesis by appearance flow. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 286–301. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_18

    Chapter  Google Scholar 

  126. Zollhöfer, M., et al.: State of the art on monocular 3D face reconstruction, tracking, and applications. Comput. Graph. Forum 37(2), 523–550 (2018). https://doi.org/10.1111/cgf.13382

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Bath, Bath, UK

    Christian Richardt

  2. Brown University, Providence, USA

    James Tompkin

  3. Stanford University, Stanford, USA

    Gordon Wetzstein

Authors
  1. Christian Richardt
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. James Tompkin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gordon Wetzstein
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christian Richardt .

Editor information

Editors and Affiliations

  1. TU Braunschweig, Brunswick, Germany

    Marcus Magnor

  2. Facebook Zurich, Zürich, Switzerland

    Alexander Sorkine-Hornung

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Richardt, C., Tompkin, J., Wetzstein, G. (2020). Capture, Reconstruction, and Representation of the Visual Real World for Virtual Reality. In: Magnor, M., Sorkine-Hornung, A. (eds) Real VR – Immersive Digital Reality. Lecture Notes in Computer Science(), vol 11900. Springer, Cham. https://doi.org/10.1007/978-3-030-41816-8_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-41816-8_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-41815-1

  • Online ISBN: 978-3-030-41816-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics