Skip to main content
SpringerLink
Log in
Book cover

Video Processing and Computational Video pp 1–24Cite as

Towards Plenoptic Raumzeit Reconstruction

  • Chapter

  • 815 Accesses

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 7082))

Abstract

The goal of image-based rendering is to evoke a visceral sense of presense in a scene using only photographs or videos. A huge variety of different approaches have been developed during the last decade. Examining the underlying models we find three different main categories: view interpolation based on geometry proxies, pure image interpolation techniques and complete scene flow reconstruction. In this paper we present three approaches for free-viewpoint video, one for each of these categories and discuss their individual benefits and drawbacks. We hope that studying the different approaches will help others in making important design decisions when planning a free-viewpoint video system.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   54.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Eisemann, M., De Decker, B., Magnor, M., Bekaert, P., de Aguiar, E., Ahmed, N., Theobalt, C., Sellent, A.: Floating Textures. Computer Graphics Forum 27, 409–418 (2008)

    Article  Google Scholar 

  2. Stich, T., Linz, C., Wallraven, C., Cunningham, D., Magnor, M.: Perception-motivated Interpolation of Image Sequences. In: Symposium on Applied Perception in Graphics and Visualization, pp. 97–106 (2008)

    Google Scholar 

  3. Stich, T., Linz, C., Albuquerque, G., Magnor, M.: View and Time Interpolation in Image Space. Computer Graphics Forum 27, 1781–1787 (2008)

    Article  Google Scholar 

  4. Stich, T.: Space-Time Interpolation Techniques. PhD thesis, Computer Graphics Lab, TU Braunschweig, Germany (2009)

    Google Scholar 

  5. Klose, F., Lipski, C., Magnor, M.: Reconstructing Shape and Motion from Asynchronous Cameras. In: Proceedings of Vision, Modeling, and Visualization (VMV 2010), Siegen, Germany, pp. 171–177 (2010)

    Google Scholar 

  6. Adelson, E.H., Bergen, J.R.: The Plenoptic Function and the Elements of Early Vision. In: Landy, M., Movshon, J.A. (eds.) Computational Models of Visual Processing, pp. 3–20 (1991)

    Google Scholar 

  7. Levoy, M., Hanrahan, P.: Light Field Rendering. In: SIGGRAPH, pp. 31–42 (1996)

    Google Scholar 

  8. Fujii, T., Tanimoto, M.: Free viewpoint TV system based on ray-space representation. In: SPIE, vol. 4864, pp. 175–189 (2002)

    Google Scholar 

  9. Matusik, W., Pfister, H.: 3D TV: A scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes. ACM Transactions on Graphics 23, 814–824 (2004)

    Article  Google Scholar 

  10. Wilburn, B., Joshi, N., Vaish, V., Talvala, E.V., Antunez, E., Barth, A., Adams, A., Horowitz, M., Levoy, M.: High performance imaging using large camera arrays. ACM Transactions on Graphics 24, 765–776 (2005)

    Article  Google Scholar 

  11. Chai, J.X., Chan, S.C., Shum, H.Y., Tong, X.: Plenoptic Sampling. In: SIGGRAPH, pp. 307–318 (2000)

    Google Scholar 

  12. Lin, Z., Shum, H.Y.: On the Number of Samples Needed in Light Field Rendering with constant-depth assumption. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 579–588 (2000)

    Google Scholar 

  13. Lin, Z., Shum, H.Y.: A Geometric Analysis of Light Field Rendering. International Journal of Computer Vision 58, 121–138 (2004)

    Article  Google Scholar 

  14. Stewart, J., Yu, J., Gortler, S.J., McMillan, L.: A New Reconstruction Filter for Undersampled Light Fields. In: Eurographics Workshop on Rendering, pp. 150–156 (2003)

    Google Scholar 

  15. Zwicker, M., Matusik, W., Durand, F., Pfister, H.: Antialiasing for Automultiscopic 3D Displays. In: Eurographics Symposium on Rendering, pp. 107–114 (2006)

    Google Scholar 

  16. Eisemann, M., Sellent, A., Magnor, M.: Filtered Blending: A new, minimal Reconstruction Filter for Ghosting-Free Projective Texturing with Multiple Images. In: Vision, Modeling, and Visualization, pp. 119–126 (2007)

    Google Scholar 

  17. Gortler, S.J., Grzeszczuk, R., Szeliski, R., Cohen, M.F.: The Lumigraph. In: SIGGRAPH, pp. 43–54 (1996)

    Google Scholar 

  18. Debevec, P.E., Taylor, C.J., Malik, J.: Modeling and Rendering Architecture from Photographs: A Hybrid Geometry- and Image-Based Approach. In: SIGGRAPH, pp. 11–20 (1996)

    Google Scholar 

  19. Starck, J., Hilton, A.: Surface capture for performance based animation. IEEE Computer Graphics and Applications 27, 21–31 (2007)

    Article  Google Scholar 

  20. Hornung, A., Kobbelt, L.: Interactive pixel-accurate free viewpoint rendering from images with silhouette aware sampling. Computer Graphics Forum 28, 2090–2103 (2009)

    Article  Google Scholar 

  21. Goldluecke, B., Cremers, D.: A superresolution framework for high-accuracy multiview reconstruction. In: Proc. DAGM Pattern Recognition, Jena, Germany (2009)

    Google Scholar 

  22. Furukawa, Y., Curless, B., Seitz, S.M., Szeliski, R.: Towards Internet-scale multi-view stereo. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1434–1441 (2010)

    Google Scholar 

  23. Carranza, J., Theobalt, C., Magnor, M., Seidel, H.P.: Free-viewpoint video of human actors. ACM Transaction on Graphics 22, 569–577 (2003)

    Article  Google Scholar 

  24. de Aguiar, E., Stoll, C., Theobalt, C., Ahmed, N., Seidel, H.P., Thrun, S.: Performance capture from sparse multi-view video. ACM Transactions on Graphics 27(3), 1–10 (2008)

    Article  Google Scholar 

  25. Hasler, N., Stoll, C., Sunkel, M., Rosenhahn, B., Seidel, H.P.: A statistical model of human pose and body shape. Computer Graphics Forum 28 (2009)

    Google Scholar 

  26. Goldluecke, B., Magnor, M.: Weighted Minimal Hypersurfaces and Their Applications in Computer Vision. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3022, pp. 366–378. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  27. Vedula, S., Baker, S., Kanade, T.: Image based spatio-temporal modeling and view interpolation of dynamic events. ACM Transactions on Graphics 24, 240–261 (2005)

    Article  Google Scholar 

  28. Aganj, E., Monasse, P., Keriven, R.: Multi-view texturing of imprecise mesh. In: Asian Conference on Computer Vision, pp. 468–476 (2009)

    Google Scholar 

  29. Takai, T., Hilton, A., Matsuyama, T.: Harmonized Texture Mapping. In: International Symposium on 3D Data Processing, Visualization and Transmission, pp. 1–8 (2010)

    Google Scholar 

  30. Chen, S.E., Williams, L.: View Interpolation for Image Synthesis. In: SIGGRAPH, pp. 279–288 (1993)

    Google Scholar 

  31. Mark, W., McMillan, L., Bishop, G.: Post-Rendering 3D Warping. In: Symposium on Interactive 3D Graphics, pp. 7–16 (1997)

    Google Scholar 

  32. Beier, T., Neely, S.: Feature-based Image Metamorphosis. In: SIGGRAPH, pp. 35–42 (1992)

    Google Scholar 

  33. Seitz, S., Dyer, C.: Physically-valid view synthesis by image interplation. In: IEEE Workshop on Representation of Visual Scenes, pp. 18–26 (1995)

    Google Scholar 

  34. Seitz, S., Dyer, C.: View Morphing. In: SIGGRAPH, pp. 21–30 (1996)

    Google Scholar 

  35. Horn, B., Schunck, B.: Determining Optical Flow. Artificial Intelligence 16, 185–203 (1981)

    Article  Google Scholar 

  36. Lucas, B., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: International Joint Conference on Artificial Intelligence, pp. 674–679 (1981)

    Google Scholar 

  37. Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  38. Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime tv-l1 optical flow. In: Hamprecht, F.A., Schnörr, C., Jähne, B. (eds.) DAGM 2007. LNCS, vol. 4713, pp. 214–223. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  39. Werlberger, M., Trobin, W., Pock, T., Wedel, A., Cremers, D., Bischof, H.: Anisotropic Huber-L1 optical flow. In: British Machine Vision Conference (2009)

    Google Scholar 

  40. Einarsson, P., Chabert, C.F., Jones, A., Ma, W.C., Lamond, B., Hawkins, T., Bolas, M., Sylwan, S., Debevec, P.: Relighting Human Locomotion with Flowed Reflectance Fields. In: Eurographics Symposium on Rendering, pp. 183–194 (2006)

    Google Scholar 

  41. Fitzgibbon, A., Wexler, Y., Zisserman, A.: Image-based rendering using image-based priors. International Journal of Computer Vision 63, 141–151 (2005)

    Article  Google Scholar 

  42. Mahajan, D., Huang, F.C., Matusik, W., Ramamoorthi, R., Belhumeur, P.: Moving Gradients: A Path-Based Method for Plausible Image Interpolation. ACM Transactions on Graphics 28, 1–10 (2009)

    Article  Google Scholar 

  43. Linz, C., Lipski, C., Magnor, M.: Multi-image Interpolation based on Graph-Cuts and Symmetric Optic Flow. In: Vision, Modeling and Visualization, pp. 115–122 (2010)

    Google Scholar 

  44. Ballan, L., Brostow, G.J., Puwein, J., Pollefeys, M.: Unstructured video-based rendering: Interactive exploration of casually captured videos. ACM Transactions on Graphics, 1–11 (2010)

    Google Scholar 

  45. Segal, M., Korobkin, C., van Widenfelt, R., Foran, J., Haeberli, P.: Fast Shadows and Lighting Effects using Texture Mapping. Computer Graphics 26, 249–252 (1992)

    Article  Google Scholar 

  46. Pock, T., Urschler, M., Zach, C., Beichel, R., Bischof, H.: A Duality Based Algorithm for TV-L1-Optical-Flow Image Registration. In: International Conference on Medical Image Computing and Computer Assisted Intervention, pp. 511–518 (2007)

    Google Scholar 

  47. Buehler, C., Bosse, M., McMillan, L., Gortler, S., Cohen, M.: Unstructured Lumigraph Rendering. In: Conference on Computer Graphics and Interactive Techniques, SIGGRAPH, pp. 425–432 (2001)

    Google Scholar 

  48. Franco, J.S., Boyer, E.: Exact polyhedral visual hulls. In: British Machine Vision Conference, pp. 329–338 (2003)

    Google Scholar 

  49. Wallach, H.: Über visuell wahrgenommene Bewegungsrichtung. Psychologische Forschung 20, 325–380 (1935)

    Article  Google Scholar 

  50. Reichardt, W.: Autocorrelation, A principle for the evaluation of sensory information by the central nervous system. In: Rosenblith, W. (ed.) Sensory Communication, pp. 303–317. MIT Press-Willey, New York (1961)

    Google Scholar 

  51. Qian, N., Andersen, R.: A physiological model for motion-stereo integration and a unified explanation of Pulfrich-like phenomena. Vision Research 37, 1683–1698 (1997)

    Article  Google Scholar 

  52. Heeger, D., Boynton, G., Demb, J., Seidemann, E., Newsome, W.: Motion opponency in visual cortex. Journal of Neuroscience 19, 7162–7174 (1999)

    Google Scholar 

  53. Ruzon, M., Tomasi, C.: Color Edge Detection with the Compass Operator. In: Conference on Computer Vision and Pattern Recognition, pp. 160–166 (1999)

    Google Scholar 

  54. Belongie, S., Malik, J., Puzicha, J.: Matching Shapes. In: International Conference on Computer Vision, pp. 454–461 (2001)

    Google Scholar 

  55. Wertheimer, M.: Laws of organization in perceptual forms. In: Ellis, W. (ed.) A Source Book of Gestalt Psychology, pp. 71–88. Trubner & Co. Ltd., Kegan Paul (1938)

    Chapter  Google Scholar 

  56. Felzenszwalb, P., Huttenlocher, D.: Efficient Graph-Based Image Segmentation. International Journal of Computer Vision 59, 167–181 (2004)

    Article  Google Scholar 

  57. Fischler, M., Bolles, R.: Random Sample Consensus. A Paradigm for Model Fitting With Applications to Image Analysis and Automated Cartography. Communications of the ACM 24, 381–395 (1981)

    Article  MathSciNet  Google Scholar 

  58. Perona, P., Malik, J.: Scale-Space and Edge Detection using Anisotropic Diffusion. Transactions on Pattern Analysis and Machine Intelligence 12, 629–639 (1990)

    Article  Google Scholar 

  59. Yuille, A.L., Poggio, T.A.: Scaling theorems for zero crossings. Transactions on Pattern Analyis and Machine Intelligence 8, 15–25 (1986)

    Article  MATH  Google Scholar 

  60. Baker, S., Scharstein, D., Lewis, J., Roth, S., Black, M., Szeliski, R.: A Database and Evaluation Methodology for Optical Flow. In: International Conference on Computer Vision, pp. 1–8 (2007)

    Google Scholar 

  61. Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: Exploring photo collections in 3d. ACM Transactions on Graphics 25, 835–846 (2006)

    Article  Google Scholar 

  62. Meyer, B., Stich, T., Magnor, M., Pollefeys, M.: Subframe Temporal Alignment of Non-Stationary Cameras. In: British Machine Vision Conference (2008)

    Google Scholar 

  63. Hasler, N., Rosenhahn, B., Thormählen, T., Wand, M., Gall, J., Seidel, H.P.: Markerless motion capture with unsynchronized moving cameras. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 224–231 (2009)

    Google Scholar 

  64. Bay, H., Ess, A., Tuytelaars, T., Gool, L.V.: Surf: Speeded up robust features. Computer Vision and Image Understanding 110, 346–359 (2008)

    Article  Google Scholar 

  65. Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 519–528 (2006)

    Google Scholar 

  66. Kazhdan, M., Bolitho, M., Hoppe, H.: Poisson surface reconstruction. In: Proceedings of the Fourth Eurographics Symposium on Geometry Processing, pp. 61–70 (2006)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Graphics Lab, TU Braunschweig, Germany

    Martin Eisemann, Felix Klose & Marcus Magnor

Authors
  1. Martin Eisemann
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Felix Klose
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marcus Magnor
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Computer Science, TU München, Germany

    Daniel Cremers

  2. Computer Graphics Lab, Mühlenpfordstrasse 23, 38106, Braunschweig, Germany

    Marcus Magnor

  3. Technische Universität München, Germany

    Martin R. Oswald

  4. Technion, Haifa, Israel

    Lihi Zelnik-Manor

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Eisemann, M., Klose, F., Magnor, M. (2011). Towards Plenoptic Raumzeit Reconstruction. In: Cremers, D., Magnor, M., Oswald, M.R., Zelnik-Manor, L. (eds) Video Processing and Computational Video. Lecture Notes in Computer Science, vol 7082. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-24870-2_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-24870-2_1

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-24869-6

  • Online ISBN: 978-3-642-24870-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation