Skip to main content
SpringerLink
Log in

Real-time free viewpoint video from a range sensor and color cameras

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

We propose a method for computing a depth map at interactive rates from a set of closely spaced calibrated video cameras and a Time-of-Flight (ToF) camera. The objective is to synthesize free viewpoint videos in real-time. All computations are performed on the graphics processing unit, leaving the CPU available for other tasks. Depth information is computed from color camera data in textured regions and from ToF data in textureless ones. The trade-off between these two sources is determined locally based on the reliability of the depth estimates obtained from the color images. For this purpose, a confidence measure taking into account the shape of the photo-consistency score as a function of depth is used. The final depth map is computed by minimizing a cost function. This approach offers a significant time savings relative to other methods that apply denoising to the photo-consistency score maps, obtained at every depth, and importantly, still obtains acceptable quality of the rendered image.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Audet, S., Okutomi, M.: A user-friendly method to geometrically calibrate projector-camera systems. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)—Workshops (Procams), pp. 47–54 (2009)

  2. Bartczak, B., Koch, R.: Dense depth maps from low resolution time-of-flight depth and high resolution color views. In: Proceedings of the 5th International Symposium on Advances in Visual Computing: Part II, ISVC ’09, pp. 228–239. Springer-Verlag, Berlin, Heidelberg (2009)

  3. Campbell, N.D.F., Vogiatzis, G., Hernández, C., Cipolla, R.: Using multiple hypotheses to improve depth-maps for multi-view stereo. In: European Conference on Computer Vision, pp. 766–779 (2008)

  4. Chai, J., Chan, S., Shum, H., Tong, X.: Plenoptic sampling. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 307–318 (2000)

  5. Collins, R.: A space-sweep approach to true multi-image matching. In: 1996 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1996. Proceedings CVPR’96, pp. 358–363 (1996)

  6. Debevec, P., Yu, Y., Borshukov, G.: Efficient view-dependent image-based rendering with projective texture-mapping. In: Proceedings of Eurographics Rendering Workshop (1998)

  7. Diebel, J., Thrun, S.: An application of markov random fields to range sensing. In: Proceedings of Conference on Neural Information Processing Systems (NIPS). MIT Press, Cambridge (2005)

  8. Fernando R., Kilgard M.J.: The Cg Tutorial: The Definitive Guide to Programmable Real-Time Graphics. Addison-Wesley, Boston (2003)

    Google Scholar 

  9. Fiala M., Shu C.: Self-identifying patterns for plane-based camera calibration. Mach. Vis. Appl. 19, 209–216 (2008)

    Article  MATH  Google Scholar 

  10. Geman D., Reynolds G.: Constrained restoration and the recovery of discontinuities. IEEE Trans. Pattern Anal. Mach. Intell. 14(3), 367–383 (1992)

    Article  Google Scholar 

  11. Geman D., Yang C.: Nonlinear image recovery with half-quadratic regularization. IEEE Trans. Image Proces. 4(7), 932–946 (1995)

    Article  Google Scholar 

  12. Gudmundsson S.A., Aanaes H., Larsen R.: Fusion of stereo vision and Time-of-Flight imaging for improved 3D estimation. Int. J. Intell. Syst. Technol. Appl. 5, 425–433 (2008)

    Google Scholar 

  13. Hartley R.I, Zisserman A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2004) ISBN: 0521540518

    Book  MATH  Google Scholar 

  14. Huber P.J.: Robust Statistics. Wiley, New York (1981)

    Book  MATH  Google Scholar 

  15. Jianfeng, Y., Cooperstock, J.R.: A new photo consistency test for voxel coloring. In: Proceedings of the Second Canadian Conference on Computer and Robot Vision (CRV). Victoria (2005)

  16. Kraft, H., Frey, J., Moeller, T., Albrecht, M., Grothof, M., Schink, B., Hess, H., Buxbaum, B.: 3D-camera of high 3D-frame rate, depth-resolution and background light elimination based on improved PMD (photonic mixer device)-technologies. In: OPTO (2004)

  17. Kuhnert, K.D., Stommel, M.: Fusion of stereo-camera and PMD-camera data for real-time suited precise 3D environment reconstruction. In: In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4780–4785. Edinburgh, UK (2006)

  18. Mühlmann K., Maier D., Hesser J., Männer R.: Dense disparity maps from color stereo images, an efficient implementation. Int. J. Comput. Vis. 47, 79–88 (2002)

    Article  MATH  Google Scholar 

  19. Reddy M.: Perceptually optimized 3D graphics. IEEE Comput. Graphics Appl. 21, 68–75 (2001)

    Article  Google Scholar 

  20. Scharstein D., Szeliski R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47(1), 7–42 (2002)

    Article  MATH  Google Scholar 

  21. Schultz R.R., Stevenson R.L.: A bayesian approach to image expansion for improved definition. IEEE Trans. Image Process. 3(3), 233–242 (1994)

    Article  Google Scholar 

  22. Seitz, S., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: International Conference on Computer Vision and Pattern Recognition, pp. 519–528 (2006)

  23. Shewchuk J.R.: An introduction to the conjugate gradient method without the agonizing pain. Tech. rep. Carnegie Mellon University, Pittsburgh (1994)

    Google Scholar 

  24. Taguchi, Y., Takahashi, K., Naemura, T.: Real-time all-in-focus video-based rendering using a network camera array. In: Proc. 3DTV-Conference, pp. 241–244 (2008)

  25. Taguchi, Y., Wilburn, B., Zitnick, C.: Stereo reconstruction with mixed pixels using adaptive over-segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, 2008. CVPR 2008, pp. 1–8 (2008)

  26. Takahashi K., Naemura T.: Layered light-field rendering with focus measurement. Signal Process. Image Commun. 21(6), 519–530 (2006)

    Article  Google Scholar 

  27. Yang Q, Tan KH, Culbertson B, Apostolopoulos J (2010) Fusion of active and passive sensors for fast 3D capture. In: MMSP

  28. Yang Q, Yang R, Davis J, Nistér D (2007) Spatial-depth super resolution for range images. In: CVPR

  29. Zhu J., Wang L., Gao J., Yang R.: fusion for high accuracy depth maps using dynamic MRFs. IEEE Trans. Pattern Anal. Mach. Intell. 32, 899–909 (2010)

    Article  Google Scholar 

  30. Zhu, J., Wang, L., Yang, R., Davis, J.: Fusion of Time-of-Flight depth and stereo for high accuracy depth maps. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2008)

  31. Zitnick C., Kang S.: Stereo for image-based rendering using image over-segmentation. Int. J. Comput. Vis. 75(1), 49–65 (2007)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, McGill University, Montreal, QC, Canada

    Stéphane Pelletier & Jeremy R. Cooperstock

Authors
  1. Stéphane Pelletier
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jeremy R. Cooperstock
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stéphane Pelletier.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Pelletier, S., Cooperstock, J.R. Real-time free viewpoint video from a range sensor and color cameras. Machine Vision and Applications 24, 739–751 (2013). https://doi.org/10.1007/s00138-012-0428-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-012-0428-2

Keywords

Search

Navigation