Skip to main content
SpringerLink
Log in

A Survey on Time-of-Flight Stereo Fusion

  • Chapter
Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 8200))

Abstract

Due to the demand for depth maps of higher quality than possible with a single depth imaging technique today, there has been an increasing interest in the combination of different depth sensors to produce a “super-camera” that is more than the sum of the individual parts. In this survey paper, we give an overview over methods for the fusion of Time-of-Flight (ToF) and passive stereo data as well as applications of the resulting high quality depth maps. Additionally, we provide a tutorial-based introduction to the principles behind ToF stereo fusion and the evaluation criteria used to benchmark these methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Park, J., Kim, H., Tai, Y., Brown, M., Kweon, I.: High quality depth map upsampling for 3d-tof cameras. In: IEEE Proc. ICCV (2011)

    Google Scholar 

  2. Huhle, B., Fleck, S., Schilling, A.: Integrating 3d time-of-flight camera data and high resolution images for 3dtv applications. In: Proc. 3DTV Conf. IEEE (2007)

    Google Scholar 

  3. Castaneda, V., Mateus, D., Navab, N.: Stereo time-of-flight. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 1684–1691. IEEE (2011)

    Google Scholar 

  4. Kuhnert, K., Stommel, M.: Fusion of stereo-camera and pmd-camera data for real-time suited precise 3d environment reconstruction. In: Int. Conf. on Intelligent Robots and Systems, pp. 4780–4785. IEEE (2006)

    Google Scholar 

  5. Fischer, J., Arbeiter, G., Verl, A.: Combination of time-of-flight depth and stereo using semiglobal optimization. In: Int. Conf. on Robotics and Automation (ICRA), pp. 3548–3553. IEEE (2011)

    Google Scholar 

  6. Eisemann, E., Durand, F.: Flash photography enhancement via intrinsic relighting. ACM Transactions on Graphics (Proc. of SIGGRAPH) 23 (2004)

    Google Scholar 

  7. Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., Toyama, K.: Digital photography with flash and no-flash image pairs. ACM Trans. Graph. 23(3), 664–672 (2004)

    Article  Google Scholar 

  8. Chen, J., Paris, S., Wang, J., Matusik, W., Cohen, M., Durand, F.: The video mesh: A data structure for image-based three-dimensional video editing. In: Proc. of the International Conference on Computional Photography (ICCP) (2011)

    Google Scholar 

  9. Lo, W.Y., van Baar, J., Knaus, C., Zwicker, M., Gross, M.: Stereoscopic 3d copy & paste. ACM Trans. Graph. 29(6), 147:1–147:10 (2010)

    Google Scholar 

  10. Zitnick, C.L., Kang, S.B., Uyttendaele, M., Winder, S.A.J., Szeliski, R.: High-quality video view interpolation using a layered representation. ACM Trans. Graph. 23(3), 600–608 (2004)

    Article  Google Scholar 

  11. Devernay, F., Beardsley, P.: Stereoscopic Cinema. In: Ronfard, R., Taubin, G. (eds.) Image and Geometry Processing for 3-D Cinematography. Geometry and Computing, vol. 5, pp. 11–51. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  12. Fei, Y., Yu, M., Shao, F., Jiang, G.: A color correction algorithm of multi-view video based on depth segmentation. In: International Symposium on Computer Science and Computational Technology (ISCSCT 2008), vol. 2, pp. 206–209 (2008)

    Google Scholar 

  13. Wilkes, L.: The role of ocula in stereo post production. The Foundry. Whitepaper (2009)

    Google Scholar 

  14. Templin, K., Didyk, P., Ritschel, T., Myszkowski, K., Seidel, H.P.: Highlight microdisparity for improved gloss depiction. ACM Transactions on Graphics (Proc. SIGGRAPH) 31(4) (2012)

    Google Scholar 

  15. Mcmillan, L., Gortler, S.: Image-based rendering: A new interface between computer vision and computer graphics. SIGGRAPH Comput. Graph. 33, 61–64 (2000)

    Article  Google Scholar 

  16. Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV 47(1), 7–42 (2002)

    Article  MATH  Google Scholar 

  17. Chiu, W.C., Blanke, U., Fritz, M.: Improving the kinect by cross-modal stereo. In: British Machine Vision Conf. BMVA, pp. 116–1 (2011)

    Google Scholar 

  18. Hrkać, T., Kalafatić, Z., Krapac, J.: Infrared-visual image registration based on corners and hausdorff distance. In: Ersbøll, B.K., Pedersen, K.S. (eds.) SCIA 2007. LNCS, vol. 4522, pp. 383–392. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  19. Bilodeau, G., Torabi, A., Morin, F.: Visible and infrared image registration using trajectories and composite foreground images. Image and Vision Computing 29(1), 41–50 (2011)

    Article  Google Scholar 

  20. Toet, A., Van Ruyven, L.J., Valeton, J.M.: Merging thermal and visual images by a contrast pyramid. Optical Engineering 28(7), 287789–287789 (1989)

    Article  Google Scholar 

  21. Wu, C.: Visualsfm: A visual structure from motion system (2011)

    Google Scholar 

  22. Newcombe, R.A., Lovegrove, S.J., Davison, A.J.: Dtam: Dense tracking and mapping in real-time. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 2320–2327. IEEE (2011)

    Google Scholar 

  23. Horn, B.K., Brooks, M.J.: Shape from shading. MIT Press (1989)

    Google Scholar 

  24. Barron, J.T., Malik, J.: Shape, albedo, and illumination from a single image of an unknown object. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 334–341. IEEE (2012)

    Google Scholar 

  25. Sturmer, M., Penne, J., Hornegger, J.: Standardization of intensity-values acquired by time-of-flight-cameras. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW 2008), pp. 1–6. IEEE (2008)

    Google Scholar 

  26. Kim, Y., Theobalt, C., Diebel, J., Kosecka, J., Miscusik, B., Thrun, S.: Multi-view image and tof sensor fusion for dense 3d reconstruction. In: ICCV Workshops, pp. 1542–1549. IEEE (2009)

    Google Scholar 

  27. Zhu, J., Wang, L., Gao, J., Yang, R.: Spatial-temporal fusion for high accuracy depth maps using dynamic mrfs. IEEE Transactions on Pattern Analysis and Machine Intelligence 32(5), 899–909 (2010)

    Article  Google Scholar 

  28. Ghobadi, S.E., Loepprich, O.E., Lottnera, O., Ahmadov, F., Hartmann, K., Weihs, W., Loffeld, O.: Analysis of the personnel safety in a man-machine-cooperation using 2d/3d images. In: Proceedings of the EURON/IARP International Workshop on Robotics for Risky Interventions and Surveillance of the Environment, Benicassim, Spain (January 2008)

    Google Scholar 

  29. Bradski, G.: The OpenCV Library. Dr. Dobb’s Journal of Software Tools (2000)

    Google Scholar 

  30. Bouguet, J.Y.: Camera calibration toolbox for matlab (2004)

    Google Scholar 

  31. Hahne, U., Alexa, M.: Combining time-of-flight depth and stereo images without accurate extrinsic calibration. IJISTA 5(3), 325–333 (2008)

    Article  Google Scholar 

  32. Nair, R., Lenzen, F., Meister, S., Schäfer, H., Garbe, C., Kondermann, D.: High accuracy TOF and stereo sensor fusion at interactive rates. In: Fusiello, A., Murino, V., Cucchiara, R. (eds.) ECCV 2012 Ws/Demos, Part II. LNCS, vol. 7584, pp. 1–11. Springer, Heidelberg (2012)

    Chapter  Google Scholar 

  33. Dal Mutto, C., Zanuttigh, P., Cortelazzo, G.M.: A probabilistic approach to tof and stereo data fusion. In: 3DPVT, Paris, France, vol. 2 (2010)

    Google Scholar 

  34. Schiller, I., Beder, C., Koch, R.: Calibration of a pmd-camera using a planar calibration pattern together with a multi-camera setup. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences 37, 297–302 (2008)

    Google Scholar 

  35. Zhu, J., Wang, L., Yang, R., Davis, J.: Fusion of time-of-flight depth and stereo for high accuracy depth maps. In: Proc. CVPR, pp. 1–8. IEEE (2008)

    Google Scholar 

  36. Kim, Y.M., Chan, D., Theobalt, C., Thrun, S.: Design and calibration of a multi-view tof sensor fusion system. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW 2008), pp. 1–7. IEEE (2008)

    Google Scholar 

  37. Horn, B.K.: Closed-form solution of absolute orientation using unit quaternions. JOSA A 4(4), 629–642 (1987)

    Article  MathSciNet  Google Scholar 

  38. Guan, L., Pollefeys, M., et al.: A unified approach to calibrate a network of camcorders and tof cameras. In: Workshop on Multi-Camera and Multi-Modal Sensor Fusion Algorithms and Applications (M2SFA2 2008) (2008)

    Google Scholar 

  39. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press (2004)

    Google Scholar 

  40. Dal Mutto, C., Zanuttigh, P., Mattoccia, S., Cortelazzo, G.: Locally consistent toF and stereo data fusion. In: Fusiello, A., Murino, V., Cucchiara, R. (eds.) ECCV 2012 Ws/Demos, Part I. LNCS, vol. 7583, pp. 598–607. Springer, Heidelberg (2012)

    Chapter  Google Scholar 

  41. Zhang, L., Curless, B., Seitz, S.M.: Spacetime stereo: Shape recovery for dynamic scenes. In: Proceedings of the 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. II–367. IEEE (2003)

    Google Scholar 

  42. Yang, Q., Tan, K.H., Culbertson, B., Apostolopoulos, J.: Fusion of active and passive sensors for fast 3d capture. In: MMSP (2010)

    Google Scholar 

  43. Reynolds, M., Dobos, J., Peel, L., Weyrich, T., Brostow, G.J.: Capturing time-of-flight data with confidence. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 945–952. IEEE (2011)

    Google Scholar 

  44. Zhu, J., Wang, L., Yang, R., Davis, J., et al.: Reliability fusion of time-of-flight depth and stereo for high quality depth maps. TPAMI (99), 1 (2011)

    Google Scholar 

  45. Song, Y., Glasbey, C.A., van der Heijden, G.W.A.M., Polder, G., Dieleman, J.A.: Combining stereo and time-of-flight images with application to automatic plant phenotyping. In: Heyden, A., Kahl, F. (eds.) SCIA 2011. LNCS, vol. 6688, pp. 467–478. Springer, Heidelberg (2011)

    Chapter  Google Scholar 

  46. Gudmundsson, S., Aanaes, H., Larsen, R.: Fusion of stereo vision and time-of-flight imaging for improved 3d estimation. IJISTA 5(3), 425–433 (2008)

    Article  Google Scholar 

  47. Bartczak, B., Koch, R.: Dense depth maps from low resolution time-of-flight depth and high resolution color views. In: Bebis, G., Boyle, R., Parvin, B., Koracin, D., Kuno, Y., Wang, J., Pajarola, R., Lindstrom, P., Hinkenjann, A., Encarnação, M.L., Silva, C.T., Coming, D. (eds.) ISVC 2009, Part II. LNCS, vol. 5876, pp. 228–239. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  48. Hahne, U., Alexa, M.: Depth imaging by combining time-of-flight and on-demand stereo. In: Kolb, A., Koch, R. (eds.) Dyn3D 2009. LNCS, vol. 5742, pp. 70–83. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  49. Dal Mutto, C., Zanuttigh, P., Cortelazzo, G.M.: A probabilistic approach to tof and stereo data fusion. In: 3DPVT, Paris, France (May 2010)

    Google Scholar 

  50. Beder, C., Bartczak, B., Koch, R.: A combined approach for estimating patchlets from PMD depth images and stereo intensity images. In: Hamprecht, F.A., Schnörr, C., Jähne, B. (eds.) DAGM 2007. LNCS, vol. 4713, pp. 11–20. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  51. Ruhl, K., Klose, F., Lipski, C., Magnor, M.: Integrating approximate depth data into dense image correspondence estimation. In: Proc. European Conference on Visual Media Production (CVMP) (August 2012)

    Google Scholar 

  52. Gandhi, V., Cech, J., Horaud, R.: High-resolution depth maps based on tof-stereo fusion. In: 2012 IEEE International Conference on Robotics and Automation (ICRA), pp. 4742–4749. IEEE (2012)

    Google Scholar 

  53. Van Meerbergen, G., Vergauwen, M., Pollefeys, M., Van Gool, L.: A hierarchical symmetric stereo algorithm using dynamic programming. International Journal of Computer Vision 47(1-3), 275–285 (2002)

    Article  MATH  Google Scholar 

  54. Gallup, D., Frahm, J.M., Mordohai, P., Yang, Q., Pollefeys, M.: Real-time plane-sweeping stereo with multiple sweeping directions. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2007), pp. 1–8. IEEE (2007)

    Google Scholar 

  55. Kolmogorov, V., Zabih, R.: Computing visual correspondence with occlusions using graph cuts. In: Proceedings of the Eighth IEEE International Conference on Computer Vision (ICCV 2001), vol. 2, pp. 508–515. IEEE (2001)

    Google Scholar 

  56. Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(11), 1222–1239 (2001)

    Article  Google Scholar 

  57. Sun, J., Zheng, N.N., Shum, H.Y.: Stereo matching using belief propagation. IEEE Transactions on Pattern Analysis and Machine Intelligence 25(7), 787–800 (2003)

    Article  Google Scholar 

  58. Ihler, A.T., Mcallester, D.A.: Particle belief propagation. In: International Conference on Artificial Intelligence and Statistics, pp. 256–263 (2009)

    Google Scholar 

  59. Birchfield, S., Tomasi, C.: A pixel dissimilarity measure that is insensitive to image sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(4), 401–406 (1998)

    Article  Google Scholar 

  60. Hirschmüller, H., Scharstein, D.: Evaluation of cost functions for stereo matching. In: Proc. CVPR, pp. 1–8. IEEE (2007)

    Google Scholar 

  61. Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  62. Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime TV-L1 optical flow. In: Pattern Recognition: 29th DAGM Symposium, vol. 29, pp. 214–223 (2007)

    Google Scholar 

  63. Nocedal, J.: Updating quasi-newton matrices with limited storage. Mathematics of Computation 25, 773–782 (1980)

    Article  MathSciNet  Google Scholar 

  64. Hirschmüller, H.: Stereo processing by semiglobal matching and mutual information. TPAMI 30(2), 328–341 (2008)

    Article  Google Scholar 

  65. Mattoccia, S.: A locally global approach to stereo correspondence. In: 2009 IEEE 12th International Conference on Computer Vision Workshops (ICCV Workshops), pp. 1763–1770. IEEE (2009)

    Google Scholar 

  66. Cech, J., Sara, R.: Efficient sampling of disparity space for fast and accurate matching. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2007), pp. 1–8. IEEE (2007)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Heidelberg Collaboratory for Image Processing (HCI), Heidelberg University, Germany

    Rahul Nair, Frank Lenzen, Stephan Meister, Henrik Schäfer, Christoph S. Garbe & Daniel Kondermann

  2. Intel Visual Computing Institute, Saarland University, Germany

    Rahul Nair, Frank Lenzen, Stephan Meister, Henrik Schäfer, Christoph S. Garbe & Daniel Kondermann

  3. Technische Universität Braunschweig, Germany

    Kai Ruhl, Martin Eisemann & Marcus Magnor

Authors
  1. Rahul Nair
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kai Ruhl
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Frank Lenzen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Stephan Meister
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Henrik Schäfer
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Christoph S. Garbe
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Martin Eisemann
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Marcus Magnor
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Daniel Kondermann
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Pattern Recognition Group, University of Siegen, Siegen, Germany

    Marcin Grzegorzek

  2. Max-lanck-Institute, Graphics, Vision & Video Gruop, Saarbrücken, Germany

    Christian Theobalt

  3. Multimedia Information Processing Group, University of Kiel, Kiel, Germany

    Reinhard Koch

  4. Computer Graphics and Multimedia Systems Group, University of Siegen, Siegen, Germany

    Andreas Kolb

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Nair, R. et al. (2013). A Survey on Time-of-Flight Stereo Fusion. In: Grzegorzek, M., Theobalt, C., Koch, R., Kolb, A. (eds) Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications. Lecture Notes in Computer Science, vol 8200. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-44964-2_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-44964-2_6

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-44963-5

  • Online ISBN: 978-3-642-44964-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation