Skip to main content

Advertisement

Log in

K-nearest neighborhood based integration of time-of-flight cameras and passive stereo for high-accuracy depth maps

  • Published:
Journal of Zhejiang University SCIENCE C Aims and scope Submit manuscript

Abstract

Both time-of-flight (ToF) cameras and passive stereo can provide the depth information for their corresponding captured real scenes, but they have innate limitations. ToF cameras and passive stereo are intrinsically complementary for certain tasks. It is desirable to appropriately leverage all the available information by ToF cameras and passive stereo. Although some fusion methods have been presented recently, they fail to consider ToF reliability detection and ToF based improvement of passive stereo. As a result, this study proposes an approach to integrating ToF cameras and passive stereo to obtain high-accuracy depth maps. The main contributions are: (1) An energy cost function is devised to use data from ToF cameras to boost the stereo matching of passive stereo; (2) A fusion method is used to combine the depth information from both ToF cameras and passive stereo to obtain high-accuracy depth maps. Experiments show that the proposed approach achieves improved results with high accuracy and robustness.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  • Attamimi, M., Mizutani, A., Nakamura, T., et al., 2010. Real-time 3D visual sensor for robust object recognition. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, p.4560–4565. [doi:10.1109/IROS.2010.5650455]

    Google Scholar 

  • Buehler, C., Bosse, M., McMillan, L., et al., 2001. Unstructured lumigraph rendering. Proc. 28th Annual Conf. on Computer Graphics and Interactive Techniques, p.425–432. [doi:10.1145/383259.383309]

    Google Scholar 

  • Canesta, 2006. Canestavision Electronic Perception Development Kit. Available from http://www.canesta.com/

    Google Scholar 

  • Chen, Q., Li, D., Tang, C., 2012. KNN matting. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, p.869–876. [doi:10.1109/CVPR.2012.6247760]

    Google Scholar 

  • De-Maeztu, L., Mattoccia, S., Villanueva, A., et al., 2011. Linear stereo matching. Proc. IEEE Int. Conf. on Computer Vision, p.1708–1715. [doi:10.1109/ICCV.2011.6126434]

    Google Scholar 

  • Diebel, J., Thrun, S., 2005. An application of Markov random fields to range sensing. Adv. Neur. Inform. Process. Syst., 18:291–298.

    Google Scholar 

  • Gandhiy, V., Cech, J., Horaud, R., 2012. High-resolution depth maps based on TOF-stereo fusion. IEEE Int. Conf. on Robotics and Automation, p.4742–4749. [doi:10.1109/ICRA.2012.6224771]

    Google Scholar 

  • Gudmundsson, S.A., Aanaes, H., Larsen, R., 2008. Fusion of stereo vision and time-of-flight imaging for improved 3D estimation. Int. J. Intell. Syst. Technol. Appl., 5(3–4):425–433. [doi:10.1504/IJISTA.2008.021305]

    Google Scholar 

  • Kanade, T., Okutomi, M., 1994. A stereo matching algorithm with an adaptive window: theory and experiment. IEEE Trans. Patt. Anal. Mach. Intell., 16(9):920–932. [doi:10.1109/34.310690]

    Article  Google Scholar 

  • Lee, C., Song, H., Choi, B., et al., 2011. 3D scene capturing using stereoscopic cameras and a time-of-flight camera. IEEE Trans. Consum. Electron., 57(3):1370–1376. [doi:10.1109/TCE.2011.6018896]

    Article  Google Scholar 

  • Lindner, M., Kolb, A., Hartmann, K., 2007. Data-fusion of PMD-based distance-information and high-resolution RGB-images. Proc. Int. Symp. on Signals, Circuits and Systems, p.1–4. [doi:10.1109/ISSCS.2007.4292666]

    Google Scholar 

  • May, S., Werner, B., Surmann, H., et al., 2006. 3D time-of-flight cameras for mobile robotics. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, p.790–795. [doi:10.1109/IROS.2006.281670]

    Google Scholar 

  • Opencv, 2012. Open Source Computer Vision Library (opencv). Available from www.intel.com/technology/computing/opencv/

    Google Scholar 

  • PMD, 2009. Camcube Series. Available from http://www.pmdtec.com/

    Google Scholar 

  • PMD, 2010. Camcube 3.0 Products. Available from http://www.pmdtec.com/products-services/pmdvisionr-cameras/pmdvisionr-camcube-30/

    Google Scholar 

  • Ringbeck, T., Hagebeuker, B., 2007. A 3D time of flight camera for object detection. Proc. 8th Conf. on Optical 3-D Measurement Techniques.

    Google Scholar 

  • Scharstein, D., Szeliski, R., 2002a. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis., 47(1—3):7–42. [doi:10.1023/A:1014573219977]

    Article  MATH  Google Scholar 

  • Scharstein, D., Szeliski, R., 2002b. Middlebury Stereo Evaluation — version 2. Available from http://vision.middlebury.edu/stereo/eval

    Google Scholar 

  • Wang, L., Lou, L., Yang, C., et al., 2013. Portrait drawing from corresponding range and intensity images. J. Zhejiang Univ.-Sci. C (Comput. & Electron.), 14(7):530–541. [doi:10.1631/jzus.CIDE1306]

    Article  Google Scholar 

  • Xu, Z., Schwarte, R., Heinol, H., et al., 1998. Smart pixelphotonic mixer device (PMD). Proc. 5th Int. Conf. on Mechatronics and Machine Vision in Practice, p.259–264.

    Google Scholar 

  • Yang, Q., 2012. A non-local cost aggregation method for stereo matching. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, p.1402–1409. [doi:10.1109/CVPR.2012.6247827]

    Google Scholar 

  • Yao, L., Li, D., Zhang, J., et al., 2012 Accurate real-time stereo correspondence using intra- and inter-scanline optimization. J. Zhejiang Univ.-Sci. C (Comput. & Electron.), 13(6):472–482. [doi:10.1631/jzus.C1100311]

    Article  Google Scholar 

  • Z-cam, 2004. 3DV Systems. Available from http://www.3dvsystems.com

    Google Scholar 

  • Zhang, J., Li, D., Zhang, M., 2010. Fast stereo matching algorithm based on adaptive window. Proc. Int. Conf. on Audio Language and Image Processing, p.138–142. [doi:10.1109/ICALIP.2010.5684994]

    Google Scholar 

  • Zhang, Z., 1999. Flexible camera calibration by viewing a plane from unknown orientations. Proc. 7th IEEE Int. Conf. on Computer Vision, p.666–673. [doi:10.1109/ICCV.1999.791289]

    Chapter  Google Scholar 

  • Zhu, J., Wang, L., Yang, R., et al., 2008. Fusion of time-of-flight depth and stereo for high accuracy depth maps. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, p.1–8. [doi:10.1109/CVPR.2008.4587761]

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Liang-hao Wang.

Additional information

Project supported by the National Natural Science Foundation of China (Nos. 61072081 and 61271338), the National High-Tech R&D Program (863) of China (No. 2012AA011505), the National Science and Technology Major Project of the Ministry of Science and Technology of China (No. 2009ZX01033-001-007), the Key Science and Technology Innovation Team of Zhejiang Province (No. 2009R50003), and the China Postdoctoral Science Foundation (No. 2012T50545)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Liu, Lw., Li, Y., Zhang, M. et al. K-nearest neighborhood based integration of time-of-flight cameras and passive stereo for high-accuracy depth maps. J. Zhejiang Univ. - Sci. C 15, 174–186 (2014). https://doi.org/10.1631/jzus.C1300194

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1631/jzus.C1300194

Key words

CLC number

Navigation