Abstract
Both time-of-flight (ToF) cameras and passive stereo can provide the depth information for their corresponding captured real scenes, but they have innate limitations. ToF cameras and passive stereo are intrinsically complementary for certain tasks. It is desirable to appropriately leverage all the available information by ToF cameras and passive stereo. Although some fusion methods have been presented recently, they fail to consider ToF reliability detection and ToF based improvement of passive stereo. As a result, this study proposes an approach to integrating ToF cameras and passive stereo to obtain high-accuracy depth maps. The main contributions are: (1) An energy cost function is devised to use data from ToF cameras to boost the stereo matching of passive stereo; (2) A fusion method is used to combine the depth information from both ToF cameras and passive stereo to obtain high-accuracy depth maps. Experiments show that the proposed approach achieves improved results with high accuracy and robustness.
Similar content being viewed by others
References
Attamimi, M., Mizutani, A., Nakamura, T., et al., 2010. Real-time 3D visual sensor for robust object recognition. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, p.4560–4565. [doi:10.1109/IROS.2010.5650455]
Buehler, C., Bosse, M., McMillan, L., et al., 2001. Unstructured lumigraph rendering. Proc. 28th Annual Conf. on Computer Graphics and Interactive Techniques, p.425–432. [doi:10.1145/383259.383309]
Canesta, 2006. Canestavision Electronic Perception Development Kit. Available from http://www.canesta.com/
Chen, Q., Li, D., Tang, C., 2012. KNN matting. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, p.869–876. [doi:10.1109/CVPR.2012.6247760]
De-Maeztu, L., Mattoccia, S., Villanueva, A., et al., 2011. Linear stereo matching. Proc. IEEE Int. Conf. on Computer Vision, p.1708–1715. [doi:10.1109/ICCV.2011.6126434]
Diebel, J., Thrun, S., 2005. An application of Markov random fields to range sensing. Adv. Neur. Inform. Process. Syst., 18:291–298.
Gandhiy, V., Cech, J., Horaud, R., 2012. High-resolution depth maps based on TOF-stereo fusion. IEEE Int. Conf. on Robotics and Automation, p.4742–4749. [doi:10.1109/ICRA.2012.6224771]
Gudmundsson, S.A., Aanaes, H., Larsen, R., 2008. Fusion of stereo vision and time-of-flight imaging for improved 3D estimation. Int. J. Intell. Syst. Technol. Appl., 5(3–4):425–433. [doi:10.1504/IJISTA.2008.021305]
Kanade, T., Okutomi, M., 1994. A stereo matching algorithm with an adaptive window: theory and experiment. IEEE Trans. Patt. Anal. Mach. Intell., 16(9):920–932. [doi:10.1109/34.310690]
Lee, C., Song, H., Choi, B., et al., 2011. 3D scene capturing using stereoscopic cameras and a time-of-flight camera. IEEE Trans. Consum. Electron., 57(3):1370–1376. [doi:10.1109/TCE.2011.6018896]
Lindner, M., Kolb, A., Hartmann, K., 2007. Data-fusion of PMD-based distance-information and high-resolution RGB-images. Proc. Int. Symp. on Signals, Circuits and Systems, p.1–4. [doi:10.1109/ISSCS.2007.4292666]
May, S., Werner, B., Surmann, H., et al., 2006. 3D time-of-flight cameras for mobile robotics. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, p.790–795. [doi:10.1109/IROS.2006.281670]
Opencv, 2012. Open Source Computer Vision Library (opencv). Available from www.intel.com/technology/computing/opencv/
PMD, 2009. Camcube Series. Available from http://www.pmdtec.com/
PMD, 2010. Camcube 3.0 Products. Available from http://www.pmdtec.com/products-services/pmdvisionr-cameras/pmdvisionr-camcube-30/
Ringbeck, T., Hagebeuker, B., 2007. A 3D time of flight camera for object detection. Proc. 8th Conf. on Optical 3-D Measurement Techniques.
Scharstein, D., Szeliski, R., 2002a. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis., 47(1—3):7–42. [doi:10.1023/A:1014573219977]
Scharstein, D., Szeliski, R., 2002b. Middlebury Stereo Evaluation — version 2. Available from http://vision.middlebury.edu/stereo/eval
Wang, L., Lou, L., Yang, C., et al., 2013. Portrait drawing from corresponding range and intensity images. J. Zhejiang Univ.-Sci. C (Comput. & Electron.), 14(7):530–541. [doi:10.1631/jzus.CIDE1306]
Xu, Z., Schwarte, R., Heinol, H., et al., 1998. Smart pixelphotonic mixer device (PMD). Proc. 5th Int. Conf. on Mechatronics and Machine Vision in Practice, p.259–264.
Yang, Q., 2012. A non-local cost aggregation method for stereo matching. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, p.1402–1409. [doi:10.1109/CVPR.2012.6247827]
Yao, L., Li, D., Zhang, J., et al., 2012 Accurate real-time stereo correspondence using intra- and inter-scanline optimization. J. Zhejiang Univ.-Sci. C (Comput. & Electron.), 13(6):472–482. [doi:10.1631/jzus.C1100311]
Z-cam, 2004. 3DV Systems. Available from http://www.3dvsystems.com
Zhang, J., Li, D., Zhang, M., 2010. Fast stereo matching algorithm based on adaptive window. Proc. Int. Conf. on Audio Language and Image Processing, p.138–142. [doi:10.1109/ICALIP.2010.5684994]
Zhang, Z., 1999. Flexible camera calibration by viewing a plane from unknown orientations. Proc. 7th IEEE Int. Conf. on Computer Vision, p.666–673. [doi:10.1109/ICCV.1999.791289]
Zhu, J., Wang, L., Yang, R., et al., 2008. Fusion of time-of-flight depth and stereo for high accuracy depth maps. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, p.1–8. [doi:10.1109/CVPR.2008.4587761]
Author information
Authors and Affiliations
Corresponding author
Additional information
Project supported by the National Natural Science Foundation of China (Nos. 61072081 and 61271338), the National High-Tech R&D Program (863) of China (No. 2012AA011505), the National Science and Technology Major Project of the Ministry of Science and Technology of China (No. 2009ZX01033-001-007), the Key Science and Technology Innovation Team of Zhejiang Province (No. 2009R50003), and the China Postdoctoral Science Foundation (No. 2012T50545)
Rights and permissions
About this article
Cite this article
Liu, Lw., Li, Y., Zhang, M. et al. K-nearest neighborhood based integration of time-of-flight cameras and passive stereo for high-accuracy depth maps. J. Zhejiang Univ. - Sci. C 15, 174–186 (2014). https://doi.org/10.1631/jzus.C1300194
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1631/jzus.C1300194