Abstract
Depth information is an important ingredient in many multiview applications including image-based rendering (IBR). With the advent of electronics, low-cost and high-speed depth cameras, such as the Microsoft Kinect, are getting increasingly popular. In this paper, we propose a superpixel-based joint color–depth restoration approach for Kinect depth camera and study its application to view synthesis in IBR systems. Thus, an edge-based matching method is proposed to reduce the color–depth registration errors. Then the Kinect depth map is restored based on probabilistic color–depth superpixels, probabilistic local polynomial regression and joint color–depth matting. The proposed restoration algorithm does not only inpaint the missing data, but also correct and refine the depth map to provide better color–depth consistency. Last but not the least, a dynamic background modeling scheme is proposed to address the disocclusion problem in the view synthesis for dynamic environment. The experimental results show the effectiveness of the proposed algorithm and system.
Similar content being viewed by others
References
Chan, S.C., Shum, H.Y., Ng, K.T.: Image-based rendering and synthesis. IEEE Signal Process. Mag. 24(6), 22–33 (2007). doi:10.1109/MSP.2007.905702
Shum, H.Y., Chan, S.C., Kang, S.B.: Image-based rendering. Springer, New york (2008)
Han, J., Pauwels, E., de Zeeuw, P., de With, P.: Employing a rgb-d sensor for real-time tracking of humans across multiple re-entries in a smart environment. IEEE Trans. Consum. Electron. 58(2), 255–263 (2012). doi:10.1109/TCE.2012.6227420
Li, H., Vouga, E., Gudym, A., Luo, L., Barron, J.T., Gusev, G.: 3D self-portraits. ACM Trans. Graph. (TOG) 32(6), 187 (2013)
Pedersoli, F., Benini, S., Adami, N., Leonardi, R.: Xkin: an open source framework for hand pose and gesture recognition using kinect. Vis. Comput. 30(10), 1107 (2014). doi:10.1007/s00371-014-0921-x
Wang, C., Liu, Z., Chan, S.C.: Superpixel-based hand gesture recognition with kinect depth camera. IEEE Trans. Multimed. 17(1), 29–39 (2015). doi:10.1109/TMM.2014.2374357
Khoshelham, K., Elberink, S.O.: Accuracy and resolution of kinect depth data for indoor mapping applications. Sensors 12(2), 1437–1454 (2012)
Burrus, N.: Kinect calibration. (2014). http://nicolas.burrus.name/index.php/Research/KinectCalibration
Herrera, C.D., Kannala, J., Heikkilä, J.: Joint depth and color camera calibration with distortion correction. IEEE Trans. Pattern Anal. Mach. Intell. 34(10), 2058–2064 (2012). doi:10.1109/TPAMI.2012.125
Zhang, C., Zhang, Z.: Calibration between depth and color sensors for commodity depth cameras. In: Proc. IEEE Int Multimedia and Expo (ICME) Conf, pp 1–6, doi: 10.1109/ICME.2011.6012191(2011)
Wang, Y., Zhong, F., Peng, Q., Qin, X.: Depth map enhancement based on color and depth consistency. Vis. Comput. 30(10), 1157 (2014). doi:10.1007/s00371-013-0896-z
Zhu, Z.Y., Zhang, S., Chan, S.C., Shum, H.Y.: Object-based rendering and 3-D reconstruction using a moveable image-based system. IEEE Trans. Circuits Syst. Video Technol. 22(10), 1405–1419 (2012). doi:10.1109/TCSVT.2012.2198133
Janoch, A., Karayev, S., Jia, Y., Barron, J., Fritz, M., Saenko, K., Darrell, T.: A category-level 3-D object dataset: Putting the kinect to work. In: Proc. IEEE Int Computer Vision Workshops (ICCV Workshops) Conf, pp 1168–1174, doi: 10.1109/ICCVW.2011.6130382 (2011)
Silberman, N., Fergus, R.: Indoor scene segmentation using a structured light sensor. In: Proc. IEEE Int Computer Vision Workshops (ICCV Workshops) Conf, pp. 601–608, doi:10.1109/ICCVW.2011.6130298 (2011)
Ding, K., Chen, W., Wu, X.: Optimum inpainting for depth map based on l0 total variation. Vis. Comput. 30(12), 1311 (2014). doi:10.1007/s00371-013-0888-z
Matyunin, S., Vatolin, D., Berdnikov, Y., Smirnov, M.: Temporal filtering for depth maps generated by kinect depth camera. In: Proc. 3DTV Conf.: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON), pp. 1–4, doi:10.1109/3DTV.2011.5877202 (2011)
Butler, D.A., Izadi, S., Hilliges, O., Molyneaux, D., Hodges, S., Kim, D.: Shake’n’sense: reducing interference for overlapping structured light depth cameras. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’12, pp. 1933–1936. ACM, New York. doi: 10.1145/2207676.2208335 (2012)
Zitnick, C.L., Kang, S.B.: Stereo for image-based rendering using image over-segmentation. Int. J. Comput. Vis. 75(1), 49–65 (2007)
Chaurasia, G., Duchene, S., Sorkine-Hornung, O., Drettakis, G.: Depth synthesis and local warps for plausible image-based navigation. ACM Trans. Graph. (TOG) 32(3), 30 (2013)
Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Süsstrunk, S.: Slic superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2274–2282 (2012). doi:10.1109/TPAMI.2012.120
Chen, X., Zou, D., Zhou, S., Zhao, Q., Tan, P.: Image matting with local and nonlocal smooth priors. In: Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 1902–1907. doi:10.1109/CVPR.2013.248 (2013)
Kong, X., Ng, M., Zhou, Z.H.: Transductive multilabel learning via label set propagation. IEEE Trans. Knowl. Data Eng. 25(3), 704–719 (2013). doi:10.1109/TKDE.2011.141
Takeda, H., Farsiu, S., Milanfar, P.: Kernel regression for image processing and reconstruction. IEEE Trans. Image Process. 16(2), 349–366 (2007). doi:10.1109/TIP.2006.888330
Zhang, Z., Chan, S.C., Wang, C.: A new regularized adaptive windowed lomb periodogram for time-frequency analysis of nonstationary signals with impulsive components. IEEE Trans. Instrum. Meas. 61(8), 2283–2304 (2012). doi:10.1109/TIM.2012.2186655
Katkovnik, V., Egiazarian, K., Astola, J.: A spatially adaptive nonparametric regression image deblurring. IEEE Trans. Image Process. 14(10), 1469–1478 (2005). doi:10.1109/TIP.2005.851705
Wang, C., Chan, S.C.: A new bandwidth adaptive non-local kernel regression algorithm for image/video restoration and its GPU realization. In: Proc. IEEE Int Circuits and Systems (ISCAS) Symp, pp. 1388–1391. doi:10.1109/ISCAS.2013.6572114 (2013)
Zhang, Z., Chan, S.C.: On kernel selection of multivariate local polynomial modelling and its application to image smoothing and reconstruction. J. Signal Process. Syst. 64(3), 361–374 (2011). doi:10.1007/s11265-010-0495-4
Paris, S., Durand, F.: A fast approximation of the bilateral filter using a signal processing approach. Int. J. Comput. Vis. 81(1), 24–52 (2009). doi:10.1007/s11263-007-0110-8
Wang, C., Zhu, Z.Y., Chan, S.C., Shum, H.Y.: Real-time depth image acquisition and restoration for image based rendering and processing systems. J. Signal Process. Syst. 79(1), 1–18 (2013). doi:10.1007/s11265-013-0819-2
Vázquez, C., Tam, W.J., Speranza, F.: Stereoscopic imaging: filling disoccluded areas in depth image-based rendering. In: Proc. SPIE 6392, Three-Dimensional TV, Video, and Display V, 63920D, vol 6392, pp. 1–12. doi: 10.1117/12.685047 (2006)
Liu, W., Zhang, D., Cui, M., Ding, J.: An enhanced depth map based rendering method with directional depth filter and image inpainting. Visu. Comput. 32(5), 579 (2016). doi:10.1007/s00371-015-1074-2
Zivkovic, Z., van der Heijden, F.: Efficient adaptive density estimation per image pixel for the task of background subtraction. Pattern Recogn. Lett. 27(7), 773–780 (2006). doi:10.1016/j.patrec.2005.11.005
Wren, C., Azarbayejani, A., Darrell, T., Pentland, A.: Pfinder: real-time tracking of the human body. IEEE Trans. Pattern Anal. Mach. Intell. 19(7):780–785. doi:10.1109/34.598236 (1997)
Berman, A., Dadourian, A., Vlahos, P.: Method for removing from an image the background surrounding a selected object. US Patent 6,134,346 (2000)
Munshi, A.: The opencl specification 2.0. https://www.khronos.org/registry/cl/specs/opencl-2.0-openclc.pdf (2014)
Author information
Authors and Affiliations
Corresponding author
Additional information
This work was supported in part by K.C. Wong Magna Fund in Ningbo University; National Natural Science Foundation of China (61603202); Zhejiang Open Foundation from Information and Communication Engineering of the Most Important Subjects, China (xkxl1512, xkxl1526); the Open Project Program of the State Key Lab of CAD\( { \& }\)CG in Zhejiang University (A1606); the Research Foundation of Education Department of Zhejiang Province, China (Y201533827); Zhejiang Provincial Natural Science Foundation, China (LQ16F030001); Ningbo Natural Science Foundation, China (2016A610070) and the General Research Fund (GRF) of Hong Kong Research Grant Council (RGC).
Electronic supplementary material
Below is the link to the electronic supplementary material.
Supplementary material 1 (mp4 29564 KB)
Rights and permissions
About this article
Cite this article
Wang, C., Chan, SC., Zhu, ZY. et al. Superpixel-based color–depth restoration and dynamic environment modeling for Kinect-assisted image-based rendering systems. Vis Comput 34, 67–81 (2018). https://doi.org/10.1007/s00371-016-1312-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00371-016-1312-2