Abstract
This paper introduces an optimization energy framework based on infrared guidance to improve depth consistency in Time of Flight image systems. The primary objective is to formulate the problem as an image energy optimization task, aimed at maximizing the coherence between the depth map and the corresponding infrared image, both captured simultaneously from the same Time of Flight sensor. The concept of depth consistency relies on the underlying hypothesis concerning the correlation between depth maps and their corresponding infrared images. The proposed optimization framework adopts a weighted approach, leveraging an iterative estimator. The image energy is characterized by introducing spatial conditional entropy as a correlation measure and spatial error as image regularization. To address the issue of missing depth values, a preprocessing step is initially applied, by using a depth completion method based on infrared guided belief propagation, which was proposed in a previous work. Subsequently, the proposed framework is employed to regularize and enhance the inpainted depth. The experimental results demonstrate a range of qualitative improvements in depth map reconstruction, with a particular emphasis on the sharpness and continuity of edges.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Achaibou, A., Sanmartín-Vich, N., Pla, F., Calpe, J.: Guided depth completion using active infrared images in time of flight systems. In: Pertusa, A., Gallego, A.J., Sánchez, J.A., Domingues, I. (eds.) Pattern Recognition and Image Analysis: 11th Iberian Conference, IbPRIA 2023, Alicante, Spain, 27–30 June 2023, Proceedings, pp. 323–335. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-36616-1_26
Ahmed, Z., Shahzad, A., Ali, U.: Enhancement of depth map through weighted combination of guided image filters in shape-from-focus. In: 2022 2nd International Conference on Digital Futures and Transformative Technologies (ICoDT2), pp. 1–7. IEEE (2022)
Diebel, J., Thrun, S.: An application of Markov random fields to range sensing. Adv. Neural Inf. Process. Syst. 18 (2005)
Ferstl, D., Reinbacher, C., Ranftl, R., Rüther, M., Bischof, H.: Image guided depth upsampling using anisotropic total generalized variation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 993–1000 (2013)
Gong, X., Liu, J., Zhou, W., Liu, J.: Guided depth enhancement via a fast marching method. Image Vis. Comput. 31(10), 695–703 (2013)
Gu, S., Xie, Q., Meng, D., Zuo, W., Feng, X., Zhang, L.: Weighted nuclear norm minimization and its applications to low level vision. Int. J. Comput. Vision 121, 183–208 (2017)
Hui, T.W., Ngan, K.N.: Depth enhancement using RGB-D guided filtering. In: 2014 IEEE International Conference on Image Processing (ICIP), pp. 3832–3836. IEEE (2014)
Izadi, S., et al.: Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, pp. 559–568 (2011)
Jiang, Z., Hou, Y., Yue, H., Yang, J., Hou, C.: Depth super-resolution from RGB-D pairs with transform and spatial domain regularization. IEEE Trans. Image Process. 27(5), 2587–2602 (2018)
Kwon, H., Tai, Y.W., Lin, S.: Data-driven depth map refinement via multi-scale sparse representation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 159–167 (2015)
Liu, J., Gong, X.: Guided depth enhancement via anisotropic diffusion. In: Huet, B., Ngo, C.-W., Tang, J., Zhou, Z.-H., Hauptmann, A.G., Yan, S. (eds.) PCM 2013. LNCS, vol. 8294, pp. 408–417. Springer, Cham (2013). https://doi.org/10.1007/978-3-319-03731-8_38
Liu, W., Chen, X., Yang, J., Wu, Q.: Robust color guided depth map restoration. IEEE Trans. Image Process. 26(1), 315–327 (2016)
Lu, S., Ren, X., Liu, F.: Depth enhancement via low-rank matrix completion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3390–3397 (2014)
Or-El, R., Rosman, G., Wetzler, A., Kimmel, R., Bruckstein, A.M.: RGBD-fusion: real-time high precision depth recovery. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5407–5416 (2015)
Shen, Y., Li, J., Lü, C.: Depth map enhancement method based on joint bilateral filter. In: 2014 7th International Congress on Image and Signal Processing, pp. 153–158. IEEE (2014)
Valsesia, D., Fracastoro, G., Magli, E.: Deep graph-convolutional image denoising. IEEE Trans. Image Process. 29, 8226–8237 (2020)
Yang, S., Liu, J., Fang, Y., Guo, Z.: Joint-feature guided depth map super-resolution with face priors. IEEE Trans. Cybernet. 48(1), 399–411 (2016)
Yi, K., Zhao, Y., Lei, Y., Pan, J.: Depth enhancement with improved inpainting order and smoothing method. J. Phys. Conf. Ser. 1187, 042065 (2019). IOP Publishing
Zhang, L., et al.: Depth enhancement with improved exemplar-based inpainting and joint trilateral guided filtering. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 4102–4106. IEEE (2016)
Zhang, X., Wu, R.: Fast depth image denoising and enhancement using a deep convolutional network. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2499–2503. IEEE (2016)
Zhang, X., Ye, P., Xiao, G.: Vifb: a visible and infrared image fusion benchmark supplementary document
Zhu, J., Zhang, J., Cao, Y., Wang, Z.: Image guided depth enhancement via deep fusion and local linear regularization. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 4068–4072. IEEE (2017)
Acknowledgments
This work was partially supported by Analog Devices, Inc. and by the Agencia Valenciana de la Innovacion of the Generalitat Valenciana under program “Plan GEnT. Doctorados Industriales. Innodocto”.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 Springer Nature Switzerland AG
About this paper
Cite this paper
Achaibou, A., Pla, F., Calpe, J. (2024). IR-Guided Energy Optimization Framework for Depth Enhancement in Time of Flight Imaging. In: Vasconcelos, V., Domingues, I., Paredes, S. (eds) Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications. CIARP 2023. Lecture Notes in Computer Science, vol 14469. Springer, Cham. https://doi.org/10.1007/978-3-031-49018-7_46
Download citation
DOI: https://doi.org/10.1007/978-3-031-49018-7_46
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-49017-0
Online ISBN: 978-3-031-49018-7
eBook Packages: Computer ScienceComputer Science (R0)
