Skip to main content
SpringerLink
Log in

Fixing algorithm of Kinect depth image based on non-local means

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The three-dimensional (3D) geometrical information that depth maps contain is useful in many applications such as 3D reconstruction or simultaneous localization and mapping (SLAM). Kinect is widely used in depth image acquisition due to its low cost and good real-time performance. However, the quality of depth images obtained by Kinect is influenced by holes which make depth image inadequate for further applications. To suppress the influence of holes on a subsequent application, a fixing algorithm of Kinect depth image based on non-local means (NLM) is proposed in this paper. The holes in depth image are filled using the weights which are calculated on the corresponding gray image by distance factor and value consistent factor. And the experiment results demonstrate that the proposed method achieves good performance in both evaluation in metrics and subjectively visual effect. This research provides a solution idea for depth image fixing algorithm with low complexity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Algorithm 2
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Achanta SDM, Karthikeyan T, Vinothkanna RA (2019) Novel hidden Markov model-based adaptive dynamic time warping (HMDTW) gait analysis for identifying physically challenged persons. Soft Comput 23:8359–8366

    Article  Google Scholar 

  2. Behroozpour B, Sandborn PAM, Wu MC, Boser BE (2017) Lidar system architectures and circuits. IEEE Commun Mag 55:135–142

    Article  Google Scholar 

  3. Buades A, Coll B, Morel JM (2005) Image denoising by non-local averaging. In: Proceedings. (ICASSP ’05). IEEE International conference on acoustics, speech, and signal processing, vol 2, pp 25–28

  4. Cai M, Wang Y, Wang S, Wang R, Tan M (2019) ROS-based depth control for hybrid-driven underwater vehicle-manipulator system. In: 2019 Chinese Control Conference (CCC), pp 4576–4580

  5. Chang T-A, Liao W-C, Yang J-F (2018) Robust depth enhancement based on texture and depth consistency. IET Signal Proc 12(1):119–128

    Article  Google Scholar 

  6. Chen M, Chiang C, Lu Y (2014) Depth estimation for hand-held light field cameras under low light conditions. In: 2014 International conference on 3D imaging (IC3d), pp 1–4

  7. Chen Z, Wang H, Wu L, Zhou Y, Wu D (2020) Spatiotemporal guided self-supervised depth completion from liDAR and monocular camera. In: 2020 IEEE International conference on visual communications and image processing (VCIP), pp 54–57

  8. Cho J, Park S, Chien S (2020) Hole-filling of RealSense depth images using a color edge map. IEEE Access 8:53901–53914

    Article  Google Scholar 

  9. Criminisi A, Perez P, Toyama K (2004) Area filling and object removal by exemplar-based image inpainting. IEEE Trans Image Process 13 (9):1200–1212

    Article  Google Scholar 

  10. Feng C, Dai SL (2014) Adaptive depth map enhancement based on joint bilateral filter. In: Proceedings of 2014 IEEE Chinese guidance, navigation and control conference, pp 2568–2571

  11. Hirschmuller H, Scharstein D (2007) Evaluation of cost functions for stereo matching. In: 2007 IEEE Conference on computer vision and pattern recognition, pp 1–8

  12. Hung S, Lo S, Hang H (2019) Incorporating luminance, depth and color information by a fusion-based network for semantic segmentation. In: 2019 IEEE International conference on image processing (ICIP), pp 2374–2378

  13. Jayachandran A, Preetha VH (2016) Application of exemplar-based inpainting in depth image based rendering. In: 2016 IEEE International conference on recent trends in electronics, information & communication technology (RTEICT), pp 1117–1121

  14. Jing N, Ma X, Guo W (2018) 3D reconstruction of underground tunnel using Kinect camera. In: 2018 International symposium on computer, consumer and control (IS3C), pp 278–281

  15. Lee G-W, Han J-K (2021) Hole concealment for depth image using pixel classification in multiview system. In: 2021 IEEE International conference on consumer electronics (ICCE), pp 1–5

  16. Liang C, Su S, Chen M (2017) Non-pre-process calibration of depth image based on fuzzy c-mean. In: 2017 International conference on system science and engineering (ICSSE), pp 125–128

  17. Liao X, Zhang X (2017) Multi-scale mutual feature convolutional neural network for depth image denoise and enhancement. In: 2017 IEEE Visual communications and image processing (VCIP) 1–4

  18. Nguyen T-D, Kim B, Hong M-C (2019) New Hole-Filling method using extrapolated Spatio-Temporal background information for a synthesized Free-View. IEEE Trans Multimed 21(6):1345–1358

    Article  Google Scholar 

  19. Pan L, Dai Y, Liu M, Porikli F (2018) Depth map completion by jointly exploiting blurry color images and sparse depth maps. In: 2018 IEEE winter conference on applications of computer vision (WACV), pp 1377–1386

  20. Rossi M, Gheche ME, Kuhn A, Frossard P (2020) Joint graph-based depth refinement and normal estimation. In: 2020 IEEE/CVF Conference on computer vision and pattern recognition (CVPR), pp 12151–12160

  21. Scharstein D, Hirschmuller H, Kitajima Y, Krathwohl G, Nesic N, Wang X, Westling P (2014) High-resolution stereo datasets with subpixel-accurate ground truth Pattern Recognition. GCPR 2014(8753):31–42

    Google Scholar 

  22. Sen X, Huiping D, Lei Z, Jin W, Li Y (2019) Exemplar-based depth inpainting with arbitrary-shape patches and cross-modal matching. Signal Process: Image Commun 71:56–65

    Google Scholar 

  23. Sharma N, Achanta SDM, Karthikeyan T, Kumari CU, Jagan BOL (2020) Gait diagnosis using fuzzy logic with wearable tech for prolonged disorders of diabetic cardiomyopathy Gait diagnosis using fuzzy logic with wearable tech for prolonged disorders of diabetic cardiomyopathy: Materials Today. Proceedings

  24. Song Z (2018) High-speed 3D shape measurement with structured light methods: A review. Opt Lasers Eng 106:119–131

    Article  Google Scholar 

  25. Song W, Le AV, Yun S, Jung S-W, Won CS (2017) Depth completion for kinect v2 sensor. Multimed Tools Appl 76(3):4357–4380

    Article  Google Scholar 

  26. Sun M-J, Zhang J-M (2019) Single-pixel imaging and its application in three-dimensional reconstruction: A brief review. Sensors 19(3):732

    Article  Google Scholar 

  27. Xiaodong B, Bailin Y, Jia Z, Tianxiang W, Yiming X (2021) A novel holes filling method based on layered depth map and patch sparsity for complex-scene images. Microelectron J 114:105140

    Article  Google Scholar 

  28. Zhong Y, Pei Y, Li P, Guo Y, Ma G, Liu M, Bai W, Wu W, Zha H (2021) Depth-based 3D face reconstruction and pose estimation using shape-preserving domain adaptation. IEEE Trans Biom Behav Identity Sci 3:6–15

    Article  Google Scholar 

  29. Zhou H, Li Y, Tian X, Li X, Zhang XA (2015) Robust iterative nonlocal means method for electrocardiogram signal denoising. In: 2015 IET International conference on biomedical image and signal processing (ICBISP) 2015, pp 1–5

  30. Xin J, Yatong X, Qionghai D (2016) Depth dithering based on texture edge-assisted classification. Signal Process: Image Commun 47:56–71

    Google Scholar 

  31. Zhang L, Xia H, Qiao Y (2020) Texture synthesis repair of RealSense D435i depth images with object-oriented RGB image Segmentation. Sensors 20 (23):6725

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by Key Research and Development Program of Shaanxi under Grant 2020KW-010, in part by National Natural Science Foundation of China under Grant 61801384 and Grant 61971343, in part by Natural Science Basic Research Plan in Shaanxi Province of China under Grant 2020JM-415, and in part by Northwest University Paleontological Bioinformatics Innovation Team under Grant 2019TD-012.

Author information

Authors and Affiliations

  1. School of Information Science and Technology, Northwest University, Shaanxi, Xi’an, 710127, China

    Lin Wang, Chengfeng Liao, Rui Zhang, Wanxu Zhang, Xiaoxuan Chen, Na Meng, Zenghui Yan, Bo Jiang & Cheng Liu

  2. College of Artificial Intelligence, Xi’an Jiaotong University, Shaanxi, Xi’an, 710049, China

    Runzhao Yao

Authors
  1. Lin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chengfeng Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Runzhao Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rui Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wanxu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xiaoxuan Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Na Meng
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Zenghui Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Bo Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Cheng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Bo Jiang or Cheng Liu.

Ethics declarations

Conflict of Interests

All authors declare that there is no conflict of interest regarding the publication of this article.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, L., Liao, C., Yao, R. et al. Fixing algorithm of Kinect depth image based on non-local means. Multimed Tools Appl 83, 787–806 (2024). https://doi.org/10.1007/s11042-023-15194-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15194-3

Keywords

Search

Navigation