Skip to main content
Log in

Probability contour guided depth map inpainting and superresolution using non-local total generalized variation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper proposes an image-guided depth super-resolution framework to improve the quality of depth map captured by low-cost depth sensors, like the Microsoft Kinect. First, a contour-guided fast marching method is proposed to preprocess the raw depth map for recovering the missing data. Then, by using the non-local total generalized variation (NL-TGV) regularization, a convex optimization model is constructed to up-sample the preprocessed depth map to a high-resolution one. To preserve the sharpness of depth discontinuities, the color image and its multi-level segmentation information are utilized to assign the weights within the NL-TGV through a novel weight combining scheme. The texture energy from color image and local structure coherence around neighbor pixels in low-resolution depth map are applied to adjust the combination weights for further suppressing texture-transfer. Quantitative and qualitative evaluations of the proposed method on the Middlebury datasets and real-sensor datasets show the promising results in quality.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Arbelaez P, Maire M, Fowlkes C, Malik J (2011) Contour detection and hierarchical image segmentation. IEEE Trans Pattern Anal Mach Intell 33(5):898–916

    Article  Google Scholar 

  2. Buades A, Coll B, Morel J-M (2011) Non-local means denoising. Image Processing On Line 1:6

    MATH  Google Scholar 

  3. Chambolle A, Pock T (2011) A first-order primal-dual algorithm for convex problems with applications to imaging. J Math Imaging Vis 40(1):120–145

    Article  MathSciNet  MATH  Google Scholar 

  4. Chan D, Buisman H, Theobalt C, Thrun S (2008) A noise-aware filter for real-time depth upsampling Proc. ECCV workshop multi-camera multi-modal sensor fus. Algorithms Appl.

    Google Scholar 

  5. Cui Y, Schuon S, Chan D, Thrun S, Theobalt C (2010) 3d shape scanning with a time-of-flight camera Proc. Comput. Vis. Pattern Recognit (CVPR), pp 1173–1180

    Google Scholar 

  6. Diebel J, Thrun S (2005) An application of markov random fields to range sensing Advances in neural information processing systems (NIPS), vol 5, pp 291–298

  7. Dollár P, Zitnick CL (2015) Fast edge detection using structured forests. IEEE Trans Pattern Anal Mach Intell 37(8):1558–1570

    Article  Google Scholar 

  8. Ferstl D, Reinbacher C, Ranftl R, Rüther M, Bischof H (2013) Image guided depth upsampling using anisotropic total generalized variation Proc Int. Conf. Comput. Vis. (ICCV), pp 993–1000

  9. Ferstl D, Ruther M, Bischof H (2015) Variational depth superresolution using example-based edge representations Proc Int. Conf. Comput. Vis. (ICCV), pp 513–521

    Google Scholar 

  10. He K, Sun J, Tang X (2013) Guided image filtering. IEEE Trans Pattern Anal Mach Intell 35(6):1397–1409

    Article  Google Scholar 

  11. Hui T-W, Loy CC, Tang X (2016) Depth map super-resolution by deep multi-scale guidance Proc Eur. Conf. Comput. Vis. (ECCV). Springer, pp 353–369

  12. Kiechle M, Hawe S, Kleinsteuber M (2013) A joint intensity and depth co-sparse analysis model for depth map super-resolution Proc Int. Conf. Comput. Vis. (ICCV), pp 1545–1552

  13. Kopf J, Cohen MF, Lischinski D, Uyttendaele M (2007) Joint bilateral upsampling. ACM Trans Graph 26(3):96

    Article  Google Scholar 

  14. Kwon H, Tai Y-W, Lin S (2015) Data-driven depth map refinement via multi-scale sparse representation Proc. Comput. Vis. Pattern Recognit (CVPR), pp 159–167

    Google Scholar 

  15. Li Y, Min D, Do MN, Lu J (2016) Fast guided global interpolation for depth and motion Proc Eur. Conf. Comput. Vis. (ECCV). Springer, pp 717–733

  16. Liu M-Y, Tuzel O, Taguchi Y (2013) Joint geodesic upsampling of depth images Proc. Comput. Vis. Pattern Recognit (CVPR), pp 169–176

    Google Scholar 

  17. Liu W, Jia S, Li P, Chen X, Yang J, Wu Q (2015) An mrf-based depth upsampling: upsample the depth map with its own property. IEEE Signal Process Lett 22(10):1708–1712

    Article  Google Scholar 

  18. Lu J, Shi K, Min D, Lin L, Do MN (2012) Cross-based local multipoint filtering Proc. Comput. Vis. Pattern Recognit (CVPR), pp 430–437

    Google Scholar 

  19. Min D, Lu J, Do MN (2012) Depth video enhancement based on weighted mode filtering. IEEE Trans Image Process 21(3):1176–1190

    Article  MathSciNet  MATH  Google Scholar 

  20. Min D, Choi S, Lu J, Ham B, Sohn K, Do MN (2014) Fast global image smoothing based on weighted least squares. IEEE Trans Image Process 23(12):5638–5653

    Article  MathSciNet  MATH  Google Scholar 

  21. Nair R, Lenzen F, Meister S, Schäfer H, Garbe C, Kondermann D (2012) High accuracy tof and stereo sensor fusion at interactive rates Proc Eur. Conf. Comput. Vis. (ECCV), pp 1–11

    Google Scholar 

  22. Nair R, Ruhl K, Lenzen F, Meister S, Schäfer H, Garbe CS, Eisemann M, Magnor M, Kondermann D (2013) A survey on time-of-flight stereo fusion Time-of-Flight and depth imaging. Sensors, Algorithms, and Applications. Springer, pp 105–127

  23. Parikh N, Boyd SP (2014) Proximal algorithms. Found Trends Optim 1 (3):127–239

    Article  Google Scholar 

  24. Park J, Kim H, Tai Y-W, Brown MS, Kweon I (2011) High quality depth map upsampling for 3d-tof cameras Proc Int. Conf. Comput. Vis. (ICCV). IEEE, pp 1623–1630

  25. Park J, Kim H, Tai Y-W, Brown MS, Kweon IS (2014) High-quality depth map upsampling and completion for rgb-d cameras. IEEE Trans Image Process 23(12):5559–5572

    Article  MathSciNet  MATH  Google Scholar 

  26. Pock T, Chambolle A (2011) Diagonal preconditioning for first order primal-dual algorithms in convex optimization Proc Int. Conf. Comput. Vis. (ICCV), pp 1762–1769

  27. Ranftl R, Bredies K, Pock T (2014) Non-local total generalized variation for optical flow estimation Proc Eur. Conf. Comput. Vis. (ECCV), pp 439–454

  28. Riegler G, Rüther M, Bischof H (2016) Atgv-net: accurate depth super-resolution Proc Eur. Conf. Comput. Vis. (ECCV). Springer, pp 268–284

  29. Scharstein D, Pal C (2007) Learning conditional random fields for stereo Proc. Comput. Vis. Pattern Recognit (CVPR), pp 1–8

    Google Scholar 

  30. Schuon S, Theobalt C, Davis J, Thrun S (2008) High-quality scanning using time-of-flight depth superresolution Proc. Comput. Vis. Pattern Recognit Workshop. (CVPRW), pp 1–7

    Google Scholar 

  31. Schuon S, Theobalt C, Davis J, Thrun S (2009) Lidarboost: Depth superresolution for tof 3d shape scanning Proc. Comput. Vis. Pattern Recognit. (CVPR), pp 343–350

    Google Scholar 

  32. Tan X, Sun C, Pham TD (2014) Multipoint filtering with local polynomial approximation and range guidance Proc. Comput. Vis. Pattern Recognit (CVPR), pp 2941–2948

    Google Scholar 

  33. Telea A (2004) An image inpainting technique based on the fast marching method. J Graphics Tools 9(1):23–34

    Article  Google Scholar 

  34. Wang L, Jin H, Yang R, Gong M (2008) Stereoscopic inpainting: joint color and depth completion from stereo images Proc. Comput. Vis. Pattern Recognit (CVPR), pp 1–8

    Google Scholar 

  35. Xiaofeng R, Bo L (2012) Discriminatively trained sparse code gradients for contour detection Advances in neural information processing systems (NIPS), pp 584–592

    Google Scholar 

  36. Yang Q, Yang R, Davis J, Nistér D (2007) Spatial-depth super resolution for range images Proc. Comput. Vis. Pattern Recognit (CVPR), pp 1–8

    Google Scholar 

  37. Yang Q, Ahuja N, Yang R, Tan K-H, Davis J, Culbertson B, Apostolopoulos J, Wang G (2013) Fusion of median and bilateral filtering for range image upsampling. IEEE Trans Image Process 22(12):4841–4852

    Article  MathSciNet  MATH  Google Scholar 

  38. Yang J, Ye X, Li K, Hou C, Wang Y (2014) Color-guided depth recovery from rgb-d data using an adaptive autoregressive model. IEEE Trans Image Process 23 (8):3443–3458

    Article  MathSciNet  MATH  Google Scholar 

  39. Yu L-F, Yeung S-K, Tai Y-W, Lin S (2013) Shading-based shape refinement of rgb-d images Proc. Comput. Vis. Pattern Recognit (CVPR), pp 1415–1422

    Google Scholar 

  40. Zhang S, Wang C, Chan S-C (2015) A new high resolution depth map estimation system using stereo vision and kinect depth sensing. J Signal Process Sys 79(1):19–31

    Article  Google Scholar 

  41. Zhu J, Wang L, Yang R, Davis JE, Pan Z (2011) Reliability fusion of time-of-flight depth and stereo geometry for high quality depth maps. IEEE Trans Pattern Anal Mach Intell 33(7):1400–1414

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Zeng-Fu Wang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, HT., Yu, J. & Wang, ZF. Probability contour guided depth map inpainting and superresolution using non-local total generalized variation. Multimed Tools Appl 77, 9003–9020 (2018). https://doi.org/10.1007/s11042-017-4791-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-4791-x

Keywords

Navigation