Skip to main content
SpringerLink
Log in

Non-convex joint bilateral guided depth upsampling

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Blurring depth edges and texture copy artifacts are challenging issues for guided depth map upsampling. They are caused by the inconsistency between depth edges and corresponding color edges. In this paper, we extend the well-known Joint Bilateral Upsampling (JBU) (Kopf et al. 2007) with a novel non-convex optimization framework for guided depth map upsampling, which is denoted as Non-Convex JBU (NCJBU). We show that the proposed NCJBU can well handle the edge inconsistency by making use of the property of both the guidance color image and the depth map. Through comprehensive experiments, we show that our NCJBU can preserve sharp depth edges and properly suppress texture copy artifacts. In addition, we present a data driven scheme to properly determine the parameter in our model such that fine details and sharp depth edges are well preserved even for a large upsampling factor (e.g., 8 ×). Experimental results on both simulated and real data show the effectiveness of our method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. Of course, there are plenty of variants of JBU. We just point out a small fraction of them.

References

  1. Buades A, Coll B, Morel J-M (2005) A non-local algorithm for image denoising. In: IEEE conference on computer vision and pattern recognition, vol 2, pp 60–65

  2. Diebel J, Thrun S (2005) An application of Markov random fields to range sensing. In: Proceedings of the advances in neural information processing systems

  3. Farsiu S, Robinson MD, Elad M, Milanfar P (2004) Fast and robust multiframe super resolution. IEEE Trans Image Process 13(10):1327–1344

    Article  Google Scholar 

  4. Ferstl D, Rüther M, Bischof H (2015) Variational depth superresolution using example-based edge representations. In: IEEE international conference on computer vision

  5. Ferstl D, Reinbacher C, Ranftl R, Rüther M, Bischof H (2013) Image guided depth upsampling using anisotropic total generalized variation. In: IEEE international conference on computer vision, pp 993–1000

  6. Gao Y, Wang M, Ji R, Wu X, Dai Q (2014) 3-D object retrieval with Hausdorff distance learning. IEEE Trans Ind Electron 61(4):2088–2098

    Article  Google Scholar 

  7. Hahne U, Alexa M (2011) Exposure Fusion for time-of-flight imaging. Computer graphics forum 30(7):1887–1894

    Article  Google Scholar 

  8. He K, Sun J, Tang X (2013) Guided image filtering. IEEE Trans Pattern Anal Mach Intell 35(6):1397–1409

    Article  Google Scholar 

  9. Hornacek M, Rhemann C, Gelautz M, Rother C (2013) Depth super resolution by rigid body self-similarity in 3D. In: IEEE conference on computer vision and pattern recognition

  10. Huhle B, Schairer T, Jenke P, Straßer W (2008) Robust non-local denoising of colored depth data. In: CVPRW, pp 1–7

  11. Jiang X, Yao H, Zhao S (2017) Text image deblurring via two-tone prior. Neurocomputing 242:1–14

    Article  Google Scholar 

  12. Kiechle M, Hawe S, Kleinsteuber M (2013) A joint intensity and depth co-sparse analysis model for depth map super-resolution. In: IEEE international conference on computer vision, pp 1545–1552

  13. Kopf J, Cohen MF, Lischinski D, Uyttendaele M (2007) Joint bilateral upsampling. In: ACM transactions on graphics, vol 26, p 96

  14. Li J, Lu Z, Zeng G, Gan R, Zha H (2014) Similarity-aware patchwork assembly for depth image super-resolution. In: IEEE conference on computer vision and pattern recognition

  15. Liu M-Y, Tuzel O, Taguchi Y (2013) Joint geodesic upsampling of depth images. In: IEEE conference on computer vision and pattern recognition

  16. Liu W, Chen X, Yang J, Wu Q (2017) Robust color guided depth map restoration. IEEE Trans Image Process 26(1):315–327

    Article  MathSciNet  Google Scholar 

  17. Mac Aodha O, Campbell ND, Nair A, Brostow GJ (2012) Patch based synthesis for single depth image super-resolution. In: European conference on computer vision

  18. Park J, Kim H, Tai Y-W, Brown MS, Kweon I (2011) High quality depth map upsampling for 3d-tof cameras. In: IEEE international conference on computer vision

  19. Perona P, Malik J (1990) Scale-space and edge detection using anisotropic diffusion. IEEE Trans Pattern Anal Mach Intell 12(7):629–639

    Article  Google Scholar 

  20. Pizarro L, Mrázek P, Didas S, Grewenig S, Weickert J (2010) Generalised nonlocal image smoothing. Int J Comput Vis 90(1):62–87

    Article  MathSciNet  Google Scholar 

  21. Rajagopalan AN, Bhavsar AV, Wallhoff F, Rigoll G (2008) Resolution enhancement of PMD range maps. In: Pattern recognition, 30th DAGM symposium, Munich, Germany, pp 304–313

  22. Schuon S, Theobalt C, Davis J, Thrun S (2009) Lidarboost: depth superresolution for tof 3d shape scanning. In: IEEE conference on computer vision and pattern recognition

  23. Tomasi C, Manduchi R (1998) Bilateral filtering for gray and color images. In: IEEE international conference on computer vision

  24. Yan J, Wang J, Zha H, Yang X, Chu SM (2015) Consistency-driven alternating optimization for multigraph matching: A unified approach. IEEE Trans. Image Processing 24(3):994–1009

    Article  MathSciNet  Google Scholar 

  25. Yan J, Cho M, Zha H, Yang X, Chu SM (2016) Multi-graph matching via affinity optimization with graduated consistency regularization. IEEE Trans. Pattern Anal. Mach. Intell. 38(6):1228–1242

    Article  Google Scholar 

  26. Yang J, Ye X, Li K, Hou C (2012) Depth recovery using an adaptive color-guided auto-regressive model. In: European conference on computer vision

  27. Yang J, Ye X, Li K, Hou C, Wang Y (2014) Color-guided depth recovery from rgb-d data using an adaptive autoregressive model. IEEE Trans Image Process 23 (8):3443–3458

    Article  MathSciNet  MATH  Google Scholar 

  28. Yang Q, Yang R, Davis J, Nistér D (2007) Spatial-depth super resolution for range images. In: IEEE conference on computer vision and pattern recognition

  29. Zhao S, Chen L, Yao H, Zhang Y, Sun X (2015) Strategy for dynamic 3d depth data matching towards robust action retrieval. Neurocomputing 151:533–543

    Article  Google Scholar 

  30. Zhao S, Yao H, Zhang Y, Wang Y, Liu S (2015) View-based 3d object retrieval via multi-modal graph learning. Signal Process 112:110–118

    Article  Google Scholar 

  31. Zimmer H, Bruhn A, Weickert J (2011) Optic flow in harmony. Int J Comput Vis 93(3):368–388

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Key Basic Research and Development Program of China under Grant 2012CB719903, the Science Fund for Creative Research Groups of the National Natural Science Foundation of China under Grant 61221003, the National Natural Science Foundation of China under Grant 41071256, 41571402, the National Science Foundation of China Youth Program under Grant 41101386.

Author information

Authors and Affiliations

  1. Automation of Shanghai Jiao Tong University, 800, Dongchuan, Shanghai, China

    Xiankai Lu, Yiyou Guo, Na Liu, Lihong Wan & Tao Fang

Authors
  1. Xiankai Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yiyou Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Na Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lihong Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tao Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tao Fang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lu, X., Guo, Y., Liu, N. et al. Non-convex joint bilateral guided depth upsampling. Multimed Tools Appl 77, 15521–15544 (2018). https://doi.org/10.1007/s11042-017-5131-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-5131-x

Keywords

Search

Navigation