skip to main content
10.1145/3338533.3366582acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Surface Normal Data Guided Depth Recovery with Graph Laplacian Regularization

Published: 10 January 2020 Publication History

Abstract

High-quality depth information has been increasingly used in many real-world multimedia applications in recent years. Due to the limitation of depth sensor and sensing technology, actually, the captured depth map usually has low resolution and black holes. In this paper, inspired by the geometric relationship between surface normal of a 3D scene and their distance from camera, we discover that surface normal map can provide more spatial geometric constraints for depth map reconstruction, as depth map is a special image with spatial information, which we called 2.5D image. To exploit this property, we propose a novel surface normal data guided depth recovery method, which uses surface normal data and observed depth value to estimate missing or interpolated depth values. Moreover, to preserve the inherent piecewise smooth characteristic of depth maps, graph Laplacian prior is applied to regularize the inverse problem of depth maps recovery and a graph Laplacian regularizer(GLR) is proposed. Finally, the spatial geometric constraint and graph Laplacian regularization are integrated into a unified optimization framework, which can be efficiently solved by conjugate gradient(CG). Extensive quantitative and qualitative evaluations compared with state-of-the-art schemes show the effectiveness and superiority of our method.

References

[1]
2014. Middlebury Datasets. http://vision.middlebury.edu/stereo/data/
[2]
Weisheng Dong, Guangming Shi, Xin Li, Kefan Peng, Jinjian Wu, and Zhenhua Guo. 2016. Color-guided depth recovery via joint local structural and nonlocal low-rank regularization. IEEE Transactions on Multimedia 19, 2 (2016), 293--301.
[3]
David Ferstl, Christian Reinbacher, Rene Ranftl, Matthias Rüther, and Horst Bischof. 2013. Image guided depth upsampling using anisotropic total generalized variation. In Proceedings of the IEEE International Conference on Computer Vision. 993--1000.
[4]
C. Häne, L. Ladický, and M. Pollefeys. 2015. Direction matters: Depth estimation with a surface normal classifier. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 381--389. https://doi.org/10.1109/CVPR.2015.7298635
[5]
Yu Li, Dongbo Min, Minh N Do, and Jiangbo Lu. 2016. Fast guided global interpolation for depth and motion. In European Conference on Computer Vision. Springer, 717--733.
[6]
Wei Liu, Xiaogang Chen, Jie Yang, and Qiang Wu. 2016. Robust color guided depth map restoration. IEEE Transactions on Image Processing 26, 1 (2016), 315--327.
[7]
Xianming Liu, Gene Cheung, Xiaolin Wu, and Debin Zhao. 2016. Random walk graph Laplacian-based smoothness prior for soft decoding of JPEG images. IEEE Transactions on Image Processing 26, 2 (2016), 509--524.
[8]
Xiaojuan Qi, Renjie Liao, Zhengzhe Liu, Raquel Urtasun, and Jiaya Jia. 2018. Geonet: Geometric neural network for joint depth and surface normal estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 283--291.
[9]
AN Rajagopalan, Arnav Bhavsar, Frank Wallhoff, and Gerhard Rigoll. 2008. Resolution enhancement of pmd range maps. In Joint Pattern Recognition Symposium. Springer, 304--313.
[10]
Sebastian Schuon, Christian Theobalt, James Davis, and Sebastian Thrun. 2009. Lidarboost: Depth super-resolution for tof 3d shape scanning. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 343--350.
[11]
Xiaoyong Shen, Chao Zhou, Li Xu, and Jiaya Jia. 2015. Mutual-structure for joint filtering. In Proceedings of the IEEE International Conference on Computer Vision. 3406--3414.
[12]
Jingyu Yang, Xinchen Ye, Kun Li, Chunping Hou, and Yao Wang. 2014. Color-guided depth recovery from RGB-D data using an adaptive autoregressive model. IEEE transactions on image processing 23, 8 (2014), 3443--3458.
[13]
Yinda Zhang and Thomas Funkhouser. 2018. Deep depth completion of a single rgb-d image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 175--185.

Cited By

View all
  • (2020)Superb Monocular Depth Estimation Based on Transfer Learning and Surface Normal GuidanceSensors10.3390/s2017485620:17(4856)Online publication date: 27-Aug-2020

Index Terms

  1. Surface Normal Data Guided Depth Recovery with Graph Laplacian Regularization

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MMAsia '19: Proceedings of the 1st ACM International Conference on Multimedia in Asia
    December 2019
    403 pages
    ISBN:9781450368414
    DOI:10.1145/3338533
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 10 January 2020

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. depth map
    2. graph Laplacian regularizer(GLR)
    3. super-resolution(SR)
    4. surface normal data

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Funding Sources

    Conference

    MMAsia '19
    Sponsor:
    MMAsia '19: ACM Multimedia Asia
    December 15 - 18, 2019
    Beijing, China

    Acceptance Rates

    MMAsia '19 Paper Acceptance Rate 59 of 204 submissions, 29%;
    Overall Acceptance Rate 59 of 204 submissions, 29%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)11
    • Downloads (Last 6 weeks)1
    Reflects downloads up to 16 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2020)Superb Monocular Depth Estimation Based on Transfer Learning and Surface Normal GuidanceSensors10.3390/s2017485620:17(4856)Online publication date: 27-Aug-2020

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media