Local-linear-fitting-based matting for joint hole filling and depth upsampling of RGB-D images

Yanfu Zhang; Li Ding; Gaurav Sharma

doi:10.1117/1.JEI.28.3.033019

4 June 2019 Local-linear-fitting-based matting for joint hole filling and depth upsampling of RGB-D images

Yanfu Zhang, Li Ding, Gaurav Sharma

Author Affiliations +

Journal of Electronic Imaging, Vol. 28, Issue 3, 033019 (June 2019). https://doi.org/10.1117/1.JEI.28.3.033019

Abstract

We propose an approach for jointly filling holes and upsampling depth information for RGB-D images captured with common acquisition systems, where RGB color information is available at all pixel locations whereas depth information is only available at lower resolution and entirely missing in small regions referred to as “holes.” Depth information completion is formulated as a minimization of an objective function composed of two additive terms. The first data fidelity term penalizes disagreement with the observed low-resolution data. The second regularization term penalizes weighted depth deviations from a local linear model in spatial coordinates, where the weights are experimentally determined to ensure consistency between the RGB color image and the estimated depth image. Analogous to techniques used for optimization formulations of image matting, the completed depth image is then obtained by solving a large sparse linear system of equations. We also propose a memory-efficient implementation of the proposed method based on the conjugate gradient method. Visual evaluation of results obtained with the proposed algorithm demonstrates that the method provides high-resolution depth maps that are consistent with the color images. Furthermore, the memory-efficient implementation significantly reduces memory requirements, allowing for computation of the upsampled, hole-filled depth maps for typical RGB-D images on normal workstation hardware. Quantitative comparisons demonstrate that the method offers an improvement in accuracy over the current state-of-the-art techniques for depth information completion. Importantly, statistical analysis, which we present in this paper, also reveals that prior evaluations of depth upsampling accuracy are potentially biased because the evaluations inappropriately used preprocessed hole-filled data as “ground truth.” An implementation of the proposed algorithm can be accessed and executed through Code Ocean: https://codeocean.com/capsule/5103691/tree/v1.

Citation Download Citation

Yanfu Zhang, Li Ding, and Gaurav Sharma "Local-linear-fitting-based matting for joint hole filling and depth upsampling of RGB-D images," Journal of Electronic Imaging 28(3), 033019 (4 June 2019). https://doi.org/10.1117/1.JEI.28.3.033019

Received: 4 January 2019; Accepted: 7 May 2019; Published: 4 June 2019

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available