Joint Residual Pyramid for Depth Map Super-Resolution

Abstract

High-resolution (HR) depth map can be better inferred from a low-resolution (LR) one with the guidance of an additional HR texture map of the same scene. Recently, deep neural networks with large receptive fields are shown to benefit applications such as image completion. Our insight is that super resolution (SR) is similar to image completion, where only parts of the depth values are precisely known. However, large receptive fields in general will increase the depth and the number of parameters of the network, which may cause degradation and large memory consumption. To solve these problems, we adapt the convolutional neural pyramid (CNP) structure by introducing residual block and linear interpolation layer, and adopt the CNP in the joint super-resolution framework. We call this convolutional neural model joint residual pyramid (JRP). Our JRP model consists of three sub-networks, two convolutional neural residual pyramids concatenated by a normal convolutional neural network. The convolutional neural residual pyramids extract information from large receptive fields of the depth map and guidance map, while the convolutional neural network effectively transfers useful structures of the guidance image to the depth image. Experimental results show that our model outperforms existing state-of-the-art algorithms not only on data pairs of RGB/depth images, but also on other data pairs like color/saliency and color-scribbles/colorized images.

The work is supported by the National Key Research and Development Program of China(Grant Num: 2018YFB0203900), NSFC from PRC (Grant Num.: 61502158), Hunan NSF (Grant Num.: 2017JJ3042, 2018JJ3067), and China Postdoctoral Foundation (Grant Num.: 2016M590740).