Shuo Qin
ABSTRACT
Depth images acquired by consumer depth sensors, such as Kinect and ToF, usually are of low resolution and insufficient quality. This limits the application of these depth sensors. Therefore, depth map enhancement is essential to its application. Most existing depth map super-resolution methods employ an RGB image of the same scene in the depth image as a guidance to up-sample the depth map. However, due to part of edges in RGB image do not occurrence in depth image, such as texture in RGB image, most existing methods will introduce a problem of texture-copy in these areas. To address this problem, we propose an approach that introduce semantic information of RGB image. On the other hand, existing methods rely on various kinds of explicit filter construction or hand-designed objective function. It is thus difficult to understand, improve, and accelerate them in a coherent framework. In this paper we use a learning-based approach to construct a joint filter based on Convolutional Neural Networks. In contrast to existing methods that consider only the RGB guidance image, our method can suppress the texture-copy problem. We validate the effectiveness of the proposed method through extensive comparisons with state-of-the-art methods on NYU v2 dataset. Experiment results show that our method suppress the texture-copy problem.
- Aodha, O. M., Campbell, N. D. F., Nair, A., and Brostow, G. J. 2012. Patch based synthesis for single depth image super-resolution. In: European Conference on Computer Vision. pp. 71--84. Google Scholar
Digital Library
- Chan, D., Buisman, H., Theobalt, C., and Thrun, S. 2008. A noise-aware filter for real-time depth up-sampling. In: The Workshop on Multi-Camera and Multi-Modal Sensor Fusion Algorithms and Applications.Google Scholar
- Diebel, J. and Thrun, S. 2005. An application of markov random fields to range sensing. Advances in Neural Information Processing Systems. pp. 291--298. Google ScholarDigital Library
- Dong, C., Chen, C. L., He, K., and Tang, X. 2016. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence. 38(2), 295--307. Google ScholarDigital Library
- Ferstl, D., Reinbacher, C., Ranftl, R., Ruether, M., and Bischof, H. 2014. Image guided depth up-samjpling using anisotropic total generalized variation. In: IEEE International Conference on Computer Vision. pp. 993--1000. Google ScholarDigital Library
- Ferstl, D., Ruther, M., and Bischof, H. 2015. Variational depth super-resolution using example-based edge representations. In: IEEE International Conference on Computer Vision. pp. 513--521. Google ScholarDigital Library
- Gregor, K. and Lecun, Y. 2010. Learning fast approximations of sparse coding. In: International Conference on International Conference on Machine Learning. pp. 399--406. Google ScholarDigital Library
- Ham, B., Cho, M., and Ponce, J. 2015. Robust image filtering using joint static and dynamic guidance. In: Computer Vision and Pattern Recognition. pp. 4823--4831.Google Scholar
- He, K., Sun, J., and Tang, X. 2010. Guided image filtering. In: European Conference on Computer Vision. pp. 1--14.Google Scholar
- Hirschmuller, H. and Scharstein, D. 2007. Evaluation of cost functions for stereo matching. In: Computer Vision and Pattern Recognition, 2007. CVPR '07. IEEE Conference on. pp. 1--8.Google Scholar
- Kopf, J., Cohen, M. F., Lischinski, D., Uyttendaele, M. 2007. Joint bilateral up-sampling. In: Acm Siggraph. p. 96. Google ScholarDigital Library
- Li, Y., Huang, J. B., Ahuja, N., and Yang, M. H. 2016. Deep Joint Image Filtering. Springer International Publishing.Google Scholar
- Liu, M. Y., Tuzel, O., and Taguchi, Y. 2013. Joint geodesic up-sampling of depth images. In: Computer Vision and Pattern Recognition. pp. 169--176. Google ScholarDigital Library
- Mandal, S., Bhavsar, A., and Sao, A. K. 2014. Hierarchical example-based range-image super-resolution with edge-preservation. pp. 3867--3871.Google Scholar
- Osendorfer, C., Soyer, H., and Smagt, P.V. D. 2014. Image super-resolution with fast approximate convolutional sparse coding. Lecture Notes in Computer Science 8836, 250--257.Google ScholarCross Ref
- Park, J., Kim, H., Tai, Y.W., Brown, M. S., and Kweon, I. 2011. High quality depth map up-sampling for 3d-tof cameras. In: IEEE International Conference on Computer Vision. pp. 1623--1630. Google ScholarDigital Library
- Scharstein, D. and Pal, C. 2007. Learning conditional random fields for stereo. In: Computer Vision and Pattern Recognition, 2007. CVPR '07. IEEE Conference on. pp. 1--8.Google Scholar
- Silberman, N., Hoiem, D., Kohli, P., and Fergus, R. 2012. Indoor Segmentation and Support Inference from RGBD Images. Springer Berlin Heidelberg.Google Scholar
- Tomasi, C. and Manduchi, R. 1998. Bilateral filtering for gray and color images iccv. Proc ICCV p. 839. Google ScholarDigital Library
- Vedaldi, A. and Lenc, K. 2014. Matconvnet: Convolutional neural networks for matlab pp. 689--692. Google ScholarDigital Library
- Wang, Y., Ortega, A., Tian, D., and Vetro, A. 2014. A graph-based joint bilateral approach for depth enhancement pp. 885--889.Google Scholar
- Wang, Z., Liu, D., Yang, J., Han, W., and Huang, T. 2016. Deep networks for image super-resolution with sparse prior. In: IEEE International Conference on Computer Vision. pp. 370--378. Google ScholarDigital Library
- Yang, Q., Yang, R., Davis, J., and Nister, D. 2007. Spatial-depth super resolution for range images. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 1--8.Google Scholar
Index Terms
- Depth Image Super-Resolution with Semantic and RGB Images Using CNN
Recommendations
Single Depth Map Super-resolution with Local Self-similarity
ICVIP '18: Proceedings of the 2018 2nd International Conference on Video and Image ProcessingConsumer depth sensors such as time-of-flight camera or Kinect have gained significant popularity in recently. However, the captured depth maps suffer from limited spatial resolution and a variety of noise, making such depth maps difficult to be ...
Symmetric Uncertainty-Aware Feature Transmission for Depth Super-Resolution
MM '22: Proceedings of the 30th ACM International Conference on MultimediaColor-guided depth super-resolution (DSR) is an encouraging paradigm that enhances a low-resolution (LR) depth map guided by an extra high-resolution (HR) RGB image from the same scene. Existing methods usually use interpolation to upscale the depth ...
Depth map Super-Resolution based on joint dictionary learning
Although Time-of-Flight (ToF) camera can provide real-time depth information from a real scene, the resolution of depth map captured by ToF camera is rather limited compared to HD color cameras, and thus it cannot be directly used in 3D reconstruction. ...
Comments