Skip to main content
SpringerLink
Log in

Color image-guided very low-resolution depth image reconstruction

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Deep learning-based image super-resolution research allows reconstruction of detailed images from low-resolution images in a short time. However, the existing depth image super-resolution reconstruction techniques focus on the reconstruction of low-resolution images with smaller downsampling multipliers, and the low-resolution images still contain richer depth information, and the superiority of super-resolution reconstruction techniques is not obvious enough. The reconstruction of depth images can be assisted by fully exploiting the rich details within the aligned 2D color images of the same scene and using the information contained internally. In this paper, we propose a network for reconstructing very low-resolution depth images with high-resolution 2D color images, mining the in-scale non-local features, cross-scale non-local features and its own priori information features of 2D color and depth images in the network. Fusing these features to greatly improve the detail accuracy of the reconstruction of very low-resolution depth images, we evaluated our method by comparing it with state-of-the-art methods comprehensively, and the experimental results demonstrate the superiority of our method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data availability

The data sets supporting the results of this article are included within the article.

References

  1. Szeliski, R., Zabih, R., Scharstein, D., Veksler, O., Kolmogorov, V., Agarwala, A., Tappen, M., Rother, C.: A comparative study of energy minimization methods for Markov random fields with smoothness-based priors. IEEE Trans. Pattern Anal. Mach. Intell. 30(6), 1068–1080 (2008)

    Article  Google Scholar 

  2. Scharstein, D., Szeliski, R., Zabih, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Stereo and Multi-Baseline Vision, 2001. (SMBV 2001). Proceedings. IEEE Workshop on (2002)

  3. Park, J., Kim, H., Tai, Y.W., Brown, M.S., Kweon, I.: High quality depth map upsampling for 3d-tof cameras. In: IEEE International Conference on Computer Vision, pp. 1623–1630 (2011)

  4. Park, J., Kim, H., Tai, Y.-W., Brown, M.S., Kweon, I.S.: High-quality depth map upsampling and completion for rgb-d cameras. IEEE Trans. Image Process. 23(12), 5559–5572 (2014). https://doi.org/10.1109/TIP.2014.2361034

    Article  MathSciNet  MATH  Google Scholar 

  5. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)

    Article  Google Scholar 

  6. Wronski, B., Garcia-Dorado, I., Ernst, M., Kelly, D., Krainin, M., Liang, C.-K., Levoy, M., Milanfar, P.: Handheld multi-frame super-resolution. Assoc. Comput. Mach. 38(4), 18 (2019)

    Google Scholar 

  7. Dong, C., Loy, C.C., Tang, X.: Accelerating the Super-Resolution Convolutional Neural Network, pp. 391–407. Springer, Cham (2016)

    Google Scholar 

  8. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1646–1654 (2016). https://doi.org/10.1109/CVPR.2016.182

  9. Kim, J., Lee, J.K., Lee, K.M.: Deeply-recursive convolutional network for image super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1637–1645 (2016). https://doi.org/10.1109/CVPR.2016.181

  10. Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2790–2798 (2017). https://doi.org/10.1109/CVPR.2017.298

  11. Wei, L., XuDong, Z.: Depth image super-resolution reconstruction based on convolution neural network. J. Electron. Meas. Instrum. 31(12), 1918–1928 (2017)

    Google Scholar 

  12. Yi, X., Xiang, C., Yan, Z., Xianyi, Z.: Joint residual pyramid for depth map super-resolution. PRICAI Trends Artif. Intell. 11012, 797–810 (2018)

    Google Scholar 

  13. Glasner, D., Bagon, S., Irani, M.: Super-resolution from a single image. In: IEEE 12th International Conference on Computer Vision, pp. 349–356 (2009). https://doi.org/10.1109/ICCV.2009.5459271

  14. Zontak, M., Irani, M.: Internal statistics of a single natural image. CVPR 2011, 977–984 (2011). https://doi.org/10.1109/CVPR.2011.5995401

    Article  Google Scholar 

  15. Dai, T., Cai, J., Zhang, Y., Xia, S.-T., Zhang, L.: Second-order attention network for single image super-resolution. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11057–11066 (2019). https://doi.org/10.1109/CVPR.2019.01132

  16. Liu, D., Wen, B., Fan, Y., Loy, C.C., Huang, T.S.: Non-local recurrent network for image restoration. Adv. Neural Inf. Process. Syst. 31 (2018)

  17. Mei, Y., Fan, Y., Zhou, Y., Huang, L., Huang, T.S., Shi, H.: Image super-resolution with cross-scale non-local attention and exhaustive self-exemplars mining. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5689–5698 (2020). https://doi.org/10.1109/CVPR42600.2020.00573

  18. Liu, W., Chen, X., Yang, J., Wu, Q.: Robust color guided depth map restoration. IEEE Trans. Image Process. 26(1), 315–327 (2017)

  19. Celik, T., Tjahjadi, T.: Image resolution enhancement using dual-tree complex wavelet transform. IEEE Geosci. Remote Sens. Lett. 7, 554–557 (2010)

    Article  Google Scholar 

  20. Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1132–1140 (2017). https://doi.org/10.1109/CVPRW.2017.151

  21. Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. Adv. Neural. Inf. Process. Syst. 3, 2672–2680 (2014)

    Google Scholar 

  22. Ferstl, D., Rüther, M., Bischof, H.: Variational depth super resolution using example-based edge representations. In: IEEE International Conference on Computer Vision (ICCV), pp. 513–521 (2015). https://doi.org/10.1109/ICCV.2015.66

  23. Song, X., Dai, Y., Zhou, D., Liu, L., Li, W., Li, H., Yang, R.: Channel attention based iterative residual learning for depth map super-resolution. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5630–5639 (2020). https://doi.org/10.1109/CVPR42600.2020.00567

  24. Jiang, Z., Yue, H., Lai, Y.-K., Yang, J., Hou, Y., Hou, C.: Deep edge map guided depth super resolution. Signal Process. Image Commun. (2021). https://doi.org/10.1016/j.image.2020.116040

    Article  Google Scholar 

  25. Diebel, J., Thrun, S.: An application of Markov random fields to range sensing. DBLP 18, 291–298 (2005)

    Google Scholar 

  26. Wang, Y., Zhong, F., Peng, Q., Qin, X.: Depth map enhancement based on color and depth consistency. Vis. Comput. 30(10), 1157–1168 (2014)

    Article  Google Scholar 

  27. Choi, O., Jung, S.-W.: A consensus-driven approach for structure and texture aware depth map upsampling. IEEE Trans. Image Process. 23(8), 3321–3335 (2014). https://doi.org/10.1109/TIP.2014.2329766

    Article  MathSciNet  MATH  Google Scholar 

  28. Zuo, Y., Wu, Q., Zhang, J., An, P.: Explicit edge inconsistency evaluation model for color-guided depth map enhancement. IEEE Trans. Circuits Syst. Video Technol. 28(2), 439–453 (2018). https://doi.org/10.1109/TCSVT.2016.2609438

    Article  Google Scholar 

  29. Zuo, Y., Wu, Q., Fang, Y., An, P., Huang, L., Chen, Z.: Multi-scale frequency reconstruction for guided depth map super-resolution via deep residual network. IEEE Trans. Circuits Syst. Video Technol. 30(2), 297–306 (2020). https://doi.org/10.1109/TCSVT.2018.2890271

    Article  Google Scholar 

  30. Yang, J., Ye, X., Li, K., Hou, C., Wang, Y.: Color-guided depth recovery from rgb-d data using an adaptive autoregressive model. IEEE Trans. Image Process. 23(8), 3443–3458 (2015)

  31. Plank, H., Holweg, G., Herndl, T., Druml, N.: High performance time-of-flight and color sensor fusion with image-guided depth super resolution. In: Design, Automation Test in Europe Conference Exhibition (DATE), pp. 1213–1218 (2016)

  32. Chen, R., Gao, W.: Color-guided depth map super-resolution using a dual-branch multi-scale residual network with channel interaction. Sensors (Basel, Switzerland) 20(6), 1560 (2020)

    Article  Google Scholar 

  33. Zuo, Y., Fang, Y., Yang, Y., Shang, X., Wu, Q.: Depth map enhancement by revisiting multi-scale intensity guidance within coarse-to-fine stages. IEEE Trans. Circuits Syst. Video Technol. 30(12), 4676–4687 (2020). https://doi.org/10.1109/TCSVT.2019.2962867

    Article  Google Scholar 

  34. Sil Be Rman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from rgbd images. In: Proceedings of the 12th European conference on Computer Vision, vol. V (2012)

  35. Scharstein, D., Pal, C.: Learning conditional random fields for stereo. In: Cvpr, pp. 1–8 (2007)

  36. Irani, M., Peleg, S.: Motion analysis for image enhancement: resolution, occlusion, and transparency. J Vis. Commun. Image Represent. 4(4), 324–335 (1993)

    Article  Google Scholar 

  37. Zhang, L., Zhang, L., Mou, X., Zhang, D.: Fsim: a feature similarity index for image quality assessment. IEEE Trans. Image Process. 20(8), 2378–2386 (2011). https://doi.org/10.1109/TIP.2011.2109730

Download references

Funding

Not applicable.

Author information

Authors and Affiliations

  1. College of Telecommunications and Information Engineering, Nanjing University of Posts and Telecommunications, Nanjing, 210003, China

    Chen Chen, Ze Lin, Hao She, Yan Huang, Heng Liu, Qin Wang & Shipeng Xie

Authors
  1. Chen Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ze Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hao She
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yan Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Heng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Qin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Shipeng Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

CC wrote the main manuscript text. CCn and ZL were responsible for the experimental design and main program. CC, SX and QW were responsible for the experimental data compilation and analysis. HS assisted in writing the relevant research part of the manuscript. YH and HL assisted in writing the data preprocessing program. All authors reviewed the manuscript.

Corresponding author

Correspondence to Shipeng Xie.

Ethics declarations

Conflict of interest

The authors declared no potential conflicts of interest with respect to the research, authorship, and publication of this article.

Ethical approval

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, C., Lin, Z., She, H. et al. Color image-guided very low-resolution depth image reconstruction. SIViP 17, 2111–2120 (2023). https://doi.org/10.1007/s11760-022-02425-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-022-02425-7

Keywords

Search

Navigation