Skip to main content
SpringerLink
Log in
Book cover

Computer Graphics International Conference

CGI 2021: Advances in Computer Graphics pp 67–79Cite as

Progressive Multi-scale Reconstruction for Guided Depth Map Super-Resolution via Deep Residual Gate Fusion Network

  • Conference paper
  • First Online:

  • 1907 Accesses

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 13002))

Abstract

Depth maps obtained by consumer depth sensors are often accompanied by two challenging problems: low spatial resolution and insufficient quality, which greatly limit the potential applications of depth images. To overcome these shortcomings, some depth map super-resolution (DSR) methods tend to extrapolate a high-resolution depth map from a low-resolution depth map with the additional guidance of the corresponding high-resolution intensity image. However, these methods are still prone to texture copying and boundary discontinuities due to improper guidance. In this paper, we propose a deep residual gate fusion network (DRGFN) for guided depth map super-resolution with progressive multi-scale reconstruction. To alleviate the misalignment between color images and depth maps, DRGFN applies a color-guided gate fusion module to acquire content-adaptive attention for better fusing the color and depth features. To focus on restoring details such as boundaries, DRGFN applies a residual attention module to highlight the different importance of different channels. Furthermore, DRGFN applies a multi-scale fusion reconstruction module to make use of multi-scale information for better image reconstruction. Quantitative and qualitative experiments on several benchmarks fully show that DRGFN obtains the state-of-the-art performance for depth map super-resolution.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Yang, Q., Yang, R., Davis, J., Nistér, D.: Spatial-depth super resolution for range images. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007)

    Google Scholar 

  2. Liu, M.-Y., Tuzel, O., Taguchi, Y.: Joint geodesic upsampling of depth images. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 169–176 (2013)

    Google Scholar 

  3. Ferstl, D., Reinbacher, C., Ranftl, R., Rüther, M., Bischof, H.: Image guided depth upsampling using anisotropic total generalized variation. In: IEEE International Conference on Computer Vision, pp. 993–1000 (2013)

    Google Scholar 

  4. Jiang, Z., Hou, Y., Yue, H., Yang, J., Hou, C.: Depth super-resolution from rgb-d pairs with transform and spatial domain regularization. IEEE Trans. Image Process. 27(5), 2587–2602 (2018)

    Article  MathSciNet  Google Scholar 

  5. Song, X., Dai, Y., Qin, X.: Deep depth super-resolution: learning depth super-resolution using deep convolutional neural network. In: Lai, S.-H., Lepetit, V., Nishino, K., Sato, Y. (eds.) ACCV 2016. LNCS, vol. 10114, pp. 360–376. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-54190-7_22

    Chapter  Google Scholar 

  6. Hui, T.-W., Loy, C.C., Tang, X.: Depth map super-resolution by deep multi-scale guidance. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 353–369. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_22

    Chapter  Google Scholar 

  7. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2015)

    Article  Google Scholar 

  8. Zhang, Z., Wang, Z., Lin, Z., Qi, H.: Image super-resolution by neural texture transfer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7982–7991 (2019)

    Google Scholar 

  9. Yang, F., Yang, H., Fu, J., Lu, H., Guo, B.: Learning texture transformer network for image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5791–5800 (2020)

    Google Scholar 

  10. Hu, Y., Li, J., Huang, Y., Gao, X.: Channel-wise and spatial feature modulation network for single image super-resolution. IEEE Trans. Circ. Syst. Video Technol. 30, 3911–3927 (2019)

    Article  Google Scholar 

  11. Dai, T., Cai, J., Zhang, Y., Xia, S.-T., Zhang, L.: Second-order attention network for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11 065–11 074 (2019)

    Google Scholar 

  12. Shi, W., Jiang, F., Zhao, D.: Single image super-resolution with dilated convolution based multi-scale information learning inception module. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 977–981. IEEE (2017)

    Google Scholar 

  13. Gu, S., Zuo, W., Guo, S., Chen, Y., Chen, C., Zhang, L.: Learning dynamic guidance for depth image enhancement. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017, pp. 712–721 (2017)

    Google Scholar 

  14. Ye, X., Duan, X., Li, H.: Depth super-resolution with deep edge-inference network and edge-guided depth filling. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1398–1402. IEEE (2018)

    Google Scholar 

  15. Wen, Y., Sheng, B., Li, P., Lin, W., Feng, D.D.: Deep color guided coarse-to-fine convolutional network cascade for depth image super-resolution. IEEE Trans. Image Process. 28(2), 994–1006 (2019)

    Article  MathSciNet  Google Scholar 

  16. Lutio, R.D., D’aronco, S., Wegner, J.D., Schindler, K.: Guided super-resolution as pixel-to-pixel transformation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 8829–8837 (2019)

    Google Scholar 

  17. Ye, X., Sun, B., Wang, Z., Yang, J., Xu, R., Li, H., Li, B.: PMBANet: progressive multi-branch aggregation network for scene depth super-resolution. IEEE Trans. Image Process. 29, 7427–7442 (2020)

    Article  Google Scholar 

  18. Wang, F., et al.: Residual attention network for image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3156–3164 (2017)

    Google Scholar 

  19. Zhang, X., Dong, H., Hu, Z., Lai, W.-S., Wang, F., Yang, M.-H.: Gated fusion network for joint image deblurring and super-resolution. In: BMVC (2018)

    Google Scholar 

  20. Glasner, D., Bagon, S., Irani, M.: Super-resolution from a single image. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 349–356. IEEE (2009)

    Google Scholar 

  21. Fu, H., Gong, M., Wang, C., Batmanghelich, K., Tao, D.: Deep ordinal regression network for monocular depth estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2002–2011 (2018)

    Google Scholar 

  22. Guo, J., Ma, S., Guo, S.: MAANet: multi-view aware attention networks for image super-resolution. CoRR, vol. abs/1904.06252 (2019)

    Google Scholar 

  23. Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell. 40(4), 834–848 (2017)

    Article  Google Scholar 

  24. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. CoRR, vol. abs/1807.02758 (2018)

    Google Scholar 

  25. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43

    Chapter  Google Scholar 

  26. Scharstein, D., Szeliski, R.: High-accuracy stereo depth maps using structured light. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 195–202 (2003)

    Google Scholar 

  27. Lu, J., Shi, K., Min, D., Lin, L., Do, M.N.: Cross-based local multipoint filtering. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 430–437 (2012)

    Google Scholar 

  28. Xie, J., Chou, C.-C., Feris, R., Sun, M.-T.: Single depth image super resolution and denoising via coupled dictionary learning with local constraints and shock filtering. In: IEEE International Conference on Multimedia and Expo, pp. 1–6 (2014)

    Google Scholar 

  29. Xie, J., Feris, R.S., Yu, S.-S., Sun, M.-T.: Joint super resolution and denoising from a single depth image. IEEE Trans. Multimedia 17(9), 1525–1537 (2015)

    Article  Google Scholar 

  30. Xie, J., Feris, R.S., Sun, M.-T.: Edge-guided single depth image super resolution. IEEE Trans. Image Process. 25(1), 428–438 (2016)

    Article  MathSciNet  Google Scholar 

  31. Mac Aodha, O., Campbell, N.D.F., Nair, A., Brostow, G.J.: Patch based synthesis for single depth image super-resolution. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7574, pp. 71–84. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33712-3_6

    Chapter  Google Scholar 

  32. Huang, J.-B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5197–5206 (2015)

    Google Scholar 

  33. Wang, Z., Liu, D., Yang, J., Han, W., Huang, T.: Deep networks for image super-resolution with sparse prior. In: IEEE International Conference on Computer Vision, pp. 370–378 (2015)

    Google Scholar 

  34. Riegler, G., Rüther, M., Bischof, H.: ATGV-Net: accurate depth super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 268–284. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_17

    Chapter  Google Scholar 

  35. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: International Conference on Curves and Surfaces, pp. 711–730 (2012)

    Google Scholar 

Download references

Acknowledgement

This work was supported in part by the National Natural Science Foundation of China under Grants 62077037 and 61872241, in part by Shanghai Municipal Science and Technology Major Project under Grant 2021SHZDZX0102, in part by the Science and Technology Commission of Shanghai Municipality under Grants 18410750700 and 17411952600, in part by Shanghai Lin-Gang Area Smart Manufacturing Special Project under Grant ZN2018020202-3, and in part by Project of Shanghai Municipal Health Commission(2018ZHYL0230).

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China

    Yang Wen & Bin Sheng

  2. Shanghai University of Sport, Shanghai, China

    Jihong Wang, Zhen Li & Lijuan Mao

  3. The Hong Kong Polytechnic University, Kowloon, Hong Kong

    Ping Li

  4. Qingdao Research Institute of Beihang University, Qingdao, China

    Xiaoyu Chi

Authors
  1. Yang Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jihong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhen Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bin Sheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ping Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xiaoyu Chi
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Lijuan Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Jihong Wang or Lijuan Mao .

Editor information

Editors and Affiliations

  1. University of Geneva, Carouge, Switzerland

    Nadia Magnenat-Thalmann

  2. University of Minnesota, Minneapolis, MN, USA

    Victoria Interrante

  3. EPFL, Lausanne, Switzerland

    Daniel Thalmann

  4. University of Crete, Heraklion, Crete, Greece

    George Papagiannakis

  5. Shanghai Jiao Tong University, Shanghai, China

    Bin Sheng

  6. University of Sydney, Sydney, NSW, Australia

    Jinman Kim

  7. University of Calgary, Calgary, AB, Canada

    Marina Gavrilova

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wen, Y. et al. (2021). Progressive Multi-scale Reconstruction for Guided Depth Map Super-Resolution via Deep Residual Gate Fusion Network. In: Magnenat-Thalmann, N., et al. Advances in Computer Graphics. CGI 2021. Lecture Notes in Computer Science(), vol 13002. Springer, Cham. https://doi.org/10.1007/978-3-030-89029-2_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-89029-2_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-89028-5

  • Online ISBN: 978-3-030-89029-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation