Abstract
Although color-guided Depth map Super-Resolution (DSR) task has made great progress with the help of deep learning, this task still suffers from some issues: 1) many DSR networks are short of good interpretability; 2) most of the popular DSR methods cannot achieve arbitrary-scale up-sampling for practical applications; 3) dual-modality gaps between color image and depth map may give rise to texture-copying problem. As for these problems, we build a new joint optimization model for two tasks of high-low frequency decomposition and arbitrary-scale DSR. According to alternatively-iterative update formulas of the solution for these two tasks, the proposed model is unfolded as Deep Arbitrary-Scale Unfolding Network (DASU-Net). In the DASU-Net, we propose a Continuous Up-Sampling Fusion (CUSF) module to address two problems of arbitrary-scale feature up-sampling and dual-modality inconsistency during color-depth feature fusion. A large number of experiments have demonstrated that the proposed DASU-Net achieves more significant reconstruction results as compared with several state-of-the-art methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Barron, J.T., Poole, B.: The fast bilateral solver. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision ?C ECCV 2016. pp. 617C632. Springer International Publishing, Cham (2016)
Diebel, J., Thrun, S.: An application of markov random fields to range sensing. In: Proceedings of the 18th International Conference on Neural Information Processing Systems. p. 291298. NIPS05, MIT Press, Cambridge, MA, USA (2005)
He, K., Sun, J., Tang, X.: Guided image filtering. IEEE Transactions on Pattern Analysis and Machine Intelligence 35(6), 1397 C1409 (2013). https://doi.org/10.1109/TPAMI.2012.213
Hui, T.W., Loy, C.C., Tang, X.: Depth map super-resolution by deep multi-scale guidance. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision C ECCV 2016. pp. 353C369. Springer International Publishing, Cham (2016)
Kim, B., Ponce, J., Ham, B.: Deformable kernel networks for joint image filtering. International Journal of Computer Vision 129(2), 579 C 600 (2021), https://doi.org/10.1007/s11263-020-01386-z
Kopf, J., Cohen, M.F., Lischinski, D., Uyttendaele, M.: Joint bilateral upsampling. In: ACM SIGGRAPH 2007 Papers. SIGGRAPH 07, Association for Computing Machinery, New York, NY, USA (2007)
Li, Y., Huang, J.B., Ahuja, N., Yang, M.H.: Joint image filtering with deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence 41(8), 1909C1923 (2019)
Tang, J., Chen, X., Zeng, G.: Joint implicit image function for guided depth super-resolution. In: Proceedings of the 29th ACM International Conference on Multi-media. ACM (oct 2021). DOI: https://doi.org/10.1145/3474085.3475584
Wang, X., Chen, X., Ni, B., Tong, Z., Wang, H.: Learning continuous depth representation via geometric spatial aggregator (2022), https://doi.org/10.48550/arXiv.2212.03499
Zhang, Z., Zheng, H., Hong, R., Xu, M., Yan, S., Wang, M.: Deep color consistent network for low-light image enhancement. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1889C1898 (2022). https://doi.org/10.1109/CVPR52688.2022.00194
Zhao, Z., Zhang, J., Xu, S., Lin, Z., Pfister, H.: Discrete cosine transform network for guided depth map super-resolution. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5687C5697 (2022). DOI: https://doi.org/10.1109/CVPR52688.2022.00561
Zhong, Z., Liu, X., Jiang, J., Zhao, D., Chen, Z., Ji, X.: High-resolution depth maps imaging via attention-based hierarchical multi-modal fusion. IEEE Transactions on Image Processing 31, 648C663 (2022). DOI: https://doi.org/10.1109/TIP.2021.3131041
Zhou, M., Yan, K., Pan, J., Ren, W., Xie, Q., Cao, X.: Memory-augmented Deep Unfolding Network for Guided Image Super-resolution. arXiv e-prints arXiv:2203.04960 (Feb 2022)
Acknowledgements
This work was supported by National Natural Science Foundation of China Youth Science Foundation Project (No.62202323), Fundamental Research Program of Shanxi Province (No.202103021223284), Taiyuan University of Science and Technology Scientific Research Initial Funding (No.20192023, No.20192055), Graduate Education Innovation Project of Taiyuan University of Science and Technology in 2022 (SY2022027), National Natural Science Foundation of China (No.62072325).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Zhang, J., Zhao, L., Zhang, J., Chen, B., Wang, A. (2024). Deep Arbitrary-Scale Unfolding Network for Color-Guided Depth Map Super-Resolution. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14434. Springer, Singapore. https://doi.org/10.1007/978-981-99-8549-4_19
Download citation
DOI: https://doi.org/10.1007/978-981-99-8549-4_19
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8548-7
Online ISBN: 978-981-99-8549-4
eBook Packages: Computer ScienceComputer Science (R0)