skip to main content
10.1145/3394171.3413874acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Depth Super-Resolution via Deep Controllable Slicing Network

Published: 12 October 2020 Publication History

Abstract

Due to the imaging limitation of depth sensors, high-resolution (HR) depth maps are often difficult to be acquired directly, thus effective depth super-resolution (DSR) algorithms are needed to generate HR output from its low-resolution (LR) counterpart. Previous methods treat all depth regions equally without considering different extents of degradation at region-level, and regard DSR under different scales as independent tasks without considering the modeling of different scales, which impede further performance improvement and practical use of DSR. To alleviate these problems, we propose a deep controllable slicing network from a novel perspective. Specifically, our model is to learn a set of slicing branches in a divide-and-conquer manner, parameterized by a distance-aware weighting scheme to adaptively aggregate different depths in an ensemble. Each branch that specifies a depth slice (e.g., the region in some depth range) tends to yield accurate depth recovery. Meanwhile, a scale-controllable module that extracts depth features under different scales is proposed and inserted into the front of slicing network, and enables finely-grained control of the depth restoration results of slicing network with a scale hyper-parameter. Extensive experiments on synthetic and real-world benchmark datasets demonstrate that our method achieves superior performance.

Supplementary Material

MP4 File (3394171.3413874.mp4)
In this paper, we propose a deep controllable slicing network from a novel perspective. Specifically, our model is to learn a set of slicing branches in a divide-and-conquer manner, parameterized by a distance-aware weighting scheme to adaptively aggregate different depths in an ensemble. Each branch that specifies a depth slice (e.g., the region in some depth range) tends to yield accurate depth recovery. Meanwhile, a scale-controllable module that extracts depth features under different scales is proposed and inserted into the front of slicing network, and enables finely-grained control of the depth restoration results of slicing network with a scale hyper-parameter.

References

[1]
Badour Albahar and Jia-Bin Huang. 2019. Guided Image-to-Image Translation With Bi-Directional Feature Transformation. In IEEE ICCV. IEEE, 9015--9024.
[2]
Simon Baker, Daniel Scharstein, J. P. Lewis, Stefan Roth, Michael J. Black, and Richard Szeliski. 2011. A Database and Evaluation Methodology for Optical Flow. International Journal of Computer Vision, Vol. 92, 1 (2011), 1--31.
[3]
Riccardo de Lutio, Stefano D'Aronco, Jan Dirk Wegner, and Konrad Schindler. 2019. Guided Super-Resolution As Pixel-to-Pixel Transformation. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019. IEEE, 8828--8836.
[4]
Thomas G. Dietterich. 2000. Ensemble Methods in Machine Learning. In Multiple Classifier Systems, First International Workshop, MCS, Cagliari, Italy, Proceedings (Lecture Notes in Computer Science), Josef Kittler and Fabio Roli (Eds.), Vol. 1857. Springer, 1--15.
[5]
T. Edeler, K. Ohliger, S. Hussmann, and A. Mertins. 2010. Time-of-flight depth image denoising using prior noise information. In IEEE 10th International Conference on Signal Processing Proceedings. 119--122.
[6]
David Ferstl, Christian Reinbacher, Rene Ranftl, Matthias Rüther, and Horst Bischof. 2013a. Image Guided Depth Upsampling using Anisotropic Total Generalized Variation. In Proc. ICCV.
[7]
David Ferstl, Christian Reinbacher, René Ranftl, Matthias Rü ther, and Horst Bischof. 2013b. Image Guided Depth Upsampling Using Anisotropic Total Generalized Variation. In IEEE International Conference on Computer Vision, ICCV, Sydney, Australia. IEEE Computer Society, 993--1000.
[8]
Zhihang Fu, Zhongming Jin, Guo-Jun Qi, Chen Shen, Rongxin Jiang, Yaowu Chen, and Xian-Sheng Hua. 2018. Previewer for Multi-Scale Object Detector. In 2018 ACM Multimedia Conference on Multimedia Conference, MM, Seoul, Republic of Korea, October 22--26. 265--273.
[9]
Shangqi Gao and Xiahai Zhuang. 2019. Multi-Scale Deep Neural Networks for Real Image Super-Resolution. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops, Long Beach, CA, USA. Computer Vision Foundation / IEEE, 0.
[10]
Xu Gao and Tingting Jiang. [n.d.]. OSMO: Online Specific Models for Occlusion in Multiple Object Tracking under Surveillance Scene. In ACM Multimedia Conference on Multimedia Conference, MM 2018, Seoul, Republic of Korea. ACM, 201--210.
[11]
Zan Gao, Li-Shuai Gao, Hua Zhang, Zhiyong Cheng, and Richang Hong. [n.d.]. Deep Spatial Pyramid Features Collaborative Reconstruction for Partial Person ReID. In ACM International Conference on Multimedia, MM 2019, Nice, France. ACM, 1879--1887.
[12]
Shuhang Gu, Wangmeng Zuo, Shi Guo, Yunjin Chen, Chongyu Chen, and Lei Zhang. 2017. Learning Dynamic Guidance for Depth Image Enhancement. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Honolulu, HI, USA. 712--721.
[13]
Chunle Guo, Chongyi Li, Jichang Guo, Runmin Cong, Huazhu Fu, and Ping Han. 2019. Hierarchical Features Driven Residual Learning for Depth Map Super-Resolution. IEEE Trans. Image Processing, Vol. 28, 5 (2019), 2545--2557.
[14]
Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. 2018. Deep Back-Projection Networks for Super-Resolution. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Salt Lake City, UT, USA. 1664--1673.
[15]
Heiko Hirschmü ller and Daniel Scharstein. 2007. Evaluation of Cost Functions for Stereo Matching. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition CVPR, Minneapolis, Minnesota, USA.
[16]
Xuecai Hu, Haoyuan Mu, Xiangyu Zhang, Zilei Wang, Tieniu Tan, and Jian Sun. 2019. Meta-SR: A Magnification-Arbitrary Network for Super-Resolution. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Long Beach, CA, USA. Computer Vision Foundation / IEEE, 1575--1584.
[17]
Tak-Wai Hui, Chen Change Loy, and Xiaoou Tang. 2016. Depth Map Super-Resolution by Deep Multi-Scale Guidance. In Computer Vision - ECCV - 14th European Conference, Amsterdam, The Netherlands, Proceedings, Part III. 353--369.
[18]
Sunghoon Im, Hyowon Ha, Gyeongmin Choe, Hae-Gon Jeon, Kyungdon Joo, and In So Kweon. 2019. Accurate 3D Reconstruction from Small Motion Clip for Rolling Shutter Cameras. IEEE Trans. Pattern Anal. Mach. Intell., Vol. 41, 4 (2019), 775--787.
[19]
Younghyun Jo, Seoung Wug Oh, Jaeyeon Kang, and Seon Joo Kim. 2018. Deep Video Super-Resolution Network Using Dynamic Upsampling Filters Without Explicit Motion Compensation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Salt Lake City, UT, USA. 3224--3232.
[20]
Beomjun Kim, Jean Ponce, and Bumsub Ham. 2019. Deformable kernel networks for guided depth map upsampling. CoRR, Vol. abs/1903.11286 (2019). arxiv: 1903.11286 http://arxiv.org/abs/1903.11286
[21]
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Accurate Image Super-Resolution Using Very Deep Convolutional Networks. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Las Vegas, NV, USA. 1646--1654.
[22]
Johannes Kopf, Michael F. Cohen, Dani Lischinski, and Matthew Uyttendaele. 2007. Joint bilateral upsampling. ACM Trans. Graph., Vol. 26, 3 (2007), 96.
[23]
Chao Li, Dongliang He, Xiao Liu, Yukang Ding, and Shilei Wen. 2019 a. Adapting Image Super-Resolution State-Of-The-Arts and Learning Multi-Model Ensemble for Video Super-Resolution. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops, Long Beach, CA, USA.
[24]
Yijun Li, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. 2016a. Deep Joint Image Filtering. In Computer Vision - ECCV - 14th European Conference, Amsterdam, The Netherlands, Proceedings, Part IV. 154--169.
[25]
Yijun Li, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. 2019 b. Joint Image Filtering with Deep Convolutional Networks. IEEE Trans. Pattern Anal. Mach. Intell., Vol. 41, 8 (2019), 1909--1923.
[26]
Yu Li, Dongbo Min, Minh N. Do, and Jiangbo Lu. 2016b. Fast Guided Global Interpolation for Depth and Motion. In Computer Vision - ECCV - 14th European Conference, Amsterdam, The Netherlands, Proceedings, Part III. 717--733.
[27]
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017. Enhanced Deep Residual Networks for Single Image Super-Resolution. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops, Honolulu, HI, USA. 1132--1140.
[28]
Jinshan Pan, Jiangxin Dong, Jimmy S. J. Ren, Liang Lin, Jinhui Tang, and Ming-Hsuan Yang. 2019. Spatially Variant Linear Representation Models for Joint Filtering. In IEEE CVPR. 1702--1711.
[29]
Jaesik Park, Hyeongwoo Kim, Yu-Wing Tai, Michael S. Brown, and In-So Kweon. 2011. High quality depth map upsampling for 3D-TOF cameras. In IEEE International Conference on Computer Vision, ICCV, Barcelona, Spain. 1623--1630.
[30]
Guo-Jun Qi. 2016. Hierarchically Gated Deep Networks for Semantic Segmentation. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Las Vegas, NV, USA, June 27--30. 2267--2275.
[31]
Jinghui Qin, Ziwei Xie, Yukai Shi, and Wushao Wen. 2019. Difficulty-Aware Image Super Resolution via Deep Adaptive Dual-Network. In IEEE International Conference on Multimedia and Expo, ICME, Shanghai, China. IEEE, 586--591.
[32]
Gernot Riegler, Matthias Rü ther, and Horst Bischof. 2016. ATGV-Net: Accurate Depth Super-Resolution. In Computer Vision - ECCV - 14th European Conference, Amsterdam, The Netherlands, Proceedings, Part III. 268--284.
[33]
Daniel Scharstein, Heiko Hirschmü ller, York Kitajima, Greg Krathwohl, Nera Nesic, Xi Wang, and Porter Westling. 2014. High-Resolution Stereo Datasets with Subpixel-Accurate Ground Truth. In Pattern Recognition - 36th German Conference, GCPR, Mü nster, Germany, Proceedings. 31--42.
[34]
Daniel Scharstein and Chris Pal. 2007. Learning Conditional Random Fields for Stereo. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition CVPR, Minneapolis, Minnesota, USA.
[35]
Daniel Scharstein and Richard Szeliski. 2003. High-Accuracy Stereo Depth Maps Using Structured Light. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition CVPR, Madison, WI, USA. 195--202.
[36]
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. 2012. Indoor Segmentation and Support Inference from RGBD Images. In ECCV.
[37]
Hang Su, Varun Jampani, Deqing Sun, Orazio Gallo, Erik G. Learned-Miller, and Jan Kautz. 2019. Pixel-Adaptive Convolutional Neural Networks. In IEEE CVPR. IEEE, 11166--11175.
[38]
Xintao Wang, Ke Yu, Chao Dong, Xiaoou Tang, and Chen Change Loy. 2019. Deep Network Interpolation for Continuous Imagery Effect Transition. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR.
[39]
Zhihui Wang, Xinchen Ye, Baoli Sun, Jingyu Yang, Rui Xu, and Haojie Li. 2020. Depth upsampling based on deep edge-aware learning. Pattern Recognition, Vol. 103 (2020), 107274.
[40]
Yang Wen, Bin Sheng, Ping Li, Weiyao Lin, and David Dagan Feng. 2019. Deep Color Guided Coarse-to-Fine Convolutional Network Cascade for Depth Image Super-Resolution. IEEE Trans. Image Processing, Vol. 28, 2 (2019), 994--1006.
[41]
Xinchen Ye, Xiangyue Duan, and Haojie Li. 2018. Depth Super-Resolution with Deep Edge-Inference Network and Edge-Guided Depth Filling. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, Calgary, AB, Canada. 1398--1402.
[42]
Le Zhang, Zenglin Shi, Ming-Ming Cheng, Yun Liu, and Zeng Zeng. 2019. Nonlinear Regression via Deep Negative Correlation Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. PP, 99 (2019), 1--1.
[43]
Kecheng Zheng, Zheng-Jun Zha, Yang Cao, Xuejin Chen, and Feng Wu. [n.d.]. LA-Net: Layout-Aware Dense Network for Monocular Depth Estimation. In ACM Multimedia Conference on Multimedia Conference, MM 2018, Seoul, Republic of Korea. ACM, 1381--1388.

Cited By

View all
  • (2024)IGAF: Incremental Guided Attention Fusion for Depth Super-ResolutionSensors10.3390/s2501002425:1(24)Online publication date: 24-Dec-2024
  • (2024)Deep Attentional Guided Image FilteringIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2023.325347235:9(12236-12250)Online publication date: Sep-2024
  • (2024)Learning Hierarchical Color Guidance for Depth Map Super-ResolutionIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.338116873(1-13)Online publication date: 2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
MM '20: Proceedings of the 28th ACM International Conference on Multimedia
October 2020
4889 pages
ISBN:9781450379885
DOI:10.1145/3394171
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 October 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. controllable
  2. distance-aware
  3. scene depth
  4. slicing
  5. super-resolution

Qualifiers

  • Research-article

Conference

MM '20
Sponsor:

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)32
  • Downloads (Last 6 weeks)1
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)IGAF: Incremental Guided Attention Fusion for Depth Super-ResolutionSensors10.3390/s2501002425:1(24)Online publication date: 24-Dec-2024
  • (2024)Deep Attentional Guided Image FilteringIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2023.325347235:9(12236-12250)Online publication date: Sep-2024
  • (2024)Learning Hierarchical Color Guidance for Depth Map Super-ResolutionIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.338116873(1-13)Online publication date: 2024
  • (2024)Latent Edge Guided Depth Super-Resolution Using Attention-Based Hierarchical Multi-Modal FusionIEEE Access10.1109/ACCESS.2024.343550412(114512-114526)Online publication date: 2024
  • (2024)DSRNet: Depth Super-Resolution Network guided by blurry depth and clear intensity edgesSignal Processing: Image Communication10.1016/j.image.2023.117064121(117064)Online publication date: Feb-2024
  • (2023)Depth Map Super-Resolution Reconstruction Based on Multi-Channel Progressive Attention Fusion NetworkApplied Sciences10.3390/app1314827013:14(8270)Online publication date: 17-Jul-2023
  • (2023)Saving Bits Using Multi-Sensor CollaborationIEEE Access10.1109/ACCESS.2023.323491711(4869-4878)Online publication date: 2023
  • (2022)WAFP-Net: Weighted Attention Fusion Based Progressive Residual Learning for Depth Map Super-ResolutionIEEE Transactions on Multimedia10.1109/TMM.2021.311828224(4113-4127)Online publication date: 2022
  • (2021)Joint Implicit Image Function for Guided Depth Super-ResolutionProceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3475584(4390-4399)Online publication date: 17-Oct-2021
  • (2021)PRNetProceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3475517(3537-3545)Online publication date: 17-Oct-2021

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media