skip to main content
10.1145/3551626.3564940acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Deep Weighted Guided Upsampling Network for Depth of Field Image Upsampling

Published:13 December 2022Publication History

ABSTRACT

Depth-of-field (DoF) rendering is an important technique in computational photography that simulates the human visual attention system. Existing DoF rendering methods usually suffer from a high computational cost. The task of DoF rendering can be accelerated by guided upsampling methods. However, the state-of-the-art guided upsampling methods fail to distinguish the focus and defocus areas, resulting in unsatisfying DoF effects. In this paper, we propose a novel deep weighted guided upsampling network (DWGUN) based on a encoder and decoder framework to jointly upsample the low-resolution DoF image under the guidance of the corresponding high-resolution all-in-focus image. Due to the intuitive weight design, the traditional weighted image upsampling is not tailored to DoF image upsampling. We propose a deep refocus-defocus edge-aware module (DREAM) to learn the spatially-varying weights and embed them in the deep weighted guided upsampling block (DWGUB). We have conducted comprehensive experiments to evaluate the proposed method. Rigorous ablation studies are also conducted to validate the rationality of the proposed components.

References

  1. Jonathan T. Barron, Andrew Adams, YiChang Shih, and Carlos Hernández. 2015. Fast bilateral-space stereo for synthetic defocus. In IEEE Conference on Computer Vision and Pattern Recognition. 4466--4474.Google ScholarGoogle ScholarCross RefCross Ref
  2. Benjamin Busam, Matthieu Hog, Steven McDonagh, and Gregory G. Slabaugh. 2019. SteReFo: Efficient Image Refocusing with Stereo Vision. In International Conference on Computer Vision Workshop. 3295--3304.Google ScholarGoogle Scholar
  3. Jiawen Chen, Andrew Adams, Neal Wadhwa, and Samuel W. Hasinoff. 2016. Bilateral guided upsampling. ACM Trans. Graph. 35, 6 (2016), 203.Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Alireza Esmaeilzehi, M. Omair Ahmad, and M. N. S. Swamy. 2021. SRNSSI: A Deep Light-Weight Network for Single Image Super Resolution Using Spatial and Spectral Information. IEEE Trans. Computational Imaging 7 (2021), 409--421.Google ScholarGoogle ScholarCross RefCross Ref
  5. Rahul Garg, Neal Wadhwa, Sameer Ansari, and Jonathan T. Barron. 2019. Learning Single Camera Depth Estimation Using Dual-Pixels. In International Conference on Computer Vision. 7627--7636.Google ScholarGoogle Scholar
  6. Yong Guo, Jian Chen, Jingdong Wang, Qi Chen, Jiezhang Cao, Zeshuai Deng, Yanwu Xu, and Mingkui Tan. 2020. Closed-Loop Matters: Dual Regression Networks for Single Image Super-Resolution. In IEEE Conference on Computer Vision and Pattern Recognition. 5406--5415.Google ScholarGoogle Scholar
  7. Kaiming He, Jian Sun, and Xiaoou Tang. 2013. Guided Image Filtering. IEEE Trans. Pattern Anal. Mach. Intell. 35, 6 (2013), 1397--1409.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-Excitation Networks. In IEEE Conference on Computer Vision and Pattern Recognition. 7132--7141.Google ScholarGoogle Scholar
  9. Andrey Ignatov, Jagruti Patel, and Radu Timofte. 2020. Rendering Natural Camera Bokeh Effect with Deep Learning. In IEEE Conference on Computer Vision and Pattern Recognition. 1676--1686.Google ScholarGoogle Scholar
  10. Jing Jin, Junhui Hou, Jie Chen, Sam Kwong, and Jingyi Yu. 2020. Light Field Super-resolution via Attention-Guided Fusion of Hybrid Lenses. In International Conference on Multimedia. 193--201.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Johannes Kopf, Michael F. Cohen, Dani Lischinski, and Matthew Uyttendaele. 2007. Joint bilateral upsampling. ACM Trans. Graph. 26, 3 (2007), 96.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Fei Kou, Weihai Chen, Changyun Wen, and Zhengguo Li. 2015. Gradient Domain Guided Image Filtering. IEEE Trans. Image Process. 24, 11 (2015), 4528--4539.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Zhen Li, Jinglei Yang, Zheng Liu, Xiaomin Yang, Gwanggil Jeon, and Wei Wu. 2019. Feedback Network for Image Super-Resolution. In IEEE Conference on Computer Vision and Pattern Recognition. 3867--3876.Google ScholarGoogle Scholar
  14. Zhengguo Li, Jinghong Zheng, Zijian Zhu, Wei Yao, and Shiqian Wu. 2015. Weighted Guided Image Filtering. IEEE Trans. Image Process. 24, 1 (2015), 120--129.Google ScholarGoogle ScholarCross RefCross Ref
  15. Shunta Maeda. 2020. Unpaired Image Super-Resolution Using Pseudo-Supervision. In IEEE Conference on Computer Vision and Pattern Recognition.Google ScholarGoogle Scholar
  16. Pingfan Song, Xin Deng, João F. C. Mota, Nikos Deligiannis, Pier Luigi Dragotti, and Miguel R. D. Rodrigues. 2020. Multimodal Image Super-Resolution via Joint Sparse Representations Induced by Coupled Dictionaries. IEEE Trans. Computational Imaging 6 (2020), 57--72.Google ScholarGoogle ScholarCross RefCross Ref
  17. Carlo Tomasi and Roberto Manduchi. 1998. Bilateral Filtering for Gray and Color Images. In International Conference on Computer Vision. 839--846.Google ScholarGoogle Scholar
  18. Neal Wadhwa, Rahul Garg, David E. Jacobs, Bryan E. Feldman, Nori Kanazawa, Robert Carroll, Yair Movshovitz-Attias, Jonathan T. Barron, Yael Pritch, and Marc Levoy. 2018. Synthetic depth-of-field with a single-camera mobile phone. ACM Trans. Graph. 37, 4 (2018), 64.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Lijun Wang, Xiaohui Shen, Jianming Zhang, Oliver Wang, Zhe L. Lin, Chih-Yao Hsieh, Sarah Kong, and Huchuan Lu. 2018. DeepLens: shallow depth of field from a single image. ACM Trans. Graph. 37, 6 (2018), 245:1--245:11.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Yuwang Wang, Yebin Liu, Wolfgang Heidrich, and Qionghai Dai. 2017. The Light Field Attachment: Turning a DSLR into a Light Field Camera Using a Low Budget Camera Ring. IEEE Trans. Vis. Comput. Graph. 23, 10 (2017), 2357--2364.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Huikai Wu, Shuai Zheng, Junge Zhang, and Kaiqi Huang. 2018. Fast End-to-End Trainable Guided Filter. In IEEE Conference on Computer Vision and Pattern Recognition. 1838--1847.Google ScholarGoogle Scholar
  22. Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo. 2020. Learning Texture Transformer Network for Image Super-Resolution. In IEEE Conference on Computer Vision and Pattern Recognition. 5790--5799.Google ScholarGoogle Scholar
  23. Jaejun Yoo, Namhyuk Ahn, and Kyung-Ah Sohn. 2020. Rethinking Data Augmentation for Image Super-resolution: A Comprehensive Analysis and a New Strategy. In IEEE Conference on Computer Vision and Pattern Recognition. 8372--8381.Google ScholarGoogle ScholarCross RefCross Ref
  24. Zhifei Zhang, Zhaowen Wang, Zhe L. Lin, and Hairong Qi. 2019. Image Super-Resolution by Neural Texture Transfer. In IEEE Conference on Computer Vision and Pattern Recognition. 7982--7991.Google ScholarGoogle ScholarCross RefCross Ref
  25. Mandan Zhao, Gaochang Wu, Yipeng Li, Xiangyang Hao, Lu Fang, and Yebin Liu. 2018. Cross-Scale Reference-Based Light Field Super-Resolution. IEEE Trans. Computational Imaging 4, 3 (2018), 406--418.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Deep Weighted Guided Upsampling Network for Depth of Field Image Upsampling

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in
      • Published in

        cover image ACM Conferences
        MMAsia '22: Proceedings of the 4th ACM International Conference on Multimedia in Asia
        December 2022
        296 pages
        ISBN:9781450394789
        DOI:10.1145/3551626

        Copyright © 2022 ACM

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 13 December 2022

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article

        Acceptance Rates

        Overall Acceptance Rate59of204submissions,29%

        Upcoming Conference

        MM '24
        MM '24: The 32nd ACM International Conference on Multimedia
        October 28 - November 1, 2024
        Melbourne , VIC , Australia
      • Article Metrics

        • Downloads (Last 12 months)49
        • Downloads (Last 6 weeks)3

        Other Metrics

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader