Skip to main content
SpringerLink
Log in

LF-MAGNet: Learning Mutual Attention Guidance of Sub-Aperture Images for Light Field Image Super-Resolution

  • Conference paper
  • First Online:
Pattern Recognition and Computer Vision (PRCV 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 13021))

Included in the following conference series:

  • 2200 Accesses

Abstract

Many light field image super-resolution networks are proposed to directly aggregate the features of different low-resolution sub-aperture images (SAIs) to reconstruct high-resolution sub-aperture images. However, most of them ignore aligning different SAI’s features before aggregation, which will generate sub-optimal light field image super-resolution results. To handle this limitation, we design a mutual attention mechanism to align the SAI’s features and propose a Light Field Mutual Attention Guidance Network (LF-MAGNet) constructed by multiple Mutual Attention Guidance blocks (MAGs) in a cascade manner. MAG achieves the mutual attention mechanism between center SAI and any surrounding SAI with two modules: the center attention guidance module (CAG) and the surrounding attention guidance module (SAG). Specifically, CAG first aligns the center-SAI features and any surrounding SAI features with the attention mechanism and then guides the surrounding SAI feature to learn from the center-SAI features, generating refined-surrounding SAI features. SAG aligns the refined-surrounding SAI feature and the original surrounding SAI feature and guides the refined surrounding SAI feature to learn from the original surrounding SAI features, generating the final outputs of MAG. With the help of MAG, LF-MAGNet can efficiently utilize different SAI features and generate high-quality light field image super-resolution results. Experiments are performed on commonly-used light field image super-resolution benchmarks. Qualitative and quantitative results prove the effectiveness of our LF-MAGNet.

This is a student paper.

This work is supported by the National Natural Science Foundation of China (No. 61273273), by the National Key Research and Development Plan (No. 2017YFC0112001), and by China Central Television (JG2018-0247).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Alain, M., Smolic, A.: Light field super-resolution via lfbm5d sparse coding. In: ICIP, pp. 2501–2505 (2018)

    Google Scholar 

  2. Dai, T., Cai, J., Zhang, Y., Xia, S.T., Zhang, L.: Second-order attention network for single image super-resolution. In: CVPR, pp. 11065–11074 (2019)

    Google Scholar 

  3. Honauer, K., Johannsen, O., Kondermann, D., Goldluecke, B.: A dataset and evaluation methodology for depth estimation on 4D light fields. In: ACCV, pp. 19–34 (2016)

    Google Scholar 

  4. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: CVPR, pp. 7132–7141 (2018)

    Google Scholar 

  5. Huang, F.C., et al.: The light field stereoscope-immersive computer graphics via factored near-eye field displays with focus cues. SIGGRAPH (2015)

    Google Scholar 

  6. Jin, J., Hou, J., Chen, J., Kwong, S.: Light field spatial super-resolution via deep combinatorial geometry embedding and structural consistency regularization. In: CVPR, pp. 2260–2269 (2020)

    Google Scholar 

  7. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: CVPR, pp. 1646–1654 (2016)

    Google Scholar 

  8. Le Pendu, M., Jiang, X., Guillemot, C.: Light field inpainting propagation via low rank matrix completion. IEEE TIP 27(4), 1981–1993 (2018)

    MathSciNet  MATH  Google Scholar 

  9. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: CVPRW, pp. 136–144 (2017)

    Google Scholar 

  10. Liu, J., Tang, J., Wu, G.: Residual feature distillation network for lightweight image super-resolution. arXiv preprint arXiv:2009.11551 (2020)

  11. Lu, X., Wang, W., Danelljan, M., Zhou, T., Shen, J., Van Gool, L.: Video object segmentation with episodic graph memory networks. In: ECCV, pp. 661–679 (2020)

    Google Scholar 

  12. Park, J., Woo, S., Lee, J.Y., Kweon, I.S.: Bam: Bottleneck attention module. arXiv preprint arXiv:1807.06514 (2018)

  13. Rerabek, M., Ebrahimi, T.: New light field image dataset. In: 8th International Conference on Quality of Multimedia Experience. No. CONF (2016)

    Google Scholar 

  14. Rossi, M., Frossard, P.: Geometry-consistent light field super-resolution via graph-based regularization. IEEE TIP 27(9), 4207–4218 (2018)

    MathSciNet  MATH  Google Scholar 

  15. Sheng, H., Zhang, S., Cao, X., Fang, Y., Xiong, Z.: Geometric occlusion analysis in depth estimation using integral guided filter for light-field image. IEEE TIP 26(12), 5758–5771 (2017)

    MathSciNet  MATH  Google Scholar 

  16. Shi, J., Jiang, X., Guillemot, C.: A framework for learning depth from a flexible subset of dense and sparse light field views. IEEE TIP 28(12), 5867–5880 (2019)

    MathSciNet  MATH  Google Scholar 

  17. Vaish, V., Adams, A.: The (new) stanford light field archive. Computer Graphics Laboratory, Stanford University 6(7) (2008)

    Google Scholar 

  18. Wang, B., Yang, L., Zhao, Y.: POLO: learning explicit cross-modality fusion for temporal action localization. IEEE Signal Process. Lett. 28, 503–507 (2021)

    Article  Google Scholar 

  19. Wang, W., Zhou, T., Qi, S., Shen, J., Zhu, S.C.: Hierarchical human semantic parsing with comprehensive part-relation modeling. IEEE TPAMI (2021)

    Google Scholar 

  20. Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: CVPR, pp. 7794–7803 (2018)

    Google Scholar 

  21. Wang, Y., Wang, L., Yang, J., An, W., Yu, J., Guo, Y.: Spatial-angular interaction for light field image super-resolution. In: ECCV, pp. 290–308 (2020)

    Google Scholar 

  22. Wang, Y., Yang, J., Wang, L., Ying, X., Wu, T., An, W., Guo, Y.: Light field image super-resolution using deformable convolution. IEEE TIP 30, 1057–1071 (2020)

    Google Scholar 

  23. Wang, Y., Liu, F., Zhang, K., Hou, G., Sun, Z., Tan, T.: LFNet: a novel bidirectional recurrent convolutional neural network for light-field image super-resolution. IEEE TIP 27(9), 4274–4286 (2018)

    MathSciNet  Google Scholar 

  24. Wanner, S., Meister, S., Goldluecke, B.: Datasets and benchmarks for densely sampled 4D light fields. Vis. Modell. Visual. 13, 225–226 (2013)

    Google Scholar 

  25. Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: CBAM: convolutional block attention module. In: ECCV, pp. 3–19 (2018)

    Google Scholar 

  26. Yang, L., Peng, H., Zhang, D., Fu, J., Han, J.: Revisiting anchor mechanisms for temporal action localization. IEEE TIP 29, 8535–8548 (2020)

    Google Scholar 

  27. Yeung, H.W.F., Hou, J., Chen, X., Chen, J., Chen, Z., Chung, Y.Y.: Light field spatial super-resolution using deep efficient spatial-angular separable convolution. IEEE TIP 28(5), 2319–2330 (2018)

    MathSciNet  Google Scholar 

  28. Yoon, Y., Jeon, H.G., Yoo, D., Lee, J.Y., Kweon, I.S.: Light-field image super-resolution using convolutional neural network. IEEE Signal Process. Lett. 24(6), 848–852 (2017)

    Article  Google Scholar 

  29. Yu, J.: A light-field journey to virtual reality. IEEE Multimedia 24(2), 104–112 (2017)

    Article  Google Scholar 

  30. Yuan, Y., Cao, Z., Su, L.: Light-field image superresolution using a combined deep CNN based on EPI. IEEE Signal Process. Lett. 25(9), 1359–1363 (2018)

    Article  Google Scholar 

  31. Zhang, S., Lin, Y., Sheng, H.: Residual networks for light field image super-resolution. In: CVPR, pp. 11046–11055 (2019)

    Google Scholar 

  32. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: ECCV, pp. 286–301 (2018)

    Google Scholar 

  33. Zhou, T., Li, J., Wang, S., Tao, R., Shen, J.: MATNet: motion-attentive transition network for zero-shot video object segmentation. IEEE TIP 29, 8326–8338 (2020)

    Google Scholar 

  34. Zhou, T., Qi, S., Wang, W., Shen, J., Zhu, S.C.: Cascaded parsing of human-object interaction recognition. IEEE TPAMI (2021)

    Google Scholar 

  35. Zhou, T., Wang, S., Zhou, Y., Yao, Y., Li, J., Shao, L.: Motion-attentive transition for zero-shot video object segmentation. In: AAAI, pp. 13066–13073 (2020)

    Google Scholar 

  36. Zhou, T., Wang, W., Liu, S., Yang, Y., Van Gool, L.: Differentiable multi-granularity human representation learning for instance-aware human semantic parsing. In: CVPR, pp. 1622–1631 (2021)

    Google Scholar 

  37. Zhu, H., Wang, Q., Yu, J.: Occlusion-model guided antiocclusion depth estimation in light field. IEEE J. Selected Top. Signal Process. 11(7), 965–978 (2017)

    Article  Google Scholar 

  38. Zhu, H., Zhang, Q., Wang, Q.: 4D light field superpixel and segmentation. In: CVPR, pp. 6384–6392 (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Beijing Laboratory of Intelligent Information Technology, School of Computer Science and Technology, Beijing Institute of Technology, Beijing, 100081, China

    Zijian Wang, Yao Lu, Haowei Lu & Shunzhou Wang

  2. School of Marine Electrical Engineering, Dalian Maritime University, Dalian, 116026, China

    Yani Zhang

  3. School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University, Xi’an, 710072, China

    Binglu Wang

Authors
  1. Zijian Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yao Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yani Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Haowei Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shunzhou Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Binglu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yao Lu .

Editor information

Editors and Affiliations

  1. University of Science and Technology Beijing, Beijing, China

    Huimin Ma

  2. Chinese Academy of Sciences, Beijing, China

    Liang Wang

  3. Tsinghua University, Beijing, China

    Changshui Zhang

  4. Zhejiang University, Hangzhou, China

    Fei Wu

  5. Chinese Academy of Sciences, Beijing, China

    Tieniu Tan

  6. Hunan University, Changsha, China

    Yaonan Wang

  7. Sun Yat-Sen University, Guangzhou, Guangdong, China

    Jianhuang Lai

  8. Beijing Jiaotong University, Beijing, China

    Yao Zhao

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, Z., Lu, Y., Zhang, Y., Lu, H., Wang, S., Wang, B. (2021). LF-MAGNet: Learning Mutual Attention Guidance of Sub-Aperture Images for Light Field Image Super-Resolution. In: Ma, H., et al. Pattern Recognition and Computer Vision. PRCV 2021. Lecture Notes in Computer Science(), vol 13021. Springer, Cham. https://doi.org/10.1007/978-3-030-88010-1_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-88010-1_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-88009-5

  • Online ISBN: 978-3-030-88010-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation