Skip to main content
SpringerLink
Log in

Light Field Super-Resolution Based on Spatial and Angular Attention

  • Conference paper
  • First Online:
Wireless Algorithms, Systems, and Applications (WASA 2021)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 12937))

Abstract

Light field (LF) images captured by LF cameras can store the intensity and direction information of light rays in the scene, which have advantages in many computer vision tasks, such as 3D reconstruction, target tracking and so on. But there is a trade-off between the spatial and angular resolution of LF images due to the fixed resolution of sensor in LF cameras. So LF image super-resolution (SR) is widely explored. Most of the existing methods do not consider the different degree of importance of spatial and angular information provided by other views in LF. So we propose a LF spatial-angular attention module (LFSAA) to adjust the weights of spatial and angular information in spatial and angular domain respectively. Based on this module, a LF image SR network is designed to super-resolve all views in LF simultaneously. And we further combine the LF image SR network with single image SR network to improve the ability to explore spatial information of a single image in LF. Experiments on both synthetic and real-world LF datasets have demonstrated the performance of our method.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Wu, G., Masia, B., Jarabo, A., Zhang, Y., Wang, L., et al.: Light field image processing: an overview. IEEE J. Sel. Top. Signal Process. 11(7), 926–954 (2017)

    Article  Google Scholar 

  2. Bishop, T.E., Favaro, P.: The light field camera: extended depth of field, aliasing, and superresolution. IEEE Trans. Pattern Anal. Mach. Intell. 34(5), 972–986 (2011)

    Article  Google Scholar 

  3. Mitra, K., Veeraraghavan, A.: Light field denoising, light field superresolution and stereo camera based refocussing using a GMM light field patch prior. In: CVPRW, pp. 22–28. IEEE (2012)

    Google Scholar 

  4. Wanner, S., Goldluecke, B.: Variational light field analysis for disparity estimation and super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. 36(3), 606–619 (2013)

    Article  Google Scholar 

  5. Boominathan, V., Mitra, K., Veeraraghavan, A.: Improving resolution and depth-of-field of light field cameras using a hybrid imaging system. In: 2014 IEEE International Conference on Computational Photography, pp. 1–10. IEEE (2014)

    Google Scholar 

  6. Yoon, Y., Jeon, H.G., Yoo, D., Lee, J.Y., So Kweon, I.: Learning a deep convolutional network for light-field image super-resolution. In: ICCVW, pp. 24–32. IEEE (2015)

    Google Scholar 

  7. Yoon, Y., Jeon, H.G., Yoo, D., Lee, J.Y., Kweon, I.S.: Light-field image super-resolution using convolutional neural network. IEEE Signal Process. Lett. 24(6), 848–852 (2017)

    Article  Google Scholar 

  8. Yuan, Y., Cao, Z., Su, L.: Light-field image superresolution using a combined deep CNN based on EPI. IEEE Signal Process. Lett. 25(9), 1359–1363 (2018)

    Article  Google Scholar 

  9. Wang, Y., Liu, F., Zhang, K., Hou, G., Sun, Z., Tan, T.: LFNet: a novel bidirectional recurrent convolutional neural network for light-field image super-resolution. IEEE Trans. Image Process. 27(9), 4274–4286 (2018)

    Article  MathSciNet  Google Scholar 

  10. Zhang, S., Lin, Y., Sheng, H.: Residual networks for light field image super-resolution. In: CVPR, pp. 11046–11055. IEEE (2019)

    Google Scholar 

  11. Yeung, H.W.F., Hou, J., Chen, X., Chen, J., Chen, Z., Chung, Y.Y.: Light field spatial super-resolution using deep efficient spatial-angular separable convolution. IEEE Trans. Image Process. 28(5), 2319–2330 (2018)

    Article  MathSciNet  Google Scholar 

  12. Jin, J., Hou, J., Chen, J., Kwong, S.: Light field spatial super-resolution via deep combinatorial geometry embedding and structural consistency regularization. In: CVPR, pp. 2260–2269. IEEE (2020)

    Google Scholar 

  13. Jin, J., Hou, J., Zhu, Z., Chen, J., Kwong, S.: Deep selective combinatorial embedding and consistency regularization for light field super-resolution. arXiv preprint arXiv:2009.12537 (2020)

  14. Wang, Y., Wang, L., Yang, J., An, W., Yu, J., Guo, Y.: Spatial-angular interaction for light field image super-resolution. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020, Part XXIII. LNCS, vol. 12368, pp. 290–308. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58592-1_18

    Chapter  Google Scholar 

  15. Cai, Z., Zheng, X.: A private and efficient mechanism for data uploading in smart cyber-physical systems. IEEE Trans. Netw. Sci. Eng. 7(2), 766–775 (2020)

    Article  MathSciNet  Google Scholar 

  16. Cai, Z., He, Z.: Trading private range counting over big IoT data. In: 2019 IEEE 39th International Conference on Distributed Computing Systems, pp. 144–153. IEEE (2019)

    Google Scholar 

  17. Zheng, X., Cai, Z.: Privacy-preserved data sharing towards multiple parties in industrial IoTs. IEEE J. Sel. Areas Commun. 38(5), 968–979 (2020)

    Article  Google Scholar 

  18. Cai, Z., Xiong, Z., Xu, H., Wang, P., Li, W., Pan, Y.: Generative adversarial networks: a survey towards private and secure applications. ACM 37(4), 38 (2020)

    Google Scholar 

  19. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2015)

    Article  Google Scholar 

  20. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part IV. LNCS, vol. 8692, pp. 184–199. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10593-2_13

    Chapter  Google Scholar 

  21. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: CVPR, pp. 1874–1883. IEEE (2016)

    Google Scholar 

  22. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: CVPR, pp. 1646–1654. IEEE (2016)

    Google Scholar 

  23. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: CVPRW, pp. 136–144. IEEE (2017)

    Google Scholar 

  24. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778. IEEE (2016)

    Google Scholar 

  25. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: ECCV, pp. 286–301 (2018)

    Google Scholar 

  26. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR, pp. 4681–4690. IEEE (2017)

    Google Scholar 

  27. Honauer, K., Johannsen, O., Kondermann, D., Goldluecke, B.: A dataset and evaluation methodology for depth estimation on 4D light fields. In: Lai, S.-H., Lepetit, V., Nishino, K., Sato, Y. (eds.) ACCV 2016, Part III. LNCS, vol. 10113, pp. 19–34. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-54187-7_2

    Chapter  Google Scholar 

  28. Wanner, S., Meister, S., Goldluecke, B.: Datasets and benchmarks for densely sampled 4D light fields. In: VMV, pp. 225–226 (2013)

    Google Scholar 

  29. Rerabek, M., Ebrahimi, T.: New light field image dataset. In: 8th International Conference on Quality of Multimedia Experience (2016)

    Google Scholar 

  30. Le Pendu, M., Jiang, X., Guillemot, C.: Light field inpainting propagation via low rank matrix completion. IEEE Trans. Image Process. 27(4), 1981–1993 (2018)

    Article  MathSciNet  Google Scholar 

  31. Vaish, V., Adams, A.: The (new) Stanford light field archive. Comput. Graph. Lab. Stanford Univ. Tech. Rep. 6(7), 73 (2008)

    Google Scholar 

Download references

Acknowledgement

This study is partially supported by the National Key R&D Program of China (No. 2018YFB2100500), the National Natural Science Foundation of China (No. 61635002), the Science and Technology Development Fund, Macau SAR (File no. 0001/2018/AFJ), the Fundamental Research Funds for the Central Universities and the Open Fund of the State Key Laboratory of Software Development Environment (No. SKLSDE-2021ZX-03). Thank you for the support from HAWKEYE Group.

Author information

Authors and Affiliations

  1. State Key Laboratory of Software Development Environment, School of Computer Science and Engineering, Beihang University, Beijing, 100191, People’s Republic of China

    Donglin Li, Da Yang, Sizhe Wang & Hao Sheng

  2. Beihang Hangzhou Innovation Institute Yuhang, Beihang University, Xixi Octagon City, Yuhang District, Hangzhou, 310023, People’s Republic of China

    Donglin Li, Da Yang, Sizhe Wang & Hao Sheng

Authors
  1. Donglin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Da Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sizhe Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hao Sheng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Da Yang .

Editor information

Editors and Affiliations

  1. Nanjing University of Aeronautics and Astronautics, Nanjing, China

    Zhe Liu

  2. Shanghai Jiao Tong University, Shanghai, China

    Fan Wu

  3. Missouri University of Science and Technology, Rolla, MO, USA

    Sajal K. Das

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Li, D., Yang, D., Wang, S., Sheng, H. (2021). Light Field Super-Resolution Based on Spatial and Angular Attention. In: Liu, Z., Wu, F., Das, S.K. (eds) Wireless Algorithms, Systems, and Applications. WASA 2021. Lecture Notes in Computer Science(), vol 12937. Springer, Cham. https://doi.org/10.1007/978-3-030-85928-2_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-85928-2_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-85927-5

  • Online ISBN: 978-3-030-85928-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation