Skip to main content
SpringerLink
Log in

An Improved 4D Convolutional Neural Network for Light Field Reconstruction

  • Conference paper
  • First Online:
Mobile Networks and Management (MONAMI 2023)

Included in the following conference series:

  • 53 Accesses

Abstract

Light field (LF) camera sensors often face a trade-off between angular resolution and spatial resolution when shooting. High spatial resolution image arrays often result in lower angular resolution, and vice versa. In order to obtain high spatial resolution and at the same time have high angular resolution. In this paper, we propose an improved 4D convolutional neural network (CNN) algorithm for angular super-resolution (SR) to improve the quality of angular SR images. Firstly, to address the problem of low luminance of images captured by LF cameras, this paper uses block threshold square reinforcement (BTSR) for image luminance enhancement. Secondly, to make the reconstructed new viewpoints of higher quality, this paper improves the attention mechanism convolutional block attention module (CBAM). This paper incorporates it into a 4D dense residual network as high dimensional attention module (HDAM). HDAM generates images along two independent dimensions, spatial and channel. The HDAM generates attention maps along two independent dimensions, space and channel, which guide the network to focus on more important features for adaptive feature modification. Finally, this paper modifies the activation function to make the network perform better in the later stages of training and more suitable for LF reconstruction tasks. This paper evaluates the network on many LF data, including real-world scenes and synthetic data. The experimental results show that the improved network algorithm can achieve higher quality LF reconstruction.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Shin, C., Jeon, H.-G., Yoon, Y., Kweon, I.S., Kim, S. J.: EPINET: a fully-convolutional neural network using epipolar geometry for depth from light field images. In Proceedings of IEEE Conference on Computer Vision Pattern Recognition, pp. 4748–4757 (2018)

    Google Scholar 

  2. Mitra, K., Veeraraghavan, A.: Light field denoising, light field superresolution and stereo camera based refocussing using a GMM light field patch prior. In: Proceedings of IEEE Conference on Computer Vision Pattern Recognition Workshops, pp. 22–28 (2012)

    Google Scholar 

  3. Yucer, K., Sorkine-Hornung, A., Wang, O., Sorkine-Hornung, O.: Efficient 3D object segmentation from densely sampled light fields with applications to 3D reconstruction. ACM Trans. Graph. 35(3), 22:1–22:15 (2016)

    Google Scholar 

  4. Kim, C., Zimmer, H., Pritch, Y., Sorkine-Hornung, A., Gross, M.: Scene reconstruction from high spatio-angular resolution light fields. ACM Trans. Graph. 32(4), 73:1–73:12 (2013)

    Google Scholar 

  5. Wang, T.-C., Efros, A.A., Ramamoorthi, R.: Occlusion-awaredepth estimation using light-field cameras. In: Proceedings of IEEE International Conference on Computer Vision, pp. 3487–3495 (2015)

    Google Scholar 

  6. Pearson, J., Brookes, M., Dragotti, P.L.: Plenoptic layer-based modeling for image based rendering. IEEE Trans. Image Process. 22(9), 3405–3419 (2013)

    Article  MathSciNet  Google Scholar 

  7. Zhang, Z., Liu, Y., Dai, Q.: Light field from micro-baseline image pair. In: Proceedings of IEEE Conference on Computer Vision Pattern Recognition, pp. 3800–3809 (2015)

    Google Scholar 

  8. Zhang, F.-L., et al.: PlenoPatch: patch-based plenoptic image manipulation. IEEE Trans. Visualization Comput. Graph. 23(5), 1561–1573 (2017). https://doi.org/10.1109/TVCG.2016.2532329

    Article  Google Scholar 

  9. Chai, J.X., Tong, X., Chan, S.C., et al.: Plenoptic sampling. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 307–318 (2000)

    Google Scholar 

  10. Zhang, C., Chen, T.: Spectral analysis for sampling image-based rendering data. IEEE Trans. Circuits Syst. Video Technol. 13(11), 1038–1050 (2003)

    Article  Google Scholar 

  11. Do, M.N., Marchand-Maillet, D., Vetterli, M.: On the bandwidth of the plenoptic function. IEEE Trans. Image Process. 21(2), 708–717 (2011)

    Article  MathSciNet  Google Scholar 

  12. Zhu, C.J., Yu, L.: Spectral analysis of image-based rendering data with scene geometry. Multimedia Syst. 23, 627–644 (2017)

    Article  Google Scholar 

  13. Vagharshakyan, S., Bregovic, R., Gotchev, A.: Light field reconstruction using shearlet transform. IEEE Trans. Pattern Anal. Mach. Intell. 40(1), 133–147 (2018)

    Article  Google Scholar 

  14. Chen, W., Zhu, C.: Spectral analysis of a surface occlusion model for image-based rendering sampling. Digital Signal Process. 130, 103697 (2022)

    Article  Google Scholar 

  15. Yoon, Y., Jeon, H.G., Yoo, D., Lee, J.Y., So Kweon, I.: Learning a deep convolutional network for light-field image superresolution. In: Proceedings of IEEE International Conference on Computer Vision Workshops, pp. 24–32 (2015)

    Google Scholar 

  16. Flynn, J., Neulander, I., Philbin, J., Snavely, N.: DeepStereo: learning to predict new views from the world’s imagery. In: Proceedings of IEEE Conference on Computer Vision Pattern Recognition, pp. 5515–5524 (2016)

    Google Scholar 

  17. Wu, G., Liu, Y., Fang, L., Dai, Q., Chai, T.: Light field reconstruction using convolutional network on EPI and extended applications. IEEE Trans. Pattern Anal. Mach. Intell. 41(7), 1681–1694 (2019)

    Article  Google Scholar 

  18. Wu, G., Liu, Y., Dai, Q., Chai, T.: Learning sheared EPI structure for light field reconstruction. IEEE Trans. Image Process. 28(7), 3261–3273 (2019)

    Article  MathSciNet  Google Scholar 

  19. Wang, Y., Liu, F., Wang, Z., Hou, G., Sun, Z., Tan, T.: End-to-end view synthesis for light field imaging with Pseudo 4DCNN. In: Proceedings of European Conference on Computer Vision, pp. 333–348 (2018)

    Google Scholar 

  20. Yeung, W.F.H., Hou, J., Chen, J., Chung, Y.Y., Chen, X.: Fast light field reconstruction with deep coarse-to-fine modeling of spatial-angular clues. In: Proceedings of European Conference on Computer Vision, pp. 137–152 (2018)

    Google Scholar 

  21. Meng, N., So, H.K.H., Sun, X., et al.: High-dimensional dense residual convolutional neural network for light field reconstruction. IEEE Trans. Pattern Anal. Mach. Intell. 43(3), 873–886 (2019)

    Article  Google Scholar 

  22. Raj, S., Lowney, M., Shah, R., Wetzstein, G.: Stanford lytro light field archive (2016). http://lightfields.stanford.edu/LF2016.html.

  23. Honauer, K., Johannsen, O., Kondermann, D., Goldluecke, B.: A dataset and evaluation methodology for depth estimation on 4D light fields. In: Proceedings of Asian Conference on Computer Vision, pp. 19–34 (2016)

    Google Scholar 

Download references

Acknowledgment

This work was supported in part by National Natural Science Foundation of China (No. 62067003), Culture and Art Science Planning Project of Jiangxi Province (No. YG2018042), Humanities and Social Science Project of Jiangxi Province (No.JC18224).

Author information

Authors and Affiliations

  1. School of Software Engineering, Jiangxi University of Science and Technology, Nanchang, 330013, China

    Qiuming Liu, Ruiqin Li, Ke Yan & Yichen Wang

  2. Nanchang Key laboratory of Virtual Digital Factory and Cultural Communications, Nanchang, 330013, People’s Republic of China

    Qiuming Liu

  3. School of Software, Jiangxi Normal University, Nanchang, 330022, China

    Yong Luo

Authors
  1. Qiuming Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ruiqin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ke Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yichen Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yong Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qiuming Liu .

Editor information

Editors and Affiliations

  1. The University of Electro-Communications, Tokyo, Japan

    Celimuge Wu

  2. VTT Technical Research Centre, Helsinki, Finland

    Xianfu Chen

  3. Xidian University, Xi'an, China

    Jie Feng

  4. Jiangxi University of Technology, Nanchang, China

    Zhen Wu

Rights and permissions

Reprints and permissions

Copyright information

© 2024 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Liu, Q., Li, R., Yan, K., Wang, Y., Luo, Y. (2024). An Improved 4D Convolutional Neural Network for Light Field Reconstruction. In: Wu, C., Chen, X., Feng, J., Wu, Z. (eds) Mobile Networks and Management. MONAMI 2023. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 559. Springer, Cham. https://doi.org/10.1007/978-3-031-55471-1_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-55471-1_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-55470-4

  • Online ISBN: 978-3-031-55471-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation