Skip to main content
SpringerLink
Log in

User-dependent interactive light field video streaming system

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

A Correction to this article was published on 20 December 2021

This article has been updated

Abstract

The sheer size and complex structure of light field (LF) videos bring new challenges to their compression and transmission. There have been numerous LF video compression algorithms reported in the literature to date. All of these algorithms compress and transmit all the views of an LF video. However, in some interactive or selective applications where users can choose the area of interest to be displayed, these algorithms generate a significant computational load and enormous data redundancies. In this paper, we propose an interactive LF video streaming system based on a user-dependent view selection scheme and an LF video coding method, which streams only the required data. Specifically, by predicting trajectories and using projection models, the viewing area of users in a limited consecutive number of time slots is firstly calculated, and then a user-dependent view selection method is proposed to determine the selected views of users for streaming. Finally, with the novel LF video sequence formed by only the selected sets of views, an adaptive coding method is presented for different LF video sequences based on users’ gestures. Experimental results illustrate that the proposed interactive LF video streaming system can achieve the best performance compared with other comparison methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Change history

References

  1. Avramelos V, De Praeter J, Van Wallendael G, Lambert P (2020) Random access prediction structures for light field video coding with MV-HEVC. Multimedia Tools and Applications 79:1–21

    Article  Google Scholar 

  2. Bakir N, Hamidouche W, Déforges O, Samrouth K, Khalil M (2018) Light field image compression based on convolutional neural networks and linear approximation. 2018 25th IEEE International conference on image processing (ICIP). IEEE, Athens, pp 1128–1132

    Chapter  Google Scholar 

  3. Bjontegaard G (2001) Calculation of average psnr differences between RD-curves. VCEG-M33

  4. Chen J, Hou J, Chau LP (2017) Light field compression with disparity-guided sparse coding based on structural key views. IEEE Transactions on Image Processing 27(1):314–324

    Article  MathSciNet  Google Scholar 

  5. Dai F, Zhang J, Ma Y, Zhang Y (2015) Lenselet image compression scheme based on subaperture images streaming. 2015 IEEE International conference on image processing (ICIP). IEEE, Quebec City, QC, pp 4733–4737

    Chapter  Google Scholar 

  6. Fecker U, Kaup A, (2005) H.264, AVC-compatible coding of dynamic light fields using transposed picture ordering. In: 2005 13th European signal processing conference. IEEE, Antalya, pp 1–4

  7. Jia C, Zhang X, Wang S, Wang S, Ma S (2018) Light field image compression using generative adversarial network-based view synthesis. IEEE Journal on Emerging and Selected Topics in Circuits and Systems 9(1):177–189

    Article  Google Scholar 

  8. Khoury J, Pourazad MT, Nasiopoulos P (2019) A new prediction structure for efficient MV-HEVC based light field video compression. 2019 International conference on computing, networking and communications (ICNC). IEEE, Honolulu, HI, pp 588–591

    Chapter  Google Scholar 

  9. Kovács PT, Nagy Z, Barsi A, Adhikarla VK, Bregović R (2014) Overview of the applicability of H. 264/MVC for real-time light-field applications. In (2014) 3DTV-conference: the true vision-capture, transmission and display of 3D video (3DTV-CON). IEEE, Budapest, pp 1–4

    Google Scholar 

  10. Kurutepe E, Civanlar MR, Tekalp AM (2007) Client-driven selective streaming of multiview video for interactive 3DTV. IEEE Transactions on Circuits and Systems for Video Technology 17(11):1558–1565

    Article  Google Scholar 

  11. Lafruit G, Domański M, Wegner K, Grajek T, Senoh T, Jung J, Kovács PT, Goorts P, Jorissen L, Munteanu A et al (2016) New visual coding exploration in MPEG: Super-Multiview and Free Navigation in Free viewpoint TV. Electronic Imaging 5:1–9

    Article  Google Scholar 

  12. Levoy M, Hanrahan P (1996) Light field rendering. In: Proceedings of the 23rd international conference on computer graphics and interactive techniques. ACM, New Orleans, LA, pp 31–42

  13. Li L, Li Z, Li B, Liu D, Li H (2017) Pseudo-sequence-based 2-D hierarchical coding structure for light-field image compression. IEEE Journal of Selected Topics in Signal Processing 11(7):1107–1119

    Article  Google Scholar 

  14. Liu D, Wang L, Li L, Xiong Z, Wu F, Zeng W (2016) Pseudo-sequence-based light field image compression. 2016 IEEE International conference on multimedia & expo workshops (ICMEW). IEEE, Seattle, WA, pp 1–4

    Google Scholar 

  15. Mehajabin N, Luo SR, Yu HW, Khoury J, Kaur J, Pourazad MT (2019) An efficient random access light field video compression utilizing diagonal inter-view prediction. 2019 IEEE International conference on image processing (ICIP). IEEE, Taipei, pp 3567–3570

    Chapter  Google Scholar 

  16. Merkle P, Muller K, Smolic A, Wiegand T (2006) Efficient compression of multi-view video exploiting inter-view dependencies based on H. 264/MPEG4-AVC. In: 2006 IEEE International conference on multimedia and expo. IEEE, Toronto, ON, pp 1717–1720

  17. Ng R, Levoy M, Brédif M, Duval G, Horowitz M, Hanrahan P (2005) Light field photography with a hand-held plenoptic camera. Research Report, Stanford University

  18. Pan Z, Ikuta Y, Bandai M, Watanabe T (2011) User dependent scheme for multi-view video transmission. 2011 IEEE International Conference on Communications (ICC). IEEE, Kyoto, pp 1–5

    Google Scholar 

  19. Peixoto E, Macchiavello B, Hung EM, Dorea C, Cheung G (2017) Progressive communication for interactive light field image data streaming. 2017 IEEE International conference on image processing (ICIP). IEEE, Beijing, pp 1925–1929

    Chapter  Google Scholar 

  20. Peixoto E, Macchiavello B, Hung EM, Cheung G (2018) Progressive sub-aperture image recovery for interactive light field data streaming. 2018 25th IEEE International conference on image processing (ICIP). IEEE, Athens, pp 3289–3293

    Chapter  Google Scholar 

  21. Perra C (2016) Light field image compression based on preprocessing and high efficiency coding. 2016 24th Telecommunications forum (TELFOR). IEEE, Belgrade, pp 1–4

    Google Scholar 

  22. Perwass C, Wietzke L (2012) Single lens 3D-camera with extended depth-of-field. In: Human vision and electronic imaging XVII, international society for optics and photonics. Burlingame, CA, p 829108

  23. Ramanathan P (2005) Compression and interactive streaming of light fields. PhD thesis, Stanford University

  24. Tech G, Chen Y, Müller K, Ohm JR, Vetro A, Wang YK (2015) Overview of the multiview and 3D extensions of high efficiency video coding. IEEE Transactions on Circuits and Systems for Video Technology 26(1):35–49

    Article  Google Scholar 

  25. Vetro A, Wiegand T, Sullivan GJ (2011) Overview of the stereo and multiview video coding extensions of the H. 264/MPEG-4 AVC standard. Proceedings of the IEEE 99(4):626–642

  26. Wang B, Peng Q, Wang E, Han K, Xiang W (2019) Region-of-interest compression and view synthesis for light field video streaming. IEEE Access 7:41183–41192

    Article  Google Scholar 

  27. Wang G, Xiang W, Pickering M (2016) A cross-platform solution for light field based 3D telemedicine. Computer Methods and Programs in Biomedicine 125:103–116

    Article  Google Scholar 

  28. Wang G, Xiang W, Pickering M, Chen CW (2016) Light field multi-view video coding with two-directional parallel inter-view prediction. IEEE Transactions on Image Processing 25(11):5104–5117

    Article  MathSciNet  Google Scholar 

  29. Wang TC, Zhu JY, Kalantari NK, Efros AA, Ramamoorthi R (2017) Light field video capture using a learning-based hybrid imaging system. ACM Transactions on Graphics 36(4):1–13

    Article  Google Scholar 

  30. Wilburn B, Joshi N, Vaish V, Talvala EV, Antunez E, Barth A, Adams A, Horowitz M, Levoy M (2005) High performance imaging using large camera arrays. In: Proceedings of the 32nd International conference on computer graphics and interactive techniques. ACM, Los Angeles, CA, pp 765–776

  31. Zhao S, Chen Z (2017) Light field image coding via linear approximation prior. 2017 IEEE International conference on image processing (ICIP). IEEE, Beijing, pp 4562–4566

    Chapter  Google Scholar 

  32. Zhao S, Chen Z, Yang K, Huang H (2016) Light field image coding with hybrid scan order. 2016 Visual communications and image processing (VCIP). IEEE, Chengdu, pp 1–4

    Google Scholar 

  33. Zhao Z, Wang S, Jia C, Zhang X, Ma S, Yang J (2018) Light field image compression based on deep learning. 2018 IEEE International conference on multimedia and expo (ICME). IEEE, San Diego, CA, pp 1–6

    Google Scholar 

Download references

Acknowledgements

Thanks are due to Dr. Pan Gao for assistance with the experiments and paper revision. In addition, the work of Bing Wang was partially supported by the China Scholarship Council (CSC) under Grant 201707000093.

Author information

Authors and Affiliations

  1. School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu, 611756, China

    Bing Wang, Qiang Peng & Xiao Wu

  2. School of Science and Engineering, James Cook University, Cairns, QLD, 4878, Australia

    Eric Wang

  3. School of Engineering and Mathematical Science, La Trobe University, Melbourne, VIC, 3686, Australia

    Wei Xiang

Authors
  1. Bing Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qiang Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eric Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wei Xiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiao Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The abstract contains spelling error, Tables 2 and 3 contain added data, and the phrase “with the proposed system” in the second line of the second paragraph of section 6 has to be removed.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, B., Peng, Q., Wang, E. et al. User-dependent interactive light field video streaming system. Multimed Tools Appl 81, 1893–1918 (2022). https://doi.org/10.1007/s11042-021-11602-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11602-8

Keywords

Search

Navigation