Skip to main content
SpringerLink
Log in

Tiled streaming for layered 3D virtual reality videos with viewport prediction

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In recent years, the demand of 3D video services has gradually increased. More and more bandwidth hungry applications are proposed, such as immersive media services which need a virtual reality (VR) headset and 3D VR videos to provide users immersive experience of watching 3D VR videos. Tiled streaming is often used for providing 3D VR videos die to the high bitrates of 3D VR videos. In a VR headset the sight is limited. Users can only watch a fraction of entire 3D VR videos in the viewport. Transmitting the content of a VR video outside the viewport is unnecessary and infeasible due to the high bitarate of 3D VR videos. Generally, In VR applications, content within the viewport should keep the highest quality while providing only basic quality outside the viewprot. Such an adaptive streaming relies on the precision of viewport prediction. Errors of viewport prediction result in the expensive overhead of quality repairing and re-transmission delay on the traditional versioned coded VR videos,, where a whole new version of video content needs to be resent and the sent content of low-quality version is discarded and cannot be reused. In this paper, we propose a novel adaptative streaming approach for providing 3D VR videos using Scalable Video Coding (SVC) with viewport prediction. For better quality adaptation, we take CubeMap projection as the projection format of 3D VR videos. Besides, we use the Scalability extension of High Efficiency Video Coding (SHVC) to encode 3D VR videos to multiple layers for supplying different qualities and also use the tiling to divide videos into rectangular regions for finer quality adaptation. The experimental results show that our proposed method outperformed other previous approaches in terms of the weighted video quality and the relative time spent on the highest quality, especially with low available network bandwidth. Even under certain miss rates of tiles, compared to previous approaches, our proposed method requires fewer bandwidth overhead and shorter re-transmission delay for repairing the quality of missed tiles in most cases.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Akamai’s (2020) [state of the internet], q1 2017. https://content.akamai.com/gl-en-pg9135-q1-soti-connectivity.htmlhttps://content.akamai.com/gl-en-pg9135-q1-soti-connectivity.html

  2. Alface PR, Mac JF, Nico V (2012) Interactive omnidirectional video delivery: a Bandwidth-Effective approach. Bell Labs Tech J 16(4):135–148. https://doi.org/10.1002/bltj

    Article  Google Scholar 

  3. Boyce JM, Ye Y, Chen J, Ramasubramonian AK (2016) Overview of SHVC: Scalable extensions of the high efficiency video coding standard. IEEE Trans Circuits Syst Video Technol 26(1):20–34. https://doi.org/10.1109/TCSVT.2015.2461951

    Article  Google Scholar 

  4. Chhabra P, Garg NK, Kumar M (2020) Content-based image retrieval system using orb and sift features. Neural Comput & Applic 32(7):2725–2733. https://doi.org/10.1007/s00521-018-3677-9

    Article  Google Scholar 

  5. Dargan S, Kumar M, Ayyagari MR, Kumar G (2020) A survey of deep learning and its applications: a new paradigm to machine learning. Arch Comput Methods Eng 27 (4):1071–1092. https://doi.org/10.1007/s11831-019-09344-w

    Article  MathSciNet  Google Scholar 

  6. Dionisio JDN, III WGB, Gilbert R (2013) 3d virtual worlds and the metaverse: Current status and future possibilities. ACM Comput Surv 45(3). https://doi.org/10.1145/2480741.2480751

  7. Duan H, Li J, Fan S, Lin Z, Wu X, Cai W (2021) Metaverse for Social Good: A University Campus Prototype, Association for Computing Machinery, New York, NY, USA, p 153–161. https://doi.org/10.1145/3474085.3479238

  8. Ffmpeg (2020) https://ffmpeg.org/

  9. Github (2020) facebook/transform360: Transform360 is an equirectangular to cubemap transform for 360 video. https://github.com/facebook/transform360

  10. Gupta S, Kumar M, Garg A (2019) Improved object recognition results using sift and orb feature detector. Multimed Tools Appl 78(23):34157–34171. https://doi.org/10.1007/s11042-019-08232-6

    Article  Google Scholar 

  11. Hevc (2020) scalability extension (shvc) — jct-vc. https://hevc.hhi.fraunhofer.de/shvc

  12. High efficiency video coding (hevc) — jct-vc (2020) https://hevc.hhi.fraunhofer.de/

  13. Hooft JVD, Vega MT, Petrangeli S, Wauters T, Turck FD (2019) Tile-based adaptive streaming for virtual reality video. ACM Transactions on Multimedia Computing, Communications and Applications 15(4). https://doi.org/10.1145/3362101, https://dl.acm.org/doi/pdf/10.1145/3362101

  14. Hosseini M, Swaminathan V (2016) Adaptive 360 VR Video Streaming: Divide and Conquer! 2016 IEEE Int Symp Multimed, pp 107–110. https://doi.org/10.1109/ISM.2016.45, arXiv:1609.08729

  15. Nasrabadi AT, Mahzari A, Beshay JD, Prakash R (2017) Adaptive 360-degree video streaming using scalable video coding. In: MM 2017 - Proceedings of the 2017 ACM Multimedia Conference, pp 1689–1697. https://doi.org/10.1145/3123266.3123414, https://dl.acm.org/doi/pdf/10.1145/3123266.3123414

  16. Petrangeli S, Swaminathan V, Hosseini M, De Turck F (2017) An HTTP/2-based adaptive streaming framework for 360∘ virtual reality videos. In: MM 2017–Proceedings of the 2017 ACM Multimedia Conference, pp 306–314. https://doi.org/10.1145/3123266.3123453, https://dl.acm.org/doi/pdf/10.1145/3123266.3123453

  17. Pourazad MT, Doutre C, Azimi M, Nasiopoulos P (2012) HEVC: The new gold standard for video compression: how does hevc compare with H.264/AVC?. IEEE Consum Electron Mag 1(3):36–46. https://doi.org/10.1109/MCE.2012.2192754, https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6222536

    Article  Google Scholar 

  18. Qian F, Han B, Ji L, Gopalakrishnan V (2016) Optimizing 360 video delivery over cellular networks. In: Proceedings of the Annual International Conference on Mobile Computing and Networking, MOBICOM, pp 1–6. https://doi.org/10.1145/2980055.2980056

  19. Recommended upload encoding settings - youtube help (2020) https://support.google.com/youtube/answer/1722171

  20. Sullivan GJ, Ohm JR, Han WJ, Wiegand T (2012) Overview of the high efficiency video coding (HEVC) standard. IEEE Trans Circuits Syst Video Technol 22(12):1649–1668. https://doi.org/10.1109/TCSVT.2012.2221191

    Article  Google Scholar 

  21. Under the hood: Building 360 video - facebook engineering (2020) https://engineering.fb.com/video-engineering/under-the-hood-building-360-video/

  22. Van Der Hooft J, Torres Vega M, Petrangeli S, Wauters T, De Turck F (2019) Quality Assessment for Adaptive Virtual Reality Video Streaming: A Probabilistic Approach on the User’s Gaze. In: Proceedings of the 2019 22nd Conference on Innovation in Clouds, Internet and Networks and Workshops, ICIN 2019, pp 19–24. https://doi.org/10.1109/ICIN.2019.8685904

  23. Wiegand T, Sullivan GJ, Bjøntegaard G, Luthra A (2003) Overview of the H.264/AVC video coding standard. IEEE Trans Circuits Syst Video Technol 13(7):560–576. https://doi.org/10.1109/TCSVT.2003.815165, https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=1218189

    Article  Google Scholar 

  24. Wu C, Tan Z, Wang Z, Yang S (2017) A Dataset for Exploring User Behaviors in VR Spherical Video Streaming. In: Proceedings of the 8th acm on multimedia systems conference - MMSys’17, pp 193–198. https://doi.org/10.1145/3083187.3083210, http://dl.acm.org/citation.cfm?doid=3083187.3083210

  25. Zare A, Aminlou A, Hannuksela MM, Gabbouj M (2016) HEVC-compliant tile-based streaming of panoramic video for virtual reality applications. In: MM 2016 - Proceedings of the 2016 ACM Multimedia Conference, pp 601–605. https://doi.org/10.1145/2964284.2967292

Download references

Funding

This work was partially supported by National Science Council in Taiwan (R.O.C.) under contract MOST 107-2221-E-024-002-.

Author information

Authors and Affiliations

  1. Department of Computer Science and Information Engineering, National University of Tainan, 33, sec. 2, shu-lin st., Tainan, Taiwan, 70005, Republic of China

    Hong-Yun Chen & Chow-Sing Lin

Authors
  1. Hong-Yun Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chow-Sing Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chow-Sing Lin.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, HY., Lin, CS. Tiled streaming for layered 3D virtual reality videos with viewport prediction. Multimed Tools Appl 81, 13867–13888 (2022). https://doi.org/10.1007/s11042-022-12277-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12277-5

Keywords

Search

Navigation