Skip to main content
SpringerLink
Log in

VCFNet: video clarity-fluency network for quality of experience evaluation model of HTTP adaptive video streaming services

  • 1221: Deep Learning for Image/Video Compression and Visual Quality Assessment
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In HTTP adaptive video streaming service, video clarity and fluency are the two most important Influencing Factors (IFs) affecting user’s Quality of Experience (QoE). In this paper, a Video Clarity-Fluency Network (VCFNet) is proposed to establish a QoE evaluation model, which focuses on characterizing the clarity-fluency characteristics of video. Firstly, the Harmonic-ResNeXt101 network is constructed by introducing the Harmonic Network into ResNeXt101 to capture the clarity information of video frames. The output of the Fully Connected (FC) layer of the Harmonic-ResNeXt101 network is fed into the Gated Recurrent Unit (GRU), which is used to perform short-term temporal modeling to capture the fluency information of video chunk. Then, the final output of GRU is extracted as the clarity-fluency features, which are concatenated with the statistical features of other IFs (including video quality level, re-buffering duration, re-buffering frequency, etc.) to form the feature parameter vector of IFs. Finally, a neural network composed of One-Dimensional Convolutional Neural Network (1D CNN) layer and two FC layers is designed to establish the mapping relationship model between the feature parameter vector of IFs and Mean Opinion Score (MOS) to predict user’s QoE. Experimental results on SQoE-III and SQoE-IV datasets demonstrate that the proposed VCFNet can effectively capture the clarity-fluency information of video, and the resulted QoE model can achieve the state-of-the-art performance compared with the existing QoE evaluation models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Agarla M, Celona L, Schettini R (2020) No-reference quality assessment of in-capture distorted videos. J Imag 6(8):74. https://doi.org/10.3390/jimaging6080074

    Article  Google Scholar 

  2. Bampis CG, Bovik AC (2017) Learning to predict streaming video qoe: distortions, rebuffering and memory. arXiv:1703.00633

  3. Bentaleb A, Begen AC, Zimmermann R (2016) Sdndash: improving qoe of http adaptive streaming using software defined networking. In: Proceedings of the 24th ACM international conference on Multimedia. https://doi.org/10.1145/2964284.2964332, pp 1296–1305

  4. Chen P, Li L, Ma L, Wu J, Shi G (2020) Rirnet: recurrent-in-recurrent network for video quality assessment. In: Proceedings of the 28th ACM international conference on multimedia. https://doi.org/10.1145/3394171.3413717, p 834842

  5. Chen P, Li L, Wu J, Zhang Y, Lin W (2021) Temporal reasoning guided qoe evaluation for mobile live video broadcasting. IEEE Trans Image Process 30:3279–3292. https://doi.org/10.1109/TIP.2021.3060255

    Article  Google Scholar 

  6. Du L, Zhuo L, Li J, Zhang J, Li X, Zhang H (2020) Video quality of experience metric for dynamic adaptive streaming services using dash standard and deep spatial-temporal representation of video. Appl Sci 10(5):1793. https://doi.org/10.3390/app10051793

    Article  Google Scholar 

  7. Duanmu Z, Liu W, Chen D, Li Z, Wang Z, Wang Y, Gao W (2019) A knowledge-driven quality-of-experience model for adaptive streaming videos. arXiv:1911.07944

  8. Duanmu Z, Liu W, Li Z, Chen D, Wang Z, Wang Y, Gao W (2020) Assessing the quality-of-experience of adaptive bitrate video streaming. arXiv:2008.08804

  9. Duanmu Z, Rehman A, Wang Z (2018) A quality-of-experience database for adaptive video streaming. IEEE Trans Broadcast 64(2):474–487. https://doi.org/10.1109/TBC.2018.2822870

    Article  Google Scholar 

  10. Duanmu Z, Zeng K, Ma K, Rehman A, Wang Z (2016) A quality-of-experience index for streaming video. IEEE J Selected Topics Signal Process 11(1):154–166. https://doi.org/10.1109/JSTSP.2016.2608329

    Article  Google Scholar 

  11. Hoßfeld T, Schatz R, Biersack E, Plissonneau L (2013) Internet video delivery in youtube: from traffic measurements to quality of experience. In: Data traffic monitoring and analysis. Springer, pp 264–301. https://doi.org/10.1007/978-3-642-36784-7_11

  12. Jiang J, Sen S (2020) A new abstraction for internet qoe optimization. arXiv:2008.04128

  13. Li D, Jiang T, Jiang M (2019) Quality assessment of in-the-wild videos. In: Proceedings of the 27th ACM international conference on multimedia. https://doi.org/10.1145/3343031.3351028, pp 2351–2359

  14. Li M, Jianbin S, Hui L (2017) A determining method of frame rate and resolution to boost the video live qoe. In: 2017 2nd international conference on multimedia and image processing (ICMIP). https://doi.org/10.1109/ICMIP.2017.26. IEEE, pp 206–209

  15. Li Z, Aaron A, Katsavounidis I, Moorthy A, Manohara M (2016) Toward a practical perceptual video quality metric. The Netflix Tech Blog 6:2

    Google Scholar 

  16. Liu X, Dobrian F, Milner H, Jiang J, Sekar V, Stoica I, Zhang H (2012) A case for a coordinated internet video control plane. In: Proceedings of the ACM SIGCOMM 2012 conference on applications, technologies, architectures, and protocols for computer communication. https://doi.org/10.1145/2342356.2342431, pp 359–370

  17. Luan S, Chen C, Zhang B, Han J, Liu J (2018) Gabor convolutional networks. IEEE Trans Image Process 27(9):4357–4366. https://doi.org/10.1109/TIP.2018.2835143

    Article  MathSciNet  Google Scholar 

  18. Mok RKP, Chan EWW, Chang RKC (2011) Measuring the quality of experience of http video streaming. In: 12th IFIP/IEEE International symposium on integrated network management (IM 2011) and workshops. https://doi.org/10.1109/INM.2011.5990550. IEEE, pp 485–492

  19. Qin Y, Hao S, Pattipati KR, Qian F, Sen S, Wang B, Yue C (2019) Quality-aware strategies for optimizing abr video streaming qoe and reducing data usage. In: Proceedings of the 10th ACM multimedia systems conference. https://doi.org/10.1145/3304109.3306231, pp 189–200

  20. Rassool R (2017) Vmaf reproducibility: validating a perceptual practical video quality metric. In: 2017 IEEE international symposium on broadband multimedia systems and broadcasting (BMSB). https://doi.org/10.1109/BMSB.2017.7986143. IEEE, pp 1–2

  21. Recommendation I (2017) 1203.3, “parametric bitstream-based quality assessment of progressive download and adaptive audiovisual streaming services over reliable transport-quality integration module”. International Telecommunication Union

  22. Sector TS (2016) Series p: terminals and subjective and objective assessment methods, https://videoclarity.com/PDF/T-REC-P.913-201603.pdf. Accessed 30 June 2021

  23. Tran HTT, Nguyen DV, Nguyen DD, Ngoc NP, Thang TC (2019) An lstm-based approach for overall quality prediction in http adaptive streaming. In: IEEE INFOCOM 2019-IEEE conference on computer communications workshops (INFOCOM WKSHPS). https://doi.org/10.1109/INFCOMW.2019.8845041. IEEE, pp 702–707

  24. Ulicny M, Krylov VA, Dahyot R (2018) Harmonic networks: integrating spectral information into cnns. arXiv:1812.03205

  25. Ulicny M, Krylov VA, Dahyot R (2019) Harmonic networks for image classification. In: BMVC, p 202

  26. Ulicny M, Krylov VA, Dahyot R (2019) Harmonic networks with limited training samples. In: 2019 27th European signal processing conference (EUSIPCO). https://doi.org/10.23919/EUSIPCO.2019.8902831. IEEE, pp 1–5

  27. Wang Y, Wang H, Shang J, Tuo H (2019) Resa: a real-time evaluation system for abr. In: 2019 IEEE International conference on multimedia and expo (ICME). https://doi.org/10.1109/ICME.2019.00317. IEEE, pp 1846–1851

  28. Wang Z, Wang J, Wang F, Li C, Fei Z, Rahim T (2017) A video quality assessment method for voip applications based on user experience. Sensing and Imaging 18(1):12. https://doi.org/10.1007/s11220-017-0161-z

    Article  Google Scholar 

  29. Watanabe K, Okamoto J, Kurita T (2007) Objective video quality assessment method for evaluating effects of freeze distortion in arbitrary video scenes. In: Image quality and system performance IV. https://doi.org/10.1117/12.703870, vol 6494. International Society for Optics and Photonics, p 64940P

  30. Xue J, Zhang D-Q, Yu H, Chen CW (2014) Assessing quality of experience for adaptive http video streaming. In: 2014 IEEE International conference on multimedia and expo workshops (ICMEW). https://doi.org/10.1109/ICMEW.2014.6890604. IEEE, pp 1–6

  31. Yan S, Guo Y, Chen Y, Xie F (2018) Predicting freezing of webrtc videos in wifi networks. In: International conference on ad hoc networks. https://doi.org/10.1007/978-3-030-05888-3_27. Springer, pp 292–301

  32. Yin X, Jindal A, Sekar V, Sinopoli B (2015) A control-theoretic approach for dynamic adaptive video streaming over http. In: Proceedings of the 2015 ACM conference on special interest group on data communication. https://doi.org/10.1145/2785956.2787486, pp 325–338

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (No. 61531006, 61971016), Beijing Municipal Education Commission Cooperation Beijing Natural Science Foundation (No.KZ201910005007).

Author information

Authors and Affiliations

  1. Beijing Key Laboratory of Computational Intelligence and Intelligent System, Beijing University of Technology, Beijing, 100124, China

    Lina Du, Jiafeng Li, Li Zhuo & Shuo Yang

  2. Faculty of Information Technology, Beijing University of Technology, Beijing, 100124, China

    Lina Du, Jiafeng Li, Li Zhuo & Shuo Yang

Authors
  1. Lina Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiafeng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Li Zhuo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shuo Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Li Zhuo.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Du, L., Li, J., Zhuo, L. et al. VCFNet: video clarity-fluency network for quality of experience evaluation model of HTTP adaptive video streaming services. Multimed Tools Appl 81, 42907–42923 (2022). https://doi.org/10.1007/s11042-022-13483-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13483-x

Keywords

Search

Navigation