Elsevier

Computer Communications

Volume 57, 15 February 2015, Pages 1-12
Computer Communications

A concise review of the quality of experience assessment for video streaming

https://doi.org/10.1016/j.comcom.2014.11.005Get rights and content

Highlights

  • Concise and up-to-date review of quality assessment for video streaming services.

  • Description of a typical video assessment process.

  • Analysis of current research on subjective, objective, and hybrid QoE assesment.

  • Discussion of future trends and challenges for QoE in video streaming services.

Abstract

The widespread use of mobile and high definition video devices is changing Internet traffic, with a significant increase in multimedia content, especially video on demand (VoD) and Internet protocol television (IPTV). However, the success of these services is strongly related to the video quality perceived as by the user, also known as quality of experience (QoE). This paper reviews current methodologies used to evaluate the quality of experience in a video streaming service. A typical video assessment diagram is described, and analyses of the subjective, objective, and hybrid approaches are presented. Finally, considering the moving target scenario of mobile and high definition devices, the text outlines challenges and future research directions that should be considered in the measurement and assessment of the quality of experience for video streaming services.

Introduction

The rapid development of mobile and high definition video devices and of the network infrastructure used for video streaming requires a permanent evolution of the techniques used to assess the video quality of experience (QoE). The objective of this paper is to provide a concise, up-to-date view of this research field.

In the last decade, interactive voice traffic (Voice over IP – VoIP) has been added to the traditional network data traffic (web, email, file transfers). Today, VoIP is common in IP networks, and the trend is a rapidly increasing in video traffic, namely, video on demand (VoD) and IP television (IPTV). Moreover, the rapid popularization of mobile devices with video display support, such as notebooks, tablets and smartphones, and the dissemination of wireless networks (WLANs and 3G/4G) contribute to this scenario. In a few years, 90% of the content transmitted over the Internet will be related to videos, which will be viewed by over a billion people [1], [2].

These services are transmitted using a streaming technique through an Internet service provider or a private corporate IP network. The contents are presented to the user as they are sent by the source, without the need to store the complete file for later viewing. A buffer is used to store a few seconds of content before their display to minimize sporadic failures or delay fluctuations in the network transmission.

A typical infrastructure used to provide a video streaming service is composed of three elements (see Fig. 1). In the headend, the contents are created, edited, encoded, and stored in a multimedia database, which is made available by a streaming server. Next, the contents are divided into several IP packets and transmitted to the customers through the core network. Finally, via an access node in a customer network, the contents are displayed on the user’s device, which can be a television, a desktop computer, a notebook, a tablet, or a smartphone.

As the success of a video streaming service is heavily linked to the quality level assurance, the contents are displayed on customer devices with minimal failures or delays. Usually, a network manager monitors network information, such as bandwidth, delay, jitter, throughput, and packet loss, to provide adequate quality for each customer. However, this task becomes difficult due to the complexity of the network infrastructure, and when mobile devices are included in this scenario, the difficulties are even greater due to new problems, such as wireless signal coverage, a high rate of packet loss, and wireless channel instability.

Given the required conditions for video transmission to customers over IP networks, the features offered by the network define the concept of quality of service (QoS). However, other information can also be measured, such as resolution and codification of video contents. All of these factors strongly influence the quality as perceived by the user, which in turn determines the level of quality of experience (QoE). Presently, the rapid development of new technology allows for the emergence of devices with new resolutions, screen sizes, and contrast and brightness features. For this reason, the techniques used to measure perceived quality, as described in the remainder of this paper, must be carefully reexamined.

Various papers have explored the approaches and methodologies used to evaluate video quality in multimedia services. Winkler and Mohandas [3] discuss the evolution of subjective and objective metrics used for video quality measurement and introduce a new hybrid metric named V-Factor. A state-of-the-art perceptual-based audio and video quality assessment is described by You et al. [4] as are some relevant quality metrics to develop a joint audio and video assessment. Seshadrinathan and Bovik [5] present recent developments in a multimedia signal (audio, image, and video) quality assessment with a focus on full-reference methods. A classification scheme for full-reference and reduced-reference video quality assessment methods is introduced by Chikkerur et al. [6] that takes into account the natural visual characteristics (natural visual statistics and natural visual features) and perceptual characteristics (frequency-domain and pixel-domain methods). Yang and Wan [7] analyze the factors that may affect the quality of the networked video method and some bitstream-based methods to evaluate video quality. Finally, a classification of objective video quality and a comparison with different metrics, distortion types, and video databases is provided by Vranješ, Rimac-Drlje, and Grgić [8].

The main goal of this paper is to summarize current and emerging approaches to evaluate the quality of a video streaming service. It presents concepts related to QoS and QoE as well as factors that influence each one. A typical process of video service quality evaluation is detailed and the different assessment methods are divided into subjective, objective and hybrid approaches and compared. A discussion about future trends and challenges in video quality assessment completes the study.

The remainder of this paper is organized as follows. Section 2 defines QoS, QoE and related factors. Section 3 details the video quality assessment process, the available methodologies, and the various approaches. Section 4 discusses future trends and challenges in video quality assessment, and Section 5 presents the conclusions.

Section snippets

What is Quality of Service (QoS)?

QoS is defined by the International Telecommunication Union (ITU) as a set of characteristics of a telecommunication service that focuses on user satisfaction [9], [10], while the Internet Engineering Task Force (IETF) summarizes QoS as a collection of requirements to be met by the transport data stream of a particular service [11]. Bandwidth, delay, jitter, and packet loss rate are some of the most common parameters used to measure QoS.

In addition to QoS, the services can also be evaluated

Approaches and methodologies used in video quality assessment

The methodologies used for video quality assessment are usually divided into four stages (see Fig. 4): (1) selection of reference videos (without introduction of error); (2) creation of distorted videos; (3) evaluation of distorted videos by users; and (4) statistical assessment of distorted videos.

The reference (raw) videos are not encoded by any type of compression. The spatial resolutions commonly used range from CIF (352 × 288) to high definition (1280 × 720 and 1920 × 1080) and, recently, to

Future directions and challenges

To improve the techniques used to measure the QoE perceived by users in a video streaming service, a range of issues remains as the state-of-the-art of video quality assessment continues to face important challenges that must be addressed.

The fundamental challenge is related to the quality predictor model. Each model uses a different choice of inputs because of the existence of several information sources such as network infrastructure (jitter, delay, packet loss, and bandwidth), video

Conclusion

Favored by the popularization of mobile devices (tablets and smartphones) and improvements to the Internet infrastructure, the use of video streaming services has grown quickly, allowing customers to watch videos anywhere, anytime, anyplace. Nevertheless, none of this advanced technology has any value if the video content provider cannot guarantee the consumer high quality videos. While the QoE represents the quality level perceived by the end user, there are many challenges remaining to be

Acknowledgments

This work was supported by the following Brazilian institutions: CAPES (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior), CNPq (Conselho Nacional de Desenvolvimento Científico e Tecnológico), FAPEMIG (Fundação de Amparo à Pesquisa do Estado de Minas Gerais), IFAM (Instituto Federal do Amazonas), CT-PIM (Centro de Ciência, Tecnologia e Inovação do Pólo Industrial de Manaus), and PRPq-UFMG (Pró-Reitoria de Pesquisa da UFMG).

References (95)

  • S. Chikkerur et al.

    Objective video quality assessment methods: a classification, review, and performance comparison

    IEEE Trans. Broadcast.

    (2011)
  • F. Yang et al.

    Bitstream-based quality assessment for networked video: a review

    Commun. Mag. IEEE

    (2012)
  • ITU-T Recommendation Y.1901, Requirements for the Support of IPTV Services,...
  • ITU-T Recommendation E.800, Definitions of Terms Related to Quality of Service,...
  • IETF RFC 2386, A Framework for QoS-based Routing in the Internet, August...
  • J. Tapolcai, P. Cholda, T. Cinkler, K. Wajda, A. Jajszczyk, A. Autenrieth, S. Bodamer, D. Colle, G. Ferraris, H....
  • H. Zepernick, U. Engelke, Quality of experience of multimedia services: past, present, and future, in: Adjunct...
  • A. Raake et al.

    Quality and quality of experience

  • K. Hyun-Jong, C. Seong-Gon, A study on a QoS/QoE correlation model for QoE evaluation on IPTV service, in: 12th...
  • M. Fiedler et al.

    A generic quantitative relationship between quality of experience and quality of service

    Netw. IEEE

    (2010)
  • S. Baraković, J. Baraković, H. Bajrić, QoE dimensions and QoE measurement of NGN services, in: Proceedings of the 18th...
  • J. Xue, C. Wen Chen, A study on perception of mobile video with surrounding contextual influences, in: Fourth...
  • Patrick Le Callet, Sebastian Möller, Andrew Perkis (Eds.), Qualinet White Paper on Definitions of Quality of Experience...
  • EPFL-PoliMI Video Quality Assessment Database, May 2014....
  • F. De Simone, M. Naccari, M. Tagliasacchi, F. Dufaux, S. Tubaro, T. Ebrahimi, Subjective assessment of H.264/AVC video...
  • F. De Simone, M. Tagliasacchi, M. Naccari, S. Tubaro, T. Ebrahimi, A H.264/AVC video database for the evaluation of...
  • IRCCyN/IVC 1080i Database, May 2014....
  • S. Péchard, R. Pépion, P. Le Callet, Suitable methodology in subjective video quality assessment: a resolution...
  • IRCCyN/IVC H264 AVC vs SVC VGA Video Database, May 2014....
  • Y. Pitrey, M. Barkowsky, P. Le Callet, R. Pepion, Subjective quality assessment of MPEG-4 scalable video coding in a...
  • IRCCyN/IVC Influence Content Video VGA Database, May 2014....
  • Y. Pitrey, M. Barkowsky, R. Pépion, P. L. Callet, H. Hlavacs, Influence of the source content and encoding...
  • IT-IST Lisbon H.264/MPEG-2 Video Quality Database, May 2014....
  • T. Brandao et al.

    No-reference quality assessment of H.264/AVC encoded video

    Trans. Circ. Syst. Video Technol. IEEE

    (2010)
  • T. Brandão, M.P. Queluz, No-reference perceptual quality metric for H.264/AVC encoded video, in: Proc. of International...
  • T. Brandão, L. Roque, M.P. Queluz, Quality assessment of H.264/AVC encoded video, in: Proc. of Conf. on...
  • Video Trace Library (TRACE) – YUV Video Sequences, May 2014....
  • P. Seeling et al.

    Video transport evaluation with H.264 video traces

    Commun. Surv. Tutorials IEEE

    (2012)
  • A. Pulipaka et al.

    Traffic and statistical multiplexing characterization of 3-D video representation formats

    IEEE Trans. Broadcast.

    (2013)
  • TUM 1080p25 Data Set, May 2014....
  • C. Keimel, J. Habigt, T. Habigt, M. Rothbucher, K. Diepold, Visual quality of current coding technologies at high...
  • TUM 1080p50 Data Set, May 2014....
  • C. Keimel, A. Redl, K. Diepold, The TUM high definition video datasets, in: Fourth International Workshop on Quality of...
  • LIVE Video Quality Database, May 2014....
  • K. Seshadrinathan et al.

    Study of subjective and objective quality assessment of video

    IEEE Trans. Image Process.

    (2010)
  • LIVE Mobile Video Quality Database, May 2014....
  • A.K. Moorthy et al.

    Video quality assessment on mobile devices: subjective, behavioral and objective studies

    IEEE J. Sel. Top. Signal Process.

    (2012)
  • Cited by (43)

    • Multimedia communication over cognitive radio networks from QoS/QoE perspective: A comprehensive survey

      2020, Journal of Network and Computer Applications
      Citation Excerpt :

      This type of subjective evaluation is not considered as an efficient metric because of the limited observers and assessors, limited distortion types, and high expenditure. On the other hand, in the objective quality assessment approaches, a factor is evaluated as a function of QoS metrics (such as PLR, latency, jitter, bitrate, and frame rate) and some other external factors that include the type of content, viewer demography, and device type (Maia et al., 2015). Normally, the objective models popular metrics include moving picture quality metric (MPQM), perceptual video quality measure (PVQM), and visual signal-to-noise ratio (VSNR) (Moorthy et al., 2017).

    • Software-defined unmanned aerial vehicles networking for video dissemination services

      2019, Ad Hoc Networks
      Citation Excerpt :

      The decoder applies a Frame-Copy method for error concealment to replace each lost frame with the last received one, reducing frame loss and maintaining the video quality. In terms of video quality evaluation, QoS schemes alone are not enough to assess the quality level of multimedia applications, because they fail in capturing subjective aspects of video content related to human experience [28,29]. QoE metrics overcome those limitations, and thus we rely on two well-known objective QoE metrics, namely Structural Similarity (SSIM) and Video Quality Metric (VQM).

    • A holistic modeling for QoE estimation in live video streaming applications over LTE Advanced technologies with Full and Non Reference approaches

      2018, Computer Communications
      Citation Excerpt :

      Regarding QoE estimation, different approaches in the related work have been found. In [24], the authors review current methodologies used to evaluate QoE in a video streaming service. The authors show concepts related to QoS and QoE as well as other factors that may influence them.

    • An Efficient Stacked Deep Incremental Model for Online Streaming Video QoE Prediction

      2023, International Journal of Computing and Digital Systems
    View all citing articles on Scopus
    View full text