A concise review of the quality of experience assessment for video streaming
Introduction
The rapid development of mobile and high definition video devices and of the network infrastructure used for video streaming requires a permanent evolution of the techniques used to assess the video quality of experience (QoE). The objective of this paper is to provide a concise, up-to-date view of this research field.
In the last decade, interactive voice traffic (Voice over IP – VoIP) has been added to the traditional network data traffic (web, email, file transfers). Today, VoIP is common in IP networks, and the trend is a rapidly increasing in video traffic, namely, video on demand (VoD) and IP television (IPTV). Moreover, the rapid popularization of mobile devices with video display support, such as notebooks, tablets and smartphones, and the dissemination of wireless networks (WLANs and 3G/4G) contribute to this scenario. In a few years, 90% of the content transmitted over the Internet will be related to videos, which will be viewed by over a billion people [1], [2].
These services are transmitted using a streaming technique through an Internet service provider or a private corporate IP network. The contents are presented to the user as they are sent by the source, without the need to store the complete file for later viewing. A buffer is used to store a few seconds of content before their display to minimize sporadic failures or delay fluctuations in the network transmission.
A typical infrastructure used to provide a video streaming service is composed of three elements (see Fig. 1). In the headend, the contents are created, edited, encoded, and stored in a multimedia database, which is made available by a streaming server. Next, the contents are divided into several IP packets and transmitted to the customers through the core network. Finally, via an access node in a customer network, the contents are displayed on the user’s device, which can be a television, a desktop computer, a notebook, a tablet, or a smartphone.
As the success of a video streaming service is heavily linked to the quality level assurance, the contents are displayed on customer devices with minimal failures or delays. Usually, a network manager monitors network information, such as bandwidth, delay, jitter, throughput, and packet loss, to provide adequate quality for each customer. However, this task becomes difficult due to the complexity of the network infrastructure, and when mobile devices are included in this scenario, the difficulties are even greater due to new problems, such as wireless signal coverage, a high rate of packet loss, and wireless channel instability.
Given the required conditions for video transmission to customers over IP networks, the features offered by the network define the concept of quality of service (QoS). However, other information can also be measured, such as resolution and codification of video contents. All of these factors strongly influence the quality as perceived by the user, which in turn determines the level of quality of experience (QoE). Presently, the rapid development of new technology allows for the emergence of devices with new resolutions, screen sizes, and contrast and brightness features. For this reason, the techniques used to measure perceived quality, as described in the remainder of this paper, must be carefully reexamined.
Various papers have explored the approaches and methodologies used to evaluate video quality in multimedia services. Winkler and Mohandas [3] discuss the evolution of subjective and objective metrics used for video quality measurement and introduce a new hybrid metric named V-Factor. A state-of-the-art perceptual-based audio and video quality assessment is described by You et al. [4] as are some relevant quality metrics to develop a joint audio and video assessment. Seshadrinathan and Bovik [5] present recent developments in a multimedia signal (audio, image, and video) quality assessment with a focus on full-reference methods. A classification scheme for full-reference and reduced-reference video quality assessment methods is introduced by Chikkerur et al. [6] that takes into account the natural visual characteristics (natural visual statistics and natural visual features) and perceptual characteristics (frequency-domain and pixel-domain methods). Yang and Wan [7] analyze the factors that may affect the quality of the networked video method and some bitstream-based methods to evaluate video quality. Finally, a classification of objective video quality and a comparison with different metrics, distortion types, and video databases is provided by Vranješ, Rimac-Drlje, and Grgić [8].
The main goal of this paper is to summarize current and emerging approaches to evaluate the quality of a video streaming service. It presents concepts related to QoS and QoE as well as factors that influence each one. A typical process of video service quality evaluation is detailed and the different assessment methods are divided into subjective, objective and hybrid approaches and compared. A discussion about future trends and challenges in video quality assessment completes the study.
The remainder of this paper is organized as follows. Section 2 defines QoS, QoE and related factors. Section 3 details the video quality assessment process, the available methodologies, and the various approaches. Section 4 discusses future trends and challenges in video quality assessment, and Section 5 presents the conclusions.
Section snippets
What is Quality of Service (QoS)?
QoS is defined by the International Telecommunication Union (ITU) as a set of characteristics of a telecommunication service that focuses on user satisfaction [9], [10], while the Internet Engineering Task Force (IETF) summarizes QoS as a collection of requirements to be met by the transport data stream of a particular service [11]. Bandwidth, delay, jitter, and packet loss rate are some of the most common parameters used to measure QoS.
In addition to QoS, the services can also be evaluated
Approaches and methodologies used in video quality assessment
The methodologies used for video quality assessment are usually divided into four stages (see Fig. 4): (1) selection of reference videos (without introduction of error); (2) creation of distorted videos; (3) evaluation of distorted videos by users; and (4) statistical assessment of distorted videos.
The reference (raw) videos are not encoded by any type of compression. The spatial resolutions commonly used range from CIF (352 × 288) to high definition (1280 × 720 and 1920 × 1080) and, recently, to
Future directions and challenges
To improve the techniques used to measure the QoE perceived by users in a video streaming service, a range of issues remains as the state-of-the-art of video quality assessment continues to face important challenges that must be addressed.
The fundamental challenge is related to the quality predictor model. Each model uses a different choice of inputs because of the existence of several information sources such as network infrastructure (jitter, delay, packet loss, and bandwidth), video
Conclusion
Favored by the popularization of mobile devices (tablets and smartphones) and improvements to the Internet infrastructure, the use of video streaming services has grown quickly, allowing customers to watch videos anywhere, anytime, anyplace. Nevertheless, none of this advanced technology has any value if the video content provider cannot guarantee the consumer high quality videos. While the QoE represents the quality level perceived by the end user, there are many challenges remaining to be
Acknowledgments
This work was supported by the following Brazilian institutions: CAPES (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior), CNPq (Conselho Nacional de Desenvolvimento Científico e Tecnológico), FAPEMIG (Fundação de Amparo à Pesquisa do Estado de Minas Gerais), IFAM (Instituto Federal do Amazonas), CT-PIM (Centro de Ciência, Tecnologia e Inovação do Pólo Industrial de Manaus), and PRPq-UFMG (Pró-Reitoria de Pesquisa da UFMG).
References (95)
- et al.
Perceptual-based quality assessment for audio–visual services: a survey
Signal Process. Image Commun.
(2010) - et al.
Review of objective video quality metrics and performance comparison using different databases
Signal Process. Image Commun.
(2013) - et al.
A survey of QoE assurance in converged networks
Comput. Netw. IEEE
(2011) - et al.
No-reference image and video quality estimation: applications and human-motivated design
Image Commun.
(2010) - et al.
Perceptual-based quality assessment for audio-visual services: a survey
Signal Process. Image Commun.
(2010) - et al.
Cross-layer optimization of OFDM transmission systems for MPEG-4 video streaming
Comput. Commun.
(2004) - et al.
Watching video over the web: Part 1: Streaming protocols
Internet Comput. IEEE
(2011) - Cisco Systems Inc., Cisco Visual Networking Index: Forecast and Methodology, 2012–2017, White Paper, May...
- et al.
The evolution of video quality measurement: from PSNR to hybrid metrics
IEEE Trans. Broadcast.
(2008) - et al.
Automatic prediction of perceptual quality of multimedia signals – a survey
Multimedia Tools Appl.
(2011)
Objective video quality assessment methods: a classification, review, and performance comparison
IEEE Trans. Broadcast.
Bitstream-based quality assessment for networked video: a review
Commun. Mag. IEEE
Quality and quality of experience
A generic quantitative relationship between quality of experience and quality of service
Netw. IEEE
No-reference quality assessment of H.264/AVC encoded video
Trans. Circ. Syst. Video Technol. IEEE
Video transport evaluation with H.264 video traces
Commun. Surv. Tutorials IEEE
Traffic and statistical multiplexing characterization of 3-D video representation formats
IEEE Trans. Broadcast.
Study of subjective and objective quality assessment of video
IEEE Trans. Image Process.
Video quality assessment on mobile devices: subjective, behavioral and objective studies
IEEE J. Sel. Top. Signal Process.
Cited by (43)
A learning-based approach for video streaming over fluctuating networks with limited playback buffers
2024, Computer CommunicationsMultimedia communication over cognitive radio networks from QoS/QoE perspective: A comprehensive survey
2020, Journal of Network and Computer ApplicationsCitation Excerpt :This type of subjective evaluation is not considered as an efficient metric because of the limited observers and assessors, limited distortion types, and high expenditure. On the other hand, in the objective quality assessment approaches, a factor is evaluated as a function of QoS metrics (such as PLR, latency, jitter, bitrate, and frame rate) and some other external factors that include the type of content, viewer demography, and device type (Maia et al., 2015). Normally, the objective models popular metrics include moving picture quality metric (MPQM), perceptual video quality measure (PVQM), and visual signal-to-noise ratio (VSNR) (Moorthy et al., 2017).
Software-defined unmanned aerial vehicles networking for video dissemination services
2019, Ad Hoc NetworksCitation Excerpt :The decoder applies a Frame-Copy method for error concealment to replace each lost frame with the last received one, reducing frame loss and maintaining the video quality. In terms of video quality evaluation, QoS schemes alone are not enough to assess the quality level of multimedia applications, because they fail in capturing subjective aspects of video content related to human experience [28,29]. QoE metrics overcome those limitations, and thus we rely on two well-known objective QoE metrics, namely Structural Similarity (SSIM) and Video Quality Metric (VQM).
A holistic modeling for QoE estimation in live video streaming applications over LTE Advanced technologies with Full and Non Reference approaches
2018, Computer CommunicationsCitation Excerpt :Regarding QoE estimation, different approaches in the related work have been found. In [24], the authors review current methodologies used to evaluate QoE in a video streaming service. The authors show concepts related to QoS and QoE as well as other factors that may influence them.
An Efficient Stacked Deep Incremental Model for Online Streaming Video QoE Prediction
2023, International Journal of Computing and Digital Systems