Skip to main content
SpringerLink
Log in

Assessing the QoE in Video Services Over Lossy Networks

  • Published:
Journal of Network and Systems Management Aims and scope Submit manuscript

Abstract

Quality of experience is of critical importance in streaming video services, because the traditional quality of service cannot represent the quality perceived by viewers. This work evaluates several objective quality metrics under realistic bursty packet loss conditions in the network, with the support of a packet loss model. Alignment of reference and streamed video sequences (with different levels of spatial-temporal information) are also explored as a technique to prevent inaccurate computation of objective metrics due to frame loss. Finally, the correlation between subjective and objective metrics for each motion level and the computing time of metrics are analysed. The most suitable objective metrics to characterize the real degradation in the quality perceived by viewers, for both off-line and real-time assessment, are proposed. The integration of motion, busty packet loss, sequence alignment after frame loss and computing time of metrics are the main contributions of this research work.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Altman, E., Avrachenkov, K., Barakat, C.: TCP in presence of bursty losses. Perform. Eval. 42(2–3), 129–147 (2000). doi:10.1016/S0166-5316(00)00027-4

    Article  MATH  Google Scholar 

  2. Hasslinger, G., Hohlfeld, O.: The Gilbert–Elliott model for packet loss in real time services on the internet. In: Proceedings of the 14th GI/ITG Conference on Measuring. Modelling and Evaluation of Computer and Communication Systems (MMB 2008), pp. 269–283. Dortmund, Germany (2008)

  3. Ellis, M., Pezaros, D., Kypraios, T., Perkins, C.: Modelling packet loss in RTP-based streaming video for residential users. In: Proceedings of the 37th IEEE Conference on Local Computer Networks (LCN 2012), pp. 220–223. Clearwater, USA (2012). doi:10.1109/LCN.2012.6423613

  4. Van Moorsel, A.: Metrics for the internet age: quality of experience and quality of business. In: Proceedings of the 5th Performability Workshop, pp. 26–31. Erlangen, Germany (2001)

  5. Chikkerur, S., Sundaram, V., Reisslein, M., Karam, : Objective video quality assessment methods: a classification, review, and performance comparison. IEEE Trans. Broadcast. 57(2), 165–182 (2011). doi:10.1109/TBC.2011.2104671

    Article  Google Scholar 

  6. Claypool, M., Tanner, J.: The effects of jitter on the peceptual quality of video. In: Proceedings of the 7th ACM International Conference on Multimedia (ACM Multimedia’99) (part 2), pp. 115–118. Orlando, USA (1999). doi:10.1145/319878.319909

  7. Boyce, J., Gaglianello, R.: Packet loss effects on MPEG video sent over the public internet. In: Proceedings of the 6th ACM International Conference on Multimedia (ACM Multimedia’98), pp. 181–190. Bristol, England (1998). doi:10.1145/290747.290770

  8. Greengrass, J., Evans, J., Begen, A.: Not all packets are equal, part 2: the impact of network packet loss on video quality. IEEE Internet Comput. 13(2), 74–82 (2009). doi:10.1109/MIC.2009.40

    Article  Google Scholar 

  9. Boyaci, O., Forte, A., Schulzrinne, H.: Performance of video-chat applications under congestion. In: Proceedings of the 11th IEEE International Symposium on Multimedia (ISM’09), pp. 213–218. San Diego, USA (2009). doi:10.1109/ISM.2009.45

  10. Pinson, M., Wolf, S., Cermak, G.: HDTV subjective quality of H.264 vs. MPEG-2, with and without packet loss. IEEE Trans. Broadcast. 56(1), 86–91 (2010). doi:10.1109/TBC.2009.2034511

    Article  Google Scholar 

  11. Loguinov, D., Radha, H.: Large-scale experimental study of Internet performance using video traffic. Comput. Commun. Rev. 32(1), 7–19 (2002). doi:10.1145/510726.510727

    Article  Google Scholar 

  12. Calyam, P., Lee, C.G.: Characterizing voice and video traffic behavior over the Internet. In: Proceedings of the 20th International Symposium on Computer and Information Sciences (ISCIS 2005). Istanbul, Turkey (2005)

  13. Ellis, M., Perkins, C.: Packet loss characteristics of IPTV-like traffic on residential links. In: Proceedings of the 7th Annual IEEE Consumer Communications and Networking Conference (CCNC 2010), pp. 1–5. Las Vegas, USA (2010). doi:10.1109/CCNC.2010.5421735

  14. Final reports from the video quality experts group on the validation of objective models of video and multimedia quality assessment. Technical report, Video Quality Experts Group (VQEG). “http://www.its.bldrdoc.gov/vqeg/reports.aspx”. Accessed 23 Jan 2015

  15. Besson, A., De Simone, F., Ebrahimi, T.: Objective quality metrics for video scalability. In: Proceedings of the 20th IEEE International Conference on Image Processing (ICIP), pp. 59–63. Melbourne, Australia (2013). doi: 10.1109/ICIP.2013.6738013

  16. Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004). doi:10.1109/TIP.2003.819861

    Article  Google Scholar 

  17. Wang, Z., Simoncelli, E., Bovik, A.: Multi-scale structural similarity for image quality assessment. In: Proceedings of the 37th Asilomar Conference on Signals, Systems and Computers, pp. 1398–1402. Piscataway, USA (2003). doi:10.1109/ACSSC.2003.1292216

  18. Damera-Venkata, N., Kite, T., Geisler, W., Evans, B., Bovik, A.: Image quality assessment based on a degradation model. IEEE Trans. Image Process. 9(4), 636–650 (2000). doi:10.1109/83.841940

    Article  Google Scholar 

  19. Wang, Z., Bovik, A.: A universal image quality index. IEEE Signal Process. Lett. 9(3), 81–84 (2002). doi:10.1109/97.995823

    Article  Google Scholar 

  20. Sheikh, H., Bovik, A.: Image information and visual quality. IEEE Trans. Image Process. 15(2), 430–444 (2006). doi:10.1109/TIP.2005.859378

    Article  Google Scholar 

  21. Rimac-Drlje, S., Vranjes, M., Zagar, D.: Influence of temporal pooling method on the objective video quality evaluation. In: Proceedings of the 2nd IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB ’09), pp. 1–5. Bilbao, Spain (2009). doi:10.1109/ISBMSB.2009.5133781

  22. Seufert, M., Slanina, M., Egger, S., Kottkamp, M.: “To pool or not to pool”: A comparison of temporal pooling methods for HTTP adaptive video streaming. In: Proceedings of the 5th International Workshop on Quality of Multimedia Experience (QoMEX 2013), pp. 52–57. Klagenfurt, Austria (2013). doi:10.1109/QoMEX.2013.6603210

  23. Wang, Z., Lu, L., Bovik, A.: Video quality assessment based on structural distortion measurement. Signal Process. Image 19(2), 121–132 (2004). doi:10.1016/S0923-5965(03)00076-6

    Article  Google Scholar 

  24. Sheikh, H., Bovik, A.: A visual information fidelity approach to video quality assessment. In: Proceedings of the 1st International Workshop on Video Processing and Quality Metrics for Consumer Electronics, pp. 23–25. Scottsdale, USA (2005)

  25. Politis, I., Dounis, L., Tselios, C., Kordelas, A., Dagiuklas, T., Papadakis, A.: A model of network related QoE for 3D video. In: Proceedings of the IEEE Global Communications Conference (GLOBECOM 2012), pp. 1335–1340. Anaheim, USA (2012). doi: 10.1109/GLOCOMW.2012.6477776

  26. Xiao, F.: DCT-based video quality evaluation. Technical Report, MSU Graphics and Media Lab (Video Group) (2000). “http://compression.ru/video/quality_measure/vqm”. Accessed 23 Jan 2015

  27. Watson, A., Hu, J., McGowan III, J.: DVQ: a digital video quality metric based on human vision. J. Electron. Imaging 10, 20–29 (2001). doi:10.1117/1.1329896

    Article  Google Scholar 

  28. Pinson, M., Wolf, S.: A new standardized method for objectively measuring video quality. IEEE Trans. Broadcast. 50(3), 312–322 (2004). doi:10.1109/TBC.2004.834028

    Article  Google Scholar 

  29. Seshadrinathan, K., Bovik, A.: Motion tuned spatio-temporal quality assessment of natural videos. IEEE Trans. Image Process. 19(2), 335–350 (2010). doi:10.1109/TIP.2009.2034992

    Article  MathSciNet  Google Scholar 

  30. MPEVQ, Advanced perceptual evaluation of video quality. Technical Report, OPTICOM GmbH (2005). “http://www.opticom.de/download/PEVQ-WP-v07-A4”. Accessed 23 Jan 2015

  31. Soundararajan, R., Bovik, A.: Video quality assessment by reduced reference spatio-temporal entropic differencing. IEEE Trans. Circuits Syst. Video Technol. 23(4), 684–694 (2013). doi:10.1109/TCSVT.2012.2214933

    Article  Google Scholar 

  32. Narwaria, M., Lin, W., Liu, A.: Low-complexity video quality assessment using temporal quality variations. IEEE Trans. Multimed. 14(3–1), 525–535 (2012). doi:10.1109/TMM.2012.2190589

    Article  Google Scholar 

  33. Yang, F., Song, J., Wan, S., Wu, H.: Content-adaptive packet-layer model for quality assessment of networked video services. IEEE J. Sel. Top. Signal Process. 6(6), 672–683 (2012). doi:10.1109/JSTSP.2012.2207705

    Article  Google Scholar 

  34. Joskowicz, J., Sotelo, R., López-Ardao, J.: Towards a general parametric model for perceptual video quality estimation. IEEE Trans. Broadcast. 59(4), 569–579 (2013). doi:10.1109/TBC.2013.2277951

    Article  Google Scholar 

  35. Mohamed, S., Rubino, G.: A study of real-time packet video quality using random neural networks. IEEE Trans. Circuits Syst. Video Technol. 12(12), 1071–1083 (2002). doi:10.1109/TCSVT.2002.806808

    Article  Google Scholar 

  36. Khan, A., Sun, L., Ifeachor, E., Fajardo, J., Liberal, F.: Impact of RLC losses on quality prediction for H.264 video over UMTS networks. In: Proceedings of the 11th IEEE International Conference on Multimedia and Expo (ICME), pp. 702–707. Singapore (2010). doi: 10.1109/ICME.2010.5583033

  37. Mok, R., Chan, E., Chang, R.: Measuring the quality of experience of HTTP video streaming. In: Proceedings of the 12th IFIP/IEEE International Symposium on Integrated Network Management (IM 2011), pp. 485–492. Dublin, Ireland (2011). doi: 10.1109/INM.2011.5990550

  38. Reibman, A., Vaishampayan, V., Sermadevi, Y.: Quality monitoring of video over a packet network. IEEE Trans. Multimed. 6(2), 327–334 (2004). doi:10.1109/TMM.2003.822785

    Article  Google Scholar 

  39. Babu, R., Bopardikar, A., Perkis, A., Hillestad, O.: No-reference metrics for video streaming applications. In: Proceedings of the 14th International Workshop on Packet Video (PV 2004). Irvine, USA (2004)

  40. Venkataraman, M., Chatterjee, M., Chattopadhyay, S.: Evaluating quality of experience for streaming video in real time. In: Proceedings of the 24th IEEE Global Communications Conference (GLOBECOM 2009), pp. 1–6. Honolulu, USA (2009). doi:10.1109/GLOCOM.2009.5425806

  41. Laghari, K., Pham, T., Nguyen, H., Crespi, N.: QoM: A new quality of experience framework for multimedia services. In: Proceedings of the 17th IEEE Symposium on Computers and Communications (ISCC 2012), pp. 851–856. Cappadocia, Turkey (2012). doi: 10.1109/ISCC.2012.6249408

  42. Staelens, N., Van Wallendael, G., Crombecq, K., Vercammen, N., De Cock, J., Vermeulen, B., Van de Walle, R., Dhaene, T., Demeester, P.: No-reference bitstream-based visual quality impairment detection for high definition H.264/AVC encoded video sequences. IEEE Trans. Broadcast. 58(2), 187–199 (2012). doi:10.1109/TBC.2012.2189334

    Article  Google Scholar 

  43. Chan, A., Zeng, K., Mohapatra, P., Lee, S., Banerjee, S.: Metrics for evaluating video streaming quality in lossy IEEE 802.11 wireless networks. In: Proceedings of the 29th IEEE International Conference on Computer Communications (INFOCOM 2010), pp. 1613–1621. San Diego, USA (2010). doi:10.1109/INFCOM.2010.5461979

  44. McDonagh, P., Pande, A., Murphy, L., Mohapatra, P.: Toward deployable methods for assessment of quality for scalable IPTV services. IEEE Trans. Broadcast. 59(2), 223–237 (2013). doi:10.1109/TBC.2013.2255776

    Article  Google Scholar 

  45. Subjective video quality assessment methods for multimedia applications (ITU-T P.910 Recommendation). Technical Report, International Telecommunication Union (ITU) (2009). “http://www.itu.int/rec/T-REC-P.910-200804-I”. Accessed 23 Jan 2015

  46. Gilbert, E.: Capacity of a burst-noise channel. Bell Syst. Tech. J. 39(5), 1253–1265 (1960). doi:10.1002/j.1538-7305.1960.tb03959.x

    Article  Google Scholar 

  47. Elliott, E.: Estimates of error rates for codes on burst-noise channels. Bell Syst. Tech. J. 42(5), 1977–1997 (1963). doi:10.1002/j.1538-7305.1963.tb00955.x

    Article  Google Scholar 

  48. Salsano, S., Ludovici, F., Ordine, A., Giannuzzi, D.: Definition of a general and intuitive loss model for packet networks and its implementation in the Netem module in the Linux kernel. Technical Report, University of Rome (2012). “http://netgroup.uniroma2.it/TR/TR-loss-netem”. Accessed 23 Jan 2015

  49. Reference algorithm for computing peak signal to noise ratio of a processed video sequence with compensation for constant spatial shifts, constant temporal shift, and constant luminance gain and offset (ITU-T J.340 Recommendation). Technical Report, International Telecommunication Union (ITU) (2010). “https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-J.340-201006-I!!PDF-E&type=items”. Accessed 23 Jan 2015

  50. VLC media player. “http://www.videolan.org/vlc/”. VideoLAN Organization. Accessed 23 Jan 2015

  51. García, P.: Python visual quality metric package (PyMetrikz). “https://bitbucket.org/kuraiev/pymetrikz”. University of Brasilia. Accessed 23 Jan 2015

  52. Moskow State University (MSU) Video quality measurement tool. “http://compression.ru/video/quality_measure/video_measurement_tool_en.html”. MSU Graphics and Media Lab (Video Group). Accessed 23 Jan 2015

  53. Visual quality metric (VQM). “http://www.its.bldrdoc.gov/resources/video-quality-research/software.aspx”. Institute of Telecommunication Sciences (ITS), US National Telecommunication and Information Administration (NTIA). Accessed 23 Jan 2015

  54. Test video sequences. “http://www.atc.uniovi.es/personal/fran/papers/SuarezFJ13-videos.rar”. Computer Engineering Group at University of Oviedo. Accessed 23 Jan 2015

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, University of Oviedo, 33203, Gijón, Spain

    Francisco J. Suárez, Juan C. Granda, Daniel F. García & Pelayo Nuño

  2. A.G. Ingeniera y Sistemas S.L. (labSens), Avenida de Asturias 12, 5ºA, 33600, Mieres, Spain

    Andrés García

Authors
  1. Francisco J. Suárez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrés García
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Juan C. Granda
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Daniel F. García
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Pelayo Nuño
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Francisco J. Suárez.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Suárez, F.J., García, A., Granda, J.C. et al. Assessing the QoE in Video Services Over Lossy Networks. J Netw Syst Manage 24, 116–139 (2016). https://doi.org/10.1007/s10922-015-9343-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10922-015-9343-y

Keywords

Search

Navigation