Skip to main content
Springer Nature Link
Log in

Subjective and objective quality assessment of videos in error-prone network environments

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Compression and transmission are two fundamental stages involved in wireless video communications, each of which may cause degradation of the quality of experience (QoE) of end users by producing compression artifacts and packet loss artifacts, respectively. They have their own unique perceptual influences. To provide insight for designing QoE-aware content delivery applications, this paper studies subjective and objective quality of videos containing both types of artifacts. First, subjective quality assessment is conducted, from which interaction between the two types of artifacts during quality perception is investigated. Second, using the subjective data, the performance of the state-of-the-art objective quality metrics is evaluated, with the aim of examining suitability of the existing metrics for their use in error-prone video communication applications. Finally, the developed data set is made publicly available for the community.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Bauer DF (1972) Constructing confidence sets using rank statistics. J Amer Stat Assoc 67(339):687–690

    Article  MATH  Google Scholar 

  2. Boulos F, Parrein B, Le Callet P, Hands David S (2009) Perceptual effects of packet loss on H.264/AVC encoded videos. In: Proceedings of the fourth international workshop on video processing and quality metrics for consumer electronics, pp 1–6

  3. Bradley RA, Terry ME (1952) Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 39:324–345

    MathSciNet  MATH  Google Scholar 

  4. Chandler DM, Hemami SS (2007) VSNR: A aavelet-based visual signal-to-noise ratio for natural images. IEEE Trans Image Proc 16(9):2284–2298

    Article  MathSciNet  Google Scholar 

  5. Chikkerur S, Sundaram V, Reisslein M, Karam L (2011) Objective video quality assessment methods: a classification, review, and performance comparison. IEEE Trans Broadcast 57(2):165–182

    Article  Google Scholar 

  6. De Simone F, Tagliasacchi M, Naccari M, Tubaro S, Ebrahimi T (2010) A H.264/AVC video database for the evaluation of quality metrics. In: Proceedings of the IEEE international conference on acoustics speech and signal processing (ICASSP). IEEE, pp 2430–2433

  7. ETSI, TS 101 154 V1.9.1: Digital Video Broadcasting (DVB); specification for the use of video and audio coding in broadcasting applications based on the MPEG-2 transport stream (2010)

  8. Gilbert EN (1960) Capacity of a burst-noise channel. Bell Syst Tech J 39(1960):1253–1265

    Article  MathSciNet  Google Scholar 

  9. Glickman ME (1999) Parameter estimation in large dynamic paired comparison experiments. J Roy Stat Soc Ser C (Appl Statist) 48(3):377–394

    Article  MATH  Google Scholar 

  10. Hong D, Horowitz M, Eleftheriadis A, Wiegand T (2010) H.264 hierarchical P coding in the context of ultra-low delay, low complexity applications. In: Proceedings of the picture coding symposium (PCS), pp 146–149

  11. Hore A, Ziou F (2010) Image quality metrics: PSNR vs. SSIM. In: Proc. IEEE int. conf. on pattern recognition, vol 10, pp 2366–2369

  12. [ITU], Available: http://www.its.bldrdoc.gov/n3/video/vqmsoftware.htm: NTIA General Video Quality Metric (VQM) Software (2013)

  13. Joint Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG: H.264/AVC JM Reference Software. Available: http://iphome.hhi.de/suehring/tml/ (2012)

  14. Justel A, Pena D, Zamar R (1997) A multivariate Kolmogorov-Smirnov test of goodness of fit. Stat Prob Lett 35(3):251–259

    Article  MathSciNet  MATH  Google Scholar 

  15. Kim SJ, Chae CB, Lee JS (2012) Quality perception of coding artifacts and packet loss in networked video communications. In: Proc. IEEE glob. telecom. conf., pp 1357–1361

  16. Lee JS, Goldmann L, Ebrahimi T (2013) Paired comparison-based subjective quality assessment of stereoscopic images. Multimed Tools Appl 67(1):31–48

    Article  Google Scholar 

  17. Lee JS, Simone FD, Ebrahimi T (2011) Subjective quality assessment of paried comparison: application to scalable video coding. IEEE Trans Multimed 13(5):882–893

    Article  Google Scholar 

  18. Liu T, Boyce J, Yang H, Wu Z (2009) A novel video quality metric for low bit-late video considering both coding and packet-loss artifacts. IEEE J Select Topics Sig Proc 3(2):280–293

    Article  Google Scholar 

  19. Mitsa T, Varkur K (1993) Evaluation of contrast sensitivity functions for formulation of quality measures incorporated in halftoning algorithms. In: Proc. IEEE int. conf. acoust., speech and sig. proc., vol 5, pp 301–304

  20. The SVT high definition multi format test Set. Available: http://www.its.bldrdoc.gov/vqeg (2000)

  21. Ou YF, Ma Z, Wang Y (2009) A novel quality metric for compressed video considering both frame rate and quantization artifacts. In: Proceedings of the international workshop video processing and quality metrics for consumer (VPQM), pp 1–5

  22. Park J, Seshadrinathan K, Lee S, Bovik AC (2013) Video quality pooling adaptive to perceptual distortion severity. IEEE Trans Image Proc 22(2):610–620

    Article  MathSciNet  Google Scholar 

  23. Peli E (1990) Contrast in complex images. J. Opt. Soc. Amer. A 7(10):2032–2039

    Article  Google Scholar 

  24. Pitrey Y, Engelke U, Barkowsky M, Pépion R, Le Callet P et al (2011) Aligning subjective tests using a low cost common set. In: QoE for multimedia content sharing

  25. Recommendation ITU com 9-80-E: final report from the video quality experts group on the validation of objective models of video quality assessment (2000)

  26. Recommendation ITU-R BT.500-13: methodology for the subjective assessment of the quality of television pictures (2012)

  27. Recommendation ITU-T J.144: objective perceptual video quality measurement techniques for digital cable television in the presence of a full reference (2004)

  28. Recommendation ITU-T P.910: subjective video quality assessment methods for multimedia applications (1999)

  29. Seshadrinathan K, Bovik AC (2010) Motion tuned spatio-temporal quality assessment of natural videos. IEEE Trans Image Proc 19(2):335–350

    Article  MathSciNet  Google Scholar 

  30. Seshadrinathan K, Soundararajan R, Bovik AC, Cormack LK (2010) Study of subjective and objective quality assessment of video. IEEE Trans Image Proc 19(6):1427–1441

    Article  MathSciNet  Google Scholar 

  31. Sheikh HR, Bovik AC (2006) Image information and visual quality. IEEE Trans Image Proc 15(2):430–444

    Article  Google Scholar 

  32. Simone FD, Naccari M, Tagliasacchi M, Dufaux F, Tubaro S, Ebrahimi T (2011) Subjective quality assessment of H.264/AVC video streaming with packet losses. EURASIP J Image Video Proc 2011:1–12

    Article  Google Scholar 

  33. Spearman C (1904) The proof and measurement of association between two things, vol 15

  34. Tan W, Patti A (2010) Improving error resilience of scalable H.264 (SVC) via drift control. In: Proc. IEEE int. conf. acoust., speech and sig. proc., pp 2302–2305

  35. Venkata ND, Kite T, Geisler W, Evans B, Bovik A (2000) Image quality assessment based on a degradation model. IEEE Trans Image Proc 9(4):636–650

    Article  Google Scholar 

  36. Wang Z, Bovik AC (2002) A universal image quality index. IEEE Sig Proc Lett 9(2):81–84

    Article  Google Scholar 

  37. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Proc 16(9):600–612

    Article  Google Scholar 

  38. Wang Z, Simoncelli EP, Bovik AC (2003) Multiscale structural similarity for image quality assessment. In: Proc. of asilomar conf. on sign., syst. and computers, vol 2, pp 1398–1402

  39. Wickelmaier F, Schmid C (2004) A MATLAB function to estimate choice model parameters from paired-comparison data. Behav Res Methods Instruments Comp 36:29–40

    Article  Google Scholar 

  40. Winkler S (2012) Analysis of public image and video databases for quality assessment. IEEE J Select Topics Sig Proc 6(6):1–10

    Article  Google Scholar 

  41. Wu HR, Rao KR (2005) Digital video image quality and perceptual coding. CRC Press

  42. You J, Korhonen J, Perkis A (2010) Spatial and temporal pooling of image quality metrics for perceptual video quality assessment on packet loss streams. In: Proc. IEEE int. conf. acoust., speech and sig. proc., vol 7491, pp 1002–1005

  43. Zhang F, Li S, Ma L, Wong YC, Ngan KN (2011) IVP subjective quality video database. http://ivp.ee.cuhk.edu.hk/research/database/subjective/

Download references

Acknowledgment

This work was supported in part by the Students’ Association of the Graduate School of Yonsei University funded by the Graduate School of Yonsei University, in part by the MSIP(Ministry of Science, ICT & Future Planning), Korea in the ICT R&D Program 2013 (KCA-2012-911-01-106) and in part by the IT Consilience Creative Program (NIPA-2014-H0201-14-1002) supervised by the NIPA (National IT Industry Promotion Agency).

Author information

Authors and Affiliations

  1. School of Integrated Technology, Yonsei University, 85 Songdogwahak-ro, Yeonsu-gu, Incheon, 406-840, Korea

    Soo-Jin Kim, Chan-Byoung Chae & Jong-Seok Lee

Authors
  1. Soo-Jin Kim
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Chan-Byoung Chae
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Jong-Seok Lee
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding authors

Correspondence to Chan-Byoung Chae or Jong-Seok Lee.

Additional information

A preliminary version of this paper was presented at the Workshop on Quality of Experience for Multimedia Communications (QoEMC) at the IEEE Global Communications Conference (GLOBECOM) in 2012 [15].

The database used in this paper is available for download at: http://jongseoklee.org/downloads.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, SJ., Chae, CB. & Lee, JS. Subjective and objective quality assessment of videos in error-prone network environments. Multimed Tools Appl 75, 6849–6870 (2016). https://doi.org/10.1007/s11042-015-2613-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-015-2613-6

Keywords