Skip to main content
SpringerLink
Log in

Efficient, robust and divisible paired comparison for subjective quality assessment

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The scale of a database is important for machine learning based image and video quality assessment. Nevertheless, it is greatly limited by the subjective test method. Among the various methods, Paired Comparison(PC) is acknowledged as the most reliable one. However, the test duration grows with square of the number of samples. To solve the dilemma, we propose an improved paired comparison method in this paper. Three types of priori are incorporated to cut down the test duration, including the long-term priori as experience results condensed in existing quality metric, the short-term priori as the subjective scores calculated by the predecessor in ongoing session, and the dynamic priori as the previous decision made by the current assessor. Based on these priori knowledge, only indispensable part of decision is needed to be made by the assessor. Equivalent performance could be achieved in one-tenth of the time used in full paired comparison method. While it is robust to mis-click and divisible to expand the database to a large scale.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Bosc E, Péepion R, Le Callet P, Köppel M, Ndjiki-Nya P, Pressigout M, Morin L (2011) Towards a new quality metric for 3-D synthesized view assessment. IEEE J Sel Top Signal Process 5(7):1332–1343. doi:10.1109/JSTSP.2011.2166245

    Article  Google Scholar 

  2. Bradley RA, Terry ME (1952) Rank analysis of incomplete block designs: i. The method of paired comparisons. Biometrika 39(3):324–345

    MathSciNet  MATH  Google Scholar 

  3. Chandler DM (2010) Most apparent distortion: full-reference image quality assessment and the role of strategy. J Electron Imaging 19(1):011,006. doi:10.1117/1.3267105

    Article  Google Scholar 

  4. Chandler DM, Hemami SS (2007) VSNR: A wavelet-based visual signal-to-noise ratio for natural images. IEEE Trans Image Process 16(9):2284–98

    Article  MathSciNet  Google Scholar 

  5. Daniel WW (1990) Spearman rank correlation coefficient. In: Applied nonparametric statistics, 2nd edn. PWS-Kent, Boston, pp 358–365

  6. Farrell J (2001) Efficient method for paired comparison. J Electron Imaging 10(2):394. doi:10.1117/1.1344187

    Article  Google Scholar 

  7. Glickman ME, Jensen ST (2005) Adaptive paired comparison design. J Stat Plan Infer 127(1-2):279–293. doi:10.1016/j.jspi.2003.09.022

    Article  MathSciNet  MATH  Google Scholar 

  8. Huynh-Thu Q, Ghanbari M (2005) A comparison of subjective video quality assessment methods for low-bit rate and low-resolution video. In: Proceedings of the IASTED international conference on signal and image processing. ACTA Press, pp 70–76

  9. ITU (1996) Recommendation ITU-T P.910, Subjective video quality assessment methods for multimedia applications. Tech. rep., ITU-T, Geneva

  10. ITU (2002) BT.500-11: Methodology for the subjective assessment of the quality of television pictures. Tech. Rep. BT.500-11, International Telecommunication Union. http://www.itu.int/rec/R-REC-BT.500/en

  11. ITU-R (2012) ITU-R BT.2022: General viewing conditions for subjective assessment of quality of SDTV and HDTV television pictures on flat panel displays. Tech. rep., International Telecommunication Union

  12. ITU-R (2012) Recommendation ITU-R BT.2021: Subjective methods for the assessment of stereoscopic 3DTV systems. Tech. rep., International Telecommunication Union

  13. Kendall MG (1938) A new measure of rank correlation. Biometrika 30(1/2):81

    Article  MATH  Google Scholar 

  14. Lee JS, De Simone F, Ebrahimi T (2011) Subjective quality evaluation via paired comparison: application to scalable video coding. IEEE Trans Multimed 13 (5):882–893. doi:10.1109/TMM.2011.2157333

    Article  Google Scholar 

  15. Lee JS, Goldmann L, Ebrahimi T (2012) Paired comparison-based subjective quality assessment of stereoscopic images. Multimed Tools Appl 67(1):31–48. doi:10.1007/s11042-012-1011-6

    Article  Google Scholar 

  16. Li J, Barkowsky M, Le Callet P (2012) Analysis and improvement of a paired comparison method in the application of 3DTV subjective experiment. In: 2012 19th IEEE international conference on image processing. IEEE, pp 629–632. doi:10.1109/ICIP.2012.6466938

  17. Li J, Barkowsky M, Le Callet P (2013) Subjective assessment methodology for preference of experience in 3DTV. In: IVMSP 2013. IEEE, pp 1–4. doi:10.1109/IVMSPW.2013.6611917

  18. Lin JY, Song R, Wu CH, Liu T, Wang H, Kuo CCJ (2015) MCL-V: a streaming video quality assessment database. J Vis Commun Image Represent 30:1–9

    Article  Google Scholar 

  19. Liu TJ, Lin W, Kuo CC, Jay Kuo CC (2013) Image quality assessment using multi-method fusion. IEEE Trans Image Process 22(5):1793–1807. doi:10.1109/TIP.2012.2236343

    Article  MathSciNet  MATH  Google Scholar 

  20. Ninassi A (2006) Pseudo no reference image quality metric using perceptual data hiding. Proc SPIE 3(1):60,570G–60,570G–12. doi:10.1117/12.650780

    Article  Google Scholar 

  21. Otto Dykstra J (1960) Rank analysis of incomplete block designs: a method of paired comparisons employing unequal repetitions on pairs. Biometrics 16(2):176–188

    Article  MATH  Google Scholar 

  22. Pinson M, Janowski L, Papir Z (2015) Video quality assessment [Subjective testing of entertainment scenes]. IEEE Signal Proc Mag 32(1):101–114. doi:10.1109/MSP.2013.2292535

    Article  Google Scholar 

  23. Pinson MH, Janowski L, Pepion R, Huynh-Thu Q, Schmidmer C, Corriveau P, Younkin A, Le Callet P, Barkowsky M, Ingram W (2012) The influence of subjects and environment on audiovisual subjective tests: An international study. IEEE J Select Topics Signal Process 6(6):640–651. doi:10.1109/JSTSP.2012.2215306

    Article  Google Scholar 

  24. Ponomarenko N, Ieremeiev O, Lukin V, Jin L, Egiazarian K, Astola J, Vozel B, Chehdi K, Carli M, Battisti F, Kuo CC (2013) A new color image database TID2013: Innovations and results. In: Advanced concepts for intelligent vision systems, lecture notes in computer science, vol 8192. Springer International Publishing, pp 402–413

  25. Ponomarenko N, Lukin V, Egiazarian K, Astola J, Carli M, Battisti F (2008) Color image database for evaluation of image quality metrics. In: 2008 IEEE 10th workshop on multimedia signal processing, pp 403–408, doi:10.1109/MMSP.2008.4665112

  26. Ponomarenko N, Lukin V, Zelensky A, Egiazarian K, Carli M, Battisti F (2009) Tid2008-a database for evaluation of full-reference visual quality assessment metrics. Adv Modern Radioelectron 10(4):30– 45

    Google Scholar 

  27. Sheikh H, Sabir M, Bovik A (2006) A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Trans Image Process 15(11):3440–3451. doi:10.1109/TIP.2006.881959

    Article  Google Scholar 

  28. Silverstein D, Farrell J (1998) Quantifying perceptual image quality. In: The society for imaging science and technology, pp 242–246

    Google Scholar 

  29. Song R, Ko H, Kuo CCJ (2015) MCL-3D: A database for stereoscopic image quality assessment using 2D-image-plus-depth source. J Inf Sci Eng 31(5):1593–1611

    Google Scholar 

  30. Xu Q, Huang Q, Jiang T, Yan B, Lin W, Yao Y (2012) HodgeRank on random graphs for subjective video quality assessment. IEEE Trans Multimed 14(3):844–857. doi:10.1109/TMM.2012.2190924

    Article  Google Scholar 

  31. Xu Q, Jiang T, Yao Y, Huang Q, Yan B, Lin W (2011) Random partial paired comparison for subjective video quality assessment via hodgerank. In: Proceedings of the 19th ACM international conference on Multimedia - MM ’11. ACM Press, New York, p 393, doi:10.1145/2072298.2072350

Download references

Acknowledgements

This work was supported by NSFC Grant No. 61401337, 61222101, the Programme of Introducing Talents of Discipline to Universities (111 Project), Grant No. B08038, Innovation ability support plan of Shaanxi Province, No.606160967047, and Key Laboratory of Infrared System Detection and Imaging Technology, Shanghai Institute of Technical Physics, Chinese Academy of Sciences.

Author information

Authors and Affiliations

  1. State Key Laboratory of ISN, Xidian University, Xi’an, 710071, China

    Rui Song, Yunsong Li, Yuan Jia & Yangli Wang

  2. Key Laboratory of Infrared System Detection and Imaging Technology, Shanghai Institute of Technical Physics, Chinese Academy of Sciences, Shanghai, 200083, People’s Republic of China

    Rui Song & Peng Rao

Authors
  1. Rui Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yunsong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuan Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yangli Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Peng Rao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rui Song.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Song, R., Li, Y., Jia, Y. et al. Efficient, robust and divisible paired comparison for subjective quality assessment. Multimed Tools Appl 77, 13597–13613 (2018). https://doi.org/10.1007/s11042-017-4977-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-4977-2

Keywords

Search

Navigation