Abstract
The scale of a database is important for machine learning based image and video quality assessment. Nevertheless, it is greatly limited by the subjective test method. Among the various methods, Paired Comparison(PC) is acknowledged as the most reliable one. However, the test duration grows with square of the number of samples. To solve the dilemma, we propose an improved paired comparison method in this paper. Three types of priori are incorporated to cut down the test duration, including the long-term priori as experience results condensed in existing quality metric, the short-term priori as the subjective scores calculated by the predecessor in ongoing session, and the dynamic priori as the previous decision made by the current assessor. Based on these priori knowledge, only indispensable part of decision is needed to be made by the assessor. Equivalent performance could be achieved in one-tenth of the time used in full paired comparison method. While it is robust to mis-click and divisible to expand the database to a large scale.
Similar content being viewed by others
References
Bosc E, Péepion R, Le Callet P, Köppel M, Ndjiki-Nya P, Pressigout M, Morin L (2011) Towards a new quality metric for 3-D synthesized view assessment. IEEE J Sel Top Signal Process 5(7):1332–1343. doi:10.1109/JSTSP.2011.2166245
Bradley RA, Terry ME (1952) Rank analysis of incomplete block designs: i. The method of paired comparisons. Biometrika 39(3):324–345
Chandler DM (2010) Most apparent distortion: full-reference image quality assessment and the role of strategy. J Electron Imaging 19(1):011,006. doi:10.1117/1.3267105
Chandler DM, Hemami SS (2007) VSNR: A wavelet-based visual signal-to-noise ratio for natural images. IEEE Trans Image Process 16(9):2284–98
Daniel WW (1990) Spearman rank correlation coefficient. In: Applied nonparametric statistics, 2nd edn. PWS-Kent, Boston, pp 358–365
Farrell J (2001) Efficient method for paired comparison. J Electron Imaging 10(2):394. doi:10.1117/1.1344187
Glickman ME, Jensen ST (2005) Adaptive paired comparison design. J Stat Plan Infer 127(1-2):279–293. doi:10.1016/j.jspi.2003.09.022
Huynh-Thu Q, Ghanbari M (2005) A comparison of subjective video quality assessment methods for low-bit rate and low-resolution video. In: Proceedings of the IASTED international conference on signal and image processing. ACTA Press, pp 70–76
ITU (1996) Recommendation ITU-T P.910, Subjective video quality assessment methods for multimedia applications. Tech. rep., ITU-T, Geneva
ITU (2002) BT.500-11: Methodology for the subjective assessment of the quality of television pictures. Tech. Rep. BT.500-11, International Telecommunication Union. http://www.itu.int/rec/R-REC-BT.500/en
ITU-R (2012) ITU-R BT.2022: General viewing conditions for subjective assessment of quality of SDTV and HDTV television pictures on flat panel displays. Tech. rep., International Telecommunication Union
ITU-R (2012) Recommendation ITU-R BT.2021: Subjective methods for the assessment of stereoscopic 3DTV systems. Tech. rep., International Telecommunication Union
Kendall MG (1938) A new measure of rank correlation. Biometrika 30(1/2):81
Lee JS, De Simone F, Ebrahimi T (2011) Subjective quality evaluation via paired comparison: application to scalable video coding. IEEE Trans Multimed 13 (5):882–893. doi:10.1109/TMM.2011.2157333
Lee JS, Goldmann L, Ebrahimi T (2012) Paired comparison-based subjective quality assessment of stereoscopic images. Multimed Tools Appl 67(1):31–48. doi:10.1007/s11042-012-1011-6
Li J, Barkowsky M, Le Callet P (2012) Analysis and improvement of a paired comparison method in the application of 3DTV subjective experiment. In: 2012 19th IEEE international conference on image processing. IEEE, pp 629–632. doi:10.1109/ICIP.2012.6466938
Li J, Barkowsky M, Le Callet P (2013) Subjective assessment methodology for preference of experience in 3DTV. In: IVMSP 2013. IEEE, pp 1–4. doi:10.1109/IVMSPW.2013.6611917
Lin JY, Song R, Wu CH, Liu T, Wang H, Kuo CCJ (2015) MCL-V: a streaming video quality assessment database. J Vis Commun Image Represent 30:1–9
Liu TJ, Lin W, Kuo CC, Jay Kuo CC (2013) Image quality assessment using multi-method fusion. IEEE Trans Image Process 22(5):1793–1807. doi:10.1109/TIP.2012.2236343
Ninassi A (2006) Pseudo no reference image quality metric using perceptual data hiding. Proc SPIE 3(1):60,570G–60,570G–12. doi:10.1117/12.650780
Otto Dykstra J (1960) Rank analysis of incomplete block designs: a method of paired comparisons employing unequal repetitions on pairs. Biometrics 16(2):176–188
Pinson M, Janowski L, Papir Z (2015) Video quality assessment [Subjective testing of entertainment scenes]. IEEE Signal Proc Mag 32(1):101–114. doi:10.1109/MSP.2013.2292535
Pinson MH, Janowski L, Pepion R, Huynh-Thu Q, Schmidmer C, Corriveau P, Younkin A, Le Callet P, Barkowsky M, Ingram W (2012) The influence of subjects and environment on audiovisual subjective tests: An international study. IEEE J Select Topics Signal Process 6(6):640–651. doi:10.1109/JSTSP.2012.2215306
Ponomarenko N, Ieremeiev O, Lukin V, Jin L, Egiazarian K, Astola J, Vozel B, Chehdi K, Carli M, Battisti F, Kuo CC (2013) A new color image database TID2013: Innovations and results. In: Advanced concepts for intelligent vision systems, lecture notes in computer science, vol 8192. Springer International Publishing, pp 402–413
Ponomarenko N, Lukin V, Egiazarian K, Astola J, Carli M, Battisti F (2008) Color image database for evaluation of image quality metrics. In: 2008 IEEE 10th workshop on multimedia signal processing, pp 403–408, doi:10.1109/MMSP.2008.4665112
Ponomarenko N, Lukin V, Zelensky A, Egiazarian K, Carli M, Battisti F (2009) Tid2008-a database for evaluation of full-reference visual quality assessment metrics. Adv Modern Radioelectron 10(4):30– 45
Sheikh H, Sabir M, Bovik A (2006) A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Trans Image Process 15(11):3440–3451. doi:10.1109/TIP.2006.881959
Silverstein D, Farrell J (1998) Quantifying perceptual image quality. In: The society for imaging science and technology, pp 242–246
Song R, Ko H, Kuo CCJ (2015) MCL-3D: A database for stereoscopic image quality assessment using 2D-image-plus-depth source. J Inf Sci Eng 31(5):1593–1611
Xu Q, Huang Q, Jiang T, Yan B, Lin W, Yao Y (2012) HodgeRank on random graphs for subjective video quality assessment. IEEE Trans Multimed 14(3):844–857. doi:10.1109/TMM.2012.2190924
Xu Q, Jiang T, Yao Y, Huang Q, Yan B, Lin W (2011) Random partial paired comparison for subjective video quality assessment via hodgerank. In: Proceedings of the 19th ACM international conference on Multimedia - MM ’11. ACM Press, New York, p 393, doi:10.1145/2072298.2072350
Acknowledgements
This work was supported by NSFC Grant No. 61401337, 61222101, the Programme of Introducing Talents of Discipline to Universities (111 Project), Grant No. B08038, Innovation ability support plan of Shaanxi Province, No.606160967047, and Key Laboratory of Infrared System Detection and Imaging Technology, Shanghai Institute of Technical Physics, Chinese Academy of Sciences.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Song, R., Li, Y., Jia, Y. et al. Efficient, robust and divisible paired comparison for subjective quality assessment. Multimed Tools Appl 77, 13597–13613 (2018). https://doi.org/10.1007/s11042-017-4977-2
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-017-4977-2