Abstract
The Dynamic Reference (DR) method has been developed for subjective image quality experiments in which original or undistorted images are unavailable. The DR method creates reference image series from test images. Reference images are presented to observers as a slide show prior to evaluating their quality. As the observers view the set of reference images, they determine the overall variation in quality within the set of test images. This study compared the performance of the DR method to that of the standardized absolute category rating (ACR) and paired comparison (PC) methods. We measured the performance of each method in terms of time effort and discriminability. The results showed that the DR method is faster than the PC method and more accurate than the ACR method. The DR method is especially suitable for experiments that require highly accurate results in a short time.
Similar content being viewed by others
References
Bagiella E, Sloan R, Heitjan D (2000) Mixed-effects models in psychophysiology. Psychophysiology 37(1):13–20
Bland M (1995) An introduction to medical statistics, 2nd edn. Oxford Medical Publications
Blin J (2006) New quality evaluation method suited to multimedia context: samviq. In: Proceeding of the 2nd International Workshop on Video Processing and Quality Metrics, (VPQM)
Engelke U, Maeder A, Zepernick HJ (2012) Human observer confidence in image quality assessment. Signal Process Image Commun 27(9):935–947
Greenwood P, Nikulin M (1996) A guide to chi-squared testing. A Wiley-Interscience Publication. Wiley. http://books.google.fi/books?id=bc8zfQSKOwIC
I3A (2007) CPIQ initiative phase 1 white paper fundamentals and review of considered test methods
ISO 20462 (2012) Psychophysical experimental methods for estimating image quality – Part 3: Quality ruler method
ITU-R, Rec.BT.1788 (2007) Methodology for the subjective assessment of video quality in multimedia applications
ITU-R BT.500 (2012) Methodology for the subjective assessment of the quality of television pictures
ITU-T Rec. P. 910 (2008) Subjective video quality assessment methods for multimedia applications
Jianping Z, Glotzbach J (2007) Image pipeline tuning for digital cameras. In: Proceedings of IEEE International Symposium on Consumer Electronics, (ISCE), pp 1–4
Jin E, Keelan B (2010) Slider-adjusted softcopy ruler for calibrated image quality assessment. J Electron Imaging 19(1):011,009
Kuang J, Yamaguchi H, Liu C, Johnson G., Fairchild M (2007) Evaluating HDR rendering algorithms. ACM Trans Appl Percept 4(2)
Leisti T, Radun J, Virtanen T, Halonen R, Nyman G (2009) Subjective experience of image quality: attributes, definitions, and decision making of subjective image quality. In: Proceedings of the SPIE 7242, Image Quality and System Performance VI, pp 72,420D–72,420D–9
Mantiuk R, Tomaszewska A, Mantiuk R (2012) Comparison of four subjective methods for image quality assessment. Comput Graph Forum 31(8):2478–2491
Nikkanen J, Gerasimow T, Kong L (2008) Subjective effects of white-balancing errors in digital photography. Opt Eng 47(11):113,201. doi:10.1117/1.3013232
Nuutinen M, Orenius O, Säämänen T, Oittinen P (2012) A framework for measuring sharpness in natural images captured by digital cameras base on reference image and local areas. EURASIP J Image Video Process 2012(13640):1–15
Nuutinen M, Virtanen T, Oittinen P (2012) Features for predicting quality of images captured by digital cameras. In: Proceedings of IEEE International Symposium on Multimedia, pp 165–168
Petit J, Mantiuk R (2013) Assessment of video tone-mapping: are cameras’ s-shaped tone-curves good enough?. J Vis Commun Image Represent 24(7):1020–1030
Ponomarenko V, Lukin V, Zelensky A, Egiazarian K, Carli M, Battisti F (2009) Tid2008 a database for evaluating of full-reference visual quality assessment metrics. Adv Modern Radioelectron 10:30–45
Radun J, Leisti T, Virtanen T, Häkkinen J, Vuori T, Nyman G (2010) Evaluating the multivariate visual quality performance of image-processing components. ACM Trans Appl Percept 7(3):16:1–16:16
Ramanath R, Snyder W, Yoo Y, Drew M (2005) Color image processing pipeline. IEEE Signal Proc Mag 22(1):34–43
Redi J, Liu H, Alers H, Zunino R, Heynderickx I (2010) Comparing subjective image quality measurement methods for the creation of public databases. In: Proceedings of the SPIE 7529, Image Quality and System Performance VII, pp 752,903–752,903–11
Sheikh HR, Sabir MF, Bovik AC (2006) A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Trans Image Process 15(11):3440–3451
Silverstein D, Farrell J (2001) Efficient method for paired comparison. Jof Electron Imaging 10(2):394–398
Taichi K., Kazuhisa Y., Takanori H. (2012) Performance comparison of subjective assessment methods for 3d video quality. In: Proceedings of International Workshop on Quality of Multimedia Experience, (QoMEx), pp 218–223
Teunissen K (1996) The validity of CCIR quality indicators along a graphical scale. SMPTE Motion Imaging J 105(3):144–149
Ween B, Kristoffersen D, Hamilton G, Olsen D (2005) Image quality preferences among radiographers and radiologists. a conjoint analysis. Radiography 11(3):191–197
Winkler S (2009) On the properties of subjective ratings in video quality experiments. In: Proceedings of international workshop on quality of multimedia experience, (QoMEx), pp 139–144
Winkler S (2012) Analysis of public image and video databases for quality assessment. IEEE J Select Topics Signal Process 6(6):616–625
Author information
Authors and Affiliations
Corresponding author
Appendix
Appendix
Rights and permissions
About this article
Cite this article
Nuutinen, M., Virtanen, T., Leisti, T. et al. A new method for evaluating the subjective image quality of photographs: dynamic reference. Multimed Tools Appl 75, 2367–2391 (2016). https://doi.org/10.1007/s11042-014-2410-7
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-014-2410-7