Skip to main content
Log in

A new method for evaluating the subjective image quality of photographs: dynamic reference

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The Dynamic Reference (DR) method has been developed for subjective image quality experiments in which original or undistorted images are unavailable. The DR method creates reference image series from test images. Reference images are presented to observers as a slide show prior to evaluating their quality. As the observers view the set of reference images, they determine the overall variation in quality within the set of test images. This study compared the performance of the DR method to that of the standardized absolute category rating (ACR) and paired comparison (PC) methods. We measured the performance of each method in terms of time effort and discriminability. The results showed that the DR method is faster than the PC method and more accurate than the ACR method. The DR method is especially suitable for experiments that require highly accurate results in a short time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. http://www.helsinki.fi/psychology/groups/visualcognition

References

  1. Bagiella E, Sloan R, Heitjan D (2000) Mixed-effects models in psychophysiology. Psychophysiology 37(1):13–20

    Article  Google Scholar 

  2. Bland M (1995) An introduction to medical statistics, 2nd edn. Oxford Medical Publications

  3. Blin J (2006) New quality evaluation method suited to multimedia context: samviq. In: Proceeding of the 2nd International Workshop on Video Processing and Quality Metrics, (VPQM)

  4. Engelke U, Maeder A, Zepernick HJ (2012) Human observer confidence in image quality assessment. Signal Process Image Commun 27(9):935–947

    Article  Google Scholar 

  5. Greenwood P, Nikulin M (1996) A guide to chi-squared testing. A Wiley-Interscience Publication. Wiley. http://books.google.fi/books?id=bc8zfQSKOwIC

  6. I3A (2007) CPIQ initiative phase 1 white paper fundamentals and review of considered test methods

  7. ISO 20462 (2012) Psychophysical experimental methods for estimating image quality – Part 3: Quality ruler method

  8. ITU-R, Rec.BT.1788 (2007) Methodology for the subjective assessment of video quality in multimedia applications

  9. ITU-R BT.500 (2012) Methodology for the subjective assessment of the quality of television pictures

  10. ITU-T Rec. P. 910 (2008) Subjective video quality assessment methods for multimedia applications

  11. Jianping Z, Glotzbach J (2007) Image pipeline tuning for digital cameras. In: Proceedings of IEEE International Symposium on Consumer Electronics, (ISCE), pp 1–4

  12. Jin E, Keelan B (2010) Slider-adjusted softcopy ruler for calibrated image quality assessment. J Electron Imaging 19(1):011,009

    Article  Google Scholar 

  13. Kuang J, Yamaguchi H, Liu C, Johnson G., Fairchild M (2007) Evaluating HDR rendering algorithms. ACM Trans Appl Percept 4(2)

  14. Leisti T, Radun J, Virtanen T, Halonen R, Nyman G (2009) Subjective experience of image quality: attributes, definitions, and decision making of subjective image quality. In: Proceedings of the SPIE 7242, Image Quality and System Performance VI, pp 72,420D–72,420D–9

  15. Mantiuk R, Tomaszewska A, Mantiuk R (2012) Comparison of four subjective methods for image quality assessment. Comput Graph Forum 31(8):2478–2491

    Article  Google Scholar 

  16. Nikkanen J, Gerasimow T, Kong L (2008) Subjective effects of white-balancing errors in digital photography. Opt Eng 47(11):113,201. doi:10.1117/1.3013232

    Article  Google Scholar 

  17. Nuutinen M, Orenius O, Säämänen T, Oittinen P (2012) A framework for measuring sharpness in natural images captured by digital cameras base on reference image and local areas. EURASIP J Image Video Process 2012(13640):1–15

    Google Scholar 

  18. Nuutinen M, Virtanen T, Oittinen P (2012) Features for predicting quality of images captured by digital cameras. In: Proceedings of IEEE International Symposium on Multimedia, pp 165–168

  19. Petit J, Mantiuk R (2013) Assessment of video tone-mapping: are cameras’ s-shaped tone-curves good enough?. J Vis Commun Image Represent 24(7):1020–1030

    Article  Google Scholar 

  20. Ponomarenko V, Lukin V, Zelensky A, Egiazarian K, Carli M, Battisti F (2009) Tid2008 a database for evaluating of full-reference visual quality assessment metrics. Adv Modern Radioelectron 10:30–45

    Google Scholar 

  21. Radun J, Leisti T, Virtanen T, Häkkinen J, Vuori T, Nyman G (2010) Evaluating the multivariate visual quality performance of image-processing components. ACM Trans Appl Percept 7(3):16:1–16:16

    Article  Google Scholar 

  22. Ramanath R, Snyder W, Yoo Y, Drew M (2005) Color image processing pipeline. IEEE Signal Proc Mag 22(1):34–43

    Article  Google Scholar 

  23. Redi J, Liu H, Alers H, Zunino R, Heynderickx I (2010) Comparing subjective image quality measurement methods for the creation of public databases. In: Proceedings of the SPIE 7529, Image Quality and System Performance VII, pp 752,903–752,903–11

  24. Sheikh HR, Sabir MF, Bovik AC (2006) A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Trans Image Process 15(11):3440–3451

    Article  Google Scholar 

  25. Silverstein D, Farrell J (2001) Efficient method for paired comparison. Jof Electron Imaging 10(2):394–398

    Article  Google Scholar 

  26. Taichi K., Kazuhisa Y., Takanori H. (2012) Performance comparison of subjective assessment methods for 3d video quality. In: Proceedings of International Workshop on Quality of Multimedia Experience, (QoMEx), pp 218–223

  27. Teunissen K (1996) The validity of CCIR quality indicators along a graphical scale. SMPTE Motion Imaging J 105(3):144–149

    Google Scholar 

  28. Ween B, Kristoffersen D, Hamilton G, Olsen D (2005) Image quality preferences among radiographers and radiologists. a conjoint analysis. Radiography 11(3):191–197

    Article  Google Scholar 

  29. Winkler S (2009) On the properties of subjective ratings in video quality experiments. In: Proceedings of international workshop on quality of multimedia experience, (QoMEx), pp 139–144

  30. Winkler S (2012) Analysis of public image and video databases for quality assessment. IEEE J Select Topics Signal Process 6(6):616–625

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Mikko Nuutinen.

Appendix

Appendix

Fig. 12
figure 12

Test images were captured from six scenes

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nuutinen, M., Virtanen, T., Leisti, T. et al. A new method for evaluating the subjective image quality of photographs: dynamic reference. Multimed Tools Appl 75, 2367–2391 (2016). https://doi.org/10.1007/s11042-014-2410-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-014-2410-7

Keywords

Navigation