CONFERENCE PROCEEDINGS
Advanced Search
Home > Proceedings > Volume 9396 > Article
Paper
8 February 2015 Alternative performance metrics and target values for the CID2013 database
T. Virtanen, Mikko Nuutinen, J. Radun, Tuomas M. Leisti, Jukka P. Häkkinen
Author Affiliations +
Proceedings Volume 9396, Image Quality and System Performance XII; 93960Q (2015) https://doi.org/10.1117/12.2079100
Event: SPIE/IS&T Electronic Imaging, 2015, San Francisco, California, United States
Abstract
An established way of validating and testing new image quality assessment (IQA) algorithms have been to compare how well they correlate with subjective data on various image databases. One of the most common measures is to calculate linear correlation coefficient (LCC) and Spearman’s rank order correlation coefficient (SROCC) against the subjective mean opinion score (MOS). Recently, databases with multiply distorted images have emerged 1,2. However with multidimensional stimuli, there is more disagreement between observers as the task is more preferential than that of distortion detection. This reduces the statistical differences between image pairs. If the subjects cannot distinguish a difference between some of the image pairs, should we demand any better performance with IQA algorithms? This paper proposes alternative performance measures for the evaluation of IQA’s for the CID2013 database. One proposed alternative performance measure is root-mean-square-error (RMSE) value for the subjective data as a function of the number of observers. The other alternative performance measure is the number of statistical differences between image pairs. This study shows that after 12 subjects the RMSE value saturates around the level of three, meaning that a target RMSE value for an IQA algorithm for CID2013 database should be three. In addition, this study shows that the state-of-the-art IQA algorithms found the better image from the image pairs with a probability of 0.85 when the image pairs with statistically significant differences were taken into account.
© (2015) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.
Citation Download Citation
T. Virtanen, Mikko Nuutinen, J. Radun, Tuomas M. Leisti, and Jukka P. Häkkinen "Alternative performance metrics and target values for the CID2013 database", Proc. SPIE 9396, Image Quality and System Performance XII, 93960Q (8 February 2015); https://doi.org/10.1117/12.2079100
ACCESS THE FULL ARTICLE
ORGANIZATIONAL
Sign in with credentials provided by your organization.
INSTITUTIONAL
Select your institution to access the SPIE Digital Library.
SELECT YOUR INSTITUTION
PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.
PERSONAL SIGN IN
No SPIE Account? Create one
PURCHASE THIS CONTENT
SUBSCRIBE TO DIGITAL LIBRARY
50 downloads per 1-year subscription
Members: $195
Non-members: $335 ADD TO CART
25 downloads per 1 - year subscription
Members: $145
Non-members: $250 ADD TO CART
PURCHASE SINGLE ARTICLE
Includes PDF, HTML & Video, when available
Members: $17.00
Non-members: $21.00 ADD TO CART
PROCEEDINGS
 11 PAGES
DOWNLOAD PAPER SAVE TO MY LIBRARY
GET CITATION
Advertisement
Advertisement
RIGHTS & PERMISSIONS
Get copyright permission  Get copyright permission on Copyright Marketplace
KEYWORDS
Image quality
Databases
Data modeling
Image enhancement
Detection and tracking algorithms
Molybdenum
Cameras
RELATED CONTENT
Subscribe to Digital Library
Receive Erratum Email Alert
Back to Top