Feature comparison and analysis for new challenging research fields of image quality assessment
Introduction
Over the past decades, considerable efforts have been done on image quality assessment (IQA) tasks based on various strategies such as subjective testing, statistical model, brain science, perceptual model, saliency detection and machine learning. IQA technologies can be mainly categorized into subjective assessment and objective assessment. Towards directly measuring image quality by the perception of the human visual system (HVS), the first subjective assessment is implemented based on a carefully-prepared testing environment, the organizers invite sufficient inexperienced observers to rank testing images in a randomized presentation order, and then yield the final mean opinion scores (MOSs) by averaging all the valid observers' scores after some necessary post-processing procedures such as outliers screening. However, subjective evaluation is usually time-consuming, cumbersome and expensive. As a result, the field of objective IQA has obtained significant progress in the last several years [1], [2].
Depending on the availability of lossless image, objective IQA algorithms can be further classified into three categories: full reference (FR) IQA, no-reference (NR) IQA and reduced-reference (RR) IQA. Among them, FR-IQA evaluates distorted images quality compared with the original image which is undistorted. RR-IQA measures only use part information from the reference image to assess distorted images quality. However, in most cases, the reference image (complete or partial) may not be available, so FR and RR IQA metrics are limited by the dependence of the reference image in practical applications. As thus, blind/NR-IQA metrics without the help of the original references are highly expected recently. During the past decade, great amount of blind image quality measures have been proposed. One type of NR-IQA algorithms works under the condition that the distortion type is known before. Typical distortion-specific quality measures are devoted to compression [3], [4], sharpness/blurriness [5], [6], contrast alteration [7], [8], sonar image IQA [13], [14], augmented reality IQA [15], etc. The other type of general-purpose blind IQA metrics concentrate on evaluating various types of distortion simultaneously.
There are several useful distortion-specific blind IQA measures [9], [10], [11], [12] and general-purpose blind IQA methods [16], [17], [18], [19]. The spectral and spatial sharpness (S3) metric was addressed in [11]. The slope of magnitude spectrum was used to measure the attenuation degree of high-frequency information. Total spatial variation and the slope of the magnitude spectrum was integrated by a weighted geometric method to give image sharpness scores. Gu et al. proposed a NR sharpness method in the autoregressive parameter space. The energy and contrast differences were first calculated in the locally estimated AR coefficients in a point-wise way, then the image sharpness score was obtained by percentile pooling [10]. Mittal et al. constructed a collection of features and fitted them to a multivariate Gaussian model (MVG) to predict the quality of an image [19]. BRISQUE [17] used spatial natural scene statistics of locally normalized luminance coefficients to quantify possible losses of naturalness in the distorted image. NFERM [18] constructed a collection of features based on free energy and classical human visual system to evaluate the image quality.
In recent years, deep learning technology has made remarkable achievements in many application fields. And the deep learning technology is also used in image quality evaluation methods [20], [21], [22], [23], [24]. Hou et al. trained a discriminative deep model to classify features, then converted the qualitative labels into scores [22]. Jia et al. proposed a blind IQA method using deep Convolutional Neural Network (CNN) to extract features of high dynamic range images,then human visual system is introduced to CNN. This blind IQA method competitive with full reference IQA methods in HDR blind IQA experiment [23]. Kim et al. proposed IQA model using Convolutional Neural Network to seek the optimal visual weight based on database without prior knowledge of the human visual system [24]. And this method delivers the state-of-the-art performance among FR IQA models.
This paper first introduces new challenging benchmark databases for validating IQA metrics in accordance with human perceptions. Second, we compare the performance of the existing algorithms, analyze their advantages and disadvantages, and emphasize the difficulties for the existing algorithms.
Section snippets
Overview of IQA experimental protocol
To compare the performance of different IQA metrics, performance criterion is necessary. In light of the video quality experts group (VQEG)'s suggestion, commonly used criteria were mentioned as follows [25]: Spearman's rank ordered correlation coefficient (SRCC) is adopted to evaluate the prediction monotonicity; Kendall's rank correlation coefficient (KRCC) is another metric used to evaluate the prediction monotonicity; Pearson's (linear) correlation coefficient (PLCC) is used to evaluate the
Conclusion
IQA has become an important topic. Because it can be widely used as an optimization criterion for various IQA metrics in image processing and computer vision, such as image/video compression, restoration, denoising and enhancement. In this paper, we have conducted feature comparison and analysis on some new challenging research fields including contrast-distorted, screen content, multiply-distorted, DIBR, authentically distorted and tone-mapped IQA. Some more effective and robust features have
Acknowledgements
This work was supported in part by the Natural Science Research of Jiangsu Higher Education Institutions of China under Grant 18KJB52, and the Qinglan Project of Jiangsu, 2018.
Lijuan Tang received the master's and PhD degrees from the University of Mining and Technology, China, in 2008 and 2018, respectively. She is an Associate Professor in the School of Information and Electrical Engineering, Jiangsu Vocational College of Business. Her research interests include image quality assessment and visual perceptual modeling.
References (68)
- et al.
Perceptual visual quality metrics: a survey
J. Vis. Commun. Image Represent.
(2011) - et al.
No-reference image quality assessment based on spatial and spectral entropies
Signal Process. Image Commun.
(Sep. 2014) - et al.
Learning a blind quality evaluation engine of screen content images
Neurocomputing
(Jul. 2016) - et al.
Image quality assessment based on multi-scale representation of structure
Digit. Signal Process.
(2014) - et al.
Blind quality index for camera images with natural scene statistics and patch-based sharpness assessment
J. Vis. Commun. Image Represent.
(2016) - et al.
A fast reliable image quality predictor by fusing micro- and macro-structures
IEEE Trans. Ind. Electron.
(May 2017) - et al.
Referenceless measure of blocking artifacts by Tchebichef kernel analysis
IEEE Signal Process. Lett.
(2014) - et al.
Unified blind quality assessment of compressed natural, graphic and screen content images
IEEE Trans. Image Process.
(2017) - et al.
No-reference image sharpness assessment in autoregressive parameter space
IEEE Trans. Image Process.
(Oct. 2015) - et al.
No-reference image blur assessment based on discrete orthogonal moments
IEEE Trans. Cybern.
(Jan. 2016)
Subjective and objective quality assessment for images with contrast change
The analysis of image contrast: from quality assessment to automatic enhancement
IEEE Trans. Cybern.
A fast wavelet-based algorithm for global and local image sharpness estimation
IEEE Signal Process. Lett.
No-reference image sharpness assessment in autoregressive parameter space
IEEE Trans. Image Process.
S3: a spectral and spatial measure of local perceived sharpness in natural images
IEEE Trans. Image Process.
Highly efficient picture-based prediction of PM2.5 concentration
IEEE Trans. Ind. Electron.
Partial-reference sonar image quality assessment for underwater transmission
IEEE Trans. Aerosp. Electron. Syst.
Statistical and structural information backed full-reference quality measure of compressed sonar images
IEEE Trans. Circu. Sys. Video Tech.
Towards an efficient methodology for evaluation of quality of experience in augmented reality
No-reference image quality assessment in the spatial domain
IEEE Trans. Image Process.
Using free energy principle for blind image quality assessment
IEEE Trans. Multimed.
Making a completely blind image quality analyzer
IEEE Signal Process. Lett.
No-reference image quality assessment using Prewitt magnitude based on convolutional neural networks
Signal Image Video Process.
Blind image quality assessment via deep learning
IEEE Trans. Neural Netw. Learn. Syst.
Blind high dynamic range image quality assessment using deep learning
Deep learning of human visual sensitivity in image quality assessment framework
Final report from the video quality experts group on the validation of objective models of video quality assessment
Hybrid no-reference quality metric for singly and multiply distorted images
IEEE Trans. Broadcast.
CID2013: a database for evaluating no-reference image quality assessment algorithms
IEEE Trans. Image Process.
Massive online crowdsourced study of subjective and objective picture quality
IEEE Trans. Image Process.
Objective quality assessment of tone-mapped images
IEEE Trans. Image Process.
Blind quality assessment of tone-mapped images via analysis of information, naturalness and structure
IEEE Trans. Multimed.
Cited by (0)
Lijuan Tang received the master's and PhD degrees from the University of Mining and Technology, China, in 2008 and 2018, respectively. She is an Associate Professor in the School of Information and Electrical Engineering, Jiangsu Vocational College of Business. Her research interests include image quality assessment and visual perceptual modeling.
Kezheng Sun received the B.E. degree from Jiangsu Normal University, China, in 2005, and the master's degree with the Jiangsu University, China, in 2012. He is an Associate Professor in the School of Information and Electrical Engineering, Jiangsu Vocational College of Business. His research interests include image quality assessment and image processing.
Jing Bi received the Ph.D. degree from Northeastern University, Shenyang, China. Dr. Bi was IEEE Senior Member and the recipient of the 2009 IBM Ph.D. Fellowship Award. Her research interests include cloud computing, dynamic resource optimization, parallel and distributed computing, big data analysis and machine learning and energy-efficient computing and communication.
Jiheng Wang received the M.Math. degree in Statistics-Computing from the University of Waterloo, Waterloo, ON, Canada, in 2011, and the Ph.D. degree in electrical and computer engineering from the University of Waterloo, in 2016. His research interests include biomedical image and signal processing, image and video quality assessment, perceptual video coding, and deep learning.