Elsevier

Digital Signal Processing

Volume 91, August 2019, Pages 3-10
Digital Signal Processing

Feature comparison and analysis for new challenging research fields of image quality assessment

https://doi.org/10.1016/j.dsp.2018.12.002Get rights and content

Abstract

Nowadays, objective image quality assessment (IQA) issue becomes increasingly important for both the practical application and scientific research in digital image processing systems. In this paper, we conduct feature comparison and analysis on new challenging research fields including contrast-distorted, screen content, multiply-distorted, tone-mapped, Depth Image Based Rendering (DIBR) and authentically distorted IQA. We describe the performance criteria, design rationale, and benchmark databases for validating IQA metrics in accordance with human perceptions. Then we provide some important conclusions of feature selection in terms of different IQA problems. In this work, our goal is dominantly towards judging the robustness of mainstream features used in the IQA field, which can help current and emerging researchers for better and faster grasping the speciality of each of novel IQA directions reported.

Introduction

Over the past decades, considerable efforts have been done on image quality assessment (IQA) tasks based on various strategies such as subjective testing, statistical model, brain science, perceptual model, saliency detection and machine learning. IQA technologies can be mainly categorized into subjective assessment and objective assessment. Towards directly measuring image quality by the perception of the human visual system (HVS), the first subjective assessment is implemented based on a carefully-prepared testing environment, the organizers invite sufficient inexperienced observers to rank testing images in a randomized presentation order, and then yield the final mean opinion scores (MOSs) by averaging all the valid observers' scores after some necessary post-processing procedures such as outliers screening. However, subjective evaluation is usually time-consuming, cumbersome and expensive. As a result, the field of objective IQA has obtained significant progress in the last several years [1], [2].

Depending on the availability of lossless image, objective IQA algorithms can be further classified into three categories: full reference (FR) IQA, no-reference (NR) IQA and reduced-reference (RR) IQA. Among them, FR-IQA evaluates distorted images quality compared with the original image which is undistorted. RR-IQA measures only use part information from the reference image to assess distorted images quality. However, in most cases, the reference image (complete or partial) may not be available, so FR and RR IQA metrics are limited by the dependence of the reference image in practical applications. As thus, blind/NR-IQA metrics without the help of the original references are highly expected recently. During the past decade, great amount of blind image quality measures have been proposed. One type of NR-IQA algorithms works under the condition that the distortion type is known before. Typical distortion-specific quality measures are devoted to compression [3], [4], sharpness/blurriness [5], [6], contrast alteration [7], [8], sonar image IQA [13], [14], augmented reality IQA [15], etc. The other type of general-purpose blind IQA metrics concentrate on evaluating various types of distortion simultaneously.

There are several useful distortion-specific blind IQA measures [9], [10], [11], [12] and general-purpose blind IQA methods [16], [17], [18], [19]. The spectral and spatial sharpness (S3) metric was addressed in [11]. The slope of magnitude spectrum was used to measure the attenuation degree of high-frequency information. Total spatial variation and the slope of the magnitude spectrum was integrated by a weighted geometric method to give image sharpness scores. Gu et al. proposed a NR sharpness method in the autoregressive parameter space. The energy and contrast differences were first calculated in the locally estimated AR coefficients in a point-wise way, then the image sharpness score was obtained by percentile pooling [10]. Mittal et al. constructed a collection of features and fitted them to a multivariate Gaussian model (MVG) to predict the quality of an image [19]. BRISQUE [17] used spatial natural scene statistics of locally normalized luminance coefficients to quantify possible losses of naturalness in the distorted image. NFERM [18] constructed a collection of features based on free energy and classical human visual system to evaluate the image quality.

In recent years, deep learning technology has made remarkable achievements in many application fields. And the deep learning technology is also used in image quality evaluation methods [20], [21], [22], [23], [24]. Hou et al. trained a discriminative deep model to classify features, then converted the qualitative labels into scores [22]. Jia et al. proposed a blind IQA method using deep Convolutional Neural Network (CNN) to extract features of high dynamic range images,then human visual system is introduced to CNN. This blind IQA method competitive with full reference IQA methods in HDR blind IQA experiment [23]. Kim et al. proposed IQA model using Convolutional Neural Network to seek the optimal visual weight based on database without prior knowledge of the human visual system [24]. And this method delivers the state-of-the-art performance among FR IQA models.

This paper first introduces new challenging benchmark databases for validating IQA metrics in accordance with human perceptions. Second, we compare the performance of the existing algorithms, analyze their advantages and disadvantages, and emphasize the difficulties for the existing algorithms.

Section snippets

Overview of IQA experimental protocol

To compare the performance of different IQA metrics, performance criterion is necessary. In light of the video quality experts group (VQEG)'s suggestion, commonly used criteria were mentioned as follows [25]: Spearman's rank ordered correlation coefficient (SRCC) is adopted to evaluate the prediction monotonicity; Kendall's rank correlation coefficient (KRCC) is another metric used to evaluate the prediction monotonicity; Pearson's (linear) correlation coefficient (PLCC) is used to evaluate the

Conclusion

IQA has become an important topic. Because it can be widely used as an optimization criterion for various IQA metrics in image processing and computer vision, such as image/video compression, restoration, denoising and enhancement. In this paper, we have conducted feature comparison and analysis on some new challenging research fields including contrast-distorted, screen content, multiply-distorted, DIBR, authentically distorted and tone-mapped IQA. Some more effective and robust features have

Acknowledgements

This work was supported in part by the Natural Science Research of Jiangsu Higher Education Institutions of China under Grant 18KJB52, and the Qinglan Project of Jiangsu, 2018.

Lijuan Tang received the master's and PhD degrees from the University of Mining and Technology, China, in 2008 and 2018, respectively. She is an Associate Professor in the School of Information and Electrical Engineering, Jiangsu Vocational College of Business. Her research interests include image quality assessment and visual perceptual modeling.

References (68)

  • K. Gu et al.

    Subjective and objective quality assessment for images with contrast change

  • K. Gu et al.

    The analysis of image contrast: from quality assessment to automatic enhancement

    IEEE Trans. Cybern.

    (Jan. 2016)
  • P.V. Vu et al.

    A fast wavelet-based algorithm for global and local image sharpness estimation

    IEEE Signal Process. Lett.

    (Jul. 2012)
  • K. Gu et al.

    No-reference image sharpness assessment in autoregressive parameter space

    IEEE Trans. Image Process.

    (Oct. 2015)
  • T. Vu et al.

    S3: a spectral and spatial measure of local perceived sharpness in natural images

    IEEE Trans. Image Process.

    (May 2012)
  • K. Gu et al.

    Highly efficient picture-based prediction of PM2.5 concentration

    IEEE Trans. Ind. Electron.

    (2019)
  • W. Chen et al.

    Partial-reference sonar image quality assessment for underwater transmission

    IEEE Trans. Aerosp. Electron. Syst.

    (2018)
  • W. Chen et al.

    Statistical and structural information backed full-reference quality measure of compressed sonar images

    IEEE Trans. Circu. Sys. Video Tech.

    (2019)
  • J. Puig et al.

    Towards an efficient methodology for evaluation of quality of experience in augmented reality

  • A. Mittal et al.

    No-reference image quality assessment in the spatial domain

    IEEE Trans. Image Process.

    (Dec. 2012)
  • K. Gu et al.

    Using free energy principle for blind image quality assessment

    IEEE Trans. Multimed.

    (Jan. 2015)
  • A. Mittal et al.

    Making a completely blind image quality analyzer

    IEEE Signal Process. Lett.

    (Mar. 2013)
  • K. Gu, G. Zhai, X. Yang, W. Zhang, Deep learning network for blind image quality assessment, in: Proc. IEEE Int. Conf....
  • J. Li et al.

    No-reference image quality assessment using Prewitt magnitude based on convolutional neural networks

    Signal Image Video Process.

    (Apr. 2016)
  • W. Hou et al.

    Blind image quality assessment via deep learning

    IEEE Trans. Neural Netw. Learn. Syst.

    (2017)
  • S. Jia et al.

    Blind high dynamic range image quality assessment using deep learning

  • J. Kim et al.

    Deep learning of human visual sensitivity in image quality assessment framework

  • VQEG

    Final report from the video quality experts group on the validation of objective models of video quality assessment

  • D. Jayaraman, A. Mittal, A.K. Moorthy, A.C. Bovik, Objective quality assessment of multiply distorted images, in: Proc....
  • K. Gu et al.

    Hybrid no-reference quality metric for singly and multiply distorted images

    IEEE Trans. Broadcast.

    (2014)
  • T. Virtanen et al.

    CID2013: a database for evaluating no-reference image quality assessment algorithms

    IEEE Trans. Image Process.

    (Jan. 2015)
  • D. Ghadiyaram et al.

    Massive online crowdsourced study of subjective and objective picture quality

    IEEE Trans. Image Process.

    (2016)
  • H. Yeganeh et al.

    Objective quality assessment of tone-mapped images

    IEEE Trans. Image Process.

    (2013)
  • K. Gu et al.

    Blind quality assessment of tone-mapped images via analysis of information, naturalness and structure

    IEEE Trans. Multimed.

    (Jan. 2016)
  • Cited by (0)

    Lijuan Tang received the master's and PhD degrees from the University of Mining and Technology, China, in 2008 and 2018, respectively. She is an Associate Professor in the School of Information and Electrical Engineering, Jiangsu Vocational College of Business. Her research interests include image quality assessment and visual perceptual modeling.

    Kezheng Sun received the B.E. degree from Jiangsu Normal University, China, in 2005, and the master's degree with the Jiangsu University, China, in 2012. He is an Associate Professor in the School of Information and Electrical Engineering, Jiangsu Vocational College of Business. His research interests include image quality assessment and image processing.

    Jing Bi received the Ph.D. degree from Northeastern University, Shenyang, China. Dr. Bi was IEEE Senior Member and the recipient of the 2009 IBM Ph.D. Fellowship Award. Her research interests include cloud computing, dynamic resource optimization, parallel and distributed computing, big data analysis and machine learning and energy-efficient computing and communication.

    Jiheng Wang received the M.Math. degree in Statistics-Computing from the University of Waterloo, Waterloo, ON, Canada, in 2011, and the Ph.D. degree in electrical and computer engineering from the University of Waterloo, in 2016. His research interests include biomedical image and signal processing, image and video quality assessment, perceptual video coding, and deep learning.

    View full text