Linking visual saliency deviation to image quality degradation: A saliency deviation-based image quality index

Highlights

• Visual quality degradation is linked and modelled with saliency deviation for the first time.

• The proposed metric avoids ad-hoc saliency weighting and corresponding computational cost.

• Both saliency and quality information of the test stimuli are provided.

Abstract

Advances in image quality research have shown the benefits of modeling functional components of the human visual system in image quality metrics. Recently, visual saliency, an important aspect of the human visual system, is increasingly investigated in relation to visual quality perception. Existing studies have showed that incorporating visual saliency leads to improved performance of image quality metrics. However, current applications of visual saliency in image quality metrics mainly focus on the extension of a specific metric with a specific visual saliency model. Issues regarding the optimal use of visual saliency in image quality metrics remain. Psychophysical experiments conducted in the literature have revealed that visual artifacts occurring in an image can change fixation deployment relative to that of the image without distortion. As such, instead of using saliency models as add-ons to image quality metrics, we explored the approach of directly assessing image quality by measuring the visual saliency deviation triggered by visual artifacts. We first analyzed the relationship between visual saliency deviation and image quality degradation on the basis of a large-scale eye-tracking dataset. A saliency deviation-based image quality index was then devised. Experimental results showed that the proposed metric features high prediction accuracy and relatively low computational cost.

Introduction

In current image processing pipeline, image signals are subject to visual distortions in the phases including data acquisition, compression, transmission and reproduction [1]. The perceived visual artifacts occurred in images could affect the visual experience of end users and may even cause interpretation errors in visual inspection tasks. Therefore, computational image quality metrics (IQMs) have emerged as an important component in modern imaging systems for the automatic assessment of perceived image quality [2]. Nowadays, IQMs designed for different purposes are widely available for a broad range of applications, e.g., for fine tuning image and video processing pipelines, evaluating image and video enhancement algorithms, and quality monitoring and control of displays. Depending on to what extent they utilize the undistorted reference, IQMs can be classified into full-reference (FR), reduced-reference (RR) and no-reference (NR) metrics. In this paper, we focused on devising a full-reference metric where the distortion-free image is required.

As a traditional pixel-wise fidelity metric, peak signal-to-noise ratio (PSNR) directly measures the quality as the intensity change of image signals and it does not well correlate with the human quality perception. Over the past decades, substantial progress has been made on the exploring and modeling the functionality aspects of the HVS. Advances in human vision research have allowed us to integrate HVS features into the design of the so-called perceptual-driven metrics [3], [4], [5], [6], [7]. PSNR-HA and PSNR-HMA [3] were proposed by taking into account the mean shift sensitivity and contrast sensitivity feature of the HVS when calculating the PSNR. Visual Signal-to Noise Ratio (VSNR) [4] was designed by weighting the Signal-to-noise ratio (SNR) with the wavelet-based model of visual masking. CSV [5] models the image quality using contrast sensitivity of retinal cells along with the differences of color and structural information between original and distorted images. Noise Quality Measure (NQM) [6] was proposed considering the contrast sensitivity, local luminance, contrast interaction between spatial frequencies and contrast masking effects of the HVS. Most Apparent Distortion (MAD) [7] models the local luminance and contrast masking features of the HVS and estimates image quality using variable strategies depending on the strength of distortions.

Instead of simulating specific features of the HVS, another kind of IQMs, namely the signal-driven IQMs concern the overall functionality of the HVS and focus on the analysis of image statistics and distortions [8], [9], [10], [11], [12]. By using the observation that HVS is highly adapted to extract structural information from visual scenes, Structure SIMilarity Index (SSIM) [8] quantifies image quality using luminance, contrast and structure features. It was further extended to an information weighted (IW-SSIM) [9] version with the information content pooling strategy. Similarly, based on the observation that visual system focus on structures and segments, Perceptual SIMilarity (PerSIM) metric [10] estimates image quality based on Log features and chroma similarity. Feature Similarity Index (FSIM) [11] utilizes the phase congruency and gradient to calculate the local distortions. Visual Information Fidelity (VIF) metric [12]estimates image quality by quantifies how much of the information present in the reference image can be extracted from the distorted image.

In recent years, designing learning-based IQMs have gained popularity with the development of artificial intelligence. The basic strategy of this kind is to learn a regression model that maps the image features to a quality score. Various regression methods, including support vector regression (SVR) [13], neural network [14], [15], random forest regression [16] and deep learning framework [17], [18], [19] are widely used for model learning. In [13], inspired by the free-energy-based brain theory, structural-related features and HVS-inspired features are extracted. A quality estimator is then derived using the support vector machine based regression module. In [14], authors proposed a compact multi-task convolutional neural network to simultaneously estimate image quality and identify distortion type in the image. In [15], researchers developed an IQM based on phase, entropy and gradient features fed to a regression neural network. In [16], authors represented an image using a feature vector and adopted a random forest to train the regression model which maps feature vector to the subjective score. In [17], a deep learning network was designed to classify an image into five quality grades and then convert the these five qualitative labels to numerical scores. In [18], author developed a convolutional networks for image quality assessment combining the learning and regression as a complete optimization process. In [20], an Unsupervised Image Quality Estimation (UNIQUE) metric was proposed based on comparing the monotonicity of sparse representations learned from generic image databases.

Recent research in [21] have showed that current visual quality modeling is lack of sophistication when dealing with real-world complexity, thus making image quality assessment a continuous research topic. Improvement the reliability of IQMs, therefore, rests with the further understanding of the HVS as well as modeling its aspects that are relevant to visual quality perception. To this end, one of the growing trends in image quality research is to investigate how visual attention impact the image quality judgement. Visual attention is a mechanism of the HVS that drives the selection of visual information in a visual scene [22]. The bottom-up, stimulus-driven part of the attentional mechanism is often referred as visual saliency in computer vision community [23]. A computational saliency model generates a topographic map to represent the conspicuousness of scene locations [24].

The underlying reason to incorporate saliency information in IQMs is that visual artifacts occurred in salient regions are considered to have higher impact on image quality perception than those in the non-salient regions [25]. As such, research in the literature mainly focused on the extension of a specific metric with a specific visual saliency model. The most common method adopted is to weight local distortions with local saliency [26], [27], [28], [29], resulting in a so-called saliency-weighted metric. For example, Moorthy et al. [26] integrated an existing saliency model called GAFFE [30] in the SSIM metric, achieving an improvement of 1% to 4% in terms of metric performance. In [27], an NR metric for evaluating the JPEG2000 compression artifacts was designed using the saliency model proposed in [31]. Experimental results demonstrated that the saliency information yielded significant improvements in the performance of the IQM without saliency. It should be noted this saliency weighting approach implicitly assumes that the attentional mechanism of the HVS functions in a post-processing manner when assessing image quality. The complicated interaction between visual saliency and image quality is, however, neglected. The added value of visual saliency in IQMs strongly depends on the saliency model, the IQM, and the characteristics of the test image [32]. Therefore, the obtained added value is often limited and even to be negative [33]. Moreover, this saliency weighting approach requires extra computational cost for generating saliency maps and refining the importance of local distortions. It may further limit the deployment of these saliency-weighted metric in any real-time applications. Therefore, exploring perceptually optimized ways to utilize saliency information in IQMs is worth further investigation.

In this paper, rather than using saliency as an add-on to extend existing IQMs, we explored the approach of modeling image quality by quantifying the saliency deviation induced by visual artifacts. We hypothesized that saliency deviation caused by visual artifacts may be used as a proxy for the visual quality degradation also caused the artifacts. To validate this hypothesis, we first investigated the relationship between saliency deviation and quality degradation on the basis of a large-scale eye-tracking database. We then considered how the findings can be used to propose an IQM, resulting in a saliency deviation-based IQMs which get rid of the disadvantages existed in the traditional saliency-weighting approach.