Saliency-induced reduced-reference quality index for natural scene and screen content images

Highlights

• We develop a saliency-induced reduced-reference (SIRR) IQA measure.

• In SIRR, image quality is described by the similarity between two images’ saliency maps.

• SIRR is a cross-content-type measure, which works efficiently for both natural scene images (NSIs) and screen content images (SCIs).

• Experimental results show that SIRR is comparable to state-of-the-art full-reference and reduced-reference IQA measures in NSIs, and it can outperform most competitors in SCIs.

Abstract

Massive content composed of both natural scene and screen content has been generated with the increasing use of wireless computing and cloud computing, which call for general image quality assessment (IQA) measures working for both natural scene images (NSIs) and screen content images (SCIs). In this paper, we develop a saliency-induced reduced-reference (SIRR) IQA measure for both NSIs and SCIs. Image quality and visual saliency are two widely studied and closely related research topics. Traditionally, visual saliency is often used as a weighting map in the final pooling stage of IQA. Instead, we detect visual saliency as a quality feature since different types and levels of degradation can strongly influence saliency detection. Image quality is described by the similarity between two images’ saliency maps. In SIRR, saliency is detected through a binary image descriptor called “image signature”, which significantly reduces the reference data. We perform extensive experiments on five large-scale NSI quality assessment databases including LIVE, TID2008, CSIQ, LIVEMD, CID2013, as well as two recently constructed SCI QA databases, i.e., SIQAD and QACS. Experimental results show that SIRR is comparable to state-of-the-art full-reference and reduced-reference IQA measures in NSIs, and it can outperform most competitors in SCIs. The most important is that SIRR is a cross-content-type measure, which works efficiently for both NSIs and SCIs. The MATLAB source code of SIRR will be publicly available with this paper.

Introduction

The quick advancements of transmission technologies have boosted various remote applications such as telecommuting and cloud computing, which bring massive computer-generated content called “screen content”. The so-called screen content has some distinctive characteristics different from natural scene because of the contained computer generated content, e.g., texts, icons, tables, graphics, etc. Those distinctive characteristics which sometimes violate natural scene statistics (NSS) cause some failures in traditional natural scene image (NSI) based applications. Hence some specialized technologies for screen content image (SCI) have been proposed, such as screen content video compression [1].

The booming of screen content also calls for SCI-specific image quality measures. Limited work has been done concerning SCI quality assessment (QA). In [2], the authors constructed a screen image quality assessment database (SIQAD), which shows that state-of-the-art image quality assessment (IQA) measures do not work efficiently for SCIs. It is reasonable since current IQA measures are implicitly designed for NSIs and somehow rely on NSS. Wang et al. [3] also constructed a database called quality assessment of compressed SCI (QACS). In [4], the authors proposed a full-reference (FR) saliency-guided quality measure named SQMS for SCI. SQMS exploits gradient magnitude similarity as the quality map, which is then weighted by a specific saliency map. Gu et al. [5], [6] learned blind quality evaluation engines for SCI from a huge group of SCIs and corresponding objective quality scores calculated by FR measures.

Although dozens of NSI quality estimators [7], [8], [9], [10], [11], [12], [13], [14], [15], [16] and several limited SCI quality measures [2], [3], [4], [5], [6] are proposed, they are either implicitly designed for NSIs or specifically developed for SCIs. Only very few quality measures can work for NSIs and SCIs simultaneously. Min et al. [17] proposed a blind blockiness measure which works for JPEG compressed NSIs and SCIs. Xu et al. [18] developed a measure for NSIs and SCIs. In [19], Min et al. constructed a cross-content-type database, and proposed a unified content-type adaptive blind IQA measure for compressed natural, graphic and screen content images. In practical multimedia communication systems, we may encounter both types of images, and sometimes we do not have any prior knowledge about the image types. Efficient general quality measures ignoring image types are highly needed in such circumstances. In this paper, we extract quality features efficient for both types of images and develop a general reduced-reference (RR) quality measure without any explicit image type classification.

The proposed method is based on visual saliency detection. Visual saliency detection is an important research topic in areas of psychology, image processing and computer vision [20]. Visual attention and quality assessment are two closely related research topics [7], [9], [10], [12], [13], [21], [22], [23]. Quality degradation can influence visual attention [21]. Contrarily, visually salient positions should be more carefully processed since subjects judge the image quality according to the observations of some limited positions, and a typical use of visual attention model is to optimize resource allocation and improve the perceptual quality under the constraints of bandwidth [24], [25], [26], [27], [28].

Motivated by the interaction between visual attention and quality assessment, some researchers used visual attention map as a weighting map during the quality pooling stage of IQA [9], [10], [13], [22], [23]. Min et al. [13] collected some visual attention data for main-stream IQA databases. Zhang et al. [22], [23] studied the use of saliency model in objective quality assessment models. Liu et al. [9] used the saliency map to highlight the visually salient areas. Besides highlighting the salient regions, Saha and Wu [10] used the dissimilarity between saliency maps of the reference and distorted images to highlight the more distorted image content. Besides visual attention maps, some measures utilize other kinds of weighting maps such as phase congruency map [7] and gradient magnitude map [12]. Although without explicit visual attention prediction or visual saliency detection processes, such kinds of weighting maps have also highlighted the visually salient positions, which can be also deemed as one kind of visual saliency.

Instead of as a weighting map, visual saliency can be also used as a quality feature since quality degradation can strongly affect saliency detection. Zhang et al. [8] proposed a FR IQA method named VSI by measuring the similarity between the reference image’s and the distorted image’s visual saliency. VSI is a FR measure since it utilizes not only saliency, but also the gradient magnitude and chrominance. All extracted feature maps have the same resolution as the reference image. Actually, deriving a gray scale saliency map from a color image is an operation of dimension reduction, which motivates us to develop a saliency-induced reduced-reference (SIRR) IQA measure.

SIRR detects saliency map of the reference image as the reference data, and then measure the similarity between the reference and distorted images’ saliency maps. We try to reduce the reference data from two aspects. First, we down-sample the reference image to a coarser scale to detect saliency, whose resolution is only one over sixty-four of the original resolution. We take full advantage of such down-sampling operation to reduce the reference data. Second, we exploit a binary image descriptor called “image signature” [29] to detect image saliency. The image saliency is represented by the binary image signature, which also significantly reduces the reference data. The final quality is described by the similarity between two images’ saliency maps. In this work, the similarity is evaluated by the classical image fidelity measure SSIM [30].

We perform extensive experiments to test the proposed SIRR in both NSIs and SCIs. Five large-scale NSI QA databases and two recent SCI QA databases are used. Among the NSI databases, LIVE [31], TID2008 [32] and CSIQ [33] are general-purpose IQA databases, whereas LIVEMD [34] focuses on multiply distorted images and CID2013 [35] consists of contrast changed images. As it to the SCI databases, SIQAD [2] is a general-purpose one and QACS [3] concentrates on compressed SCIs. The all seven databases can give an overall description of both NSIs and SCIs. As will be presented in the experiments part, the proposed SIRR is efficient for both types of images. SIRR can be comparable to or outperform state-of-the-art FR and RR IQA measures on all seven IQA databases.

The remainder of this paper is organized as follows. Section 2 describes the proposed saliency-induced reduced-reference quality measure. Experimental results are given in Section 3. We compare the proposed method with state-of-the-art FR and RR quality measures in this section. Section 4 concludes this paper.