Unified quality assessment of natural and screen content images via adaptive weighting on double scales

https://doi.org/10.1016/j.image.2021.116446Get rights and content

Highlights

  • Different characteristics between NSIs and SCIs.

  • Analyze the scale level of structures conveyed in images.

  • Unified IQA model for cross-content-type images.

Abstract

In multimedia communication systems, digital images may contain various visual contents, among which the natural scene images (NSIs) and screen content images (SCIs) are two important and common types. The existing full-reference image quality assessment (IQA) metrics are designed for only one type of images, but cannot precisely perceive the visual quality of another type. It is still unclear what the different characteristics are between NSIs and SCIs resulting in this failure. Inspired by some psychological studies, we figure out that it is due to the different structural scale levels between NSIs and SCIs. Given this observation, this paper introduces the gradient degradation of Gaussians (GDoG) to analyze the images’ structural scale level, proposes a fast unified IQA index for both NSIs and SCIs by incorporating an adaptive weighting strategy on double scales. Experimental results conducted on several databases verify the effectiveness and efficiency of the proposed unified IQA index for both types of images, also demonstrate that the adaptive weighting strategy based on GDoG can improve the existing models for cross-content-type images.

Introduction

In the era of information, digital images are distributed everywhere for sharing information. Apart from natural scene images (NSIs) captured from real-world scenes, screen content images (SCIs), which are created using computers, have been increasingly involved in various multimedia communication and information sharing systems, such as online education, remote computing, cloud gaming, etc [1], [2], [3]. The contents of SCIs are often a mixture of pictures, texts, graphics and other computer-generated patterns. They share distinct characteristics from NSIs.

Digital images, no matter NSIs or SCIs, are subject to degradations during processing. Therefore, image quality assessment (IQA) algorithms are presented to evaluate the perceptual quality of distorted images. According to the availability of a reference image, IQA metrics can be classified as full reference (FR), no reference (NR) and reduced reference methods [4]. This paper focuses on FR models, where the reference image is fully accessible when evaluating the distorted image.

In the past decades, numerous traditional IQA methods have been developed for visual quality perception. These methods are natural image quality assessment (NIQA) models, because they are mainly designed for evaluating the perceptual quality of the NSIs, but do not work well on the SCIs [1]. Over the past few years, some methods have been proposed for screen image quality assessment (SIQA). Likewise, these SIQA models perform well for SCIs, but cannot accurately predict the visual quality of NSIs. In fact, neither these NIQA nor SIQA models take the content type variations into account, so they cannot effectively generalize from one type to another.

Many application systems may contain not only one content type of images, but also a mixture of NSIs and SCIs. For example, in online education, the image content presented to users is generally a mixture of natural and screen content. In such a scenario, a straightforward solution is to classify image content type and then select adaptive NIQA or SIQA techniques. However, it might be difficult to accurately classify each image, because images may contain both natural scene regions and text regions. Therefore, it is worth investigating why traditional NIQA models cannot apply to SCIs effectively and how to build unified IQA models to handle both NSIs and SCIs, which can be called cross-content-type IQA models.

In this paper, we analyze the different characteristics between NSIs and SCIs, discuss the reason why SIQA and NIQA models cannot apply to each other, and propose a strategy to build unified IQA models. Overall, our main contributions are listed as follows:

(1) Inspired by some psychological conclusions, this paper argues that it is the different structural scale levels between NSIs and SCIs leading to NIQA models’ failure on SCIs. We also find out that different scale selection can crucially affect the accuracy of quality evaluation on NSIs and SCIs.

(2) This paper links the structural scale levels and image quality by introducing the GDoG to analyze the scale level of structures conveyed in images, and proposing an adaptive weighting strategy associated with image content type to improve IQA performance for cross-content-type images.

(3) We propose a fast unified IQA model, which uses the gradient magnitude as the only feature and adopts the calculation framework of GDoG to perceive image structural degradation. Experimental results conducted on six databases demonstrate that it is with low computational complexity, and delivers promising performance on cross-content-type images.

Section snippets

NIQA models

The most known IQA metrics are mean-squared error (MSE) and peak signal-to-noise ratio (PSNR), which directly compute the error on the intensity of images. PSNR and MSE do not consider the characteristics of the human visual system (HVS) [4], and their assessment results are frequently inconsistent with human perception.

Recently, many models make use of structural information to guide quality prediction. The well-known structural similarity (SSIM) index [4] verifies that a measure of structural

Image structural scales

The performance of existing traditional IQA models designed for NSIs significantly drops when applying to SCIs. But what is the exact reason behind it? What is the specific difference of characteristics resulting in such a result? Inspired by some psychological studies, we point out that it is the different structural scale levels between NSIs and SCIs.

According to the two-process model proposed by Farah [34], the cognitive process of object recognition task distinguishes two forms of analysis:

Gradient degradation of Gaussians

In this section, we introduce the gradient degradation of Gaussians (GDoG) to analyze the scale level of structures conveyed in images, then propose an adaptive weighting strategy associated with image content type to build unified IQA models for cross-content-type images.

It is known that the HVS is highly adapted to extract structural information from images [4]. As stated in Section 3, the representation of images at coarse scale reserves only the coarse-scale structures but loses fine-scale

Experimental results

In this section, we show the experimental results on several IQA databases. Note that the proposed fast IQA index is denoted as AWDS because it is based on the adaptive weighting strategy on double scales.

Conclusion

In this paper, we discuss the different characteristics between NSIs and SCIs, in terms of structural scale levels, which result in the NIQA models’ failure on SCIs. Inspired by this, we introduce the GDoG feature to analyze the structural scale level of images, proposes a unified IQA model based on adaptive weighting strategy on double scales. Experimental results show its promising performance and relatively low computational cost in perceiving the visual quality of both NSIs and SCIs. The

CRediT authorship contribution statement

Li Ding: Conceptualization, Methodology, Investigation, Software, Writing - original draft. Ping Wang: Visualization, Supervision. Hua Huang: Writing - review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References (51)

  • MaZ. et al.

    Advanced screen content coding using color table and index map

    IEEE Trans. Image Process.

    (2014)
  • WangZ. et al.

    Image quality assessment: from error visibility to structural similarity

    IEEE Trans. Image Process.

    (2004)
  • Z. Wang, E. Simoncelli, A. Bovik, Multiscale structural similarity for image quality assessment, in: The Thrity-Seventh...
  • LiuA. et al.

    Image quality assessment based on gradient similarity

    IEEE Trans. Image Process.

    (2012)
  • ZhangL. et al.

    FSIM: A feature similarity index for image quality assessment

    IEEE Trans. Image Process.

    (2011)
  • LinZ. et al.

    Vsi: a visual saliency-induced index for perceptual image quality assessment

    IEEE Trans. Image Process.

    (2014)
  • W.X. et al.

    Gradient magnitude similarity deviation: A highly efficient perceptual image quality index

    IEEE Trans. Image Process.

    (2014)
  • SheikhH.R. et al.

    Image information and visual quality

    IEEE Trans. Image Process.

    (2006)
  • BosseS. et al.

    Deep neural networks for no-reference and full-reference image quality assessment

    IEEE Trans. Image Process.

    (2017)
  • SeoS. et al.

    JND-SalCAR: A novel JND-based saliency-channel attention residual network for image quality prediction

    (2019)
  • GuK. et al.

    Saliency-guided quality assessment of screen content images

    IEEE Trans. Multimed.

    (2016)
  • NiZ.K. et al.

    Gradient direction for screen content image quality assessment

    IEEE Signal Process. Lett.

    (2016)
  • GuK. et al.

    Screen image quality assessment incorporating structural degradation measurement

  • FangY. et al.

    Objective quality assessment of screen content images by uncertainty weighting

    IEEE Trans. Image Process.

    (2017)
  • NiZ. et al.

    ESIM: Edge similarity for screen content image quality assessment

    IEEE Trans. Image Process.

    (2017)
  • View full text