Unified quality assessment of natural and screen content images via adaptive weighting on double scales
Introduction
In the era of information, digital images are distributed everywhere for sharing information. Apart from natural scene images (NSIs) captured from real-world scenes, screen content images (SCIs), which are created using computers, have been increasingly involved in various multimedia communication and information sharing systems, such as online education, remote computing, cloud gaming, etc [1], [2], [3]. The contents of SCIs are often a mixture of pictures, texts, graphics and other computer-generated patterns. They share distinct characteristics from NSIs.
Digital images, no matter NSIs or SCIs, are subject to degradations during processing. Therefore, image quality assessment (IQA) algorithms are presented to evaluate the perceptual quality of distorted images. According to the availability of a reference image, IQA metrics can be classified as full reference (FR), no reference (NR) and reduced reference methods [4]. This paper focuses on FR models, where the reference image is fully accessible when evaluating the distorted image.
In the past decades, numerous traditional IQA methods have been developed for visual quality perception. These methods are natural image quality assessment (NIQA) models, because they are mainly designed for evaluating the perceptual quality of the NSIs, but do not work well on the SCIs [1]. Over the past few years, some methods have been proposed for screen image quality assessment (SIQA). Likewise, these SIQA models perform well for SCIs, but cannot accurately predict the visual quality of NSIs. In fact, neither these NIQA nor SIQA models take the content type variations into account, so they cannot effectively generalize from one type to another.
Many application systems may contain not only one content type of images, but also a mixture of NSIs and SCIs. For example, in online education, the image content presented to users is generally a mixture of natural and screen content. In such a scenario, a straightforward solution is to classify image content type and then select adaptive NIQA or SIQA techniques. However, it might be difficult to accurately classify each image, because images may contain both natural scene regions and text regions. Therefore, it is worth investigating why traditional NIQA models cannot apply to SCIs effectively and how to build unified IQA models to handle both NSIs and SCIs, which can be called cross-content-type IQA models.
In this paper, we analyze the different characteristics between NSIs and SCIs, discuss the reason why SIQA and NIQA models cannot apply to each other, and propose a strategy to build unified IQA models. Overall, our main contributions are listed as follows:
(1) Inspired by some psychological conclusions, this paper argues that it is the different structural scale levels between NSIs and SCIs leading to NIQA models’ failure on SCIs. We also find out that different scale selection can crucially affect the accuracy of quality evaluation on NSIs and SCIs.
(2) This paper links the structural scale levels and image quality by introducing the GDoG to analyze the scale level of structures conveyed in images, and proposing an adaptive weighting strategy associated with image content type to improve IQA performance for cross-content-type images.
(3) We propose a fast unified IQA model, which uses the gradient magnitude as the only feature and adopts the calculation framework of GDoG to perceive image structural degradation. Experimental results conducted on six databases demonstrate that it is with low computational complexity, and delivers promising performance on cross-content-type images.
Section snippets
NIQA models
The most known IQA metrics are mean-squared error (MSE) and peak signal-to-noise ratio (PSNR), which directly compute the error on the intensity of images. PSNR and MSE do not consider the characteristics of the human visual system (HVS) [4], and their assessment results are frequently inconsistent with human perception.
Recently, many models make use of structural information to guide quality prediction. The well-known structural similarity (SSIM) index [4] verifies that a measure of structural
Image structural scales
The performance of existing traditional IQA models designed for NSIs significantly drops when applying to SCIs. But what is the exact reason behind it? What is the specific difference of characteristics resulting in such a result? Inspired by some psychological studies, we point out that it is the different structural scale levels between NSIs and SCIs.
According to the two-process model proposed by Farah [34], the cognitive process of object recognition task distinguishes two forms of analysis:
Gradient degradation of Gaussians
In this section, we introduce the gradient degradation of Gaussians (GDoG) to analyze the scale level of structures conveyed in images, then propose an adaptive weighting strategy associated with image content type to build unified IQA models for cross-content-type images.
It is known that the HVS is highly adapted to extract structural information from images [4]. As stated in Section 3, the representation of images at coarse scale reserves only the coarse-scale structures but loses fine-scale
Experimental results
In this section, we show the experimental results on several IQA databases. Note that the proposed fast IQA index is denoted as AWDS because it is based on the adaptive weighting strategy on double scales.
Conclusion
In this paper, we discuss the different characteristics between NSIs and SCIs, in terms of structural scale levels, which result in the NIQA models’ failure on SCIs. Inspired by this, we introduce the GDoG feature to analyze the structural scale level of images, proposes a unified IQA model based on adaptive weighting strategy on double scales. Experimental results show its promising performance and relatively low computational cost in perceiving the visual quality of both NSIs and SCIs. The
CRediT authorship contribution statement
Li Ding: Conceptualization, Methodology, Investigation, Software, Writing - original draft. Ping Wang: Visualization, Supervision. Hua Huang: Writing - review & editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
References (51)
- et al.
A Haar wavelet-based perceptual similarity index for image quality assessment
Signal Process., Image Commun.
(2018) - et al.
Perceptual image quality assessment through spectral analysis of error representations
Signal Process., Image Commun.
(2019) - et al.
SSVD: Structural SVD-based image quality assessment
Signal Process., Image Commun.
(2019) - et al.
Evaluation of color differences in natural scene color images
Signal Process., Image Commun.
(2019) - et al.
Deepsim: Deep similarity for image quality assessment
Neurocomputing
(2017) An ensemble image quality assessment algorithm based on deep feature clustering
Signal Process., Image Commun.
(2020)- et al.
Eye movements in reading and information processing: Keith Rayner’s 40 year legacy
J. Mem. Lang.
(2016) - et al.
Scale and the differential structure of images
Image Vis. Comput.
(1992) - et al.
Perceptual quality assessment of screen content images
IEEE Trans. Image Process.
(2015) - et al.
Screen content coding based on HEVC framework
IEEE Trans. Multimed.
(2014)
Advanced screen content coding using color table and index map
IEEE Trans. Image Process.
Image quality assessment: from error visibility to structural similarity
IEEE Trans. Image Process.
Image quality assessment based on gradient similarity
IEEE Trans. Image Process.
FSIM: A feature similarity index for image quality assessment
IEEE Trans. Image Process.
Vsi: a visual saliency-induced index for perceptual image quality assessment
IEEE Trans. Image Process.
Gradient magnitude similarity deviation: A highly efficient perceptual image quality index
IEEE Trans. Image Process.
Image information and visual quality
IEEE Trans. Image Process.
Deep neural networks for no-reference and full-reference image quality assessment
IEEE Trans. Image Process.
JND-SalCAR: A novel JND-based saliency-channel attention residual network for image quality prediction
Saliency-guided quality assessment of screen content images
IEEE Trans. Multimed.
Gradient direction for screen content image quality assessment
IEEE Signal Process. Lett.
Screen image quality assessment incorporating structural degradation measurement
Objective quality assessment of screen content images by uncertainty weighting
IEEE Trans. Image Process.
ESIM: Edge similarity for screen content image quality assessment
IEEE Trans. Image Process.
Cited by (3)
New Enhancement Techniques for Optimizing Multimedia Visual Representations in Music Pedagogy
2023, Traitement du Signal