The context effect for blind image quality assessment

doi:10.1016/j.neucom.2022.11.026

Neurocomputing

Volume 521, 7 February 2023, Pages 172-180

https://doi.org/10.1016/j.neucom.2022.11.026 Get rights and content

Abstract

Image quality assessment (IQA) is a process of visuo-cognitive, which is an essential stage in human interaction with the environment. The study of the context effect (Brown and Daniel, 1987) also shows that the evaluation results made by the human vision system (HVS) is related to the contrast between the distorted image and the background environment. However, the existing IQA methods carry out the quality evaluation that only depends on the distorted image itself and ignores the impact of environment to human perception. In this paper, we propose a novel blind image quality assessment(BIQA) based on the context effect. At first, we use a graphical model to describe how the context effect influences human perception of image quality. Based on the established graph, we construct the context relation between the distorted image and the background environment by the X. Han et al. (2015). Then the context features are extracted from the constructed relation, and the quality-related features are extracted by the fine-tuned neural network from the distorted image in pixel-wise. Finally, these features are concatenated to quantify image quality degradations and then regress to quality scores. In addition, the proposed method is adaptive to various deep neural networks. Experimental results show that the proposed method not only has the state-of-art performance on the synthetic distorted images, but also has a great improvement on the authentic distorted images.

Introduction

In the area of Image Quality Assessment (IQA), in more general terms measurement metric categorized into subjective and objective quality assessment.Currently,objective image quality assessment can be broadly divided into Opinion-aware and Opinion-unaware.In Opinion-aware,objective image quality assessment algorithms based on the mean opinion score(MOS)[9] obtained from subjective experiments.Over last two decades years, many significant works have been well-established.

IQA plays a significant role in many foundational visual problems.The capability of diverse electronic visual content continues to advance rapidly in network applications.However,there is a wide variety of annoying distortions when these diversity of images are being accessed, uploaded, and displayed.Therefore,How to accurately describe this quality change is very important for the optimization of system parameters or the transmission costs[3].This quantization process has a wide range of applications.In the literature, some full reference image quality assessment (FR-IQA) methods[4] inspired from humanvisial system(HVS).On the other hand,some NR-IQA methods[5], [6] which rely on natural scene statistics(NSS) considered image spatial domain and transform domain features.Since the reference image is not required, the application scenario of NR-IQA evaluation is broader than FR-IQA. Based on whether human subjective scores need to be used in the process of designing a perceptual model to evaluate.

Following Marr [7] and Newell [8], the quality of an image cannot just be simply defined as a visibility of its distortions, but should be regarded as the adequacy of this image as input to the vision stage of the interaction process. This interaction of visuo-cognitive is not an isolated process but instead as an essential stage in human interaction with the environment [8]. Image quality assessment can also be regarded as a process of visuo-cognitive. However, the existing IQA methods ignores the impact of environment to human perception. For example, when evaluating the image quality, the existing IQA methods only rely on some regularity of distorted image and use the common machine learning model to build the fitting function of MOS. But the MOS is generated by human subjective evaluation, which means it is influenced by both visuo-cognitive and background environment. Those image-based regularity cannot depicts the relationship between the human perception and the human interaction with the environment. Therefore, the evaluation results cannot really reflect the process of human perception to image quality.

In this work, inspired by the study of cognitive psychology, we propose a novel BIQA method based on the context effect [1]. The context effect refers to a psychological effect that the quality evaluation made by the HVS is related to the contrast between the distorted image and the background environment. According to this fact, we use a graphical model to describe the mechanism of context effect on IQA.Based on the established graph, we construct the context relation between the distorted image and the background environment by Matchnet[2]. The Matchnet is trained to rank images in terms of perceptual quality, so the context relation is actually the quality difference between the distorted image and the other image in the background environment. After that, a context feature is constructed based on the obtained context relation, and a quality-related feature is extracted by a deep neural network from the pixel-level of the distorted image. Finally, The context feature and the quality-relatd feature are combined to form the final quality descriptor, and then regress to quality score.

In summary, the main contributions of this paper are as follows:

1.
It is the first time to take the process of human interaction with the environment into account when we do the IQA task, and the context effect is introduced to describe the impact of environment to human perception.
2.
We build a graphical model to describe the mechanism of the context effect on IQA, which makes it have good interpretability.
3.
We propose a novel BIQA method based on the context effect, which can be combined with other deep neural network to get siginificant performance improvement.

Section snippets

Related work

The idea of using hand-crafted features to construct BIQA methods has been the mainstream for a long time. The early research was focused on the distortion-specific features which are designed for particular distortion type. Then is the well-known features family of NSS which assume that natural scenes have statistical regularities that can describe the change of perceptual quality[9]. In addition, there are also some other methods such as using the semantic saliency or codebooks to construct

Proposed model

In this section, we establish a graphical model to describe the mechanism of the context effect on IQA, and then propose a novel BIQA method, which is called DeepCE. The implementation of DeepCE can be divided into three phases. First, a Matchnet is trained to generate a feature which is about the context effect. Then a classical deep neural network which has pre-trained in the ImageNet[35] is fine-tuned on the IQA dataset. Finally, the feature of context effect and the feature extracted by the

Experimental results

In this section, we first describe the experimental protocol, including database, criteria, and implementation details. Then, we compare with state-of-the-art BIQA methods in terms of full-reference and no-reference performance, respectively. comparisons separately. We design a series of experiments to verify the effect of background images.

Conclusion

In this paper, we establish a graphical model to describe the context effect in IQA for the first time. Then based on this graph, the role of context effect is refined into the CE-feature. By combining the CE-feature with other pixel-level quality-related features, we propose a novel BIQA model, which is called DeepCE. Experimental results show that the proposed DeepCE is comparable with other state-of-the art BIQA methods. In addition, the other contribution of our work is that it is possible

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This research was supported in part by the National Natural Science Foundation of China (Grant No. 61871311), the Key Industrial Innovation Chain Project in Industrial Domain of Shaanxi Province (Grant No. 2020ZDLGY05-01). Aeronautical Science Foundation of China (Grant No. 2020Z071081004).

Zehong Liang received the B.S. degree from the School of Electronic Engineering, Xidian University, Xian,China, in 2021, where he is currently pursuing the masters degree with the School of Electronic Engineering. His current research interests include machine learning, visual information processing and image quality assessment.

References (48)

T.C. Brown et al.
Context effects in perceived environmental quality assessment: scene selection and landscape quality ratings
Journal of Environmental Psychology
(1987)
Y. Liu et al.
Abnet: Adaptive balanced network for multiscale object detection in remote sensing imagery
IEEE Transactions on Geoscience and Remote Sensing
(2022)
N. Ponomarenko et al.
Image Database TID2013: Peculiarities, results and perspectives
Signal Processing: Image Communication
(2015)
X. Han, T. Leung, Y. Jia, R. Sukthankar, A.C. Berg, Matchnet: Unifying feature and metric learning for patch-based...
Z. Wang, A.C. Bovik, L. Lu, Why is image quality assessment so difficult?, in: Proceedings of International Conference...
S. Wang et al.
Human visual system-based fundus image quality assessment of portable fundus camera photographs
IEEE transactions on medical imaging
(2015)
B. Yan et al.
Naturalness-aware deep no-reference image quality assessment
IEEE Transactions on Multimedia
(2019)
K.-Y. Lin, G. Wang, Hallucinated-iqa: No-reference image quality assessment via adversarial learning, in: Proceedings...
D. Marr
Vision: a computational investigation into the human representation and processing of visual information
(1982)
A. Newell
Unified theories of cognition
(1994)

A.K. Moorthy et al.

Blind image quality assessment: From natural scene statistics to perceptual quality

IEEE transactions on Image Processing

(2011)

J. Xu et al.

Blind image quality assessment based on high order statistics aggregation

IEEE Transactions on Image Processing

(2016)

X. Liu, J. van de Weijer, A.D. Bagdanov, Rankiqa: Learning from rankings for no-reference image quality assessment, in:...

L. Kang, P. Ye, Y. Li, D. Doermann, Convolutional neural networks for no-reference image quality assessment, in:...

H. Talebi et al.

Nima: Neural image assessment

IEEE transactions on image processing

(2018)

S. Su et al.

Blindly assess image quality in the wild guided by a self-adaptive hyper network, in

IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

(2020)

W. Zhang et al.

Uncertainty-aware blind image quality assessment in the laboratory and wild

IEEE Transactions on Image Processing

(2021)

K. Simonyan et al.

Very deep convolutional networks for large- scale image recognition

Computational and Biological Learning Society

(2015)

S. Bosse et al.

Deep neural networks for no-reference and full-reference image quality assessment

IEEE Transactions on Image Processing

(2018)

H. Zeng, L. Zhang, A. C. Bovik, Blind image quality assessment with a probabilistic quality representation, in:...

S. Bianco et al.

On the use of deep learning for blind image quality assessment

Signal, Image and Video Processing

(2018)

K. Ma et al.

dipiq: Blind image quality assessment by learning-to-rank discriminable image pairs

IEEE Transactions on Image Processing

(2017)

K. Ma et al.

End-to-end blind image quality assessment using deep neural networks

IEEE Transactions on Image Processing

(2018)

J. Kim et al.

Fully deep blind image quality predictor

IEEE Journal of selected topics in signal processing

(2017)

Cited by (5)

MFHOD: Multi-modal image fusion method based on the higher-order degradation model
2024, Expert Systems with Applications
The task of multimodal image fusion aims to preserve the respective advantages of each modality, such as the detailed texture information from visible light images and the salient target information from infrared images. However, images in real environments are often not perfect high-quality images but are affected by various degradation factors such as noise, blur, and glare. Existing unsupervised multimodal image fusion algorithms use undegraded source images as input to fusion networks and generate high-quality fusion images through complex fusion strategies. This single linear learning approach from high-quality to high-quality cannot be applied to the fusion task of degraded multimodal images in real environments. To address this issue, this paper proposes an end-to-end multimodal image fusion network based on a high-order local degradation model (MFHOD). Firstly, inspired by the idea of probabilistic degradation, we propose a high-order local random degradation model (HODM), which inputs the source multimodal images into the degradation model to obtain degraded images before feeding them into the network. Secondly, we design a simple and efficient dual-branch feature extraction encoder to extract deep features from images. Then, from the perspectives of image pixels, brightness, and gradients, we propose an improved composite loss composed of multiple loss functions to constrain network training. Finally, we propose a $L_{2}$ -norm fusion strategy for preserving brightness information in low-light nighttime images. Our MFHOD demonstrates good performance on infrared and visible light image datasets as well as medical image datasets. Experimental results show that MFHOD can effectively suppress the effects of glare, noise, and smoke in adverse environments, and also improve the quality of fusion images in low-light and nighttime environments.
Blind image quality assessment based on hierarchical dependency learning and quality aggregation
2024, Neurocomputing
Image quality assessment (IQA) aims to build a quality prediction model to assess image quality automatically rather than artificially. Due to a lack of reference images, blind image quality assessment (BIQA) has become an attractive yet challenging research topic. Inspired by the hierarchical perception mechanism in the human visual system, some existing BIQA methods aggregate multi-stage features of a convolutional neural network (CNN). However, they are regardless of the latent dependencies. To solve this problem, we propose a novel BIQA method based on hierarchical dependency learning and quality aggregation (HDLaQA). The proposed method includes multi-stage feature extraction, hierarchical dependency learning, and quality aggregation. In multi-stage feature extraction, a CNN is used as the feature extractor and multi-stage features are output for further learning. In hierarchical dependency learning, spatial and channel dependencies among the multi-stage features are modeled. To this end, a dual-head spatial dependency (DSD) module is designed to harvest the spatial dependencies between the adjacent-stage features and deliver these dependencies to the next stage. Moreover, exponential bilinear pooling (EBP) is presented to learn the channel dependencies, which is more stable than commonly used BP. In quality aggregation, multiple quality scores are predicted based on the learned dependencies, and multiple learnable weights are used to measure the importance of the predicted scores for final quality evaluation. Experimental results on seven IQA databases demonstrate the competitiveness of the proposed method on both synthetic and authentic distortions.
Combining CNN and transformers for full-reference and no-reference image quality assessment
2023, Neurocomputing
Most deep learning approaches for image quality assessment use regression from deep features extracted by CNN (Convolutional Neural Networks). However, non-local information is usually neglected in existing methods. Motivated by the recent success of transformers in modeling contextual information, we propose a hybrid framework that utilizes a vision transformer backbone to extract features and a CNN decoder for quality estimation. We propose a shared feature extraction scheme for both FR and NR settings. A two-branch structured attentive quality predictor is devised for quality prediction. Evaluation experiments on various IQA datasets, including LIVE, CSIQ and TID2013, LIVE-Challenge, KADID-10 K, and KONIQ-10 K, show that our proposed models achieve outstanding performance for both FR and NR settings.
SherVine: A graphical dependency modeling for shearlet transform and its application in image quality assessment
2023, Expert Systems with Applications
Many applications of image processing, statistical modeling of frequency domain coefficients is considered by researchers as an approach. In this paper, we present a probabilistic graphical model based on the Vine Copula approach called SherVine(Shearlet Vine) to model the shearlet transform coefficients jointly. The proposed model is an extension of joint modeling with the help of Copula’s theory, which can diversely model a pair of shearlet coefficients. Also, the graphical representation of SherVine leads to a better understanding of dependencies types between shearlet coefficients. In the second part of this paper, we propose a novel image quality assessment(IQA) approach based on SherVine called IQA-SherVine. IQA-SherVine uses the parameters of the SherVine model as a feature in the IQA framework. The extracted features are modeled using the decision tree algorithm in IQA-SherVine. The results show that IQA-SherVine performs very well compared to the statistical and deep learning based state-of-the-art methods.
No-Reference Image Quality Assessment Based on a Multitask Image Restoration Network
2023, Applied Sciences (Switzerland)

Wen Lu received the BSc, MSc and PhD degrees in signal and information processing from Xidian University, Xi an, China, in 2002, 2006 and 2009, respectively. Since 2009, he joined the School of School of Electronic Engineering at Xidian University. From 2010 to 2012, he was a Post-doctoral Research Fellow with the Department of Electronic Engineering, Stanford University, U.S. He is currently a professor at School of Electronic Engineering, Xidian University. His current research interests include multimedia analysis, computer vision, pattern recognition, deep learning. He has published 2 books and around 50 technical articles in refereed journals and proceedings including IEEE Transactions on Image Processing, IEEE Transactions on Cybernetics, Information Science, Neurocomputing, etc. He is also on the editorial boards and serves as reviewers for many journals, such as IEEE Transactions on Image Processing, IEEE Transactions on Multimedia, etc.

Yong Zheng received the B.S. degree from the School of Electronic Engineering, Xidian University, Xian,China, in 2020, where he is currently pursuing the masters degree with the School of Electronic Engineering. His current research interests include machine learning, visual information processing and image quality assessment.

Weiquan He received the B.S. degree from the School of Telecommunication and Information Engineering, Xi an University of Posts and Telecommunications, Xian,China, in 2017. He is currently working at Alibaba,research interests include machine learning, visual information processing and image quality assessment.

Jiachen Yang IEEE Senior Member; a member of New Century Outstanding Talents Support Program of the Ministry of Education; professor of Communication and Information Engineering, Tianjin University; Doctoral supervisor. Has been in charge of more than 30 projects in the past five years, including two projects of National Natural Science Foundation, and has gotten more than 15 million yuan cumulative funds. He has published more than 40 papers on the SCI. He also has applied for more than 40 patents, and more than 20 of them have been authorized.

View full text

The context effect for blind image quality assessment

Abstract

Introduction

Section snippets

Related work

Proposed model

Experimental results

Conclusion

Declaration of Competing Interest

Acknowledgments

Journal of Environmental Psychology

IEEE Transactions on Geoscience and Remote Sensing

Signal Processing: Image Communication

Human visual system-based fundus image quality assessment of portable fundus camera photographs

IEEE transactions on medical imaging

Naturalness-aware deep no-reference image quality assessment

IEEE Transactions on Multimedia

Vision: a computational investigation into the human representation and processing of visual information

Unified theories of cognition

Blind image quality assessment: From natural scene statistics to perceptual quality

IEEE transactions on Image Processing

Blind image quality assessment based on high order statistics aggregation

IEEE Transactions on Image Processing

Nima: Neural image assessment

IEEE transactions on image processing

Blindly assess image quality in the wild guided by a self-adaptive hyper network, in

IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

Uncertainty-aware blind image quality assessment in the laboratory and wild

IEEE Transactions on Image Processing

Very deep convolutional networks for large- scale image recognition

Computational and Biological Learning Society

Deep neural networks for no-reference and full-reference image quality assessment

IEEE Transactions on Image Processing

On the use of deep learning for blind image quality assessment

Signal, Image and Video Processing

dipiq: Blind image quality assessment by learning-to-rank discriminable image pairs

IEEE Transactions on Image Processing

End-to-end blind image quality assessment using deep neural networks

IEEE Transactions on Image Processing

Fully deep blind image quality predictor

IEEE Journal of selected topics in signal processing