Perceptual quality assessment for no-reference image via optimization-based meta-learning

Highlights

• We integrate the image quality assessment with meta-learning strategy.

• This model optimizes initialization parameter and learning rate simultaneously.

• Our framework can learn various distortion knowledge and complex real distortions.

Abstract

Image quality assessment (IQA) is a critical issue in computer vision, which intends to simulate human visual system (HVS) with a view to evaluating the error degree of distorted images from reference images. The algorithms based on deep learning have been successfully introduced to quality evaluation of no-referenced images in the last years. Unfortunately, they are confronted with the problems of over-fitting and weak generalization ability caused by insufficient labeled data. The emergence of meta-learning has brought new ideas, which have been proved to address the issues regarding few-shot learning. However, the commonly used meta-learning metrics only learn the initialization of weights, which can’t guarantee the optimal gradient direction. The manual design mode leads to lower accuracy and speed. In our work, an improved meta-learning framework is applied to no-reference (NR) IQA to meet the above challenges. It can achieve maximum generalization performance through only a few update iterations. Specifically, we collected a great many NR-IQA tasks with different distortions to pre-train meta-model and optimizer to learn a general weight initialization and optimization rule. Then, the meta-model acquired meta-knowledge and learned unique learning rate for each task. Finally, it could be directly adapted to new NR-IQA tasks only by fine-tuning a few images. Experiments on synthetic and authentic datasets proved that our approach has more vital learning ability and better generalization performance for evaluating real distorted images, effectively reducing dependence on manual marking.

Introduction

In recent years, with the popularization of intelligent mobile devices and the advancement of multimedia technology, the application of digital images is becoming more and more widespread in our daily lives. At any stage of the entire media technology chain, digital images will inevitably be degraded and distorted. However, it requires lengthy and expensive subjective experiments to make human observers directly assess digital images’ visual quality. Therefore, objective image quality assessment (IQA) [1] solves this problem by predicting image quality with a specific model which could simulate human visual perception system. Due to the advantages of convenience, speed, stability, and reliability, objective IQA methods have a broad application prospect in many fields, such as image fusion and image restoration [2], [3].

In practical applications, NR-IQA [4], also called blind IQA (BIQA), is more attractive than full-reference (FR) and reduced-reference (RR). However, it is more challenging to use NR-IQA because reference information is often unavailable in many scenarios. In the light of the type of distortion, NR-IQA methods can be categorized into distortion-specific methods and general-purpose methods. Distortion-specific methods assume that digital images have specific noise or artifacts, such as JPEG compression [5], blur/noise [6] and JPEG2000 compression [7]. Since images usually contain various unknown distortion types, increasing number of general-purpose models have been proposed. At the same time, with the breakthrough of machine learning in computer vision, researchers have begun to introduce it into IQA. According to the features made by hand or obtained through learning, these methods do not have to predict the distortion type but characterize the deviation degree of the distorted images. Natural scene statistics (NSS) of images can be applicable to design hand-made features [8], [9]. With the rise of data-driven machine learning research, NR-IQA methods based on learning [10], [11] have become very popular to learn quality perception features from given images.

In recent years, the learning-based image quality metrics (IQMs) have performed well because of the powerful fitting capability of deep neural networks (DNNs) [12], [13]. Although DNNs have proven its adaptability to different learning tasks, a large amount of labeled data and computing resources are necessary for DNNs model training. Since IQA is usually a small sample problem, it has become a challenging task to train deep IQMs from scratch. To resolve this issue, researchers have developed new methods to overcome the need for massive datasets. As shown in Fig. 1(a), some studies employed a transfer learning-based approach using pre-trained convolutional networks to train non-IQA data, such as ImageNet [14]. However, the limitation of these methods lies in the low correlation between image classification and IQA, which requires lots of fine-tuning steps. Moreover, recent researches could also learn NR-IQA models by using information about the relative ranking from distortion specifications [15], FR-IQA models [16] and human data [17]. But the degradation process of these methods was clearly defined, so it only had good performance in synthetic distortions. Therefore, they could not be extended to unknown distortions.

It is difficult to find neural network weights that generalize well from small datasets since many deep learning metrics learn each task independently. In contrast, humans can generalize to the evaluation of images with unknown distortions using high-quality prior knowledge, and do not require large amounts of data. In addition, most IQA models spend a lot of time in the initial training stage [18], while in some practical applications, such as monitoring, real-time processing is a need. Therefore, we consider improving the fast-learning ability of the network. In particular, humans can regard IQA of each known distortion as a task, and utilize few-shot learning (FSL) strategy [19] to achieve rapid adaptation to new tasks. As a promising approach in FSL, meta-learning has been successfully used in various fields of computer vision [20]. For unseen distortion tasks with few instances, meta-learning can get prior knowledge from NR-IQA tasks of various given distortion types, then guide the model to update the parameters accurately and quickly, as shown in Fig. 1(b).

In this paper, we try to solve the problem about insufficient generalization ability of deep IQA models and propose an improved meta-learning framework applied to NR-IQA, as shown in Fig. 2. To this end, combined with an optimization-based meta-learning strategy, the meta-model and optimizer parameters are trained through the test loss obtained from a great deal of synthetic distorted NR-IQA tasks. In this way, the gradient of the model is reduced to the most suitable position for subsequent updates. During training process, the meta-learner guides the network to optimize its weights in each task to adapt to the current task quickly. The meta-learner is then updated with the experience learned from a series of tasks to find a set of potential initialization parameters for new tasks. The contributions of this paper are summarized as follows:

• (1)We integrate the deep IQA model with an optimization-based meta-learning strategy into a meta-model. The model can easily handle unseen distortions by leveraging prior knowledge of various tasks without using some hand-designed training techniques, complex network structure design or hyperparameters adjustment. Also, the trained meta-model could learn a general initial representation and gradient direction of the IQA model for rapid parameter updating with a small number of training examples.

• (2)We introduce an improved meta-learning metric to make the network learn to learn, which can optimize the initialization parameter and learning rate of the model simultaneously in one meta-learning step instead of setting manually. In this way, the convergence of the model can be accelerated.

• (3)Experiments demonstrated that our meta-learning framework has the ability to learn various distortion knowledge and deal with complex real distortions. It indicates that our approach can be extended to any NR-IQA task, which significantly improves the generalization performance.

The rest of this paper is structured as follows. Section 2 briefly introduces the related work of NR-IQA and meta-learning algorithm. In the Section 3, we introduce the meta-learning framework for NR-IQA problem based on optimization in detail. Section 4 gives our experimental results. At last, the conclusion is given in the Section 5.