Content-oriented image quality assessment with multi-label SVM classifier

https://doi.org/10.1016/j.image.2019.07.018Get rights and content

Abstract

Image content is a fundamental attribute of images and plays an important role in human perception of image information. However, the influence of image content type, which is derived based on the classification of the image content, has been largely ignored in the image quality assessment (IQA). In this paper, a new IQA database based on the classification of image content is built. In particular, the database contains four content types, including landscape, human face, handcrafted scene and the hybrid scene. In total, 80 reference images with 20 images for each type of content are involved, and 1600 distorted images with mean opinion scores (MOSs) are generated by using five types and four levels of distortion. Furthermore, to classify these images, especially for the hybrid case, a Support Vector Machine (SVM) based multi-label (ML) classification is presented. Extensive experiments based on existing no reference IQA (NR-IQA) models show that content classification can greatly facilitate the image quality evaluation. The database and code are made publicly available at: https://github.com/jingchao17/Content-oriented-Database.

Introduction

Recently the diversity and multi-functionality of the displays have increased the demands of more advanced image processing methods, such that the quality evaluation and improvement of digital images during the image processing become vital. Subjective testing is the most reliable approach to evaluate the quality of images, and numerous image quality assessment (IQA) databases, including LIVE [1] and TID [2], [3], have been established. However, subjective testing is time-consuming and expensive, which inspires the objective IQA methods that can automatically evaluate the image quality with computational models.

Objective IQA aims to predict the image quality and perform consistently with the human visual system (HVS), and it is widely used in image processing, such as acquisition, compression, transmission, and enhancement [4]. Depending on the availability of the pristine image, the objective IQA methods can be classified into three types: full-reference (FR), reduced-reference (RR) and no-reference (NR). FR-IQA means that the reference image is available during the quality prediction [5], while only some features of the reference image can be utilized for RR-IQA methods [6]. NR-IQA methods evaluate image quality without any information of the corresponding reference image. Among these methods, NR-IQA is regarded as the most difficult task as the reference image is unavailable. As such, the distortion-type aware IQA has been intensively studied to obtain the reliable quality score. Generally speaking, NR-IQA methods can be categorized into one or the combination of these three categories: (1) Natural scene statistics (NSS) approaches, which are commonly based on the hypothesis that images of the natural world occupy a small subspace of all possible images, leading to the method of finding the distance between the test image and the subspace of natural world [7], [8], [9], [10]; (2) Distortion-specific approaches: which seek to use a specific distortion model to measure the perceptual quality [11], [12], [13], [14]; (3) Training-based approaches: rely on extracting some features from an image, based on which a quality assessment model is learned to predict the image quality [15], [16], [17] (see Table 1).

In essence, the existing databases consist of different kinds of images, such as handcrafted scenes, natural images, animals and human beings, and a number of distortion types including blur, white noise, JPEG, and JPEG2K are injected to generate the distorted image. However, different kinds of images always have different characteristics, which have not been explicitly investigated in the literature. For example, handcrafted images rely on sharp edges to characterize the object information. For the natural scene, textures are more apparent and exhibit strong masking effect. The quality assessment model should be able to adapt to the image content type as well. This motivates us to explore the image quality from the perspective of image content type, and a new IQA database based on the classification of content types is built. In particular, this database contains four types of images: landscape, handcrafted scene, human face and hybrid (based on the combination of the first three types), with 20 reference images for each content type. Five distortion types including Gaussian blur, HEVC compression, JPEG compression, JPEG2000 compression and white Gaussian noise with four distortion levels are considered in the construction of the database, resulting in 80 reference images and 1600 distorted images in total. To the best of our knowledge, this is the first IQA database that considers the content type classification.

There are two parts in our method: (1) a classifier to classify the images based on content, (2) IQA models to predict quality for the images in our proposed database. Since the images in hybrid type have multiple labels (i.e., handcrafted scene, natural scene, and human being), we propose to use multi-label (ML) learning for the classification. The task of ML learning is to construct models based on ML training set, in order to accurately predict the label sets of testing samples [20]. Usually, there are two categories of ML learning algorithms: (1) problem transformation methods, and (2) algorithm adaption methods. A problem transformation method transforms the original ML problem into a set of single-label problems. Compared with algorithm adaption method, it is easy to implement and can be achieved with any single-label learner. Thus, in this paper, we adopt problem transformation method, and achieve it with binary relevance (BR) strategy, where support vector machine (SVM) is used as the base classifier. Finally, a SVM-based ML model is constructed to classify the images in the Content-oriented Database before testing on IQA metrics.

The rest of this paper is organized as follows. In Section 2, the related works are discussed. In Section 3, we describe the construction of the IQA database. The ML model to classify image types is presented in Section 4. Validations by considering the image type as an attribute of IQA are provided in Section 5, and we conclude this paper in Section 6.

Section snippets

Existing IQA databases

There exists numerous IQA databases which are widely applied to evaluate the accuracy of IQA models. In 2006, Sheikh et al. proposed the LIVE database in [1], which was extensively adopted to evaluate and analyze the performance of IQA algorithms. The LIVE database consists of 29 reference and 779 distorted images with five distortion types such as Gaussian blur, white Gaussian noise, JPEG, JPEG2000 and fast fading transmission error. More than 25000 individual human quality judgments were used

Description of the database

The new database includes 80 reference images with four types of image content, including face, handcrafted scene, landscape and the hybrid scene with the previous three categories. The reason why we just use these four classes is that our aim is to build the new database to investigate the influence of the image content on the IQA performance. These four classes of the images include indoor and outdoor scenes, human beings, and many kinds of artificial products, which are sufficient to verify

SVM-based ML model

In order to classify the images in the database, we adopt SVM-based ML learning algorithm. As the most commonly used strategy, BR is utilized, which constructs a binary classifier for each decision component, and gets the final results by combining the predictions of the multiple classifiers. Suppose Y={ y1, y2, …, yM } denotes a label set with M possible class labels and Rd denotes the d-dimensional sample space. As a remark, we have M=3 in this paper, and y1, y2, y3 correspond to face,

Experimental results

In this section, extensive experimental results are provided to verify the influence of image content on image quality assessment. Before testing the IQA algorithms on the Content-oriented Database, we first automatically classify the testing images. We use SVM as our single-label base classifier, and adopt a BR strategy to achieve SVM-based ML classification. The experiments for classification and IQA algorithms are carried out under MATLAB R2016a, which are executed on a computer with

Conclusion

We have described the content-oriented database, which is among of the largest databases with subjective testing for IQA research. In order to test the performance of IQA models, we use images with three kinds of content and combine them together. Before testing IQA performance, we first classify the testing images of the content-oriented database by using a SVM-based ML classifier. Then the IQA models that trained on each content based sub-database are used to predict the quality of images in

Acknowledgments

This work was supported in part by Hong Kong RGC General Research Fund 9042489 under Grant CityU 11206317, in part by Hong Kong RGC General Research Fund 9042322 under Grant CityU 11200116, in part by the Hong Kong RGC Early Career Scheme under Grant 9048122 (CityU 21211018), in part by the City University of Hong Kong under Grant 7200539/CS and in part by the National Natural Science Foundation of China under Grant 61772344 and Grant 61811530324.

References (50)

  • WangZ. et al.

    Quality-aware images

    IEEE Trans. Image Process.

    (2006)
  • MoorthyA.K. et al.

    A two-step framework for constructing blind image quality indices

    IEEE Signal Process. Lett.

    (2010)
  • MoorthyA.K. et al.

    Blind image quality assessment: From natural scene statistics to perceptual quality

    IEEE Trans. Image Process.

    (2011)
  • GuK. et al.

    Model-based referenceless quality metric of 3D synthesized images using local image description

    IEEE Trans. Image Process.

    (2018)
  • SaadM.A. et al.

    Blind image quality assessment: a natural scene statistics approach in the DCT domain

    IEEE Trans. Image Process. Publ. IEEE Signal Process. Soc.

    (2012)
  • WangZ. et al.

    Blind measurement of blocking artifacts in images

  • SazzadZ.M.P. et al.

    No-reference image quality assessment for JPEG2000 based on spatial features

    Signal Process., Image Commun.

    (2000)
  • ZhuX. et al.

    A no-reference sharpness metric sensitive to blur and noise

  • GuK. et al.

    No-reference image sharpness assessment in autoregressive parameter space

    IEEE Trans. Image Process.

    (2015)
  • JungM. et al.

    Univariant assessment of the quality of images

    J. Electron. Imaging

    (2002)
  • CharrierChristophe et al.

    A machine learning-based color image quality metric

  • MoorthyA.K. et al.

    A two-step framework for constructing blind image quality indices

    IEEE Signal Process. Lett.

    (2010)
  • LarsonE.C. et al.

    Most apparent distortion: full-reference image quality assessment and the role of strategy

    J. Electron. Imaging

    (2010)
  • VirtanenT. et al.

    CID2013: a database for evaluating no-reference image quality assessment algorithms

    IEEE Trans. Image Process.

    (2013)
  • TsoumakasG. et al.

    Tutorial on learning from multi-label data

  • Cited by (0)

    No author associated with this paper has disclosed any potential or pertinent conflicts which may be perceived to have impending conflict with this work. For full disclosure statements refer to https://doi.org/10.1016/j.image.2019.07.018.

    View full text