Content-oriented image quality assessment with multi-label SVM classifier☆
Introduction
Recently the diversity and multi-functionality of the displays have increased the demands of more advanced image processing methods, such that the quality evaluation and improvement of digital images during the image processing become vital. Subjective testing is the most reliable approach to evaluate the quality of images, and numerous image quality assessment (IQA) databases, including LIVE [1] and TID [2], [3], have been established. However, subjective testing is time-consuming and expensive, which inspires the objective IQA methods that can automatically evaluate the image quality with computational models.
Objective IQA aims to predict the image quality and perform consistently with the human visual system (HVS), and it is widely used in image processing, such as acquisition, compression, transmission, and enhancement [4]. Depending on the availability of the pristine image, the objective IQA methods can be classified into three types: full-reference (FR), reduced-reference (RR) and no-reference (NR). FR-IQA means that the reference image is available during the quality prediction [5], while only some features of the reference image can be utilized for RR-IQA methods [6]. NR-IQA methods evaluate image quality without any information of the corresponding reference image. Among these methods, NR-IQA is regarded as the most difficult task as the reference image is unavailable. As such, the distortion-type aware IQA has been intensively studied to obtain the reliable quality score. Generally speaking, NR-IQA methods can be categorized into one or the combination of these three categories: (1) Natural scene statistics (NSS) approaches, which are commonly based on the hypothesis that images of the natural world occupy a small subspace of all possible images, leading to the method of finding the distance between the test image and the subspace of natural world [7], [8], [9], [10]; (2) Distortion-specific approaches: which seek to use a specific distortion model to measure the perceptual quality [11], [12], [13], [14]; (3) Training-based approaches: rely on extracting some features from an image, based on which a quality assessment model is learned to predict the image quality [15], [16], [17] (see Table 1).
In essence, the existing databases consist of different kinds of images, such as handcrafted scenes, natural images, animals and human beings, and a number of distortion types including blur, white noise, JPEG, and JPEG2K are injected to generate the distorted image. However, different kinds of images always have different characteristics, which have not been explicitly investigated in the literature. For example, handcrafted images rely on sharp edges to characterize the object information. For the natural scene, textures are more apparent and exhibit strong masking effect. The quality assessment model should be able to adapt to the image content type as well. This motivates us to explore the image quality from the perspective of image content type, and a new IQA database based on the classification of content types is built. In particular, this database contains four types of images: landscape, handcrafted scene, human face and hybrid (based on the combination of the first three types), with 20 reference images for each content type. Five distortion types including Gaussian blur, HEVC compression, JPEG compression, JPEG2000 compression and white Gaussian noise with four distortion levels are considered in the construction of the database, resulting in 80 reference images and 1600 distorted images in total. To the best of our knowledge, this is the first IQA database that considers the content type classification.
There are two parts in our method: (1) a classifier to classify the images based on content, (2) IQA models to predict quality for the images in our proposed database. Since the images in hybrid type have multiple labels (i.e., handcrafted scene, natural scene, and human being), we propose to use multi-label (ML) learning for the classification. The task of ML learning is to construct models based on ML training set, in order to accurately predict the label sets of testing samples [20]. Usually, there are two categories of ML learning algorithms: (1) problem transformation methods, and (2) algorithm adaption methods. A problem transformation method transforms the original ML problem into a set of single-label problems. Compared with algorithm adaption method, it is easy to implement and can be achieved with any single-label learner. Thus, in this paper, we adopt problem transformation method, and achieve it with binary relevance (BR) strategy, where support vector machine (SVM) is used as the base classifier. Finally, a SVM-based ML model is constructed to classify the images in the Content-oriented Database before testing on IQA metrics.
The rest of this paper is organized as follows. In Section 2, the related works are discussed. In Section 3, we describe the construction of the IQA database. The ML model to classify image types is presented in Section 4. Validations by considering the image type as an attribute of IQA are provided in Section 5, and we conclude this paper in Section 6.
Section snippets
Existing IQA databases
There exists numerous IQA databases which are widely applied to evaluate the accuracy of IQA models. In 2006, Sheikh et al. proposed the LIVE database in [1], which was extensively adopted to evaluate and analyze the performance of IQA algorithms. The LIVE database consists of 29 reference and 779 distorted images with five distortion types such as Gaussian blur, white Gaussian noise, JPEG, JPEG2000 and fast fading transmission error. More than 25000 individual human quality judgments were used
Description of the database
The new database includes 80 reference images with four types of image content, including face, handcrafted scene, landscape and the hybrid scene with the previous three categories. The reason why we just use these four classes is that our aim is to build the new database to investigate the influence of the image content on the IQA performance. These four classes of the images include indoor and outdoor scenes, human beings, and many kinds of artificial products, which are sufficient to verify
SVM-based ML model
In order to classify the images in the database, we adopt SVM-based ML learning algorithm. As the most commonly used strategy, BR is utilized, which constructs a binary classifier for each decision component, and gets the final results by combining the predictions of the multiple classifiers. Suppose , , …, denotes a label set with possible class labels and denotes the -dimensional sample space. As a remark, we have M3 in this paper, and , , correspond to face,
Experimental results
In this section, extensive experimental results are provided to verify the influence of image content on image quality assessment. Before testing the IQA algorithms on the Content-oriented Database, we first automatically classify the testing images. We use SVM as our single-label base classifier, and adopt a BR strategy to achieve SVM-based ML classification. The experiments for classification and IQA algorithms are carried out under MATLAB R2016a, which are executed on a computer with
Conclusion
We have described the content-oriented database, which is among of the largest databases with subjective testing for IQA research. In order to test the performance of IQA models, we use images with three kinds of content and combine them together. Before testing IQA performance, we first classify the testing images of the content-oriented database by using a SVM-based ML classifier. Then the IQA models that trained on each content based sub-database are used to predict the quality of images in
Acknowledgments
This work was supported in part by Hong Kong RGC General Research Fund 9042489 under Grant CityU 11206317, in part by Hong Kong RGC General Research Fund 9042322 under Grant CityU 11200116, in part by the Hong Kong RGC Early Career Scheme under Grant 9048122 (CityU 21211018), in part by the City University of Hong Kong under Grant 7200539/CS and in part by the National Natural Science Foundation of China under Grant 61772344 and Grant 61811530324.
References (50)
- et al.
Image database TID2013: Peculiarities, results and perspectives
Signal Process., Image Commun.
(2015) - et al.
MDID: a multiply distorted image database for image quality assessment
Pattern Recognit.
(2017) - et al.
Learning multi-label scene classification
Pattern Recognit.
(2004) - et al.
A comparative study of texture measures with classification based on featured distributions
Pattern Recognit.
(1996) - et al.
Learning a blind quality evaluation engine of screen content images
Neurocomputing
(2016) - et al.
Blind image quality assessment by relative gradient statistics and adaboosting neural network
Signal Process., Image Commun.
(2016) - et al.
A statistical evaluation of recent full reference image quality assessment algorithms
IEEE Trans. Image Process.
(2006) - et al.
TID2008 - A database for evaluation of full-reference visual quality assessment metrics
Adv. Mod. Radioelectron.
(2009) - et al.
Modern image quality assessment
Synth. Lect. Image Video Multimedia Process.
(2006) Visible differences predictor: an algorithm for the assessment of image fidelity
Quality-aware images
IEEE Trans. Image Process.
A two-step framework for constructing blind image quality indices
IEEE Signal Process. Lett.
Blind image quality assessment: From natural scene statistics to perceptual quality
IEEE Trans. Image Process.
Model-based referenceless quality metric of 3D synthesized images using local image description
IEEE Trans. Image Process.
Blind image quality assessment: a natural scene statistics approach in the DCT domain
IEEE Trans. Image Process. Publ. IEEE Signal Process. Soc.
Blind measurement of blocking artifacts in images
No-reference image quality assessment for JPEG2000 based on spatial features
Signal Process., Image Commun.
A no-reference sharpness metric sensitive to blur and noise
No-reference image sharpness assessment in autoregressive parameter space
IEEE Trans. Image Process.
Univariant assessment of the quality of images
J. Electron. Imaging
A machine learning-based color image quality metric
A two-step framework for constructing blind image quality indices
IEEE Signal Process. Lett.
Most apparent distortion: full-reference image quality assessment and the role of strategy
J. Electron. Imaging
CID2013: a database for evaluating no-reference image quality assessment algorithms
IEEE Trans. Image Process.
Tutorial on learning from multi-label data
Cited by (0)
- ☆
No author associated with this paper has disclosed any potential or pertinent conflicts which may be perceived to have impending conflict with this work. For full disclosure statements refer to https://doi.org/10.1016/j.image.2019.07.018.