Full-reference IPTV image quality assessment by deeply learning structural cues

Highlights

• We leverage structural cues to calculate the similarity between the original image and the test one.

• A novel distance metric is proposed to measure the difference between the reference image and each test image.

• We propose a structure-preserved deep neural networks to extract deep representation from these images.

Abstract

Image quality assessment (IQA) attempts to quantify the quality-aware visual attributes perceived by humans. They can be divided into subjective and objective image quality assessment. Subjective IQA algorithms rely on human judgment of image quality, where the human visual perception functions as the dominant factor However, they cannot be widely applied in practice due to the heavy reliance on different individuals. Motivated by the fact that objective IQA largely depends on image structural information, we propose a structural cues-based full-reference IPTV IQA algorithm. More specifically, we first design a grid-based object detection module to extract multiple structural information from both the reference IPTV image (i.e., video frame) and the test one. Afterwards, we propose a structure-preserved deep neural networks to generate the deep representation for each IPTV image. Subsequently, a new distance metric is proposed to measure the similarity between the reference image and the evaluated image. A test IPV image with a small calculated distance is considered as a high quality one. Comprehensive comparative study with the state-of-the-art IQA algorithms have shown that our method is accurate and robust.

Introduction

Image quality assessment (IQA) is an indispensable technique in computer vision and pattern recognition, such as image clustering, retrieval, compression and classification. The target of IQA is to assess image quality objectively based on some designed algorithm, which attempts to mimic human visual perception. That means the IQA score obtained by the algorithm should approximate the visual quality assessment by human beings [1]. Nowadays, IQA plays a significant role in modern advertising systems as well as photo sharing platform. For example, IQA-guided recommendation systems can display photographs or video clips according to users’ favorite content. This function replaces the traditional manual selection of various sections of website. In addition, IQA-guided image sharing platform (e.g., Flickr and Google Picasa) can intelligently construct user groups by analyzing the quality tendency of the uploaded images.

Image quality is determined by multiple factors, e.g., color distribution, illumination, and objects, within each image. For example, harmonic color distribution can reflect the high image quality, while low-resolution or illumination images typically exhibit poor quality. As shown in Fig. 1, the aesthetically pleasing illumination and color harmony will lead to a high image quality. The color hue wheels proposed by Luo et al. [2] can quantify the color harmony within each image. They argue that an image can be considered as harmonic if its hue overlaps most regions in the pre-defined hue wheels. In their work, seven different hue wheels were proposed, each of which represents a particular color harmony sample. The hue of image is related to illumination and color matching. Images with harmonic hue are deemed as high quality.

In general, images with various visual distortion have low quality score, e.g., images with the Gaussian blur, salt-pepper impulsive noise. Subjective IQA can conveniently distinguish different quality scores of images. However, this is still a challenge task for objective IQA algorithms which aim to design mathematical models that mimic human visual perception of different images. Realizing that structural cues within each image are perceived by humans when they viewing scenes, we consider that structural cues play the most important role in IQA. As far as we know, however, such informative attribute has not been explicitly encoded. Besides, some other limitations of the conventional IQA algorithms are as follows:

• Existing designed mathematical model cannot reflect human visual perception effectively. Thus, the results of objective IQA algorithms are far from understanding human visual perception.

• A carefully-designed distance metric is significant to IQA, since the inherent differences between the distorted images and the original one might be difficult to distinguish. Nevertheless, conventional IQA methods cannot exploit distance metric effectively.

To solve these limitations, we propose a structural cues based full-reference IQA algorithm. Specifically, we first propose a grid-based object detection algorithm to extract structural clues from both the reference image and the evaluated one. Afterwards, we propose a structure-reservation based deep neural network to generate the deep feature. Subsequently, a new distance metric is constructed to calculate similarity between reference image and evaluated image. A test image with a small calculated distance is deemed as high quality and thus will be assigned with a high score. Extensive experimental results on popular IQA databases have demonstrated the usefulness of our method. The main contributions of our proposed IQA method are as follows: (1) We leverage structural cues to calculate the similarity between the original image and the test one, where both local and global image features are exploited. (2) A novel distance metric is proposed to measure the difference between the reference image and each test image. (3) We propose a structure-preserved deep neural networks to extract deep representation from these images, where image structural feature can be well preserved.