Learning quality assessment of retargeted images

Highlights

• Propose an open framework for image retargeting quality assessment.

• Embody a novel image retargeting quality assessment model which combines CW-SSIM, SIFT and image saliency.

• Embody a novel no-reference image aesthetics quality assessment method for retargeted images.

Abstract

Content-aware image resizing (or image retargeting) enables images to be fit to different display devices having different aspect ratios while preserving salient image content. There are many approaches to retargeting, although no “best” method has been agreed upon. Therefore, finding ways to assess the quality of image retargeting has become a prominent challenge. Traditional image quality assessment methods are not directly applicable to image retargeting because the retargeted image size is not same as the original one. In this paper, we propose an open framework for image retargeting quality assessment, where the quality prediction engine is a trained Radial Basis Function (RBF) neural network. Broadly, our approach is motivated by the observation that no single method can be expected to perform well on all types of content. We train the network on ten perceptually relevant features, including a saliency-weighted, SIFT-directed complex wavelet structural similarity (CW-SSIM) index, and a new image aesthetics evaluation method. These two features and eight other features are used by the neural network to learn to assess the quality of retargeted images. The accuracy of the new model is extensively verified by simulations.

Introduction

As the dimensions and sizes of display devices on mobile phones and tablets continues to diversify, image retargeting has become an important way to adjust original images to fit user-defined resolutions while simultaneously preserving important relationships between objects and picture content. The general idea is to shrink image regions of less importance while causing as little salient change as possible [1].

Recent image retargeting methods can be classified into two main types: discrete and continuous methods. Discrete methods including seam carving [2], [3] and shift map [4], which remove and shift pixels of an image. Continuous methods including scale-and-stretch [5] and warping [6], which are processed on a quad mesh and merge pixels of an image. Other types of approaches, such as [7], apply a sequence of operators such as seam carving, scaling and cropping, objectively optimizing each step of the process. Given the rapid growth and use of automatic image retargeting techniques, it is becoming more necessary to also develop methods to automatically ensure the high quality of the resulting retargeted images.

If the perceptual quality of an image is high, then the retargeted image should also be obtained with good quality. Thus a perceptual evaluation process is needed to aid image retargeting. While human subjective judgments of images, including retargeted images, are the most reliable assessment, these are time consuming and difficult to obtain. Although highly effective image quality models exist [8], [9], [10], [11], they require considerable modification to be able to be applied to assess the quality of retargeted images. This motivates the development of special-purpose objective retargeted image assessment models.

In [12], Rubinstein et $\mathit{al}.$ conducted a large scale subjective study to compare eight state-of-the-art retargeting algorithms. They made algorithm comparisons against a variety of objective distance metrics, including a color layout descriptor, bidirectional warping and an edge histogram, to assess the quality of retargeted images. The authors of [13] also built a subject image retargeting database that includes 171 images produced by several representative retargeting methods on 57 natural source image contents. Each image has a mean opinion score (MOS) drawn from the subjective ratings of at least 30 viewers.

The authors of [14] measured global geometric structures and local pixel correspondences to evaluate the visual quality of retargeted images, using an objective metric. The method is top-down, organizing the features from global to local scales. The method proposed in [15] creates a SSIM [8] quality map that indicates, at each spatial location of the reference image, how well the structural information is preserved in a corresponding retargeted image. A saliency map is generated to spatially weight a computed SSIM map to estimate the visual quality of a retargeted image. In [16], three factors predictive of human judgements of the visual quality experienced when viewing retargeted images were analyzed. In [17], Hsu et al. proposed a full-reference objective metric for assessing the visual quality of a retargeted image, based on measurements of perceptual geometric distortion and information loss. This method is highly predictive of the subject quality of retargeted images, but when it is applied on images containing redundant, repeated texture patterns or very smooth areas, the SIFT flow [18] algorithm used to establish correspondences between the original and retargeted images may fail. In [19], Zhang et al. developed an aspect ratio similarity (ARS) metric to evaluate the visual quality of retargeted images by measuring local block changes. This method achieved state-of-the-art performance on two image retargeting quality assessment databases. Although many approaches have been devised for assessing the quality of retargeted images, no single method has achieved good results on the database in [12]. Given the generality and complexity of the problem, modern machine learning based methods offer a possible way to achieve acceptable performance [20].

The aesthetic value of a retargeted image is also relevant to image retargeting quality assessment. In [21], Tong et al. developed a method of image classification to identify whether photos were professional or amateur snapshots. They deployed a large set of heuristic low-level features. In [22], Murray et al. built a large scale database to assist with the development of aesthetic visual analysis models, called AVA. It contains more than 250,000 images, each supplied with a subjective aesthetic score and a semantic label from over 60 categories. The authors of [23], [24], [25] designed perceptual features which they used to create predictive algorithms for image aesthetics assessment. Ke et al. [23] proposed a principled method based on high level semantic features to determine perceptual aesthetic differences. Datta et al. [24] extracted 56 attributes from photos which they used to train an SVM classifier to classify the photos into two categories. They obtained an accuracy of approximately 70%. Cerosaletti and Loui [25] designed a few image features for image aesthetics prediction which they demonstrate on 450 photos. Jiang et al. [26] used the same data to train and verify their model. Many existing techniques for image aesthetics evaluation seek to classify photos into only two classes. Certainly, since retargeting affects image aesthetics as well as (distortion related) quality, it is of great interest to find ways to automatically assign aesthetics scores to retargeted images. The relationships between quality assessment and aesthetics evaluation on retargeted images is also potentially quite interesting. In [27], Liang et al. used aesthetics evaluation as part of a method of quality assessment on retargeted images. Only two aesthetic features were used: one descriptive of the rule of thirds, and the other of visual balance. However, many other aesthetics features have been developed that could be used in this application.

Here we describe an open framework for image retargeting quality assessment that builds on several existing quality related features, a novel aesthetics evaluation method, and a new saliency weighted CW-SSIM feature. We use these features to train a radial basis function (RBF) neural network to predict retargeted image quality. Ten of the features used as input to the neural network are the scalar outputs of independent image retargeting quality assessment methods. The experimental results show that the integrated method outperforms all of these ten methods.

The main contributions of our work are: i) We propose an open framework for image retargeting quality assessment; ii) that embodies a novel image retargeting quality assessment model which combines CW-SSIM [28], SIFT [29] and image saliency [30]; iii) and that also embodies a novel no-reference image aesthetics quality assessment method designed for retargeted images.

The rest of the paper is organized as follows. In Section 2, ten features are introduced including two important new ones, a saliency-weighted, SIFT-directed CW-SSIM feature and an integrated aesthetics feature. The radial basis function (RBF) network and the training process are described in Section 3. In Section 4, we present experimental results. The training and testing data are presented and the results of our integrated method are demonstrated and compared with existing retargeting assessment approaches. Finally, we draw conclusions in Section 5.