Learning quality assessment of retargeted images
Introduction
As the dimensions and sizes of display devices on mobile phones and tablets continues to diversify, image retargeting has become an important way to adjust original images to fit user-defined resolutions while simultaneously preserving important relationships between objects and picture content. The general idea is to shrink image regions of less importance while causing as little salient change as possible [1].
Recent image retargeting methods can be classified into two main types: discrete and continuous methods. Discrete methods including seam carving [2], [3] and shift map [4], which remove and shift pixels of an image. Continuous methods including scale-and-stretch [5] and warping [6], which are processed on a quad mesh and merge pixels of an image. Other types of approaches, such as [7], apply a sequence of operators such as seam carving, scaling and cropping, objectively optimizing each step of the process. Given the rapid growth and use of automatic image retargeting techniques, it is becoming more necessary to also develop methods to automatically ensure the high quality of the resulting retargeted images.
If the perceptual quality of an image is high, then the retargeted image should also be obtained with good quality. Thus a perceptual evaluation process is needed to aid image retargeting. While human subjective judgments of images, including retargeted images, are the most reliable assessment, these are time consuming and difficult to obtain. Although highly effective image quality models exist [8], [9], [10], [11], they require considerable modification to be able to be applied to assess the quality of retargeted images. This motivates the development of special-purpose objective retargeted image assessment models.
In [12], Rubinstein et conducted a large scale subjective study to compare eight state-of-the-art retargeting algorithms. They made algorithm comparisons against a variety of objective distance metrics, including a color layout descriptor, bidirectional warping and an edge histogram, to assess the quality of retargeted images. The authors of [13] also built a subject image retargeting database that includes 171 images produced by several representative retargeting methods on 57 natural source image contents. Each image has a mean opinion score (MOS) drawn from the subjective ratings of at least 30 viewers.
The authors of [14] measured global geometric structures and local pixel correspondences to evaluate the visual quality of retargeted images, using an objective metric. The method is top-down, organizing the features from global to local scales. The method proposed in [15] creates a SSIM [8] quality map that indicates, at each spatial location of the reference image, how well the structural information is preserved in a corresponding retargeted image. A saliency map is generated to spatially weight a computed SSIM map to estimate the visual quality of a retargeted image. In [16], three factors predictive of human judgements of the visual quality experienced when viewing retargeted images were analyzed. In [17], Hsu et al. proposed a full-reference objective metric for assessing the visual quality of a retargeted image, based on measurements of perceptual geometric distortion and information loss. This method is highly predictive of the subject quality of retargeted images, but when it is applied on images containing redundant, repeated texture patterns or very smooth areas, the SIFT flow [18] algorithm used to establish correspondences between the original and retargeted images may fail. In [19], Zhang et al. developed an aspect ratio similarity (ARS) metric to evaluate the visual quality of retargeted images by measuring local block changes. This method achieved state-of-the-art performance on two image retargeting quality assessment databases. Although many approaches have been devised for assessing the quality of retargeted images, no single method has achieved good results on the database in [12]. Given the generality and complexity of the problem, modern machine learning based methods offer a possible way to achieve acceptable performance [20].
The aesthetic value of a retargeted image is also relevant to image retargeting quality assessment. In [21], Tong et al. developed a method of image classification to identify whether photos were professional or amateur snapshots. They deployed a large set of heuristic low-level features. In [22], Murray et al. built a large scale database to assist with the development of aesthetic visual analysis models, called AVA. It contains more than 250,000 images, each supplied with a subjective aesthetic score and a semantic label from over 60 categories. The authors of [23], [24], [25] designed perceptual features which they used to create predictive algorithms for image aesthetics assessment. Ke et al. [23] proposed a principled method based on high level semantic features to determine perceptual aesthetic differences. Datta et al. [24] extracted 56 attributes from photos which they used to train an SVM classifier to classify the photos into two categories. They obtained an accuracy of approximately 70%. Cerosaletti and Loui [25] designed a few image features for image aesthetics prediction which they demonstrate on 450 photos. Jiang et al. [26] used the same data to train and verify their model. Many existing techniques for image aesthetics evaluation seek to classify photos into only two classes. Certainly, since retargeting affects image aesthetics as well as (distortion related) quality, it is of great interest to find ways to automatically assign aesthetics scores to retargeted images. The relationships between quality assessment and aesthetics evaluation on retargeted images is also potentially quite interesting. In [27], Liang et al. used aesthetics evaluation as part of a method of quality assessment on retargeted images. Only two aesthetic features were used: one descriptive of the rule of thirds, and the other of visual balance. However, many other aesthetics features have been developed that could be used in this application.
Here we describe an open framework for image retargeting quality assessment that builds on several existing quality related features, a novel aesthetics evaluation method, and a new saliency weighted CW-SSIM feature. We use these features to train a radial basis function (RBF) neural network to predict retargeted image quality. Ten of the features used as input to the neural network are the scalar outputs of independent image retargeting quality assessment methods. The experimental results show that the integrated method outperforms all of these ten methods.
The main contributions of our work are: i) We propose an open framework for image retargeting quality assessment; ii) that embodies a novel image retargeting quality assessment model which combines CW-SSIM [28], SIFT [29] and image saliency [30]; iii) and that also embodies a novel no-reference image aesthetics quality assessment method designed for retargeted images.
The rest of the paper is organized as follows. In Section 2, ten features are introduced including two important new ones, a saliency-weighted, SIFT-directed CW-SSIM feature and an integrated aesthetics feature. The radial basis function (RBF) network and the training process are described in Section 3. In Section 4, we present experimental results. The training and testing data are presented and the results of our integrated method are demonstrated and compared with existing retargeting assessment approaches. Finally, we draw conclusions in Section 5.
Section snippets
Quality-aware retargeting features
The success of any quality assessment model depends heavily on the features being used. Here we describe two new and effective features: a method of applying CW-SSIM to retargeted images by a SIFT-directed mapping process, which weights the CW-SSIM values using a saliency model, and a new method of evaluating aesthetics of retargeted images. We also briefly summarize the other eight existing retargeting assessment features.
RBF neural network
We learn the image retargeting assessment model using an RBF neural network [39], [40]. Artificial neural networks have a long history of applications in image analysis [41]. Among the various flavors of neural networks, the RBF neural network model is particularly well suited for learning to approximate continuous or piecewise continuous real-valued mappings, when the input dimension is sufficiently small.
The radial basis function network is an artificial neural network that uses radial basis
Dataset
We conducted the following experiment to examine the validity of our integrated model. We tested our integrated model on two benchmark datasets: the RetargetMe dataset [12] and the CUHK dataset [13]. In this experiment, we used the images and content in the RetargetMe and CUHK datasets.
When using the RetargetMe dataset, we chose the 37 analysis images for which users' votes are supplied. These images have one or more attributes from six major attributes: Line/Edge, Face/People, Foreground
Conclusion
We have described an open framework for image retargeting quality assessment that uses a saliency-weighted, SIFT-directed CW-SSIM model and an aesthetics evaluation model, that are combined with eight existing retargeting assessment methods. These new and old models supply features that we used to train a radial basis function network to learn to predict the quality of retargeted images. We used the images and subjective votes in the RetargetMe dataset to train and test our integrated
References (49)
- et al.
Backward registration based aspect ratio similarity (ars) for image retargeting quality assessment
IEEE Trans. Image Process.
(2016) - et al.
Objective quality prediction of image retargeting algorithms
IEEE Trans. Vis. Comput. Graph. PP
(2016) - et al.
Experiments in segmenting texton patterns using localized spatial filters
Pattern Recognit.
(1989) - et al.
Three learning phases for radial-basis-function networks
Neural Netw.
(2001) - et al.
Video quality assessment based on structural distortion measurement
Signal Process. Image Commun.
(2004) - et al.
Perceptually guided photo retargeting
IEEE Trans. Cybern. PP
(2016) - S. Avidan, A. Shamir, Seam carving for content-aware image resizing, in: ACM Transactions on Graphics (TOG), vol. 26,...
- M. Rubinstein, A. Shamir, S. Avidan, Improved seam carving for video retargeting, in: ACM Transactions on Graphics...
- Y. Pritch, E. Kav-Venaki, S. Peleg, Shift-map image editing, in: Proceedings of the 2009 IEEE 12th International...
- Y.-S. Wang, C.-L. Tai, O. Sorkine, T.-Y. Lee, Optimized scale-and-stretch for image resizing, in: ACM Transactions on...
Multi-operator media retargeting
ACM Trans. Graph. (TOG)
Image quality assessment: from error visibility to structural similarity
IEEE Trans. Image Process.
Image quality assessment using multi-method fusion
IEEE Trans. Image Process.
Automatic prediction of perceptual image and video quality
Proc. IEEE
Visual quality assessment algorithms: what does the future hold?
Multimed. Tools Appl.
Objective quality assessment for image retargeting based on structural similarity
IEEE J. Emerg. Sel. Top. Circuits Syst.
Objective quality assessment for image retargeting based on perceptual geometric distortion and information loss
IEEE J. Sel. Top. Signal Process.
Visual Quality Assessment by Machine Learning
Cited by (9)
A Comprehensive Review on Content-Aware Image Retargeting: From Classical to State-of-the-art Methods
2022, Signal ProcessingCitation Excerpt :These results were reported in benchmark datasets and related publications. According to Table 11, the metric used in [105] provides the best overall mean KRCC, and the metric in [107] gives the lowest standard deviation KRCC on the RetargetMe dataset. Since the HDPM metric [103] was not tested on this dataset, the results of this metric are not available for this dataset and are not included in Table 11.
Progress of image retargeting quality evaluation:a survey
2024, Journal of Image and GraphicsObjective quality assessment of retargeted images based on RBF neural network with structural distortion and content change
2023, Multimedia Tools and ApplicationsImage retargeting quality assessment: A survey
2023, Journal of Intelligent and Fuzzy SystemsFine-Grained Image Quality Assessment: A Revisit and Further Thinking
2022, IEEE Transactions on Circuits and Systems for Video TechnologyTransformation-Aware Similarity Measurement for Image Retargeting Quality Assessment via Bidirectional Rewarping
2021, IEEE Transactions on Systems, Man, and Cybernetics: Systems