A flexible framework of adaptive method selection for image saliency detection
Introduction
A large number of data analysis approaches for various applications (e.g., visual saliency detection and image segmentation) have been developed recently. However, it is usually difficult to choose a proper one for new instances, especially when the overall performances of these methods are not that different from each other on the training data. Typically, one method may outperform the others on one dataset, but degrade on another dataset. The intrinsic reason is the performance of an analysis method often varies with individual instances. Though one method may achieve the best performance on the whole dataset, it may be not the best one for every instance.
In this paper, we take the saliency detection [8] as an example, which is used to discover the visual saliency on the images. To exploit the multiple saliency methods, the aggregation strategy is proposed by [12]. The combination of saliency analysis methods can achieve a better performance than a single method. However, the aggregation way often suffers from the following two main limitations. First, the aggregation-based methods need to conduct all of these candidate methods in advance, consequently weights each saliency map corresponding to each method to obtain the final saliency map. Therefore, it meets high computation cost. Second, as shown in [12], if there are some relatively bad results, the traditional aggregation method may be badly affected by them. Based on the above observations, we present a novel framework, taking both the effectiveness and efficiency into account simultaneously. The motivation of our method is that a method tends to work well for the similar instances. As shown in Fig. 1, images A and B are very similar and the same method always achieves the similar performance on them. Furthermore, the best results of image A and B are both from the same method HC [4]. On the other hand, we argue that there exists the most suitable processing method for a test instance. As shown in Fig. 1, the best method for image C is the method CA [5], though the method is not better than HC on average. Therefore, if we choose the best method for the test instances adaptively, then we do not need to pre-process the instances with all of the candidate methods and hence, the computation cost will be reduced with guaranteed performance.
The key contributions of this paper are highlighted as follows: First, to well balance the effectiveness and efficiency, we propose a general framework for adaptively selecting a single method for a new instance, which executes in the data-driven manner with the learned metric. Second, our framework transforms the method selection task into a multi-label annotation problem, which allows flexible application of large amount of machine learning techniques. Third, we have conducted extensive experiments on the saliency detection task to demonstrate the effectiveness of the proposed method.
The rest of this paper is organized as follows: In Section 2, we give a formal description of the proposed framework. We report quantitative experiment results in Section 3. Finally, we conclude the paper in Section 4.
Section snippets
Framework overview
Fig. 2 gives the architecture of our framework. First, the training image xi is pre-processed by M candidate saliency detection methods, then M saliency maps are obtained. Based on these saliency maps and the ground truth, we calculate the AUC score (or F-measure) for each saliency map to get a real-value vector zi for the image xi. The score in the vector zi indicates the fitness of the mth method for the specific image xi. We consider the method corresponding to the highest AUC
Data set & methods
We test our algorithm on two benchmarks: MSRA-B [10], [9], and CMU-Cornell iCoseg dataset [2]. The first one is mainly used for single image saliency detection and the other is used for co-saliency detection. For training, we use the transfer learning strategy. That is to say, we learn the distance metrics on one dataset and use the learned metrics on another one in test stage. For evaluation, we use the leave-one-out way as in [12]. That is to say, for a test image, we use the rest of the
Conclusion
In this paper, we have proposed a framework for adaptive method selection. Unlike the aggregation strategy, our framework balances well the efficiency and effectiveness. By proper transformation, the method selection task is transformed into the multi-label annotation one, which is a well-studied field. We introduce the retrieve-based metric learning algorithm, which effectively exploits the Mahalanobis distance for similarity measure. We have conducted extensive empirical studies on two image
Acknowledgement
This work was supported by National Natural Science Foundation of China (No. 61422213, 61332012), National Basic Research Program of China (2013CB329305), National High-tech R&D Program of China (2014BAK11B03), and 100 Talents Programme of the Chinese Academy of Sciences.
References (22)
- et al.
A rapid learning algorithm for vehicle classification
Inform. Sciences
(2015) - et al.
Frequency-tuned saliency detection model
(2009) - et al.
Interactively co-segmentating topically related images with intelligent scribble guidance
Int. J. Comput. Vis.
(2011) - et al.
Integrating constraints and metric learning in semi-supervised clustering
(2004) - et al.
Global contrast based salient region detection
(2011) - et al.
Context-aware saliency detection
IEEE Trans. Pattern Anal. Mach. Intell.
(2012) - et al.
Graph-based visual saliency
(2006) - et al.
Saliency detection: a spectral residual approach
(2007) - et al.
A model of saliency-based visual attention for rapid scene analysis
IEEE Trans. Pattern Anal. Mach. Intell.
(1998) - et al.
Salient object detection: a discriminative regional feature integration approach
(2013)
Learning to detect a salient object
IEEE Trans. Pattern Anal. Mach. Intell.
Cited by (10)
Saliency detection based on background seeds by object proposals and extended random walk
2018, Journal of Visual Communication and Image RepresentationCitation Excerpt :Later, a decomposition model was developed to increase the precision of the saliency-detection task, by analyzing high-dimensional tensor-like visual data [45]. In [46], an instance-aware model was devised for image-saliency detection, which aims to adaptively select the best method for a specific instance. Recently, Li et al. [47] developed a sparse and double low-rank decomposition framework (SDLRD) to formulate saliency fusion, as a subspace decomposition problem.
Automatic image segmentation method for cotton leaves with disease under natural environment
2018, Journal of Integrative AgricultureShort time fourier transform with coefficient optimization for detecting salient regions in stereoscopic 3D images: GSDU
2020, Multimedia Tools and ApplicationsVideo saliency detection via sparsity-based reconstruction and propagation
2019, IEEE Transactions on Image ProcessingSG-FCN: A Motion and Memory-Based Deep Learning Model for Video Saliency Detection
2019, IEEE Transactions on CyberneticsGrey wolf optimization on modeling saliency detection in stereoscopic 3D images
2019, International Journal of Recent Technology and Engineering
This paper has been recommended for acceptance by Egon L. van den Broek.