Elsevier

Pattern Recognition Letters

Volume 63, 1 October 2015, Pages 66-70
Pattern Recognition Letters

A flexible framework of adaptive method selection for image saliency detection

https://doi.org/10.1016/j.patrec.2015.06.012Get rights and content

Highlights

  • A flexible framework of adaptive method selection is proposed.

  • The approach adaptively selects the best candidate method for new testing instance.

  • It is more efficient than traditional aggregation ways.

  • It yields competitive results on saliency detection datasets.

Abstract

For most of the data analysis tasks (e.g. visual saliency detection), there are usually plenty of candidate methods to be selected. However, it is very difficult to choose a proper one for new instances, especially when the performances of these methods are with little difference overall. Though aggregation strategy aims to take advantage of the different methods, it often has the following weaknesses. Firstly, these methods often tend to combine the results from these candidate methods. Therefore, they suffer from high computation cost. Secondly, the performance may significantly degrade when there are obviously poor results. To address the two limitations above, we propose an instance-aware method selection approach which aims to select a single method instead of aggregating the results of all candidate ones. The proposed approach is based on the following observations: different methods often perform differently and the performance of a method often varies with respect to different instances. Hence, we devise the method selection manner to adaptively choose the best method for a specific instance. We transform the method selection problem into a multi-label annotation problem, which makes it general for many applications and flexible to employ metric learning technique.

Introduction

A large number of data analysis approaches for various applications (e.g., visual saliency detection and image segmentation) have been developed recently. However, it is usually difficult to choose a proper one for new instances, especially when the overall performances of these methods are not that different from each other on the training data. Typically, one method may outperform the others on one dataset, but degrade on another dataset. The intrinsic reason is the performance of an analysis method often varies with individual instances. Though one method may achieve the best performance on the whole dataset, it may be not the best one for every instance.

In this paper, we take the saliency detection [8] as an example, which is used to discover the visual saliency on the images. To exploit the multiple saliency methods, the aggregation strategy is proposed by [12]. The combination of saliency analysis methods can achieve a better performance than a single method. However, the aggregation way often suffers from the following two main limitations. First, the aggregation-based methods need to conduct all of these candidate methods in advance, consequently weights each saliency map corresponding to each method to obtain the final saliency map. Therefore, it meets high computation cost. Second, as shown in [12], if there are some relatively bad results, the traditional aggregation method may be badly affected by them. Based on the above observations, we present a novel framework, taking both the effectiveness and efficiency into account simultaneously. The motivation of our method is that a method tends to work well for the similar instances. As shown in Fig. 1, images A and B are very similar and the same method always achieves the similar performance on them. Furthermore, the best results of image A and B are both from the same method HC [4]. On the other hand, we argue that there exists the most suitable processing method for a test instance. As shown in Fig. 1, the best method for image C is the method CA [5], though the method is not better than HC on average. Therefore, if we choose the best method for the test instances adaptively, then we do not need to pre-process the instances with all of the candidate methods and hence, the computation cost will be reduced with guaranteed performance.

The key contributions of this paper are highlighted as follows: First, to well balance the effectiveness and efficiency, we propose a general framework for adaptively selecting a single method for a new instance, which executes in the data-driven manner with the learned metric. Second, our framework transforms the method selection task into a multi-label annotation problem, which allows flexible application of large amount of machine learning techniques. Third, we have conducted extensive experiments on the saliency detection task to demonstrate the effectiveness of the proposed method.

The rest of this paper is organized as follows: In Section 2, we give a formal description of the proposed framework. We report quantitative experiment results in Section 3. Finally, we conclude the paper in Section 4.

Section snippets

Framework overview

Fig. 2 gives the architecture of our framework. First, the training image xi is pre-processed by M candidate saliency detection methods, then M saliency maps {Sim}m=1M are obtained. Based on these saliency maps and the ground truth, we calculate the AUC score (or F-measure) for each saliency map to get a real-value vector zi for the image xi. The score zim in the vector zi indicates the fitness of the mth method for the specific image xi. We consider the method corresponding to the highest AUC

Data set & methods

We test our algorithm on two benchmarks: MSRA-B [10], [9], and CMU-Cornell iCoseg dataset [2]. The first one is mainly used for single image saliency detection and the other is used for co-saliency detection. For training, we use the transfer learning strategy. That is to say, we learn the distance metrics on one dataset and use the learned metrics on another one in test stage. For evaluation, we use the leave-one-out way as in [12]. That is to say, for a test image, we use the rest of the

Conclusion

In this paper, we have proposed a framework for adaptive method selection. Unlike the aggregation strategy, our framework balances well the efficiency and effectiveness. By proper transformation, the method selection task is transformed into the multi-label annotation one, which is a well-studied field. We introduce the retrieve-based metric learning algorithm, which effectively exploits the Mahalanobis distance for similarity measure. We have conducted extensive empirical studies on two image

Acknowledgement

This work was supported by National Natural Science Foundation of China (No. 61422213, 61332012), National Basic Research Program of China (2013CB329305), National High-tech R&D Program of China (2014BAK11B03), and 100 Talents Programme of the Chinese Academy of Sciences.

References (22)

  • X. Wen et al.

    A rapid learning algorithm for vehicle classification

    Inform. Sciences

    (2015)
  • R. Achanta et al.

    Frequency-tuned saliency detection model

    (2009)
  • D. Batra et al.

    Interactively co-segmentating topically related images with intelligent scribble guidance

    Int. J. Comput. Vis.

    (2011)
  • M. Bilenko et al.

    Integrating constraints and metric learning in semi-supervised clustering

    (2004)
  • M.-M. Cheng et al.

    Global contrast based salient region detection

    (2011)
  • S. Goferman et al.

    Context-aware saliency detection

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2012)
  • J. Harel et al.

    Graph-based visual saliency

    (2006)
  • X. Hou et al.

    Saliency detection: a spectral residual approach

    (2007)
  • L. Itti et al.

    A model of saliency-based visual attention for rapid scene analysis

    IEEE Trans. Pattern Anal. Mach. Intell.

    (1998)
  • H. Jiang et al.

    Salient object detection: a discriminative regional feature integration approach

    (2013)
  • T. Liu et al.

    Learning to detect a salient object

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2011)
  • Cited by (10)

    • Saliency detection based on background seeds by object proposals and extended random walk

      2018, Journal of Visual Communication and Image Representation
      Citation Excerpt :

      Later, a decomposition model was developed to increase the precision of the saliency-detection task, by analyzing high-dimensional tensor-like visual data [45]. In [46], an instance-aware model was devised for image-saliency detection, which aims to adaptively select the best method for a specific instance. Recently, Li et al. [47] developed a sparse and double low-rank decomposition framework (SDLRD) to formulate saliency fusion, as a subspace decomposition problem.

    • Grey wolf optimization on modeling saliency detection in stereoscopic 3D images

      2019, International Journal of Recent Technology and Engineering
    View all citing articles on Scopus

    This paper has been recommended for acceptance by Egon L. van den Broek.

    View full text