Saliency based multiple object cosegmentation by ensemble MIML learning

Abstract

As an interesting and emerging topic, multiple foreground cosegmentation (MFC) aims at extracting a finite number of common objects from an image collection, which is useful to variety of visual media applications. Although a number of approaches have been proposed to address this problem, many of them are designed with the misleading consistent information, suboptimal image representation, or inefficient segmentation assist and thus still suffer from certain limitations, which reduces their capability in the real-world scenarios. To alleviate these limitations, we propose a novel unsupervised MFC framework, which is composed of three components: unsupervised label generation, saliency based pseudo-annotation and cosegmentation by MIML learning. Specifically, we combine the high-level and low-level feature to represent the proposal objects, and adopt a novel SPAP clustering scheme to obtain more accurate consistent information of common objects. Then the saliency based pseudo-annotation help us reformulate the MFC problem as a Multi-Instance Multi-Label (MIML) learning problem by label propagation. Finally, by introducing a novel ensemble MIML learning scheme, the consistent information of common objects can more efficiently assist the segmentation of the images and get the more accurate segmentation results. We evaluate our framework on widely used public databases including the ICoseg dataset, MSRC dataset and FlickrMFC dataset for single and multiple common object cosegmentation respectively. Comparison results show that the proposed methods reach advanced and efficient performance.

Appendix: : SPAP clustering algorithm

The SPAP clustering algorithm we used is a novel clustering algorithm which is based on spectral clustering method. Generally, given a dataset Z = (z₁,z₂,z₃...z_n),n is the data number of this dataset. To use spectral clustering, we should first get its similarity matrix by Radial basis function kernel:

$$ M_{ij} = exp\left( \frac{ || z_{i} - z_{j} ||^{2}_{2} }{ 2\sigma^{2} }\right) $$

(9)

M is similarity matrix, $|| z_{i} - z_{j} ||^{2}_{2}$ is Euclidean distance between z_i and z_j,σ is a free parameter and we set σ = 1 here. According similarity matrix M, we can then get its adjacency matrix W, degree matrix D, and Laplacian matrix L by L = D − W. After having Laplacian matrix L, we then construct a standardized Laplacian matrix. We use

$$ L_{s} = D^{-1/2}LD^{-1/2} $$

(10)

to get standardized Laplacian matrix L_s. Then we can get the most m minimum eigenvalues of L_s. By the way, each eigenvalue has an eigenvector f and the shape of f is (n, 1). Finally, we can combine these m eigenvectors to construct an n × m matrix F and we standardize F row by row to get final eigenmatrix FS.

Normally, in spectral clustering, the k-means algorithm are used on matrix FS to get clustering result. The k-means algorithm requires the number of clusters. However, the group number of all common objects are unknown in practice. Hence, we use Affinity propagation to replace k-means, which does not need the number of clusters before clustering. Affinity propagation has the concept of similarity, responsibility, and availability to together determine whether one point can be the center. These points are called exemplars. First of all, similaritys(i,k) indicates how well the data point with index k is suited to be the exemplar for data point i. Obviously, k means the row k of eigenmatrix FS. When the goal is to minimize squared error, each similarity is set to a negative squared error(Euclidean distance): for point FS_i and $FS_{k}, s(i,k) = || FS_{i} - FS_{k} ||^{2}_{2}$. Alternatively, when appropriate, similarities may be set by hand. Because the similarity describes each point’s ability whether it can be chosen as an exemplar, in the beginning, we set each s(i,i) the same value. Then, the responsibility r(i,k), sent from data point i to candidate exemplar point k, reflects the accumulated evidence for how well-suited point k is to serve as the exemplar for point i, taking into account other potential exemplars for point i. The availability a(i,k), sent from candidate exemplar point k to point i, reflects the accumulated evidence for how appropriate it would be for point i to choose point k as its exemplar, taking into account the support from other points that point k should be an exemplar. To begin with, the availabilities are initialized to zero: a(i,k) = 0. Then, the responsibilities are computed using the rule:

$$ r(i,k) \gets s(i,k) - \max\limits_{k^{\prime}s.t k^{\prime} \not= k}(\ a(i,k^{\prime} ) + s(i,k^{\prime} )\ ) $$

(11)

Whereas the above responsibility update lets all candidate exemplars compete for ownership of a data point, the following availability update gathers evidence from data points as to whether each candidate exemplar would make a good exemplar:

$$ \begin{array}{@{}rcl@{}} &&a(i,k) \gets s(i,k) - min(\ 0,r(k,k)\\ &&+\sum\limits_{(i^{\prime} s.t. i^{\prime} \not \in(i,k)}max(\ 0,r(i^{\prime},k) \ ) \ )\ ) \end{array} $$

(12)

According to the above rules, if the self-responsibilityr(k,k) is negative, the availability of point k as an exemplar can be increased if some other points have positive responsibilities for point k being their exemplar. To limit the influence of strong incoming positive responsibilities, the total sum is threshold so that it cannot go above zero. So the self-availabilitya(k,k) is updated differently:

$$ a(k,k) \gets \sum\limits_{(i^{\prime} s.t. i^{\prime} \not \in(i,k)}max(\ 0,r(i^{\prime},k)\ ) $$

(13)

Hence, after some iterations , availability and responsibility can be combined to identify exemplars. For point k, the value of that maximizes r(i,k) + a(i,k) either identifies point k as an exemplar if k = i, or identifies the data point is the exemplar for point i. The whole SPAP method is shown in Algorithm 1.