Multi-view spectral clustering via integrating nonnegative embedding and spectral embedding

Highlights

• The nonnegative embedding and spectral embedding are integrated into a unified formulation.

• A parameter-free model is proposed for multi-view clustering.

• An efficient optimization method based on Majorization-Minimization is developed to solve the involved objective.

Abstract

The application of most existing multi-view spectral clustering methods is generally limited by the following three deficiencies. First, the requirement to post-processing, such as K-means or spectral rotation. Second, the susceptibility to parameter selection. Third, the high computation cost. To this end, in this paper we develop a novel method that integrates nonnegative embedding and spectral embedding into a unified framework. Two promising advantages of proposed method include 1) the learned nonnegative embedding directly reveals the consistent clustering result, such that the uncertainty brought by post-processing can be avoided; 2) the involved model is parameter-free, which makes our method more applicable than existing algorithms that introduce many additional parameters. Furthermore, we develop an efficient inexact Majorization-Minimization method to solve the involved model which is non-convex and non-smooth. Experiments on multiple benchmark datasets demonstrate that our method achieves state-of-the-art performance.

Introduction

Clustering is a fundamental problem that arises in many fields, including data mining, computer vision, and machine learning. Conceptually, given n samples, the goal of clustering is to partition them into k subsets. In general, for each sample one can describe it from different views, and leveraging the information of multiple views simultaneously is beneficial for achieving a better clustering result. Such a problem refers to multi-view clustering. Most existing approaches for dealing with this issue can be roughly divided into four groups, including graph-based methods [1], [2], [3], [4], [5], [6], matrix factorization methods [7], [8], [9], [10], [11], multiple kernel-based methods [12], [13], [14], and subspace learning-based methods [15], [16]. In particular, due to the ability in digging non-linear structure information, graph-based methods generally outperform others in terms of clustering precision.

Different graph-based methods usually vary in how they learn the consistent spectral embedding. One simple alternative is multi-view spectral clustering that directly uses multiple graphs, constructed by k-nearest neighbors (or other similar approaches), to learn a consistent spectral embedding [1], [2], [17]. However, the noise levels of different views are generally different, which implies that the quality difference exists in different graphs. To alleviate this issue, multi-view subspace clustering [18], [19], [20] searches for first learning a consistent similarity matrix, i.e., a graph from multiple views, and then obtaining the final spectral embedding via traditional single-view spectral embedding methods, such as Ncut.

There are three deficiencies that are usually encountered in graph-based methods. First, the requirement to post-processing. For graph-based methods, the final clustering result is generally returned by conducting K-means or spectral rotation to consistent spectral embedding, which inevitably introduce the uncertainty caused by initialization. Second, the susceptibility to parameter selection. The models of most existing graph-based methods introduce some additional parameters, while parameter selection is not an easy thing for clustering, an unsupervised task. Third, the high computation cost. Eigenvalue decomposition with computation complexity $\mathcal{O}\left(n^3\right)$ is required by most multi-view spectral clustering methods, and matrix inversion with computation complexity $\mathcal{O}\left(n^3\right)$ is required by most multi-view subspace clustering methods when solve the involved models. Both eigenvalue decomposition and matrix inversion are time consuming for large scale data.

Another line of studies focus on the matrix factorization methods [7], [9], [21], [22] that generally provide a large advantage over graph-based methods in terms of time cost. However, these methods cannot tackle with the data with non-linear structure. In short, graph-based methods perform better but are limited by the high computation cost. On the other hand, matrix factorization methods are efficient but unable to provide a satisfactory clustering result. Such an issue motivates us to combine the advantages of them. In this work, we propose to implement spectral embedding and nonnegative embedding simultaneously. Our basic idea is partially motivated by Kuang et al. [23], where the relation between symmetric nonnegative matrix factorization and spectral clustering is discussed in single-view case. The main contributions of this work can be summarized as follows:

• We provide a novel multi-view spectral clustering algorithm, namely NESE (multi-view spectral clustering via integrating Nonnegative Embedding and Spectral Embedding). It inherits the advantages of both graph-based and matrix factorization methods. Specifically, the model of NESE is parameter-free, which makes it more applicable than existing methods. Moreover, the solution returned by NESE directly reveals the consistent clustering result. Such that the uncertainty brought by post-processing, such as K-means and spectral rotation can be avoided.

• We provide an efficient optimization approach, namely inexact Majorization-Minimization (inexact-MM), to solve the non-convex and non-smooth objective involved in NESE. The computation complexity of inexact-MM is approximately $\mathcal{O}\left(n{k}^2\right),$ where n and k are the number of samples and clusters, respectively.

• We conduct numerous experiments to verify the performance of NESE. And the experimental results demonstrate that our method can achieve comparable and even better clustering results. We provide the datasets and code in https://github.com/sudalvxin/SMSC.git.

We report the notations that are widely used in this paper in Table 1. The remainder of this work is organized as follows. We introduce the related works in Section 2, and present proposed method NESE in Section 3. The optimization details w.r.t NESE is summarized in Section 4. We report comparison results in Section 5, and conclude this work in Section 6.