Locally discriminative spectral clustering with composite manifold
Introduction
With the rapid development of information technology, a large volume of data are generated by human activities every day in numerous real-world applications, e.g., photos of various albums in social networks, funny pictures for entertainment, innovative artworks for flyers, and diseases' symptom images in medical diagnosis. We are always drowning in such large amount of data but starved of valuable information. The pattern of the data, as one kind of useful information, can be effectively discovered by capturing the cluster membership of a given data collection using existing clustering methods, such as k-means [5], spectral clustering [11], [27], [34], [35], [41], matrix factorization based clustering [10], [23]. Traditional k-means is a classical clustering algorithm, which minimizes some distance (e.g., Euclidean distance) between the centers and the data points. As one kind of representative methods, spectral clustering finds the cluster partitions by spectral decomposition on an affinity matrix, e.g., normalized cut for image segmentation [33]. The most popular matrix factorization based clustering approach is nonnegative matrix factorization [23], which approximates the original data matrix by the product of one basis and one coefficient matrix. The coefficient matrix is often treated as the new data representation, which is used for clustering in the lower dimensional data space. Actually, nonnegative matrix factorization is equivalent to kernel k-means and Laplacian-based spectral clustering [14]. These clustering methods have been successfully applied to a wide range of real-world applications, such as image annotation [21] and image retrieval [13].
In this work, we focus on studying spectral clustering approach. The goal is to determine the cluster membership of the partition matrix learned by spectral clustering. Once the partition matrix is available, one can use either traditional k-means or spectral rotation [41] to discretize it to reveal the cluster membership of the data samples. Its quality has direct impacts on the ultimate clustering results. Hence, the partition matrix is a crucial component for spectral clustering. In the last decade, some effective spectral clustering methods have been developed to obtain an informative partition matrix, e.g., some researchers consider the discriminant structure [1], [36], [9] or both the manifold and discriminant structure [37], [38], [39]. Among them, the Laplacian-based methods need to estimate the intrinsic manifold determined by some hyperparameters. However, there is no principled way to determine suitable hyperparameters for estimating the intrinsic manifold of the data. Actually, this is a very challenging task. Heuristically, we can do exhausted searching from some pre-given parameter grid. The shortcomings of this method are that it costs much time and might even seriously deviate from the intrinsic manifold under certain adverse condition.
Recently, an ensemble manifold regularization framework was proposed in [15], which was proved to be a promising way to approximate the intrinsic manifold for semi-supervised learning problems using support vector machine. Inspired by this work, we propose to employ composite manifold to approximate the intrinsic manifold for spectral clustering in a sensible manner. In particular, we assume that a set of candidate manifolds are pre-given in a convex hull, and construct the composite manifold by a convex combination of these candidates, which are able to provide much richer information because of their diversity nature. Just like other Laplacian-based methods [7], [12], [20], [24], [25], [28], we make an assumption that the data points are sampled from the ambient nonlinear manifold, and nearby data points are likely to be grouped into the same cluster, called manifold assumption [3]. Besides, we strive to capture the discriminant structure by local discriminant learning using local linear regression [43]. In this way, the obtained partition matrix could be equipped with more discriminating power, thus improving the clustering performance. Therefore, we introduce an integrated clustering approach named Spectral Clustering via Composite manifold and Local discriminant learning, and we refer it to as SCCL for short. The significant advantage of this method is that it exploits composite manifold to approximate the intrinsic manifold of the data as much as possible, which can better preserve the local geometrical structure of the data space. Essentially, the basic idea of this work is motivated from the previous work in [15], [1].
It is worthwhile to highlight several aspects of this work as follows.
- •
We present a simple, yet effective, spectral clustering approach for clustering analysis. This method provides an automatic way to estimate the intrinsic manifold by using composite manifold, which is scalable for a large number of candidate hyperparameters. Moreover, we embed the discriminant information into the partition matrix through local discriminant learning, in order to obtain better clustering results.
- •
The composite manifold coefficient learning algorithm is introduced to capture the coefficient distribution of the composite manifold using a convex combination of some pre-given candidate manifolds. It can be viewed as a nonlinear projected-subgradient method with a global efficiency estimate.
- •
To optimize the objective function, we adopt an alternative optimization framework to update the partition matrix and the composite manifold coefficient vector alternatively.
- •
We conduct comprehensive experiments on a range of real-world databases to investigate the performances of several clustering algorithms. Results have validated the efficacy of the proposed method.
The rest of this paper is organized as follows. Section 2 briefly reviews some closely related works. We introduce our SCCL approach in Section 3 and report the experimental results as well as the model selection in Section 4. Finally, the concluding remarks and the future work discussion are provided in Section 5.
Section snippets
Related works
In this section, we briefly review some works that are closely related to our SCCL approach.
Clustering, as a fundamental topic in data mining community, aims to group the similar data points into the same cluster. For analysis, it discovers the hidden pattern of the data in an unsupervised manner from the perspective of machine learning. The classical k-means is a widely used clustering method in both academic and industrial societies. It assigns each data point to the corresponding cluster in
The proposed method
In this section, we introduce our new clustering method and strive to explain it in clarity. Two main components, i.e., composite manifold and local discriminant learning, will be explicitly elaborated. Besides, an alternative optimization framework including the composite manifold coefficient learning algorithm is provided. Before proceeding to details, we begin with a concise problem formulation.
Experiments
In this section, we conduct extensive experiments to empirically compare the clustering performance of SCCL with several state-of-the-art approaches on a series of real-world data collections. To further explore the parameter sensitivity of the proposed algorithm, model selections are provided as well.
Conclusion and future work
In this paper, we present a novel spectral clustering approach that incorporates composite manifold and local discriminant learning for clustering analysis. We assume a set of candidate manifolds are available in a convex hull. The convex combination of these candidate manifolds is employed to construct the composite manifold, whose coefficient vector is obtained by using the composite manifold coefficient learning algorithm. In this way, the learned manifold can approximate the intrinsic
Acknowledgments
This work was supported in part by National Natural Science Foundation of China under Grants 91120302, 61222207, 61173185 and 61173186, National Basic Research Program of China (973 Program) under Grant 2013CB336500, the Fundamental Research Funds for the Central Universities under Grant 2013FZA5012, the International S&T Cooperation Program of China under Grant S2013GR0480 and the Zhejiang Province Key S&T Innovation Group Project under Grant 2009R50009.
Ping Li is currently pursuing the Ph.D. degree in Computer Science at Zhejiang University. He received the M.S. degree in Information and Communication Engineering from Central South University, China, in 2010. His research interests include machine learning, data mining and information retrieval.
References (43)
- et al.
Mirror descent and nonlinear projected subgradient methods for convex optimization
Oper. Res. Lett.
(2003) - et al.
Clustering analysis using manifold kernel concept factorization
Neurocomputing
(2012) - et al.
Diffraca discriminative and flexible framework for clustering
Adv. Neural Inf. Process. Syst.
(2007) - et al.
Manifold regularizationa geometric framework for learning from labeled and unlabeled examples
J. Mach. Learn. Res.
(2006) - et al.
Convergence of alternating optimization
Neural Parallel Sci. Computations
(2003) Pattern Recognition and Machine Learning
(2006)- et al.
Convex Optimization
(2004) - et al.
Manifold adaptive experimental design for text categorization
IEEE Trans. Knowl. Data Eng.
(2012) - et al.
Locally consistent concept factorization for document clustering
IEEE Trans. Knowl. Data Eng.
(2011) - et al.
Speed up kernel discriminant analysis
The VLDB J.
(2011)
Graph regularized nonnegative matrix factorization for data representation
IEEE Trans. Pattern Anal. Mach. Intell.
Parallel spectral clustering in distributed systems
IEEE Trans. Pattern Anal. Mach. Intell.
Nonnegative local coordinate factorization for image representation
IEEE Trans. Image Process.
Image retrievalideas, influences, and trends of the new age
ACM Computing Surv.
From few to manyillumination cone models for face recognition under variable lighting and pose
IEEE Trans. Pattern Anal. Mach. Intell.
The Elements of Statistical LearningData Mining, Inference, and Prediction
Robust principal component analysis based on maximum correntropy criterion
IEEE Trans. Image Process.
Laplacian regularized Gaussian mixture model for data clustering
IEEE Trans. Knowl. Data Eng.
Cited by (12)
Coupled local–global adaptation for multi-source transfer learning
2018, NeurocomputingCitation Excerpt :Xiong et al. [22] verify that exploiting manifold structure of latent domains can further modeling the dataset. Li et al. [23–25] investigate the manifold structures under subspace learning. For multi-source domain adaptation, another possible solution is to learn multiple cross-domain transforms, one for each source-target pair [26].
Semi-supervised classification in stratified spaces by considering non-interior points using Laplacian behavior
2017, NeurocomputingCitation Excerpt :Li. et al introduced Spectral Clustering via Composite manifold and Local discriminant learning to estimate the intrinsic structure of data by assuming that the data lies in a convex combination of some pre given manifolds [22]. As unsupervised methods do not consider the labeled data, only related semi-supervised methods are discussed in the following paragraphs.
Towards robust subspace recovery via sparsity-constrained latent low-rank representation
2016, Journal of Visual Communication and Image RepresentationCitation Excerpt :Therefore, in this paper we consider explicitly imposing the sparsity constraint on the learnt low-rank representation by LLRR, thus capturing a both lowest-rank and sparsest data representation using both the observed and hidden data as the dictionary. In addition, the core idea of this work can be extended in several aspects, such as semi-supervised learning [38], multi-view learning [39], weakly-supervised learning [40,41], discriminant learning [42,43]. In these directions, there have emerged a number of efficient computer vision methods.
Sparse fixed-rank representation for robust visual analysis
2015, Signal ProcessingCitation Excerpt :Among them, lots of contributions on low-rank based methods have been devoted to this very topic [21–24]. Generally, low-rank methods are closely related to low-dimensional representation methods, such as matrix factorization [14,25], subspace learning [26–28] and spectral decomposition methods [29,30]. Now, we begin with the low-rank based methods.
Spectral clustering with the probabilistic cluster kernel
2015, NeurocomputingA novel graph-based k-means for nonlinear manifold clustering and representative selection
2014, NeurocomputingCitation Excerpt :Goh and Vidal [10] only consider separated nonlinear manifold clustering. In recent years, spectral clustering [11–15] and manifold clustering algorithms [16–18] have been proposed to handle general manifold-distributed data sets. These approaches either do not present a meaningful subset of data points which can mostly represent the data set or need a complicated optimization process.
Ping Li is currently pursuing the Ph.D. degree in Computer Science at Zhejiang University. He received the M.S. degree in Information and Communication Engineering from Central South University, China, in 2010. His research interests include machine learning, data mining and information retrieval.
Jiajun Bu received the B.S. and Ph.D. degrees in Computer Science from Zhejiang University, China, in 1995 and 2000 respectively. He is a professor in College of Computer Science, Zhejiang University. His research interests include embedded system, data mining, information retrieval and mobile database.
Bin Xu received the B.S. degree in Computer Science from Zhejiang University, China, in 2009. He is currently a Ph.D. candidate in Computer Science at Zhejiang University. His research interests include information retrieval and machine learning.
Beidou Wang received the B.S. degree in Computer Science from Zhejiang University, China, in 2011. He is currently a Ph.D. candidate in Computer Science at Zhejiang University. His research interests include recommendation system, data mining in social networks, information retrieval and machine learning.
Chun Chen received the B.S. degree in Mathematics from Xiamen University, China, in 1981, and his M.S. and Ph.D. degrees in Computer Science from Zhejiang University, China, in 1984 and 1990 respectively. He is a professor in College of Computer Science, Zhejiang University. His research interests include information retrieval, data mining, computer vision, computer graphics and embedded technology.