Elsevier

Neurocomputing

Volume 119, 7 November 2013, Pages 243-252
Neurocomputing

Locally discriminative spectral clustering with composite manifold

https://doi.org/10.1016/j.neucom.2013.03.034Get rights and content

Abstract

A large number of data are generated in many real-world applications, e.g., photos of albums in social networks. Discovering meaningful patterns from them is desirable and still remains a big challenge. To this end, spectral clustering has established itself as a very useful tool for data analysis. It considers the manifold geometrical structure of the data and desires to estimate the intrinsic manifold. However, there exists no principled way for estimating such manifold. Thus, the clustering performance might even degrade seriously when the estimated manifold deviates far from the intrinsic manifold. To address this problem, we propose to employ composite manifold to approximate the intrinsic manifold as much as possible. This composite manifold is derived from a convex combination of some pre-given candidate manifolds in a convex hull. The diversity nature of these manifolds provides richer structure information, which is helpful to maximally estimate the intrinsic manifold. Besides, traditional spectral clustering neglects the discriminant information latent in the data space, so we incorporate the locally discriminative structure into the partition matrix by explicitly using local linear regression, for better clustering. Therefore, in this paper, we present an integrated clustering approach named Spectral Clustering via Composite manifold and Local discriminant learning, referred to as SCCL for short, which addresses the aforementioned two problems. To optimize the objective function, an alternating optimization framework is adopted to derive both the cluster membership matrix and the composite manifold coefficient. Extensive experiments are carried out on several real-world databases. Results have validated the efficacy of the proposed algorithm.

Introduction

With the rapid development of information technology, a large volume of data are generated by human activities every day in numerous real-world applications, e.g., photos of various albums in social networks, funny pictures for entertainment, innovative artworks for flyers, and diseases' symptom images in medical diagnosis. We are always drowning in such large amount of data but starved of valuable information. The pattern of the data, as one kind of useful information, can be effectively discovered by capturing the cluster membership of a given data collection using existing clustering methods, such as k-means [5], spectral clustering [11], [27], [34], [35], [41], matrix factorization based clustering [10], [23]. Traditional k-means is a classical clustering algorithm, which minimizes some distance (e.g., Euclidean distance) between the centers and the data points. As one kind of representative methods, spectral clustering finds the cluster partitions by spectral decomposition on an affinity matrix, e.g., normalized cut for image segmentation [33]. The most popular matrix factorization based clustering approach is nonnegative matrix factorization [23], which approximates the original data matrix by the product of one basis and one coefficient matrix. The coefficient matrix is often treated as the new data representation, which is used for clustering in the lower dimensional data space. Actually, nonnegative matrix factorization is equivalent to kernel k-means and Laplacian-based spectral clustering [14]. These clustering methods have been successfully applied to a wide range of real-world applications, such as image annotation [21] and image retrieval [13].

In this work, we focus on studying spectral clustering approach. The goal is to determine the cluster membership of the partition matrix learned by spectral clustering. Once the partition matrix is available, one can use either traditional k-means or spectral rotation [41] to discretize it to reveal the cluster membership of the data samples. Its quality has direct impacts on the ultimate clustering results. Hence, the partition matrix is a crucial component for spectral clustering. In the last decade, some effective spectral clustering methods have been developed to obtain an informative partition matrix, e.g., some researchers consider the discriminant structure [1], [36], [9] or both the manifold and discriminant structure [37], [38], [39]. Among them, the Laplacian-based methods need to estimate the intrinsic manifold determined by some hyperparameters. However, there is no principled way to determine suitable hyperparameters for estimating the intrinsic manifold of the data. Actually, this is a very challenging task. Heuristically, we can do exhausted searching from some pre-given parameter grid. The shortcomings of this method are that it costs much time and might even seriously deviate from the intrinsic manifold under certain adverse condition.

Recently, an ensemble manifold regularization framework was proposed in [15], which was proved to be a promising way to approximate the intrinsic manifold for semi-supervised learning problems using support vector machine. Inspired by this work, we propose to employ composite manifold to approximate the intrinsic manifold for spectral clustering in a sensible manner. In particular, we assume that a set of candidate manifolds are pre-given in a convex hull, and construct the composite manifold by a convex combination of these candidates, which are able to provide much richer information because of their diversity nature. Just like other Laplacian-based methods [7], [12], [20], [24], [25], [28], we make an assumption that the data points are sampled from the ambient nonlinear manifold, and nearby data points are likely to be grouped into the same cluster, called manifold assumption [3]. Besides, we strive to capture the discriminant structure by local discriminant learning using local linear regression [43]. In this way, the obtained partition matrix could be equipped with more discriminating power, thus improving the clustering performance. Therefore, we introduce an integrated clustering approach named Spectral Clustering via Composite manifold and Local discriminant learning, and we refer it to as SCCL for short. The significant advantage of this method is that it exploits composite manifold to approximate the intrinsic manifold of the data as much as possible, which can better preserve the local geometrical structure of the data space. Essentially, the basic idea of this work is motivated from the previous work in [15], [1].

It is worthwhile to highlight several aspects of this work as follows.

  • We present a simple, yet effective, spectral clustering approach for clustering analysis. This method provides an automatic way to estimate the intrinsic manifold by using composite manifold, which is scalable for a large number of candidate hyperparameters. Moreover, we embed the discriminant information into the partition matrix through local discriminant learning, in order to obtain better clustering results.

  • The composite manifold coefficient learning algorithm is introduced to capture the coefficient distribution of the composite manifold using a convex combination of some pre-given candidate manifolds. It can be viewed as a nonlinear projected-subgradient method with a global efficiency estimate.

  • To optimize the objective function, we adopt an alternative optimization framework to update the partition matrix and the composite manifold coefficient vector alternatively.

  • We conduct comprehensive experiments on a range of real-world databases to investigate the performances of several clustering algorithms. Results have validated the efficacy of the proposed method.

The rest of this paper is organized as follows. Section 2 briefly reviews some closely related works. We introduce our SCCL approach in Section 3 and report the experimental results as well as the model selection in Section 4. Finally, the concluding remarks and the future work discussion are provided in Section 5.

Section snippets

Related works

In this section, we briefly review some works that are closely related to our SCCL approach.

Clustering, as a fundamental topic in data mining community, aims to group the similar data points into the same cluster. For analysis, it discovers the hidden pattern of the data in an unsupervised manner from the perspective of machine learning. The classical k-means is a widely used clustering method in both academic and industrial societies. It assigns each data point to the corresponding cluster in

The proposed method

In this section, we introduce our new clustering method and strive to explain it in clarity. Two main components, i.e., composite manifold and local discriminant learning, will be explicitly elaborated. Besides, an alternative optimization framework including the composite manifold coefficient learning algorithm is provided. Before proceeding to details, we begin with a concise problem formulation.

Experiments

In this section, we conduct extensive experiments to empirically compare the clustering performance of SCCL with several state-of-the-art approaches on a series of real-world data collections. To further explore the parameter sensitivity of the proposed algorithm, model selections are provided as well.

Conclusion and future work

In this paper, we present a novel spectral clustering approach that incorporates composite manifold and local discriminant learning for clustering analysis. We assume a set of candidate manifolds are available in a convex hull. The convex combination of these candidate manifolds is employed to construct the composite manifold, whose coefficient vector is obtained by using the composite manifold coefficient learning algorithm. In this way, the learned manifold can approximate the intrinsic

Acknowledgments

This work was supported in part by National Natural Science Foundation of China under Grants 91120302, 61222207, 61173185 and 61173186, National Basic Research Program of China (973 Program) under Grant 2013CB336500, the Fundamental Research Funds for the Central Universities under Grant 2013FZA5012, the International S&T Cooperation Program of China under Grant S2013GR0480 and the Zhejiang Province Key S&T Innovation Group Project under Grant 2009R50009.

Ping Li is currently pursuing the Ph.D. degree in Computer Science at Zhejiang University. He received the M.S. degree in Information and Communication Engineering from Central South University, China, in 2010. His research interests include machine learning, data mining and information retrieval.

References (43)

  • A. Beck et al.

    Mirror descent and nonlinear projected subgradient methods for convex optimization

    Oper. Res. Lett.

    (2003)
  • P. Li et al.

    Clustering analysis using manifold kernel concept factorization

    Neurocomputing

    (2012)
  • F. Bach et al.

    Diffraca discriminative and flexible framework for clustering

    Adv. Neural Inf. Process. Syst.

    (2007)
  • M. Belkin et al.

    Manifold regularizationa geometric framework for learning from labeled and unlabeled examples

    J. Mach. Learn. Res.

    (2006)
  • J.C. Bezdek et al.

    Convergence of alternating optimization

    Neural Parallel Sci. Computations

    (2003)
  • C.M. Bishop

    Pattern Recognition and Machine Learning

    (2006)
  • S.P. Boyd et al.

    Convex Optimization

    (2004)
  • D. Cai et al.

    Manifold adaptive experimental design for text categorization

    IEEE Trans. Knowl. Data Eng.

    (2012)
  • D. Cai et al.

    Locally consistent concept factorization for document clustering

    IEEE Trans. Knowl. Data Eng.

    (2011)
  • D. Cai et al.

    Speed up kernel discriminant analysis

    The VLDB J.

    (2011)
  • D. Cai et al.

    Graph regularized nonnegative matrix factorization for data representation

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2011)
  • W.Y. Chen et al.

    Parallel spectral clustering in distributed systems

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2011)
  • Y. Chen et al.

    Nonnegative local coordinate factorization for image representation

    IEEE Trans. Image Process.

    (2013)
  • R. Datta et al.

    Image retrievalideas, influences, and trends of the new age

    ACM Computing Surv.

    (2008)
  • C. Ding, X. He, H.D. Simon, On the equivalence of nonnegative matrix factorization and spectral clustering, in:...
  • B. Geng, C. Xu, D. Tao, L. Yang, X.S. Hua, Ensemble manifold regularization, in: Proceedings of the International...
  • A.S. Georghiades et al.

    From few to manyillumination cone models for face recognition under variable lighting and pose

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2001)
  • G. Hamerly, C. Elkan, Learning the k in k-means, in: Advances in Neural Information Processing Systems, vol. 16, MIT...
  • T. Hastie et al.

    The Elements of Statistical LearningData Mining, Inference, and Prediction

    (2009)
  • R. He et al.

    Robust principal component analysis based on maximum correntropy criterion

    IEEE Trans. Image Process.

    (2011)
  • X. He et al.

    Laplacian regularized Gaussian mixture model for data clustering

    IEEE Trans. Knowl. Data Eng.

    (2011)
  • Cited by (12)

    • Coupled local–global adaptation for multi-source transfer learning

      2018, Neurocomputing
      Citation Excerpt :

      Xiong et al. [22] verify that exploiting manifold structure of latent domains can further modeling the dataset. Li et al. [23–25] investigate the manifold structures under subspace learning. For multi-source domain adaptation, another possible solution is to learn multiple cross-domain transforms, one for each source-target pair [26].

    • Semi-supervised classification in stratified spaces by considering non-interior points using Laplacian behavior

      2017, Neurocomputing
      Citation Excerpt :

      Li. et al introduced Spectral Clustering via Composite manifold and Local discriminant learning to estimate the intrinsic structure of data by assuming that the data lies in a convex combination of some pre given manifolds [22]. As unsupervised methods do not consider the labeled data, only related semi-supervised methods are discussed in the following paragraphs.

    • Towards robust subspace recovery via sparsity-constrained latent low-rank representation

      2016, Journal of Visual Communication and Image Representation
      Citation Excerpt :

      Therefore, in this paper we consider explicitly imposing the sparsity constraint on the learnt low-rank representation by LLRR, thus capturing a both lowest-rank and sparsest data representation using both the observed and hidden data as the dictionary. In addition, the core idea of this work can be extended in several aspects, such as semi-supervised learning [38], multi-view learning [39], weakly-supervised learning [40,41], discriminant learning [42,43]. In these directions, there have emerged a number of efficient computer vision methods.

    • Sparse fixed-rank representation for robust visual analysis

      2015, Signal Processing
      Citation Excerpt :

      Among them, lots of contributions on low-rank based methods have been devoted to this very topic [21–24]. Generally, low-rank methods are closely related to low-dimensional representation methods, such as matrix factorization [14,25], subspace learning [26–28] and spectral decomposition methods [29,30]. Now, we begin with the low-rank based methods.

    • A novel graph-based k-means for nonlinear manifold clustering and representative selection

      2014, Neurocomputing
      Citation Excerpt :

      Goh and Vidal [10] only consider separated nonlinear manifold clustering. In recent years, spectral clustering [11–15] and manifold clustering algorithms [16–18] have been proposed to handle general manifold-distributed data sets. These approaches either do not present a meaningful subset of data points which can mostly represent the data set or need a complicated optimization process.

    View all citing articles on Scopus

    Ping Li is currently pursuing the Ph.D. degree in Computer Science at Zhejiang University. He received the M.S. degree in Information and Communication Engineering from Central South University, China, in 2010. His research interests include machine learning, data mining and information retrieval.

    Jiajun Bu received the B.S. and Ph.D. degrees in Computer Science from Zhejiang University, China, in 1995 and 2000 respectively. He is a professor in College of Computer Science, Zhejiang University. His research interests include embedded system, data mining, information retrieval and mobile database.

    Bin Xu received the B.S. degree in Computer Science from Zhejiang University, China, in 2009. He is currently a Ph.D. candidate in Computer Science at Zhejiang University. His research interests include information retrieval and machine learning.

    Beidou Wang received the B.S. degree in Computer Science from Zhejiang University, China, in 2011. He is currently a Ph.D. candidate in Computer Science at Zhejiang University. His research interests include recommendation system, data mining in social networks, information retrieval and machine learning.

    Chun Chen received the B.S. degree in Mathematics from Xiamen University, China, in 1981, and his M.S. and Ph.D. degrees in Computer Science from Zhejiang University, China, in 1984 and 1990 respectively. He is a professor in College of Computer Science, Zhejiang University. His research interests include information retrieval, data mining, computer vision, computer graphics and embedded technology.

    View full text