Elsevier

Pattern Recognition

Volume 45, Issue 4, April 2012, Pages 1455-1470
Pattern Recognition

Geometrically local embedding in manifolds for dimension reduction

https://doi.org/10.1016/j.patcog.2011.09.022Get rights and content

Abstract

In this paper, geometrically local embedding (GLE) is presented to discover the intrinsic structure of manifolds as a method in nonlinear dimension reduction. GLE is able to reveal the inner features of the input data in the lower dimension space while suppressing the influence of outliers in the local linear manifold. In addition to feature extraction and representation, GLE behaves as a clustering and classification method by projecting the feature data into low-dimensional separable regions. Through empirical evaluation, the performance of GLE is demonstrated by the visualization of synthetic data in lower dimension, and the comparison with other dimension reduction algorithms with the same data and configuration. Experiments on both pure and noisy data prove the effectiveness of GLE in dimension reduction, feature extraction, data visualization as well as clustering and classification.

Highlights

► The geometry metric is introduced to select geometrical neighbors. ► The reliability weight is defined to suppress outlier data. ► Geometrically local embedding is proposed for dimension reduction.

Introduction

The representation of high dimensional data in low dimension space while preserving intrinsic properties is a fundamental issue in information discovery and pattern recognition. In most information processing applications, signal data such as images, texts and sound are high-dimensional [1], [2], which are usually pre-processed into a more concise format to facilitate subsequent recognition and visualization. As a machine learning technique, dimension reduction can achieve low-dimensional data and discover the intrinsic structure of manifolds, in order to facilitate data manipulation and visualization. Dimension reduction techniques are applied in different disciplines such as image processing, computer vision, speech recognition and textural information retrieval [3], [4], [5].

Two types of dimension reduction techniques are primarily studied in the machine learning field: linear and nonlinear methods. Linear dimension reduction such as principal component analysis (PCA) [6], classical multidimensional scaling (MDS) [7], independent component analysis (ICA) [8], and projection pursuit (PP) [9] is characterized by linearly mapping high-dimensional data into low-dimensional embedding. In general, the nonlinear dimension reduction, covering kernel principal component analysis (KPCA) [10], Isomap [11], locally linear embedding (LLE) [12], Laplacian eigenmaps (LE) [13], diffusion maps (DMap) [14] and so forth, performs better than linear ones due to the nonlinearity of the high-dimensional data. However, nonlinear methods suffer the problems of heavy computational burden and difficulties in incremental handling new data, as it is difficult to define the projection of a novel datum based on the trained mapping.

There has been a few works in improving the performance of dimension reduction by finding the outliers and filtering the noisy data. An outlier detection and noisy data reduction method was developed as a pre-processing procedure for manifolds learning in [15]. In the method, the iterative weight selection scheme was used to determine the weights in estimation and in finding noisy data. A robust locally linear embedding algorithm was proposed in [16] using Robust PCA, with a statistics process to decrease the influence of noisy data by computing the associated weight value for each of the neighbors. The process of noisy datum detection was integrated into the LLE and the algorithm showed good performance in handling outliers. A robust version of locally linear coordination (LLC) was also presented to achieve robust mixture modeling by combining t-distribution and probabilistic subspace mixture models [17]. The algorithm performed well in density estimation and classification of image data.

As one of the nonlinear dimension reduction algorithms, LLE computes low dimensional embedding of high dimensional data while preserving neighborhood relationships amongst data. LLE begins with the pre-processing to find neighbors in terms of the Euclidean distance between each datum. The selection of the neighbors influences the final outcome of the dimension reduction dramatically. The k nearest neighbor (kNN) is one of the simple and easy algorithms to implement for neighbor selection [18]. There are, however, intrinsic limitations of kNN, such as: (i) a large k is prone to merge different clusters, and (ii) a small k cannot reflect the local geometry and the results will be sensitive to outliers. Neighborhood locally embedding (NLE) [19], [20] presented an adaptive neighbor selection method to optimally remove redundant information in representation. Prior distribution information of the data can improve the results of neighbor selection in most applications. This prior information is, however, not available in many problems, where estimation and approximation techniques can be adopted to discover the intrinsic properties of the data. Distributed locally linear embedding [21] presented a method to estimate a density function and choose neighbors according to the local distribution. Hessian LLE (HLLE) [22] employed the basic structure of LLE in estimating the null space of the tangent Hessian functional. Although HLLE can analyze large scale data, the computation complexity is high as computation of second derivatives is required, which is also sensitive to numerical noise. The weighted locally linear embedding (WLLE) [23] performed well in discovering the intrinsic structures of data such as neighbor relationship, local distribution and clustering. This advantage was achieved by adopting a new neighbor selection criterion, which is much fairer in considering all the directions around the interested datum, to avoid the problem that the group of the neighbors with low probability distribution are not represented sufficiently. LLE has difficulties in handling novel input data, as the new datum may cause repeat of the entire embedding procedure including the computation of the mapping weight matrix. Locally linear embedding eigenspace analysis [24] provided a way to process the new datum in linear projection, as the linear approximation version of LLE. Given a new high dimensional data point, the corresponding low dimensional data is computed by linear projection. This approximation technique can also be applied to most of derivatives of LLE to facilitate the processing of new input data.

In this paper, we propose geometrically local embedding (GLE) for internal feature discovery, classification and clustering. A geometry distance is presented to measure neighborhoods within data in order to generate a clear visualization of the high dimensional data. Compared with the previous works in [20], [23], the geometry distance emphasizes the local geometrical structure of the manifold spanned by central vectors instead of computing the pairwise metric between data. The geometry distance facilitates further in reconstructing weights to depress the affection of the noisy data and the outliers. The main contributions of this paper are highlighted as follows:

  • 1.

    the geometry metric is proposed to select neighbors with geometrical relationships in high dimensional manifolds;

  • 2.

    the reliability weight is introduced to suppress outlier data with the proposed measurement of geometry distance; and

  • 3.

    geometrically local embedding with three phases is proposed for dimension reduction, which is also efficient in visualization of high dimensional data, clustering and classification.

The rest of the paper is organized as follows. Section 2 proposes a neighbor determination technique, focusing on the mathematical computation of geometrical distances. In Section 3, the properties of GLE including the features of neighbors, classification methods and decision boundaries are discussed. Section 4 presents the experiments that are conducted based on synthetic data and actual feature data to assess the performance of GLE in data visualization, dimension reduction as well as clustering and classification. The conclusions and discussions are given in Section 5. For simplicity, the symbols employed in this paper are tabulated in Table 1.

Section snippets

Overview of GLE

The problem of dimension reduction is mapping the high-dimensional data into a low dimensional space while emphasizing interested information during the process. Given a data set X=[xi]i=1N where xiRD is a vector in the high dimensional space, we desire to find a set of representation Y=[yi]i=1N with yiRd and d<D, satisfying that the intrinsic features of input data are preserved. In this paper, GLE is proposed with the following three stages to extract intrinsic features of input data and to

Geometry distance

Theorem 1

The volume of the parallelotope spanned by a cluster of selected neighbor vectors is invariant to the sequence of the selection of those neighbors.

Proof

See Appendix A.1.

The selection of neighbors is influenced by the sequence of neighbor selection especially by the early selected neighbors. The constructed volume is minimized greedily for each new datum, which does not guarantee the volume is minimal in all linear manifolds including that datum. In the neighbor selection process, a datum xm that

Empirical evaluation

This section presents the experimental results on synthetic data and actual application data to show the performance of GLE in dimension reduction, feature extraction, data visualization, clustering and classification. The robustness of the selective parameters is also studied. The GLE is compared with other dimension reduction algorithms including LLE, Diffusion Map, Hessian LLE and Isomap on synthetic manifold data. The configurations of all data sets used in the experiments are listed in

Conclusion

In this paper, GLE has been presented to geometrically discover the intrinsic structure of linear manifolds. The GLE algorithm performs well in extracting inner structures of input linear manifold with outliers. In addition to feature extraction and representation, GLE behaves as a clustering and classification method by projecting the feature data into low-dimensional separable regions. The major drawback of GLE is the slow computation speed compared with other dimension reduction algorithms

Acknowledgment

This research is supported by Singapore National Research Foundation, Interactive Digital Media R&D Program, under research grant R-705-000-017-279 with Social Robotics Laboratory, National University of Singapore, and the National Basic Research Program of China (973 Program) under Grant 2011CB707005.

Shuzhi Sam Ge is founding Director of Social Robotics Lab of Interactive Digital Media Institute, and Director of Edutainment Robotics Lab of the Department of Electrical and Computer Engineering, National University of Singapore.

He has (co)-authored three books: Adaptive Neural Network Control of Robotic Manipulators (WorldScientific, 1998), Stable Adaptive Neural Network Control (Kluwer, 2001) and Switched Linear Systems: Control and Design (Springer, 2055), and over 300 international journal

References (32)

  • K. Pearson

    LIII. On lines and planes of closest fit to systems of points in space

    Philosophical Magazine Series

    (1901)
  • A. O'Connell

    Modern multidimensional scaling: theory and applications

    Journal of the American Statistical Association

    (1999)
  • J. Friedman et al.

    A projection pursuit algorithm for exploratory data analysis

    Graphics

    (1988)
  • J.B. Tenenbaum et al.

    A global geometric framework for nonlinear dimensionality reduction

    Science

    (2000)
  • S.T. Roweis et al.

    Nonlinear dimensionality reduction by locally linear embedding

    Science

    (2000)
  • M. Belkin et al.

    Laplacian eigenmaps and spectral techniques for embedding and clustering

    Advances in Neural Information Processing Systems

    (2002)
  • Cited by (11)

    • Personalizing self-management via behavioral predictive analytics with health education for improved self-efficacy

      2022, Patterns
      Citation Excerpt :

      Recent clustering research focuses on minimizing dimensionality without losing meaning in the data. Ge et al.19 suggested a geometrically local embedding (GLE) process that reduces dimensionality by assigning clusters according to geometric distance in the higher dimension. After finding optimal reconstruction weights, the algorithm filters for outliers, and the manifold is mapped to a lower dimension.

    • Kernel maximum likelihood scaled locally linear embedding for night vision images

      2014, Optics and Laser Technology
      Citation Excerpt :

      WLLE uses a weighted distance measurement by giving the prototype data with high density a smaller weight scaling whereas those with low density a larger weight scaling, which supposes a standard normal distribution to estimate the probability density function [7]. GLE selects neighbors by using geometry distance, which is defined as the Euclidean distance from each point to a linear manifold spanned by the principal vectors and is represented as the volume of a parallelotope determined by the principal vectors [8]. The fundamental geometric or weighting information of input data is recovered with the assumption that data are from smooth manifolds, so that each datum and the neighbors lie on a locally approximated linear patch.

    • Feature dimensionality reduction: a review

      2022, Complex and Intelligent Systems
    • Incorporating association patterns into manifold clustering for enabling predictive analytics

      2019, Proceedings - 6th Annual Conference on Computational Science and Computational Intelligence, CSCI 2019
    View all citing articles on Scopus

    Shuzhi Sam Ge is founding Director of Social Robotics Lab of Interactive Digital Media Institute, and Director of Edutainment Robotics Lab of the Department of Electrical and Computer Engineering, National University of Singapore.

    He has (co)-authored three books: Adaptive Neural Network Control of Robotic Manipulators (WorldScientific, 1998), Stable Adaptive Neural Network Control (Kluwer, 2001) and Switched Linear Systems: Control and Design (Springer, 2055), and over 300 international journal and conference papers. He is a co-founder of Personal E-Motion Pte Ltd. dedicated to interactive multimedia digital books for education and digital publishing. He is the founding Editor-in-Chief, International Journal of Social Robotics, Springer. He has served/been serving as an Associate Editor for a number of flagship journals including IEEE Transactions on Automatic Control, IEEE Transactions on Control Systems Technology, IEEE Transactions on Neural Networks, and Automatica. He also serves as a book Editor of the Taylor & Francis Automation and Control Engineering Series. His current research interests include social robotics, multimedia fusion, adaptive control, intelligent systems and artificial intelligence.

    Hongsheng He received the M.Eng. degree from Northeastern University, China 2008. He is currently working toward a Ph.D. degree in the Department of Electrical and Computer Engineering at National University of Singapore. His research interests include machine learning, pattern recognition and computer vision.

    Chengyao Shen received the B.S. degree in microelectronics from Shanghai Jiaotong University in 2010. He is currently a Ph.D. candidate in the Social Robotics Lab, National University of Singapore. His research interests included computer vision and natural image statistics. He has been a recipient of NGS Scholarship since 2010.

    View full text