Geometrically local embedding in manifolds for dimension reduction
Graphical Abstract
Highlights
► The geometry metric is introduced to select geometrical neighbors. ► The reliability weight is defined to suppress outlier data. ► Geometrically local embedding is proposed for dimension reduction.
Introduction
The representation of high dimensional data in low dimension space while preserving intrinsic properties is a fundamental issue in information discovery and pattern recognition. In most information processing applications, signal data such as images, texts and sound are high-dimensional [1], [2], which are usually pre-processed into a more concise format to facilitate subsequent recognition and visualization. As a machine learning technique, dimension reduction can achieve low-dimensional data and discover the intrinsic structure of manifolds, in order to facilitate data manipulation and visualization. Dimension reduction techniques are applied in different disciplines such as image processing, computer vision, speech recognition and textural information retrieval [3], [4], [5].
Two types of dimension reduction techniques are primarily studied in the machine learning field: linear and nonlinear methods. Linear dimension reduction such as principal component analysis (PCA) [6], classical multidimensional scaling (MDS) [7], independent component analysis (ICA) [8], and projection pursuit (PP) [9] is characterized by linearly mapping high-dimensional data into low-dimensional embedding. In general, the nonlinear dimension reduction, covering kernel principal component analysis (KPCA) [10], Isomap [11], locally linear embedding (LLE) [12], Laplacian eigenmaps (LE) [13], diffusion maps (DMap) [14] and so forth, performs better than linear ones due to the nonlinearity of the high-dimensional data. However, nonlinear methods suffer the problems of heavy computational burden and difficulties in incremental handling new data, as it is difficult to define the projection of a novel datum based on the trained mapping.
There has been a few works in improving the performance of dimension reduction by finding the outliers and filtering the noisy data. An outlier detection and noisy data reduction method was developed as a pre-processing procedure for manifolds learning in [15]. In the method, the iterative weight selection scheme was used to determine the weights in estimation and in finding noisy data. A robust locally linear embedding algorithm was proposed in [16] using Robust PCA, with a statistics process to decrease the influence of noisy data by computing the associated weight value for each of the neighbors. The process of noisy datum detection was integrated into the LLE and the algorithm showed good performance in handling outliers. A robust version of locally linear coordination (LLC) was also presented to achieve robust mixture modeling by combining t-distribution and probabilistic subspace mixture models [17]. The algorithm performed well in density estimation and classification of image data.
As one of the nonlinear dimension reduction algorithms, LLE computes low dimensional embedding of high dimensional data while preserving neighborhood relationships amongst data. LLE begins with the pre-processing to find neighbors in terms of the Euclidean distance between each datum. The selection of the neighbors influences the final outcome of the dimension reduction dramatically. The k nearest neighbor (kNN) is one of the simple and easy algorithms to implement for neighbor selection [18]. There are, however, intrinsic limitations of kNN, such as: (i) a large k is prone to merge different clusters, and (ii) a small k cannot reflect the local geometry and the results will be sensitive to outliers. Neighborhood locally embedding (NLE) [19], [20] presented an adaptive neighbor selection method to optimally remove redundant information in representation. Prior distribution information of the data can improve the results of neighbor selection in most applications. This prior information is, however, not available in many problems, where estimation and approximation techniques can be adopted to discover the intrinsic properties of the data. Distributed locally linear embedding [21] presented a method to estimate a density function and choose neighbors according to the local distribution. Hessian LLE (HLLE) [22] employed the basic structure of LLE in estimating the null space of the tangent Hessian functional. Although HLLE can analyze large scale data, the computation complexity is high as computation of second derivatives is required, which is also sensitive to numerical noise. The weighted locally linear embedding (WLLE) [23] performed well in discovering the intrinsic structures of data such as neighbor relationship, local distribution and clustering. This advantage was achieved by adopting a new neighbor selection criterion, which is much fairer in considering all the directions around the interested datum, to avoid the problem that the group of the neighbors with low probability distribution are not represented sufficiently. LLE has difficulties in handling novel input data, as the new datum may cause repeat of the entire embedding procedure including the computation of the mapping weight matrix. Locally linear embedding eigenspace analysis [24] provided a way to process the new datum in linear projection, as the linear approximation version of LLE. Given a new high dimensional data point, the corresponding low dimensional data is computed by linear projection. This approximation technique can also be applied to most of derivatives of LLE to facilitate the processing of new input data.
In this paper, we propose geometrically local embedding (GLE) for internal feature discovery, classification and clustering. A geometry distance is presented to measure neighborhoods within data in order to generate a clear visualization of the high dimensional data. Compared with the previous works in [20], [23], the geometry distance emphasizes the local geometrical structure of the manifold spanned by central vectors instead of computing the pairwise metric between data. The geometry distance facilitates further in reconstructing weights to depress the affection of the noisy data and the outliers. The main contributions of this paper are highlighted as follows:
- 1.
the geometry metric is proposed to select neighbors with geometrical relationships in high dimensional manifolds;
- 2.
the reliability weight is introduced to suppress outlier data with the proposed measurement of geometry distance; and
- 3.
geometrically local embedding with three phases is proposed for dimension reduction, which is also efficient in visualization of high dimensional data, clustering and classification.
Section snippets
Overview of GLE
The problem of dimension reduction is mapping the high-dimensional data into a low dimensional space while emphasizing interested information during the process. Given a data set where is a vector in the high dimensional space, we desire to find a set of representation with and , satisfying that the intrinsic features of input data are preserved. In this paper, GLE is proposed with the following three stages to extract intrinsic features of input data and to
Geometry distance
Theorem 1 The volume of the parallelotope spanned by a cluster of selected neighbor vectors is invariant to the sequence of the selection of those neighbors. Proof See Appendix A.1.
The selection of neighbors is influenced by the sequence of neighbor selection especially by the early selected neighbors. The constructed volume is minimized greedily for each new datum, which does not guarantee the volume is minimal in all linear manifolds including that datum. In the neighbor selection process, a datum xm that
Empirical evaluation
This section presents the experimental results on synthetic data and actual application data to show the performance of GLE in dimension reduction, feature extraction, data visualization, clustering and classification. The robustness of the selective parameters is also studied. The GLE is compared with other dimension reduction algorithms including LLE, Diffusion Map, Hessian LLE and Isomap on synthetic manifold data. The configurations of all data sets used in the experiments are listed in
Conclusion
In this paper, GLE has been presented to geometrically discover the intrinsic structure of linear manifolds. The GLE algorithm performs well in extracting inner structures of input linear manifold with outliers. In addition to feature extraction and representation, GLE behaves as a clustering and classification method by projecting the feature data into low-dimensional separable regions. The major drawback of GLE is the slow computation speed compared with other dimension reduction algorithms
Acknowledgment
This research is supported by Singapore National Research Foundation, Interactive Digital Media R&D Program, under research grant R-705-000-017-279 with Social Robotics Laboratory, National University of Singapore, and the National Basic Research Program of China (973 Program) under Grant 2011CB707005.
Shuzhi Sam Ge is founding Director of Social Robotics Lab of Interactive Digital Media Institute, and Director of Edutainment Robotics Lab of the Department of Electrical and Computer Engineering, National University of Singapore.
He has (co)-authored three books: Adaptive Neural Network Control of Robotic Manipulators (WorldScientific, 1998), Stable Adaptive Neural Network Control (Kluwer, 2001) and Switched Linear Systems: Control and Design (Springer, 2055), and over 300 international journal
References (32)
- et al.
Complete neighborhood preserving embedding for face recognition
Pattern Recognition
(2010) - et al.
Reconstruction and analysis of multi-pose face images based on nonlinear dimensionality reduction
Pattern Recognition
(2004) Independent component analysis—a new concept?
Signal Processing
(1994)Kernel PCA for novelty detection
Pattern Recognition
(2007)- et al.
Robust locally linear embedding
Pattern Recognition
(2006) - et al.
Weighted locally linear embedding for dimension reduction
Pattern Recognition
(2009) - et al.
A novel and quick SVM-based multi-class classifier
Pattern Recognition
(2006) Discriminative features for text document classification
Pattern Analysis & Applications
(2004)- et al.
Evolutionary cross-domain discriminative hessian eigenmaps
IEEE Transactions on Image Processing
(2010) - et al.
A boosted manifold learning for automatic face recognition
International Journal of Pattern Recognition and Artificial Intelligence
(2010)
LIII. On lines and planes of closest fit to systems of points in space
Philosophical Magazine Series
Modern multidimensional scaling: theory and applications
Journal of the American Statistical Association
A projection pursuit algorithm for exploratory data analysis
Graphics
A global geometric framework for nonlinear dimensionality reduction
Science
Nonlinear dimensionality reduction by locally linear embedding
Science
Laplacian eigenmaps and spectral techniques for embedding and clustering
Advances in Neural Information Processing Systems
Cited by (11)
Personalizing self-management via behavioral predictive analytics with health education for improved self-efficacy
2022, PatternsCitation Excerpt :Recent clustering research focuses on minimizing dimensionality without losing meaning in the data. Ge et al.19 suggested a geometrically local embedding (GLE) process that reduces dimensionality by assigning clusters according to geometric distance in the higher dimension. After finding optimal reconstruction weights, the algorithm filters for outliers, and the manifold is mapped to a lower dimension.
Kernel maximum likelihood scaled locally linear embedding for night vision images
2014, Optics and Laser TechnologyCitation Excerpt :WLLE uses a weighted distance measurement by giving the prototype data with high density a smaller weight scaling whereas those with low density a larger weight scaling, which supposes a standard normal distribution to estimate the probability density function [7]. GLE selects neighbors by using geometry distance, which is defined as the Euclidean distance from each point to a linear manifold spanned by the principal vectors and is represented as the volume of a parallelotope determined by the principal vectors [8]. The fundamental geometric or weighting information of input data is recovered with the assumption that data are from smooth manifolds, so that each datum and the neighbors lie on a locally approximated linear patch.
A SYSTEMATIC REVIEW OF VARIOUS NONLINEAR DIMENSIONALITY REDUCTION TECHNIQUES FOR ANALYSIS OF BIG DATA
2023, Journal of Theoretical and Applied Information TechnologyNonparametric locality reforming projection for linear discriminative dimensionality reduction
2022, Journal of Electronic ImagingFeature dimensionality reduction: a review
2022, Complex and Intelligent SystemsIncorporating association patterns into manifold clustering for enabling predictive analytics
2019, Proceedings - 6th Annual Conference on Computational Science and Computational Intelligence, CSCI 2019
Shuzhi Sam Ge is founding Director of Social Robotics Lab of Interactive Digital Media Institute, and Director of Edutainment Robotics Lab of the Department of Electrical and Computer Engineering, National University of Singapore.
He has (co)-authored three books: Adaptive Neural Network Control of Robotic Manipulators (WorldScientific, 1998), Stable Adaptive Neural Network Control (Kluwer, 2001) and Switched Linear Systems: Control and Design (Springer, 2055), and over 300 international journal and conference papers. He is a co-founder of Personal E-Motion Pte Ltd. dedicated to interactive multimedia digital books for education and digital publishing. He is the founding Editor-in-Chief, International Journal of Social Robotics, Springer. He has served/been serving as an Associate Editor for a number of flagship journals including IEEE Transactions on Automatic Control, IEEE Transactions on Control Systems Technology, IEEE Transactions on Neural Networks, and Automatica. He also serves as a book Editor of the Taylor & Francis Automation and Control Engineering Series. His current research interests include social robotics, multimedia fusion, adaptive control, intelligent systems and artificial intelligence.
Hongsheng He received the M.Eng. degree from Northeastern University, China 2008. He is currently working toward a Ph.D. degree in the Department of Electrical and Computer Engineering at National University of Singapore. His research interests include machine learning, pattern recognition and computer vision.
Chengyao Shen received the B.S. degree in microelectronics from Shanghai Jiaotong University in 2010. He is currently a Ph.D. candidate in the Social Robotics Lab, National University of Singapore. His research interests included computer vision and natural image statistics. He has been a recipient of NGS Scholarship since 2010.