Robust data representation using locally linear embedding guided PCA

Abstract

Locally Linear Embedding (LLE) is widely used for embedding data on a nonlinear manifold. It aims to preserve the local neighborhood structure on the data manifold. Our work begins with a new observation that LLE has a natural robustness property. Motivated by this observation, we propose to integrate LLE and PCA into a LLE guided PCA model (LLE-PCA) that incorporates both global structure and local neighborhood structure simultaneously while performs robustly to outliers. LLE-PCA has a compact closed-form solution and can be efficiently computed. Extensive experiments on five datasets show promising results on data reconstruction and improvement on data clustering and semi-supervised learning tasks.

Introduction

Efficient and compact representation of data is a fundamental problem in data mining and machine learning area. In real-world datasets, data usually have high dimensionality and often contain noises and errors. One of the widely used effective approaches is to find a low-dimensional representation for high dimension data based on low-rank matrix factorization [1], [2], [3], [4], [5], [6], [7]. In low-dimensional space, the cluster distribution becomes more apparent and thus can improve the machine learning results [8], [9], [10], [11]. In additional to matrix factorization, trace norm based low-rank and sparse methods have also been widely used to conduct data representation [12], [13], [14], [15]. For example, one of popular methods is sparse representation [12] which has been successfully applied in face recognition problem. Liu et al., [13] proposed a robust subspace segmentation and data recovery method based on low rank representation (LRR). Liu and Yan [16] extend LRR to Latent Low-Rank Representation (LatLRR) which uses both observed and unobserved, hidden data and thus provides a more robust representation. Li et al., [14] proposed a robust structured subspace learning (RSSL) algorithm for data representation by integrating both image recognition and feature extraction into a general joint learning model. Zhang et al., [17] provided a matrix completion method via truncated nuclear norm regularization which can better approximate the matrix rank function. In this paper, we focus on matrix factorization based data representation methods.

Principal Component Analysis (PCA) is a classical data representation and dimension reduction technique [1], [18], [19], [20]. It has been widely used to learn a low-dimensional representation of data on a linear manifold. When PCA is applied for data representation and dimension reduction, it aims to preserve the global Euclidean structure of the data [21], [22], [23]. However, in many applications, data lie in a nonlinear manifold. One popular method is to use manifold embedding methods such as Laplacian Embedding [24], Isomap [25], Locality Preserving Projections [26], Locally Linear Embedding [27], Local Tangent Space Alignment [28], Neighborhood Preserving Embedding [29], Kernel methods [30] etc. As one of the popular manifold embedding methods, Locally Linear Embedding (LLE) [27], [30] has been widely used for embedding data on a nonlinear manifold. It aims to preserve the local neighborhood structure on the data manifold and thus performs robustly to local outliers.

In this paper, we begin with a new observation that LLE has a natural robustness property. Roughly speaking, the second step of LLE (which computes the embedding coordinates) can be approximately determined by the normal data points, without consideration of the outliers or severely corrupted data points. Once the embedding coordinates of normal data points are computed, the outlier data points are brought back to the correct data manifold through embedding in their kNN neighborhoods which consists of normal data points only.

Motivated by this observation, we propose to integrate LLE and PCA into a new robust data representation model, called LLE guided PCA (LLE-PCA), which combines the LLE robustness in local manifold structure and PCA’s good preservation of the global structure of data simultaneously. Generally, there are three main benefits of LLE-PCA method.

• It can be used for data reconstruction. Compared with traditional PCA, LLE-PCA reconstruction performs more robustly to outliers due to LLE regularization.

• It can be used for data dimension reduction. In LLE-PCA low-dimensional subspace, the cluster structure of the data generally becomes more apparent and thus more desirable for conducting data clustering and semi-supervised learning tasks.

• It has a compact closed-form solution and can be efficiently computed. This makes it simple and suitable for practical application.

Moreover, based on LLE-PCA model, we show experimentally how the data information of X can be preserved (retained) in LLE manifold embedding. This can be seen as a new feature for LLE manifold learning and embedding method. We perform experiments on several datasets. Promising results show that LLE-PCA has certain robustness and consistently improves clustering and semi-supervised learning.