Nonparametric discriminant multi-manifold learning for dimensionality reduction

Abstract

Based on that data sampled from the same class locate on one manifold and those labeled different classes reside on the corresponding manifolds, traditional data classification problem can be reasoned to multiply manifolds identification. Thus in this paper, a dimensionality reduction method titled nonparametric discirminant multi-manifold learning (NDML) is put forward and involved in different manifolds recognition. In the proposed method, a novel nonparametric manifold-to-manifold distance is defined to characterize the separability between manifolds. And then an objective function is modeled to project the original data into a low dimensional space, where the manifold-to-manifold distances can be maximized and manifolds locality will be preserved. Experiments have been carried out on benchmark face data sets with comparisons to some related dimensionality reduction methods such as Unsupervised Discriminant Projection (UDP), Constrained Maximum Variance Mapping (CMVM) and Linear Discriminant Analysis (LDA). The experimental results validate that the proposed NDML can obtain better performance than other methods.

Introduction

During last decade, dimensionality reduction has been attracting considerable attention in many fields as pattern recognition, data mining, computer vision and machine learning. As a fundamental problem in these fields, dimensionality reduction plays an important role in data analysis with the goal to find a meaningful low dimensional representation of high dimensional data. In regard to pattern recognition, dimensionality reduction is an effective and feasible way to overcome the “curse of dimensionality”. Moreover, since there are large volumes of high dimensional data in numerous real world applications, some of which are perhaps superfluous, so extracting the most useful features from the real world data using dimensionality reduction techniques not only helps to explore the essential structure of the original data, but also contributes to accomplish the task of classification at low computational cost.

Dimensionality reduction methods can be categorized into linear models and nonlinear ones. Many linear methods such as Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA) and Independent Component Analysis (ICA) have been widely used on practical applications with good performance [34]. However, they also expose their limitations when applied to nonlinear distributed data. So in the last few years, some approaches including manifold learning have been developed for nonlinear dimensionality reduction. Among all the manifold learning methods, Isometric Mapping (ISOMAP) [1], Locally Linear Embedding (LLE) [2], [3], Laplacian Eigenmaps (LE) [4], Local Tangent Space Alignment (LTSA) [5], Maximum Variance Unfolding (MVU) [6] and Riemannian Manifold Learning (RML) [7] are their representatives. It was shown by many examples that these methods have yielded impressive results on artificial and real world data sets. Compared to other nonlinear dimensionality reduction methods, manifold learning shows its superiorities in the following. On the one hand, manifold learning can explore the essential dimensions of manifold data embedded in high dimensional space. For instance, in Ref. [1], [2], the left–right pose, up–down pose and lighting direction can be found to three essential dimensions of face manifold. On the other hand, manifold learning pursuits to embed the original data with high dimensions in a lower dimensional space by locality preserving, where the locality can be approached using k Nearest Neighbors (KNN) criterion. Thus manifold learning can be efficiently applied for data visualization. The reason is concluded that both manifold learning and data visualization have the similar goal to project the original data into a 2-D or 3-D space, where their intrinsic structure information will be preserved as much as possible.

However, when employed to data classification, manifold learning appears some shortcomings. Firstly, manifold learning yields embeddings directly based on training data sets. Owing to the implicitness of nonlinear mapping, for any new test sample, the original manifold learning methods cannot easily obtain its projection in the embedding space by utilizing low-dimensional embeddings of training sets, which greatly confines applications of the original manifold learning algorithms to pattern classification. In order to overcome this out-of-sample problem [8], linearization, kernelization, tensorization and other tricks were proposed, which were validated efficient to find the low dimensional embeddings of test data on the basis of the mapping results of training samples [9], [10].

Secondly, it is unproblematic for the existing manifold learning algorithms as they try to approach a simple manifold. However, if there are many manifolds, how to clearly identify different manifolds still needs further demonstrations. For example, if face images sampled from many persons do exist in a high dimensional space, then different persons׳ face images should lie on the corresponding manifolds. So it is necessary to distinguish individual face images from different manifolds. In order to achieve optimal recognition results, the recovered embeddings associated to different manifolds should be as separable as possible in the final embedding space, which poses a problem that might be called “classification-oriented multi-manifold learning” [11]. The problem cannot be solved by some current manifold learning algorithms and their supervised versions [12], [13], [14], [15], [16], [17], [18], [19], [20] because they all just concentrate on the characterization of “locality” and do not take into account the variances among manifolds. Recently, researchers put forward some discriminant multi-manifold learning methods, where supervised graph was constructed to approach manifolds distances. Lai proposed a soft margin scatter and a soft within-class scatter, both of which were introduced to find an optimal subspace for data classification [21]. Wang presented a maximum inter-class and marginal discriminant embedding for manifolds identification [22]. On the basis of Unsupervised Discriminant Projection (UDP) [11], a locally statistical uncorrelation was advanced by Chen as a constraint [23]. Later, Lu defined an inter-manifold graph and an intra-manifold graph according to class information and then the corresponding graph Laplacian spectrum was used to search the optimal projection [24]. Similar to Lu, Chen adopted the least reconstruction trick to both inter-class graph and intra-class graph to seek a discirminant subspace [25]. All the methods mentioned above characterized the separability of manifolds globally and failed to take advantage of the local distances between manifolds.

So in this paper, a dimensionality reduction method, named Nonparametric Discirminant Multi-manifold Learning (NDML), is presented and involved in multiply manifolds identification. In the proposed NDML algorithm, a novel nonparametric manifold-to-manifold distance is defined to model separability between manifolds, where both labels and local structure information of manifolds are all considered. Moreover, the linearization trick is also introduced to avoid out-of-sample problem. At last, an objective function is constructed to explore an optimal subspace with the maximum manifold-to-manifold distances and the minimum manifolds locality.

The rest of the paper is organized as follows: Section 2 simply reviews LLE and Constrained Maximum Variance Mapping (CMVM) [26]. In Section 3, the principle of NDML is addressed in details. Experimental results on AR face data, ORL face data and YaleB face data are offered in Section 4 and the paper is finished with some conclusions in Section 5.