Supervised discriminant Isomap with maximum margin graph regularization for dimensionality reduction

Highlights

• Two novel DR methods are proposed to extract critical discriminative information.

• Neighborhood size parameters of DR models are decided adaptively via data similarity.

• Outside data can be projected directly through SD-IsoP that prevents over-fitting.

• Margins between classes in reduced space can be maximized by proposed methods.

Abstract

As one of the most popular nonlinear dimensionality reduction methods, Isomap has been widely used in pattern recognition and machine learning. However, Isomap has the following problems: (1) Isomap is an unsupervised dimensionality reduction method, it cannot use class label information to obtain discriminative low dimensional embedding for classification; (2) The embedding performance of Isomap is sensitive to neighborhood size parameter; (3) Isomap cannot deal with outside new data by direct embedding. In this paper, a novel dimensionality reduction method called supervised discriminant Isomap is proposed to solve the first two problems mentioned above. Specifically, first, raw data points are partitioned into different manifolds by using their class label information. Then, supervised discriminant Isomap aims at seeking an optimal nonlinear subspace to preserve the geometrical structure of each manifold according to the Isomap criterion, and to enhance the discriminating capability by maximizing the distances between data points of different classes and the maximum margin graph regularization term. Finally, the corresponding optimization problems are solved by using eigen-decomposition algorithm. Further, we extend supervised discriminant Isomap to a linear dimensionality reduction method called supervised discriminant Isomap projection for handling the above three problems. Moreover, our approaches have three important characteristics: (1) Proposed methods adaptively estimate the local neighborhood surrounding each sample based on data density and similarity; (2) The objective functions of proposed methods can maximize margins between the each classes in the dimension-reduced feature space; (3) The objective functions of proposed methods have closed-form solutions. Furthermore, our methods can capture more discriminative information from raw data than other Isomap based methods. Extensive experiments on nine data sets demonstrate that the proposed methods are superior to the related state-of-the-art methods.

Introduction

Dimensionality reduction (DR) method plays an increasingly important role in science, engineering and physics applications. It aims at computing relevant low-dimensional representations of high dimensional data sets to foil the curse of dimensionality (Lee & Verleysen, 2011). DR methods can be considered as a preprocessing step in pattern recognition tasks and have a wide range of applications, including recommender systems (Nilashi, Ibrahim, & Bagherifard, 2018), stock trading (Han & Ge, 2020), data visualization and classification (Raducanu and Dornaika, 2012, Rajabzadeh et al., 2021), and many others.

In recent years, many methods have been proposed to deal with DR problem. Principal component analysis (PCA) (Jolliffe, 1986), multidimensional scaling (MDS) (Borg & Groenen, 2005) and linear discriminant analysis (LDA) (Fisher, 1936, Fukunaga, 1991) are three commonly used linear DR methods. PCA projects the high-dimensional vectors by maximizing the variance or dot product preservation. MDS is closely related to principal component analysis. Different from PCA and MDS, LDA can use prior information to project the feature space into a smaller subspace, while maintaining the information of distinguishing categories. However, most of the data sets distribution in the real-world is nonlinear (Geng, Zhan, & Zhou, 2005). Therefore, various nonlinear dimensionality reduction methods have been proposed by scholars. As three early nonlinear DR methods, Isometric Mapping (Isomap) (Balasubramanian, Schwartz, Tenenbaum, Silva, & Langford, 2002), local linear embedding (LLE) (Roweis & Saul, 2000) and Laplacian Eigenmap (LE) (Belkin & Niyogi, 2003) were designed to discover the geometric properties of data. Specifically, Isomap aims at obtaining nonlinear embedding by preserving the geodesic distance of all similarity pairs, while both LLE and LE try to preserve the local neighborhood information of each object as much as possible for discovering the low-dimensional subspace. Kernel principal component analysis (KPCA) is a combination of conventional PCA and kernel trick, which has been widely used in practical applications (Schölkopf, Smola, & Müller, 1998). Beyond that, other nonlinear dimensionality reduction methods including maximum variance unfolding (Weinberger & Saul, 2006), diffusion maps (Lafon & Lee, 2006), t-distributed stochastic neighbor embedding (T-SNE) (Laurens & Hinton, 2008), and Local distances preserving based manifold learning (SLDP) (Hajizadeh, Aghagolzadeh, & Ezoji, 2020) also are widely used in various practical applications (Ayesha, Hanif, & Talib, 2020).

Isomap can reveal the intrinsic geometric structure of manifold by preserving geodesic distance. It has shown a remarkable performance for nonlinear DR in various research domains (Ayesha et al., 2020, Zheng et al., 2017). However, the original Isomap is an unsupervised method, it is not good at extracting discriminative features by using prior information for classification. To deal with this problem, supervised Isomap (S-Isomap) (Geng et al., 2005), marginal Isomap (M-Isomap) (Zhang, Chow, & Zhao, 2013) and multi-manifold discriminant Isomap (MMD-Isomap) (Yang, Xiang, & Zhang, 2016) were proposed. S-Isomap uses two parameters to define a new distance metric among the pairwise points with the same and different class labels. M-Isomap tends to offer certain advantages over S-Isomap, due to it incorporates the local pairwise constraints into Isomap to guide the discriminant manifold learning. MMD-Isomap is extension of the M-Isomap, it uses global pairwise constraints to preserve the geometrical structure of each manifold. To solve the problem that Isomap fails to deal with outside new data, some techniques are proposed. Isometric Projection (Isoprojection) (Cai, He, & Han, 2007) constructed a weighted data graph where the weights are discrete approximations of the geodesic distances on the data manifold, which explicitly takes into account the manifold structure. Both S-Isomap and MMD-Isomap handle this problem by using the back-propagation (BP) neural networks.

Schmidhuber (2015) to calculate a deterministic mapping via an independent step after the manifold feature learning process.

Although some conventional methods show their superiority in the real-world applications, most of them still have three common problems. First, the most popular graph construction manner is based on the k-nearest neighbor (k-NN) or $∊$-ball neighborhood criteria. k and $∊$ are user-defined parameters that should be fixed in advance. It is well known that the choice of these parameters can affect the performance of the embedding. Thus, most of Isomap based methods may suffer from the restricted applications. Second, most of Isomap based methods handle the outside new data by using two steps. For example, MMD-Isomap learns a mapping by using an independent process after the manifold feature learning stage. This strategy cannot explicitly ensure the mapping is optimal for data representation. Third, most of supervised Isomap based methods are not so powerful for preserving discriminative information for classification. This also indicates that the learned projections of these methods cannot preserve more discriminant information.

In this paper, we propose a novel supervised nonlinear DR method called supervised discriminant Isomap (SD-Isomap) to solve above problems. To adaptively select the size of neighborhood, two subsets can be obtained by using class label information with the same strategy in Raducanu and Dornaika (2012). Then, SD-Isomap aims at seeking an optimal nonlinear subspace to preserve the geometrical structure of each manifold according to the Isomap criterion, meanwhile, to enhance the discriminating capability by maximizing the distances between data points of different manifolds. To make SD-Isomap capture more discriminative information from data, a maximum margin graph regularization term is constructed. To make SD-Isomap obtain an explicit mapping, supervised discriminant Isomap projection (SD-IsoP) is proposed. It projects the original data to a new space of lower dimensionality by using a unified learning framework. Therefore, Our methods can capture more discriminative information from data compared to other Isomap based methods. Experimental results on nine real-world data sets illustrate the effectiveness of the proposed methods. Based on the above, the main contributions of this paper are summarized as follows:

• SD-Isomap and SD-IsoP can maximize the margins between classes in thedimension-reduced feature space and enjoy closed-form solutions.

• SD-IsoP learn an explicit mapping by using a unified learning framework, so that the learned projection can handle outside data efficiently by direct embedding.

• Proposed methods can adaptively estimate the local neighborhood surrounding each sample based on data density and similarity.

• Proposed methods can capture more discriminative information from high dimensional data than other Isomap based methods.

The rest of this paper is organized as follows. In Section 2, we review some related work about our proposed methods. SD-Isomap and SD-IsoP are proposed in Section 3. Furthermore, we provide a complexity analysis for the proposed methods in the Section 4. Finally, several experiments are conducted to prove the effectiveness of the proposed methods in Section 5.