Representative null space LDA for discriminative dimensionality reduction

Highlights

• The research reveals the main problem of the classic null space LDA method: the intrinsic overfitting problem.

• A new approach, representative null space LDA (RNLDA), is proposed to solve the overfitting problem.

• Practical and efficient RNLDA algorithms and an automatic parameter setting algorithm are proposed.

Abstract

Null space Linear Discriminant Analysis (NLDA) was proposed twenty years ago to overcome the singularity problem of LDA in practical applications. With two decades of technique development, many Discriminative Dimensionality Reduction (DDR) methods that outperform NLDA have been proposed. This paper provides new insight into NLDA and illustrates that NLDA is much more powerful after solving its inherent problem. The main problem of NLDA is the intrinsic overfitting problem. An ideal NLDA model is proposed to analyze its overfitting problem. Based on the ideal NLDA model, a more reasonable Representative NLDA (RNLDA) method is proposed to prevent overfitting. Two simple but efficient RNLDA algorithms are proposed to implement the RNLDA method with a theoretical proof. This study theoretically analyzed and indicated that applying the classical but simple hold-out pretraining method can automatically set the only parameter to achieve high performance. Extensive experiments with eight databases demonstrate the superior performance of the RNLDA method over state-of-the-art DDR methods.

Introduction

As the technique development of sensing and storage hardware improves, the processing of an increasing number of high-dimensional data is needed in many fields. Due to the curse of dimensionality [1], processing data in their original high dimension is computationally expensive or even impossible. Therefore, Dimensionality Reduction (DR) has an important role in processing these data. The main goal of DR is to map high-dimension data into a low dimension with a minimal loss of desired information. In most applications, including machine learning and pattern recognition, discriminative information is desired. Therefore, DR also refers to Discriminative Dimensionality Reduction (DDR). In addition to reducing computation and storage, DDR usually also has another role- feature selection, which renders subsequent classification more precise in the obtained low-dimensional subspace than in the original high dimension. Therefore, DDR methods are also referred to as feature selection methods [2], [3]. Depending on whether the label information is employed in training, DDR methods can be classified into three types of methods: unsupervised methods [4], [5], semisupervised methods [6], [7], and supervised [8], [9] methods. Generally, supervised DDR methods outperform the other two types of methods because they utilize label information during training.

Linear Discriminant Analysis (LDA) is a classical and famous supervised DDR method [8], [10]. LDA introduced the idea of obtaining a low-dimensional subspace that maximizes the ratio of the between-class difference to the within-class difference. Numerous variations of the original LDA method have been proposed to improve it from different perspectives.

The main problem that LDA confronted in many practical applications is the singularity, or Small Sample Size (SSS) problem [11]. When the dimension of data is considerably larger than the number of training samples, such that the within-class and total scatter matrices are singular, LDA can not be applied. To address the singularity issue, numerous variations of LDA have been proposed. Direct LDA (DLDA) [12] employs two main steps to overcome the singularity problem. First, a transformation matrix is computed to transform the training data to the range space of the between-class scatter matrix. Second, the dimensionality of the transformed data is further transformed using regulating matrices. In the Regularized LDA (RLDA) [13], [14], a small perturbation is added to the within-class scatter matrix to make it nonsingular. Subspace LDA (SLDA) is the most popular variation with extensive applications. First, the original data are projected to a lower-dimensional subspace using PCA, which makes the transformed within-class scatter matrix full rank. Second, LDA is utilized to further reduce the dimensionality [15]. Therefore, the method is also referred to as PCA + LDA. Null space LDA (NLDA) [16] projects data into the null space of the within-class scatter matrix and then maximizes the between-class difference in the null space. Many other variations overcome the singularity problem in different ways, such as Pseudo-inverse LDA [17], Orthogonal LDA [18], and Angle Linear Discriminant Embedding [19].

In addition to addressing the singularity issue, many other methods attempt to improve LDA in modeling. LDA utilizes the between-class, within-class or total scatter matrices to model the global structures of data. Many researchers have reported that leaning the local structure of data is advantageous. Based on this idea, numerous local structure-based DDR methods have been proposed. For example, Locality Sensitive Discriminant Analysis (LSDA) considers the k-nearest neighbors of each training sample [20]. Among the neighbors, pairs from the same classes are used to represent the within-class difference, and pairs from different classes are used to represent the between-class difference. In Marginal Fisher Analysis (MFA) [9], k₁-nearest neighbors within the same class of each sample are used to represent the within-class difference, and k₂ pairs of nearest samples from the different classes are used to represent the between-class difference. Manifold Partition Discriminant Analysis(MPDA) [21] combines the pairwise differences of neighboring samples with the tangent space [22] and achieves high performance. Many other local structure-based methods benefit from neighboring samples in different ways [23], [24], [25], [26]. On the other hand, in recent years, many researchers claimed that LDA is not sufficiently robust due to the use of L₂-norm and attempted to improve LDA by introducing the L₁-norm. For example, L₁-LDA [27] was reported to be more robust than LDA both theoretically and experimentally. Recently, a new formulation of LDA-Robust LDA (RLDA) [28] was proposed to improve the robustness using joint L_2,1-norm minimization on the objective function. An efficient iterative algorithm was also presented to solve the L_2,1-norm minimization optimization problem. The are also a kind of variations, that attempt to introduce a nonlinear property [29]. The underlying principle in these methods is the sparse nature of signals, and L₁-norm is a useful tool to take use of it. Very recently, further developments based on sparsity and low-rank preservation, which benefit from the sparse nature, have been made [30], [31].

As previously mentioned, NLDA, which was proposed 20 years ago, is a typical variation of LDA that is aimed to overcome the singularity problem. However, compared with SLDA (PCA+LDA), its application is not extensive. In addition, with the development of new DDR methods that were reported to achieve higher performance, e.g., the local structure-based methods and L₁-norm-based methods, the research and application of NLDA has become scarce. This paper provides new insight into NLDA and illustrates that the performance of NLDA is affected by its intrinsic overfitting problem. By solving the overfitting problem, NLDA can achieve very high performance that is even superior to that of state-of-the-art DDR methods.

The main contributions of this paper are summarized as follows:

• It is pointed out that the main problem of NLDA is the intrinsic overfitting problem. An ideal NLDA model is proposed to analyze this overfitting problem. Based on the idea NLDA model, a more reasonable Representative NLDA (RNLDA) method is proposed to prevent overfitting.

• Practical and efficient RNLDA algorithms and an automatic parameter setting algorithm are proposed to implement the RNLDA method. First, two simple but efficient RNLDA algorithms are proposed to implement the RNLDA method. Second, the results of the algorithms are theoretically proven to be the solution of the RNLDA method. Last, the study analyzed and proved that applying the classical but simple hold-out pretraining method can automatically set the only parameter to achieve high performance.

Extensive experiments on eight extensively employed and publicly available databases were conducted to verify the proposed method. The results demonstrate the superior performance of the RNLDA method over state-of-the-art DDR methods.