Improved discriminate analysis for high-dimensional data and its application to face recognition

Abstract

Many pattern recognition applications involve the treatment of high-dimensional data and the small sample size problem. Principal component analysis (PCA) is a common used dimension reduction technique. Linear discriminate analysis (LDA) is often employed for classification. PCA plus LDA is a famous framework for discriminant analysis in high-dimensional space and singular cases. In this paper, we examine the theory of this framework and find out that even if there is no small sample size problem the PCA dimension reduction cannot guarantee the subsequent successful application of LDA. We thus develop an improved discriminate analysis method by introducing an inverse Fisher criterion and adding a constrain in PCA procedure so that the singularity phenomenon will not occur. Experiment results on face recognition suggest that this new approach works well and can be applied even when the number of training samples is one per class.

Introduction

Linear discriminate analysis (LDA) is a useful tool for pattern classification. Although successful in many cases, many LDA-based algorithms suffer from the so-called “small sample size problem” (SSS) [1] which exists in high-dimensional pattern recognition tasks, where the number of available samples is smaller than the dimensionality of the samples. An active field where such problem appears is image retrieval/classification. In particular, face recognition (FR) [2], [3] technique has found a wide range of applications. As a result, numerous FR algorithms have been proposed, and theories related to these fields have been studied. Among various solutions to this problem, the most successful are those appearance-based approaches, such as Eigenfaces and Fisherfaces [4], [5], [6], [7], are built on these techniques or their variants. Since SSS problems are common, it is necessary to develop new and more effective algorithms to deal with them. A number of regularization techniques that might alleviate this problem have been suggested. Mika et al. [8], [9] used the technique of making the inner product matrix nonsingular by adding a scalar matrix. Baudat and Anouar [10] employed an orthogonal decomposition technique to avoid the singularity by removing the zero eigenvalues. In Refs. [11], [12], [13] the technique of regularization was used. A well-known approach, called Fisher discriminant analysis (FDA), to avoid the SSS problem was proposed by Belhumeur et al. [4]. This method consists of two steps. The first step is the use of principal component analysis (PCA) for dimensionality reduction. The second step is the application of LDA for the transformed data. The basic idea is that after the PCA step the within-class scatter matrix for the transformed data is not singular. Although the effectiveness of this framework in face recognition is obvious, see Refs. [3], [4], [14], [15], and the theoretical foundation for this framework has also been laid [16], [17], yet in this paper we find out that the PCA step cannot guarantee the successful application of subsequent LDA, the transformed within-class scatter matrix might still be singular.

On the other hand, many researchers have been dedicated to searching for more effective discriminant subspaces [16], [17], [18], [19], [20], [21], [22], [23]. A significant result is the finding that there exists crucial discriminative information in the null space of the within-class scatter matrix. This kind of discriminative information is called irregular discriminant information, in contrast with regular discriminant information outside of the null space [24].

Unfortunately, in order to proceed LDA after PCA, many of the above methods discard the discriminant information contained in the null space of the within-class scatter matrix, yet this discriminant information is very effective for the SSS problem. Chen et al. [19] emphasized the irregular information and proposed a more effective way to extract it, but overlooked the regular information. Yu and Yang [16] took two kinds of discriminatory information into account and suggested extracting them within the range space of the between-class scatter matrix. Since the dimension of the range space is up to $K-1$, Yu et al.'s algorithm, direct LDA (DLDA), is computationally more efficient for SSS problems in that the computational complexity is reduced to be $\mathcal{O}(K^3)$.

This paper is the full version of Ref. [25]. Motivated by the success and power of the two-phase framework (PCA plus LDA) in pattern regression and classification tasks, considering the importance of the irregular information in the null space of the within-class scatter matrix, and in view of the limitation of the PCA step, we propose a new framework for the SSS problem. The algorithm of our new method modifies the procedure of PCA and derives the regular and irregular information from the within-class scatter matrix by a new criterion, which is called inverse Fisher discriminant criterion.

The rest of this paper is organized as follows. Since our method is built on PCA and LDA, in Section 2, we start the analysis by briefly reviewing the two latter methods. We point out the deficiency of the PCA plus LDA method through an example. Following that, the proposed framework is introduced and analyzed in Section 3. The relationship of the two different frameworks is also discussed. Algorithm of the new framework and the computational complexity of the algorithm will be considered, too. In Section 4, experiments with face image data are presented to demonstrate the effectiveness of the new method. Conclusions are summarized in Section 5.