Discriminant common vectors versus neighbourhood components analysis and Laplacianfaces: A comparative study in small sample size problem

Abstract

Discriminant common vectors (DCV), neighbourhood components analysis (NCA) and Laplacianfaces (LAP) are three recently proposed methods which can effectively learn linear projection matrices for dimensionality reduction in face recognition, where the dimension of the sample space is typically larger than the number of samples in the training set and consequently the so-called small sample size (SSS) problem exists. The three methods obtained their respective projection matrices based on different objective functions and all claimed to be superior to such methods as Principal component analysis (PCA) and PCA plus Linear discriminant analysis (PCA+LDA) in terms of classification accuracy. However, in literature, no comparative study is carried out among them. In this paper, we carry out a comparative study among them in face recognition (or generally in the SSS problem), and argue that the projection matrix yielded by DCV is the optimal solution to both NCA and LAP in terms of their respective objective functions, whereas neither NCA nor LAP may get their own optimal solutions. In addition, we show that DCV is more efficient than both NCA and LAP for both linear dimensionality reduction and subsequent classification in SSS problem. Finally, experiments are conducted on ORL, AR and YALE face databases to verify our arguments and to present some insights for future study.

Introduction

In face recognition, we usually employ appearance-based methods [1], [2]. One primary advantage of appearance-based methods is that it is not necessary to create representations or models for face images, since for a given face image, its model is now implicitly defined in the face image itself [3]. When using appearance-based methods, we usually represent an image of size r×c pixels by a vector in a d-dimensional space, where d=rc. Although such an appearance based representation is simple in form, the corresponding dimensionality d is too large to realize robust and fast recognition [3], and is typically lager than the number of samples in the training set which leads to the so-called small sample size (SSS) problem. A common way to resolve this problem is to use dimensionality reduction techniques. Discriminant common vectors (DCV) [7], [8], Laplacianfaces (LAP) [17] and neighbourhood components analysis (NCA) [18] are three recently proposed methods, which can effectively learn linear projection matrices for dimensionality reduction in face recognition.

DCV [7], [8] aims at solving the small sample size (SSS) problem in linear discriminant analysis (LDA) [4], [5], [6] which maximizes the Fisher's linear discriminat criterion as follows

$$J_{\text{FLD}}(W_{\text{opt}})=\text{arg}\underset{W}{\text{max}}\left|W^{\text{T}}{S}_{\text{b}}W\right|/\left|W^{\text{T}}{S}_{\text{w}}W\right|$$

where S_w is the within-class scatter matrix and S_b is the between-scatter matrix. When the SSS problem takes place, S_w will be typically singular and LDA cannot be applied directly. DCV¹ remedies this by calculating the projection matrix in the null space of S_w, and as a result gets an optimum (infinite) of objective function (1).

LAP [17] originates from viewpoint of preserving the locality structure of the image space. To this end, it models a manifold [13], [14], [15] structure by a nearest-neighbor graph, constructs a face subspace by locality preserving projections (LPP) [16], and performs dimensionality reduction by a set of feature images called Laplacianfaces.

NCA [18] aims at learning a Mahalanobis distance measure to be used in the k nearest neighbor (KNN) classification. Subtly, it boils learning such Mahalanobis distance down to learning a linear projection (or transformation) matrix, and at the same time avoids the inverse operation of the matrix in calculating traditional Mahalanobis distance metric. The projection matrix in NCA is obtained through optimizing the KNN leave-one-out (LOO) classification performance on the training set, so the learned Mahalanobis distance metric or equivalently the projection matrix is directly related to the classification performance. This is the main characteristic of NCA, and is quite different from the dimensionality reduction methods mentioned above (e.g. DCV, LAP) whose objective functions are not directly associated with the classification decision. By restricting the projection matrix in the distance measure learning to a non-square one, NCA can be used for dimensionality reduction [18].

The three methods all claimed to be superior to principal component analysis (PCA) [1] and PCA+LDA [4], [5], [6], namely: (1) DCV is superior to PCA and PCA+LDA in terms of recognition accuracy, efficiency and numerical stability [8]; (2) PCA and PCA+LDA can be obtained from different graph models in LAP, and LAP provides a better representation and achieves lower classification error rates in face recognition [17]; and (3) When labeled data is available, NCA performs better both in terms of classification performance in the projected representation and in terms of visualization of class separation as compared to the standard methods of PCA and LDA [18]. However, there is no comparative study among them in literature. The purpose of this paper is to compensate this by a comparative study among them, and to get some sight from such comparative study. It is worthwhile to highlight our contributions in this paper as follows:

• We for the first time in literature perform a comparative study among DCV, LAP and NCA, and argue that in SSS problem (e.g. face recognition) the projection matrix yielded by DCV is the optimal solution to both NCA and LAP in terms of their respective objective functions, whereas neither NCA nor LAP may get their own optimal solutions.

• We show that DCV is more efficient than both NCA and LAP for both linear dimensionality reduction and subsequent classification in SSS problem.

• We reveal the essence of DCV, i.e. calculating the projection matrix is equivalent to solving a thin QR decomposition problem, which is easier for both understanding and being extended to its nonlinear version by kernel trick [22], [23], [24], [25].

• We experimentally give the application scope of DCV, namely, when MSV (defined in Section 4) is relatively small, it performs well while on the contrary when MSV is relatively large, it performs poorly.

The rest of the paper is organized as follows. In Section 2, DCV, NCA and LAP are, respectively, reviewed. In Section 3, we carry out a comparative study among these three methods in SSS problem. In Section 4, we report experimental results on several face databases. Finally in Section 5, we provide some concluding remarks and suggestions for future work.