Enhanced regularized least square based discriminative projections for feature extraction

Abstract

The regularized least square based discriminative projections (RLSDP) for extracting features was recently proposed, which aims to seek discriminant projection directions that maximize the between-class scatter and minimize the within-class compactness. However, in RLSDP, the retrieval samples are reconstructed by the coefficients only associated with the same class, and may have large errors. Moreover, the distances between each sample and other within-class samples characterize the most important within-class compactness information, and are not minimized in RLSDP. To deal with the above two problems, we propose an enhanced regularized least square based discriminative projections (ERLSDP). ERLSDP utilizes all the related coefficients of each sample for reconstruction and explicitly minimizes the distances between all the within-class samples, and thus it has better reconstruction accuracy and more discriminating power than that of RLSDP. Experimental results demonstrate that ERLSDP gets a clear improvement over RLSDP when the training sample size is small.

Introduction

Feature extraction, which aims to produce compact and effective low-dimensional feature representations of high-dimensional data, has been extensively studied over the past several decades. Compared with the global based principal component analysis (PCA) [1] and linear discriminant analysis (LDA) [2] approaches, manifold learning methods are more appealing since they can discover the local intrinsic structure of data. Representative manifold learning methods include locality preserving projections (LPP) [3], locality preserving discriminant projections (LPDP) [4], discriminative locality alignment (DLA) [5], discriminant locality preserving projections (DLPP) [6], marginal Fisher analysis (MFA) [7], etc. Although their motivations are different, they all can be unified in the graph embedding (GE) framework [7], and their differences lie in graph construction. Manifold learning has found its wide applications in various fields. For example, Li et al. [8] developed a discriminative distance metric learning (DML) algorithm based on manifold learning, and further derived a distributed and parallel computational scheme to deal with the large-scale metric learning problem. Reference [9] exploited the manifold learning method to analyze multivariate variable-length sequence data. Gao et al. [10] integrated local and global manifold structures for face and image classification.

Recently, sparse representation has shown its promising performance in many domains [11], [12], [13], [14], [15]. For instance, Wright et al. [11] proposed a sparse representation based classification (SRC) for face recognition. Zhou et al. [12] proposed a double shrinking algorithm (DSA) for sparse projection eigenvectors. Moreover, many research efforts [16], [17] had shown that the neighborhood relationship of each data could be adaptively obtained by sparse representation methods, and the resulted ℓ₁-graph was robust to noise. Based on ℓ₁-graph, Qiao et al. [16] proposed a sparsity preserving projections (SPP) for feature extraction, which aims at preserving the sparse reconstruction relationship of the data both in original space and low-dimensional embedding space. By combining the supervised SPP and maximum margin criterion, Gui et al. [18] introduced a discriminant sparse neighborhood preserving embedding (DSNPE) algorithm. Gao et al. [10] gave a discriminative sparsity preserving projections (DSPP), which first employs sparse representation to build an intrinsic graph and a penalty graph, and then integrates global within-class structure for dimensionality reduction. Despite their good performance, sparse representation methods need to solve ℓ₁ norm minimization problem, which has higher computational complexity.

Zhang et al. [19], [20] claimed that the collaborative representation mechanism was the key factor for the success of SRC, and proposed a collaborative representation based classification (CRC) method. CRC replaces the ℓ₁ norm in SRC with simpler ℓ₂ norm. CRC has similar properties and competitive classification performance to SRC. Based on CRC, Yang et al. [21] constructed a ℓ₂-graph and developed a collaborative representation based projections (CRP) to preserve the collaborative reconstruction relationship of the data. Yin et al. [22] proposed a collaborative representation reconstruction based projections (CRRP). The projection matrix in CRRP is obtained by maximizing the collaborative reconstruction between-class scatter and minimizing the collaborative reconstruction within-class scatter. Another method was proposed in [23], which is similar to [22]. In [24], Yang et al. developed a regularized least square based discriminative projections (RLSDP). It maximizes the between-class scatter adopted by LDA and minimizes the within-class compactness by the reconstruction residual from the same class. However, RLSDP includes two main problems. First, the reconstructions by the coefficients only corresponding to the same class will have large errors, and thus RLSDP cannot give the best reconstruction for each sample. Second, it does not minimize the distances between each sample and other within-class samples, which is important for minimizing the within-class compactness.

To address the above two problems, we propose an enhanced regularized least square based discriminative projections (ERLSDP). In ERLSDP, each sample is reconstructed by all the associated coefficients, which results in smaller reconstruction error. More importantly, the distances between each sample and all its reconstructed within-class samples, which characterize the most important within-class compactness, are minimized. The optimal discriminant projection of ERLSDP is achieved by maximizing the between-class scatter and minimizing the within-class compactness simultaneously. Experiments on three face databases indicate that our ERLSDP performs better than RLSDP.

The main contributions of our work are as follows. (1) We make use of the whole representation coefficients to reconstruct each sample. In contrast, the original RLSDP only utilizes the partial representation coefficients corresponding to the same class for reconstruction of each sample. Thus, our ERLSDP achieves smaller reconstruction error and better classification performance. (2) We build a weight matrix to explicitly characterize the within-class geometry of the data, and minimize the distances between all the within-class samples. Meanwhile, by maximizing the between-class scatter, the samples sharing the same class label will be pulled together and those from different classes will be pushed apart, which is a very desirable property for classification tasks.

The rest of this paper is structured as follows. In Section 2, the regularized least square (RLS) and RLSDP are briefly reviewed. The proposed ERLSDP is detailed in Section 3. The experimental results are illustrated in Section 4, and the conclusions are given in Section 5.