Tensor linear Laplacian discrimination (TLLD) for feature extraction

Abstract

Discriminant feature extraction plays a central role in pattern recognition and classification. In this paper, we propose the tensor linear Laplacian discrimination (TLLD) algorithm for extracting discriminant features from tensor data. TLLD is an extension of linear discriminant analysis (LDA) and linear Laplacian discrimination (LLD) in directions of both nonlinear subspace learning and tensor representation. Based on the contextual distance, the weights for the within-class scatters and the between-class scatter can be determined to capture the principal structure of data clusters. This makes TLLD free from the metric of the sample space, which may not be known. Moreover, unlike LLD, the parameter tuning of TLLD is very easy. Experimental results on face recognition, texture classification and handwritten digit recognition show that TLLD is effective in extracting discriminative features.

Introduction

Discriminant feature extraction is an important topic in pattern recognition and classification. Principal component analysis (PCA) and linear discriminant analysis (LDA) are two traditional algorithms for linear discriminant feature extraction. Both methods involve scatters computed in the Euclidean metric, i.e., the underlying assumption is that the sample space is Euclidean. Both PCA and LDA have found wide application in pattern recognition and computer vision. For example, they are known as the famous Eigenfaces method and Fisherfaces method in face recognition [2], respectively. And many variants of LDA have shown good performance in various applications [9], [12], [20], [21], [22], [27]. As the data manifold may not be linear, some nonlinear discriminant feature extraction algorithms, e.g., locality preserving projections (LPP) [8] and linear Laplacian discrimination (LLD) [31], have recently been developed. In addition, the kernel trick [15] is also widely applied to extend linear feature extraction algorithms to nonlinear ones by performing linear operations in a higher or even infinite dimensional space transformed by a kernel mapping function.

It is worth noting that most of the existing discriminant analysis methods are vector based, i.e., the input data are always (re)arranged in a vector form regardless of the inherent correlation among different dimensions. In practice, vector-based methods have been found to have some intrinsic problems [26]: singularity of within-class scatter matrices, limited available projection directions and high computational cost. Much work has been done to deal with these problems [20], [21], [22], [4], [5]. Recently, several tensor-based methods have been proposed as alternatives to overcome these drawbacks. Tensor-based methods respect the dimensional structure of data, hence can extract better discriminant features robustly. They perform well particularly when the number of samples is relatively small, a case in which vector-based methods often suffer the singularity problem. Along this line, Ye et al.'s 2DLDA [29] and Yan et al.'s DATER [26] are the tensor extensions of the popular vector-based LDA algorithm. And tensor LPP [6], [7] is an extension of LPP, also preserving local neighbor structures of tensor samples. All these methods work in tensor spaces with Euclidean metrics if metrics are to be used.

Despite the success of various subspace learning algorithms, we notice that almost all of them rely on the Euclidean assumption on the data space when computing the distance between samples, unless the appropriate metric for the data space is known, e.g., KL divergence or ${\chi }^2$ distance are suitable for histogram-based data. Distance metric learning attempts to learn metrics from data. However, it has mainly focused on finding a linear distance metric that optimizes the data compactness and separability in a global sense [23], [24], [28]. It is computationally expensive when treating high-dimensional data, and no current nonlinear dimensionality reduction approaches can learn an explicit nonlinear metric [28]. Approximated geodesic distance [18], which attempts to estimate the distances among samples, could help alleviate, but also not resolve, the issue of metrics. For example, a slenderly distributed cluster can have large geodesic distance between the samples, which makes distance-based cluster analysis error-prone. Actually, what is more important is the structure of the data, rather than the absolute distance between the data samples.

From the above observations, we propose the tensor linear Laplacian discrimination (TLLD) method for nonlinear feature extraction from tensor data. TLLD could be viewed as an extension of both LDA and LLD [31] in directions of nonlinearity and tensor representation. LLD has shown its superiority of feature extraction in nonlinear spaces [31], but it still has all the abovementioned drawbacks of vector-based methods because it has the same number of available projection directions and the same null spaces of the within-class scatter matrices as LDA (the proof is in Appendix A). And although LLD has aimed at removing the metric assumption by introducing weights to the scatter matrices, nonetheless the weights are still defined as a function of the distance in the sample space. Therefore, LLD still needs the a priori assumption on the metric of the sample space. To further reduce the dependence on the metric of the sample space, TLLD computes the weights based on the contextual distances instead, which are measured by the contribution to the structure of data in the sample space. This idea is inspired by the recent work on structural perception of data [13], [30]. In order to match the tensor nature of data, we further extend the vector-based coding length [13], [30] to tensor coding length as the contextual set [30] descriptor. Another advantage of using contextual-distance-based weights is that tuning the time variable in the weights now becomes very easy by rescaling. In short, TLLD handles two kinds of structure in the sample data, the tensor structure within each individual sample and the distributional structure across all samples, in a unified way.

The rest of this paper is organized as follows. We first present TLLD in Section 2, then discuss the choice of the weights for scatter matrices in Section 3. The experimental results are presented in Section 4 and Section 5 concludes our paper.