Incremental min–max projection analysis for classification

Abstract

For data classification, the standard implementation of projection algorithms do not scale well with large dataset size. It makes the computation of large samples infeasible. In this paper, we utilize a block optimization strategy to propose a new locally discriminant projection algorithm termed min–max projection analysis (MMPA). The algorithm takes into account both intra-class and interclass geometries and also possesses the orthogonality property. Furthermore, an incremental MMPA is proposed to learn the local discriminant subspace with newly inserted data by employing the idea of singular value decomposition updating algorithm. Moreover, we extend MMPA to the semi-supervised case and nonlinear case, namely, semi-supervised MMPA and kernel MMPA. The experimental results on image database, hand written digit database, and face database demonstrate the effectiveness of those proposed algorithms.

Introduction

The problem of feature extraction is one of the core issues for data mining and classification. Using a more efficient feature extraction method can improve the classification results in the reduced subspace. The problem of dimensionality reduction can be described as follows. Consider a data set X, which consists of n samples x_i (1≤i≤n) in a high-dimensionality space R^m. The objective of dimensionality reduction is to compute a faithful low-dimensionality representation of X, i.e. Y=[y₁,…,y_n]∈ R^d×n, where d⪡m.

Over the past decades, numerous dimension reduction methods have been proposed to find the low-dimensional feature representation. The two most popular techniques for this purpose are principal component analysis (PCA) [1] and linear discriminant analysis (LDA) [2]. PCA is an unsupervised algorithm based on the computation of low-dimensional representation of high dimensional data, which maximizes the total scatter. Comparatively LDA is a supervised feature extraction technique for pattern recognition, and it tends to find a set of projective directions which maximize the between-class scatter and simultaneously minimize the within-class scatter. An intrinsic limitation of LDA is that it usually suffers from the small sample size (SSS) problem, where the sample size is much smaller than the size of dimensionality of samples. Additionally, both PCA and LDA can only see the global Euclidean structure but cannot discover the embedding structure hidden in the high-dimensional data.

In order to exploit the local discriminative manifold structure, a lot of subspace learning techniques have been proposed, such as locality preserving projections (LPP) [3], Maximal Similarity Embedding [4], Local Spline Discriminant Projection [5], and Neighborhood Preserving Embedding [6]. Recently, some researchers pointed out that enforcing an orthogonality relationship between projection directions can achieve competitive effectiveness, and therefore the orthogonal neighborhood preserving projection (ONPP) was introduced [5], [7]. However, for classification problems, ONPP and LPP (even in a supervised setting) only focus on the intra-class geometrical information while the interaction of samples from different classes is ignored.

More recently, numerous algorithms have been proposed which take the intra-class preserving into consideration as well as the interclass discriminant [8], [9], [10], [11]. Among them, the locality sensitive discriminant analysis (LSDA) [8] and its variation maximum margin projection (MMP) [9] are two typical examples, which gain more competitive results in image recognition applications. Yan et al. [12] explained most of these manifold learning techniques as a general framework that can be defined in a graph-embedding way. Generally, a discriminative feature extraction algorithm is summarized as a graph-based constraint embedding by defining the intrinsic and penalty graphs. In other words, it finds a set of projection directions in the linear embedded subspace, i.e., J(U)=argmin{(U^TXLX^TU)/(U^TXBX^TU)} or J(U)=argmin{U^TXLX^TU}, subject to U^TXBX^TU=c, where c is a constant, X is the data matrix, and L is the Laplacian matrix of intrinsic graph, which is defined as follows: L=D−W, D_ii=∑_jW_ij. Here, W is the affinity matrix of the intrinsic graph. In addition, B can be the Laplacian matrix of penalty graph, B=D_p−W_p, where W_p indicates the adjacency matrix of penalty graph. W_p describes the similarity of interclass data which should be avoided for classification. D_p is the diagonal matrix defined in the graph-embedding framework. The general solution of the optimal U is to find the eigenvector corresponding to the smallest eigenvalue using the generalized eigenvalue decomposition (ED) XLX^TU=λXBX^TU, which has a heavy computation burden because of the high data dimensionality, especially in image and video applications.

Incremental learning has already attracted much attention as a result of the increasing demand for developing machine vision/ intelligent systems. Numerous incremental learning algorithms have been proposed, especially in the data-mining domain and the image-retrieval field [13], [14], [15]. Most of these recent works are designed for incremental principal component analysis [16], [17] and incremental linear discriminant analysis [18], [19]. Both of them are global statistic feature extraction algorithms. To our best knowledge, there are few works focusing on the incremental local discriminant embedding except for the ILDSE proposed by Miao et al. [20], which demands that B must be Laplacian matrix.

In this paper, we propose a new algorithm termed min–max projection analysis (MMPA) based on the perspective of block optimization [21]. MMPA offers three main benefits (1) the algorithm takes into account both intra-class and interclass geometries so that it can achieve better performance in classification; (2) the algorithm produces the orthogonal projection matrix; and (3) the combination matrix of this algorithm can be iteratively computed for the newly inserted samples.

Furthermore, an incremental MMPA is introduced to learn the discriminative sub-manifold structure incrementally, namely, incremental MMPA (IMMPA). This paper also extends MMPA to the semisupervised case and nonlinear case, termed semisupervised MMPA (SMMPA) and kernel MMPA (KMMPA) respectively. SMMPA is produced by incorporating the additional unlabeled samples, and KMMPA performs MMPA in reproducing kernel hilbert space (RKHS).They are powerful. For generalization, the proposed algorithm is also based on graph-embedding framework [12] that incorporates the graph adjacency to represent the discriminative weights of data.

The rest of the paper is organized as follows. Section 2 introduces MMPA algorithm as well as the incremental implementation. Subsequently, the semisupervised MMPA algorithm is proposed in Section 3. In Section 4, the algorithm is extended to the nonlinear case, termed KMMPA. The experimental performance of the proposed algorithms is presented in Section 5. Finally, we conclude this paper in Section 6.