Prototype learning and collaborative representation using Grassmann manifolds for image set classification

Highlights

• Principle component and variation subspaces are constructed for an over-complete dictionary.

• A novel prototype and variation model (P+V) based collaborative representation for Grassmann manifolds is proposed to deal with image set classification naturally.

• Previous special matrices are generalized to common sparse matrices.

• Experimental results show the superiorities of our methods.

Abstract

Image set classification using manifolds is becoming increasingly more attractive since it considers non-Euclidean geometry. However, with the success of dictionary learning for image set classification using manifolds, how to learn an over-complete dictionary is still challenging. This paper proposes a novel prototype subspace learning method, in which a set of images is represented by a linear subspace and then mapped onto a Grassmann manifold. With this subspace representation, class prototypes and intra-class differences can be represented as principal components and variation subspaces, respectively. Isometric mapping further maps the manifolds into the symmetrical space via collaborative representation, which permits a closed-term solution. The proposed method is evaluated for face recognition, object recognition and action recognition. Extensive experimental results on the Honda, Extended YaleB, ETH-80 and Cambridge-Gesture datasets verify the superiority of the proposed method over the state-of-the-art methods.

Introduction

Traditional classification techniques are used for single image classification problems. With the development of digital imaging technology, such as multiview cameras and photo albums, multiple images can be conveniently available for one subject. Therefore, image set classification has recently attracted increased attention in computer vision. Compared with a single image, a set of images comprises individual appearance changes under arbitrary poses, illumination conditions, expressions and other factors. While one subject can be described using increasingly more information, image set classification becomes increasingly challenging due to large intraclass variations.

Image set classification generally focuses on two key issues: how to effectively model an image set and how to measure the similarity between two sets. Many image set classification methods have been proposed [1], [2]. Usually, linear subspaces are used to model image sets [3], [4], and the canonical angle between two subspaces is defined as the similarity between the two sets. Among them, the mutual subspace method (MSM) [3] is the most classic image set subspace method. However, the MSM fails to distinguish the similarities among intraclass and interclass objects. As such, the constrained mutual subspace method (CMSM) [5] was proposed to map original subspaces into a difference subspace such that the difference components can be extracted. It’s notable that this constrained subspace is derived by removing the base vectors of all the reference class subspaces with large variances and keeping the base vectors with relatively small variances. Unfortunately, the CMSM neglects the relationships among subspaces from the same class. Boosted Manifold Principal Angles (BoMPA) [6] extend the concept of principal angles between linear subspaces to manifolds with arbitrary nonlinearities. By contrast, sample-based methods [7] use an affine hull/convex hull to represent an image set. These methods measure the similarity by matching the specific sample-based statics. As studied in [1], an image set can also be represented as a linear subspace, which is denoted as a point on a Grassmann manifold.

Sparse representation and dictionary learning are attracting significant attention [8] for classification. SRC [8] treats the entire training samples as a dictionary for face recognition. To have discriminative sparse coding, several methods consider non-Euclidean geometry. Harandi et al. [9] embed Grassmann manifolds into the space of symmetric matrices using an isometric mapping. Later, sparse coding using symmetric positive definite manifolds and Bregman Divergences was introduced in [10]. Sparse representation emphasizes the l₁-norm sparsity but ignores the collaborative ability [11]. It is also difficult to acquire many image sets to construct an over-complete dictionary. Recently, Deng et al. [12] proposed a superposed linear representation-based classification (SLRC) model that represents a test image using a superposition of the class centroids and the shared intraclass differences. SLRC can improve the robustness against contaminated training samples and it requires fewer samples to construct an over-complete dictionary. However, SLRC is used for single image classification, and it may not be appropriate for image set classification. Hence, designing an over-complete dictionary in the collaborative representation framework is significant for image set classification. This paper aims to introduce a novel prototype plus variation subspaces ($P+V$) model and allows this model to naturally and effectively deal with image set classification. How to derive the prototype representation and variation dictionaries in the collaborative representation framework still remains challenging.

To solve the aforementioned limitations, this paper proposes prototype learning and collaborative representation using Grassmann manifolds (GPLCR) and its generalized version (GGPLCR) for image set classification. The proposed methods learn the prototypes for image sets using Grassmann manifolds to construct dictionary to improve the discrimination. More significantly, the proposed methods jointly learn to derive an over-complete dictionary by extracting less discriminative intraclass variances. Therefore, the learned prototypes and variations of the image sets contain common class characteristics and intraclass variations, respectively. Collaborative representation using Grassmann manifolds is developed for classification that maps into the symmetrical space. This optimization with the l₂-norm regularization is simple, and it admits a closed-form solution. An illustration of the proposed methods is shown in Fig. 1.

The main contributions of this paper are summarized as follows.

• We naturally and effectively address image set classification via $P+V$ model. This model can preserve more discriminative class-specific portions and also extract less discriminative intraclass variances. A subsequent over-complete dictionary is constructed to improve the robustness against contaminated training samples and require fewer samples.

• We integrate prototype learning into the collaborative representation framework, which changes dictionary learning problems into standard Euclidean problems with L1 and L2 penalty terms. Further, we generalize previous special matrices into common sparse matrices and achieve better performance.

• The proposed method is applied to several recognition tasks including face recognition, object recognition and action recognition. Experimental results show the superiority of the proposed method over the state-of-the-art methods.

The rest of the paper is organized as follows: In Section 2, we briefly introduce the related work. After creating the P+V model, we employ this model in collaborative representation on Grassmann manifolds in Section 3. In Section 4, we give the experimental evaluation and analysis. The conclusion is drawn in Section 5.