Supervised class-specific dictionary learning for sparse modeling in action recognition

Abstract

In this paper, we propose a new supervised classification method based on a modified sparse model for action recognition. The main contributions are three-fold. First, a novel hierarchical descriptor is presented for action representation. To capture spatial information about neighboring interest points, a compound motion and appearance feature is proposed for the interest point at low level. Furthermore, at high level, a continuous motion segment descriptor is presented to combine temporal ordering information of motion. Second, we propose a modified sparse model which incorporates the similarity constrained term and the dictionary incoherence term for classification. Our sparse model not only captures the correlations between similar samples by sharing dictionary, but also encourages dictionaries associated with different classes to be independent by the dictionary incoherence term. The proposed sparse model targets classification, rather than pure reconstruction. Third, in the sparse model, we adopt a specific dictionary for each action class. Moreover, a classification loss function is proposed to optimize the class-specific dictionaries. Experiments validate that the proposed framework obtains the performance comparable to the state-of-the-art.

Introduction

In recent years, a large number of approaches have been proposed to fulfill action recognition. Among them, bag of visual words approaches are greatly popular, due to their simple implementation and good reliability. A video clip is summarized by the histogram of its local features. By fully exploiting local space–time features, the bag of visual words approaches are robust to noise, occlusion and geometric variation, without requiring reliable tracks on a particular subject. Recent work has shown promising results using local space–time features together with bag of visual words models. The methods in Refs. [1], [2], [3] are classical interest-point-based methods for action recognition. These approachs extract the local feature from a single interest point, and achieve good results. However, the conventional interest-point-based methods describe the feature of a single interest point. They are mainly based on the individual power of the interest point, and therefore do not consider the spatio-temporal relationship between them. As an improvement, Gilbert et al. [4] perform dense interest point detection, and compute the distribution of interest points in a small area. Although this approach utilizes some spatial information, it does not exhibit the temporally ordering information in actions. A key limitation of interest-point-based representation is failing to capture adequate spatial or temporal information.

Sparse representation has received a lot of attention from the signal processing community due in part to the fact that various signals such as audio and natural images can be well approximated by a linear combination of a few elements of some redundant bases, usually called dictionary. Recent publications about sparse representation have shown that this approach is very effective, leading to state-of-the-art results, e.g., in image restoration, image denoising, texture classification and texture synthesis. In the supervised or weakly supervised methods, algorithms adopt features of the sparse coding of signals for classification [5], [6], [7], [8], [9]. But the sparse models mainly consider minimizing the reconstruction error. Little attention is paid to better classification.

Recent research on dictionary learning for sparse coding has been targeted on learning discriminative sparse models instead of the purely reconstructive ones. Mairal et al. [10] generalize the reconstructive sparse dictionary learning process by optimizing the sparse reconstruction jointly with a linear prediction model. Bradley and Bagnell [11] propose a novel differentiable sparse prior rather than the conventional L₁ norm, and employ a back propagation procedure to train the dictionary for sparse coding in order to minimize the training error. These approaches need to explicitly associate each sample with a label in order to perform the supervised training. How to learn a discriminative dictionary for both sparse data representation and classification is still an open problem.

In this paper, we solve the three problems aforementioned and present a novel framework for action recognition. Fig. 1 shows the flowchart of our framework. We make the following three contributions.

First, traditional interest-point-based representation only utilizes features of a single interest point, and fails to capture adequate spatial or temporal information. It is sensitive to the noise. We consider that spatial and temporal information is very important for action representation. So a novel hierarchical descriptor is presented. The proposed compound appearance and motion feature captures spatial information of neighboring interest points. Furthermore, we propose a continuous motion segment descriptor to represent human action by capturing the temporal ordering information in actions. Spatial and temporal context information is utilized for action representation.

Second, we propose a modified sparse model for classification. Different from traditional sparse representation whose only task is to minimize the reconstruction error, the proposed sparse model targets at classification. Given K action classes, we learn K class-specific dictionaries for representing the data, and then classify the test sample into the class whose dictionary generates the minimum reconstruction error. The similarity-constrained term is utilized to project each descriptor into its local coordinate system which captures the correlations between similar descriptors by sharing bases. The dictionary incoherence term ensures that samples from different classes are reconstructed by independent dictionaries. Our proposed sparse model ensures samples are best reconstructed by their own class specific dictionary.

Third, we introduce a classification loss function for the class-specific dictionary learning. The dictionaries are trained by minimizing the classification loss function. The test sample is classified into the class whose dictionary generates the minimum reconstruction error. The learned dictionaries are remarkably more discriminative.

The remainder of this paper is organized as follows. Section 2 gives a review of related approaches for action representation and sparse representation. Section 3 introduces the compound appearance and motion feature and the continuous motion segment descriptor. Section 4 presents a modified sparse model and a supervised class-specific dictionary learning method for classification. Section 5 demonstrates experimental results. Section 6 concludes this paper.