Discriminative human action classification using locality-constrained linear coding

Highlights

• We propose using the locality-constrained linear coding for action classification.

• Our sequence descriptor includes cell, block, and subsequence descriptors.

• We use maximum pooling and a logistic regression classifier to encode each sequence.

• We demonstrate the effectiveness of our algorithm on both depth and RGB videos.

Abstract

We propose a Locality-constrained Linear Coding (LLC) based algorithm that captures discriminative information of human actions in spatio-temporal subsequences of videos. The input video is divided into equally spaced overlapping spatio-temporal subsequences. Each subsequence is further divided into blocks and then cells. The spatio-temporal information in each cell is represented by a Histogram of Oriented 3D Gradients (HOG3D). LLC is then used to encode each block. We show that LLC gives more stable and repetitive codes compared to the standard Sparse Coding. The final representation of a video sequence is obtained using logistic regression with ℓ₂ regularization and classification is performed by a linear SVM. The proposed algorithm is applicable to conventional and depth videos. Experimental comparison with ten state-of-the-art methods on three depth video and two conventional video databases shows that the proposed method consistently achieves the best performance.

Introduction

Human action classification from videos is an important problem because of its many applications in video surveillance, human-computer interaction, sports analysis, and elderly health care [1], [33]. However, automatic classification of human actions in videos is a challenging problem. For colour videos, lighting conditions in the environment and clothing worn by the human subject can both affect the performance of the action classification algorithms. While these problems are eliminated in depth videos, issues about occlusion, loose clothing, and variations in style and execution speed of actions remain.

We propose a human action classification algorithm based on Locality-constrained Linear Coding (LLC). We demonstrate improved action classification performance for actions captured by both depth and colour videos (Fig. 1). [32] compared LLC versus SC and found that LLC holds more essential information than SC for object classification. In this paper and in our earlier work in this direction [18] we investigated the use of LLC for encoding human actions. We found that to favour sparsity SC tends to select quite different elements from the action feature dictionary even for the same action. This has an adverse effect as it increases the intra-action-class variation. The LLC, on the other hand, exerts locality constraints on the features and tends to select the same elements in the dictionary for the same action. Our research contributions are fourfold:

• We propose using the locality-constrained linear coding for human action classification.

• We propose a sequence descriptor for each action video to be constructed in a hierarchical fashion, including the computation of the cell descriptor, block descriptor, and subsequence descriptor.

• We propose using maximum pooling and a logistic regression classification with ℓ₂ regularization for action classification.

• We demonstrate the effectiveness of the proposed algorithm over the existing techniques for improved action classification accuracy.

To show that LLC is effective for human action classification, we design our sequence descriptor based on LLC and compare its accuracy on human action classification with the descriptor computed using SC. Furthermore, we evaluate these sequence descriptors against ten state-of-the-art techniques. For the benchmark depth video datasets (MSRGesture3D, MSRAction3D, and MSRActionPairs3D), we compare our algorithm against the algorithms of

• [11], where the local descriptor is based on the histogram of oriented 3D spatio-temporal gradients (or HOG3D);

• [30], where the random occupancy patterns (ROP) are used;

• [31], where an actionlet ensemble model is learned;

• [17], where the histogram of oriented 4D normals (HON4D) features are used;

• [35], where space time interest points from depth sequences (DSTIP) are used;

• [20], where local features encoding variations of depths and depth gradients plus skeletal body joints are used with a random decision forest (RDF) classifier.

For the benchmark colour video datasets (Weizmann and UCFSports), we compare our algorithm with the algorithms of [11] and the following:

• [38], where the mapping between densely-sampled feature patches and the votes in a spatio-temporal action Hough space is trained using random trees;

• [42], where the HOG3D feature was computed within a small 3D cuboid centred at a space-time point and encoded in a sparse coding framework;

• [13], where a figure-centric word representation is used;

• [21], where spatio-temporal structures from clustering of point trajectories of body parts are used;

• [28], where a deformable part model is generated for each action from a collection of examples, with actions being treated as spatio-temporal patterns in the colour videos.

In all of these experiments, our proposed algorithm has consistently shown improved performance compared to the existing algorithms.