The spatial Laplacian and temporal energy pyramid representation for human action recognition using depth sequences

Abstract

Depth sequences are useful for action recognition since they are insensitive to illumination variation and provide geometric information. Many current action recognition methods are limited by being computationally expensive and requiring large-scale training data. Here we propose an effective method for human action recognition using depth sequences captured by depth cameras. A multi-resolution operation, the spatial Laplacian and temporal energy pyramid (SLTEP), decomposes the depth sequences into certain frequency bands in different space and time positions. A spatial aggregating and fusion scheme is applied to cluster the low-level features and concatenate two different feature types extracted from low and high frequency levels, respectively. We evaluate our approach on five public benchmark datasets (MSRAction3D, MSRGesture3D, MSRActionPairs, MSRDailyActivity3D, and NTU RGB+D) and demonstrate its advantages over existing methods and is likely to be highly useful for online applications.

Introduction

Human action recognition has been of longstanding interest to the computer vision community due to its widespread potential in real-word applications, for example human–computer interactions [1] and intelligent video surveillance [2]. Conventional human action recognition studies mainly focus on recognizing actions from videos captured by color cameras, and these approaches still face problems due to difficulties in handling object segmentation, illumination and texture variances. The emergence of inexpensive depth sensors such as Kinect [3], which capture depth information in real-time, has captured the imagination of many researchers, including those studying action recognition. Compared to conventional color-based cameras, subtracting the foreground from cluttered backgrounds is much easier in depth maps and, furthermore, depth sensors are insensitive to illumination variance, facilitating applications in dark environments. Depth sensors also provide a lot of geometric information, which facilitates spatial information extraction and target recognition.

An action or gesture recognition system should work independently of the actor's identity, the speed of the performance, and the inherent variance present in the realizations of action instances [4] whilst ensuring fast classification performance, especially for online action recognition. Various detection and representation methods have been proposed to acquire accurate similarity measurements and improve action recognition performance using depth sequences, e.g., cloud points [5], skeleton joints [6], [7], and hyper-surface normals [8]. Skeleton joints can be accessed in real-time by depth sensors and are computationally efficient for human action recognition but less so for other applications such as gesture recognition and human–object interactions. Cloud points and hyper-surface normals use raw depth maps as inputs and are more robust to noise and occlusion [9] but are resource intensive. In this paper, we focus on improving action recognition performance using depth maps at the lowest computational cost.

Our approach extracts features at different frequencies, which are then decomposed by the spatial Laplacian and temporal energy pyramid (SLTEP). For the low frequency component, 4D hyper-surface normals are extracted to capture spatial orientation cues. To retain the correlation between neighboring normals and make them more robust to noise, the local spatial neighborhood is introduced to cluster these low-level normal features and generate polynormal vectors. The coefficient-weighted differences between polynormals and visual words learnt by sparse coding are then computed to obtain the extra distribution information of low-level features, with the coefficient differences aggregated by average pooling over each spatial grid. For the high frequency component, we exploit maximum pooling over silhouettes to achieve motion invariance to temporal locations according to the temporal energy pyramid. Then, histogram of oriented gradient (HOG) features (introduced in [10]) are extracted from the spatial grids to record body shape information. Finally, the two different feature types are fused as the representation of the depth sequences.

The main contributions of this paper can be summarized as follows: first, by employing spatial Laplacian and temporal energy pyramids, we decompose depth sequences to extract complementary features at different frequencies; second, local sparse representation is employed to obtain extra spatial distribution information from low-level feature in the local neighborhoods; and third, we propose a spatial aggregation and fusion scheme to cluster the low-level features and concatenate the extracted complementary features as the final representation. The experimental results demonstrate the advantages of the proposed method with respect to recognition accuracy and computational efficiency.

The remainder of the paper is organized as follows. In Section 2, we review previous work related to action recognition. Section 3 describes our feature extraction and representation framework. In Section 4, we present experiments performed on five public benchmark datasets, the results of which demonstrate the effectiveness of our proposed approach compared to other published results. Finally, we conclude and discuss possibilities for future work in Section 5.