Actionness-pooled Deep-convolutional Descriptor for fine-grained action recognition

Abstract

Recognition of general actions has witnessed great success in recent years. However, the existing general action representations cannot work well to recognize fine-grained actions, which usually share high similarities in both appearance and motion pattern. To solve this problem, we introduce the visual attention mechanism into the proposed descriptor, termed Actionness-pooled Deep-convolutional Descriptor (ADD). Instead of pooling features uniformly from the entire video, we aggregate features in sub-regions that are more likely to contain actions according to actionness maps. This endows ADD with the superior capability of capturing the subtle differences between fine-grained actions. We conduct experiments on HIT Dances dataset, one of the few existing datasets for fine-grained action analysis. Quantitative results have demonstrated that ADD remarkably outperforms traditional CNN-based representations. Extensive experiments on two general action benchmarks, JHMDB and UCF101, have additionally proved that combining ADD with end-to-end ConvNet can further boost the recognition performance. Besides, taking advantage of ADD, we reveal the sparsity characteristic existing in actions and point out a potential direction to design more effective action analysis models by extracting both representative and discriminative action patterns.

Introduction

In the past decades, great efforts have been made for the general action recognition problem. Many successful representations have been proposed, including hand-crafted features such as Space Time Interest Points (STIPs) [1] and Improved Trajectories [2]. Besides, it has become popular to extract action representations by leveraging deep learning techniques. Tran et al. [3] propose a generic spatio-temporal descriptor by performing 3D convolutions and pooling to preserve both spatial and temporal information of the input signals. Simonyan and Zisserman design a two-stream architecture [4] consisting of RGB and optical flow streams to capture the appearance and motion information respectively. These ConvNets-based methods achieve the state-of-the-art performance.

However, the actions in traditional datasets, such as KTH [5] and UCF101 [6], are mostly well-defined with significant appearance and motion differences, which are easy to be distinguished as compared to the realistic ones that are often vague and uncertain. Specifically, the intra-class difference of fine-grained actions is likely to be very large due to the variations of actors, backgrounds, etc. In contrast, the similarity between different actions substantially reduces the inter-class difference. The recognition of fine-grained actions raises more challenges. First, an effective representation should be able to distinguish between actions sharing similar appearance and motion styles but having subtle differences. Take, for example, the Mongolian dance and Uygur dance are shown in Fig. 1(a), for example, the background scenes and costumes are very similar. What is more, the actors are performing very similar “Spinning” actions. Only the poses and actions of arms are slightly different. These tiny differences are very hard to be captured by existing ConvNets which are designed for general action recognition since they adopt globally sampling and pooling on entire frames (see Fig. 1(b)). Second, the lack of training data is another barrier. As we know, only with enough and diverse training samples can deep learning models work better, but there are no large-scale datasets available for fine-grained action recognition. Even the most widely used datasets for general action recognition are not large enough.

In order to circumvent the above-mentioned problems encountered with current approaches, we propose a novel feature sampling and pooling method and propose a novel representation, namely Actionness-pooled Deep-convolutional Descriptor (ADD), inspired by human visual attention mechanism that has been widely used in fine-grained image classification [7]. In this work, we treat actionness maps as the guidance of visual saliency and extract features from more discriminative patches (see Fig. 1(c) and (d)). By this means, for one thing, we endow the proposed feature descriptor with stronger sensitivity to subtle differences in local appearance and motion patterns between fine-grained actions. For another, our proposed actionness-constrained sampling and pooling serves as a kind of feasible data augmentation strategy since it builds complementary representation to traditional end-to-end ConvNet representations.

We evaluate ADD on HIT Dances dataset [8], which is originally constructed for dance video recommendation. The results demonstrate that ADD significantly outperforms 3D convolutional network (C3D) representation [3] and the traditional two-stream model [9] by 6.9% and 5.8%, respectively, on the task of fine-grained dance recognition. Besides, from extensive experiments on two traditional benchmarks, JHMDB [10] and UCF101 dataset [6], we experimentally show that the combination of different sampling and pooling strategies can further improve the performance, which indicates the complementary property between ADD and general end-to-end ConvNets. Furthermore, we take advantage of ADD to derive the segment-level representations of action videos and analyze the contributions of tiny action clips to fine-grained action classification.

In summary, the contributions of this paper are threefold:

• We propose an effective model for fine-grained action recognition which integrates a novel discriminative descriptor named ADD. We embed the attention mechanism by aggregating features following the actionness cues.

• Experiments demonstrate the superior performance of our method on fine-grained action recognition and the complementary properties between ADD and traditional end-to-end ConvNets-based representations.

• We explore the sparsity characteristic of action data by reasoning the temporal importance of long-range actions to action classification and point out a potential direction to promote future action analysis tasks.

One preliminary version of this work has been accepted by IEEE Conference on Multimedia and Expo (ICME 2018). In this paper, we have improvements mainly in the following two aspects: (1) Several additional experiments have been conducted to support our argument that ADD captures complementary information to traditional end-to-end ConvNets representation and the combinations of them contribute to further improve the recognition performance; (2) We carry out an extra exploration experiment taking advantage of our ADD representation. Specifically, we investigate the temporal importance of long-range actions when recognizing an action. By this means, we reveal that only a small subset of the video content contributes to the recognition. And we find that these action patterns are not only representative but also discriminative. This discovery will facilitate other computer vision tasks, such as action compression, summarization, and compact action representation.

The remainder of the paper is organized as follows. In Section 2, we review the related work. And then, we introduce the proposed Actionness-pooled Deep-convolutional Descriptor (ADD) in Section 3. Sections 4 and 5 present the experimental settings and experimental results, respectively. Finally, Section 6 concludes the paper.