A novel mid-level distinctive feature learning for action recognition via diffusion map

Abstract

Recent works have shown that mid-level feature is superior to low-level feature, which can not only improve discriminative power, but also enhance descriptive capability. In this paper, the classical STIP, spatial star-graph and temporal star-graph are first extracted to represent human action from multi-perspectives. Then a principled feature learning algorithm is proposed to embed these multi-cues into a unified space and enhance all low-level features using diffusion map. Unlike treating spatio-temporal patch as mid-level primitive, we use a graph to model different types of primitives, then apply graph partitioning to co-cluster them into visual-word clusters called mid-level distinctive feature, which can bridge the semantic gap across low-level features. Experimental results show that our approach can successfully classify human activities with higher accuracies both on single-person actions (KTH and UCF) and complex interactional activities (UT-Interaction and HMDB51).

Introduction

How should a video be represented? It is a question as old as the computer vision discipline itself, and it is also a key issue in human action recognition. Nowadays, a variety of visual features have been proposed spanning a wide spectrum, from very local to global volume or from low-level to high-level concepts.

In terms of spatio-temporal resolution, one extreme is treating the whole video volume [1], [2] as a primitive. However, the holistic method is suffering from two limitations in real world application. First, motion tracking and foreground segmentation is required, which is still a challenge in itself. Second, such representing method is sensitive to occlusions, noise, viewpoint and illumination. At the other extreme, the space-time interest point (STIP) [3] can be as a primitive. But there is generally not enough information at a local level to make a representative feature. For instance, some local models such as applying STIP to “bag-of-features” (BOF) [4] describe action as a collection of spatio-temporal interest points and their spatio-temporal relations are completely neglected. As a result, most researchers have started looking at using features at an intermediate scale: that of mid-level feature.

The goal of this paper is to propose a mid-level representation and learning framework for action recognition which is a balance between local feature and global template. The following desired properties are expected for our proposed mid-level feature: 1) motion saliency, it needs to capture the part with strong motion cue in video; 2) spatio-temporal representation, it requires to model contextual structure in both spatial and temporal domain; 3) to be representative and discriminative, “representative” indicates it needs to occur frequently enough in action video, and “discriminative” implies it needs to be different enough from the rest of features.

However, even if we fix the resolution of the primitive, there is still a wide range of choices regarding how to get these mid-level primitives. Two classical ways can be used to achieve these mid-level features by hand-designing [5], [6], [7] or feature learning [8], [10]. On the one hand, some hand-designed mid-level features are generated by integrating some contextual information into BOF to improve STIP based representing approach. The drawback of the hand-designed relation is that it is only representative but not be discriminative. On the other hand, some feature learning algorithms are proposed, which employ discriminative part detectors as semantic features to describe and reason about actions. But such semantic and discriminative entities are just not representative enough to act as good features. Meanwhile, there is a significant practical barrier that lots of hand-labeled training data of per each semantic entity are required in these supervised approaches.

In this paper, we propose a simple, efficient, and effective method to learn a mid-level distinctive feature (MLDF) which can satisfy the above three properties simultaneously. At first, we estimate motion saliency using spatio-temporal interest points, since they often correspond to regions with high-motion. To conform with the second property, spatial star-graph and temporal star-graph are extracted to capture sufficient spatio-temporal information around these local interest points. Finally, we learn a representative and discriminative primitive based on these three low-level features. To be representative, the standard solution is to employ some form of unsupervised clustering, which will latch onto frequently occurring primitives. However, running clustering algorithm with general distance metric cannot achieve good clusters, since it lakes of discriminating ability and does not work well for data from various sources. Hence, the key insight of this paper is to pose this as an unsupervised clustering problem with a discriminative similarity measurement on multiple but homologous features. In this paper, we apply diffusion map [26], [31] framework to fuse these multi-cues and enhance all low-level features by co-clustering them into a unified visual-word clusters called mid-level distinctive feature. There are three reasons that diffusion distance is employed as a similarity measure here. Firstly, diffusion map can capture intrinsic geometric and semantic relationships between the different low-level features on the manifold. Secondly, it is proper to adopt graph-based distance measure when clustering graph-based features. Thirdly, diffusion map can be used as a multi-scale analysis on the data, and all the paths between two points are employed to compute the distance, so that it is more robust to noise than the geodesic distance (shortest path distance). The underlying idea is to embed low-level features into a unified semantic space, so as to construct a compact yet discriminative mid-level vocabulary.

The proposed algorithm can model both single-person actions and multi-person activities, since mid-level feature often yields a representation with adequate discriminative power for more complex datasets. We conduct experiments on human action recognition on four public datasets: KTH, UCF, HMDB51 and UT-interaction. In experimental results, our MLDF outperforms almost all other related state-of-the-art methods by taking into account all the three properties.

The rest of this paper is organized as follows: The Section 2 focuses on some related works. The Section 3 details how to construct Mid-Level Distinctive Feature (MLDF) for video representation. Experiments and discussions are shown in the Section 4. Finally, the Section 5 concludes this paper.