Simultaneous multi-person tracking and activity recognition based on cohesive cluster search

Highlights

• Simultaneous multi-person tracking and activity recognition using a bootstrapping framework.

• High-order correlation formulations among social dynamics and activities using a hypergraph representation.

• Cohesive cluster search applied to solve the hypergraph optimization.

Abstract

We present a bootstrapping framework to simultaneously improve multi-person tracking and activity recognition at individual, interaction and social group activity levels. The inference consists of identifying trajectories of all pedestrian actors, individual activities, pairwise interactions, and collective activities, given the observed pedestrian detections. Our method uses a graphical model to represent and solve the joint tracking and recognition problems via three stages: (i) activity-aware tracking, (ii) joint interaction recognition and occlusion recovery, and (iii) collective activity recognition.

This full-stack problem induces great complexity in learning the representations for the sub-problems at each stage, and the complexity increases as with more stages in the system. Our solution is to make use of symbolic cues for inference at higher stages, inspired by the observations of cohesive clusters at different stages. This also avoids learning more ambiguous representations in the higher stages.

High-order correlations among the visible and occluded individuals, pairwise interactions, groups, and activities are then solved using the cohesive cluster search within a Bayesian framework. Experiments on several benchmarks show the advantages of our approach over the existing methods.

Introduction

Multi-person activity recognition is a major component of many applications, e.g, video surveillance and traffic control. The problem entails the inference of the actor activities, their motion trajectories, as well as the interactions and time dynamics of the groups for the case of multiple actors. This task is challenging, since the activities must be analyzed from both the spontaneous individual actions and the complex social dynamics involving groups and crowds (Vinciarelli et al., 2009). We aim to address the where and when problems by visual trajectory analysis, as well as the who and what problems by activity recognition.

While advanced methods for person detection are becoming more reliable (Cai et al., 2016, Yu et al., 2016), most existing activity recognition approaches rely on visual tracking following a tracking-by-detection paradigm. These methods either fail to consider social interactions while inferring activities (Ibrahim et al., 2016, Khamis et al., 2012b, Khamis et al., 2012a) or have difficulties recognizing the structural correlations of actions and interactions (Choi and Savarese, 2012, Choi and Savarese, 2014, Deng et al., 2016). In particular, there are two major challenges: (i) ineffective tracking due to frequent occlusions in groups and crowds, and (ii) the lack of a suitable methodology to infer the complex but salient structures involving social dynamics and groups.

In this paper, we address both challenges using a bootstrapping framework to simultaneously improve the two tasks of multi-person tracking and social group activity recognition. We take person detection bounding boxes (Cai et al., 2016, Yu et al., 2016) as input to perform initial multi-person tracking. We then recognize stable group structures including the temporally cohesive individual activities (such as walking) and pairwise interactions (such as walking side-by-side, see Fig. 1 to robustly infer collective social activities (such as street crossing in a group) in multiple stages. Auxiliary inputs such as body orientation detections can be considered within the stages if available. The recognized activities and salient grouping structures are used as priors to recover occluded detections and false associations to improve performance.

We explicitly explore the correlations of pairwise interactions (of two individuals) and group activities (within the group of more individuals) during the optimization. Observe in Fig. 1 that group activities generally are identified by cohesive clusters of pairwise interactions, which we have exploited in the multi-stage inference steps. In our method, multi-person tracking and individual/group activity recognition are jointly optimized, such that consistent activity labels characterizing the dynamics of the individuals and groups can be obtained. The individual and group activities are formulated using a dynamic graphical model, and high-order correlations are represented using hypergraphs. The simultaneous pedestrian tracking and multi-person activity recognition problems are then to be solved jointly using an efficient cohesive cluster search on the hypergraphs.

Main contribution of this work is two-fold. First, we propose a new framework that can jointly solve the two tasks of real-time simultaneous tracking and activity recognition. Explicit modeling of the correlations among the individual activities, pairwise interactions, and collective activities leads to a consistent solution. Second, we propose a hypergraph formulation to infer the high-order correlations among social dynamics, occlusions, groups, and activities in multi-stages. Simultaneous tracking and activity recognition are formulated as a bootstrapping framework, which can be solved efficiently using the search of cohesive clusters in the hypergraphs. This cohesive cluster search solution is general that it can be extended to include additional scenarios or constraints in new applications.

The main novelty of our work is the adaptation of cohesive cluster for trajectory tracking and activity recognition. Specifically, the optimization procedure for the cohesive cluster search preserves advantages from the previous works; new research efforts are mainly reflected in the investigation of how to construct hypergraphs for the two problems in an effective manner, such that the tracking and activity recognition can benefit each other.

Experiments on several benchmarks show the advantages of our method with improvements in both activity recognition and multi-person tracking. Our method is easily deployable to real-world applications, since: (i) our method does not depend on site knowledge, i.e., camera calibration is not required; (ii) online video streams can be processed by considering a time window in a round; (iii) the computation can be performed in real-time (about 20 FPS, excluding the input detection steps).