A labeled random finite set online multi-object tracker for video data

Highlights

• The proposed filter addresses occlusions and detection loss that exploits the advantages of both detection-based and TBD approaches to improve performance while reducing the computational cost.

• In a single Bayesian recursion the filter seamlessly integrates state estimation, track management, clutter rejection, detection loss and occlusion handling as well as prior knowledge that detection loss in the middle of the scene are likely to be due to occlusions.

• Tracking performance is compared to state-of-the-art algorithms on simulated data and well-known benchmark video datasets.

Abstract

This paper proposes an online multi-object tracking algorithm for image observations using a top-down Bayesian formulation that seamlessly integrates state estimation, track management, handling of false positives, false negatives and occlusion into a single recursion. This is achieved by modeling the multi-object state as labeled random finite set and using the Bayes recursion to propagate the multi-object filtering density forward in time. The proposed filter updates tracks with detections but switches to image data when detection loss occurs, thereby exploiting the efficiency of detection data and the accuracy of image data. Furthermore the labeled random finite set framework enables the incorporation of prior knowledge that detection loss in the middle of the scene are likely to be due to occlusions. Such prior knowledge can be exploited to improve occlusion handling, especially long occlusions that can lead to premature track termination in on-line multi-object tracking. Tracking performance is compared to state-of-the-art algorithms on synthetic data and well-known benchmark video datasets.

Introduction

In a multiple object setting, not only do the states of the objects vary with time, but the number of objects also changes due to objects appearing and disappearing. In this work, we consider the problem of jointly estimating the time-varying number of objects and their trajectories from a stream of noisy images. In particular, we are interested in multi-object tracking (MOT) solutions that compute estimates at a given time using only data up to that time. These so-called online solutions are better suited for time-critical applications.

A critical function of a multi-object tracker is track management, which concerns track initiation/termination and track labeling or identifying trajectories of individual objects. Track management is more challenging for online algorithms than for batch algorithms. Usually, track initiation/termination in online MOT algorithms is performed by examining consecutive detections in time [1], [2]. However, false positives generated by the background, compounded by false negatives (including those from object occlusions), can result in false tracks and lost tracks, especially in online algorithms. False negatives also cause track fragmentation in batch algorithms as reported in [3], [4], [5], [6]. With the exception of the recent network flow [7] techniques, track labels are assigned upon track initiation, and maintained over time until termination. An online multi-object Bayesian filter that provides systematic track labeling using labeled random finite set (RFS) was proposed in [8].

In most video MOT approaches, each image in the data sequence is compressed into a set of detections before a filtering operation is applied to keep track of the objects (including undetected ones). Typically, in the filtering module, motion correspondence or data association is first determined followed by the application of standard filtering techniques such as Kalman or sequential Monte Carlo [1], [2]. The main advantage of performing detection before filtering is the computational efficiency in the compression of images into relevant detections. The main disadvantage is the loss of information, in addition to false negatives and false positives, especially in low signal to noise ratio (SNR) applications.

Track-before-detect (TBD) is an alternative approach, which by-passes the detection module and exploits the spatio-temporal information directly from the image sequence. The TBD methodology is often required in tracking applications for low SNR image data [9], [10], [11], [12]. In visual tracking applications, perhaps the most well-known TBD MOT algorithm is BraMBLe [13]. Other visual MOT algorithms that can be categorized as TBD include [14], [15] which exploit color-based observation models, [2], [16], which exploit multi-modality of distributions, and [17] which uses multi-Bernoulli random finite set models. While the TBD approach minimizes information loss, it is computationally more expensive. So far it is not clear how we could simultaneously process detection and image measurements to exploit their complementary advantages, in a principled manner.

In this paper, we develop an efficient online MOT algorithm for video data that exploits the advantages of both detection-based and TBD approaches to improve performance while reducing the computational cost. In the visual MOT literature, simultaneous consideration of detections and image features were proposed in ad-hoc manners [1], [5], and it is not clear how to combine them in a principled way. The innovation of our proposed algorithm is the adaptive update of tracks with detections (for efficiency), or with local regions of the input image (to minimize information loss and improve accuracy). In addition, the proposed visual MOT filter seamlessly integrates state estimation, track management, clutter rejection, false negatives and occlusion handling, (which are traditionally separate functionalities) in a single Bayesian recursion.

The key technical contribution is a hybrid multi-object measurement model that simultaneously accommodates detections and image observations. Conceptually, this model is a simple generalization of the standard multi-object measurement model [18] and the separable model for image measurement [10]. Such a simple construct, however, enables us to simultaneously exploit the efficiency of the detection-based approach and the accuracy of TBD-based approach. Specifically, using the labeled RFS framework for multi-object estimation [8], we prove conjugacy of the Generalized Labelled Multi-Bernoulli (GLMB) distributions with respect to the likelihood function of the proposed measurement model. Using this conjugacy result, and the labeled RFS estimation formulation [8], we develop an analytic Bayesian MOT filter that avoids processing the entire image so as to reduce computational costs, while at the same time make use of relevant local information at the image level to reduce the effect of false negatives as well as tracking errors.

Due to the labeled RFS filtering formulation, the proposed MOT filter addresses state estimation, track management, clutter rejection, false negatives and occlusion handling, in one single recursion. Generally, an online MOT algorithm would terminate a track that has not been detected over several frames. In many visual MOT applications however, it is observed that away from designated exit regions such as scene edges, the longer an object is in the scene, the less likely it is to disappear, see for example [19], [20] which exploit theses so-called closed world assumptions. Intuitively, this observation can be used to delay the termination of tracks that have been occluded over an extended period, so as to improve occlusion handling. The labeled RFS framework provides a principled and inexpensive means to exploit this observation for improved occlusion handling.

The remainder of the paper is structured as follows. The Bayesian filtering formulation of the MOT problem using labeled RFS is given in Section 2, followed by details of the proposed solution in Section 3. Performance evaluation of the proposed MOT filter against state-of-the-art trackers is presented in Section 4, and concluding remarks are given in Section 5.