Dense convolutional feature histograms for robust visual object tracking

Highlights

• Novel visual object tracking algorithm based on extracting histograms of convolutional features

• Proposed architecture is better suited to the tracking task, facilitating the training process

• Extensive experimental validation in multiple tracking benchmarks

• Entirely offline training, realtime speeds even on embedded systems

Abstract

Despite recent breakthroughs in the field, Visual Object Tracking remains an open and challenging task in Computer Vision. Modern applications require trackers to not only be accurate but also very fast, even on embedded systems. In this work, we use features from Convolutional Neural Networks to build histograms, which are more adept at handling appearance variations, in an end-to-end trainable architecture. To deal with the internal covariate shift that occurs when extracting histograms from convolutional features as well as to incorporate informations from the multiple levels of the neural hierarchy, we propose and use a novel densely connected architecture where histograms from multiple layers are concatenated to produce the final representation. Experimental results validate our hypotheses on the benefits of using histograms as opposed to standard convolutional features, as the proposed histogram-based tracker surpasses recently proposed sophisticated trackers on multiple benchmarks. Long-term tracking results also reaffirm the usefulness of the proposed tracker in more challenging scenarios, where appearance variations are more severe and traditional trackers fail.

Introduction

Visual Object Tracking is the task of locating an arbitrary target, defined by a Region of Interest (ROI), in a video sequence. Various applications such as security and surveillance, cinematography, robotics, augmented reality, and even entertainment [1], [2], all rely on tracking as a first step. Despite recent breakthroughs in the field, owing mostly to the success of Convolutional Neural Networks (CNNs), it remains a very challenging task in the field of Computer Vision.

Typically, a tracker will generate a model of the target ROI and use this model to search for the target in subsequent frames. Challenges arise first from the discriminative ability of the target model, which should be distinguishable enough such that the target will avoid drifting to similar objects, while remaining adaptive enough to the inter-frame appearance changes of the target. Furthermore, a tracker must be equipped to deal with occlusions, fast target movements and out-of-view scenarios. Another important aspect is the speed of the tracker, i.e., the time it takes for the tracker to locate the target in each frame. Real-time requirements impose hard constraints on the per frame processing time of the tracker.

Recently, CNNs have been excelled in other Computer Vision tasks, such as image classification, object detection and semantic segmentation [3]. Their success can be attributed to the semantically meaningful representations they extract from visual information. Hence, CNNs begun to be used in tracking tasks. In general, most recently released CNN-based trackers work by extracting a convolutional feature-based of the target as well as from the subsequent frames and cross-correlating the target model with the frame model to locate the target. They rely either on online optimization of the network parameters or offline learning to produce discriminative features while maintaining high tracking speed on GPUs.

To address the challenges imposed by modern applications, such as those on embedded machines and robots [2], a tracking algorithm must push the speed-accuracy trade-off to its limit. For CNN-based trackers, this means using modern neural architectures while keeping the number of layers relatively short. Learning a discriminative model is a matter of training the network well offline as well as updating the network and the model online. The abundance of annotated video datasets such as ImageNet VID [4], or TrackingNet [5] facilitates the training process of such trackers.

Before convolutional features, histogram-based descriptors were used, such as Histogram of Oriented Gradients (HOG), descriptors in the Discriminative Correlation Filter (DCF) paradigm [6], or color histograms in the Mean Shift (MS) algorithm [7]. Depending on the encoded feature type, histograms can more robust to appearance changes while being computationally efficient. Based on the success of histogram-based trackers, as well as the powerful representations extracted by deep CNNs, in this work, combine the two in a single neural architecture, where histograms are extracted from deep convolutional features, to exploit the best attributes of each.

The main contribution of this work is a novel visual object tracking algorithm, based on extracting histograms of convolutional features in a single, end-to-end learnable architecture. Using histograms, the proposed tracker becomes robust to the appearance variations of the target, while remaining discriminative due to the use of convolutional features, inspired by recent works coupling encoding methods with CNNs [23], [24]. We furthermore introduce an architecture which is better suited to the tracking task and facilitates the training process of neural networks using histograms. The effectiveness of the histogram-based representations is validated experimentally in various tracking scenarios. Our tracker is trained entirely offline, thus remaining very fast during deployment and being capable of running at real-time even on embedded systems.

The rest of this paper is organized as follows. Section 2 is a summary of related work on visual object tracking. Section 3 introduces and analyzes in depth the proposed tracker and methodology. The experimental results, which validate our hypotheses, are presented in Section 4, and finally Section 5 summarizes our findings.