Mean Shift tracking with multiple reference color histograms

Abstract

The Mean Shift tracker is a widely used tool for robustly and quickly tracking the location of an object in an image sequence using the object’s color histogram. The reference histogram is typically set to that in the target region in the frame where the tracking is initiated. Often, however, no single view suffices to produce a reference histogram appropriate for tracking the target. In contexts where multiple views of the target are available prior to the tracking, this paper enhances the Mean Shift tracker to use multiple reference histograms obtained from these different target views. This is done while preserving both the convergence and the speed properties of the original tracker. We first suggest a simple method to use multiple reference histograms for producing a single histogram that is more appropriate for tracking the target. Then, to enhance the tracking further, we propose an extension to the Mean Shift tracker where the convex hull of these histograms is used as the target model. Many experimental results demonstrate the successful tracking of targets whose visible colors change drastically and rapidly during the sequence, where the basic Mean Shift tracker obviously fails.

Introduction

The target’s color histogram is widely used for visual tracking (e.g. [1], [2], [3], [4], [5]) and, as was shown by Comaniciu et al. [3], [6], tracking using this feature may be performed very quickly via the Mean Shift procedure [7]. This paper extends Comaniciu et al.’s tracker in [3], [6], which will be referred to in this paper by its common name Mean Shift tracker.

The Mean Shift tracker works by searching in each frame for the location of an image region whose color histogram is closest to the reference color histogram of the target. The distance between two histograms is measured using their Bhattacharyya coefficient, and the search is performed by seeking the target location via Mean Shift iterations beginning from the target location estimated in the previous frame (the tracker is outlined in Section 2).

In the Mean Shift tracker, as well as in the other trackers cited previously, the reference color histogram is approximated according to a single view of the target, typically as it appears in the first frame of the sequence. Although using this method for obtaining the reference histogram proved to be very robust in many scenarios, it produces, in many cases, a poor representation of the target, which might result in poor tracking. More seriously, the support of a reference histogram obtained by this method may become non-overlapping with the support of the target’s histogram as it appears in the sequence, usually resulting in target loss. Indeed, for many objects, any viewing direction may be replaced with a different viewing direction where all the object’s colors apparent in the latter view differ from those in the former. An (unscrambled) Rubik Cube is an extreme example of such an object; each side is a different color, and three sides at most are visible from any viewing direction. Major changes in the apparent colors of a target may also result from changes in the actual target’s colors, as when a person puts on or removes a piece of clothing, or as in the case of an alternating street advertisement.

Often, several different views of the target are available prior to the tracking, either from images that were previously acquired (e.g. [8], [9], [10], [11]) or when performing off-line tracking (e.g. [12], [13], [14], [15]). In these contexts, this paper extends the Mean Shift tracker to using multiple reference color histograms. At first we suggest a simple method to combine these histograms into a single histogram that is more appropriate for tracking the target. In order to enhance the tracking further, we then propose an extension to the Mean Shift tracker, where the convex hull of these histograms is used as the target model. That is, rather than searching for the image region whose color histogram is closest to a single reference histogram, we search for the image region by minimizing the distance of its color histogram from the convex hull of several reference histograms.

Time-varying histograms of colors (e.g. [5], [16]) or of other features such as filter responses (e.g. [17]) have been used for target modeling before, and many trackers have modeled the target’s 2D appearance as being time-varying within a subspace (e.g. [8], [9], [18], [19], [20]). In the latter group of trackers, the search for the target (and possibly for additional transformation parameters) is performed by minimizing the distance of its appearance in the current frame from that subspace. This approach is applied here by modeling the target’s color histogram as being a time-varying linear combination of several reference histograms, under the restriction that the mixture coefficients are nonnegative and sum to unity (so that the linear combination will be a histogram mixture).

Section 2 outlines the original Mean Shift tracker [3]. A simple method for combining multiple reference histograms into one is proposed in Section 3. Section 4 describes the proposed extension of the Mean Shift tracker to use the convex hull-based target model. Experimental results are described in Section 5, Section 6 includes a discussion, and a paper summary is provided in Section 7.