Robust tracking using visual cue integration for mobile mixed images

Highlights

• Maximizing observation likelihood to optimize particle weights under reflections.

• Combining co-inference and maximum likelihood for visual cue integration.

• Co-inference is combined with SIR to avoid the particle degeneracy problem.

• Maximum likelihood selects the more reliable cue for co-inference fused state.

• Motion compensation for both layer separation and prediction of the particle filter.

Abstract

The transmitted scene superposed with the reflected scene from a transparent surface leads to mixed images. Few methods have been devoted for tracking on mixed images while such images are ubiquitous in the real world. Thus, this paper proposes a robust single object tracking scheme for mixed images acquired by mobile cameras. Layer separation that decomposes mixed images extracts intrinsic dynamic layers before tracking. In order to make the tracker robust against camera motion, motion compensation is applied to both layer separation and prediction stage of the particle filter. To maximize the observation likelihood and thus optimize particle weights in the face of reflections, the proposed scheme combines sequential importance resampling (SIR) based co-inference and maximum likelihood for multi-cue integration. Experimental results show that the proposed scheme effectively improves tracking accuracy on mixed images with camera motion.

Introduction

Visual tracking is essential to many applications of computer vision. Previous trackers tackle problems such as occlusion, illumination variations, pose changes, cluttered background, complex trajectory, and multi-target. But few of them focus on the reflection interference problem. In fact, mixed images contain both reflections from the transparent surface and the transmitted scene behind the surface. Due to being superposed with the reflected scene, the appearance of the target and background in the transmitted scene change significantly in the mixed images and thus inaccurate tracking is easily raised. In complex and dynamic environments, multi-cue integration can improve the robustness of particle filter based trackers. Serby et al. combine multiple low-level features, including interesting points, edges, homogeneous and textured regions, into a particle filter framework for tracking [1]. And the multi-feature observation likelihood is the product of individual likelihoods of different features. To maximize the discriminability of multiple cues, Yang et al. use object detectors to adapt the target observation model [2]. Li et al. propose the weighted Dempter-Shafer fusion to combine evidences from different spatio-temporal SVMs (supper vector machine) into the observation likelihood of a particle filter [3]. The phenomenon of co-inference is proposed by Wu and Huang in 2001 and is refined in [4]. By using the structured variational inference to decouple the dynamics of multiple hidden states, Wu et al. propose co-inference tracking of multiple modalities. The variational parameters of one modality are inferred by the other modalities to maximize the observation likelihood [4]. Sparse representation can be also integrated into the particle filter for data fusion [5], [6], [7], [8]. Such trackers are robust against occlusion, illumination variations, and cluttered background. Wu et al. solves a $l_1$-regularized least squares problem to estimate sparse coefficients of the target candidate [5].

Reflection separation is to estimate the transmitted scene and reflection image from the transparent surface, e.g., glass [9]. Blind source separation aims at estimating the unknown source signals and mixing matrix from a set of mixed signals [10]. For computer vision, reflection separation can take use of blind source separation to estimate source layers. Independent component analysis (ICA) based separation works under assumptions of independent source layers and static mixing, e.g. [11]. Given two mixed images, two source images can be separated by minimizing their structural correlations, e.g. [12]. Under the sparse prior over derivative filters on natural images [13], manually marked edges can help separation from a single image [14]. Automatic separation of weak reflection from a single image can be achieved by using the smoothness constraint and reducing the structural correlation between layers [15]. In [16], several structural priors in the transmitted and reflected layers are combined. Then the geometrical alignment of the reflection region in multiple mixed images is optimized using the augmented Lagrangian multiplier. Gai et al. consider the diversities of layer motions and model the transformation of reflected layers in a parametric way [17]. Instead, Li et al. assume the parametric transformation for transmitted layers so that variations of reflection layers can be handled [18]. The method in [19] tracks reflection regions in video frames. Another kind of layer separation is to derive intrinsic images including the illumination and reflectance images where the input image is the product of separated layers [13], [20]. The intrinsic image, the mid-level description of scenes, is defined by Barrow and Tenenbaum [20]. Sometimes, computer vision algorithms working on such descriptions achieve better performance.

Nowadays, few methods have been proposed for tracking on mixed images. Before using the Kanade–Lucas–Tomasi feature tracker for tracking an object in regions of reflections, the method in [21] applies layer separation [13] to temporally aligned frames to extract the background and foreground layers. Tracks of separated layers are longer than those of the mixed images. Since the focus of tracking is the target but neither the transmitted scene nor the reflected scene, layer separation [13] also extracts the dynamic layer before single object tracking in [22]. Based on the framework of particle filter with compensated motion model [23], the correction stage reweights particles using RGB and [I, R-G, Y-B] color histograms of the mixed images [20]. The [I, R-G, Y-B] color histogram is generated with the aid of a mask, indicating the dynamic regions on the mixed image. Then each particle weight is optimized using maximum likelihood. One problem with Chen et al. [22] is that tracking accuracy will decrease if videos have camera motion. This is mainly because the inaccurate static layer (i.e., reflectance image) makes edges of background and reflections contaminate the dynamic layer (i.e. illumination image). The measurement that refers to the inaccurate mask decreases estimation accuracy of the correction stage. Since few of previous trackers tackle the problem of reflection interference and multi-target tracking should discuss the areas of interactive multiple motion (IMM) model, data association, and state estimation in depth [24], this paper focuses on how to achieve robust single object tracking using multiple cue integration for mobile mixed images. This paper improves the work in [22] and its major contributions are stated as follows. (1) The proposed scheme improves tracking accuracy under the condition of reflections by combining co-inference [4] and maximum likelihood for visual cue integration. (2) The proposed particle filter based scheme realizes co-inference using sequential importance resampling (SIR) [25] instead of sequential important sampling (SIS) to avoid the degeneracy problem. (3) Layer separation with motion compensation for mobile images is proposed to extract objects with active motion. As a result, the proposed scheme significantly improves tracking accuracy on mobile mixed images. The remainder of this paper is organized as follows. Section 2 reviews the compensated motion model for tracking with mobile cameras [23]. Section 3 proposes motion compensated layer separation for images with camera motion. Section 4 proposes a robust single object tracking scheme that combines co-inference [4] and maximum likelihood for mixed images. Section 5 analyzes experimental results and Section 6 concludes this paper.