Adaptive low-rank subspace learning with online optimization for robust visual tracking

Abstract

In recent years, sparse and low-rank models have been widely used to formulate appearance subspace for visual tracking. However, most existing methods only consider the sparsity or low-rankness of the coefficients, which is not sufficient enough for appearance subspace learning on complex video sequences. Moreover, as both the low-rank and the column sparse measures are tightly related to all the samples in the sequences, it is challenging to incrementally solve optimization problems with both nuclear norm and column sparse norm on sequentially obtained video data. To address above limitations, this paper develops a novel low-rank subspace learning with adaptive penalization (LSAP) framework for subspace based robust visual tracking. Different from previous work, which often simply decomposes observations as low-rank features and sparse errors, LSAP simultaneously learns the subspace basis, low-rank coefficients and column sparse errors to formulate appearance subspace. Within LSAP framework, we introduce a Hadamard production based regularization to incorporate rich generative/discriminative structure constraints to adaptively penalize the coefficients for subspace learning. It is shown that such adaptive penalization can significantly improve the robustness of LSAP on severely corrupted dataset. To utilize LSAP for online visual tracking, we also develop an efficient incremental optimization scheme for nuclear norm and column sparse norm minimizations. Experiments on 50 challenging video sequences demonstrate that our tracker outperforms other state-of-the-art methods.

Introduction

Visual tracking refers to establish the correspondences of the interesting object in dynamic scene between the successive frames (Chen et al., 2016, Song, 2014, Song et al., 2017, Yang et al., 2016, Zhang, Zhang et al., 2014). As one of the core problems of computer vision, visual tracking has numerous advanced applications such as gesture recognition, video surveillance, medical image analysis and human computer interaction. As heavy occlusion, illumination change, and complex object motion may happen in complex and dynamic scenes, visual tracking is a challenging task. In the past decades, various algorithms have been proposed for visual tracking to solve these difficulties. These existing methods can be roughly divided into two categories: discriminative algorithms (e.g., Avidan, 2004; Babenko, Yang, & Belongie, 2009; Hare, Saffari, & Torr, 2011; Kalal, Mikolajczyk, & Matas, 2012; Zhang, Zhang, & Yang, 0000) and generative algorithms (e.g., Kwon & Lee, 2011; Liu, Jin, Su, & Zhang, 2014; Liu, Bai, Su, Zhang, & Sun, 2015; Mei & Ling, 2009; Zhang, Ghanem, Liu, & Ahuja, 2012a).

The discriminative algorithms formulate visual tracking as a binary classification problem which is to extract discriminative features to distinguish the target from the local background. For example, online boosting methods (Grabner and Bischof, 2006, Grabner et al., 2006) design an online boosting classifier to extract and maintain the best discriminative features for visual tracking. Yang, Lu, and Yang (2014) present a discriminative appearance model based on superpixels and facilitate a tracker to distinguish the target and the background with midlevel cues. However, when the limited data is available, discriminative models cannot perform better than generative models (Ng & Jordan, 2002).

In contrast, the generative algorithms model the appearance of the target and search for the candidate which is most similar to the object. In recent years, subspace, as an efficient geometric, has become the most popular choice in generative visual tracking. For example, incremental visual tracking (IVT) (Ross, Lim, Lin, & Yang, 2008) formulates the target applying principal component analysis (PCA) (Jolliffe, 2002, Liu et al., 2012) subspace. Moreover, its incrementally updated strategy can adapt to the appearance changes efficiently. However, IVT learns and updates the object subspace by using corrupted observations; thus, the brittleness of PCA for partial occlusions may decrease its performance.

Sparse representation has proven to be an efficient method to model subspace structure in vision data (Wright, Yang, Ganesh, Sastry, & Ma, 2009). Motivated by this idea, Mei and Ling (2009) proposed a L1 minimization based formula to model appearance subspace for tracking. Each target candidate is represented as a sparse linear combination of the target and trivial templates. However, the tracker cannot exploit redundant and rich image properties. So sparse prototypes (SP) (Wang, Lu, & Yang, 2013) exploit the advantages of both PCA and sparse representation to model the appearance of the target. Unfortunately, as these methods find the sparsest representation coefficients for each frame individually, they cannot capture the global subspace structure of the target.

It is also well known that under some suitable conditions, the low-rank and sparse decomposition based robust principal component analysis (RPCA) (Candès, Li, Ma, & Wright, 2011) model can also exactly recover a low-rank matrix from corrupted observations to represent the subspace. But unfortunately, the optimization of RPCA requires to solve nuclear norm minimization on the whole data matrix, which leads to high computational complexity. Moreover, in RPCA, the nuclear norm minimization problem has to be recomputed on the whole dataset if new samples arrive. Thus, standard RPCA is not suitable for sequential data analysis problems, such as visual tracking. To address above limitations, Liu, Lin, Su, and Gao (2014) develop a linear time optimization scheme for RPCA. Following this idea, the work in Wang, Liu, and Su (2015) and Zhang, Liu, Qiu, and Su (2014) also proposes an incremental extension of RPCA to learn low-rank features for object tracking. However, as low-rank features cannot be used to explicitly represent the subspace, additional postprocess has to be used on the learned low-rank features to obtain final appearance subspace representation. More importantly, all the above-mentioned low-rank methods are designed for general data. Thus no specific structure constraints for appearance subspace in object tracking problems are considered in these formulations. It can be seen in Fig. 1 that the general designed subspace learning based trackers cannot track the target well in challenging sequences. In addition, the filtering scheme proposed in Liu, Lin et al. (2014) cannot be used for column sparse optimization (e.g., ${\ell }_{2,1}$ norm minimization), which are more suitable for frame specified corruptions in video sequences.

In this paper, we design a novel subspace learning model, called low-rank subspace learning with adaptive penalization (LSAP), to specifically address appearance subspace learning problems for object tracking. Different from general subspace learning methods, which do not consider specific constraints for tracking problems, we introduce an adaptive penalization to incorporate structure constraints of appearance subspace into our model. First, by using sample-subspace distance as a generative penalization, we can successfully remove samples which are far from the previously learned appearance subspace. Second, we train a confidence measure of foreground/background as a discriminative penalization. In this way, LSAP can remove candidate samples in the background region thus build more accurate appearance subspace for the object in the foreground region. To formulate frame specified corruptions in the video sequences, we utilize ${\ell }_{2,1}$ norm rather than ${\ell }_1$ norm as sparse regularization in our model. However, due to the attribute of ${\ell }_{2,1}$ norm, we cannot use the online updating scheme in Liu, Lin et al. (2014) to solve LSAP. So as a nontrivial byproduct, we develop a new online optimization algorithm to incrementally update LSAP on sequential data. Comparing with the existing methods, the contributions of our work can be summarized as follows:

• Compared with RPCA, which needs additional postprocess to build explicit subspace representation (e.g., subspace basis) on the learned low-rank features, LSAP can simultaneously learn the subspace basis, low-rank coefficients and column sparse errors by a single optimization model.

• Most general subspace learning models do not consider any specific structure constraints for the object tracking problem. In contrast, we introduce an adaptive penalization to incorporate rich generative and discriminative information into LSAP model for specific appearance subspace learning. Thus, LSAP can extract correct appearance even when the observations are corrupted by severe corruptions (see Fig. 5, Fig. 6).

• To address online optimization issues in LSAP, we extend the filtering idea in Liu, Lin et al. (2014) and develop an efficient incremental optimization scheme for ${\ell }_{2,1}$ norm minimization. It should be noticed that our proposed numerical scheme can also be used to solve other ${\ell }_{2,1}$ regularized low-rank learning problems, such as Liu et al. (2013). Table 1 shows that the speed of our method is much faster than standard RPCA.

• By incorporating a novel convolutional location estimation process into our framework, we successfully provide address.

The remaining of the paper is organized as follows. In Section  2, the most relevant work are reviewed. Section  3 gives a detailed description of the proposed approach and the construction of penalization $W$. The online optimization scheme of our model is proposed in Section  4 in detail. Section  5 introduces visual applications of LSAP model. Experimental results of visual tracking are reported and analyzed in Section  6. In Section  7, we conclude our work and future work.