This paper presents a robust object tracking algorithm using a collaborative model. Under the framework of particle filtering, we develop a multi-task learning based generative and discriminative classifier model. In the generative model, we propose a histogram-based subspace learning method that takes advantage of adaptive template update. In the discriminative model, we introduce an effective method to compute the confidence value that assigns more weights to the foreground than the background. A decomposition model is employed to take the outliers of each particle into consideration. The alternating direction method of multipliers (ADMM) algorithm guarantees the optimization problem can be solved robustly and accurately. Qualitative and quantitative comparison with ten state-of-the-art methods demonstrates the effectiveness and efficiency of our method in handling various challenges during tracking.