In this paper, we cast tracking as a novel multi-task learning problem and exploit various types of visual features. We use an on-line feature selection mechanism based on the two-class variance ratio measure, applied to log likelihood distributions computed with respect to a given feature from samples of object and background pixels. The proposed method is integrated in a particle filtering framework. We jointly consider the underlying relationship across different particles, and tackle it in a unified robust multi-task formulation. We show that the proposed formulation can be efficiently solved using the Alternating Direction Method of Multipliers (ADMM) with a small number of closed-form updates. Both the qualitative and quantitative results demonstrate the superior performance of the proposed approach compared to several state of-the-art trackers.