This paper presents a novel visual tracking method based on linear representation. First, we present a novel probability continuous outlier model (PCOM) to depict the continuous outliers within the linear representation model. In the proposed model, the element of the noisy observation sample can be either represented by a principle component analysis subspace with small Guassian noise or treated as an arbitrary value with a uniform prior, in which a simple Markov random field model is adopted to exploit the spatial consistency information among outliers (or inliners). Then, we derive the objective function of the PCOM method from the perspective of probability theory. The objective function can be solved iteratively by using the outlier-free least squares and standard max-flow/min-cut steps. Finally, for visual tracking, we develop an effective observation likelihood function based on the proposed PCOM method and background information, and design a simple update scheme. Both qualitative and quantitative evaluations demonstrate that our tracker achieves considerable performance in terms of both accuracy and speed.