This letter presents a novel visual tracking algorithm by using Incremental Non-negative Matrix Factorization (INMF) and dual ${\ell _1}$-norm constraints. Firstly, we introduce one ${\ell _1}$ regularization into the NMF reconstruction, which enables appearance model to tolerate different noises to some extent. Meanwhile, we enforce another ${\ell _1}$ regularization on the projection coefficients when using iterative operators to obtain NMF basis vectors for the effective tracking. Secondly, to obtain the sparse error and projection coefficient matrice, we present an iterative algorithm to solve the optimal problem, which ensures the representation is more robust. Finally, we take partial occlusion into construct likelihood function, and combined with INMF learning to update appearance model for alleviating tracking drift. Experimental results compared with the state-of-the-art tracking methods demonstrate the proposed algorithm achieves favorable performance when the object undergoes large occlusion, motion blur and illumination changes.