We present a patch-based tracking algorithm in which both appearance and spatial information are taken into account for target localization. We decompose a target into several patches based on appearance similarity and spatial distribution. Each patch has its distinctive appearance and spatial distribution. Appearance information is described by kernels which are non-parametric; while spatial information is represented by spatial Gaussians. The overall motion is estimated by mean shift algorithm. The motion is refined based on the likelihood images computed using pixel classification. The proposed tracker provides better position and likelihood images.