Tracking moving objects in video sequences requires a robust technique. In this paper, a novel object tracking method is discussed. The color distributions histograms is integrated into particle filter, which is also considered the target’s shape as a necessary factor in target model. To compare the target model with the current hypotheses of the particle filter, the Bhattacharyya coefficient is applied. Experimental results demonstrate new method is able to dealing with the complex backgrounds such as clutter and partial occlusion, target rotation, scale change, and even when the target disappears for a while or the camera is moving with high accuracy.