Motion vectors extracted from a compressed video file can be used to track objects in the video and it could be efficient as motion vectors provide trajectory information of the objects. However, tracking objects represented by the motion vectors can be inaccuracy because of camera movement, small size sets of motion vectors acting as noise, unmoving of the object and occlusion. These are conditions in most real world video application. The system in this paper uses the statistical and distributional information of motion vectors to overcome the problems with three stages. 1) Frame preprocessing uses a Mode reduction technique to remove unwanted motion vectors created from camera movements. 2) Intra-frame processing: k-means is used to segment and cluster moving objects. Statistical standard deviation is used to extract objects' torso and remove small size sets of motion vectors. 3) Inter-frame processing: By comparing the positional information between successive frames, tracking object in successive frames is assigned a same label. A copying rule is used to represent the stopping of the tracking object. The direction and velocity information of motion vector is used for the occlusion problems. Overall, an experiment on tracking multiple basketball players demonstrates a good result of the system.