During tracking process in optical flow, some points are normally easy to be lost or become outliers, due to the fact that the object undergoes changes of illumination or becomes partially occluded. The paper presents a novel scheme, Heterogeneity Elimination Individually (HEI), for outlier rejections from the tracking results of optical flow. HEI determines the most poisonous element by the distance between the tracked point and the projected point which is determined by its complements set from the source to the target one. Then HEI eliminates the farthest point every time from the element heterogeneous sequence until all remaining points are within the error tolerance and ensures the accuracy of the optical flow tracking results. Finally, we make an experiment by comparing our method with a popularly used method, RANSAC. The comparison results show the validity and effectiveness of our proposed method for outlier rejection.