This paper addresses optical flow estimation and motion segmentation through a highly robust estimator vbQMDPE [Wang, H., and Suter, D., Oct. 2003]. A coarse-to-fine model based optical flow estimation technique through the robust local regression is first presented. Then, the estimated optical flow is further optimised in each hierarchical level based on piecewise spatial coherence constraint using a robust M-estimator [Black, M. J. and Anandan, P., Jan. 1996]. This technique can produce an accurate optical flow estimate for image sequence containing multiple motion and large motion. In the motion segmentation, we present a model-based motion segmentation technique through the mean shift based robust estimator vbQMDPE. The experimental results have demonstrated that our approach achieved good performance in most synthetic and real video sequences.