This paper presents a method for optical flow estimation and segmentation through polynomial surface fitting, robust least-median-squares regression, and robust statistic clustering mean shift. This approach consists of three stages. First, a standard polynomial surface fitting is used to smooth an image, and least-median-of-squares (LMedS) robust regression is used to calculate the optical flow, which can tolerate up to 50% outlier contamination. Second, the estimated optical flow map is segmented through a mean shift technique. Third, an affine flow model is employed to fit the coarse flow estimates within the segmented regions, and the affine fitted motion of the regions is refined with a robust least median squares process based on optical flow constraints. The experimental results have demonstrated that our approach achieved good performance in most synthetic and real video sequences.