This paper presents a novel outlier removal method which is capable of fitting ellipse in real-time under high outlier rate, based on the phenomenon that outliers generated by ellipse edge point detector are likely to appear as groups due to real-world nuisances, such as under partial occlusion or illumination change. To confront the grouped outliers while maintaining the fitting efficiency, we introduce a proximity-based `split and merge' approach to cluster the edge points into subsets, followed by a breath-first outlier removal process. The experiment shows that our algorithm achieves high performance under a wide range of inlier ratio and noise level with various types of realistic nuisances.