In this paper, an algorithm, fine granularity successive elimination (FGSE), is proposed for fast optimal block matching in motion estimation. The FGSE is featured by providing a sequence of fine grained boundary levels in an aim to reject a checking candidate as early as possible, thus reducing computational load as much as possible. It is shown that the multilevel successive elimination algorithm (MSEA) is just a degraded case of FGSE with much coarser boundary levels. Comparative experiments are presented to verify the substantial speed-up gain of the proposed algorithm over the MSEA.