In this paper, we present a new sorting-based partial distortion elimination (PDE) algorithm for fast optimal motion estimation. By analyzing the contributions to the true sum of absolute differences (SAD), we found that there is a close relationship between the distances from the mean value of the current block and the contributions to the true SAD. By sorting the distances from the pixels to the current block and subtracting them from a mean value, then applying this order to the typical PDE, we can eliminate impossible candidates faster and save substantial computations. Experimental results show that the proposed algorithm reduces computational complexity by about 45% on average compared with the typical PDE.