The latest video coding standard HEVC (High Efficiency Video Coding) provides up to 35 intra prediction modes to achieve more accurate prediction. However, a larger number of prediction modes leads to higher encoding complexity. To accelerate the intra mode decision in HEVC, an improved algorithm based on numerical analysis of the neighboring reference samples of coding units is proposed. Four filters are designed to reduce the candidate modes. And an algorithm to seek the proper thresholds for the filters is developed. The experimental results show that when compared to the test model HM 12.0 of HEVC, the proposed method can save up to 28.40% of encoder time with slight increase of BD-rate and negligible loss of PSNR. Furthermore, it is easy to combine our approach with other fast algorithms to achieve more improvement.