In order to achieve compression efficiency, the rate distortion cost (RDO) technique and full checking strategy are used to select the optimal intra prediction mode in the latest video coding standard H.264/AVC. As a result, the computational complexity increases drastically. In this paper, a novel fast mode decision algorithm for intra prediction is proposed to improve the coding efficiency. In this algorithm, the histogram of each prediction mode is directly calculated based on the relationship between the pixels and coding modes. Then, the several most possible coding modes are selected from all prediction modes by using histogram. Compared with full search algorithm, the proposed algorithm can reduce 77% coding time on average, with a slight peak signal-to-noise ratio loss of 0.08 dB, a bit rate increment of 4.3%. Furthermore, performance comparison is also given to show the improvement of the proposed algorithm over two methods in the previous literature.