AVS video coding standard can achieve considerably high coding efficiency. Unfortunately this comes at a come in considerably increased complexity at the encoder due to mode decision and motion estimation. In this paper, we propose fast inter-prediction mode decision based on probability distribution algorithm and improved small diamond-shaped pattern search algorithm to optimize motion estimation and mode decision. The experimental results show that the proposed methods provide considerable reduction in computational complexity while maintaining coding rate and quality.