To exploit both spatial and temporal correlation of video signal, a Second Order Prediction (SOP) scheme has been presented to eliminate the spatial correlation existing in motion compensated residue. Although SOP achieves better RD performance by adopting residue prediction in Mode Decision (MD) stage, it can be further improved by combining residue prediction into Motion Estimation (ME). Our analysis demonstrates that a more accurate motion vector could be obtained if both ME and MD take residue prediction into account. Implementation of the proposed algorithm into the H.264/AVC reference software demonstrates that the enhanced SOP can achieve up to 18.46% of bit-rate saving at equivalent objective quality.