Based on a relatively simple motion assumption, the technique of motion-compensated prediction will result in large residue when dealing with video contents containing complex motions. To achieve higher coding efficiency, this paper presents a method named second order prediction (SOP) to exploit remaining signal correlation after motion-compensated prediction. Demonstration and analysis on the SOP concept are provided along with its technical details. Experimental results show that the proposed SOP can outperform H.264/AVC P-picture coding 10.28% bit-rate saving under the same PSNR.