With the increasing demand of high video quality and large image size, adaptive interpolation filter (AIF) addresses these issues and conquers the time varying effects resulting in increased coding efficiency, comparing with recent H.264 standard. However, currently most AIF algorithms are based on either frame level or macroblock (MB) level, which are not flexible enough for different video contents in a real codec system, and most of them are facing a severe time consuming problem. This paper proposes a content based coarse to fine AIF algorithm, which can adapt to video contents by adding different filters and conditions from coarse to fine. The overall algorithm has been mainly made up by 3 schemes: frequency analysis based frame level skip interpolation, motion vector modeling based region level interpolation, and edge detection based macroblock level interpolation. According to the experiments, AIF are discovered to be more effective in the high frequency frames, therefore, the condition to skip low frequency frames for generating AIF coefficients has been set. Moreover, by utilizing the motion vector information of previous frames the region level based interpolation has been designed, and Laplacian of Gaussian based macroblock level interpolation has been proposed to drive the interpolation process from coarse to fine. Six 720p and six 1080p video sequences which cover most typical video types have been tested for evaluating the proposed algorithm. The experimental results show that the proposed algorithm reduce total encoding time about 41% for 720p and 25% for 1080p sequences averagely, comparing with Key Technology Areas (KTA) Enhanced AIF algorithm, while obtains a BDPSNR gain up to 0.004 and 3.122 BDBR reduction.