Visual features of images and video frames have become pervasive and maturely developed in extensive research fields such as computer vision and visual search. In more and more cases, the visual feature becomes necessary information which needs to be transmitted and stored at server side in cloud. Among visual features, the local feature descriptors extracted by SIFT can represent both translational and non-translational motion, such as orientation and zooming. On the other hand, only translational motion can be represented by the Motion Vector (MV) in current MV based block video coding standard. Inspired by these properties, a method that utilizes the available feature to optimize inter prediction video coding is proposed in this paper. In this method, the localization, orientation and scale parameters of matching features extracted by SIFT are delivered to inter prediction to provide non-translational motion estimation (ME) and optimized merge mode. Experimental results have shown that the proposed method can efficiently improve the coding performance according to the accurate feature-matching.