To provide more powerful video enabled applications, e.g. in video surveillance environments, it is increasingly more critical not only to have access to the decoded video but also to, e.g. efficiently search for similar videos. In this context, this paper proposes a feature-based video coding solution adopting a hybrid approach where both pixels and local visual features are exploited for coding. In this novel solution, part of the frames are coded using a set of key point matches, thus allowing not only to decode the usual frames for visualization but also valuable key point information extracted from uncompressed frames which is instrumental for searching. Experimental results for video surveillance like sequences and conditions show bitrate savings regarding the state-of-the-art HEVC standard while additionally facilitating more accurate searching.