This paper presents a method for detection of motion vector-based video steganography. First, the modification on the least significant bit of the motion vector is modeled. The influence of the embedding operation on the sum of absolute difference (SAD) is illustrated, which allows us to focus on the difference between the actual SAD and the locally optimal SAD after the adding-or-subtracting-one operation on the motion value. Finally, based on the fact that most motion vectors are locally optimal for most video codecs, two feature sets are extracted and used for classification. Experiments are carried out on videos corrupted by various steganography methods and encoded by various motion estimation methods, in various bit rates, and in various video codecs. Performance results demonstrate that our scheme outperforms previous works in general, and is more favorable for real-world applications.