The Passive Keyless Entry and Start system (PKES) has become an essential element of vehicle systems since it allows owners to lock or unlock their properties without having to take out the keys. However, the system suffers from a potential and serious security problem because of a relay attack. This paper introduces dynamic biometrics to resolve such potential problems. Precisely, it opts for hand-gesture and proposes a Multi-Task Siamese Feature-pyramid network with Bidirectional Gated Recurrent Units (MTSFBet) against relay attack. MTSFBet is made up of a Dynamic Biometrics Extraction Module (DBEM) and Multi-Task Operation Module (MTOM). DBEM extracts trajectory and kinematic features from the gesture data of owners, whereas MTOM performs gesture classification and identity authentication using these features. Moreover, we designed a loss function for the identity authentication part. Eventually, based on the dataset collected by mobile devices, we constructed comparative experiments and ablation studies to demonstrate the effectiveness of the method. Our comprehensive model achieves an accuracy of 0.86 in gesture classification and an accuracy of 0.73 in identity authentication. The results showed that our MTSFBet significantly outperforms other comparison methods. The MTSFBet can also be used in any scenario associated with identity authentication based on hand gestures.