Video-based hand gesture authentication is a challenging fine-grained spatiotemporal analysis task, which requires models with the ability to extract stable physiological and behavioral features from dynamic gestures. Inspired by biological studies, we present a physiological–behavioral characteristic understanding network (PB-Net) for hand gesture authentication. According to the properties of physiological and behavioral characteristics, we design two branches, the physiological branch (P-Branch) and the behavioral branch (B-Branch), as well as corresponding data-tailoring strategies for the PB-Net. The data-tailoring strategies can produce two customized videos for the two branches, which can facilitate the analyses of physiological and behavioral characteristics. Besides, the data-tailoring strategies can remove significant redundant information and thus can improve running efficiency. The P-Branch and B-Branch do not interfere with each other and focus on the distillation of physiological and behavioral features separately. Considering that the important degree of the physiological and behavioral features could be different and changeable, we devise an adaptive physiological–behavioral feature fusion (APBF) module to automatically assign appropriate weights for the two features and merge them together to obtain a more satisfactory identity feature. Finally, the rationality, validity, and superiority of the PB-Net are fully demonstrated by extensive ablation experiments and sufficient comparisons with 23 excellent video understanding networks on the SCUT-DHGA dataset. The code is available at https://github.com/SCUT-BIP-Lab/PB-Net.