Traditional solutions for human action recognition usually rely on sensor or video methods. However, these methods have some limitations, such as inconvenient portability, light intensity influence, privacy protection, etc. In this article, an RFID-based nonwearable human action recognition scheme is proposed. In order to reduce the occlusion effect of the human body on the signal and increase the diversity of the reflected signal, a tags array is constructed. The data of phase and RSSI are fused as feature data to enhance the diversity of data. Furthermore, a combined processing method is proposed to eliminate thermal noise generated by the equipment and reduce the interference caused by the environment. Then, an action segmentation algorithm is designed to align the RF signals of human action. Finally, an efficient human action signal classification model is constructed using the spatiotemporal graph convolutional neural network (STGCN). Extensive experiments demonstrate that the overall accuracy rate of the system for human action recognition is 92.8%. Compared with the comparative mainstream recognition algorithms, STGCN shows better classification performance in terms of identification precision. In addition, multimodal RFID data fusion also improves the accuracy of identification.