Skeleton-based action recognition is a crucial but challenging task in the application of engineering algorithms. However, due to the inaccurate estimation quality, certain joints that should theoretically lack dynamic information show irregular jitter, which affects the recognition accuracy. In this article, we propose a cascade interaction spatial-temporal transformer network (CI-STFormer) to address abnormal joint jitter and multiscale aggregation of features effectively. The network comprises three parts: 1) spatial sparse-level optimization and interaction module. Relying on the topology map, the sparse-level interaction adjacency matrix is constructed through cascading interaction to mask the jitter nondiscriminative joints, and then the transformer is applied to realize the sparse-level internal feature update. 2) Spatial primal-level fusion and interaction module. Multihead self-attention is employed to enhance the original feature representation. Then, the sparse-level update features are cascaded to construct the scale-cascaded interaction adjacency matrix to achieve the balance of noise caused by the jitter and the fusion of different scale features. 3) Temporal domain scale-level temporal convolutional network (TCN) combines the multiscale filter and the temporal self-attention channel interaction algorithm to extract different temporal features and the interaction and update of global temporal information. The experimental results show that the proposed method performs excellently on the four datasets of NTU RGB+D 60, NTU RGB+D 120, UAV-Human, and NW-UCLA and achieves state-of-the-art performance on the transformer-based track. Related code will be available on https://github.com/Xdu-Liu/CI-STFormer.git.