Human activity recognition plays a fundamental role in smart home systems and contributes to remote health monitoring for the elderly or disabled. However, the shallow architecture and heavy parameters of the current 3-D convolutional networks (3-D ConvNets) still restrict the recognition efficiency and spatiotemporal representations. For these issues, this article proposes a framework called LMFNet, which is mainly composed of a deep attentive 3-D residual network (A3D ResNet) and a multistage fusion strategy. Specifically, LMFNet changes the information flow of transmission in the C3D network and implements a residual learning method for efficient training. Besides, the two-stream-fused spatiotemporal attention 3-D ConvNets (2S-F_STA3DCN) are built based on the A3D ResNet. The experimental results show that the proposed LMFNet can achieve a higher recognition accuracy and satisfactory training efficiency compared with the existing methods.