Remote photoplethysmography (rPPG) is a non-contact measurement method for estimating heart rate, which has great application potential in medical ward monitoring and remote physiological health monitoring. Existing methods for measuring remote physiological signals are susceptible to different degrees of light changes, head movements, and other conditions. In this paper, we design a multi-scale physiological signal feature extraction module (MPSFE) and a plug-and-play time difference global attention module (TDGAM). The MPSFE module extracts different degrees of 3-dimensional convolution of shallow features in the early stage. It obtains more robust semantic information while ensuring the complete position and detailed information to enhance the accuracy and generalization ability of the model. The TDGAM module combines temporal differential convolution with spatial attention and channel attention to enhance the network’s localization of the region of interest (ROI) in the case of light changes and head movement. We conduct many experiments on the three datasets, UBFC-rPPG, COHFACE, and PURE. The results shows that our modules significantly improves the accuracy of heart rate measurement.