We consider the task of real-time pulse rate estimation from the facial video of a subject recorded in a natural setting in an office environment. For estimating the pulse rate, we exploit the fact that the pulse rate does not vary drastically from one analysis window to the next. For this purpose we sparsify the spectra of windowed traces obtained by independent component analysis of the average RGB profile over the face of the subject. This is done by preserving the top few significant peaks of the spectra which are used to compute multiple candidate pulse rate trajectories among which one is chosen for predicting the pulse rate in the current window. The selection of the best trajectory is done such that it passes closely through the peaks of the spectra in consecutive analysis windows. Experiments with video recordings of fifteen subjects using two different camera types and three different camera-subject distances reveal that the estimated pulse rate accuracy improves by 6.71 beats per minute (averaged across different subjects and recording conditions) when the slow-varying nature of the pulse rate is exploited compared to when pulse rate is estimated independently in each analysis window.