In a video streaming system, the user’s experience is affected not only by the average video quality but also by some disturbing events such as rebuffering, bitrate switch, etc. How to combine these factors and accurately model the overall user-perceived quality-of-experience (QoE) is a challenging problem. In this paper we propose a time-variant QoE model that takes into account the temporal properties of rebuffering events during the user’s watching process. From data collected in a real video streaming system, we observe that the user’s tolerance to rebuffering changes dynamically with the video playing progress. Based on that, a novel QoE function is defined, in which the penalty weight of each rebuffering event varies with its occurring time. Experimental results show that the proposed model fits better to subject-rated QoE databases, and also brings improvement to the performance of rate adaptation algorithms.