The rapid growth of mobile video traffic and user demand poses a more stringent requirement for efficient bandwidth allocation in mobile networks where multiple users may share a bottleneck link. This provides content providers an opportunity to jointly optimize multiple users’ experiences but users often suffer short connection durations and frequent handoffs because of their high mobility. In this paper, we propose an end-to-end scheme, VSiM, for supporting mobile video streaming applications in heterogeneous wireless networks. The key idea is allocating bottleneck bandwidth among multiple users based on their mobility profiles and Quality of Experience (QoE)-related knowledge to achieve max-min QoE fairness. Besides, the QoE of buffer-sensitive clients is further improved by the novel server push strategy based on HTTP/3 protocol without affecting the existing bandwidth allocation approach or sacrificing other clients’ view quality. VSiM is lightweight and easy to deploy in the real world without touching the underlying network infrastructure. We evaluated VSiM experimentally in both simulations and a lab testbed on top of the HTTP/3 protocol. We find that the clients’ QoE fairness of VSiM achieves more than 40% improvement compared with state-of-the-art solutions, i.e., the viewing quality of clients in VSiM can be improved from 720p to 1080p in resolution. Meanwhile, VSiM provides about 20% improvement of average QoE.