Abstract
Now, short video application users have reached 1.02 billion and accounted for 94.8% of the total Internet users. The preload strategy for short video is the key to guarantee the Quality of Experience (QoE) of users. However, the design of preload strategy is challenging because the performance is influenced by factors including network bandwidth, video types, and user behavior. Existing preload strategies suffer from two issues. First, the impact of current decision on the future decision is ignored and each decision is evaluated independently, leading to local optimal decision. Second, the learning-based preload strategies predict the QoE of decisions as the rewards, which may deviate from the actual rewards of the decisions. To address these issues, we design the Reinforcement Learning based Preload Strategy (RLPS) for short video to improve QoE in this work. Specifically, RLPS constructs a delayed feedback mechanism to obtain the actual reward of each decision. In this way, the impacts of current decision on the future decision are also involved in the reward function. Simulation results confirm the advantages of RLPS under different scenarios. Specifically, compared with the state-of-the-art strategy PDAS, RLPS improves the combination score of QoE and bandwidth usage by more than 17.3%.
This work is supported by the Nation Natural Science Foundation of China (No. 61972421), the Key R&D Plan of Hunan Province (No. 2022SK2107), the Excellent Youth Foundation of Hunan Province (No. 2022JJ20078), and the Fundamental Research Funds for the Central Universities of Central South University in China (No. 2022ZZTS0705). This work uses the computing resources at the High Performance Computing Center of Central South University.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
CNNIC. the 51th Statistical Report on the Development of Internet in China. https://cnnic.cn/n4/2023/0302/c199-10755.htmll. Accessed 2 Mar 2023
Qian, S.Z., Xie, Y., Pan, Z., et al.: DAM: deep reinforcement learning based preload algorithm with action masking for short video streaming. In: Proceedings of the 30th ACM International Conference on Multimedia, pp. 7030–7034 (2022)
Guo, J., Zhang, G.A.: video-quality driven strategy in short video streaming. In: Proceedings of the 24th International ACM Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems, pp. 221–228 (2021)
Zhou, C., Ban, Y., Zhao, Y., et al.: PDAS: probability-driven adaptive streaming for short video. In: Proceedings of the 30th ACM International Conference on Multimedia, pp. 7021–7025 (2022)
Konda, V., Tsitsiklis, J.: Actor-critic algorithms. Adv. Neural Inf. Process. Syst. 12 (1999)
Huang, T., Zhou, C., Zhang, R.X., et al.: Stick: a harmonious fusion of buffer-based and learning-based approach for adaptive streaming. In: IEEE INFOCOM 2020-IEEE Conference on Computer Communications, pp. 1967–1976. IEEE (2020)
Lv, G., Wu, Q., Wang, W., et al.: Lumos: towards better video streaming QOE through accurate throughput prediction. In: IEEE INFOCOM 2022-IEEE Conference on Computer Communications, pp. 650–659. IEEE (2022)
Yin, X., Jindal, A., Sekar, V., et al.: A control-theoretic approach for dynamic adaptive video streaming over HTTP. In: Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication, pp. 325−338 (2015)
Zhang, X., Ou, Y., Sen, S., et al.: SENSEI: aligning video streaming quality with dynamic user sensitivity. In: 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21), pp. 303–320 (2021)
Yan, F.Y., Ayers, H., Zhu, C., et al.: Learning in situ: a randomized experiment in video streaming. In: 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20), pp. 495–511 (2020)
Zuo, X., Li, Y., Xu, M., et al.: Bandwidth-efficient multi-video prefetching for short video streaming. arXiv preprint arXiv:2206.09839 (2022)
Timmerer, C.: HTTP Streaming of MPEG Media [EB/OL]. [2022–04–15]. https://multimediacommunication.blogspot.co.at/2010/05/http-streaming-of-mpeg-media.html
Wu, X., Zhang, L., Cui, L.: QoE-aware download control and bitrate adaptation for short video streaming. In: Proceedings of the 30th ACM International Conference on Multimedia, pp. 7115–7119 (2022)
He, K., Gkioxari, G., Dollár, P., et al. Mask r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969 (2017)
Bishop, C.M. (ed.): Pattern Recognition and Machine Learning. ISS, Springer, New York (2006). https://doi.org/10.1007/978-0-387-45528-0
Mao, H., Netravali, R., Alizadeh, M.: Neural adaptive video streaming with pensieve. In: Proceedings of the Conference of the ACM Special Interest Group on Data Communication, pp. 197–210 (2017)
Kim, J., Jung, Y., Yeo, H., et al.: Neural-enhanced live streaming: Improving live video ingest via online learning. In: Proceedings of the Annual Conference of the ACM Special Interest Group on Data Communication on the Applications, Technologies, Architectures, and Protocols for Computer Communication, pp. 107–125 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Ren, Z., Shan, Y., Jiang, W., Shan, Y., Shan, D., Wang, J. (2023). Reinforcement-Learning Based Preload Strategy for Short Video. In: Huang, DS., Premaratne, P., Jin, B., Qu, B., Jo, KH., Hussain, A. (eds) Advanced Intelligent Computing Technology and Applications. ICIC 2023. Lecture Notes in Computer Science(), vol 14090. Springer, Singapore. https://doi.org/10.1007/978-981-99-4761-4_28
Download citation
DOI: https://doi.org/10.1007/978-981-99-4761-4_28
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-4760-7
Online ISBN: 978-981-99-4761-4
eBook Packages: Computer ScienceComputer Science (R0)