Skip to main content

Federated Deep Reinforcement Learning - Based Bitrate Adaptation for Dynamic Adaptive Streaming over HTTP

  • Conference paper
  • First Online:
Recent Challenges in Intelligent Information and Database Systems (ACIIDS 2023)

Abstract

In video streaming over HTTP, the bitrate adaptation selects the quality of video chunks depending on the current network condition. Some previous works have applied deep reinforcement learning (DRL) algorithms to determine the chunk’s bitrate from the observed states to maximize the quality-of-experience (QoE). However, to build an intelligent model that can predict in various environments, such as 3G, 4G, Wifi, etc., the states observed from these environments must be sent to a server for training centrally.

In this work, we integrate federated learning (FL) to DRL-based rate adaptation to train a model appropriate for different environments. The clients in the proposed framework train their model locally and only update the weights to the server. The simulations show that our federated DRL-based rate adaptations, called FDRLABR with different DRL algorithms, such as deep Q-learning, advantage actor-critic, and proximal policy optimization, yield better performance than the traditional bitrate adaptation methods in various environments.

This research is funded by Vietnam National University HoChiMinh City (VNU-HCM) under grant number DS2020-28-01, and in part by the Vietnam National Foundation for Science and Technology Development (NAFOSTED) under Grant 102.02-2019.321.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/toiuuvagiaithuat/FDRLABR.

References

  1. Stockhammer, T.: Dynamic adaptive streaming over HTTP- standards and design principles. In: Proceedings of the Second Annual ACM Conference on Multimedia Systems, pp. 133–144 (2011)

    Google Scholar 

  2. Lederer, S., Müller, C., Timmerer, C.: Dynamic adaptive streaming over HTTP dataset. In: Proceedings of the 3rd Multimedia Systems Conference, pp. 89–94 (2012). https://ftp.itec.aau.at/datasets/mmsys12/ElephantsDream/ed_4s/

  3. Spiteri, K., Urgaonkar, R., Sitaraman, R.K.: BOLA: near-optimal bitrate adaptation for online videos. IEEE/ACM Trans. Netw. 28(4), 1698–1711 (2020)

    Article  Google Scholar 

  4. dash.js. https://github.com/Dash-Industry-Forum/dash.js/

  5. Claeys, M., Latré, S., Famaey, J., Wu, T., Van Leekwijck, W., De Turck, F.: Design and optimisation of a (FA) Q-learning-based HTTP adaptive streaming client. Connect. Sci. 26(1), 25–43 (2014)

    Article  Google Scholar 

  6. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT press, Cambridge (2018)

    Google Scholar 

  7. Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)

    Article  Google Scholar 

  8. Mnih, V., et al.: Asynchronous methods for deep reinforcement learning. In: International Conference on Machine Learning PMLR, pp. 1928–1937 (2016)

    Google Scholar 

  9. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)

  10. Mao, H., Netravali, R., Alizadeh, M.: Neural adaptive video streaming with pensieve. In: Proceedings of the Conference of the ACM Special Interest Group on Data Communication, pp. 197–210 (2017)

    Google Scholar 

  11. Gadaleta, M., Chiariotti, F., Rossi, M., Zanella, A.: D-DASH: a deep Q-learning framework for DASH video streaming. IEEE Trans. Cogn. Commun. Netw. 3(4), 703–718 (2017)

    Article  Google Scholar 

  12. Liu, J., Tao, X., Lu, J.: QoE-oriented rate adaptation for DASH with enhanced deep Q-learning. IEEE Access 7, 8454–8469 (2018)

    Article  Google Scholar 

  13. Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., Meger, D.: Deep reinforcement learning that matters. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1 (2018)

    Google Scholar 

  14. McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial Intelligence and Statistics PMLR, pp. 1273–1282 (2017)

    Google Scholar 

  15. Qi, J., Zhou, Q., Lei, L., Zheng, K.: Federated reinforcement learning: techniques, applications, and open challenges. arXiv preprint arXiv:2108.11887 (2021)

  16. Zhang, C., Xie, Y., Bai, H., Yu, B., Li, W., Gao, Y.: A survey on federated learning. Knowl.-Based Syst. 216, 106775 (2021)

    Article  Google Scholar 

  17. Raca, D., Quinlan, J.J., Zahran, A.H., Sreenan, C.J.: Beyond throughput: a 4G LTE dataset with channel and context metrics. In: Proceedings of the 9th ACM Multimedia Systems Conference, pp. 460–465 (2018)

    Google Scholar 

  18. US Federal Communications Commission (FCC). https://data.fcc.gov/download/measuring-broadband-america/2019/data-raw-2019-sept.tar.gz

  19. Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., Dormann, N.: Stable-Baselines3: reliable reinforcement learning implementations. J. Mach. Learn. Res. 22(1), 12348–12355 (2021)

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Phuong L. Vo .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Vo, P.L., Nguyen, N.T., Luu, L., Dinh, C.T., Tran, N.H., Le, TA. (2023). Federated Deep Reinforcement Learning - Based Bitrate Adaptation for Dynamic Adaptive Streaming over HTTP. In: Nguyen, N.T., et al. Recent Challenges in Intelligent Information and Database Systems. ACIIDS 2023. Communications in Computer and Information Science, vol 1863. Springer, Cham. https://doi.org/10.1007/978-3-031-42430-4_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-42430-4_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-42429-8

  • Online ISBN: 978-3-031-42430-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics