Skip to main content
SpringerLink
Log in

Common Structures in Resource Management as Driver for Reinforcement Learning: A Survey and Research Tracks

  • Conference paper
  • First Online:
Machine Learning for Networking (MLN 2018)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11407))

Included in the following conference series:

Abstract

In the era of growing digitalization, dynamic resource management becomes one of the critical problems in many application fields where, due to the permanently evolving environment, the trade-off between cost and system performance needs to be continuously adapted. While traditional approaches based on prior system specification or model learning are challenged by the complexity and the dynamicity of these systems, a new paradigm of learning in interaction brings a strong promise - based on the toolset of model-free Reinforcement Learning (RL) and its great success stories in various domains. However, current RL methods still struggle to learn rapidly in incremental, online settings, which is a barrier to deal with many practical problems. To address the slow convergence issue, one approach consists in exploiting the system’s structural properties, instead of acting in full model-free mode. In this paper, we review the existing resource management systems and unveil their common structural properties. We propose a meta-model and discuss the tracks on how these properties can enhance general purpose RL algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Sutton, R.S., Barto, A.G.: RL: An Introduction, 2nd edn. The MIT Press, Cambridge, London (2017)

    Google Scholar 

  2. Clark, J. This Preschool is for Robots. Bloomberg (2015)

    Google Scholar 

  3. Gu, S., Holly, E., et al.: Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In: IEEE International Conference on Robotics and Automation (ICRA), Singapore (2017)

    Google Scholar 

  4. Pit.ai. https://www.pit.ai/

  5. Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Human-level control through deep Reinforcement Learning. Nature 518, 529–533 (2015)

    Article  Google Scholar 

  6. Silver, D., Hassabis, D.: AlphaGo: mastering the ancient game of Go with Machine Learning. Google Research Blog (2016)

    Google Scholar 

  7. Jin, Y., Bouzid, M., Kostadinov, D., Aghasaryan, A.: Model-free resource management of cloud-based applications using RL. In: International Workshop on Network Intelligence (NI/ICIN2018), Paris, France (2018)

    Google Scholar 

  8. Liu, Y., Watt, W.: Stabilizing customer abandonment in many-server queues with time-varying arrivals. Oper. Res. 60(6), 1551–1564 (2012)

    Article  MathSciNet  Google Scholar 

  9. Fu, M.C., Marcus, S.I., Wang, I.: Monotone optimal policies for a transient queueing staffing problem. Oper. Res. 48(2), 327–331 (2000)

    Article  Google Scholar 

  10. Bassamboo, A., Harrison, J.M., Zeevi, A.: Design and control of a large call center: asymptotic analysis of an LP-based method. Oper. Res. 54(3), 419–435 (2006)

    Article  MathSciNet  Google Scholar 

  11. Defraeye, M., Van Nieuwenhuyse, I.: Staffing and scheduling under nonstationary demand for service: a literature review. Omega 58, 4–25 (2016)

    Article  Google Scholar 

  12. Gans, N., Koole, G., Mandelbaum, A.: Telephone call centers: tutorial, review, and research prospects. Manuf. Serv. Oper. Manage. 5(2), 79–141 (2003)

    Article  Google Scholar 

  13. Tan, T., Alp, O.: An integrated approach to inventory and flexible capacity management subject to fixed costs and non-stationary stochastic demand. OR Spectrum 31(2), 337–360 (2009)

    Article  MathSciNet  Google Scholar 

  14. Buyukkaramikli, N.C., van Ooijen, H.P., Bertrand, J.W.: Integrating inventory control and capacity management at a maintenance service provider. Ann. Oper. Res. 231(1), 185–206 (2015)

    Article  MathSciNet  Google Scholar 

  15. Bradley, J.R., Glynn, P.W.: Managing capacity and inventory jointly in manufacturing systems. Manage. Sci. 48(2), 273–288 (2002)

    Article  Google Scholar 

  16. Snyder, L.V., Atan, Z., Peng, P., Rong, Y., Schmitt, A.J., Sinsoysal, B.: OR/MS models for supply chain disruptions: a review. IIE Trans. 48(2), 89–109 (2015)

    Article  Google Scholar 

  17. Parikh, S., Patel, N., Prajapati, H.: Resource management in cloud computing: classification and taxonomy. CoRR (2017)

    Google Scholar 

  18. Jennings, B., Stadler, R.: Resource management in clouds: survey and research challenges. J. Netw. Syst. Manage. 23, 567–619 (2015)

    Article  Google Scholar 

  19. Mann, Z.A.: Allocation of virtual machines in cloud data centers - a survey of problem models and optimization algorithms. ACM Comput. Surv. 48(1), 11 (2015)

    Article  Google Scholar 

  20. Amazon: AWS Auto Scaling. https://aws.amazon.com/autoscaling/

  21. Jacobson, D., Yuan, D., Joshi, N.: Scryer: Netflix’s Predictive Auto Scaling Engine. Netflix Technology Blog (2013)

    Google Scholar 

  22. Roy, N., Dubey, A., Gokhale, A.: Efficient autoscaling in the cloud using predictive models for workload forecasting. In: IEEE CLOUD 2011, Washington, pp. 500–507 (2011)

    Google Scholar 

  23. Li, H., Venugopal, S.: Using RL for controlling an elastic web application hosting platform. In: International Conference on Automatic Computing, pp. 205–208 (2011)

    Google Scholar 

  24. Rao, J., Bu, X., Xu, C.-Z., Wang, K.: A distributed self-learning approach for elastic provisioning of virtualized cloud resources. In: 19th Annual IEEE International Symposium on Modelling, Analysis, and Simulation of Computer and Telecommunication Systems, pp. 45–54 (2011)

    Google Scholar 

  25. Manvi, S.S., Shyam, G.K.: Resource management for Infrastructure as a Service (IaaS) in cloud computing: a survey. J. Netw. Comput. Appl. 41, 424–440 (2014)

    Article  Google Scholar 

  26. SON: Self-Organizing Networks. https://www.3gpp.org/technologies/keywords-acronyms/105-son

  27. Hämäläinen, S., Sanneck, H., Sartori, C.: LTE self-organising networks (SON): Network Management Automation for Operational Efficiency. Wiley, Chichester (2012)

    Google Scholar 

  28. Sesia, S., Toufik, I., Baker, M.: LTE - The UMTS Long Term Evolution: From Theory to Practice, 2nd edn. Wiley, Chichester (2011)

    Book  Google Scholar 

  29. Rodriguez, J.: Fundamentals of 5G Mobile Networks. Wiley, Chichester (2015)

    Book  Google Scholar 

  30. Network Functions Virtualisation – Update White Paper. ETSI (2013)

    Google Scholar 

  31. Evolution of the cloud-native mobile core, Nokia White Paper (2017)

    Google Scholar 

  32. Evolving Mobile Core to Being Cloud Native. Cisco White Paper (2017)

    Google Scholar 

  33. Project Clearwater - IMS in the Cloud. http://www.projectclearwater.org/

  34. Pearl, J.: Causality: Models, Reasoning and Inference, 2nd edn. Cambridge University Press, New York (2009)

    Book  Google Scholar 

  35. Yoo, J.: Queueing models for staffing service operations. Ph.D. dissertation. University of Maryland, College Park, MD (1996)

    Google Scholar 

  36. Djonin, D.V., Krishnamurthy, V.: Q-learning algorithms for constrained markov decision processes with randomized monotone policies: application to MIMO transmission control. IEEE Trans. Signal Process. 55(5), 2170–2181 (2007)

    Article  MathSciNet  Google Scholar 

  37. Djonin, D.V., Krishnamurthy, V.: MIMO transmission control in fading channels—a constrained markov decision process formulation with monotone randomized policies. IEEE Trans. Signal Process. 55(10), 5069–5083 (2007)

    Article  MathSciNet  Google Scholar 

  38. Krishnamurthy, V.: Structural Results for Partially Observed Markov Decision Processes (2015). arXiv:1512.03873. https://arxiv.org/abs/1512.03873

  39. Rosenbaum, P.: Design of Observational Studies. Springer, New York (2010). https://doi.org/10.1007/978-1-4419-1213-8

    Book  MATH  Google Scholar 

  40. Shanmugam, K., Kocaoglu, M., Dimakis, A., Vishwanath, S.: Learning causal graphs with small interventions. In: NIPS 2015, Cambridge, MA, USA, pp. 3195–3203 (2015)

    Google Scholar 

  41. Le, T., Hoang, T., Li, J., Liu, L., Liu, H.: A fast PC algorithm for high dimensional causal discovery with multi-core PCs. In: IEEE/ACM Transactions on Computational Biology and Bioinformatics (2015). https://doi.org/10.1109/tcbb.2016.2591526

  42. Spirtes, P., Glymour, C., Scheines, R.: Causation, Prediction, and Search, 2nd edn. MIT Press, Cambridge (2000)

    MATH  Google Scholar 

  43. Ruder, S.: Transfer Learning - Machine Learning’s Next Frontier. Blog post (2017). http://ruder.io/transfer-learning/

  44. Bingel, J., Søgaard, A.: Identifying beneficial task relations for multi-task learning in deep neural networks. In: EACL, pp. 164–169 (2017)

    Google Scholar 

  45. Glorot, X., Bordes, A., Bengio, Y.: Domain adaptation for large-scale sentiment classification: a deep learning approach. In: 28th International Conference on Machine Learning, pp. 513–520 (2011)

    Google Scholar 

  46. Taylor, M., Stone, P.: Transfer learning for reinforcement learning domains: a survey. J. Mach. Learn. Res. 10, 1633–1685 (2009)

    MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Nokia Bell Labs, Paris-Saclay, Nozay, France

    Yue Jin, Dimitre Kostadinov, Makram Bouzid & Armen Aghasaryan

Authors
  1. Yue Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dimitre Kostadinov
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Makram Bouzid
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Armen Aghasaryan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yue Jin .

Editor information

Editors and Affiliations

  1. Télécom SudParis, Évry, France

    Éric Renault

  2. Inria, Paris, France

    Paul Mühlethaler

  3. CNAM/CEDRIC, Paris, France

    Selma Boumerdassi

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jin, Y., Kostadinov, D., Bouzid, M., Aghasaryan, A. (2019). Common Structures in Resource Management as Driver for Reinforcement Learning: A Survey and Research Tracks. In: Renault, É., Mühlethaler, P., Boumerdassi, S. (eds) Machine Learning for Networking. MLN 2018. Lecture Notes in Computer Science(), vol 11407. Springer, Cham. https://doi.org/10.1007/978-3-030-19945-6_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-19945-6_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-19944-9

  • Online ISBN: 978-3-030-19945-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation