Abstract
Cloud computing enables effortless access to a seemingly infinite shared pool of resources, on a pay-per-use basis. As a result, a new challenge has emerged: designing control mechanisms to precisely meet the actual workload requirements of cloud applications in an online manner. To this end, a variety of complex resource management issues have to be addressed, because workloads in the cloud are of a dynamic and heterogeneous nature, and traditional algorithms do not cope well within this context. In this work, we adopt the point of view of the user of a cloud infrastructure and focus on the task of controlling leased resources. We formulate this task as a Reinforcement Learning problem and we simulate the decision-making process of a controller implementing the Q-learning algorithm. We conduct an experimental study, the outcomes of which offer valuable insight into the advantages and shortcomings of using Reinforcement Learning to implement such adaptive cloud resource controllers.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Also referred to as an objective function or return function.
- 2.
Optimality arguments explain empirical regularities through objective maximization.
- 3.
\(-2\) for the two unavailable actions.
- 4.
A Broker allocates resources from multiple cloud providers.
References
Agmon Ben-Yehuda, O., Ben-Yehuda, M., Schuster, A., Tsafrir, D.: The rise of raas: the resource-as-a-service cloud. CACM 57(7), 76–84 (2014)
Aldhalaan, A., Menasc, D.A.: Near-optimal allocation of vms from iaas providers by saas providers. In: ICCAC, pp. 228–231 (2015)
Antonio, P., Stefano, B., Francisco, F., Alessandro, G., Guido, O., Martina, P., Vincenzo, S.: Resource management in multi-cloud scenarios via reinforcement learning. In: Control Conference (CCC), pp. 9084–9089 (2015)
Ardagna, D., Casale, G., Ciavotta, M., Pérez, J.F., Wang, W.: Quality-of-service in cloud computing: modeling techniques and their applications. J. Internet Serv. Appl. 5(1), 1–17 (2014)
Bellman, R.: A markovian decision process. Ind. Univ. Math. J. 6, 679–684 (1957)
Bertsekas, D.P.: Dynamic Programming and Optimal Control, vol. II, 4th edn. Athena Scientific, Belmont (2012)
Bertsekas, D.P.: Abstract Dynamic Programming, 1st edn. Athena Scientific, Belmont (2013)
Bu, X., Rao, J., Xu, C.Z.: Coordinated self-configuration of virtual machines and appliances using a model-free learning approach. IEEE Trans. Parallel Distrib. Syst. 24(4), 681–690 (2013)
Doyle, J.R.: Survey of time preference, delay discounting models. Judgm. Decis. Making 8(2), 116–135 (2013)
Dreyfus, S.: Richard bellman on the birth of dynamic programming. Oper. Res. 50(1), 48–51 (2002)
Dutreilh, X., Kirgizov, S., Melekhova, O., Malenfant, J., Rivierre, N., Truck, I.: Using reinforcement learning for autonomic resource allocation in clouds: towards a fully automated workflow. In: ICAS, pp. 67–74, May 2011
Han, R., Ghanem, M.M., Guo, L., Guo, Y., Osmond, M.: Enabling cost-aware and adaptive elasticity of multi-tier cloud applications. Fut. Gen. Comp. Syst. 32, 82–98 (2014)
Howard, R.A.: Dynamic programming and markov processes (1960)
Islam, S., Keung, J., Lee, K., Liu, A.: Empirical prediction models for adaptive resource provisioning in the cloud. Fut. Gen. Comp. Syst. 28(1), 155–162 (2012)
Jamshidi, P., Sharifloo, A.M., Pahl, C., Metzger, A., Estrada, G.: Self-learning cloud controllers: fuzzy q-learning for knowledge evolution. In: ICCAC, pp. 208–211 (2015)
Jennings, B., Stadler, R.: Resource management in clouds: survey and research challenges. J. Netw. Syst. Manag. 23(3), 567–619 (2014)
Kassela, E., Boumpouka, C., Konstantinou, I., Koziris, N.: Automated workload-aware elasticity of nosql clusters in the cloud, pp. 195–200 (2014)
Malekimajd, M., Ardagna, D., Ciavotta, M., Rizzi, A.M., Passacantando, M.: Optimal map reduce job capacity allocation in cloud systems. SIGMETRICS Perform. Eval. Rev. 42(4), 51–61 (2015)
Markov, A.A., Schorr-Kon, J.J.: The theory of algorithms (1954)
Marshall, P., Keahey, K., Freeman, T.: Elastic site: using clouds to elastically extend site resources. In: CCGRID, pp. 43–52. IEEE Computer Society (2010)
Nguyen, H., Shen, Z., Gu, X., Subbiah, S., Wilkes, J.: AGILE: elastic distributed resource scaling for infrastructure-as-a-service. In: ICAC, pp. 69–82 (2013)
Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming, 1st edn. Wiley, New York (1994)
Rao, J., Wei, Y., Gong, J., Xu, C.Z.: Dynaqos: model-free self-tuning fuzzy control of virtualized resources for qos provisioning. In: IEEE International Workshop on Quality of Service (IWQoS), pp. 1–9 (2011)
Sutton, R.S., Barto, A.G.: Introduction to Reinforcement Learning, 1st edn. MIT Press, Cambridge (1998)
Tan, Y., Xia, C.H.: An adaptive learning approach for efficient resource provisioning in cloud services. SIGMETRICS Perform. Eval. Rev. 42(4), 3–11 (2015)
Tsoumakos, D., Konstantinou, I., Boumpouka, C., Sioutas, S., Koziris, N.: Automated, elastic resource provisioning for nosql clusters using TIRAMOLA. In: CCGrid, pp. 34–41 (2013)
Watkins, C.: Learning from delayed rewards. Ph.D. thesis, King’s College (1989)
Yang, J., Qiu, J., Li, Y.: A profile-based approach to just-in-time scalability for cloud applications. In: IEEE CLOUD, pp. 9–16 (2009)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing AG
About this paper
Cite this paper
Kontarinis, A., Kantere, V., Koziris, N. (2016). Cloud Resource Allocation from the User Perspective: A Bare-Bones Reinforcement Learning Approach. In: Cellary, W., Mokbel, M., Wang, J., Wang, H., Zhou, R., Zhang, Y. (eds) Web Information Systems Engineering – WISE 2016. WISE 2016. Lecture Notes in Computer Science(), vol 10041. Springer, Cham. https://doi.org/10.1007/978-3-319-48740-3_34
Download citation
DOI: https://doi.org/10.1007/978-3-319-48740-3_34
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-48739-7
Online ISBN: 978-3-319-48740-3
eBook Packages: Computer ScienceComputer Science (R0)