Abstract
The rapid growth of Cloud Computing (CC) has increased the variety of computing resources, storage, and communication services that pose significant new challenges for the efficient use of cloud resources. The cost-efficient allocation of cloud resources has become a decisive premise for the adoption of CC services. The cost-efficient selection and scheduling of these resources to meet the demands of a scientific workflow is a challenging problem that is exacerbated by the inclusion of multiple CC providers. In this paper, we present a new strategy for the cost-efficient selection of CC resources using Lagrange relaxation. Our approach is based on preselection of resources and demand decomposition to create a series of smaller sub-problems, which allow the estimation of the best cost-structures and selection of CC service providers for a subset of the time period of the planning horizon. Decomposition of the demand is achieved through the boundary analysis of a continuous relaxation of the problem. Using the metrics defined in terms of the cost and time of completion, we demonstrate excellent performance with respect to optimal solutions. Our method reduced the computational time from hours to seconds for a representative 36-month problem and provided high-quality solutions (<0.05% relative error). Given the importance of selecting resources and scheduling complex scientific workflows, we believe that this novel strategy will be beneficial for many researchers and users of cloud computing resources.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Asner, D.M., Dart, E., Hara, T.: Belle II: experiment network and computing. Technical report arXiv:1308.0672. PNNL-SA-97204, August 2013. Contributed to CSS2013 (Snowmass)
Friese, R.D., Halappanavar, M., Sathanur, A.V., Schram, M., Kerbyson, D.J., de la Torre, L.: Towards efficient resource allocation for distributed workflows under demand uncertainties. In: Proceedings of the 21st Workshop on Job Scheduling Strategies for Parallel Processing (2015)
Gao, P.X., Curtis, A.R., Wong, B., Keshav, S.: It’s not easy being green. ACM SIGCOMM Comput. Commun. Rev. 42(4), 211–222 (2012)
Graham, R.L., Lawler, E.L., Lenstra, J.K., Rinnooy Kan, A.H.G.: Optimization and approximation in deterministic sequencing and scheduling: a survey. Ann. Discret. Math. 5(2), 287–326 (1979)
Halappanavar, M., Schram, M., de la Torre, L., Barker, K., Tallent, N.R., Kerbyson, D.J.: Towards efficient scheduling of data intensive high energy physics workflows. In: Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science. WORKS ’15, pp. 3:1–3:9. ACM, New York, NY, USA (2015)
Hara, T.: Belle II: computing and network requirements. In: Proceedings of the Asia-Pacific Advanced Network, pp. 115–122 (2014)
Harvey, N.J.A., Ladner, R.E., Lovász, L., Tamir, T.: Semi-matchings for bipartite graphs and load balancing. J. Algorithms 59(1), 53–78 (2006)
Juve, G., Deelman, E.: Scientific workflows and clouds. XRDS 16(3), 14–18 (2010). https://doi.org/10.1145/1734160.1734166
Kamiński, B., Szufel, P.: On optimization of simulation execution on amazon ec2 spot market. Simul. Model. Pract. Theory 58, 172–187 (2015)
Mallipeddi, R., Suganthan, P.N.: Unit commitment - a survey and comparison of conventional and nature inspired algorithms. Int. J. Bio-Inspired Comput. 6(2), 71–90 (2014). https://doi.org/10.1504/IJBIC.2014.060609
Roland, J., Figueira, J.R., De Smet, Y.: The inverse 0,1-knapsack problem: theory, algorithms and computational experiments. Discret. Optim. 10(2), 181–192 (2013). https://doi.org/10.1016/j.disopt.2013.03.001, https://www.sciencedirect.com/science/article/pii/S1572528613000066
Saravanan, B., Das, S., Sikri, S., Kothari, D.P.: A solution to the unit commitment problem–a review. Front. Energy 7(2), 223–236 (2013)
Schwartz, R., Dodge, J., Smith, N.A., Etzioni, O.: Green AI. Commun. ACM 63(12), 54–63 (2020). https://doi.org/10.1145/3381831
Singer, G., Livenson, I., Dumas, M., Srirama, S.N., Norbisrath, U.: Towards a model for cloud computing cost estimation with reserved instances. In: Proceedings of 2nd International ICST Conference on Cloud Computing, CloudComp 2010 (2010)
Tarplee, K.M., Friese, R., Maciejewski, A.A., Siegel, H.J.: Efficient and scalable pareto front generation for energy and makespan in heterogeneous computing systems. In: Fidanova, S. (ed.) Recent Advances in Computational Optimization. SCI, vol. 580, pp. 161–180. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-12631-9_10
Wright, B.: A review of unit commitment. ELENE4511, 28 May 2013
Acknowledgment
The research is supported in part by the U.S. DOE Exascale Computing Project’s (ECP) (17-SC-20-SC) ExaGraph codesign center and Laboratory Directed Research and Development Program at Pacific Northwest National Laboratory (PNNL).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
de la Torre, L., Halappanavar, M. (2023). Scaling Optimal Allocation of Cloud Resources Using Lagrange Relaxation. In: Klusáček, D., Corbalán, J., Rodrigo, G.P. (eds) Job Scheduling Strategies for Parallel Processing. JSSPP 2023. Lecture Notes in Computer Science, vol 14283. Springer, Cham. https://doi.org/10.1007/978-3-031-43943-8_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-43943-8_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43942-1
Online ISBN: 978-3-031-43943-8
eBook Packages: Computer ScienceComputer Science (R0)