Abstract
Cost-efficient selection and scheduling of a subset of geographically distributed resources to meet the demands of a scientific workflow is a challenging problem. The problem is exacerbated by uncertainties in demand and availability of resources. In this paper, we present a stochastic optimization based framework for robust decision making in the selection of distributed resources over a planning horizon under demand uncertainty. We present a novel two-stage stochastic programming model for resource selection, and implement an L-shaped decomposition algorithm to solve this model. A Sample Average Approximation algorithm is integrated to enable stochastic optimization to solve problems with a large number of scenarios. Using the metric of stochastic solution, we demonstrate up to 30% cost reduction relative to solutions without explicit consideration of demand uncertainty for a 24-month problem. We also demonstrate up to 54% cost reduction relative to a previously developed solution for a 36-month problem. We further argue that the composition of resources selected is superior to solutions computed without explicit consideration of uncertainties. Given the importance of resource selection and scheduling of complex scientific workflows, especially in the context of commercial cloud computing, we believe that our novel stochastic programming framework will benefit many researchers as well as users of distributed computing resources.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Benders, J.F.: Partitioning procedures for solving mixed-variables programming problems. Numerische mathematik 4(1), 238–252 (1962)
Bhuiyan, T.H., Nandi, A.K., Medal, H., Halappanavar, M.: Minimizing expected maximum risk from cyber-attacks with probabilistic attack success. In: 2016 IEEE Symposium on Technologies for Homeland Security (HST), pp. 1–6. IEEE (2016)
Birge, J.R., Louveaux, F.: Introduction to Stochastic Programming, 2nd edn. Springer Publishing Company, New York (2011). https://doi.org/10.1007/978-1-4614-0237-4
Chaisiri, S., Lee, B.S., Niyato, D.: Optimization of resource provisioning cost in cloud computing. IEEE Trans. Serv. Comput. 5(2), 164–177 (2012)
Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE Trans. Evol. Comput. 6(2), 182–197 (2002)
Friese, R.D., Halappanavar, M., Sathanur, A.V., Schram, M., Kerbyson, D.J., de la Torre, L.: Towards efficient resource allocation for distributed workflows under demand uncertainties. In: Klusáček, D., Cirne, W., Desai, N. (eds.) JSSPP 2017. LNCS, vol. 10773, pp. 103–121. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-77398-8_6
Gurobi, O.: Gurobi optimizer reference manual (2015). http://www.gurobi.com
Halappanavar, M., Schram, M., de la Torre, L., Barker, K., Tallent, N.R., Kerbyson, D.J.: Towards efficient scheduling of data intensive high energy physics workflows. In: Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science, WORKS 2015, pp. 3:1–3:9. ACM, New York, USA (2015)
Hara, T.: Belle II: Computing and network requirements. In: Proceedings of the Asia-Pacific Advanced Network, pp. 115–122 (2014)
Huang, Z.C., He, C., Gu, L., Wu, J.F.: On-demand service in grid: architecture, design and implementation. In: 2005 Proceedings of 11th International Conference on Parallel and Distributed Systems, vol. 2, pp. 674–678. IEEE (2005)
Jirutitijaroen, P., Singh, C.: Reliability constrained multi-area adequacy planning using stochastic programming with sample-average approximations. IEEE Trans. Power Syst. 23(2), 504–513 (2008)
Johannes, A., Borhan, N., Liu, C., Ranjan, R., Chen, J.: A user demand uncertainty based approach for cloud resource management. In: 2013 IEEE 16th International Conference on Computational Science and Engineering (CSE), pp. 566–571. IEEE (2013)
Kleywegt, A.J., Shapiro, A., Homem-de Mello, T.: The sample average approximation method for stochastic discrete optimization. SIAM J. Optim. 12(2), 479–502 (2002)
Kusic, D., Kandasamy, N.: Risk-aware limited lookahead control for dynamic resource provisioning in enterprise computing systems. Cluster Comput. 10(4), 395–408 (2007)
Mak, W.K., Morton, D.P., Wood, R.K.: Monte carlo bounding techniques for determining solution quality in stochastic programs. Oper. Res. Lett. 24(1), 47–56 (1999)
Medernach, E., Sanlaville, E.: Fair resource allocation for different scenarios of demands. Eur. J. Oper. Res. 218(2), 339–350 (2012)
Rodriguez, M.A., Buyya, R.: Deadline based resource provisioningand scheduling algorithm for scientific workflows on clouds. IEEE Trans. Cloud Comput. 2(2), 222–235 (2014)
Santoso, T., Ahmed, S., Goetschalckx, M., Shapiro, A.: A stochastic programming approach for supply chain network design under uncertainty. Eur. J. Oper. Res. 167(1), 96–115 (2005)
Saravanan, B., Das, S., Sikri, S., Kothari, D.: A solution to the unit commitment problem-a review. Front. Energy 7(2), 223 (2013)
Shapiro, A., Homem-de Mello, T.: A simulation-based approach to two-stage stochastic programming with recourse. Math. Program. 81(3), 301–325 (1998)
Van Slyke, R.M., Wets, R.: L-shaped linear programs with applications to optimal control and stochastic programming. SIAM J. Appl. Math. 17(4), 638–663 (1969)
Yang, J., Qiu, J., Li, Y.: A profile-based approach to just-in-time scalability for cloud applications. In: 2009 IEEE International Conference on Cloud Computing, CLOUD 2009, pp. 9–16. IEEE (2009)
Zhang, Q., Zhu, Q., Boutaba, R.: Dynamic resource allocation for spot markets in cloud computing environments. In: 2011 Fourth IEEE International Conference on Utility and Cloud Computing (UCC), pp. 178–185. IEEE (2011)
Zheng, Q.P., Wang, J., Pardalos, P.M., Guan, Y.: A decomposition approach to the two-stage stochastic unit commitment problem. Annal. Oper. Res. 210(1), 387–410 (2013)
Acknowledgements
This work was supported by the Integrated End-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows (IPPD) Project. IPPD is funded by the U.S. Department of Energy Awards FWP-66406 and DE-SC0012630 at the Pacific Northwest National Laboratory. The work of Luis de la Torre was supported in part by the U.S. Department of Energy, Office of Science, Office of Workforce Development for Teachers and Scientists (WDTS) under the Visiting Faculty Program (VFP).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Bhuiyan, T.H. et al. (2019). Stochastic Programming Approach for Resource Selection Under Demand Uncertainty. In: Klusáček, D., Cirne, W., Desai, N. (eds) Job Scheduling Strategies for Parallel Processing. JSSPP 2018. Lecture Notes in Computer Science(), vol 11332. Springer, Cham. https://doi.org/10.1007/978-3-030-10632-4_6
Download citation
DOI: https://doi.org/10.1007/978-3-030-10632-4_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-10631-7
Online ISBN: 978-3-030-10632-4
eBook Packages: Computer ScienceComputer Science (R0)