Skip to main content

Stochastic Programming Approach for Resource Selection Under Demand Uncertainty

  • Conference paper
  • First Online:
Job Scheduling Strategies for Parallel Processing (JSSPP 2018)

Abstract

Cost-efficient selection and scheduling of a subset of geographically distributed resources to meet the demands of a scientific workflow is a challenging problem. The problem is exacerbated by uncertainties in demand and availability of resources. In this paper, we present a stochastic optimization based framework for robust decision making in the selection of distributed resources over a planning horizon under demand uncertainty. We present a novel two-stage stochastic programming model for resource selection, and implement an L-shaped decomposition algorithm to solve this model. A Sample Average Approximation algorithm is integrated to enable stochastic optimization to solve problems with a large number of scenarios. Using the metric of stochastic solution, we demonstrate up to 30% cost reduction relative to solutions without explicit consideration of demand uncertainty for a 24-month problem. We also demonstrate up to 54% cost reduction relative to a previously developed solution for a 36-month problem. We further argue that the composition of resources selected is superior to solutions computed without explicit consideration of uncertainties. Given the importance of resource selection and scheduling of complex scientific workflows, especially in the context of commercial cloud computing, we believe that our novel stochastic programming framework will benefit many researchers as well as users of distributed computing resources.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Benders, J.F.: Partitioning procedures for solving mixed-variables programming problems. Numerische mathematik 4(1), 238–252 (1962)

    Article  MathSciNet  Google Scholar 

  2. Bhuiyan, T.H., Nandi, A.K., Medal, H., Halappanavar, M.: Minimizing expected maximum risk from cyber-attacks with probabilistic attack success. In: 2016 IEEE Symposium on Technologies for Homeland Security (HST), pp. 1–6. IEEE (2016)

    Google Scholar 

  3. Birge, J.R., Louveaux, F.: Introduction to Stochastic Programming, 2nd edn. Springer Publishing Company, New York (2011). https://doi.org/10.1007/978-1-4614-0237-4

    Book  MATH  Google Scholar 

  4. Chaisiri, S., Lee, B.S., Niyato, D.: Optimization of resource provisioning cost in cloud computing. IEEE Trans. Serv. Comput. 5(2), 164–177 (2012)

    Article  Google Scholar 

  5. Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE Trans. Evol. Comput. 6(2), 182–197 (2002)

    Article  Google Scholar 

  6. Friese, R.D., Halappanavar, M., Sathanur, A.V., Schram, M., Kerbyson, D.J., de la Torre, L.: Towards efficient resource allocation for distributed workflows under demand uncertainties. In: Klusáček, D., Cirne, W., Desai, N. (eds.) JSSPP 2017. LNCS, vol. 10773, pp. 103–121. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-77398-8_6

    Chapter  Google Scholar 

  7. Gurobi, O.: Gurobi optimizer reference manual (2015). http://www.gurobi.com

  8. Halappanavar, M., Schram, M., de la Torre, L., Barker, K., Tallent, N.R., Kerbyson, D.J.: Towards efficient scheduling of data intensive high energy physics workflows. In: Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science, WORKS 2015, pp. 3:1–3:9. ACM, New York, USA (2015)

    Google Scholar 

  9. Hara, T.: Belle II: Computing and network requirements. In: Proceedings of the Asia-Pacific Advanced Network, pp. 115–122 (2014)

    Article  Google Scholar 

  10. Huang, Z.C., He, C., Gu, L., Wu, J.F.: On-demand service in grid: architecture, design and implementation. In: 2005 Proceedings of 11th International Conference on Parallel and Distributed Systems, vol. 2, pp. 674–678. IEEE (2005)

    Google Scholar 

  11. Jirutitijaroen, P., Singh, C.: Reliability constrained multi-area adequacy planning using stochastic programming with sample-average approximations. IEEE Trans. Power Syst. 23(2), 504–513 (2008)

    Article  Google Scholar 

  12. Johannes, A., Borhan, N., Liu, C., Ranjan, R., Chen, J.: A user demand uncertainty based approach for cloud resource management. In: 2013 IEEE 16th International Conference on Computational Science and Engineering (CSE), pp. 566–571. IEEE (2013)

    Google Scholar 

  13. Kleywegt, A.J., Shapiro, A., Homem-de Mello, T.: The sample average approximation method for stochastic discrete optimization. SIAM J. Optim. 12(2), 479–502 (2002)

    Article  MathSciNet  Google Scholar 

  14. Kusic, D., Kandasamy, N.: Risk-aware limited lookahead control for dynamic resource provisioning in enterprise computing systems. Cluster Comput. 10(4), 395–408 (2007)

    Article  Google Scholar 

  15. Mak, W.K., Morton, D.P., Wood, R.K.: Monte carlo bounding techniques for determining solution quality in stochastic programs. Oper. Res. Lett. 24(1), 47–56 (1999)

    Article  MathSciNet  Google Scholar 

  16. Medernach, E., Sanlaville, E.: Fair resource allocation for different scenarios of demands. Eur. J. Oper. Res. 218(2), 339–350 (2012)

    Article  MathSciNet  Google Scholar 

  17. Rodriguez, M.A., Buyya, R.: Deadline based resource provisioningand scheduling algorithm for scientific workflows on clouds. IEEE Trans. Cloud Comput. 2(2), 222–235 (2014)

    Article  Google Scholar 

  18. Santoso, T., Ahmed, S., Goetschalckx, M., Shapiro, A.: A stochastic programming approach for supply chain network design under uncertainty. Eur. J. Oper. Res. 167(1), 96–115 (2005)

    Article  MathSciNet  Google Scholar 

  19. Saravanan, B., Das, S., Sikri, S., Kothari, D.: A solution to the unit commitment problem-a review. Front. Energy 7(2), 223 (2013)

    Article  Google Scholar 

  20. Shapiro, A., Homem-de Mello, T.: A simulation-based approach to two-stage stochastic programming with recourse. Math. Program. 81(3), 301–325 (1998)

    Article  MathSciNet  Google Scholar 

  21. Van Slyke, R.M., Wets, R.: L-shaped linear programs with applications to optimal control and stochastic programming. SIAM J. Appl. Math. 17(4), 638–663 (1969)

    Article  MathSciNet  Google Scholar 

  22. Yang, J., Qiu, J., Li, Y.: A profile-based approach to just-in-time scalability for cloud applications. In: 2009 IEEE International Conference on Cloud Computing, CLOUD 2009, pp. 9–16. IEEE (2009)

    Google Scholar 

  23. Zhang, Q., Zhu, Q., Boutaba, R.: Dynamic resource allocation for spot markets in cloud computing environments. In: 2011 Fourth IEEE International Conference on Utility and Cloud Computing (UCC), pp. 178–185. IEEE (2011)

    Google Scholar 

  24. Zheng, Q.P., Wang, J., Pardalos, P.M., Guan, Y.: A decomposition approach to the two-stage stochastic unit commitment problem. Annal. Oper. Res. 210(1), 387–410 (2013)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

This work was supported by the Integrated End-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows (IPPD) Project. IPPD is funded by the U.S. Department of Energy Awards FWP-66406 and DE-SC0012630 at the Pacific Northwest National Laboratory. The work of Luis de la Torre was supported in part by the U.S. Department of Energy, Office of Science, Office of Workforce Development for Teachers and Scientists (WDTS) under the Visiting Faculty Program (VFP).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Mahantesh Halappanavar .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bhuiyan, T.H. et al. (2019). Stochastic Programming Approach for Resource Selection Under Demand Uncertainty. In: Klusáček, D., Cirne, W., Desai, N. (eds) Job Scheduling Strategies for Parallel Processing. JSSPP 2018. Lecture Notes in Computer Science(), vol 11332. Springer, Cham. https://doi.org/10.1007/978-3-030-10632-4_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-10632-4_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-10631-7

  • Online ISBN: 978-3-030-10632-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics