Skip to main content
SpringerLink
Log in

Cost Optimization of Execution of Multi-level Deadline-Constrained Scientific Workflows on Clouds

  • Conference paper
  • First Online:
Parallel Processing and Applied Mathematics (PPAM 2013)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 8384))

Abstract

This paper introduces a cost optimization model for scientific workflows on IaaS clouds such as Amazon EC2 or RackSpace. We assume multiple IaaS clouds with heterogeneous VM instances, with limited number of instances per cloud and hourly billing. Input and output data are stored on a Cloud Object Store such as Amazon S3. Applications are scientific workflows modeled as DAGs as in the Pegasus Workflow Management System. We assume that tasks in the workflows are grouped into levels of identical tasks. Our model is specified in AMPL modeling language and allows us to minimize the cost of workflow execution under deadline constraints. We present results obtained using our model and the benchmark workflows representing real scientific applications such as Montage, Epigenomics, LIGO. We indicate how this model can be used for scenarios that require resource planning for scientific workflows and their ensembles.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Abrishami, S., Naghibzadeh, M., Epema, D.H.: Deadline-constrained workflow scheduling algorithms for infrastructure as a service clouds. Future Gener. Comput. Syst. 29(1), 158–169 (2013). http://www.sciencedirect.com/science/article/pii/S0167739X12001008

  2. AWS: AWS public datasets. http://aws.amazon.com/publicdatasets/ (2013)

  3. Barrionuevo, J.J.D., Fard, H.M., Prodan, R.: Moheft: a multi-objective list-based method for workflow scheduling. In: 4th IEEE International Conference on Cloud Computing Technology and Science Proceedings, CloudCom 2012, Taipei, Taiwan, 3–6 December 2012, pp. 185–192 (2012)

    Google Scholar 

  4. Bharathi, S., Chervenak, A., Deelman, E., Mehta, G., Su, M.H., Vahi, K.: Characterization of scientific workflows. In: Third Workshop on Workflows in Support of Large-Scale Science, WORKS 2008, pp. 1–10. IEEE (2008). http://dx.doi.org/10.1109/WORKS.2008.4723958

  5. Bittencourt, L.F., Madeira, E.R.M.: Hcoc: a cost optimization algorithm for workflow scheduling in hybrid clouds. J. Internet Serv. Appl. 2(3), 207–227 (2011)

    Article  Google Scholar 

  6. den Bossche, R.V., Vanmechelen, K., Broeckhove, J.: Online cost-efficient scheduling of deadline-constrained workloads on hybrid clouds. Future Gener. Comput. Syst. 29(4), 973–985 (2013). http://www.sciencedirect.com/science/article/pii/S0167739X12002324

  7. Bubak, M., Kasztelnik, M., Malawski, M., Meizner, J., Nowakowski, P., Varma, S.: Evaluation of cloud providers for VPH applications. In: CCGrid2013 - 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid, Computing, May 2013. http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6546092

  8. Chen, J., Wang, C., Zhou, B.B., Sun, L., Lee, Y.C., Zomaya, A.Y.: Tradeoffs between profit and customer satisfaction for service provisioning in the cloud. In: Proceedings of the 20th International Symposium on High Performance Distributed Computing, HPDC ’11, pp. 229–238. ACM, New York (2011)

    Google Scholar 

  9. CloudHarmony: Benchmarks. http://cloudharmony.com/benchmarks (2011)

  10. Deelman, E., Juve, G., Malawski, M., Nabrzyski, J.: Hosted science: managing computational workflows in the cloud. Parallel Process. Lett. 23(2), June 2013. http://www.worldscientific.com/doi/abs/10.1142/S0129626413400045

  11. Duan, R., Prodan, R., Li, X.: A sequential cooperative game theoretic approach to storage-aware scheduling of multiple large-scale workflow applications in grids. In: 2012 ACM/IEEE 13th International Conference on Grid Computing (GRID), pp. 31–39. IEEE (2012). http://dx.doi.org/10.1109/Grid.2012.14

  12. Fourer, R., Gay, D.M., Kernighan, B.W.: AMPL: A Modeling Language for Mathematical Programming. Duxbury Press, Belmont (2002)

    Google Scholar 

  13. IBM: IBM ILOG CPLEX Optimization Studio - CPLEX User’s Manual. http://pic.dhe.ibm.com/infocenter/cosinfoc/v12r5/index.jsp (2013)

  14. Kim, H., El-Khamra, Y., Rodero, I., Jha, S., Parashar, M.: Autonomic management of application workflows on hybrid computing infrastructure. Sci. Program. 19, 75–89 (2011)

    Google Scholar 

  15. Malawski, M., Figiela, K., Nabrzyski, J.: Cost minimization for computational applications on hybrid cloud infrastructures. Future Gener. Comput. Syst. 29(7), 1786–1794 (2013). http://www.sciencedirect.com/science/article/pii/S0167739X13000186

  16. Malawski, M., Juve, G., Deelman, E., Nabrzyski, J.: Cost- and deadline-constrained provisioning for scientific workflow ensembles in IaaS clouds. In: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, SC ’12. IEEE Computer Society Press (2012). http://portal.acm.org/citation.cfm?id=2389026

  17. Mao, M., Humphrey, M.: Auto-scaling to minimize cost and meet application deadlines in cloud workflows. In: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’11. ACM, New York (2011). http://dx.doi.org/10.1145/2063384.2063449

  18. Pandey, S., Barker, A., Gupta, K.K., Buyya, R.: Minimizing execution costs when using globally distributed cloud services. In: 24th IEEE International Conference on Advanced Information Networking and Applications, pp. 222–229. IEEE (2010)

    Google Scholar 

  19. Tolosana-Calasanz, R., Banares, J.A., Pham, C., Rana, O.F.: Enforcing QoS in scientific workflow systems enacted over cloud infrastructures. J. Comput. Syst. Sci. 78(5), 1300–1315 (2012). http://www.sciencedirect.com/science/article/pii/S0022000011001607

Download references

Acknowledgement

This research was partially supported by the EC ICT VPH-Share Project (contract 269978), the KI AGH grant, and by the National Science Foundation under grant OCI-1148515.

Author information

Authors and Affiliations

  1. Department of Computer Science, AGH, al. Mickiewicza 30, 30-059, Kraków, Poland

    Maciej Malawski, Kamil Figiela & Marian Bubak

  2. ACC CYFRONET AGH, ul. Nawojki 11, 30-950, Kraków, Poland

    Marian Bubak

  3. USC Information Sciences Institute, Admiralty Way, Marina Del Rey, CA, 4676, USA

    Ewa Deelman

  4. Center for Research Computing, University of Notre Dame, Notre Dame, IN, USA

    Jarek Nabrzyski

Authors
  1. Maciej Malawski
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kamil Figiela
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marian Bubak
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ewa Deelman
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jarek Nabrzyski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maciej Malawski .

Editor information

Editors and Affiliations

  1. Institute of Computer and Information Science, Czestochowa University of Technology, Czestochowa, Poland

    Roman Wyrzykowski

  2. University of Tennessee, Department of Computer Science, Knoxville, Tennessee, USA

    Jack Dongarra

  3. Institute of Computer and Information Science, Czestochowa University of Technology, Czestochowa, Poland

    Konrad Karczewski

  4. Technical University of Denmark Informatics and Mathematical Modelling, Kongens Lyngby, Denmark

    Jerzy Waśniewski

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Malawski, M., Figiela, K., Bubak, M., Deelman, E., Nabrzyski, J. (2014). Cost Optimization of Execution of Multi-level Deadline-Constrained Scientific Workflows on Clouds. In: Wyrzykowski, R., Dongarra, J., Karczewski, K., Waśniewski, J. (eds) Parallel Processing and Applied Mathematics. PPAM 2013. Lecture Notes in Computer Science(), vol 8384. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-55224-3_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-55224-3_24

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-55223-6

  • Online ISBN: 978-3-642-55224-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation