Skip to main content
SpringerLink
Log in

Improving Existing WMS for Reduced Makespan of Workflows with Lambda

  • Conference paper
  • First Online:
Euro-Par 2020: Parallel Processing Workshops (Euro-Par 2020)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 12480))

Included in the following conference series:

  • 843 Accesses

Abstract

Scientific workflows are increasingly important for complex scientific applications. Recently, Function as a Service (FaaS) has emerged as a platform for processing non-interactive tasks. FaaS (such as AWS Lambda and Google Cloud Functions) can play an important role in processing scientific workflows. A number of works have demonstrated their ability to process these workflows. However, some issues were identified when workflows executed on cloud functions due to their limits (e.g., stateless behaviour). A major issue is the additional data transfer during the execution between object storage and the FaaS invocation environment. This leads to increased communication costs. DEWE v3 is one of the Workflow Management Systems (WMSs) that already had foundations for processing workflows with cloud functions. In this paper, we have modified the job dispatch algorithm of DEWE v3 on a function environment to reduce data dependency transfers. Our modified algorithm schedules jobs with precedence constraints to be executed in a single function invocation. Therefore, later jobs can utilise output files generated from their predecessor job in the same invocation. This reduces the makespan of workflow execution. We have evaluated the improved scheduling algorithm and the original with small- and large-scale Montage workflows. The experimental results show that our algorithm can reduce the overall makespan in contrast to the original DEWE v3 by about 10%.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://aws.amazon.com/lambda/.

  2. 2.

    https://cloud.google.com/functions/.

  3. 3.

    https://docs.microsoft.com/en-us/azure/azure-functions/functions-overview.

  4. 4.

    https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-getting-started.

  5. 5.

    https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html.

References

  1. Abramovici, A., et al.: LIGO: the laser interferometer gravitational-wave observatory. Science 256(5055), 325–333 (1992)

    Article  Google Scholar 

  2. Altintas, I., Berkley, C., Jaeger, E., Jones, M., Ludascher, B., Mock, S.: Kepler: an extensible system for design and execution of scientific workflows. In: Proceedings. 16th International Conference on Scientific and Statistical Database Management, pp. 423–424. IEEE (2004)

    Google Scholar 

  3. Deelman, E., et al.: Pegasus: mapping scientific workflows onto the grid. In: Dikaiakos, M.D. (ed.) AxGrids 2004. LNCS, vol. 3165, pp. 11–20. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-28642-4_2

    Chapter  Google Scholar 

  4. Graves, R., et al.: CyberShake: a physics-based seismic hazard model for southern California. Pure Appl. Geophys. 168(3–4), 367–381 (2011)

    Article  Google Scholar 

  5. Jacob, J.C., et al.: Montage: a grid portal and software toolkit for science-grade astronomical image mosaicking. arXiv preprint arXiv:1005.4454 (2010)

  6. Jiang, Q., Lee, Y.C., Zomaya, A.Y.: Serverless execution of scientific workflows. In: Maximilien, M., Vallecillo, A., Wang, J., Oriol, M. (eds.) ICSOC 2017. LNCS, vol. 10601, pp. 706–721. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-69035-3_51

    Chapter  Google Scholar 

  7. Juve, G., Deelman, E.: Resource provisioning options for large-scale scientific workflows. In: 2008 IEEE Fourth International Conference on eScience, pp. 608–613. IEEE (2008)

    Google Scholar 

  8. Kecskemeti, G.: DISSECT-CF: a simulator to foster energy-aware scheduling in infrastructure clouds. Simul. Model. Pract. Theory 58, 188–218 (2015)

    Article  Google Scholar 

  9. Kijak, J., Martyna, P., Pawlik, M., Balis, B., Malawski, M.: Challenges for scheduling scientific workflows on cloud functions. In: 2018 IEEE 11th International Conference on Cloud Computing (CLOUD), pp. 460–467. IEEE (2018)

    Google Scholar 

  10. Lee, H., Satyam, K., Fox, G.: Evaluation of production serverless computing environments. In: 2018 IEEE 11th International Conference on Cloud Computing (CLOUD), pp. 442–450. IEEE (2018)

    Google Scholar 

  11. Malawski, M.: Towards serverless execution of scientific workflows-hyperflow case study. In: Works@ Sc, pp. 25–33 (2016)

    Google Scholar 

  12. Malawski, M., Gajek, A., Zima, A., Balis, B., Figiela, K.: Serverless execution of scientific workflows: experiments with hyperflow, aws lambda and google cloud functions. Future Gener. Comput. Syst. (2017)

    Google Scholar 

  13. Pawlik, M., Figiela, K., Malawski, M.: Performance considerations on execution of large scale workflow applications on cloud functions. arXiv preprint arXiv:1909.03555 (2019)

Download references

Acknowledgements

This work was supported in part by the Hungarian Scientific Research Fund under Grant agreement OTKA FK 131793.

Author information

Authors and Affiliations

  1. Institute of Information Technology, University of Miskolc, Miskolc, Hungary

    Ali Al-Haboobi & Gabor Kecskemeti

  2. Department of Computer Sceince, Liverpool John Moores University, Liverpool, UK

    Gabor Kecskemeti

Authors
  1. Ali Al-Haboobi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gabor Kecskemeti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Ali Al-Haboobi or Gabor Kecskemeti .

Editor information

Editors and Affiliations

  1. AGH University of Science and Technology, Krakow, Poland

    Bartosz Balis

  2. CiTIUS, Santiago de Compostela, Spain

    Dora B. Heras

  3. ICAR-CNR, Naples, Italy

    Laura Antonelli

  4. University of Stirling, Stirling, UK

    Andrea Bracciali

  5. Friedrich-Alexander-Universität, Erlangen, Germany

    Thomas Gruber

  6. Konkuk University, Seoul, Korea (Republic of)

    Jin Hyun-Wook

  7. Otto von Guericke University Magdeburg, Magdeburg, Germany

    Michael Kuhn

  8. Tennessee Tech University, Cookeville, TN, USA

    Stephen L. Scott

  9. Koç University, Istanbul, Turkey

    Didem Unat

  10. Czestochowa University of Technology, Czestochowa, Poland

    Roman Wyrzykowski

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Al-Haboobi, A., Kecskemeti, G. (2021). Improving Existing WMS for Reduced Makespan of Workflows with Lambda. In: Balis, B., et al. Euro-Par 2020: Parallel Processing Workshops. Euro-Par 2020. Lecture Notes in Computer Science(), vol 12480. Springer, Cham. https://doi.org/10.1007/978-3-030-71593-9_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-71593-9_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-71592-2

  • Online ISBN: 978-3-030-71593-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation