Skip to main content
SpringerLink
Log in

Real-Time Workflow Scheduling in Cloud with Recursive Neural Network and List Scheduling

  • Conference paper
  • First Online:
Hybrid Artificial Intelligent Systems (HAIS 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14001))

Included in the following conference series:

  • 545 Accesses

Abstract

This paper investigates the problem of dynamic workflow scheduling in cloud computing. In a real-time scenario, the only available information is input data size, and the other task execution requirements, such as execution time, memory consumption, and output data size, must be estimated. In this study, we ask whether a more accurate estimation of task execution requirements can be obtained if workflow structure is taken into account explicitly and whether such estimations can result in more efficient task resource allocations and better computing resource utilization. We compare the estimation accuracy of a graph learning neural network, e.g., Recursive Neural Network (RecNN), with two standard prediction models (that do not consider the workflow structure), e.g., a linear and non-linear regression. We used two types of scientific workflows, Montage and LIGO, to train the prediction models. The execution time (makespan) comparison of the newly generated workflows with the original set of workflows shows that the RecNN model estimates the task information more accurately than linear and non-linear regression models, and the makespan of the workflow generated by the estimated values by RecNN is closer to the makespan of the original workflows. To the best of our knowledge, we are the first to consider the overall workflow topological structure in real-time workflow scheduling scenarios. The result shows that explicitly considering the workflow structure through structure learning models such as RecNN can considerably improve workflow scheduling in the cloud.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The number of recursive connections of a recursive neuron should be equal to the \(max\_degree\) of the domain D, even if not all of them will be used for computing the output of a vertex v with \(out\_deg(v)< max\_degree\).

References

  1. Abramovici, A., et al.: LIGO: the laser interferometer gravitational-wave observatory. Science 256(5055), 325–333 (1992). https://doi.org/10.2307/2877074

    Article  Google Scholar 

  2. Kintsakis, A.M., Psomopoulos, F.E., Mitkas, P.A.: Reinforcement learning based scheduling in a workflow management system. Eng. Appl. Artif. Intell. 81, 94–106 (2019). ISSN 0952–1976, https://doi.org/10.1016/j.engappai.2019.02.013

  3. Shirazi, B., Wang, M., Pathak, G.: Analysis and evaluation of heuristic methods for static task scheduling. J. Parallel Distrib. Comput. 10(3), 222–232 (1990). ISSN 0743–7315, https://doi.org/10.1016/0743-7315(90)90014-G

  4. Kerbyson, D.J., Alme, H.J., Hoisie, A., Petrini, F., Wasserman, H.J., Gittings, M.: Predictive performance and scalability modeling of a large-scale application. In: Proceedings of the 2001 ACM/IEEE Conference on Supercomputing. New York, NY, USA, p. 37 (2001) https://doi.org/10.1145/582034.582071

  5. Nurmi, D., Mandal, A., Brevik, J., Koelbel, C., Wolski, R., Kennedy, K.: Evaluation of a workflow scheduler using integrated performance modelling and batch queue wait time prediction. In: Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC 2006, Tampa, FL, USA, p. 29 (2006). https://doi.org/10.1109/SC.2006.29

  6. Nurmi, D.C., Brevik, J., Wolski, R.: QBETS: queue bounds estimation from time series. In: Proceedings of the 2007 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS 2007). New York, NY, USA, pp. 379–380 (2007). https://doi.org/10.1145/1254882.1254939

  7. Deelman, E., et al.: The future of scientific workflows. Int. J. High Perform. Comput. Appl. 32(1), 159–175 (2018). https://doi.org/10.1177/1094342017704893

    Article  Google Scholar 

  8. Deelman, E., et al.: Pegasus, a workflow management system for science automation, Future Gener. Comput. Syst. 46, 17–35 (2015). ISSN 0167–739X, https://doi.org/10.1016/j.future.2014.10.008

  9. Da Silva, R.F., Juve, G., Rynge, M., Deelman, E., Livny, M.: Online task resource consumption prediction for scientific workflows. Parallel Process. Lett. 25(03), 1541003 (2015). https://doi.org/10.1142/S0129626415410030

  10. Lee, H., Cho, S., Jang, Y., Lee, J., Woo, H.: A global DAG task scheduler using deep reinforcement learning and graph convolution network. IEEE Access 9, 158548–158561 (2021). https://doi.org/10.1109/ACCESS.2021.3130407

    Article  Google Scholar 

  11. Brevik, J., Nurmi, D., Wolski, R.: Predicting bounds on queuing delay for batch-scheduled parallel machines. In: Proceedings of the eleventh ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP 2006), New York, NY, USA, pp. 110–118 (2006). https://doi.org/10.1145/1122971.1122989

  12. Pollack, J.B.: Recursive distributed representations. Artif. Intell. 46(1–2), 77–105 (1990). ISSN 0004–3702, https://doi.org/10.1016/0004-3702(90)90005-K

  13. Jacob, J.C., et al.: Montage: a grid portal and software toolkit for science-grade astronomical image mosaicking. Int. J. Comput. Sci. Eng. 4(2), 73–87 (2009). https://doi.org/10.1504/IJCSE.2009.026999

    Article  Google Scholar 

  14. Rodriguez, M.A., Buyya, R.: Scheduling dynamic workloads in multi-tenant scientific workflow as a service platforms. Future Gener. Comput. Syst. 79, 739–750 (2018). Part 2, ISSN 0167–739X, https://doi.org/10.1016/j.future.2017.05.009

  15. Buyya, R., Broberg, J., Goscinski, A.M.: Cloud Computing: Principles and Paradigms. Wiley, Hoboken (2011). ISBN: 978-0-470-88799-8

    Google Scholar 

  16. Socher, R., et al.: Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631–1642, Seattle, Washington, USA (2013)

    Google Scholar 

  17. Socher, R., Huang, E., Pennin, J., Manning, C.D., Ng, A.: Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In: Proceedings of the 24th International Conference on Neural Information Processing Systems (NIPS 2011), Red Hook, NY, USA, pp. 801–809 (2011). https://doi.org/10.5555/2986459.2986549

  18. Calheiros, R.N., Ranjan, R., Beloglazov, A., De Rose, C.A., Buyya, R.: CloudSim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms. Softw. Pract. Exper. 41(1), 23–50 (2011). https://doi.org/10.1002/spe.995

    Article  Google Scholar 

  19. Pllana, S., Fahringer, T.: Performance prophet: a performance modeling and prediction tool for parallel and distributed programs. In: 2005 International Conference on Parallel Processing Workshops (ICPPW 2005), Oslo, Norway, 2005, pp. 509–516 (2005). https://doi.org/10.1109/ICPPW.2005.72

  20. Samandi, V., Tiňo, P., Bahsoon, R.: Duplication scheduling with bottom-up top-down recursive neural network. In: Yin, H., Camacho, D., Tino, P. (eds.) Intelligent Data Engineering and Automated Learning - IDEAL 2022, IDEAL 2022. Lecture Notes in Computer Science, vol. 13756, pp. 170–178. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-21753-1_17

    Chapter  Google Scholar 

  21. Sonmez, O., Yigitbasi, N., Abrishami, S., Iosup, A., Epema, D.: Performance analysis of dynamic workflow scheduling in multicluster grids. In: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, ACM, pp. 49–60 (2010). https://doi.org/10.1145/1851476.1851483

  22. Sperduti, A., Starita, A.: Supervised neural networks for the classification of structures. IEEE Trans. Neural Netw. 8(3), 714–735 (1997). https://doi.org/10.1109/72.572108

    Article  Google Scholar 

  23. Ma, X., Xu, H., Gao, H., Bian, M.: Real-time multiple-workflow scheduling in cloud environments. IEEE Trans. Netw. Serv. Manage. 18(4), 4002–4018 (2021). https://doi.org/10.1109/TNSM.2021.3125395

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science, University of Birmingham, Birmingham, B15 2TT, UK

    Vahab Samandi, Peter Tiňo & Rami Bahsoon

Authors
  1. Vahab Samandi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peter Tiňo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rami Bahsoon
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vahab Samandi .

Editor information

Editors and Affiliations

  1. University of Deusto, Bilbao, Spain

    Pablo García Bringas

  2. University of Leon, León, Spain

    Hilde Pérez García

  3. University of La Rioja, Logroño, La Rioja, Spain

    Francisco Javier Martínez de Pisón

  4. Pablo de Olavide University, Seville, Spain

    Francisco Martínez Álvarez

  5. Pablo de Olavide University, Seville, Spain

    Alicia Troncoso Lora

  6. University of Burgos, Burgos, Spain

    Álvaro Herrero

  7. University of A Coruña, Ferrol - Coruña, Spain

    José Luis Calvo Rolle

  8. University of A Coruña, Ferrol - Coruña, Spain

    Héctor Quintián

  9. University of Salamanca, Salamanca, Spain

    Emilio Corchado

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Samandi, V., Tiňo, P., Bahsoon, R. (2023). Real-Time Workflow Scheduling in Cloud with Recursive Neural Network and List Scheduling. In: García Bringas, P., et al. Hybrid Artificial Intelligent Systems. HAIS 2023. Lecture Notes in Computer Science(), vol 14001. Springer, Cham. https://doi.org/10.1007/978-3-031-40725-3_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-40725-3_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-40724-6

  • Online ISBN: 978-3-031-40725-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation