Abstract
The objective of this study is to predict the wait time in job schedulers with high accuracy. Job executions in supercomputers or data centers are typically managed by job schedulers to efficiently utilize computing resources. A possible disadvantage is that, depending on resource availability and scheduling policy, the job waits for a long time before being executed. Therefore, providing the predicted wait time for individual jobs can contribute to the users’ research planning. Additionally, the job wait time potentially becomes an important input for the scheduling policy. However, the prediction of the job wait time is a challenging task because the state of the scheduling system changes dynamically by many uncertainty factors. To address this problem, a graph neural network architecture of deep learning, which is a novel approach for processing job information in the scheduler, was employed in this study. Our experiments using real historical logs confirmed that the proposed deep learning model achieved 0.3–7.9% higher prediction accuracy compared to the boosted decision tree and multi-layer perceptron models. An extensive analysis of the proposed deep learning model was performed to improve the explainability of the experimental results. In particular, the visualization of attention weights in the graph neural network expanded our understanding of the behavior of the proposed deep learning model.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Catalina scheduler. https://www.sdsc.edu/catalina/. Accessed 30 Nov 2022
Cobalt scheduler. https://trac.mcs.anl.gov/projects/cobalt/. Accessed 30 Nov 2022
Deepbatch. https://github.com/ktomoe/deepbatch/. Accessed 16 Dec 2022
Maui scheduler. http://docs.adaptivecomputing.com/maui/. Accessed 30 Nov 2022
Parallel workloads archive. https://www.cs.huji.ac.il/labs/parallel/workload/index.html. Accessed 30 Nov 2022
Battaglia, P.W., et al.: Relational inductive biases, deep learning, and graph networks (2018). https://doi.org/10.48550/ARXIV.1806.01261, https://arxiv.org/abs/1806.01261
Bos, K., et al.: LHC computing grid: technical design report. version 1.06 (20 Jun 2005), Technical design report, LCG, CERN, Geneva (2005). http://cds.cern.ch/record/840543
Brody, S., Alon, U., Yahav, E.: How Attentive are Graph Attention Networks? In: International Conference on Learning Representations (2022). https://openreview.net/forum?id=F72ximsx7C1
Brown, N., Gibb, G., Belikov, E., Nash, R.: Predicting batch queue job wait times for informed scheduling of urgent HPC workloads (2022). https://doi.org/10.48550/ARXIV.2204.13543, https://arxiv.org/abs/2204.13543
Chen, T., Guestrin, C.: XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785–794. KDD 2016, ACM, New York (2016). https://doi.org/10.1145/2939672.2939785
Feitelson, D.G., Tsafrir, D., Krakov, D.: Experience with using the parallel workloads archive. J. Parallel Distrib. Comput. 74(10), 2967–2982 (2014). https://doi.org/10.1016/j.jpdc.2014.06.013. www.sciencedirect.com/science/article/pii/S0743731514001154
Gombert, L., Suter, F.: Learning-based approaches to estimate job wait time in HTC datacenters. In: Klusáček, D., Cirne, W., Rodrigo, G. (eds.) 24th Workshop on Job Scheduling Strategies for Parallel Processing. Job Scheduling Strategies for Parallel Processing. JSSPP 2021, vol. 12985, pp. 101–125. Portland, United States (2021). https://doi.org/10.1007/978-3-030-88224-2_6, https://hal.archives-ouvertes.fr/hal-03357129
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition (2015). https://doi.org/10.48550/ARXIV.1512.03385, https://arxiv.org/abs/1512.03385
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, p. 448–456. ICML 2015, JMLR.org (2015)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2017)
Klusáček, D., Chlumský, V.: Evaluating the impact of soft walltimes on job scheduling performance. In: Klusáček, D., Cirne, W., Desai, N. (eds.) Job Scheduling Strategies for Parallel Processing, pp. 15–38. Springer International Publishing, Cham (2019)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017). https://doi.org/10.1145/3065386
Li, H., Groep, D., Wolters, L.: Efficient response time predictions by exploiting application and resource state similarities. In: The 6th IEEE/ACM International Workshop on Grid Computing, p. 8 (2005). https://doi.org/10.1109/GRID.2005.1542747
Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol.32, pp. 8024–8035. Curran Associates, Inc. (2019). http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2014). https://doi.org/10.48550/ARXIV.1409.1556, https://arxiv.org/abs/1409.1556
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(56), 1929–1958 (2014). http://jmlr.org/papers/v15/srivastava14a.html
Thain, D., Tannenbaum, T., Livny, M.: Distributed computing in practice: the condor experience. Concurrency Pract. Experience 17(2–4), 323–356 (2005)
Tsafrir, D., Etsion, Y., Feitelson, D.G.: Backfilling using system-generated predictions rather than user runtime estimates. IEEE Trans. Parallel Distrib. Syst. 18(6), 789–803 (2007). https://doi.org/10.1109/TPDS.2007.70606
Vaswani, A., et al.: Attention is all you need. In: Guyon, I., (eds.) Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017). https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
Wang, M., et al.: Deep graph library: a graph-centric, highly-performant package for graph neural networks. arXiv preprint arXiv:1909.01315 (2019)
Wang, Q., Zhang, H., Qu, C., Shen, Y., Liu, X., Li, J.: RLSchert: an HPC job scheduler using deep reinforcement learning and remaining time prediction. Appl. Sci. 11(20), 9448 (2021). https://doi.org/10.3390/app11209448. https://www.mdpi.com/2076-3417/11/20/9448
Yoo, A.B., Jette, M.A., Grondona, M.: Slurm: simple Linux utility for resource management. In: Feitelson, D., Rudolph, L., Schwiegelshohn, U. (eds.) Job Scheduling Strategies for Parallel Processing, pp. 44–60. Springer, Berlin Heidelberg, Berlin, Heidelberg (2003)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kishimoto, T., Nakamura, T. (2023). An Efficient Approach Based on Graph Neural Networks for Predicting Wait Time in Job Schedulers. In: Klusáček, D., Corbalán, J., Rodrigo, G.P. (eds) Job Scheduling Strategies for Parallel Processing. JSSPP 2023. Lecture Notes in Computer Science, vol 14283. Springer, Cham. https://doi.org/10.1007/978-3-031-43943-8_7
Download citation
DOI: https://doi.org/10.1007/978-3-031-43943-8_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43942-1
Online ISBN: 978-3-031-43943-8
eBook Packages: Computer ScienceComputer Science (R0)