Abstract
Understanding HPC facilities users’ behaviors and how computational resources are requested and utilized is not only crucial for the cluster productivity but also essential for designing and constructing future exascale HPC systems.
This paper tackles Challenge 4, ‘Analyzing Resource Utilization and User Behavior on Titan Supercomputer’, of the 2021 Smoky Mountains Conference Data Challenge. Specifically, we dig deeper inside the records of Titan to discover patterns and extract relationships.
This paper explores the workload distribution and usage patterns from resource manager system logs, GPU traces, and scientific areas information collected from the Titan supercomputer. Furthermore, we want to know how resource utilization and user behaviors change over time.
Using data science methods, such as correlations, clustering, or neural networks, our findings allow us to investigate how projects, jobs, nodes, GPUs and memory are related. We provide insights about seasonality usage of resources and a predictive model for forecasting utilization of Titan Supercomputer. In addition, the described methodology can be easily adopted in other HPC clusters.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
References
Patel, T., Liu, Z., Kettimuthu, R., Rich, P., Allcock, W., Tiwari, D.: Job characteristics on large-scale systems: long-term analysis, quantification, and implications. In: SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–17 (2020)
Dash, S., Paul, A.K., Wang, F., Oral, S., Technology Integration, SMC Data Challenge 2021: Analyzing Resource Utilization and User Behavior on Titan Supercomputer (2021). https://smc-datachallenge.ornl.gov/wp-content/uploads/2021/05/C4-SMC_DataChallenge_2021.pdf
Top500 the list. https://www.top500.org. Accessed 04 Aug 2021
Oak Ridge National Laboratory, ORNL Debuts Titan Supercomputer (2012). https://www.olcf.ornl.gov/wp-content/themes/olcf/titan/Titan_Debuts.pdf
Wang, F., Oral, S., Sen, S., Imam, N.: Learning from five-year resource-utilization data of Titan system. In: Proceedings - IEEE International Conference on Cluster Computing, ICCC 2019, September (2019). https://doi.org/10.1109/CLUSTER.2019.8891001
Ostrouchov, G., Maxwell, D., Ashraf, R.A., Engelmann, C., Shankar, M., Rogers, J.H.: GPU lifetimes on Titan supercomputer: survival analysis and reliability. In: International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, November (2020)
Jin, X., Han, J.: K-means clustering. In: Sammut, C., Webb, G.I. (eds.) Encyclopedia of Machine Learning, pp. 563–564. Springer, Boston (2010). https://doi.org/10.1007/978-0-387-30164-8_425
Cleveland, R.B., Cleveland, W.S., McRae, J.E., Terpenning, I.: STL: a seasonal-trend decomposition. J. Off. Stat. 6(1), 3–73 (1990)
Kingma, D.P., Ba, J.L.: Adam: a method for stochastic optimization. In: 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings, pp. 1–15 (2015). arXiv:1412.6980
Acknowledgements
S. Iserte was supported by the postdoctoral fellowship APOSTD/2020/026 from Valencian Region Government and European Social FundsFootnote 2. The study on Cori supercomputer was carried out during an internship funded under HiPEAC Collaboration Grant H2020-ICT-2017-779656Footnote 3. Finally, the author wants to thank the anonymous reviewers whose suggestions significantly improved the quality of this manuscript.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Iserte, S. (2022). An Study on the Resource Utilization and User Behavior on Titan Supercomputer. In: Nichols, J., et al. Driving Scientific and Engineering Discoveries Through the Integration of Experiment, Big Data, and Modeling and Simulation. SMC 2021. Communications in Computer and Information Science, vol 1512. Springer, Cham. https://doi.org/10.1007/978-3-030-96498-6_23
Download citation
DOI: https://doi.org/10.1007/978-3-030-96498-6_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-96497-9
Online ISBN: 978-3-030-96498-6
eBook Packages: Computer ScienceComputer Science (R0)