Abstract
After a seven-year long-term development process, the supercomputer Fugaku was officially launched as the successor to the K computer in March 2021. During this development process, we upgraded various system components and the data center infrastructure for official service in Fugaku. It was also necessary to upgrade the K computer operational data collection/monitoring platform for use in Fugaku. As a result, we are now in the process of developing and deploying an operational data collection/monitoring platform based on a three-tier pipeline architecture. In the first stage, the HPC system produces various types of log/metric data that are used to identify and monitor troubleshooting issues. Additionally, several thousand sensors operated by the building management system (BMS) generate metrics for power supply and cooling equipment. In the second stage, we aggregate the data into time-series databases and then visualize the results via a dashboard in the third stage. The dashboard provides an interactive interface for multiple data of the HPC system and data center infrastructure. During the course of this project, we resolved some issues found in the previous K computer platform. By using the redundant cores of the A64FX to allocate agents, it was determined that the new platform takes less than 20 s to collect metrics from over 150k compute nodes and finally write them to persistent storage. This paper introduces the design of the system architecture and reports on the current state of the platform renewal project, and provides overviews of two use cases encountered during the prelaunch service period.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Elastic. github.com/elastic
Grafana. github.com/grafana/grafana
Kibana. github.com/elastic/kibana
logstash. github.com/elastic/logstash
Project Jupyter. github.com/jupyter
Prometheus. github.com/prometheus
redash. github.com/getredash/redash
Top500. www.top500.org/system/179807/
Wikipedia. en.wikipedia.org/wiki/Fugaku(supercomputer)
Bates, N., Hsu, C., Imam, N., Wilde, T., Sartor, D.: Re-examining HPC energy efficiency dashboard elements. In: 2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp. 1106–1109 (2016)
Bautista, E., Romanus, M., Davis, T., Whitney, C., Kubaska, T.: Collecting, monitoring, and analyzing facility and systems data at the national energy research scientific computing center. In: Proceedings of the 48th International Conference on Parallel Processing: Workshops. ICPP 2019, Association for Computing Machinery (2019)
Bourassa, N., et al.: Operational data analytics: optimizing the national energy research scientific computing center cooling systems. In: Proceedings of the 48th International Conference on Parallel Processing: Workshops. ICPP 2019, Association for Computing Machinery (2019)
Chen, J., Tan, R., Xing, G., Wang, X.: Ptec: a system for predictive thermal and energy control in data centers. In: 2014 IEEE Real-Time Systems Symposiumm, pp. 218–227 (2014)
Fujitsu: A64fx microarchitecture manual. github.com/fujitsu/A64FX
Matsuda, M., Matsuba, H., Nonaka, J., Yamamoto, K., Shibata, H., Tsukamoto, T.: Modeling the existing cooling system to learn its behavior for post-k supercomputer at riken r-ccs. In: Proceedings of the 48th International Conference on Parallel Processing: Workshops. ICPP 2019, Association for Computing Machinery (2019)
Minet, P., Renault, E., Khoufi, I., Boumerdassi, S.: Analyzing traces from a Google Data Center. In: 2018 14th International Wireless Communications Mobile Computing Conference (IWCMC), pp. 1167–1172 (2018). https://doi.org/10.1109/IWCMC.2018.8450304
Netti, A., et al.: Dcdb wintermute: enabling online and holistic operational data analytics on HPC systems. In: Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing (2020)
Nonaka, J., Hanawa, T., Shoji, F.: Analysis of cooling water temperature impact on computing performance and energy consumption. In: 2020 IEEE International Conference on Cluster Computing (CLUSTER), pp. 169–175 (2020)
Nonaka, J., Yamamoto, K., Kuroda, A., Tsukamoto, T., Koiso, K., Sakamoto, N.: A view from the facility operations side on the water/air cooling system of the k computer (2019). sc19.supercomputing.org/proceedings/tech/poster/tech/poster/pages/rpost246.html
Okazaki, R., et al.: Supercomputer fugaku CPU A64FX realizing high performance, high-density packaging, and low power consumption. Tech. Rep. Fujitsu Tech. Rev. (2020)
Ott, M., et al.: Global experiences with HPC operational data measurement, collection and analysis. In: 2020 IEEE International Conference on Cluster Computing (CLUSTER), pp. 499–508 (2020)
Santos, D., Mataloto, B., Ferreira, J.C.: Data center environment monitoring system. In: CCIOT 2019: Proceedings of the 2019 4th International Conference on Cloud Computing and Internet of Things, pp. 75–81. CCIOT 2019, Association for Computing Machinery (2019)
Sartor, D., Mahdavi, R., Radhakrishnan, B.D., Bates, N., Bailey, A.M., Wescott, R.: General recommendations for high performance computing data center energy management dashboard display. In: 2013 IEEE International Symposium on Parallel Distributed Processing, Workshops and Phd Forum, pp. 892–898 (2013)
(SRCC), S.R.C.C.: HPC dashboards. github.com/stanford-rc/hpc-dashboards
Terai, M., Shoji, F., Tsukamoto, T., Yamochi, Y.: A study of operational impact on power usage effectiveness using facility metrics and server operation logs in the K computer. In: 2020 IEEE International Conference on Cluster Computing (CLUSTER), pp. 509–513 (2020)
Terai, M., Tsukamoto, T., Shoji, F.: Study on the facility enhancement by operational data analysis: a comparison of the operations in the K computer and fugaku. In: ISC 2021 Digital Research Poster (2021)
Yamamoto, K.: Operational data processing pipeline. In: SC19 BoF: Operational Data Analytics (2019). eehpcwg.llnl.gov/assets/sc19/bof/operational/data/processing/pipeline.pdf
Acknowledgements
We would like to thank colleagues, especially Dr. Toshiyuki Tsukamoto (RIKEN R-CCS), who provided excellent advice and abundant materials regarding facility operations, and Dr. Yuichi Tsujita (RIKEN R-CCS) for valuable comments based on his expertise. We thank Mr. Kensuke Matsumoto (Fujitsu Limited), Mr. Naoki Ikeda (Fujitsu Limited), Mr. Yoshitaka Furutani (Fujitsu Limited), and Mr. Nobuo Ohgushi (Fujitsu Limited) for technical support and comments during the course of this project.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Terai, M., Yamamoto, K., Miura, S., Shoji, F. (2021). An Operational Data Collecting and Monitoring Platform for Fugaku: System Overviews and Case Studies in the Prelaunch Service Period. In: Jagode, H., Anzt, H., Ltaief, H., Luszczek, P. (eds) High Performance Computing. ISC High Performance 2021. Lecture Notes in Computer Science(), vol 12761. Springer, Cham. https://doi.org/10.1007/978-3-030-90539-2_24
Download citation
DOI: https://doi.org/10.1007/978-3-030-90539-2_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-90538-5
Online ISBN: 978-3-030-90539-2
eBook Packages: Computer ScienceComputer Science (R0)