Skip to main content
SpringerLink
Log in

An AutoML Based Algorithm for Performance Prediction in HPC Systems

  • Conference paper
  • First Online:
Parallel and Distributed Computing, Applications and Technologies (PDCAT 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13798))

  • 548 Accesses

Abstract

Neural networks are extensively utilized for building performance prediction models for high-performance computing systems. It is challenging to construct the neural network architecture that provides accurate performance (runtime) predictions by exploring the number of layers and neurons in each layer. Automated machine learning (AutoML) using neural architecture search (NAS) frameworks such as Auto-Keras have been proposed to tune neural networks automatically. Researchers have utilized AutoML frameworks for many application domains to build neural networks automatically, providing accurate predictions. However, AutoML has not been explored for performance prediction domain based on our literature survey. Hence, our goal is to show the feasibility of AutoML-based algorithm for performance prediction of HPC systems. In this paper, we propose a novel AutoML based algorithm that builds a neural network model layer-by-layer for performance prediction. Our algorithm takes decisions based on prediction accuracy metric values to add a new layer to tune the network. We have performed extensive experiments utilizing applications from SPEC CPU 2006, SPEC CPU 2017 benchmark suites. Performance prediction accuracy results of RMSE (Root Mean Squared Error), MedAE (Median Absolute Error) and MedAPE (Median Absolute Percentage Error) from our experiments demonstrate superior performance of our algorithm compared to the state-of-the-art Auto-Keras framework.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Agarwal, R., Chopra, S., Christophides, V., Georgantas, N., Issarny, V.: Detecting mobile crowdsensing context in the wild. In: Proceedings - IEEE International Conference on Mobile Data Management, June 2019, pp. 170–175. Institute of Electrical and Electronics Engineers Inc. (2019). https://doi.org/10.1109/MDM.2019.00-60

  2. Chen, W., Dong, X., Chen, H., Wang, Q., Yu, X., Zhang, X.: Performance evaluation of convolutional neural network on tianhe-3 prototype. J. Supercomput. 77(11), 12647–12665 (2021). https://doi.org/10.1007/S11227-021-03759-8/FIGURES/10, https://link.springer.com/article/10.1007/s11227-021-03759-8

  3. Chollet, F., et al.: Keras. https://keras.io (2015)

  4. Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: a survey. J. Mach. Learn. Res. 20, 1–21 (2019). https://doi.org/10.5555/3322706.3361996, http://jmlr.org/papers/v20/18-598.html

  5. Erickson, N., et al.: AutoGluon-tabular: robust and accurate AutoML for structured data (2020). https://arxiv.org/abs/2003.06505v1

  6. Ferreira, L., Pilastri, A., Martins, C.M., Pires, P.M., Cortez, P.: A comparison of automl tools for machine learning, deep learning and XGBoost. In: Proceedings of the International Joint Conference on Neural Networks, July 2021. Institute of Electrical and Electronics Engineers Inc. (2021). https://doi.org/10.1109/IJCNN52387.2021.9534091

  7. Feurer, M., Eggensperger, K., Falkner, S., Lindauer, M., Hutter, F.: Auto-sklearn 2.0: hands-free AutoML via meta-learning (2020). https://arxiv.org/abs/2007.04074v2

  8. García-Domínguez, M., Domínguez, C., Heras, J., Mata, E., Pascual, V.: UFOD: an AutoML framework for the construction, comparison, and combination of object detection models. Pattern Recogn. Lett. 145, 135–140 (2021). https://doi.org/10.1016/J.PATREC.2021.01.022

    Article  Google Scholar 

  9. Gupta, G., Katarya, R.: EnPSO: an AutoML technique for generating ensemble recommender system. Arab. J. Sci. Eng. 46(9), 8677–8695 (2021). https://doi.org/10.1007/S13369-021-05670-Z/FIGURES/8, https://link.springer.com/article/10.1007/s13369-021-05670-z

  10. Halvari, T., Nurminen, J.K., Mikkonen, T.: Robustness of AutoML for time series forecasting in sensor networks. In: 2021 IFIP Networking Conference, IFIP Networking 2021. Institute of Electrical and Electronics Engineers Inc. (2021). https://doi.org/10.23919/IFIPNETWORKING52078.2021.9472199

  11. Jin, H., Song, Q., Hu, X.: Auto-Keras: an efficient neural architecture search system. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, New York, NY, USA (2019). https://doi.org/10.1145/3292500, https://doi.org/10.1145/3292500.3330648

  12. Kulkarni, G.N., Ambesange, S., Vijayalaxmi, A., Sahoo, A.: Comparision of diabetic prediction AutoML model with customized model. In: Proceedings - International Conference on Artificial Intelligence and Smart Systems, ICAIS 2021, pp. 842–847. Institute of Electrical and Electronics Engineers Inc. (2021). https://doi.org/10.1109/ICAIS50930.2021.9395775

  13. Kwok, T.Y., Yeung, D.Y.: Constructive algorithms for structure learning in feedforward neural networks for regression problems. IEEE Trans. Neural Netw. 8(3), 630–645 (1997). https://doi.org/10.1109/72.572102

    Article  Google Scholar 

  14. Lee, M., Ahn, H., Hong, C.H., Nikolopoulos, D.S.: gShare: a centralized GPU memory management framework to enable GPU memory sharing for containers. Future Gener. Comput. Syst. 130, 181–192 (2022). https://doi.org/10.1016/J.FUTURE.2021.12.016, https://linkinghub.elsevier.com/retrieve/pii/S0167739X21004970

  15. Liu, D., et al.: AutoGenome: an AutoML tool for genomic research. Artif. Intell. Life Sci. 1, 100017 (2021). https://doi.org/10.1016/J.AILSCI.2021.100017

    Article  Google Scholar 

  16. Lopez, L., Guynn, M., Lu, M.: Predicting computer performance based on hardware configuration using multiple neural networks. In: Proceedings - 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018, pp. 824–827. Institute of Electrical and Electronics Engineers Inc. (2019). https://doi.org/10.1109/ICMLA.2018.00132

  17. Luo, X., Liu, D., Huai, S., Kong, H., Chen, H., Liu, W.: Designing efficient DNNs via hardware-aware neural architecture search and beyond. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. (2021). https://doi.org/10.1109/TCAD.2021.3100249

    Article  Google Scholar 

  18. Malakar, P., Balaprakash, P., Vishwanath, V., Morozov, V., Kumaran, K.: Benchmarking machine learning methods for performance modeling of scientific applications. In: Proceedings of PMBS 2018: Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 33–44. Institute of Electrical and Electronics Engineers Inc., Dallas, Texas, USA (2019). https://doi.org/10.1109/PMBS.2018.8641686

  19. Mankodi, A., Bhatt, A., Chaudhury, B.: Evaluation of neural network models for performance prediction of scientific applications. In: IEEE Region 10th Annual International Conference, Proceedings/TENCON. November 2020, pp. 426–431. Institute of Electrical and Electronics Engineers Inc. (2020). https://doi.org/10.1109/TENCON50793.2020.9293788

  20. Mariani, G., Anghel, A., Jongerius, R., Dittmann, G.: Predicting cloud performance for HPC applications before deployment. Futur. Gener. Comput. Syst. 87, 618–628 (2018). https://doi.org/10.1016/j.future.2017.10.048

    Article  Google Scholar 

  21. Nikitin, N., et al.: Automated evolutionary approach for the design of composite machine learning pipelines. Futur. Gener. Comput. Syst. 127, 109–125 (2022). https://doi.org/10.1016/J.FUTURE.2021.08.022

    Article  Google Scholar 

  22. Pedregosa, F., et al.: Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825–2830 (2011). http://scikit-learn.sourceforge.net

  23. Rakhshani, H., et al.: Automated machine learning for information retrieval in scientific articles. In: 2020 IEEE Congress on Evolutionary Computation, CEC 2020 - Conference Proceedings. Institute of Electrical and Electronics Engineers Inc. (2020). https://doi.org/10.1109/CEC48606.2020.9185893

  24. Ren, P., et al.: A comprehensive survey of neural architecture search. ACM Comput. Surv. (CSUR), 54(4), 76 (2021). https://doi.org/10.1145/3447582

  25. Srivastava, A., Zhang, N., Kannan, R., Prasanna, V.K.: Towards high performance, portability, and productivity: lightweight augmented neural networks for performance prediction. In: Proceedings - 2020 IEEE 27th International Conference on High Performance Computing, Data, and Analytics, HiPC 2020, pp. 21–30. Institute of Electrical and Electronics Engineers Inc. (2020). https://doi.org/10.1109/HIPC50609.2020.00016

  26. Sun, J., Sun, G., Zhan, S., Zhang, J., Chen, Y.: Automated performance modeling of HPC applications using machine learning. IEEE Trans. Comput. 69(5), 749–763 (2020). https://doi.org/10.1109/TC.2020.2964767

    Article  MATH  Google Scholar 

  27. Wang, K., Guo, P.: A robust automated machine learning system with pseudoinverse learning. Cogn. Comput. 13(3), 724–735 (2021). https://doi.org/10.1007/S12559-021-09853-6, https://link.springer.com/article/10.1007/s12559-021-09853-6

  28. Wang, K., Liu, Z., Lin, Y., Lin, J., Han, S.: Hardware-centric AutoML for mixed-precision quantization. Int. J. Comput. Vis. 128(8–9), 2035–2048 (2020). https://doi.org/10.1007/S11263-020-01339-6/FIGURES/11, https://link.springer.com/article/10.1007/s11263-020-01339-6

  29. Wang, C.C., Liao, Y.C., Kao, M.C., Liang, W.Y., Hung, S.H.: Toward accurate platform-aware performance modeling for deep neural networks. ACM SIGAPP Appl. Comput. Rev. 21(1), 50–61 (2021). https://doi.org/10.1145/3477133.3477137, https://dl.acm.org/doi/abs/10.1145/3477133.3477137

  30. Yang, J., Shi, R., Ni, B.: MedMNIST classification decathlon: a lightweight automl benchmark for medical image analysis. In: Proceedings - International Symposium on Biomedical Imaging, April 2021, pp. 191–195. IEEE Computer Society (2021). https://doi.org/10.1109/ISBI48211.2021.9434062

  31. Zhang, L., et al.: AutoGGN: a gene graph network AutoML tool for multi-omics research. Artif. Intell. Life Sci. 1, 100019 (2021). https://doi.org/10.1016/J.AILSCI.2021.100019

    Article  Google Scholar 

  32. Zimmer, L., Lindauer, M., Hutter, F.: Auto-pytorch tabular: multi-fidelity metalearning for efficient and robust AutoDL. IEEE Trans. Pattern Anal. Mach. Intell. 43(9), 3079–3090 (2020). https://doi.org/10.1109/TPAMI.2021.3067763, https://arxiv.org/abs/2006.13799v3

  33. Zöller, M.A., Huber, M.F.: Benchmark and survey of automated machine learning frameworks. J. Artif. Intell. Res. 70, 409–472 (2021). https://doi.org/10.1613/JAIR.1.11854, https://dl.acm.org/doi/abs/10.1613/jair.1.11854

Download references

Author information

Authors and Affiliations

  1. Dhirubhai Ambani Institute of Information and Communication Technology, Gandhinagar, India

    Amit Mankodi, Amit Bhatt & Bhaskar Chaudhury

Authors
  1. Amit Mankodi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Amit Bhatt
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bhaskar Chaudhury
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Amit Mankodi .

Editor information

Editors and Affiliations

  1. Tohoku University, Aoba-ku, Japan

    Hiroyuki Takizawa

  2. Sun Yat-sen University, Guangzhou, China

    Hong Shen

  3. The University of Tokyo, Tokyo, Japan

    Toshihiro Hanawa

  4. Seoul National University of Science and Technology, Seoul, Korea (Republic of)

    Jong Hyuk Park

  5. Griffith University, Queensland, QLD, Australia

    Hui Tian

  6. Tokyo Denki University, Tokyo, Japan

    Ryusuke Egawa

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Mankodi, A., Bhatt, A., Chaudhury, B. (2023). An AutoML Based Algorithm for Performance Prediction in HPC Systems. In: Takizawa, H., Shen, H., Hanawa, T., Hyuk Park, J., Tian, H., Egawa, R. (eds) Parallel and Distributed Computing, Applications and Technologies. PDCAT 2022. Lecture Notes in Computer Science, vol 13798. Springer, Cham. https://doi.org/10.1007/978-3-031-29927-8_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-29927-8_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-29926-1

  • Online ISBN: 978-3-031-29927-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation