Skip to main content

Machine Learning Based Parallel I/O Predictive Modeling: A Case Study on Lustre File Systems

  • Conference paper
  • First Online:
High Performance Computing (ISC High Performance 2018)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 10876))

Included in the following conference series:

  • 2378 Accesses

Abstract

Parallel I/O hardware and software infrastructure is a key contributor to performance variability for applications running on large-scale HPC systems. This variability confounds efforts to predict application performance for characterization, modeling, optimization, and job scheduling. We propose a modeling approach that improves predictive ability by explicitly treating the variability and by leveraging the sensitivity of application parameters on performance to group applications with similar characteristics. We develop a Gaussian process-based machine learning algorithm to model I/O performance and its variability as a function of application and file system characteristics. We demonstrate the effectiveness of the proposed approach using data collected from the Edison system at the National Energy Research Scientific Computing Center. The results show that the proposed sensitivity-based models are better at prediction when compared with application-partitioned or unpartitioned models. We highlight modeling techniques that are robust to the outliers that can occur in production parallel file systems. Using the developed metrics and modeling approach, we provide insights into the file system metrics that have a significant impact on I/O performance.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/jhammond/lltop.

References

  1. Barker, K.J., Davis, K., Kerbyson, D.J.: Performance modeling in action: performance prediction of a Cray XT4 system during upgrade. In: International Symposium on Parallel & Distributed Processing, pp. 1–8. IEEE (2009)

    Google Scholar 

  2. Behzad, B., Byna, S., Wild, S.M., Prabhat, M., Snir, M.: Improving parallel I/O autotuning with performance modeling. In: 23rd International Symposium on High-Performance Parallel and Distributed Computing, pp. 253–256. ACM (2014)

    Google Scholar 

  3. Betke, E., Kunkel, J.: Real-time I/O-monitoring of HPC applications with SIOX, elasticsearch, Grafana and FUSE. In: Kunkel, J.M., Yokota, R., Taufer, M., Shalf, J. (eds.) ISC High Performance 2017. LNCS, vol. 10524, pp. 174–186. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67630-2_15

    Chapter  Google Scholar 

  4. Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, New York (2006)

    MATH  Google Scholar 

  5. Cao, Z., Tarasov, V., Raman, H.P., Hildebrand, D., Zadok, E.: On the performance variation in modern storage stacks. In: FAST, pp. 329–344 (2017)

    Google Scholar 

  6. Chen, T., Guestrin, C.: XGBoost: a scalable tree boosting system. In: 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785–794. ACM (2016)

    Google Scholar 

  7. DOE-ASCR: storage systems and input/output to support extreme scale science. In: DOE Workshops on Storage Systems and Input/Output (2014)

    Google Scholar 

  8. Dorier, M., Antoniu, G., Cappello, F., Snir, M., Sisneros, R., Yildiz, O., Ibrahim, S., Peterka, T., Orf, L.: Damaris: addressing performance variability in data management for post-petascale simulations. ACM Trans. Parallel Comput. 3(3), 15:1–15:43 (2016)

    Article  Google Scholar 

  9. Dorier, M., Antoniu, G., Ross, R., Kimpe, D., Ibrahim, S.: CALCioM: mitigating I/O interference in HPC systems through cross-application coordination. In: 28th International Parallel and Distributed Processing Symposium, pp. 155–164. IEEE (2014)

    Google Scholar 

  10. Feroz, F., Hobson, M., Cameron, E., Pettitt, A.: Importance nested sampling and the MultiNest algorithm. arXiv preprint arXiv:1306.2144 (2013)

  11. Gelman, A., Carlin, J.B., Stern, H.S., Dunson, D.B., Vehtari, A., Rubin, D.B.: Bayesian Data Analysis, 2nd edn. CRC Press, Boca Raton (2014)

    MATH  Google Scholar 

  12. Geurts, P., Ernst, D., Wehenkel, L.: Extremely randomized trees. Mach. Learn. 63(1), 3–42 (2006)

    Article  Google Scholar 

  13. Geurts, P., Louppe, G.: Learning to rank with extremely randomized trees. In: JMLR: Workshop and Conference Proceedings, vol. 14, pp. 49–61 (2011)

    Google Scholar 

  14. Gulati, A., Merchant, A., Varman, P.J.: mClock: handling throughput variability for hypervisor IO scheduling. In: 9th USENIX Conference on Operating Systems Design and Implementation, pp. 437–450. USENIX Association (2010)

    Google Scholar 

  15. Habib, S., Morozov, V., Finkel, H., Pope, A., Heitmann, K., Kumaran, K., Peterka, T., Insley, J., Daniel, D., Fasel, P., et al.: The universe at extreme scale: multi-petaflop sky simulation on the BG/Q. In: International Conference on High Performance Computing, Networking, Storage and Analysis, p. 4. IEEE (2012)

    Google Scholar 

  16. Inacio, E.C., Barbetta, P.A., Dantas, M.A.: A statistical analysis of the performance variability of read/write operations on parallel file systems. Procedia Comput. Sci. 108, 2393–2397 (2017)

    Article  Google Scholar 

  17. Isaila, F., Balaprakash, P., Wild, S.M., Kimpe, D., Latham, R., Ross, R., Hovland, P.: Collective I/O tuning using analytical and machine learning models. In: International Conference on Cluster Computing, pp. 128–137. IEEE (2015)

    Google Scholar 

  18. Jordan, M.I., Ghahramani, Z., Jaakkola, T.S., Saul, L.K.: An introduction to variational methods for graphical models. Mach. Learn. 37(2), 183–233 (1999)

    Article  Google Scholar 

  19. Kunkel, J., Zimmer, M., Betke, E.: Predicting performance of non-contiguous I/O with machine learning. In: Kunkel, J.M., Ludwig, T. (eds.) ISC High Performance 2015. LNCS, vol. 9137, pp. 257–273. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-20119-1_19

    Chapter  Google Scholar 

  20. Kuo, C.S., Nomura, A., Matsuoka, S., Shah, A., Wolf, F., Zhukov, I.: Environment matters: how competition for I/O among applications degrades their performance. IPSJ SIG Technical report 2013-HPC-142(11), 1–7 (2013)

    Google Scholar 

  21. Lee, E.K., Katz, R.H.: An analytic performance model of disk arrays. In: ACM SIGMETRICS Performance Evaluation Review, vol. 21, pp. 98–109. ACM (1993)

    Article  Google Scholar 

  22. Lockwood, G.K., Snyder, S., Yoo, W., Harms, K., Nault, Z., Byna, S., Carns, P., Wright, N.J.: UMAMI: a recipe for generating meaningful metrics through holistic I/O performance analysis. In: 2nd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems (PDSW-DISCS 2017) (2017)

    Google Scholar 

  23. Lofstead, J., Zheng, F., Liu, Q., Klasky, S., Oldfield, R., Kordenbrock, T., Schwan, K., Wolf, M.: Managing variability in the IO performance of petascale storage systems. In: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–12. IEEE (2010)

    Google Scholar 

  24. Madireddy, S., Balaprakash, P., Carns, P., Latham, R., Ross, R., Snyder, S., Wild, S.M.: Analysis and correlation of application I/O performance and system-wide I/O activity. In: International Conference on Networking, Architecture, and Storage, pp. 1–10. IEEE (2017)

    Google Scholar 

  25. Mann, H.B., Whitney, D.R.: On a test of whether one of two random variables is stochastically larger than the other. Ann. Math. Stat. 18, 50–60 (1947)

    Article  MathSciNet  Google Scholar 

  26. van der Matthews, A.G.D.G., Wilk, M., Nickson, T., Fujii, K., Boukouvalas, A., León-Villagrá, P., Ghahramani, Z., Hensman, J.: GPflow: a gaussian process library using TensorFlow. J. Mach. Learn. Res. 18(40), 1–6 (2017)

    MathSciNet  MATH  Google Scholar 

  27. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)

    MathSciNet  MATH  Google Scholar 

  28. Snyder, S., Carns, P., Harms, K., Ross, R., Lockwood, G.K., Wright, N.J.: Modular HPC I/O characterization with Darshan. In: Workshop on Extreme-Scale Programming Tools (2016)

    Google Scholar 

  29. Son, S.W., Sehrish, S., Liao, W., Oldfield, R., Choudhary, A.: Reducing I/O variability using dynamic I/O path characterization in petascale storage systems. J. Supercomput. 73(5), 2069–2097 (2017)

    Article  Google Scholar 

  30. Stein, M.L.: Interpolation of Spatial Data: Some Theory for Kriging. Springer, New York (2012). https://doi.org/10.1007/978-1-4612-1494-6

    Book  Google Scholar 

  31. Xie, B., Huang, Y., Chase, J.S., Choi, J.Y., Klasky, S., Lofstead, J., Oral, S.: Predicting output performance of a petascale supercomputer. In: 26th International Symposium on High-Performance Parallel and Distributed Computing, pp. 181–192. ACM, New York (2017)

    Google Scholar 

  32. Yildiz, O., Dorier, M., Ibrahim, S., Ross, R., Antoniu, G.: On the root causes of cross-application I/O interference in HPC storage systems. In: International Parallel and Distributed Processing Symposium, pp. 750–759. IEEE (2016)

    Google Scholar 

Download references

Acknowledgment

This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, under Contract DE-AC02-06CH11357. This research used resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Sandeep Madireddy .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Madireddy, S. et al. (2018). Machine Learning Based Parallel I/O Predictive Modeling: A Case Study on Lustre File Systems. In: Yokota, R., Weiland, M., Keyes, D., Trinitis, C. (eds) High Performance Computing. ISC High Performance 2018. Lecture Notes in Computer Science(), vol 10876. Springer, Cham. https://doi.org/10.1007/978-3-319-92040-5_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-92040-5_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-92039-9

  • Online ISBN: 978-3-319-92040-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics