Abstract
As distributed systems dramatically grow in terms of scale, complexity, and usage, understanding the hidden interactions among system and workload properties becomes an exceedingly difficult task. Machine learning models for prediction of system behavior (and analysis) are increasingly popular but their effectiveness in answering what and why is not always the most favorable. In this talk I will present two reliability analysis studies from two large, distributed systems: one that looks into GPGPU error prediction at the Titan, a large scale high-performance-computing system at ORNL, and one that analyzes the failure characteristics of solid state drives at a Google data center and hard disk drives at the Backblaze data center. Both studies illustrate the difficulty of untangling complex interactions of workload characteristics that lead to failures and of identifying failure root causes from monitored symptoms. Nevertheless, this difficulty can occasionally manifest in spectacular results where failure prediction can be dramatically accurate.
The work was partially supported by NSF grants CCF-1649087, CCF-1717532, and IIS-1838022. The work presented here was done in collaboration with J. Alter, L. Yang, B. Nie, J. Xue, R. Pinciroli, D. Tiwari, A. Jog, A. Dimnaku, R. Birke, and L. Chen.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Top500 Supercomputer Sites, November 2018. https://www.top500.org/lists/2018/11/
Alter, J., Xue, J., Dimnaku, A., Smirni, E.: SSD failures in the field: symptoms, causes, and prediction models. In: Taufer, M., Balaji, P., Peña, A.J. (eds.) Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2019, Denver, Colorado, USA, 17–19 November 2019, pp. 75:1–75:14. ACM (2019). https://doi.org/10.1145/3295500.3356172
Backblaze: Hard drive data and stats. https://www.backblaze.com/b2/hard-drive-test-data.html. Accessed 28 Apr 2020
Birke, R., Björkqvist, M., Chen, L.Y., Smirni, E., Engbersen, T.: (Big)data in a virtualized world: volume, velocity, and variety in cloud datacenters. In: Schroeder, B., Thereska, E. (eds.) Proceedings of the 12th USENIX conference on File and Storage Technologies, FAST 2014, Santa Clara, CA, USA, 17–20 February 2014, pp. 177–189. USENIX (2014). https://www.usenix.org/conference/fast14/technical-sessions/presentation/birke
Nie, B., Jog, A., Smirni, E.: Characterizing accuracy-aware resilience of GPGPU applications. In: 20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing, CCGRID 2020, Melbourne, Australia, 11–14 May 2020, pp. 111–120. IEEE (2020). https://doi.org/10.1109/CCGrid49817.2020.00-82
Nie, B., Tiwari, D., Gupta, S., Smirni, E., Rogers, J.H.: A large-scale study of soft-errors on GPUs in the field. In: 2016 IEEE International Symposium on High Performance Computer Architecture, HPCA 2016, Barcelona, Spain, 12–16 March 2016, pp. 519–530. IEEE Computer Society (2016). https://doi.org/10.1109/HPCA.2016.7446091
Nie, B., Xue, J., Gupta, S., Engelmann, C., Smirni, E., Tiwari, D.: Characterizing temperature, power, and soft-error behaviors in data center systems: insights, challenges, and opportunities. In: 25th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems, MASCOTS 2017, Banff, AB, Canada, 20–22 September 2017, pp. 22–31. IEEE Computer Society (2017). https://doi.org/10.1109/MASCOTS.2017.12
Nie, B., et al.: Machine learning models for GPU error prediction in a large scale HPC system. In: 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018, Luxembourg City, Luxembourg, 25–28 June 2018, pp. 95–106. IEEE Computer Society (2018). https://doi.org/10.1109/DSN.2018.00022
Nie, B., Yang, L., Jog, A., Smirni, E.: Fault site pruning for practical reliability analysis of GPGPU applications. In: 51st Annual IEEE/ACM International Symposium on Microarchitecture, MICRO 2018, Fukuoka, Japan, 20–24 October 2018, pp. 749–761. IEEE Computer Society (2018). https://doi.org/10.1109/MICRO.2018.00066
Pinciroli, R., Yang, L., Alter, J., Smirni, E.: The life and death of SSDs and HDDs: Similarities, differences, and prediction models, pp. 1–14 (2020). (under submission)
Xue, J., Birke, R., Chen, L.Y., Smirni, E.: Managing data center tickets: prediction and active sizing. In: 46th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2016, Toulouse, France, 28 June–1 July 2016, pp. 335–346. IEEE Computer Society (2016). https://doi.org/10.1109/DSN.2016.38
Xue, J., Birke, R., Chen, L.Y., Smirni, E.: Tale of tails: anomaly avoidance in data centers. In: 35th IEEE Symposium on Reliable Distributed Systems, SRDS 2016, Budapest, Hungary, 26–29 September 2016, pp. 91–100. IEEE Computer Society (2016). https://doi.org/10.1109/SRDS.2016.021
Xue, J., Birke, R., Chen, L.Y., Smirni, E.: Spatial-temporal prediction models for active ticket managing in data centers. IEEE Trans. Netw. Serv. Manag. 15(1), 39–52 (2018). https://doi.org/10.1109/TNSM.2018.2794409
Xue, J., Nie, B., Smirni, E.: Fill-in the gaps: spatial-temporal models for missing data. In: 13th International Conference on Network and Service Management, CNSM 2017, Tokyo, Japan, 26–30 November 2017, pp. 1–9. IEEE Computer Society (2017). https://doi.org/10.23919/CNSM.2017.8255983
Xue, J., Yan, F., Birke, R., Chen, L.Y., Scherer, T., Smirni, E.: PRACTISE: robust prediction of data center time series. In: Tortonesi, M., Schönwälder, J., Madeira, E.R.M., Schmitt, C., Serrat, J. (eds.) 11th International Conference on Network and Service Management, CNSM 2015, Barcelona, Spain, 9–13 November 2015, pp. 126–134. IEEE Computer Society (2015). https://doi.org/10.1109/CNSM.2015.7367348
Xue, J., Yan, F., Riska, A., Smirni, E.: Scheduling data analytics work with performance guarantees: queuing and machine learning models in synergy. Clust. Comput. 19(2), 849–864 (2016). https://doi.org/10.1007/s10586-016-0563-z
Yang, L., Nie, B., Jog, A., Smirni, E.: Practical resilience analysis of GPGPU applications in the presence of single- and multi-bit faults. IEEE Trans. Comput. (2020). https://doi.org/10.1109/TC.2020.2980541
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Smirni, E. (2020). Machine Learning for Reliability Analysis of Large Scale Systems. In: Gribaudo, M., Jansen, D.N., Remke, A. (eds) Quantitative Evaluation of Systems. QEST 2020. Lecture Notes in Computer Science(), vol 12289. Springer, Cham. https://doi.org/10.1007/978-3-030-59854-9_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-59854-9_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-59853-2
Online ISBN: 978-3-030-59854-9
eBook Packages: Computer ScienceComputer Science (R0)