Abstract
MLPerf has emerged as a frontrunner in benchmarking AI performance by having support of main players in the industry. At the same time, official scores uncover challenges for measuring distributed AI performance: a 2/3 throughput loss at a large scale and longer number of epochs needed to reach the required accuracy. Furthermore, no distributed scored have been submitted for Tensorflow, the most popular AI framework. Our work investigates these issues and suggests ways for overcoming challenges facing benchmarking at scale. Focusing on Tensorflow, wee show how efficient distributed scores can be obtained with appropriate software and hardware choices. Results for various Lenovo servers and Nvidia GPUs (V100 and T4) are also presented. Finally, we examine the utility of MLPerf for evaluating scale-up hardware and propose augmenting the main MLPerf score by an additional score that takes into account computational efficiency. Several options for the score are explored and analyzed in detail.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
MLPerf. https://www.mlperf.org/
Bench Research: Deep Bench. https://github.com/baidu-research/DeepBench
Coleman, C.A., et al.: DAWNBench: an end-to-end deep learning benchmark and competition. In: Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017) (2017)
Bourrasset, C., et al.: Requirements for an enterprise AI benchmark. In: Nambiar, R., Poess, M. (eds.) TPCTC 2018. LNCS, vol. 11135, pp. 71–81. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11404-6_6
TensorFlow. https://tensorflow.org
Hodak, M., Dholakia, A.: Towards evaluation of tensorflow performance in a distributed compute environment. In: Nambiar, R., Poess, M. (eds.) TPCTC 2018. LNCS, vol. 11135, pp. 82–93. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11404-6_7
Tensorflow Benchmarks Issue # 247. https://github.com/tensorflow/benchmarks/issues/247
Hodak, M., Ellison, D., Seidel, P., Dholakia, A.: Performance implications of big data in scalable deep learning: on the importance on bandwidth and caching. In: 2018 IEEE International Conference on Big Data, pp. 1945–1950 (2018)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Hodak, M., Dholakia, A. (2020). Challenges in Distributed MLPerf. In: Nambiar, R., Poess, M. (eds) Performance Evaluation and Benchmarking for the Era of Cloud(s). TPCTC 2019. Lecture Notes in Computer Science(), vol 12257. Springer, Cham. https://doi.org/10.1007/978-3-030-55024-0_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-55024-0_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-55023-3
Online ISBN: 978-3-030-55024-0
eBook Packages: Computer ScienceComputer Science (R0)