Accelerating Model Training: Performance Antipatterns Eliminator Framework
Pages 163 - 170
Abstract
In the realm of ML/DL training pipelines, the training-specific data preparation of complex models may consume up to 87% of the total training time. A data scientist may build training pipelines using Python data structures on GPU while being unaware of the performance antipatterns that arise due to communication between CPU and GPU during model training, etc. These antipatterns may not be easily identifiable using traditional profiling tools alone. In this paper, we propose Performance Antipatterns Eliminator Framework (PAEF), a framework to identify six performance antipatterns occurring due to data movements between CPU and GPU during training. Our framework co-relates profiles of CPU and GPU executions of the pipeline along with the static analysis of the code to identify the performance antipatterns. We further replace these antipatterns with their performant versions. We evaluate the benefits of PAEF for two industrial recommendation models, where we showcase up to 7X speedup by using PAEF over the original pipeline.
References
[1]
Stefan Behnel, Robert Bradshaw, Craig Citro, Lisandro Dalcin, Dag Sverre Seljebotn, and Kurt Smith. 2011. Cython: The best of both worlds. Computing in Science & Engineering 13, 2 (2011), 31--39.
[2]
Emery D. Berger. 2020. Scalene: Scripting-Language Aware Profiling for Python. arXiv:2006.03879 [cs.PL]
[3]
Jesse Daniel. 2019. Data Science with Python and Dask. Simon and Schuster.
[4]
Diksha Garg, Priyanka Gupta, Pankaj Malhotra, Lovekesh Vig, and Gautam Shroff. 2020. Batch-Constrained Distributional Reinforcement Learning for Session-based Recommendation. CoRR abs/2012.08984 (2020). arXiv:2012.08984 https://arxiv.org/abs/2012.08984
[5]
Priyanka Gupta, Diksha Garg, Pankaj Malhotra, Lovekesh Vig, and Gautam Shroff. 2019. NISER: Normalized Item and Session Representations with Graph Neural Networks. CoRR abs/1909.04276 (2019). arXiv:1909.04276 http://arxiv.org/abs/1909.04276
[6]
Siu Kwan Lam, Antoine Pitrou, and Stanley Seibert. 2015. Numba: A llvm-based python jit compiler. In Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC. 1--6.
[7]
Yuetian Mao, Shuai Yuan, Nan Cui, Tianjiao Du, Beijun Shen, and Yuting Chen. 2021. DeFiHap: Detecting and Fixing HiveQL Anti-Paterns. In In proceedings of International Conference on Very Large Data Bases (VLDB).
[8]
Mayank Mishra, Archisman Bhowmick, and Rekha Singhal. 2021. FASCA: Framework for Automatic Scalable Acceleration of ML Pipeline. In 2021 IEEE International Conference on Big Data (Big Data). 1867--1876.
[9]
Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. 2018. Ray: A Distributed Framework for Emerging AI Applications. In Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementation (Carlsbad, CA, USA) (OSDI'18). USENIX Association, USA, 561--577.
[10]
Shoumik Palkar, James Thomas, Deepak Narayanan, Anil Shanbhag, Rahul Palamuttam, Holger Pirk, Malte Schwarzkopf, Saman Amarasinghe, Samuel Madden, and Matei Zaharia. 2017. Weld: Rethinking the interface between data-intensive applications. arXiv preprint arXiv:1709.06416 (2017).
[11]
Shoumik Palkar and Matei Zaharia. 2019. Optimizing data-intensive computations in existing libraries with split annotations. In Proceedings of the 27th ACM Symposium on Operating Systems Principles. 291--305.
[12]
David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. 2015. Hidden technical debt in machine learning systems. Advances in neural information processing systems 28 (2015), 2503--2511.
Index Terms
- Accelerating Model Training: Performance Antipatterns Eliminator Framework
Recommendations
Accelerating Training Process in Logistic Regression Model using OpenCL Framework
In this paper, the authors propose a new parallel implemented approach on Graphics Processing Units GPU for training logistic regression model. Logistic regression has been applied in many machine learning applications to build building predictive ...
Accelerating PQMRCGSTAB algorithm on GPU
UCHPC-MAW '09: Proceedings of the combined workshops on UnConventional high performance computing workshop plus memory access workshopThe general computations on GPU are becoming more and more popular because of GPU's powerful computing ability. In this paper, how to use GPU to accelerate sparse linear system solver, preconditioned QMRCGSTAB (PQMRCGSTAB for short), is our concern. We ...
Accelerating the Training of HTK on GPU with CUDA
IPDPSW '12: Proceedings of the 2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD ForumThe training procedure of Hidden Markov Model (HMM) based Speech Recognition is often very time consuming because of its high computational complexity. The new parallel hardware like GPU can provide multi-thread processing and very high floating-point ...
Comments
Information & Contributors
Information
Published In
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 08 May 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
EuroMLSys '23
Sponsor:
Acceptance Rates
Overall Acceptance Rate 18 of 26 submissions, 69%
Upcoming Conference
EuroSys '25
- Sponsor:
- sigops
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 62Total Downloads
- Downloads (Last 12 months)32
- Downloads (Last 6 weeks)6
Reflects downloads up to 08 Mar 2025
Other Metrics
Citations
Cited By
View allView Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in