Elastic distributed training with fast convergence and efficient resource utilization
- ORNL
Distributed learning is now routinely conducted on cloud as well as dedicated clusters. Training with elastic resources brings new challenges and design choices. Prior studies focus on runtime performance and assume a static algorithmic behavior. In this work, by analyzing the impact of of resource scaling on convergence, we introduce schedules for synchronous stochastic gradient descent that proactively adapt the number of learners to reduce training time and improve convergence. Our approach no longer assumes a constant number of processors throughout training. In our experiment, distributed stochastic gradient descent with dynamic schedules and reduction momentum achieves better convergence and significant speedups over prior static ones. Numerous distributed training jobs running on cloud may benefit from our approach.
- Research Organization:
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
- Sponsoring Organization:
- USDOE
- DOE Contract Number:
- AC05-00OR22725
- OSTI ID:
- 1843691
- Resource Relation:
- Conference: International Conference on Machine Learning and Applications (ICMLA) - remote, California, United States of America - 12/13/2021 10:00:00 AM-12/16/2021 5:00:00 AM
- Country of Publication:
- United States
- Language:
- English
Similar Records
Adaptive elasticity policies for staging-based in situ visualization
High-Level Synthesis of Parallel Specifications Coupling Static and Dynamic Controllers