Workflow Improvement for KubeFlow DL Performance over Cloud-Native SmartX AI Cluster

Abstract

Cloud-native Kubernetes-based orchestration is widely adopted to take advantage of building large-scale resource pools by flexibly expanding the size of pools with the insertion of additional worker nodes. To meet the emerging demand for AI (Artificial Intelligence)-inspired HPC (High Performance Computing)/HPDA (High Performance Data Analytics) workloads, versatile AI clusters driven by open-source KubeFlow software have been rapidly developed by leveraging for various ML (Machine Learning)/DL (Deep Learning) tools and frameworks. However, since the current version of KubeFlow is not fully aware of underlying GPU (Graphics Processing Unit) resources, special attention should be made to smoothly execute the ML/DL workloads. Thus, in this paper, we explore tentative options to improve the ML/DL workflow under a KubeFlow-enabled AI cluster, which focus on GPU utilization efficiency with the assistance of Prometheus open-source monitoring.