Abstract
TensorFlow, a popular machine learning (ML) platform, allows users to transparently exploit both GPUs and CPUs to run their applications. Since GPUs are optimized for compute-intensive workloads (e.g., matrix calculus), they help boost executions, but introduce resource heterogeneity. TensorFlow neither provides efficient heterogeneous resource management nor allows for the enforcement of user-defined constraints on the execution time. Most of the works address these issues in the context of creating models on existing data sets (training phase), and only focus on scheduling algorithms. This paper focuses on the inference phase, that is, on the application of created models to predict the outcome on new data interactively, and presents a comprehensive resource management solution called ROMA (Resource Constrained ML Applications). ROMA is an extension of TensorFlow that (a) provides means to easily deploy multiple TensorFlow models in containers using Kubernetes b) allows users to set constraints on response times, (c) schedules the execution of requests on GPUs and CPUs using heuristics, and (d) dynamically refines the CPU core allocation by exploiting control theory. The assessment conducted on four real-world benchmark applications compares ROMA against four different systems and demonstrates a significant reduction ( \(75\%\)) in constraint violations and \(24\%\) saved resources on average.
This work has been partially supported by the SISMA national research project, which has been funded by the MIUR under the PRIN 2017 program (Contract 201752ENYB) and by the European Commission grant no. 825480 (H2020), SODALITE.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
Source code is available at https://github.com/deib-polimi/ROMA.
- 3.
- 4.
- 5.
- 6.
This average execution time is computed by considering the different executions of the same model executable over a given time window.
- 7.
In December 2020, the Kubernetes team announced that the Docker runtime will be considered deprecated in future versions [14]. Docker will not be removed from Kubernetes at least until late 2021. While the evaluation of ROMA in Sect. 5 is based on the described Docker-dependent implementation, we are already developing a version of ROMA that does not require Docker and that supports other container runtimes as, for example, containerd [6].
- 8.
To avoid ambiguities, in this section a dynamic model is a mathematical representation of the controlled system, that is, of the ML application.
References
Abadi, M., et al.: TensorFlow: a system for large-scale machine learning. In: 12th Symposium on Operating Systems Design and Implementation, pp. 265–283. USENIX (2016)
Baresi, L., Guinea, S., Leva, A., Quattrocchi, G.: A discrete-time feedback controller for containerized cloud applications. In: Proceedings of the 2016 24th International Symposium on Foundations of Software Engineering, pp. 217–228. ACM (2016)
Baresi, L., Leva, A., Quattrocchi, G.: Fine-grained dynamic resource allocation for big-data applications. IEEE Trans. Softw. Eng. 47(8), 1668–1682 (2021)
Chen, L., Huo, X., Agrawal, G.: Accelerating MapReduce on a coupled CPU-GPU architecture. In: Hollingsworth, J.K. (ed.) SC Conference on High Performance Computing Networking, Storage and Analysis, pp. 1–11. IEEE/ACM (2012)
Chen, T., Li, M., et al.: MXNet: a Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems. arXiv (2015)
Containerd: an industry-standard container runtime with an emphasis on simplicity, robustness and portability (2021). https://containerd.io
Ding, J., Cao, R., Saravanan, I., Morris, N., Stewart, C.: Characterizing service level objectives for cloud services: realities and myths. In: 2019 IEEE International Conference on Autonomic Computing (ICAC), pp. 200–206. IEEE (2019)
Farokhi, S., Lakew, E.B., Klein, C., Brandic, I., Elmroth, E.: Coordinating CPU and memory elasticity controllers to meet service response time constraints. In: 2015 International Conference on Cloud and Autonomic Computing, pp. 69–80 (2015)
Fedorov, R., Camerada, A., et al.: Estimating snow cover from publicly available images. IEEE Trans. Multimed. 18(6), 1187–1200 (2016)
Forbes: TensorFlow Turns 5 - Five Reasons Why it is the Most Popular ML Framework. http://tiny.cc/Forbes-TF (2020)
He, K., Zhang, X., et al.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Jahani, A., Lattuada, M., Ciavotta, M., Ardagna, D., Amaldi, E., Zhang, L.: Optimizing on-demand GPUs in the cloud for deep learning applications training. In: 2019 4th International Conference on Computing, Communications and Security (ICCCS), pp. 1–8 (2019)
Khalid, Y.N., Aleem, M., Prodan, R., Iqbal, M.A., Islam, M.A.: E-OSched: a load balancing scheduler for heterogeneous multicores. J. Supercomput. 74(10), 5399–5431 (2018)
Kubernetes: Don’t Panic: Kubernetes and Docker (2020). https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker
Kubernetes: Schedule GPUs (2020). https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/
Lakew, E., Papadopoulos, A., Maggio, M., Klein, C., Elmroth, E.: KPI-agnostic control for fine-grained vertical elasticity. In: Proceedings of the 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, pp. 589–598. IEEE (2017)
Mittal, S., Vetter, J.S.: A survey of CPU-GPU heterogeneous computing techniques. ACM Comput. Surv. 47(4), 69:1–69:35 (2015)
Nozal, R., Bosque, J.L., Beivide, R.: EngineCL: usability and performance in heterogeneous computing. Future Gener. Comput. Syst. 107, 522–537 (2020)
Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. Adv. Neural Inf. Proc. Syst. 32, 8024–8035 (2019)
Szegedy, C., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)
Verma, A., Cherkasova, L., et al.: Deadline-based workload management for MapReduce environments: pieces of the performance puzzle. In: 2012 IEEE Network Operations and Management Symposium, pp. 900–905. IEEE (2012)
Zhang, X., Zou, J., He, K., Sun, J.: Accelerating very deep convolutional networks for classification and detection. IEEE Trans. Pattern Anal. Mach. Intell. 38(10), 1943–1955 (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Baresi, L., Quattrocchi, G., Rasi, N. (2021). Resource Management for TensorFlow Inference. In: Hacid, H., Kao, O., Mecella, M., Moha, N., Paik, Hy. (eds) Service-Oriented Computing. ICSOC 2021. Lecture Notes in Computer Science(), vol 13121. Springer, Cham. https://doi.org/10.1007/978-3-030-91431-8_15
Download citation
DOI: https://doi.org/10.1007/978-3-030-91431-8_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-91430-1
Online ISBN: 978-3-030-91431-8
eBook Packages: Computer ScienceComputer Science (R0)