Abstract
The resource scheduling of data center is a research hotspot of cloud computing. The exiting research work is concerned with the issue of fairness, resource utilization and energy efficiency, which are only applicable to the same cluster environment or specific application situations. First, the default scheduling algorithm (DRF) of Mesos is analyzed. The DRF algorithm does not consider machine performance and task types. Then, this paper presents a heterogeneous cluster multi-resource fair scheduling algorithm based on machine learning to solve the problem. The algorithm is to test the performance of the machine and use the machine learning method to classify the computing tasks and reach the goal of reasonable resource allocation. Finally, the experimental results show that the method presented in this paper not only ensures the fairness of resource allocation, but also makes the system more reasonable allocation of resources and further improves the system’s resource utilization.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Hindman, B., Konwinski, A., Zaharia, M., Ghodsi, A., Joseph, A., et al.: Mesos: a platform for fine-grained resource sharing in the data center. In: Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation, pp. 429–483. USENIX Association (2013)
Delimitrou, C., Kozyrakis, C.: Quasar: resource-efficient and QoS-aware cluster management. ACM SIGPLAN Not. 49(4), 127–144 (2014). ACM
Li, Y., Zhang, J., Zhang, W., Liu, Q.: Cluster resource adjustment based on an improved artificial fish swarm algorithm in Mesos. In: IEEE International Conference on Signal Processing, pp. 1843–1847. IEEE (2017)
Ghodsi, A., Zaharia, M., Hindman, B., Konwinski, A., Shenker, S., et al.: Dominant resource fairness: fair allocation of multiple resource types. In: Usenix Conference on Networked Systems Design and Implementation, pp. 323–336. USENIX Association (2011)
Wang, W., Liang, B., Li, B.: Multi-resource fair allocation in heterogeneous cloud computing systems. IEEE Trans. Parallel Distrib. Syst. 26(10), 2822–2835 (2015)
Tang, H., Li, Y., Wang, L., et al.: Predicting misconfiguration-induced unsuccessful executions of jobs in big data system. In: Computer Software and Applications Conference, pp. 772–777. IEEE (2017)
Cernadas, E., Amorim, D.: Do we need hundreds of classifiers to solve real world classification problems? J. Mach. Learn. Res. 15(1), 3133–3181 (2014)
Chen, T., Guestrin, C.: XGBoost: a scalable tree boosting system. In: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 785–794. ACM (2016)
Isard, M., Prabhakaran, V., Currey, J., Wieder, U., Talwar, K., Goldberg, A.: Quincy: fair scheduling for distributed computing clusters. In: ACM SIGOPS, Symposium on Operating Systems Principles, pp. 261–276. ACM (2009)
Ghodsi, A., Zaharia, M., Shenker, S., Stoica, I.: Choosy: max-min fair sharing for datacenter jobs with constraints. In: Proceedings of the 8th ACM European Conference on Computer Systems, pp. 365–378. ACM (2013)
Ousterhout, K., Wendell, P., Zaharia, M., Stoica, I. Sparrow: distributed, low latency scheduling. In: Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, pp. 69–84. ACM (2013)
Grandl, R., Chowdhury, M., Akella, A., Ananthanarayanan, G.: Altruistic scheduling in multi-resource clusters. In: 12th USENIX Symposium on Operating Systems Design and Implementation, pp. 65–80 (2016)
Ukidave, Y., Li, X,, Kaeli, D.: Mystic: predictive scheduling for GPU based cloud servers using machine learning. In: IEEE International Parallel and Distributed Processing Symposium, pp. 353–362. IEEE (2016)
Chen, X., Wu, H., Wu, Y., Lu, Z., Zhang, W.: Large-scale resource scheduling method based on minimum cost maximum flow. J. Softw. 28(3), 598–610 (2017)
Acknowledgments
This work is supported by the Natural Science Foundation of China (No. 61762008), the Natural Science Foundation Project of Guangxi (No. 2017GXNSFAA198141), the Key R&D project of Guangxi (No. GuiKE AB17195014), and the R&D Project of Nanning (No. 20173161).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Liu, W., Chen, N., Li, H., Tang, Y., Liang, B. (2018). A Heterogeneous Cluster Multi-resource Fair Scheduling Algorithm Based on Machine Learning. In: Zhou, Q., Gan, Y., Jing, W., Song, X., Wang, Y., Lu, Z. (eds) Data Science. ICPCSEE 2018. Communications in Computer and Information Science, vol 901. Springer, Singapore. https://doi.org/10.1007/978-981-13-2203-7_44
Download citation
DOI: https://doi.org/10.1007/978-981-13-2203-7_44
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-2202-0
Online ISBN: 978-981-13-2203-7
eBook Packages: Computer ScienceComputer Science (R0)