Abstract
Existing big data computing and storage platforms are generally based on traditional virtual machine technology, which often results in low resource utilization, a long time for flexible scaling and expanding clusters. To deal with these problems, this paper proposes an improved container scheduling algorithm, Kubernetes-based Particle Swarm Optimization (K-PSO), for big data applications based on Particle Swarm Optimization (PSO). The K-PSO algorithm converges faster than the basic PSO algorithm, and the algorithm running time is reduced by about half. The K-PSO capacity for big data applications is implemented in the Kubernetes container cloud system. The experimental results show that the node resource utilization rate of the improved scheduling strategy based on K-PSO algorithm is about 20% higher than that of Kube-Scheduler default strategy, BalancedQosPriority strategy, ESS strategy, and PSO strategy while the average I/O performance and average computing performance of Hadoop cluster are not degraded.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Felter, W, Ferreira, A, Rajamony, R, et al.: An updated performance comparison of virtual machines and linux containers. In: 2015 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 171–172. IEEE (2015)
Alfonso, C.D., Calatrava, A., Moltó, G.: Container-based virtual elastic clusters. J. Syst. Softw. 127, 1–11 (2017)
Li, Z., Zhang, Y., Liu, Y.: Towards a full-stack DevOps environment (platform-as-a-service) for cloud-hosted applications. Tsinghua Sci. Technol. 22(1), 1–9 (2017)
Pahl, C., Brogi, A., Soldani, J., et al.: Cloud container technologies: a state-of-the-art review. IEEE Trans. Cloud Comput. 99, 1–1 (2017)
Gandhi, A., Thota, S., Dube, P., et al.: Autoscaling for Hadoop clusters. In: 2016 IEEE International Conference on Cloud Engineering (IC2E). IEEE (2016)
Khan, M., Jin, Y., Li, M., et al.: Hadoop performance modeling for job estimation and resource provisioning. IEEE Trans. Parallel Distrib. Syst. 99, 441–454 (2015)
Naik, N.: Docker container-based big data processing system in multiple clouds for everyone. In: 2017 IEEE International Systems Engineering Symposium (ISSE), pp. 1–7. IEEE (2017)
Xu, Z., Yang, H.: Quality of service based on Kubernetes scheduler. Softw. Guide 17(11), 77–80 (2018)
Zhang, K., Peng, L., Lu, X., et al.: Kubernetes elastic scheduling on open source cloud. Comput. Technol. Dev. 29(02), 115–120 (2019)
Weiwei, L., Dejun, Q.: Review of cloud computing resource scheduling. Comput. Sci. 39(10), 1–6 (2012)
Fernández-Baca, D.: Allocating modules to processors in a distributed system. IEEE Trans. Softw. Eng. 15(11), 1427–1436 (1989)
Bernstein, D.: Containers and cloud: from LXC to Docker to Kubernetes. IEEE Cloud Comput. 1(3), 81–84 (2014)
Hindman, B., Konwinski, A., Zaharia, M., et al.: Mesos: a platform for fine-grained resource sharing in the data center. In: NSDI, vol. 11, no. 2011, p. 22 (2011)
Jie, L., Guangzhong, L.: Research on automated container deployment of Hadoop distributed cluster. Comput. Appl. Res. 33(11), 3404–3407 (2016)
Liu, B., Li, P., Lin, W., et al.: A new container scheduling algorithm based on multi-objective optimization. Soft Comput. 22(23), 7741–7752 (2018)
Lin, W., Wang, Z.: Docker cluster scheduling strategy based on genetic algorithm. J. S. China Univ. Technol. (Nat. Sci. Ed.) 46(3), 19 (2018)
Sujana, J.A.J., Revathi, T., Priya, T.S.S., et al.: Smart PSO-based secured scheduling approaches for scientific workflows in cloud computing. Soft Comput. 23(5), 1745–1765 (2019)
Zhou, Z., Chang, J., Hu, Z., et al.: A modified PSO algorithm for task scheduling optimization in cloud computing. Concurr. Comput. Pract. Exp. 30(24), e4970. 3404–3407 (2018)
Adhikari, M., Srirama, S.N.: Multi-objective accelerated particle swarm optimization with a container-based scheduling for Internet-of-Things in cloud environment. J. Netw. Comput. Appl. 137, 35–61 (2019)
Zhang, L., Tang, Y., Hua, C., et al.: A new particle swarm optimization algorithm with adaptive inertia weight based on Bayesian techniques. Appl. Soft Comput. 28, 138–149 (2015)
Nobile, M.S., Cazzaniga, P., Besozzi, D., et al.: Fuzzy self-tuning PSO: a settings-free algorithm for global optimization. Swarm Evol. Comput. 39, 70–85 (2018)
Taherkhani, M., Safabakhsh, R.: A novel stability-based adaptive inertia weight for particle swarm optimization. Appl. Soft Comput. 38, 281–295 (2016)
Deng, W., Yao, R., Zhao, H., et al.: A novel intelligent diagnosis method using optimal LS-SVM with improved PSO algorithm. Soft Comput. 23(7), 2445–2462 (2019)
Clerc, M.: The swarm and the queen: towards a deterministic and adaptive particle swarm optimization. In: Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406). IEEE (2002)
Acknowledgment
This work is supported by National Natural Science Foundation of China (Grant Nos. 61772205, 61872084), Guangdong Science and Technology Department (Grant No. 2017B010126002), Guangzhou Science and Technology Program key projects (Grant Nos. 201802010010, 201807010052, 201902010040 and 201907010001), Nansha Science and Technology Projects (Grant No. 2017GJ001), Guangzhou Development Zone Science and Technology (Grant No. 2018GH17) and the Fundamental Research Funds for the Central Universities, SCUT (Grant No. 2019ZD26).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Li, J., Liu, B., Lin, W., Li, P., Gao, Q. (2019). An Improved Container Scheduling Algorithm Based on PSO for Big Data Applications. In: Vaidya, J., Zhang, X., Li, J. (eds) Cyberspace Safety and Security. CSS 2019. Lecture Notes in Computer Science(), vol 11982. Springer, Cham. https://doi.org/10.1007/978-3-030-37337-5_42
Download citation
DOI: https://doi.org/10.1007/978-3-030-37337-5_42
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-37336-8
Online ISBN: 978-3-030-37337-5
eBook Packages: Computer ScienceComputer Science (R0)