Abstract
In order to overcome the shortcomings of the existing scheduling allocation methods for Hadoop clusters in heterogeneous resource environments, an adaptive scheduling algorithm NCAS (node capacity adaptive scheduling) based on node capability is proposed. Firstly, NCAS algorithm calculates the scheduling factor according to the node performance and task characteristics; then, the scheduling factor determines the amount of data and task slots that each node should share; finally, the data and tasks are distributed more to fast nodes and less to slow nodes. The experimental results show that compared with the traditional scheduling algorithm, NCAS algorithm greatly reduces the number of backup tasks to start, significantly reduces job completion time, and improves the efficiency of task execution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Glushkova D, Jovanovic P, Abello A (2019) Mapreduce performance model for Hadoop 2.x. Inf Syst 79:32–43
Qin P, Dai B, Huang BX et al (2017) Bandwidth-aware scheduling with SDN in Hadoop: a new trend for big data. IEEE Syst J 11(4):2337–2344
Pastorelli M, Carra D, Dell’Amico M et al (2017) HFSP: bringing size-based scheduling to Hadoop. IEEE Trans Cloud Comput 5(1):43–56
Quresh NMF, Shin DR, Siddiqui IF et al (2017) Storage-tag-aware scheduler for Hadoop cluster. IEEE Access 5:13742–13755
Guo YF, Rao J, Jiang CJ et al (2017) Moving Hadoop into the cloud with flexible slot management and speculative execution. IEEE Trans Parallel Distrib Syst 28(3):798–812
Ferrucci F, Salza P, Sarro F (2018) Using Hadoop MapReduce for parallel genetic algorithms: a comparison of the global, grid and island models. Evol Comput 26(4):535–567
Lin CY, Lin YC (2017) An overall approach to achieve load balancing for Hadoop distributed file system. Int J Web Grid Serv 13(4):448–466
Alarabi L, Mokbel MF, Musleh M (2018) ST-Hadoop: a MapReduce framework for spatio-temporal data. Geoinformatica 22(4):785–813
Lu XZ, Phang K (2018) An enhanced Hadoop heartbeat mechanism for MapReduce task scheduler using dynamic calibration. China Commun 15(11):93–110
Oo MN, Parvin S, Thein T (2018) Forensic investigation through data remnants on Hadoop big data storage system. Comput Syst Sci Eng 33(3):203–217
Acknowledgements
This work was supported by grants from The National Natural Science Foundation of China (No. 61862056), the Guangxi Natural Science Foundation (No. 2017GXNSFAA198148) foundation of Wuzhou University (No. 2017B001) and Guangxi Colleges and Universities Key Laboratory of Professional Software Technology, Wuzhou University.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Zheng, M., Zhuo, M. (2020). Adaptive Scheduling Algorithm for Hadoop Node Capability in Heterogeneous Resource Environment. In: Xu, Z., Choo, KK., Dehghantanha, A., Parizi, R., Hammoudeh, M. (eds) Cyber Security Intelligence and Analytics. CSIA 2019. Advances in Intelligent Systems and Computing, vol 928. Springer, Cham. https://doi.org/10.1007/978-3-030-15235-2_182
Download citation
DOI: https://doi.org/10.1007/978-3-030-15235-2_182
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-15234-5
Online ISBN: 978-3-030-15235-2
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)