Abstract
Hadoop is a well-known parallel computing system for processing massive amounts of data, but there is a task in Hadoop called “Straggling task” that has a significant impact on Hadoop. Speculative Execution (SE) is a good technique to deal with “Straggling tasks” since it monitors the rate of running processes in real time. backing up the “Straggler” on another node to increase the opportunity of completing the backup task ahead of the original. This proposal tackles dealing with the “Straggling task” by creating a strategy able to deal with misjudgment, improper selection of backup nodes, and making speculative tasks start from the checkpoint, by leveraging the checkpoint of the original tasks. BY this work we can achieve comparing with common methods in this sector, such as LATE, ESAMR, and the real remaining time for Wordcount and Sort benchmarks, it was demonstrated to be capable of detecting straggler tasks and properly estimating execution time. It also allows for job execution to be sped up.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Kaseb, M.R., Khafagy, M.H., Ali, I.A., Saad, E.M.: Multi-split HDFS technique for improving data confidentiality in big data replication. World Conference on Information Systems and Technologies, pp. 132–142 (2019)
Gill, S.S., Ouyang, X., Garraghan, P.: Tails in the cloud: a survey and taxonomy of straggler management within large-scale cloud data centres. J. Supercomput. 76(12), 10050–10089 (2020). https://doi.org/10.1007/s11227-020-03241-x
Ozfatura, E., Ulukus, S., Gündüz, D.: Straggler-aware distributed learning: Communication-computation latency trade-off. Entropy 22(5), 1–30 (2020). https://doi.org/10.3390/E22050544
Editor, J., S. M. K.: Improving Hadoop Performance 1(4), 2020, [Online]. Available: www.jcsis.org/
Oo, Z.Z., Phyu, S.: Improving Hadoop MapReduce Performance Using Speculative Execution Strategy in a Heterogeneous Environment.
Javadpour, A., Wang, G., Rezaei, S., Li, K.-C.: Detecting straggler MapReduce tasks in big data processing infrastructure by neural network. J. Supercomput. 76(9), 6969–6993 (2020). https://doi.org/10.1007/s11227-019-03136-6
Farhang, M., Safi-Esfahani, F.: Recognizing mapreduce straggler tasks in big data infrastructures using artificial neural networks. J. Grid Comput. 18(4), 879–901 (2020). https://doi.org/10.1007/s10723-020-09514-2
Katrawi, A.H., Abdullah, R., Anbar, M., Abasi, A.K.: Earlier stage for straggler detection and handling using combined CPU test and LATE methodology. Int. J. Electr. Comput. Eng. 10(5), 4910–4917 (2020). https://doi.org/10.11591/ijece.v10i5.pp4910-4917
Van Veen, C.J.: Een bijdrage tot de kennis van de jeugdige commune delinquent. Ned. Tijdschr. Psychol. 4(4), 319–339 (1949)
Wang, Y., Lu, W., Lou, R., Wei, B.: Improving mapreduce performance with partial speculative execution. J. Grid Comput. 13(4), 587–604 (2015). https://doi.org/10.1007/s10723-015-9350-y
Liu, X., Liu, Q.: An Optimized Speculative Execution Strategy Based on Local Data Prediction in a Heterogeneous Hadoop Environment. In: Proc. - 2017 IEEE Int. Conf. Comput. Sci. Eng. IEEE/IFIP Int. Conf. Embed. Ubiquitous Comput. CSE EUC 2017, vol. 2, no. 3, pp. 128–131 (2017) doi: https://doi.org/10.1109/CSE-EUC.2017.208
Chen, Q., Liu, C., Xiao, Z.: Improving mapreduce performance using smart speculative execution strategy. IEEE Trans Comput 63(4), 954–967 (2014)
Yoo, D.G., Sim, K.M.: A comparative review of job scheduling for MapReduce. In: Proc. IEEE International Conference on Cloud Computing and Intelligence Systems (2011)
Ananthanarayanan, G., Ghodsi, A., Shenker, S., Stoica, I.: Effective straggler mitigation: attack of the clones (2013). [4] Matei Zaharia, Andy Konwinski, Anthony D. Joseph, Randy Katz, Ion Stoica University of California, Berkeley (2014)
Wang, Y., Lu, W., Lou, R.,·Wei, B.: (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Hussien, G.K., Khafagy, M.H., Ibrahim, M.H., Kaseb, M.R. (2022). A Comparative Study on Improving Straggler Tasks in Hadoop. In: Rocha, A., Adeli, H., Dzemyda, G., Moreira, F. (eds) Information Systems and Technologies. WorldCIST 2022. Lecture Notes in Networks and Systems, vol 468. Springer, Cham. https://doi.org/10.1007/978-3-031-04826-5_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-04826-5_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-04825-8
Online ISBN: 978-3-031-04826-5
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)