Abstract
Heterogeneity is common in parallel and distributed environments used for extensive computations such as MapReduce. Stragglers are the tasks that are running on inferior performing nodes in the cluster. Early detection of stragglers is always challenging in such environments. In the previously proposed approaches, late detection of straggler tasks and estimation of time to end (TTE) for all the tasks running in a heterogeneous environment delays the entire job execution. Early straggler detection help to speculate a task at the early stages of task execution which indirectly improves the complete job execution. This article proposed early straggler detection by a recurrent neural network (ESDRNN) that collects the task and node information every three seconds from ApplicationMaster to train the RNN. It classifies the straggler tasks pretty early by RNN, between thirty to forty seconds of task execution, and transfers a list of classified tasks to an agent running on ResourceManager. RNN is a type of artificial neural network that is prevalent for processing sequential time-series data. Then, the agent predicts the TTE of these classified tasks by the Autoregressive integrated moving average (ARIMA) model. Finally, it sorts and refreshes the list with higher TTE after every ten seconds and speculates the tasks for the early completion of the MapReduce job. This proposed technique’s performance is evaluated on the HiBench benchmark suite of Hadoop’s most popular benchmark. Finally, compared with the default speculation technique and different techniques, the proposed speculation technique detects the stragglers early within 35 to 40 seconds of task execution. As a result, it decreases the job execution time by an average of 21% to 38% significantly for different workloads in a heterogeneous Hadoop cluster.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Ahmad F, Chakradhar ST, Raghunathan A, Vijaykumar T (2012) Tarazu: optimizing mapreduce on heterogeneous clusters. In: ACM SIGARCH Computer architecture news. ACM, vol 40, pp 61–74
Ananthanarayanan G, Kandula S, Greenberg AG, Stoica I, Lu Y, Saha B, Harris E (2010) Reining in the outliers in map-reduce clusters using mantri. In: Osdi, vol 10, p 24
Arasanal RM, Rumani DU (2013) Improving mapreduce performance through complexity and performance based data placement in heterogeneous hadoop clusters. In: International conference on distributed computing and internet technology. Springer, pp 115–125
Bawankule KL, Dewang RK, Singh AK (2021) Load balancing approach for a mapreduce job running on a heterogeneous hadoop cluster. In: International conference on distributed computing and internet technology. Springer, pp 289–298
Bawankule KL, Dewang RK, Singh AK (2021) Historical data based approach for straggler avoidance in a heterogeneous hadoop cluster J Ambient Intell Human Comput. https://doi.org/10.1007/s12652-020-02699-0
Bawankule KL, Dewang RK, Singh AK (2021) Performance analysis of hadoop yarn job schedulers in a multi-tenant environment on hibench benchmark suite. International Journal of Distributed Systems and Technologies (IJDST) 12(3):64–82
Bawankule KL, Dewang RK, Singh AK (2022) A classification framework for straggler mitigation and management in a heterogeneous hadoop cluster: A state-of-art survey Journal of King Saud University-Computer and Information Sciences
Bawankule KL, Dewang RK, Singh AK (2022) Historical data based approach to mitigate stragglers from the reduce phase of mapreduce in a heterogeneous hadoop cluster. Cluster Computing. pp 1–19
Bousbaci A, Kamel N (2018) Efficient data distribution and results merging for parallel data clustering in mapreduce environment. Appl Intell 48(8):2408–2428
Chang F, Dean J, Ghemawat S, Hsieh WC, Wallach DA, Burrows M, Chandra T, Fikes A, Gruber RE (2008) Bigtable: a distributed storage system for structured data. ACM Transactions on Computer Systems (TOCS) 26(2):1–26
Chen Q, Guo M, Deng Q, Zheng L, Guo S, Shen Y (2013) Hat: history-based auto-tuning mapreduce in heterogeneous environments. The Journal of Supercomputing 64(3):1038–1054
Chen Q, Zhang D, Guo M, Deng Q, Guo S (2010) Samr: a self-adaptive mapreduce scheduling algorithm in heterogeneous environment. In: 2010 10Th IEEE international conference on computer and information technology. IEEE, pp 2736–2743
Dean J, Ghemawat S (2008) Mapreduce: simplified data processing on large clusters. Commun ACM 51(1):107–113
Frnda J, Pavlicko M, Durica M, Sevcik L, Voznak M, Fournier-Viger P, Lin JCW (2021) A new perceptual evaluation method of video quality based on neural network. Intelligent Data Analysis 25(3):571–587
Fujita H (2017) Challenges on big data based clouds health-care for risk predictions based on ensemble classifiers and subjective analysis. In: CLOSER, p. 9
Ghemawat S, Gobioff H, Leung ST (2003) The google file system
Glushkova D, Jovanovic P, Abelló A. (2019) Mapreduce performance model for hadoop 2. x. Inf Syst 79:32–43
Gupta S, Fritz C, Price B, Hoover R, Dekleer J, Witteveen C (2013) Throughputscheduler: Learning to schedule on heterogeneous hadoop clusters. In: 10Th international conference on autonomic computing ({ICAC} 13), pp 159–165
He Z, Cao Y, Du L, Xu B, Yang J, Cao Y, Tang S, Zhuang Y (2019) Mrfn: Multi-receptive-field network for fast and accurate single image super-resolution. IEEE Trans Multimed 22(4):1042–1054
Huang S, Huang J, Dai J, Xie T, Huang B (2010) The hibench benchmark suite: Characterization of the mapreduce-based data analysis. In: 2010 IEEE 26Th international conference on data engineering workshops (ICDEW 2010). IEEE, pp 41–51
Javadpour A, Wang G, Rezaei S, Li KC (2020) Detecting straggler mapreduce tasks in big data processing infrastructure by neural network. The Journal of Supercomputing. pp 1–25
Jin H, Yang X, Sun X, Raicu I (2012) Adapt: Availability-aware mapreduce data placement for non-dedicated distributed computing. In: 2012 IEEE 32Nd international conference on distributed computing systems. IEEE, pp 516–525
Lee CW, Hsieh KY, Hsieh SY, Hsiao HC (2014) A dynamic data placement strategy for hadoop in heterogeneous environments. Big Data Research 1:14–22
Li H, Wei X, Fu Q, Luo Y (2014) Mapreduce delay scheduling with deadline constraint. Concurrency and Computation:, Practice and Experience 26(3):766–778
Li Q, Cao Z, Ding W, Li Q (2020) A multi-objective adaptive evolutionary algorithm to extract communities in networks. Swarm and Evolutionary Computation 52:100629
Li Q, Li L, Wang W, Li Q, Zhong J (2020) A comprehensive exploration of semantic relation extraction via pre-trained cnns. Knowl-Based Syst 194:105488
Li Y, Yang Q, Lai S, Li B (2015) A new speculative execution algorithm based on c4. 5 decision tree for hadoop. In: International conference of young computer scientists, engineers and educators. Springer, pp 284–291
Lin JCW, Li Y, Fournier-Viger P, Djenouri Y, Wang LSL (2019) Mining high-utility sequential patterns from big datasets. In: 2019 IEEE International conference on big data (big data). IEEE, pp 2674–2680
Pandey V, Saini P (2020) A heuristic method towards deadline-aware energy-efficient mapreduce scheduling problem in hadoop yarn. Cluster Computing. pp 1–17
Pedrycz W, Chen SM (2014) Information granularity, big data, and computational intelligence, vol. 8 Springer
Saleti S, Subramanyam R (2019) A novel mapreduce algorithm for distributed mining of sequential patterns using co-occurrence information. Appl Intell 49(1):150–171
Shvachko K, Kuang H, Radia S, Chansler R, et al. (2010) The hadoop distributed file system. In: MSST, vol 10, pp 1–10
Sun X, He C, Lu Y (2012) Esamr: an enhanced self-adaptive mapreduce scheduling algorithm. In: 2012 IEEE 18Th international conference on parallel and distributed systems. IEEE, pp 148–155
Tang S, Yu F (2021) Construction and verification of retinal vessel segmentation algorithm for color fundus image under bp neural network model. J Supercomput 77(4):3870–3884
Vavilapalli VK, Murthy AC, Douglas C, Agarwal S, Konar M, Evans R, Graves T, Lowe J, Shah H, Seth S et al (2013) Apache hadoop yarn: Yet another resource negotiator. In: Proceedings of the 4th annual symposium on cloud computing, p 5. ACM
Wang B, Jiang J, Yang G (2015) Actcap: Accelerating mapreduce on heterogeneous clusters with capability-aware data placement. In: 2015 IEEE Conference on computer communications (INFOCOM). IEEE, pp 1328–1336
Wang S, Cong Y, Zhu H, Chen X, Qu L, Fan H, Zhang Q, Liu M (2020) Multi-scale context-guided deep network for automated lesion segmentation with endoscopy images of gastrointestinal tract. IEEE J Biomed Health Inf 25(2):514–525
Wang T, Li J, Guo J (2021) A scalable parallel chinese online encyclopedia knowledge denoising method based on entry tags and spark cluster. Appl Intell 51(10):7573–7599
White T (2012) Hadoop: The definitive guide. “O’Reilly Media Inc.”
Whitehead BA, Kiech EL, Ali M (1990) Learning and diagnosing faults using neural networks
Xie J, Yin S, Ruan X, Ding Z, Tian Y, Majors J, Manzanares A, Qin X (2010) Improving mapreduce performance through data placement in heterogeneous hadoop clusters. In: 2010 IEEE International symposium on parallel & distributed processing, workshops and phd forum (IPDPSW). IEEE, pp 1–9
Xiong R, Du Y, Jin J, Luo J (2018) Hadaap: a hotness-aware data placement strategy for improving storage efficiency in heterogeneous hadoop clusters. Concurrency and Computation:, Practice and Experience 30(20):e4830
Xu H, Lau WC (2016) Optimization for speculative execution in big data processing clusters. IEEE Trans Parallel Distrib Syst 28(2):530–545
Ying C, Huang Z, Ying C (2018) Accelerating the image processing by the optimization strategy for deep learning algorithm dbn. EURASIP J Wirel Commun Netw 2018(1):1–8
Zaharia M, Borthakur D, Sen Sarma J, Elmeleegy K, Shenker S, Stoica I (2010) Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling. In: Proceedings of the 5th european conference on computer systems, pp 265–278. ACM
Zaharia M, Konwinski A, Joseph AD, Katz RH, Stoica I (2008) Improving mapreduce performance in heterogeneous environments. In: Osdi, vol 8, p 7
Zhang X, Wu Y, Zhao C (2016) Mrheter: improving mapreduce performance in heterogeneous environments. Clust Comput 19(4):1691–1701
Funding
The authors would like to thank the Quality Improvement Program of All India Council for Technical Education(AICTE), India, to support the research.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interests
On behalf of all authors, the corresponding author states that there is no conflict of interest.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Bawankule, K.L., Dewang, R.K. & Singh, A.K. Early straggler tasks detection by recurrent neural network in a heterogeneous environment. Appl Intell 53, 7369–7389 (2023). https://doi.org/10.1007/s10489-022-03837-1
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-022-03837-1