Skip to main content
SpringerLink
Log in

Early straggler tasks detection by recurrent neural network in a heterogeneous environment

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Heterogeneity is common in parallel and distributed environments used for extensive computations such as MapReduce. Stragglers are the tasks that are running on inferior performing nodes in the cluster. Early detection of stragglers is always challenging in such environments. In the previously proposed approaches, late detection of straggler tasks and estimation of time to end (TTE) for all the tasks running in a heterogeneous environment delays the entire job execution. Early straggler detection help to speculate a task at the early stages of task execution which indirectly improves the complete job execution. This article proposed early straggler detection by a recurrent neural network (ESDRNN) that collects the task and node information every three seconds from ApplicationMaster to train the RNN. It classifies the straggler tasks pretty early by RNN, between thirty to forty seconds of task execution, and transfers a list of classified tasks to an agent running on ResourceManager. RNN is a type of artificial neural network that is prevalent for processing sequential time-series data. Then, the agent predicts the TTE of these classified tasks by the Autoregressive integrated moving average (ARIMA) model. Finally, it sorts and refreshes the list with higher TTE after every ten seconds and speculates the tasks for the early completion of the MapReduce job. This proposed technique’s performance is evaluated on the HiBench benchmark suite of Hadoop’s most popular benchmark. Finally, compared with the default speculation technique and different techniques, the proposed speculation technique detects the stragglers early within 35 to 40 seconds of task execution. As a result, it decreases the job execution time by an average of 21% to 38% significantly for different workloads in a heterogeneous Hadoop cluster.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26

Similar content being viewed by others

References

  1. Ahmad F, Chakradhar ST, Raghunathan A, Vijaykumar T (2012) Tarazu: optimizing mapreduce on heterogeneous clusters. In: ACM SIGARCH Computer architecture news. ACM, vol 40, pp 61–74

  2. Ananthanarayanan G, Kandula S, Greenberg AG, Stoica I, Lu Y, Saha B, Harris E (2010) Reining in the outliers in map-reduce clusters using mantri. In: Osdi, vol 10, p 24

  3. Arasanal RM, Rumani DU (2013) Improving mapreduce performance through complexity and performance based data placement in heterogeneous hadoop clusters. In: International conference on distributed computing and internet technology. Springer, pp 115–125

  4. Bawankule KL, Dewang RK, Singh AK (2021) Load balancing approach for a mapreduce job running on a heterogeneous hadoop cluster. In: International conference on distributed computing and internet technology. Springer, pp 289–298

  5. Bawankule KL, Dewang RK, Singh AK (2021) Historical data based approach for straggler avoidance in a heterogeneous hadoop cluster J Ambient Intell Human Comput. https://doi.org/10.1007/s12652-020-02699-0

  6. Bawankule KL, Dewang RK, Singh AK (2021) Performance analysis of hadoop yarn job schedulers in a multi-tenant environment on hibench benchmark suite. International Journal of Distributed Systems and Technologies (IJDST) 12(3):64–82

    Article  Google Scholar 

  7. Bawankule KL, Dewang RK, Singh AK (2022) A classification framework for straggler mitigation and management in a heterogeneous hadoop cluster: A state-of-art survey Journal of King Saud University-Computer and Information Sciences

  8. Bawankule KL, Dewang RK, Singh AK (2022) Historical data based approach to mitigate stragglers from the reduce phase of mapreduce in a heterogeneous hadoop cluster. Cluster Computing. pp 1–19

  9. Bousbaci A, Kamel N (2018) Efficient data distribution and results merging for parallel data clustering in mapreduce environment. Appl Intell 48(8):2408–2428

    Article  Google Scholar 

  10. Chang F, Dean J, Ghemawat S, Hsieh WC, Wallach DA, Burrows M, Chandra T, Fikes A, Gruber RE (2008) Bigtable: a distributed storage system for structured data. ACM Transactions on Computer Systems (TOCS) 26(2):1–26

    Article  Google Scholar 

  11. Chen Q, Guo M, Deng Q, Zheng L, Guo S, Shen Y (2013) Hat: history-based auto-tuning mapreduce in heterogeneous environments. The Journal of Supercomputing 64(3):1038–1054

    Article  Google Scholar 

  12. Chen Q, Zhang D, Guo M, Deng Q, Guo S (2010) Samr: a self-adaptive mapreduce scheduling algorithm in heterogeneous environment. In: 2010 10Th IEEE international conference on computer and information technology. IEEE, pp 2736–2743

  13. Dean J, Ghemawat S (2008) Mapreduce: simplified data processing on large clusters. Commun ACM 51(1):107–113

    Article  Google Scholar 

  14. Frnda J, Pavlicko M, Durica M, Sevcik L, Voznak M, Fournier-Viger P, Lin JCW (2021) A new perceptual evaluation method of video quality based on neural network. Intelligent Data Analysis 25(3):571–587

    Article  Google Scholar 

  15. Fujita H (2017) Challenges on big data based clouds health-care for risk predictions based on ensemble classifiers and subjective analysis. In: CLOSER, p. 9

  16. Ghemawat S, Gobioff H, Leung ST (2003) The google file system

  17. Glushkova D, Jovanovic P, Abelló A. (2019) Mapreduce performance model for hadoop 2. x. Inf Syst 79:32–43

    Article  Google Scholar 

  18. Gupta S, Fritz C, Price B, Hoover R, Dekleer J, Witteveen C (2013) Throughputscheduler: Learning to schedule on heterogeneous hadoop clusters. In: 10Th international conference on autonomic computing ({ICAC} 13), pp 159–165

  19. He Z, Cao Y, Du L, Xu B, Yang J, Cao Y, Tang S, Zhuang Y (2019) Mrfn: Multi-receptive-field network for fast and accurate single image super-resolution. IEEE Trans Multimed 22(4):1042–1054

    Article  Google Scholar 

  20. Huang S, Huang J, Dai J, Xie T, Huang B (2010) The hibench benchmark suite: Characterization of the mapreduce-based data analysis. In: 2010 IEEE 26Th international conference on data engineering workshops (ICDEW 2010). IEEE, pp 41–51

  21. Javadpour A, Wang G, Rezaei S, Li KC (2020) Detecting straggler mapreduce tasks in big data processing infrastructure by neural network. The Journal of Supercomputing. pp 1–25

  22. Jin H, Yang X, Sun X, Raicu I (2012) Adapt: Availability-aware mapreduce data placement for non-dedicated distributed computing. In: 2012 IEEE 32Nd international conference on distributed computing systems. IEEE, pp 516–525

  23. Lee CW, Hsieh KY, Hsieh SY, Hsiao HC (2014) A dynamic data placement strategy for hadoop in heterogeneous environments. Big Data Research 1:14–22

    Article  Google Scholar 

  24. Li H, Wei X, Fu Q, Luo Y (2014) Mapreduce delay scheduling with deadline constraint. Concurrency and Computation:, Practice and Experience 26(3):766–778

    Article  Google Scholar 

  25. Li Q, Cao Z, Ding W, Li Q (2020) A multi-objective adaptive evolutionary algorithm to extract communities in networks. Swarm and Evolutionary Computation 52:100629

    Article  Google Scholar 

  26. Li Q, Li L, Wang W, Li Q, Zhong J (2020) A comprehensive exploration of semantic relation extraction via pre-trained cnns. Knowl-Based Syst 194:105488

    Article  Google Scholar 

  27. Li Y, Yang Q, Lai S, Li B (2015) A new speculative execution algorithm based on c4. 5 decision tree for hadoop. In: International conference of young computer scientists, engineers and educators. Springer, pp 284–291

  28. Lin JCW, Li Y, Fournier-Viger P, Djenouri Y, Wang LSL (2019) Mining high-utility sequential patterns from big datasets. In: 2019 IEEE International conference on big data (big data). IEEE, pp 2674–2680

  29. Pandey V, Saini P (2020) A heuristic method towards deadline-aware energy-efficient mapreduce scheduling problem in hadoop yarn. Cluster Computing. pp 1–17

  30. Pedrycz W, Chen SM (2014) Information granularity, big data, and computational intelligence, vol. 8 Springer

  31. Saleti S, Subramanyam R (2019) A novel mapreduce algorithm for distributed mining of sequential patterns using co-occurrence information. Appl Intell 49(1):150–171

    Article  Google Scholar 

  32. Shvachko K, Kuang H, Radia S, Chansler R, et al. (2010) The hadoop distributed file system. In: MSST, vol 10, pp 1–10

  33. Sun X, He C, Lu Y (2012) Esamr: an enhanced self-adaptive mapreduce scheduling algorithm. In: 2012 IEEE 18Th international conference on parallel and distributed systems. IEEE, pp 148–155

  34. Tang S, Yu F (2021) Construction and verification of retinal vessel segmentation algorithm for color fundus image under bp neural network model. J Supercomput 77(4):3870–3884

    Article  Google Scholar 

  35. Vavilapalli VK, Murthy AC, Douglas C, Agarwal S, Konar M, Evans R, Graves T, Lowe J, Shah H, Seth S et al (2013) Apache hadoop yarn: Yet another resource negotiator. In: Proceedings of the 4th annual symposium on cloud computing, p 5. ACM

  36. Wang B, Jiang J, Yang G (2015) Actcap: Accelerating mapreduce on heterogeneous clusters with capability-aware data placement. In: 2015 IEEE Conference on computer communications (INFOCOM). IEEE, pp 1328–1336

  37. Wang S, Cong Y, Zhu H, Chen X, Qu L, Fan H, Zhang Q, Liu M (2020) Multi-scale context-guided deep network for automated lesion segmentation with endoscopy images of gastrointestinal tract. IEEE J Biomed Health Inf 25(2):514–525

    Article  Google Scholar 

  38. Wang T, Li J, Guo J (2021) A scalable parallel chinese online encyclopedia knowledge denoising method based on entry tags and spark cluster. Appl Intell 51(10):7573–7599

    Article  Google Scholar 

  39. White T (2012) Hadoop: The definitive guide. “O’Reilly Media Inc.”

  40. Whitehead BA, Kiech EL, Ali M (1990) Learning and diagnosing faults using neural networks

  41. Xie J, Yin S, Ruan X, Ding Z, Tian Y, Majors J, Manzanares A, Qin X (2010) Improving mapreduce performance through data placement in heterogeneous hadoop clusters. In: 2010 IEEE International symposium on parallel & distributed processing, workshops and phd forum (IPDPSW). IEEE, pp 1–9

  42. Xiong R, Du Y, Jin J, Luo J (2018) Hadaap: a hotness-aware data placement strategy for improving storage efficiency in heterogeneous hadoop clusters. Concurrency and Computation:, Practice and Experience 30(20):e4830

    Article  Google Scholar 

  43. Xu H, Lau WC (2016) Optimization for speculative execution in big data processing clusters. IEEE Trans Parallel Distrib Syst 28(2):530–545

    Google Scholar 

  44. Ying C, Huang Z, Ying C (2018) Accelerating the image processing by the optimization strategy for deep learning algorithm dbn. EURASIP J Wirel Commun Netw 2018(1):1–8

    Article  MathSciNet  Google Scholar 

  45. Zaharia M, Borthakur D, Sen Sarma J, Elmeleegy K, Shenker S, Stoica I (2010) Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling. In: Proceedings of the 5th european conference on computer systems, pp 265–278. ACM

  46. Zaharia M, Konwinski A, Joseph AD, Katz RH, Stoica I (2008) Improving mapreduce performance in heterogeneous environments. In: Osdi, vol 8, p 7

  47. Zhang X, Wu Y, Zhao C (2016) Mrheter: improving mapreduce performance in heterogeneous environments. Clust Comput 19(4):1691–1701

    Article  Google Scholar 

Download references

Funding

The authors would like to thank the Quality Improvement Program of All India Council for Technical Education(AICTE), India, to support the research.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Motilal Nehru National Institute of Technology Allahabad, Pryagraj, Uttar Pradesh, India

    Kamalakant Laxman Bawankule, Rupesh Kumar Dewang & Anil Kumar Singh

Authors
  1. Kamalakant Laxman Bawankule
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rupesh Kumar Dewang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anil Kumar Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kamalakant Laxman Bawankule.

Ethics declarations

Conflict of Interests

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bawankule, K.L., Dewang, R.K. & Singh, A.K. Early straggler tasks detection by recurrent neural network in a heterogeneous environment. Appl Intell 53, 7369–7389 (2023). https://doi.org/10.1007/s10489-022-03837-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03837-1

Keywords

Search

Navigation