Abstract
Hadoop is a well-known parallel computing framework for processing large-scale data, but there is such a task in the Hadoop framework called the “Straggling task” and has a serious impact on Hadoop. Speculative execution is an efficient method of processing “Straggling Tasks” by monitoring the real-time rate of running tasks and backing up “Straggler” on another node to increase the chance of an early completion of a backup task. The proposed speculative execution strategy has many problems, such as misjudgement of “Straggling task” and improper selection of backup nodes, which leads to inefficient implementation of speculative execution. This paper proposes a hybrid resource scheduling strategy in speculative execution based on non-cooperative game theory (HRSE), which transforms the resource scheduling of backup task in speculative execution into a multi-party non-cooperative game problem. The backup task group is the game participant and the game strategy is the computing node, the utility function is the overall task execution time of the cluster. When the game reaches the Nash equilibrium state, the final resource scheduling scheme is obtained. Finally, we implemented the strategy in Hadoop-2.6.0, experimental results show that the scheduling scheme can guarantee the efficiency of speculative execution and improve the fault-tolerant performance of the computation under the condition of high cluster load.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Armbrust, M., et al.: A view of cloud computing. Commun. ACM 53(4), 50–58 (2010)
Mell, P., Grance, T.: The NIST definition of cloud computing. Natl. Inst. Stand. Technol. 53(6), 50 (2011)
Kong, Y., Zhang, M., Ye, D., et al.: An intelligent agent‐based method for task allocation in competitive cloud environments. Concurr. Comput. Pract. Exp. 6, e4178 (2017)
Kong, Y., Zhang, M., Ye, D.: An auction-based approach for group task allocation in an open network environment. Comput. J. 59(3), 403–422 (2016)
Apache Hadoop. http://Hadoop.Apache.Org/. Accessed 11 Feb 2018
Ghemawat, S., Gobioff, H., Leung, S.T.: The Google file system. ACM SIGOPS Oper. Syst. Rev. 37(5), 29–43 (2003)
Dean, J., Ghemawa, S.: MapReduce: simplified data processing on large clusters. Proc. Oper. Syst. Des. Implentation 51(1), 107–113 (2004)
Apache Hive. https://hive.apache.org/. Accessed 11 Mar 2018
Vijayalakshmi, B., Ravi, P.R.: The down of big Data-Hbase. In: IT in Business, Industry and Government, pp. 1–4. IEEE (2015)
Chang, F., Dean, J., Ghemawa, S.: A distributed storage system for structured data. ACM Trans. Comput. Syst. 26(2), 1–26 (2008)
Toshniwal, A., Taneja, S., Shukla, A., et al.: Storm@ Twitter. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, pp. 147–156. ACM (2014)
Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., Stoica, I.: Spark: cluster computing with working sets. the USENIX Conference on Hot Topics in Cloud Computing, USENIX Association, pp. 1765–1773 (2010)
Isard, M., Budiu, M., Yu, Y., Birrel, A., Fetterly, D.: Dryad: distributed data-parallel programs from sequential building blocks. In: Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems, pp. 59–72. ACM (2007)
Yoo, D.G., Sim, K.M.: A comparative review of job scheduling for MapReduce. In: IEEE International Conference on Cloud Computing and Intelligence Systems (CCIS), pp. 353–358. IEEE (2011)
Dinu, F., Ng, T.S.E.: Understanding the effects and implications of compute node related failures in Hadoop. In: International Symposium on High-Performance Parallel and Distributed Computing, pp. 187–198. ACM (2012)
Nenavath, S.N., Atul, N.: A review of adaptive approaches to MapReduce scheduling in heterogeneous environments. In: International Conference on Advances in Computing, Communications and Informatics, pp. 677–683. IEEE (2014)
Liu, Q., Jin, D., Liu, X., Linge, N.: A survey of speculative execution strategy in MapReduce. In: Sun, X., Liu, A., Chao, H.-C., Bertino, E. (eds.) ICCCS 2016. LNCS, vol. 10039, pp. 296–307. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-48671-0_27
Zaharia, M., Konwinski, A., Joseph, A., Katz, R., Stoica, I.: Improving MapReduce performance in heterogeneous environments. In: Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation (OSDI), pp. 29–42 (2008)
Huang, X., Zhang, L.X., Li, R.F., Wan, L.J., Li, K.Q.: Novel Heuristic speculative execution strategies in heterogeneous distributed environments. In: Computers and Electrical Engineering (2015)
Chen, Q., Liu, C., Xiao, Z.: Improving MapReduce performance using smart speculative execution strategy. IEEE Trans. Comput. 63(4), 954–967 (2014)
Wu, H.C., Li, K., Tang, Z., Zhang, L.: A heuristic speculative execution strategy in heterogeneous distributed environments. In: Sixth International symposium on Parallel Architectures, Algorithms and Programming (PAAP), pp. 268–273 (2014)
Liu, Q., Cai, W., Shen, J., Fu, Z., Linge, N.: A smart strategy for speculative execution based on hardware resource in a heterogeneous distributed environment. Int. J. Grid Distrib. Comput. 9, 203–214 (2015)
Wang, Y., Lu, W., Lou, R., Wei, B.: Improving MapReduce performance with partial speculative execution. J. Grid Comput. 13(4), 587–604 (2015)
Liu, Q., Cai, W., Shen, J., Fu, Z., Linge, N.: A speculative approach to spatial-temporal efficiency with multi-objective optimization in a heterogeneous cloud environment. Secur. Commun. Netw. 9(17), 4002–4012 (2016)
Li, Y., Yang, Q., Lai, S., Li, B.: A new speculative execution algorithm based on C4.5 decision tree for Hadoop. In: Wang, H., et al. (eds.) ICYCSEE 2015. CCIS, vol. 503, pp. 284–291. Springer, Heidelberg (2015). https://doi.org/10.1007/978-3-662-46248-5_35
Yang, S., Chen, Y.: Design adaptive task allocation scheduler to improve MapReduce performance in heterogeneous clouds. J. Netw. Comput. Appl. 57, 61–70 (2015)
Guo, Y., Rao, J., Jiang, C., Zhou, X.: Moving Hadoop into the cloud with flexible slot management and speculative execution. IEEE Trans. Parallel Distrib. Syst. 28(3), 798–812 (2017)
Acknowledgements
This work has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No. 701697, Major Program of the National Social Science Fund of China (Grant No. 17ZDA092) and the PAPD fund.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Dannah, W., Liu, Q., Jin, D. (2018). A Hybrid Resource Scheduling Strategy in Speculative Execution Based on Non-cooperative Game Theory. In: Sun, X., Pan, Z., Bertino, E. (eds) Cloud Computing and Security. ICCCS 2018. Lecture Notes in Computer Science(), vol 11063. Springer, Cham. https://doi.org/10.1007/978-3-030-00006-6_8
Download citation
DOI: https://doi.org/10.1007/978-3-030-00006-6_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-00005-9
Online ISBN: 978-3-030-00006-6
eBook Packages: Computer ScienceComputer Science (R0)