Abstract
Efficient resource management to improve the throughput in large-scale cluster has become a research focus with the rapid development of applications of Big Data. YARN (Yet Another Resource Negotiator), as the new generation of resource management system in Hadoop, is more efficient in resource utilization and capable of handling more kinds of workload than previous systems. Due to the fact that a task usually occupies more resources than it actually uses during some stage of its life cycle, a relevant amount of resource is idle and can not be allocated to satisfy the requirements of pending tasks. In order to address the deficiencies of resource allocation in YARN, this paper presents a high concurrent elastic resource allocation strategy named Ballon, which can dynamically adjust the configured resource of a node depending on the actual resource utilization of the node. Moreover, Ballon classifies resource requests of applications into different types. Consequently the elastic resources can be allocated to proper request. Our experiments demonstrate that Ballon cluster can reduce the average execution time of application by at least 10% in most MapReduce application and can increase the resource utilization of cluster.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Shvachko, K., Kuang, H., Radia, S., et al.: The Hadoop distributed file system, pp. 1–10 (2010)
Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. In: Conference on Symposium on Operating Systems Design & Implementation. DBLP (2004)
Frampton, M.: Storing and configuring data with Hadoop, YARN, and ZooKeeper. In: Frampton, M. (ed.) Big Data Made Easy, pp. 11–56. Apress, Berkeley (2014). https://doi.org/10.1007/978-1-4842-0094-0_2
Huang, W., Meng, L., Zhang, D., et al.: In-memory parallel processing of massive remotely sensed data using an Apache Spark on Hadoop YARN model. IEEE J. Sel. Topics Appl. Earth Obs. Remote Sens. 10, 1–17 (2017)
Vavilapalli, V.K., Murthy, A.C., Douglas, C., et al.: Apache Hadoop YARN: yet another resource negotiator. In: Symposium on Cloud Computing, p. 5 (2013)
Eadline, D.: Preface to Apache Hadoop YARN: Moving beyond MapReduce and Batch Processing with Apache Hadoop 2. Pearson Schweiz Ag, Zug (2014)
Liu, Y.: High availability of network service on docker container. In: International Conference on Measurement, Instrumentation and Automation (2016)
Lee, S., Bae, M.-H., Eum, J.-H., Oh, S.: Efficient vCore based container deployment algorithm for improving heterogeneous Hadoop YARN performance. In: Kim, K., Joukov, N. (eds.) ICISA 2017. LNEE, vol. 424, pp. 191–201. Springer, Singapore (2017). https://doi.org/10.1007/978-981-10-4154-9_23
Genkin, M., Dehne, F., Pospelova, M., Chen, Y., Navarro, P.: Automatic, on-line tuning of YARN container memory and CPU parameters. In: IEEE 2nd International Conference on Data Science and Systems (HPCC/SmartCity/DSS), Sydney, NSW, pp. 317–324 (2016)
Ding, X., Liu, Y., Qian, D.: JellyFish: online performance tuning with adaptive configuration and elastic container in Hadoop YARN. In: 2015 IEEE 21st International Conference on Parallel and Distributed Systems (ICPADS), Melbourne, VIC, p. 831 (2015)
Shao, Y., Li, C., Dong, W., Liu, Y.: Energy-aware dynamic resource allocation on Hadoop YARN cluster. In: IEEE 2nd International Conference on Data Science and Systems (HPCC/SmartCity/DSS), Sydney, NSW, pp. 364–371 (2016)
Yao, Y., Gao, H., Wang, J., Mi, N., Sheng, B.: OpERA: opportunistic and efficient resource allocation in Hadoop YARN by harnessing idle resources. In: 2016 25th International Conference on Computer Communication and Networks (ICCCN), Waikoloa, HI, pp. 1–9 (2016)
Zhao, Y., Wu, G.: Yadoop: an elastic resource management solution of YARN. In: 2015 IEEE Symposium on Service-Oriented System Engineering, San Francisco Bay, CA (2015)
Lin, J., Lee, M.: Performance evaluation of job schedulers on Hadoop YARN. Concur. Comput. Pract. Exp. 28(9), 2711–2728 (2016)
Shah, P.: Adaptive application master for elastic web server farms for cloud based on Hadoop YARN. In: International Conference on Cloud Computing and Big Data, pp. 461–446 (2013)
Kakantousis, T.: Scaling YARN: a distributed resource manager for Hadoop (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
About this paper
Cite this paper
Peng, Y., Luo, D., Dong, J., Wu, Z. (2018). High Concurrent Elastic Resource Allocation in Hadoop YARN. In: Li, B., Shu, L., Zeng, D. (eds) Communications and Networking. ChinaCom 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 236. Springer, Cham. https://doi.org/10.1007/978-3-319-78130-3_54
Download citation
DOI: https://doi.org/10.1007/978-3-319-78130-3_54
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-78129-7
Online ISBN: 978-3-319-78130-3
eBook Packages: Computer ScienceComputer Science (R0)