Abstract
In order to meet the need of setting deadline for Hadoop MapReduce job and improve resource utilization of Hadoop cluster, a resource scheduler based on collecting the running information of tasks is proposed. According to the information of resource usage, the progress of job, the deadline of job, and the handling time of job, we estimate the resource demand of jobs, and then schedule these jobs according to their resource demand. Meanwhile, a method to judge whether the resource of cluster can meet the deadline of all the jobs in cluster is proposed. When the jobs will miss the deadline under the allocated resources, scheduler applies to cloud platform for extra resources. Experimental results show the on demand resource scheduler can increase the utilization of resource in Hadoop cluster and approximately ensure the deadline of jobs.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
ApacheHadoop. http://hadoop.apache.org/
Capacity Scheduler. http://hadoop.apache.org/docs/r0.20.205.0/fairscheduler.html
Fair Scheduler. http://hadoop.apache.org/docs/r0.20.205.0/fairscheduler.html
Polo, J., Carrera, D., Becerra, Y., et al.: Performance-driven task co-scheduling for MapReduce environments. In: Network Operations and Management Symposium, pp. 373–380. IEEE (2010)
Ferguson, A.D., Bodik, P., Kandula, S., et al.: Jockey: guaranteed job latency in data parallel clusters. In: Eurosys Proceedings of the European Conference on Computer Systems, pp. 99–112 (2012)
Verma, A., Cherkasova, L., Campbell, R.H., et al.: ARIA: automatic resource inference and allocation for MapReduce environments. In: International Conference on Autonomic Computing (2011)
Lee, G., Chun, B., Katz, H., et al.: Heterogeneity-aware resource allocation and scheduling in the cloud. In: Cloud Computing (2011)
Zhang, W., Rajasekaran, S., Wood, T., et al.: MIMP: Deadline and interference aware scheduling of hadoop virtual machines. In: Cluster Computing and the Grid (2014)
Romer, T.: Autoscaling Hadoop Clusters. Master’s thesis of University of Tartu faculty of mathematics and computer science (2010)
Rao, B.T., Sridevi, N.V., Reddy, V.K., et al.: Performance issues of heterogeneous Hadoop clusters in cloud computing. Comput. Sci. (2012)
Ahmad, F., Chakradhar, S.T., Raghunathan, A., et al.: Tarazu: optimizing MapReduce on heterogeneous clusters. ACM Sigarch Comput. Archit. News 40(1), 61–74 (2012)
Cheng, D., Rao, J., Jiang, C.J., et al.: Resource and deadline-aware job scheduling in dynamic hadoop clusters, pp. 956–965 (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
About this paper
Cite this paper
Chen, L., Xu, J., Li, K., Lu, Z., Qi, Q., Wang, J. (2017). On Demand Resource Scheduler Based on Estimating Progress of Jobs in Hadoop. In: Wang, S., Zhou, A. (eds) Collaborate Computing: Networking, Applications and Worksharing. CollaborateCom 2016. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 201. Springer, Cham. https://doi.org/10.1007/978-3-319-59288-6_62
Download citation
DOI: https://doi.org/10.1007/978-3-319-59288-6_62
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-59287-9
Online ISBN: 978-3-319-59288-6
eBook Packages: Computer ScienceComputer Science (R0)