Abstract
Job scheduling in hadoop is a hot topic, however, current research mainly focuses on the time optimization in scheduling. With the trend of providing hadoop as a service to the public or specified groups, more factors should be considered, such as time and cost. To solve this problem, we present a utility-driven share scheduling algorithm. Considering time and cost, algorithm offers a global optimization scheduling scheme according to the workload of the job. Furthermore, we present a model that can estimate job execute time by cost. Finally, we implement the algorithm and experiment it in a hadoop cluster.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Ghemawat, S., Gobioff, H., Leung, S.-T.: The Google file system. Operating Systems Review 37(5), 29–43 (2003)
Chang, F., et al.: Bigtable: a distributed storage system for structured data. ACM Transactions on Computer Systems 26(2), 4–30 (2008)
Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)
Hadoop: Open source implementation of MapReduce, http://hadoop.apache.org/
Hadoop CapacityScheduler, http://hadoop.apache.org/common/docs/current/capacity_scheduler.html
Hadoop FairScheduler, http://hadoop.apache.org/common/docs/current/fair_scheduler.html
Sandholm, T., Lai, K.: Dynamic Proportional Share Scheduling in Hadoop. In: Frachtenberg, E., Schwiegelshohn, U. (eds.) JSSPP 2010. LNCS, vol. 6253, pp. 110–131. Springer, Heidelberg (2010)
Xicheng, D., Ying, W., Huaming, L.: Scheduling Mixed Real-time and Non-real-time Applications in MapReduce Environment. In: 17th IEEE International Conference on Parallel and Distributed Systems (ICPADS), Tainan, pp. 9–16 (2011)
Polo, J., et al.: Performance-Driven Task Co-Scheduling for MapReduce Environments. In: 2010 IEEE/IFIP Network Operations and Management Symposium - NOMS, pp. 373–380 (2010)
Kc, K., Anyanwu, K.: Scheduling Hadoop Jobs to Meet Deadlines. In: Proceedings of the 2010 IEEE 2nd International Conference on Cloud Computing Technology and Science (CloudCom 2010), pp. 388–392 (2010)
You, H., Yang, C., Huang, J.: A load-aware scheduler for MapReduce framework in heterogeneous cloud environments. In: Proceedings of the ACM Symposium on Applied Computing, pp. 127–132 (2011)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Wan, C., Wang, C., Yuan, Y., Wang, H., Song, X. (2013). Utility-Driven Share Scheduling Algorithm in Hadoop. In: Guo, C., Hou, ZG., Zeng, Z. (eds) Advances in Neural Networks – ISNN 2013. ISNN 2013. Lecture Notes in Computer Science, vol 7952. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-39068-5_67
Download citation
DOI: https://doi.org/10.1007/978-3-642-39068-5_67
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-39067-8
Online ISBN: 978-3-642-39068-5
eBook Packages: Computer ScienceComputer Science (R0)