Abstract
With the advent of big data and cloud computing era, scheduling and executing large-scale computing tasks effectively and allocating resources to tasks reasonably are becoming a quite challenging problem. And there is theoretical significance to research on efficient scheduling algorithm to improve resource utilization and task execution efficiency. We present a scheduling algorithm based on resource attribute selection (RAS) by sending a set of test tasks to an execution node to determine its resource attributes before a task is scheduled; and then selecting the optimal node to execute a task according to its resource requirements and the fitness between the resource node and the task, which also uses history task data if exists. We (1) give a formal definition of the resource attributes and (2) compute the fitness of the resource nodes and (3) store the information of node selection for next round. We integrate our algorithm into the Gearman scheduling framework, and through comparison with three other scheduling frameworks, we find out there is significant improvement in resource selection and resource utilization using RAS. The throughput of the RAS (with work-stealing, WS) is at least 30% higher than the other frameworks and the resource utilization of RAS (WS) reaches 0.94. The algorithm can make a good model for practical large scale application scheduling.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Foster, I., Zhao, Y., Raicu, I., et al.: Cloud computing and grid computing 360-degree compared. In: Proceedings of the 2008 Grid Computing Environments Workshop, pp. 1–10. IEEE Computer Society, Washington, DC (2008)
Ilavarasan, E., Thambidurai, P., Mahilmannan, R.: Performance effective task scheduling algorithm for heterogeneous computing system. In: Proceedings of the Fourth International Symposium on Parallel and Distributed Computing, Lille, France, pp. 28–38 (2005)
Beman, F., Fox, G., Tony, H.: Grid Computing-making the Global Infrastructure a Reality, pp. 65–80. John Wiley and Sons Ltd, USA (2003)
Zhao, Y., Raicu, I., Foster, I.: Scientific Workflow Systems for 21st Century e-Science, New Bottle or New Wine?, Invited Paper. In: IEEE Workshop on Scientific Workflows 2008, Co-located with IEEE International Conference on Services Computing, SCC (2008)
Zhao, Y., Raicu, I., Foster, I., et al.: Realizing Fast, Scalable and Reliable Scientific Computations in Grid Environments. In: Grid Computing Research Progress. Nova Publisher (2008) ISBN: 978-1-60456-404-4
Raicu, I., Zhao, Y., Dumitrescu, C., Foster, I., Wilde, M.: Falkon: A Fast and Light-weight tasK executiON Framework. IEEE/ACM SC (2007)
Gearman (2013), http://gearman.org/
Ousterhout, K., Wendell, P., Zaharia, M., Stoica, I.: Batch Sampling: Low Overhead Scheduling for Sub-Second Parallel Jobs. Under Submission
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauley, M., Franklin, M.J., Shenker, S., Stoica, I.: Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. In: Proc. NSDI (2012)
Hindman, B., Konwinski, A., Zaharia, M., Ghodsi, A., Joseph, A.D., Katz, R.H., Shenker, S., Stoica, I.: Mesos: A platform for ne-grained resource sharing in the data center. Technical Report UCB/EECS-2010-87, EECS Department, University of California, Berkeley (2010)
YARN (2013), http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html
Liu, C., Zhao, Z., Liu, F.: An Insight into the Architecture of Condor - A Distributed Scheduler. In: International Symposium on Computer Network and Multimedia Technology, CNMT 2009, pp. 1–4 (2009)
Tannenbaum, T., Wright, D., Miller, K., Livny, M.: Condor - A Distributed Job Scheduler. In: Sterling, T. (ed.) Beowulf Cluster Computing with Linux. The MIT Press (2002) ISBN: 0-262-69274-0
Thain, D., Tannenbaum, T., Livny, M.: Distributed Computing in Practice: The Condor Experience. Concurrency and Computation: Practice and Experience 17(2-4), 323–356 (2005)
Coleman, N.: Distributed Policy Specification and Interpretation with Classified Advertisements. In: Russo, C., Zhou, N.-F. (eds.) PADL 2012. LNCS, vol. 7149, pp. 198–211. Springer, Heidelberg (2012)
HTCondor (2013), http://research.cs.wisc.edu/htcondor/
Ousterhout, K., Wendell, P., Zaharia, M., Stoica, I.: Sparrow: Scalable Scheduling for Sub-Second Parallel Jobs. Technical Report No. UCB/EECS-2013-29 (2013)
LSF (2013), http://en.wikipedia.org/wiki/Platform_LSF
Xu, M.Q.: Effective metacomputing using LSF Multicluster. In: Proceedings of the First IEEE/ACM International Symposium on Cluster Computing and the Grid, pp. 100–105 (2001)
Costen, F., Brooke, J., Pettipher, M.: Investigation to make best use of LSF with high efficiency. In: Proceedings of the 1st IEEE Computer Society International Workshop on Cluster Computing, Melbourne, Vic, pp. 211–220 (1999)
Day, E., Aker, B.: Gearman: Bringing the Power of Map/Reduce to Everyday Applications (Slides). In: OSCON 2009 (2009)
Kaya, K., Aykanat, C.: Iterative-Improvement-Based Heuristics for Adaptive Scheduling of Tasks Sharing Files on Heterogeneous Master-Slave Environments. IEEE Transactions on Parallel and Distributed Systems 17(8), 883–896 (2006)
He, X., Sun, X., von Laszewski, G.: QoS guided Min-Min heuristic for grid task scheduling. Journal of Computer Science and Technology 18(4), 442–451 (2003)
Yanchun, W.: On Gene Expression Programming Algorithm and its Application. Computer Applications and Software 27(6), 23–26 (2010)
Abdulal, W., Ramachandram, S.: Reliability-Aware Scheduling Based on a Novel Simulated Annealing in Grid. In: 2012 Fourth International Conference on Computational Intelligence and Communication Networks (CICN), pp. 665–670 (2012)
Lu, B., Zhang, H.: Grid Load Balancing Scheduling Algorithm Based on Statistics Thinking. In: The 9th International Conference for Young Computer Scientists, pp. 288–292 (2008)
Ku-Mahamud, K.R., Nasir, H.J.A.: Ant Colony Algorithm for Job Scheduling in Grid Computing. In: 2010 Fourth Asia International Conference on Mathematical/Analytical Modelling and Computer Simulation (AMS), pp. 40–45 (2010)
Darmawan, I., Kuspriyanto; Priyana, Y., Joseph, M.I.: Grid computing process improvement through computing resource scheduling using genetic algorithm and Tabu Search integration. In: 2012 7th International Conference on Telecommunication Systems, Services, and Applications (TSSA), pp. 330–334 (2012)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Zhao, Y., Chen, L., Li, Y., Liu, P., Li, X., Zhu, C. (2013). RAS: A Task Scheduling Algorithm Based on Resource Attribute Selection in a Task Scheduling Framework. In: Pathan, M., Wei, G., Fortino, G. (eds) Internet and Distributed Computing Systems. IDCS 2013. Lecture Notes in Computer Science, vol 8223. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-41428-2_9
Download citation
DOI: https://doi.org/10.1007/978-3-642-41428-2_9
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-41427-5
Online ISBN: 978-3-642-41428-2
eBook Packages: Computer ScienceComputer Science (R0)