Abstract
As the rapid advancement and diversity in the computing systems, it is demanding to take the most robust scheduling algorithm that guarantee an optimized performance to execute manifold applications on large-scale heterogeneous computing environments. This paper present an adaptive application-aware job scheduling optimization strategy for large-scale high throughput computing in heterogeneous infrastructures. The proposed scheduling optimization method is built on two main concepts. First, it provides application-aware job distribution weights through empirical data in large-scale heterogeneous infrastructures. Here we adopt the concept of weight, which represent the ratio of tasks that will be computed on each resource. The weights can vary in terms of application type and are optimized until the system get in steady status. Secondly, it offers an adaptive control phase that is invoked by the weight adjustment and resource scaling feature. The feedback data from monitoring module is forwarded to the control phase in order to adjust weights and over-provisioning ratio, and result in enhancing overall balance between performances and utilization of system. The experimental evaluation with the four realistic workload patterns demonstrates that, when compared to the core-based scheme which distributes tasks in proportions of each resource’s number of cores, the use of our optimization method can achieve 62 % better average throughput, 43 % shorter average queueing time, and 38 % better average utilization of the entire resources in diverse infrastructure environments by harnessing our adaptive module.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Autodock. http://autodock.scripps.edu/
Baumgärtner, A., Burkitt, A., Ceperley, D., De Raedt, H., Ferrenberg, A., Heermann, D., Herrmann, H., Landau, D., Levesque, D., von der Linden, W., et al.: The Monte Carlo method in condensed matter physics, vol. 71. Springer, Berlin (2012)
Berman, F., Wolski, R., Casanova, H., Cirne, W., Dail, H., Faerman, M., Figueira, S., Hayes, J., Obertelli, G., Schopf, J., et al.: Adaptive computing on the grid using apples. IEEE Trans. Parallel Distrib. Syst. 14(4), 369–382 (2003)
Botón-Fernández, M., Vega-Rodríguez, M.A., Prieto Castrillo, F.: Intelligent self-adaptive resources selection for grid applications. Concurr. Comput.: Pract. Exp. 27, 3539–3560 (2014)
Bryk, P., Maciej, M., Juve, G., Deelman, E.: Storage-aware algorithms for scheduling of workflow ensembles in clouds. Grid Comput 14, 359–378 (2015)
Bu, X., Rao, J., Xu, C.: Interference and locality-aware task scheduling for mapreduce application in virtual clusters. In: HPDC’13 Proceedings of the 22nd international symposium on High-Performance parallel and distributed computing, pp. 227–238 (2013)
Chang, R.S., Lin, C.Y., Lin, C.F.: An adaptive scoring job scheduling algorithm for grid computing. Inf. Sci. 207, 79–89 (2012). doi:10.1016/j.ins.2012.04.019. http://www.sciencedirect.com/science/article/pii/S0020025512002836
Foster, I., Kesselman, C.: Globus: a metacomputing infrastructure toolkit. Int. J. High Perform. Comput. Appl. 11(2), 115–128 (1997)
Frey, J., Tannenbaum, T., Livny, M., Foster, I., Tuecke, S.: Condor-g: a computation management agent for multi-institutional grids. Clust. Comput. 5(3), 237–246 (2002)
Gentzsch, W.: Sun grid engine: towards creating a compute power grid. In: First IEEE/ACM International Symposium on Cluster Computing and the Grid, IEEE Proceedings, pp. 35–36 (2001)
Gupta, A., Kale, L.V., Milojicic, D., Faraboschi, P., Balle, S.M.: Hpc-aware vm placement in infrastructure clouds. In: Cloud Engineering (IC2E), 2013 IEEE International Conference on pp. 11–20 (2013)
Mateescu, G., Gentzsch, W., Ribbens, C.J.: Hybrid computingG where hpc meets grid and cloud computing. Future Gener. Comput. Syst. 27(5), 440–453 (2011)
Moca, M., Litan, C., Silaghi, G.C., Fedak, G.: Multi-criteria and satisfaction oriented scheduling for hybrid distributed computing infrastructures. Future Gener. Comput. Syst. 55, 428–443 (2015)
Montage. http://montage.ipac.caltech.edu/
Openstack Nova. https://wiki.openstack.org/wiki/Nova
Rho, S., Kim, S., Kim, S., Kim, S., Kim, J.S., Hwang, S.: Htcaas: a large-scale high-throughput computing by leveraging grids, supercomputers and cloud. In: High Performance Computing, Networking, Storage and Analysis (SCC), IEEE 2012 SC Companion, pp. 1341–1342 (2012)
Sjstrand, T., Mrenna, S., Pythia, P.S.: 6.4 physics and manual 050262006
Thaha, A.F., Singh, M., Amin, A.H., Ahmad, N.M., Kannan, S.: Hadoop in openstack: data-location-aware cluster provisioning. In: Fourth World Congress on Information and Communication Technologies (WICT) 2014, pp. 296–301 (2014)
The Galfa-hi Survey: Data release 1. Astrophys. J. Suppl. 194(2), 13 (2011)
Yang, L., Dai, Y.: Application scheduling in cloud computing environment with the consideration of performance interference. J. Commun. 10(8), 603–609 (2015). doi:10.12720/jcm.10.8.603-609
Acknowledgments
This research was supported by Next-Generation Information Computing Development Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT & Future Plannig (2015M 3C 4A7065646r) and also supported by Institute for Information & communications Technology Promotion (IITP) grant funded by the Korea government (MSIP) (No. R0190-15-2012, High Performance Big Data Analytics Platform Performance Acceleration Technologies Development).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Kim, S., Choi, J. & Kim, Y. Adaptive application-aware job scheduling optimization strategy in heterogeneous infrastructures. Cluster Comput 19, 1515–1526 (2016). https://doi.org/10.1007/s10586-016-0588-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10586-016-0588-3