Abstract
Makespan is one for crucial factors to determine the performance of job scheduling in cloud data center, short makespan could lead to more job throughput and less energy consumption. In this paper, we study the joint task and data assignment problem to realized makespan minimization. We propose the data migration method to overcome the memory space limitation of servers, and realize better data locality for task execution. We conduct extensive simulations, and the simulation results show that our algorithm has significant improvement on makespan reduction.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Apach Hadoop. http://hadoop.apache.org
Ahmad, F., Chakradhar, S., Raghunathan, A., Vijaykumar, T.: Shufflewatcher: shuffle-aware scheduling in multi-tenant mapreduce clusters. In: USENIX ATC (2014)
Ananthanarayanan, G., Agarwal, S., Kandula, S., Greenberg, A., Stoica, I., Harlan, D., Harris, E.: Scarlett: coping with skewed content popularity in mapreduce clusters. In: EuroSys (2011)
Ananthanarayanan, G., Ghodsi, G., Wang, A., Borthakur, D., Kandula, S., Shenker, S., Stoica, I.: Pacman: coordinated memory caching for parallel jobs. In: USENIX NSDI (2012)
Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)
Eltabakh, M., Tian, Y., Ozcan, F., Gemulla, R., Krettek, A., McPherson, J.: Cohadoop: flexible data placement and its exploitation in hadoop. In: VLDB Endow (2011)
Grandl, R., Ananthanarayanan, G., Kandula, S., Rao, S., Akella, A.: Multi-resource packing for cluster schedulers. In: ACM SIGCOMM (2014)
Jalaparti, V., Bodik, P., Menache, I., Rao, S., Makarychev, K., Caesar, M.: Network-aware scheduling for data-parallel jobs: plan when you can. In: ACM SIGCOMM (2015)
Leung, J., Kelly, L., Anderson, J.H.: Handbook of Scheduling: Algorithms, Models, and Performance Analysis. CRC Press, Boca Raton (2004)
Maguluri, S.T., Srikant, R.: Scheduling jobs with unknown duration in clouds. In: IEEE INFOCOM (2013)
Tan, J., Meng, X., Zhang, L.: Coupling task progress for mapreduce resource-aware scheduling. In: IEEE INFOCOM (2013)
Verma, A., Cherkasova, L., Campbell, R.H.: Two sides of a coin: Optimizing the schedule of mapreduce jobs to minimize their makespan and improve cluster performance. In: IEEE MASCOTS (2012)
Wang, W., Zhu, K., Ying, L., Tan, J., Zhang, L.: Map task scheduling in mapreduce with data locality: throughput and heavy-traffic optimality. In: IEEE INFOCOM (2013)
Wolf, J., Rajan, D., Hildrum, K., Khandekar, R., Kumar, V., Parekh, S., Wu, K.L., Balmin, A.: Flex: a slot allocation scheduling optimizer for mapredcue workloads. In: ACM/IFIP/USENIX Middleware (2010)
Zaharia, M., Borthakur, D., Sarma, J.S., Elmeleegy, K., Shenker, S., Stoica, I.: Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling. In: EuroSys (2010)
Acknowledgments
This work is supported in part by the Jiangsu Natural Science Foundation under Grant No. BK20160813, National High Technology Research and Development Program of China under Grant No. 2015AA015303, Project Funded by China Postdoctoral Science Foundation, Fundamental Research Funds for the Central Universities under Grant NO. NS2016097.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing AG
About this paper
Cite this paper
Li, X., Tang, C. (2016). Makespan Minimization for Batch Tasks in Data Centers. In: Wang, G., Ray, I., Alcaraz Calero, J., Thampi, S. (eds) Security, Privacy and Anonymity in Computation, Communication and Storage. SpaCCS 2016. Lecture Notes in Computer Science(), vol 10067. Springer, Cham. https://doi.org/10.1007/978-3-319-49145-5_12
Download citation
DOI: https://doi.org/10.1007/978-3-319-49145-5_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-49144-8
Online ISBN: 978-3-319-49145-5
eBook Packages: Computer ScienceComputer Science (R0)