Skip to main content
Springer Nature Link
Log in

A Parallel Job Execution Time Estimation Approach Based on User Submission Patterns within Computational Grids

  • Published:
International Journal of Parallel Programming Aims and scope Submit manuscript

Abstract

Scheduling performance in computational grid can potentially benefit a lot from accurate execution time estimation for parallel jobs. Most existing approaches for the parallel job execution time estimation, however, require ample past job traces and the explicit correlations between the job execution time and the outer layout parameters such as the consumed processor numbers, the user-estimated execution time and the job ID, which are hard to obtain or reveal. This paper presents and evaluates a novel execution time estimation approach for parallel jobs, the user-behavior clustering for execution time estimation, which can give more accurate execution time estimation for parallel jobs through exploring the job similarity and revealing the user submission patterns. Experiment results show that compared to the state-of-art algorithms, our approach can improve the accuracy of the job execution time estimation up to 5.6 %, meanwhile the time that our approach spends on calculation can be reduced up to 3.8 %.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Carvalho, M., Brasileiro, F.: A user-based model of grid computing workloads. In: Proceedings of the 2012 ACM/IEEE 13th International Conference on Grid Computing, GRID ’12, pp. 40–48. IEEE Computer Society, Washington, DC, USA (2012). doi:10.1109/Grid.2012.13

  2. Castillo, C., Rouskas, G.N., Harfoush, K.: Efficient resource management using advance reservations for heterogeneous grids. In: Parallel and Distributed Processing, 2008. IPDPS 2008. IEEE International Symposium, pp. 1–12 (2008)

  3. Dan, T., Dan, T.: Using inaccurate estimates accurately. In: Frachtenberg, E., Schwiegelshohn, U. (eds.) Job Scheduling Strategies for Parallel Processing, Lecture Notes in Computer Science, vol. 6253, pp. 208–221. Springer, Berlin Heidelberg, Los Alamitos, CA, USA (2010)

  4. den Bossche, R.V., Vanmechelen, K., Broeckhove, J.: An evaluation of the benefits of fine-grained value-based scheduling on general purpose clusters. Future Gener. Comput. Syst. 27(1), 1–9 (2011). doi:10.1016/j.future.2010.06.009. http://www.sciencedirect.com/science/article/pii/S0167739X10001226

    Google Scholar 

  5. Dimitriadou, S., Karatza, H.: Job scheduling in a distributed system using backfilling with inaccurate runtime computations. In: International Conference on Complex, Intelligent and Software Intensive Systems (CISIS), 2010, pp. 329–336 (2010). doi:10.1109/CISIS.2010.65

  6. Downey, A.: Predicting queue times on space-sharing parallel computers. In: Proceedings of 11th International Parallel Processing Symposium, 1997, pp. 209–218 (1997). doi:10.1109/IPPS.1997.580894

  7. Elmroth, E., Tordsson, J.: Grid resource brokering algorithms enabling advance reservations and resource selection based on performance predictions. Future Gener. Comput. Syst. 24, 585–593 (2008)

    Article  Google Scholar 

  8. Elmroth, E., Tordsson, J.: A standards-based grid resource brokering service supporting advance reservations, coallocation, and cross-grid interoperability. Concurr. Comput. Pract. Experience 21(18), 2298–2335 (2009). doi:10.1002/cpe.v21:18

    Article  Google Scholar 

  9. Feitelson, D.: Experimental analysis of the root causes of performance evaluation results: a backfilling case study. IEEE Trans. Parallel Distrib. Systems 16(2), 175–182 (2005). doi:10.1109/TPDS.2005.18

    Article  Google Scholar 

  10. Feitelson, D.: The parallel workload archive (2010). http://www.cs.huji.ac.il/labs/parallel/workload/

  11. Feitelson, D.: Workload Modeling for Computer Systems Performance Evaluation, vol. 0.34. School of Computer Science and Engineering, The Hebrew University of Jerusalem 91904 Jerusalem, Israel (2012). [Online Book draft] www.cs.huji.ac.il/feit/wlmod/

  12. Iosup, A., Epema, D.H.J.: Grid computing workloads. IEEE Internet Comput. 15(2), 19–26 (2011)

    Article  Google Scholar 

  13. Klusavek, D., Rudova, H.: The importance of complete data sets for job scheduling simulations. In: Proceedings of the 15th International Conference on Job Scheduling Strategies for Parallel Processing, JSSPP10, pp. 132–153. Springer, Berlin, Heidelberg (2010)

  14. Lee, C.B., Snavely, A., Lee, C.B., Snavely, A.: On the user-scheduler dialogue: studies of user-provided runtime estimates and utility functions. Int. J. High Perform. Comput. Appl. 20, 495–506 (2006)

    Article  Google Scholar 

  15. Lee, C.B., Snavely, A.E.: Precise and realistic utility functions for user-centric performance analysis of schedulers. In: Proceedings of the 16th International Symposium on High Performance Distributed Computing, HPDC ’07, pp. 107–116. ACM, New York, NY, USA (2007). doi:10.1145/1272366.1272381

  16. Lee, S.B., Schwartzman, Y., Hardy, J., Snavely, A.: Are user runtime estimates inherently inaccurate? In: Feitelson, D., Rudolph, L., U. Schwiegelshohn (eds.) Job Scheduling Strategies for Parallel Processing: 10th International Workshop, JSSPP 2004, pp. 253–263. Springer (2005)

  17. Li, H.: Workload dynamics on clusters and grids. J. Supercomput. 47, 1–20 (2009). doi:10.1007/s11227-008-0189-x

    Article  MATH  Google Scholar 

  18. Liang, F., Ma, S., Luckow, A., Schnor, B.: Earliest start time estimation for advance reservation-based resource brokering within computational grids. In: International Symposium on Parallel and Distributed Processing with Applications (ISPA), 2010, pp. 8–15 (2010). doi:10.1109/ISPA.2010.30

  19. Liang, F., Ma, S., Luckow, A., Schnor, B.: Advance reservation-based computational resource brokering using earliest start time estimation. J. Comput. 7(6), 1329–1336 (2012)

    Article  Google Scholar 

  20. Machine Learning Group, The University of Waikato: WEKA 3 webpage (2013). http://www.cs.waikato.ac.nz/ml/weka/

  21. Nadeem, F., Prodan, R., Fahringer, T.: Benchmarking grid applications for performance and scalability predictions. In: Li, K.C., Hsu, C.H., Yang, L.T., Dongarra, J., Zima H. (eds.) Handbook of Research on Scalable Computing Technologies. IGI Global, Hershey, PA (2010). doi:10.4018/978-1-60566-661-7.ch005, http://www.igi-global.com/bookstore/TitleDetails.aspx?TitleId=498

  22. Nadeem, F., Yousaf, M., Prodan, R., Fahringer, T.: Soft benchmarks-based application performance prediction using a minimum training set. In: Second IEEE International Conference on e-Science and Grid Computing, 2006. e-Science ’06, p. 71 (2006). doi:10.1109/E-SCIENCE.2006.261155

  23. Oikonomakos, M., Christodoulopoulos, K., Varvarigos, E.: Profiling computation jobs in grid systems. In: Seventh IEEE International Symposium on Cluster Computing and the Grid, 2007. CCGRID 2007, pp. 197–204 (2007). doi:10.1109/CCGRID.2007.87

  24. Parallel and Distributd System Group, Technische Universiteit Delft: The grid workloads archive webpage (2010). http://gwa.ewi.tudelft.nl/pmwiki/

  25. Qu, C.: A grid advance reservation framework for co-allocation and co-reservation across heterogeneous local resource management systems. In: Wyrzykowski, R., Dongarra, J., Karczewski, K., Wasniewski, J. (eds.) 7th International Conference on Parallel Processing and Applied Mathematics, PPAM 2007, vol. 4967, pp. 770–779. Springer, Gdansk, Poland (2007)

  26. Ramírez-Alcaraz, J.M., Tchernykh, A., Yahyapour, R., Schwiegelshohn, U., Quezada-Pina, A., González-García, J.L., Hirales-Carbajal, A.: Job allocation strategies with user run time estimates for online scheduling in hierarchical grids. J. Grid Comput. 9(1), 95–116 (2011)

    Article  Google Scholar 

  27. Röblitz, T., Rzadca, K.: On the placement of reservations into job schedules. In: Nagel, W.E., Walter, W.V., Lehner, W. (eds.) Euro-Par 2006 Proceedings, Lecture Notes in Computer Science, vol. 4128. Springer, pp. 198–210 (2006)

  28. Roy, S., Mukherjee, N.: Adaptive execution of jobs in computational grid environment. J. Comput. Sci. Technol. 24, 925–938 (2009)

    Article  Google Scholar 

  29. Sarkar, M., Roy, S., Mukherjee, N.: Prediction of resource requirement using feedback on job execution performance. Procedia Comput. Sci. 3, 1271–1276 (2011)

    Article  Google Scholar 

  30. Sipser, M.: Introduction to the Theory of Computation (Ise). Thomson Course Technology, Stamford, CT (2006). http://www.google.com.tw/books?id=VJ1mQgAACAAJ

  31. Smith, W.: Prediction services for distributed computing. In: International Parallel and Distributed Processing Symposium, pp. 1–10. IEEE Computer Society, Los Alamitos, CA, USA (2007). doi:10.1109/IPDPS.2007.370276

  32. Smith, W., Foster, I., Taylor, V.: Predicting application run times with historical information. J. Parallel Distrib. Comput. 64, 1007–1016 (2004). doi:10.1016/j.jpdc.2004.06.008

    Article  MATH  Google Scholar 

  33. Takefusa, A., Nakada, H., Kudoh, T., Tanaka, Y., Sekiguchi, S.: GridARS: an advance reservation-based grid co-allocation framework for distributed computing and network resources. In: Frachtenberg, E., Schwiegelshohn, U. (eds.) 13th International Workshop on Job Scheduling Strategies for Parallel Processing, JSSPP 2007, Lecture Notes in Computer Science, vol. 4942, pp. 152–168. Springer, Seattle, WA, USA (2007)

  34. Talby, D., Tsafrir, D., Goldberg, Z., Feitelson, D.G.: Session-based, estimation-less, and information-less runtime prediction algorithms for parallel and grid job scheduling. Technical report 2006-77, School of Computer Science and Engineering, the Hebrew University, Jerusalem, Israel (2006)

  35. Tang, W., Desai, N., Buettner, D., Lan, Z.: Analyzing and adjusting user runtime estimates to improve job scheduling on the blue gene p. In: International Parallel and Distributed Processing Symposium, pp. 1–11. IEEE Computer Society, Los Alamitos, CA, USA (2010)

  36. Tong, Y., Chen, L., Ding, B.: Discovering threshold-based frequent closed itemsets over probabilistic data. In: IEEE 28th International Conference on Data Engineering (ICDE), 2012, pp. 270–281 (2012). doi:10.1109/ICDE.2012.51

  37. Tong, Y., Chen, L., Yu, P.S.: Ufimt: an uncertain frequent itemset mining toolbox. In: Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD’12, pp. 1508–1511. ACM, New York, NY, USA (2012). doi:10.1145/2339530.2339767

  38. Tran, N.M., Wolters, L.: Towards a profound analysis of bags-of-tasks in parallel systems and their performance impact. In: Proceedings of the 20th International Symposium on High Performance Distributed Computing, HPDC’11, pp. 111–122. ACM, New York, NY, USA (2011) doi:10.1145/1996130.1996148

Download references

Acknowledgments

This research work was supported by both the Self-conducted Exploratory Research Program from State Key Laboratory for Software Development Environment in China (No. SKLSDE-2013ZX-11) and the Special Program for Seism-Scientific Research in Public Interest “Research in Online Processing Technologies for Seismological Precursory Network Dynamic Monitoring and Products” (No. 201008002). Great thanks to the institutions that made the grid traces available through the Grid Workload Archive (GWA) and the GWA group for the initiative. Also thanks to Assistant Professor CHAO Wenhan from the College of Computer Science and Engineering, Beihang University, Beijing, China for his suggestions on data mining and Ph.D. candidate Carvalho Marcus from Universidade Federal de Campina Grande for his suggestions on the data analysis.

Author information

Authors and Affiliations

  1. The State Key Laboratory of Software Development Environment, Beihang University, Beijing, 100191, China

    Feng Liang, Yunzhen Liu, Hai Liu & Shilong Ma

  2. Institute of Computer Science, University of Potsdam, Potsdam, 14482, Germany

    Bettina Schnor

Authors
  1. Feng Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yunzhen Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hai Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shilong Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bettina Schnor
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Feng Liang.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Liang, F., Liu, Y., Liu, H. et al. A Parallel Job Execution Time Estimation Approach Based on User Submission Patterns within Computational Grids. Int J Parallel Prog 43, 440–454 (2015). https://doi.org/10.1007/s10766-013-0294-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10766-013-0294-1

Keywords

Search

Navigation