Abstract
Distributed computing systems are widely used for the execution of loosely coupled many-task applications, such as parameter sweeps, workflows, distributed optimization. These applications consist of a potentially large number of computational tasks that can be executed more or less independently. Since the application users often have an access to multiple computing resources, it is important to provide a convenient and efficient environment for execution of applications across the user-defined heterogeneous resource pools. The paper discusses the related challenges and presents an approach for solving them based on Everest, a web-based distributed computing platform. The presented solution supports reliable and efficient execution of many-task applications, while taking into account resource performance, adapting to queuing delays and providing a mechanism for communication between tasks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Everest. http://everest.distcomp.org/
Adam, G., et al.: IT-ecosystem of the HybriLIT heterogeneous platform for high-performance computing and training of IT-specialists. Distributed Computing and Grid-technologies in Science and Education. 2267, 638–644 (2018)
Afgan, E., Goecks, J., Baker, D., Coraor, N., Nekrutenko, A., Taylor, J.: Galaxy: a gateway to tools in e-science. In: Yang, X., Wang, L., Jie, W. (eds.) Guide to e-Science, pp. 145–177. Springer, London (2011)
Allen, G., Angulo, D., Foster, I., et al.: The Cactus Worm: experiments with dynamic resource discovery and allocation in a grid environment. Int. J. High Perform. Comput. Appl. 15(4), 345–358 (2001)
Casanova, H., Berman, F., Obertelli, G., Wolski, R.: The AppLeS parameter sweep template: User-level middleware for the grid. In: SC 2000: Proceedings of the 2000 ACM/IEEE Conference on Supercomputing, pp. 60–60. IEEE (2000)
Cirne, W., Brasileiro, F., Sauve, J., et al.: Grid computing for bag of tasks applications. In: Proceedings of the 3rd IFIP Conference on E-Commerce, E-Business and EGovernment. Citeseer (2003)
Dean, J., Barroso, L.A.: The tail at scale. Commun. ACM 56(2), 74–80 (2013)
Gamrath, G., Anderson, D., Bestuzheva, K., et al.: The SCIP Optimization Suite 7.0. Technical report, Optimization Online, March 2020
Huedo, E., Montero, R.S., Llorente, I.M.: A framework for adaptive execution in grids. Softw. Pract. Exp. 34(7), 631–651 (2004)
Kacsuk, P.: P-GRADE portal family for grid infrastructures. Concurrency Comput. Practice Exp. 23(3), 235–245 (2011)
McLennan, M., Kennell, R.: HUBzero: a platform for dissemination and collaboration in computational science and engineering. Comput. Sci. Eng. 12(2), 48–53 (2010)
Moscicki, J.T.: Diane - distributed analysis environment for grid-enabled simulation and analysis of physics data. In: 2003 IEEE Nuclear Science Symposium. Conference Record (IEEE Cat. No. 03CH37515), vol. 3, pp. 1617–1620. IEEE (2003)
Raicu, I., Foster, I.T., Zhao, Y.: Many-task computing for grids and supercomputers. In: 2008 Workshop on Many-Task Computing on Grids and Supercomputers, pp. 1–11. IEEE (2008)
Sfiligoi, I., Bradley, D.C., Holzman, B., Mhashilkar, P., Padhi, S., Wurthwein, F.: The pilot way to grid resources using glideinWMS. In: 2009 WRI World Congress on Computer Science and Information Engineering, vol. 2, pp. 428–432. IEEE (2009)
Smirnov, S., Voloshinov, V.: On domain decomposition strategies to parallelize branch-and-bound method for global optimization in Everest distributed environment. Procedia Comput. Sci. 136, 128–135 (2018)
Smirnov, S., Sukhoroslov, O., Volkov, S.: Integration and combined use of distributed computing resources with Everest. Procedia Comput. Sci. 101, 359–368 (2016)
Smirnov, S., Sukhoroslov, O., Voloshinov, V.: Using resources of supercomputing centers with everest platform. In: Voevodin, V., Sobolev, S. (eds.) RuSCDays 2018. CCIS, vol. 965, pp. 687–698. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-05807-4_59
Smirnov, S., Voloshinov, V.: Implementation of concurrent parallelization of branch-and-bound algorithm in Everest distributed environment. Procedia Comput. Sci. 119, 83–89 (2017)
Sokolov, A., Voloshinov, V.: Balanced identification as an intersection of optimization and distributed computing. arXiv preprint arXiv:1907.13444 (2019)
Sukhoroslov, O., Volkov, S., Afanasiev, A.: A web-based platform for publication and distributed execution of computing applications. In: 14th International Symposium on Parallel and Distributed Computing (ISPDC), pp. 175–184, June 2015
Sukhoroslov, O.: Integration of Everest platform with BOINC-based desktop grids. In: Third International Conference BOINC:FAST 2017, pp. 102–107 (2017)
Sukhoroslov, O.: Supporting Efficient Execution of Workflows on Everest Platform. In: Voevodin, V., Sobolev, S. (eds.) RuSCDays 2019. CCIS, vol. 1129, pp. 713–724. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-36592-9_58
Sukhoroslov, O.: Supporting efficient execution of many-task applications with Everest. In: Proceedings of GRID 2018, pp. 266–270 (2018)
Taylor, I.J., Deelman, E., Gannon, D.B., Shields, M.: Workflows for e-Science: Scientific Workflows for Grids. Springer Publishing Company, Incorporated (2014)
Thain, D., Tannenbaum, T., Livny, M.: Distributed computing in practice: the Condor experience. Concurrency Comput. Practice Exp. 17(2–4), 323–356 (2005)
Thomas, M., Burruss, J., Cinquini, L., et al.: Grid portal architectures for scientific applications. In: Journal of Physics: Conference Series, vol. 16, p. 596. IOP Publishing (2005)
Volkov, S., Sukhoroslov, O.: A generic web service for running parameter sweep experiments in distributed computing environment. Procedia Comput. Sci. 66, 477–486 (2015)
Volkov, S., Sukhoroslov, O.: Simplifying the use of clouds for scientific computing with Everest. Procedia Comput. Sci. 119, 112–120 (2017)
Voloshinov, V., Smirnov, S., Sukhoroslov, O.: Implementation and use of coarse-grained parallel branch-and-bound in Everest distributed environment. Procedia Comput. Sci. 108, 1532–1541 (2017)
Wächter, A., Biegler, L.: On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming. Math. Program. 106(1), 25–57 (2006)
Acknowledgments
This work is supported by the Russian Science Foundation (Project 16-11-10352 - Sects. 4 and 5.2) and the Russian Foundation for Basic Research (Project 20-07-00701 - Sect. 5.3, Project 18-07-00956 - all other sections). This research was supported in part through resources of supercomputer facilities provided by NRU HSE. This work has been carried out using computing resources of the federal collective usage center Complex for Simulation and Data Processing for Mega-science Facilities at NRC “Kurchatov Institute”. The research is carried out using the equipment of the shared research facilities of HPC computing resources at Lomonosov Moscow State University. Computations were held on the basis of the HybriLIT heterogeneous computing platform (LIT, JINR) [2].
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Sukhoroslov, O., Voloshinov, V., Smirnov, S. (2020). Running Many-Task Applications Across Multiple Resources with Everest Platform. In: Voevodin, V., Sobolev, S. (eds) Supercomputing. RuSCDays 2020. Communications in Computer and Information Science, vol 1331. Springer, Cham. https://doi.org/10.1007/978-3-030-64616-5_54
Download citation
DOI: https://doi.org/10.1007/978-3-030-64616-5_54
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-64615-8
Online ISBN: 978-3-030-64616-5
eBook Packages: Computer ScienceComputer Science (R0)