Abstract
Management of computational infrastructure is a complicated task which, often employs user workloads delivery across multiple clusters. Criteria for such tasks distribution may vary: priority, transport costs, utilization of data, node capabilities, etc.
Such process happens to tasks devoted to the simulation and analysis of the results of high-energy physics experiments at CERN. For task distribution on massive data streams obtained during ATLAS experiment, “Production ANd Distributed Analysis system” (PanDA) was developed. It performs management of workloads delivery and execution in a geographically distributed cluster environment. This paper is devoted to the deployment of PanDA server in a private cluster setting.
This paper presents architecture and its implementation that allows, to run and embed PanDA system into existing computational solutions. It consists of a container, that isolates PanDA server its dependencies and environment from other system processes and an embedded Web interface which simplifies task management for end-users. In other words, our approach is focused on PanDA system deployment speed up by means of security layer simplification, containerization and stateless client web service implementation. System was tested on a heterogeneous geographically distributed Azure cloud nodes.
O. Iakushkin—This research was partially supported by Russian Foundation for Basic Research grant (project no. 16-07-01113).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bogdanov, A.V., Degtyarev, A., Stankova, E.N.: Example of a potential grid technology application in shipbuilding. In: 2007 International Conference on Computational Science and Its Applications (ICCSA 2007), pp. 3–8 (2007)
Borodin, M., De, K., Garcia, J., Golubkov, D., Klimentov, A., Maeno, T., Vaniachine, A., et al.: Scaling up ATLAS production system for the LHC run 2 and beyond: project ProdSys2. J. Phys. Conf. Ser. 664, 062005 (2015). IOP Publishing
De, K., Klimentov, A., Maeno, T., Nilsson, P., Oleynik, D., Panitkin, S., Petrosyan, A., Schovancova, J., Vaniachine, A., Wenaus, T.: The future of panda in atlas distributed computing. J. Phys. Conf. Ser. 664, 062035 (2015). IOP Publishing
Dworak, A., Ehm, F., Charrue, P., Sliwinski, W.: The new cern controls middleware. J. Phys. Conf. Ser. 396, 012017 (2012). IOP Publishing
Gankevich, I., Gaiduchok, V., Gushchanskiy, D., Tipikin, Y., Korkhov, V., Degtyarev, A., Bogdanov, A., Zolotarev, V.: Virtual private supercomputer: design and evaluation. In: Ninth International Conference on Computer Science and Information Technologies Revised Selected Papers, pp. 1–6 (2013)
Gankevich, I., Korkhov, V., Balyan, S., Gaiduchok, V., Gushchanskiy, D., Tipikin, Y., Degtyarev, A., Bogdanov, A.: Constructing virtual private supercomputer using virtualization and cloud technologies. In: Murgante, B., et al. (eds.) ICCSA 2014. LNCS, vol. 8584, pp. 341–354. Springer, Cham (2014). doi:10.1007/978-3-319-09153-2_26
Grishkin, V., Iakushkin, O.: Middleware transport architecture monitoring: topology service. In: 2014 20th International Workshop on Beam Dynamics and Optimization (BDO), pp. 1–2 (2014)
Iakushkin, O.: Cloud middleware combining the functionalities of message passing and scaling control. In: EPJ Web of Conferences, vol. 108 (2016)
Iakushkin, O., Grishkin, V.: Messaging middleware for cloud applications: extending brokerless approach. In: 2014 2nd International Conference on Emission Electronics (ICEE), pp. 1–4 (2014)
Iakushkin, O., Sedova, O., Valery, G.: Application control and horizontal scaling in modern cloud middleware. In: Gavrilova, M.L., Tan, C.J.K. (eds.) Transactions on Computational Science XXVII. LNCS, vol. 9570, pp. 81–96. Springer, Heidelberg (2016). doi:10.1007/978-3-662-50412-3_6
Iakushkin, O., Grishkin, V.: Unification of control in p2p communication middleware: towards complex messaging patterns. In: AIP Conference Proceedings, vol. 1648, no. 1, p. 040004 (2015)
Iakushkin, O., Shichkina, Y., Sedova, O.: Petri nets for modelling of message passing middleware in cloud computing environments. In: Gervasi, O., et al. (eds.) ICCSA 2016. LNCS, vol. 9787, pp. 390–402. Springer, Cham (2016). doi:10.1007/978-3-319-42108-7_30
Johnston, W.E., Dart, E., Ernst, M., Tierney, B.: Enabling high throughput in widely distributed data management and analysis systems: lessons from the LHC. In: TERENA Networking Conference (TNC) (2013)
Klimentov, A., Buncic, P., De, K., Jha, S., Maeno, T., Mount, R., Nilsson, P., Oleynik, D., Panitkin, S., Petrosyan, A., et al.: Next generation workload management system for big data on heterogeneous distributed computing. J. Phys. Conf. Ser. 608, 012040 (2015). IOP Publishing
Korenkov, V., Pelevanyuk, I., Zrelov, P., Tsaregorodtsev, A.: Accessing distributed computing resources by scientific communities using dirac services (2016)
Maeno, T., De, K., Klimentov, A., Nilsson, P., Oleynik, D., Panitkin, S., Petrosyan, A., Schovancova, J., Vaniachine, A., Wenaus, T., et al.: Evolution of the atlas panda workload management system for exascale computational science. J. Phys. Conf. Ser. 513, 032062 (2014). IOP Publishing
Maeno, T., De, K., Panitkin, S.: Pd2p: Panda dynamic data placement for atlas. J. Phys. Conf. Ser. 396, 032070 (2012). IOP Publishing
Maeno, T.: Panda: distributed production and distributed analysis system for atlas. J. Phys. Conf. Ser. 119, 062036 (2008). IOP Publishing
Maeno, T., De, K., Wenaus, T., Nilsson, P., Stewart, G., Walker, R., Stradling, A., Caballero, J., Potekhin, M., Smith, D., et al.: Overview of atlas panda workload management. J. Phys. Conf. Ser. 331, 072024 (2011). IOP Publishing
Shichkina, Y., Degtyarev, A., Gushchanskiy, D., Iakushkin, O.: Application of optimization of parallel algorithms to queries in relational databases. In: Gervasi, O., et al. (eds.) ICCSA 2016. LNCS, vol. 9787, pp. 366–378. Springer, Cham (2016). doi:10.1007/978-3-319-42108-7_28
Acknowledgments.
This research was partially supported by Russian Foundation for Basic Research grant (project no. 16-07-01113). Microsoft Azure for Research Award (http://research.microsoft.com/en-us/projects/azure/) as well as the resource center “Computer Center of SPbU” (http://cc.spbu.ru/en) provided computing resources. The authors would like to acknowledge the Reviewers for the valuable recommendations that helped in the improvement of this paper.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Iakushkin, O., Malevanniy, D., Bogdanov, A., Sedova, O. (2017). Adaptation and Deployment of PanDA Task Management System for a Private Cloud Infrastructure. In: Gervasi, O., et al. Computational Science and Its Applications – ICCSA 2017. ICCSA 2017. Lecture Notes in Computer Science(), vol 10408. Springer, Cham. https://doi.org/10.1007/978-3-319-62404-4_32
Download citation
DOI: https://doi.org/10.1007/978-3-319-62404-4_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-62403-7
Online ISBN: 978-3-319-62404-4
eBook Packages: Computer ScienceComputer Science (R0)