Workflow Submit Nodes as a Service on Leadership Class Systems
DOE scientists, today, have access to high performance computing (HPC) facilities with very powerful systems that enable them to execute their computations faster, more efficiently, and at greater scales than ever before. To further their knowledge and produce new discoveries, scientists rely on workflows - sometimes very complex - that provide them with an easy way to automate, reproduce and verify their computations. However, historically, creating workflow submission environments in large HPC facilities has been cumbersome, requires expertise and many man-hours of effort due to the peculiarities, policies, and the restrictions that these systems present. In this paper we discuss the approach a large DOE facility (OLCF) is taking in order to provide containers as a service to its users. This capability is used to create Pegasus workflow management system submit nodes as a service (WSaaS) at the Oak Ridge Leadership Computing Facilities (OLCF), targeting the Summit supercomputer. This deployment builds upon the Kubernetes/Openshift cluster (Slate) that exists within OLCF’s DMZ and its automation triggers. Additionally, we evaluate our approach’s overhead and effort to deploy the solution as compared to previous solutions, such as setting up a Pegasus submission environment on OLCF’s login nodes or submitting jobs remotely via the rvGAHP.
- Research Organization:
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
- DOE Contract Number:
- AC05-00OR22725
- OSTI ID:
- 1665960
- Resource Relation:
- Conference: PEARC '20: Practice and Experience in Advanced Research Computing - Portland, Oregon, United States of America - 7/27/2020 4:00:00 AM-7/31/2020 4:00:00 AM
- Country of Publication:
- United States
- Language:
- English
Similar Records
US Department of Energy, Office of Science High Performance Computing Facility Operational Assessment 2021: Oak Ridge Leadership Computing Facility
US Department of Energy, Office of Science High Performance Computing Facility Operational Assessment 2019 Oak Ridge Leadership Computing Facility