Trey Dockendorf
ABSTRACT
The Ohio Supercomputer Center has deployed a Kubernetes cluster with tight integration to a high performance computing (HPC) environment. This deployment leverages existing file systems for data sharing between HPC systems and Kubernetes objects, monitoring, account management, resource management, and accounting systems. This paper describes the motivation and overall design, the novel methods for the implementation, and the applications supported by this new resource. It also presents a short description of future work and some of the questions raised by this design.
- Harbor Authors. 2022. Harbor. https://goharbor.io/Google Scholar
- Helm Authors. 2022. Helm. https://helm.sh/Google Scholar
- The Kubernetes Authors. 2022. Kubernetes: Production-Grade Container Orchestration. https://kubernetes.io/Google Scholar
- Joe Breen, Lincoln Bryant, Gabriele Carcassi, Jiahui Chen, Robert W Gardner, Ryan Harden, Martin Izdimirski, Robert Killen, Ben Kulbertis, Shawn McKee, 2018. Building the SLATE Platform. In Proceedings of the Practice and Experience on Advanced Research Computing. 1–7.Google ScholarDigital Library
- Ewa Deelman, Karan Vahi, Gideon Juve, Mats Rynge, Scott Callaghan, Philip J. Maechling, Rajiv Mayani, Weiwei Chen, Rafael Ferreira da Silva, Miron Livny, and Kent Wenger. 2015. Pegasus, a workflow management system for science automation. Future Generation Computer Systems 46 (2015), 17–35. https://doi.org/10.1016/j.future.2014.10.008Google ScholarDigital Library
- Paolo Di Tommaso, Maria Chatzou, Evan W Floden, Pablo Prieto Barja, Emilio Palumbo, and Cedric Notredame. 2017. Nextflow enables reproducible computational workflows. Nature biotechnology 35, 4 (2017), 316–319.Google Scholar
- Trey Dockendorf. 2022. k8-ldap-configmap. https://github.com/OSC/k8-ldap-configmapGoogle Scholar
- Red Hat. 2022. Keycloak. https://www.keycloak.org/Google Scholar
- David E. Hudak, Thomas Bitterman, Patricia Carey, Douglas Johnson, Eric Franz, Shaun Brady, and Piyush Diwan. 2013. OSC OnDemand: A Web Platform Integrating Access to HPC Systems, Web and VNC Applications. In Proceedings of the Conference on Extreme Science and Engineering Discovery Environment: Gateway to Discovery(XSEDE ’13). ACM, New York, NY, USA, Article 49, 6 pages. https://doi.org/10.1145/2484762.2484780Google ScholarDigital Library
- Johannes Köster and Sven Rahmann. 2012. Snakemake – a scalable bioinformatics workflow engine. Bioinformatics 28, 19 (2012), 2520–2522.Google ScholarDigital Library
- Kubernetes. 2022. Kubernetes Pod Security Standards. https://kubernetes.io/docs/concepts/security/pod-security-standards/Google Scholar
- Gregory M Kurtzer, Vanessa Sochat, and Michael W Bauer. 2017. Singularity: Scientific containers for mobility of compute. PloS one 12, 5 (2017), e0177459.Google Scholar
- Kyverno. 2022. Kyverno. https://kyverno.io/Google Scholar
- letsencrypt. 2022. Let’s Encrypt. https://letsencrypt.org/Google Scholar
- George Papadimitriou, Karan Vahi, Jason Kincl, Valentine Anantharaj, Ewa Deelman, and Jack Wells. 2020. Workflow Submit Nodes as a Service on Leadership Class Systems. In Practice and Experience in Advanced Research Computing. 56–63.Google Scholar
- Prometheus. 2022. Prometheus. https://prometheus.io/Google Scholar
- Robert E Settlage, Alan Chalker, Jeff Ohrstrom, Eric Franz, Doug Johnson, and David Hudak. 2021. Open OnDemand as a Platform for Virtual Learning in Higher Education. In Proceedings of Sixth International Congress on Information and Communication Technology: ICICT 2021, London, Vol. 3. Springer, 323–331.Google Scholar
- Preston M Smith, Erik Gough, Alexander Younts, Brian Werts, Thomas J Hacker, Norbert Neumeister, and Jennifer Wisecaver. 2020. The “Geddes” Composable Platform-An Evolution of Community Clusters for a Composable World. In 2020 IEEE/ACM International Workshop on Interoperability of Supercomputing and Cloud Technologies (SuperCompCloud). IEEE, 33–38.Google Scholar
Recommendations
Enabling dynamic and intelligent workflows for HPC, data analytics, and AI convergence
AbstractThe evolution of High-Performance Computing (HPC) platforms enables the design and execution of progressively larger and more complex workflow applications in these systems. The complexity comes not only from the number of elements ...
Highlights- Analysis of the HPC, Big Data and AI convergence in complex scientific workflows.
Workflows and e-Science: An overview of workflow system features and capabilities
Scientific workflow systems have become a necessary tool for many applications, enabling the composition and execution of complex analysis on distributed resources. Today there are many workflow systems, often with overlapping functionality. A key issue ...
The Organization and Management of Grid Infrastructures
Grid computing technology has become fundamental to e-Science. As the virtual organizations established by scientific communities progress from testing their applications to more routine usage, maintaining reliable and adaptive grid infrastructures ...
Comments