research-article

Open access

Continuous Integration for HPC with Github Actions and Tapis

Authors:

Benjamin Pachev,

Georgia Stuart,

Clint DawsonAuthors Info & Claims

PEARC '22: Practice and Experience in Advanced Research Computing 2022: Revolutionary: Computing, Connections, You

Article No.: 36, Pages 1 - 4

https://doi.org/10.1145/3491418.3535124

Published: 08 July 2022 Publication History

All formats PDF

Abstract

Continuous integration and deployment (CICD) are fundamental to modern software development. While many platforms such as GitHub and Atlassian provide cloud solutions for CICD, these solutions don’t fully meet the unique needs of high performance computing (HPC) applications. These needs include, but are not limited to, testing distributed memory and scaling studies, both of which require an HPC environment. We propose a novel framework for running CICD workflows on supercomputing resources. Our framework directly integrates with GitHub Actions and leverages TACC’s Tapis API for communication with HPC resources. The framework is demonstrated for PYthon Ocean PArticle TRAcking (PYOPATRA), an HPC application for Lagrangian particle tracking.

References

[1]

Chaminda Chandrasekara and Pushpa Herath. 2021. Introduction to GitHub Actions. In Hands-on GitHub Actions. Springer, 1–8.

[2]

Brian Fitzgerald and Klaas-Jan Stol. 2017. Continuous software engineering: A roadmap and agenda. Journal of Systems and Software 123 (2017), 176–189.

[3]

Dustin Heaton and Jeffrey C Carver. 2015. Claims about the use of software engineering practices in science: A systematic literature review. Information and Software Technology 67 (2015), 207–219.

Digital Library

[4]

Jez Humble and David Farley. 2010. Continuous delivery: reliable software releases through build, test, and deployment automation. Pearson Education.

[5]

Joshua Hursey, Ethan Mallove, Jeffrey M. Squyres, and Andrew Lumsdaine. 2007. An Extensible Framework for Distributed Testing of MPI Implementations. In Recent Advances in Parallel Virtual Machine and Message Passing Interface, Franck Cappello, Thomas Herault, and Jack Dongarra (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 64–72.

[6]

Jenkins 2022. Jenkins. Retrieved Mar 7, 2022 from https://jenkins.io/

[7]

Arne Johanson and Wilhelm Hasselbring. 2018. Software engineering for computational science: Past, present, future. Computing in Science & Engineering 20, 2 (2018), 90–109.

Digital Library

[8]

Samuel Khuvis, Zhi-Qiang You, Heechang Na, Scott Brozell, Eric Franz, Trey Dockendorf, Judith Gardiner, and Karen Tomko. 2019. A continuous integration-based framework for software management. In Proceedings of the Practice and Experience in Advanced Research Computing on Rise of the Machines (learning). 1–7.

Digital Library

[9]

Gregory M Kurtzer, Vanessa Sochat, and Michael W Bauer. 2017. Singularity: Scientific containers for mobility of compute. PloS one 12, 5 (2017), e0177459.

[10]

Veronica G Vergara Larrea, Wayne Joubert, and Chris Fuson. 2015. Use of continuous integration tools for application performance monitoring. Concurrency and Computation Practice and Experience on the Cray User Group (2015).

[11]

Mathias Meyer. 2014. Continuous Integration and Its Tools. IEEE Software 31, 3 (2014), 14–16. https://doi.org/10.1109/MS.2014.58

[12]

Ioannis K Moutsatsos, Imtiaz Hossain, Claudia Agarinis, Fred Harbinski, Yann Abraham, Luc Dobler, Xian Zhang, Christopher J Wilson, Jeremy L Jenkins, Nicholas Holway, 2017. Jenkins-CI, an open-source continuous integration system, as a scientific data and image-processing platform. SLAS DISCOVERY: Advancing Life Sciences R&D 22, 3 (2017), 238–249.

[13]

Puppet 2022. Puppet. Retrieved Mar 7, 2022 from https://jenkins.io/

[14]

Zebula Sampedro, Aaron Holt, and Thomas Hauser. 2018. Continuous integration and delivery for HPC: Using Singularity and Jenkins. In Proceedings of the Practice and Experience on Advanced Research Computing. 1–6.

Digital Library

[15]

Mojtaba Shahin, Muhammad Ali Babar, and Liming Zhu. 2017. Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices. IEEE Access 5(2017), 3909–3943. https://doi.org/10.1109/ACCESS.2017.2685629

[16]

Georgia Stuart and Benjamin Pachev. 2022. PYthon Ocean PArticle TRAcking (PYOPATRA). Retrieved Mar 7, 2022 from https://github.com/UT-CHG/PYOPATRA

[17]

Joe Stubbs, Richard Cardone, Mike Packard, Anagha Jamthe, Smruti Padhy, Steve Terry, Julia Looney, Joseph Meiring, Steve Black, Maytal Dahan, 2021. Tapis: An API platform for reproducible, distributed computational research. In Future of Information and Communication Conference. Springer, 878–900.

[18]

J. Towns, T. Cockerill, M. Dahan, I. Foster, K. Gaither, A. Grimshaw, V. Hazlewood, S. Lathrop, D. Lifka, G. D. Peterson, R. Roskies, J. R. Scott, and N. Wilkins-Diehr. 2014. XSEDE: Accelerating Scientific Discovery. Computing in Science & Engineering 16, 5 (Sept.-Oct. 2014), 62–74. https://doi.org/10.1109/MCSE.2014.80

[19]

Jake Tronge, Jieyang Chen, Patricia Grubel, Tim Randles, Rusty Davis, Quincy Wofford, Steven Anaya, and Qiang Guan. 2021. BeeSwarm: Enabling parallel scaling performance measurement in continuous integration for HPC applications. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). 1136–1140. https://doi.org/10.1109/ASE51524.2021.9678805

Digital Library

Cited By

Zhang YWu YChen TWang TLiu HWang HRoychoudhury APaiva AAbreu RStorey M(2024)How do Developers Talk about GitHub Actions? Evidence from Online Software Development CommunityProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3623327(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3623327

Recommendations

Continuous Integration and Delivery for HPC: Using Singularity and Jenkins
PEARC '18: Proceedings of the Practice and Experience on Advanced Research Computing: Seamless Creativity

Continuous integration, delivery, and deployment (CICD) is widely used in DevOps communities, as it allows for teams of all sizes to deploy rapidly-changing hardware and software resources quickly and confidently. In this paper, we will describe how ...
Enabling Workflow-Aware Scheduling on HPC Systems
HPDC '17: Proceedings of the 26th International Symposium on High-Performance Parallel and Distributed Computing

Scientific workflows are increasingly common in the workloads of current High Performance Computing (HPC) systems. However, HPC schedulers do not incorporate workflow-specific mechanisms beyond the capacity to declare dependencies between their jobs. ...
An Edge Service for Managing HPC Workflows
HUST'17: Proceedings of the Fourth International Workshop on HPC User Support Tools

Large experimental collaborations, such as those at the Large Hadron Collider at CERN, have developed large job management systems running hundreds of thousands of jobs across worldwide computing grids. HPC facilities are becoming more important to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

PEARC '22: Practice and Experience in Advanced Research Computing 2022: Revolutionary: Computing, Connections, You

July 2022

455 pages

ISBN:9781450391610

DOI:10.1145/3491418

Editors:
Julie Wernert
Indiana University
,
Alan Chalker
Ohio Supercomputer Center
,
Shava Smallen
San Diego Supercomputer Center, University of California, San Diego
,
Tabitha Samuel
University of Tennessee, Knoxville
,
JP Navarro
Argonne National Laboratory

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 July 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

KAUST

Conference

PEARC '22

Sponsor:

PEARC '22: Practice and Experience in Advanced Research Computing

July 10 - 14, 2022

MA, Boston, USA

Acceptance Rates

Overall Acceptance Rate 133 of 202 submissions, 66%

Upcoming Conference

PEARC '25

Sponsor:
sighpc
sighpc

Practice and Experience in Advanced Research Computing

July 20 - 24, 2025

Columbus , OH , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
468
Total Downloads

Downloads (Last 12 months)162
Downloads (Last 6 weeks)31

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang YWu YChen TWang TLiu HWang HRoychoudhury APaiva AAbreu RStorey M(2024)How do Developers Talk about GitHub Actions? Evidence from Online Software Development CommunityProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3623327(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3623327

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten