skip to main content
10.1145/3332186.3332219acmotherconferencesArticle/Chapter ViewAbstractPublication PagespearcConference Proceedingsconference-collections
research-article

A Continuous Integration-Based Framework for Software Management

Published: 28 July 2019 Publication History

Abstract

Automated software-installation tools such as EasyBuild and Spack have enabled HPC support staff to more quickly deploy software environments for HPC. However, to provide reliable user-facing software environments, any change made to the environments should be recorded and tested. In this paper, we will describe a continuous integration-based framework for software management in use at the Ohio Supercomputer Center (OSC). In this framework, we rely on homegrown tools for automated software build and installation and the ReFrame framework to setup the testing system which can be triggered on commit to a local Gitlab repository. We will document the workflow and explain how we can deploy the software environment quickly and confidently using these tools.

References

[1]
Ohio Supercomputer Center. 1987. Ohio Supercomputer Center. http://osc.edu/ark:/19495/f5s1ph73
[2]
Ohio Supercomputer Center. 2015. Ruby Supercomputer. http://osc.edu/ark:/19495/hpc93fc8
[3]
Ohio Supercomputer Center. 2016. Owens Supercomputer. http://osc.edu/ark:/19495/hpc6h5b1
[4]
Ohio Supercomputer Center. 2018. Pitzer Supercomputer. http://osc.edu/ark:/19495/hpc56htp
[5]
Kevin Colby, Amiya K. Maji, Jason Rahman, and Joseph Bottum. 2017. Testpilot: A Flexible Framework for User-centric Testing of HPC Clusters. In Proceedings of the Fourth International Workshop on HPC User Support Tools (HUST'17). ACM, New York, NY, USA, Article 5, 10 pages.
[6]
Todd Gamblin. 2019. Spack: A package manager for HPC. Retrieved April 4, 2019 from htttps://www.youtube.com/watch?v=P5c__EkiuSk Campus Champion Tech Talk Series.
[7]
Todd Gamblin, Matthew LeGendre, Michael R. Collette, Gregory L. Lee, Adam Moody, Bronis R. de Supinski, and Scott Futral. 2015. The Spack Package Manager: Bringing Order to HPC Software Chaos. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC '15). ACM, New York, NY, USA, Article 40, 12 pages.
[8]
Markus Geimer, Kenneth Hoste, and Robert Mclay. 2014. Modern Scientific Software Management Using EasyBuild and Lmod. 41--51.
[9]
Gitlab. 2019. Gitlab Webhooks Documentation. Retrieved 2019 from https://docs.gitlab.com/ee/user/project/integrations/webhooks.html
[10]
Vasileios Karakasis, Victor H. Rusu, Andreas Jocksch, Jean-Guillaume Piccinali, and Guilherme Peretti-Pezzi. 2017. A regression framework for checking the health of large HPC systems. In Proceedings of the Cray User Group Meeting (CUG 2017). Redmond, Washington. https://cug.org/proceedings/cug2017_proceedings/includes/files/pap122s2-file1.pdf
[11]
Robert McLay, Karl W.Schulz, William L.Barth, and Tommy Minyard. 2011. Best Practices for the Deployment and Management of Production HPC Clusters. In State of the Practice Reports (SC '11). ACM, New York, NY, USA, Article 9, 11 pages.
[12]
Zebula Sampedro, Aaron Holt, and Thomas Hauser. 2018. Continuous Integration and Delivery for HPC: Using Singularity and Jenkins. In Proceedings of the Practice and Experience on Advanced Research Computing (PEARC '18). ACM, New York, NY, USA, Article 6, 6 pages.
[13]
Dan Terpstra, Heike Jagode, Haihang You, and Jack Dongarra. 2010. Collecting Performance Data with PAPI-C. In Tools for High Performance Computing 2009. Springer Berlin Heidelberg, Berlin, Heidelberg, 157--173.

Cited By

View all
  • (2023)Agnostic Performance Testing FrameworkProceedings of the 2023 9th International Conference on Computer Technology Applications10.1145/3605423.3605436(179-183)Online publication date: 10-May-2023
  • (2023)Semi-Automatic Hybrid Software Deployment Workflow in a Research Computing CenterPractice and Experience in Advanced Research Computing 2023: Computing for the Common Good10.1145/3569951.3593607(68-74)Online publication date: 23-Jul-2023
  • (2022)Continuous Integration for HPC with Github Actions and TapisPractice and Experience in Advanced Research Computing 2022: Revolutionary: Computing, Connections, You10.1145/3491418.3535124(1-4)Online publication date: 8-Jul-2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
PEARC '19: Practice and Experience in Advanced Research Computing 2019: Rise of the Machines (learning)
July 2019
775 pages
ISBN:9781450372275
DOI:10.1145/3332186
  • General Chair:
  • Tom Furlani
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 July 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. automation
  2. continuous integration
  3. software management

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

PEARC '19

Acceptance Rates

Overall Acceptance Rate 133 of 202 submissions, 66%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)12
  • Downloads (Last 6 weeks)0
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Agnostic Performance Testing FrameworkProceedings of the 2023 9th International Conference on Computer Technology Applications10.1145/3605423.3605436(179-183)Online publication date: 10-May-2023
  • (2023)Semi-Automatic Hybrid Software Deployment Workflow in a Research Computing CenterPractice and Experience in Advanced Research Computing 2023: Computing for the Common Good10.1145/3569951.3593607(68-74)Online publication date: 23-Jul-2023
  • (2022)Continuous Integration for HPC with Github Actions and TapisPractice and Experience in Advanced Research Computing 2022: Revolutionary: Computing, Connections, You10.1145/3491418.3535124(1-4)Online publication date: 8-Jul-2022
  • (2020)HPC Software Tracking Strategies for a Diverse Workload2020 IEEE/ACM International Workshop on HPC User Support Tools (HUST) and Workshop on Programming and Performance Visualization Tools (ProTools)10.1109/HUSTProtools51951.2020.00008(1-9)Online publication date: Nov-2020
  • (2020)Enabling Continuous Testing of HPC Systems Using ReFrameTools and Techniques for High Performance Computing10.1007/978-3-030-44728-1_3(49-68)Online publication date: 26-Mar-2020

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media