research-article

Launcher: A Shell-based Framework for Rapid Development of Parallel Parametric Studies

Authors:

Lucas A. Wilson,

John M. FonnerAuthors Info & Claims

XSEDE '14: Proceedings of the 2014 Annual Conference on Extreme Science and Engineering Discovery Environment

Article No.: 40, Pages 1 - 8

https://doi.org/10.1145/2616498.2616534

Published: 13 July 2014 Publication History

Get Access

Abstract

Petascale computing systems have enabled tremendous advances for traditional simulation and modeling algorithms that are built around parallel execution. Unfortunately, scientific domains using data-oriented or high-throughput paradigms have difficulty taking full advantage of these resources without custom software development. This paper describes our solution for rapidly developing parallel parametric studies using sequential or threaded tasks: The launcher. We detail how to get ensembles executing quickly through common job schedulers SGE and SLURM, and the various user-customizable options that the launcher provides. We illustrate the efficiency of or tool by presenting execution results at large scale (over 65,000 cores) for varying workloads, including a virtual screening workload with indeterminate runtimes using the drug docking software Autodock Vina.

References

[1]

Texas Advanced Computing Center (TACC). http://www.tacc.utexas.edu.

Google Scholar

[2]

Wolfgang Gentzsch. Sun grid engine: Towards creating a compute power grid. In Cluster Computing and the Grid, 2001. Proceedings. First IEEE/ACM International Symposium on, pages 35--36. IEEE, 2001.

Digital Library

Google Scholar

[3]

Michael J Litzkow, Miron Livny, and Matt W Mutka. Condor-a hunter of idle workstations. In Distributed Computing Systems, 1988., 8th International Conference on, pages 104--111. IEEE, 1988.

Crossref

Google Scholar

[4]

Marc Snir. Mpi the Complete Reference: The Mpi Core, volume 1. MIT press, 1998.

Digital Library

Google Scholar

[5]

Texas Advanced Computing Center. Stampede User Guide. http://www.tacc.utexas.edu/user-services/user-guides/stampede-user-guide, 2013.

Google Scholar

[6]

Texas Advanced Computing Center (TACC). Stampede's Comprehensive Capabilities to Bolster U.S. Open Science Computational Resources. http://www.tacc.utexas.edu/news/press-releases/2011/stampede, 2011.

Google Scholar

[7]

Sudha Uda Thiagarajan, Charles Congdon, Sumedh Naik, and Loc Q Nguyen. Intel®Xeon Phi#8482; Coprocessor Developer's Quick Start Guide. http://software.intel.com/en-us/articles/intel-xeon-phi-coprocessor-developers-quick-start-guide.

Google Scholar

[8]

Oleg Trott and Arthur J Olson. Autodock vina: improving the speed and accuracy of docking with a new scoring function, efficient optimization, and multithreading. Journal of computational chemistry, 31(2):455--461, 2010.

Google Scholar

[9]

A Tsaregorodstev et al. Dirac-distributed infrastructure with remote agent conntrol. In Proc. of CHEP2003, 2003.

Google Scholar

[10]

Andy B Yoo, Morris A Jette, and Mark Grondona. Slurm: Simple linux utility for resource management. In Job Scheduling Strategies for Parallel Processing, pages 44--60. Springer, 2003.

Google Scholar

[11]

Yong Zhao, Mihael Hategan, Ben Clifford, Ian Foster, Gregor Von Laszewski, Veronika Nefedova, Ioan Raicu, Tiberiu Stef-Praun, and Michael Wilde. Swift: Fast, reliable, loosely coupled parallel computation. In Services, 2007 IEEE Congress on, pages 199--206. IEEE, 2007.

Crossref

Google Scholar

[12]

Songnian Zhou. Lsf: Load sharing in large heterogeneous distributed systems. In I Workshop on Cluster Computing, 1992.

Google Scholar

Cited By

View all

Pash GHassanaly MYellapantula S(2025)a priori uncertainty quantification of reacting turbulence closure models using Bayesian neural networksEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.109821141(109821)Online publication date: Feb-2025
https://doi.org/10.1016/j.engappai.2024.109821
Liu SEijkhout VCazes J(2021)Tools and Guidelines for Job Bundling on Modern SupercomputersPractice and Experience in Advanced Research Computing 2021: Evolution Across All Dimensions10.1145/3437359.3465569(1-8)Online publication date: 17-Jul-2021
https://dl.acm.org/doi/10.1145/3437359.3465569
Abruzzo MHaiman Z(2019)The impact of photometric redshift errors on lensing statistics in ray-tracing simulationsMonthly Notices of the Royal Astronomical Society10.1093/mnras/stz1016486:2(2730-2753)Online publication date: 18-Apr-2019
https://doi.org/10.1093/mnras/stz1016
Show More Cited By

Index Terms

Launcher: A Shell-based Framework for Rapid Development of Parallel Parametric Studies
1. Computing methodologies
  1. Concurrent computing methodologies
    1. Concurrent programming languages
2. Software and its engineering
  1. Software notations and tools
    1. General programming languages
      1. Language types
        Concurrent programming languages

Recommendations

Applying workflow as a service paradigm to application farming

Task farming is often used to enable parameter sweep for exploration of large sets of initial conditions for large scale complex simulations. Such applications occur very often in life sciences. Available solutions enable to perform parameter sweep by ...
GridPACK^™: a framework for developing power grid simulations on high performance computing platforms
WOLFHPC '14: Proceedings of the Fourth International Workshop on Domain-Specific Languages and High-Level Frameworks for High Performance Computing

This paper describes the GridPACK^™framework, which is designed to help power grid engineers develop modeling software capable of running on high performance computers. The framework makes extensive use of software templates to provide high level ...
The GridPACK™ toolkit for developing power grid simulations on high performance computing platforms
HiPCNA-PG '13: Proceedings of the 3rd International Workshop on High Performance Computing, Networking and Analytics for the Power Grid

This paper describes the GridPACK™ framework, which is designed to help power grid engineers develop modeling software capable of running on high performance computers. The framework contains modules for setting up distributed power grid networks, ...

Comments

Information & Contributors

Information

Published In

XSEDE '14: Proceedings of the 2014 Annual Conference on Extreme Science and Engineering Discovery Environment

July 2014

445 pages

ISBN:9781450328937

DOI:10.1145/2616498

General Chair:
Scott Lathrop
National Center for Supercomputing Applications
,
Program Chair:
Jay Alameda
National Center for Supercomputing Applications

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

In-Cooperation

NSF: National Science Foundation
Drexel University
Indiana University: Indiana University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

XSEDE '14

XSEDE '14: Annual Conference of the Extreme Science and Engineering Discovery Environment

July 13 - 18, 2014

GA, Atlanta, USA

Acceptance Rates

XSEDE '14 Paper Acceptance Rate 80 of 120 submissions, 67%;

Overall Acceptance Rate 129 of 190 submissions, 68%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
109
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Pash GHassanaly MYellapantula S(2025)a priori uncertainty quantification of reacting turbulence closure models using Bayesian neural networksEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.109821141(109821)Online publication date: Feb-2025
https://doi.org/10.1016/j.engappai.2024.109821
Liu SEijkhout VCazes J(2021)Tools and Guidelines for Job Bundling on Modern SupercomputersPractice and Experience in Advanced Research Computing 2021: Evolution Across All Dimensions10.1145/3437359.3465569(1-8)Online publication date: 17-Jul-2021
https://dl.acm.org/doi/10.1145/3437359.3465569
Abruzzo MHaiman Z(2019)The impact of photometric redshift errors on lensing statistics in ray-tracing simulationsMonthly Notices of the Royal Astronomical Society10.1093/mnras/stz1016486:2(2730-2753)Online publication date: 18-Apr-2019
https://doi.org/10.1093/mnras/stz1016
A. Wilson LM. Fonner JAllison JEsteban OKenya HLerner M(2017)Launcher: A simple tool for executing high throughput computing workloadsThe Journal of Open Source Software10.21105/joss.002892:16(289)Online publication date: Aug-2017
https://doi.org/10.21105/joss.00289
Peng SZhang XYang SSu WZhang ZDong DLu KLu YLiao XSchmidt BZhu WLi K(2017)mD3DOCKxbProceedings of the 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing10.1109/CCGRID.2017.131(671-676)Online publication date: 14-May-2017
https://dl.acm.org/doi/10.1109/CCGRID.2017.131
Wilson LDey S(2016)Computational science education focused on future domain scientistsProceedings of the Workshop on Education for High Performance Computing10.5555/3018088.3018092(19-24)Online publication date: 13-Nov-2016
https://dl.acm.org/doi/10.5555/3018088.3018092
Wilson LDey S(2016)Computational Science Education Focused on Future Domain Scientists2016 Workshop on Education for High-Performance Computing (EduHPC)10.1109/EduHPC.2016.008(19-24)Online publication date: Nov-2016
https://doi.org/10.1109/EduHPC.2016.008
Wilson L(2016)Using Managed High Performance Computing Systems for High-Throughput ComputingConquering Big Data with High Performance Computing10.1007/978-3-319-33742-5_4(61-79)Online publication date: 17-Sep-2016
https://doi.org/10.1007/978-3-319-33742-5_4
Seetharam AGomez APurcell CHyde JBlood PSeverin APeterson G(2015)NCBI-BLAST programs optimization on XSEDE resources for sustainable aquacultureProceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure10.1145/2792745.2792749(1-5)Online publication date: 26-Jul-2015
https://dl.acm.org/doi/10.1145/2792745.2792749

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Applying workflow as a service paradigm to application farming

GridPACK™: a framework for developing power grid simulations on high performance computing platforms

The GridPACK™ toolkit for developing power grid simulations on high performance computing platforms

Comments

Information

Published In

In-Cooperation

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations

GridPACK^™: a framework for developing power grid simulations on high performance computing platforms