skip to main content
10.1145/2616498.2616534acmotherconferencesArticle/Chapter ViewAbstractPublication PagesxsedeConference Proceedingsconference-collections
research-article

Launcher: A Shell-based Framework for Rapid Development of Parallel Parametric Studies

Published: 13 July 2014 Publication History

Abstract

Petascale computing systems have enabled tremendous advances for traditional simulation and modeling algorithms that are built around parallel execution. Unfortunately, scientific domains using data-oriented or high-throughput paradigms have difficulty taking full advantage of these resources without custom software development. This paper describes our solution for rapidly developing parallel parametric studies using sequential or threaded tasks: The launcher. We detail how to get ensembles executing quickly through common job schedulers SGE and SLURM, and the various user-customizable options that the launcher provides. We illustrate the efficiency of or tool by presenting execution results at large scale (over 65,000 cores) for varying workloads, including a virtual screening workload with indeterminate runtimes using the drug docking software Autodock Vina.

References

[1]
Texas Advanced Computing Center (TACC). http://www.tacc.utexas.edu.
[2]
Wolfgang Gentzsch. Sun grid engine: Towards creating a compute power grid. In Cluster Computing and the Grid, 2001. Proceedings. First IEEE/ACM International Symposium on, pages 35--36. IEEE, 2001.
[3]
Michael J Litzkow, Miron Livny, and Matt W Mutka. Condor-a hunter of idle workstations. In Distributed Computing Systems, 1988., 8th International Conference on, pages 104--111. IEEE, 1988.
[4]
Marc Snir. Mpi the Complete Reference: The Mpi Core, volume 1. MIT press, 1998.
[5]
Texas Advanced Computing Center. Stampede User Guide. http://www.tacc.utexas.edu/user-services/user-guides/stampede-user-guide, 2013.
[6]
Texas Advanced Computing Center (TACC). Stampede's Comprehensive Capabilities to Bolster U.S. Open Science Computational Resources. http://www.tacc.utexas.edu/news/press-releases/2011/stampede, 2011.
[7]
Sudha Uda Thiagarajan, Charles Congdon, Sumedh Naik, and Loc Q Nguyen. Intel®Xeon Phi#8482; Coprocessor Developer's Quick Start Guide. http://software.intel.com/en-us/articles/intel-xeon-phi-coprocessor-developers-quick-start-guide.
[8]
Oleg Trott and Arthur J Olson. Autodock vina: improving the speed and accuracy of docking with a new scoring function, efficient optimization, and multithreading. Journal of computational chemistry, 31(2):455--461, 2010.
[9]
A Tsaregorodstev et al. Dirac-distributed infrastructure with remote agent conntrol. In Proc. of CHEP2003, 2003.
[10]
Andy B Yoo, Morris A Jette, and Mark Grondona. Slurm: Simple linux utility for resource management. In Job Scheduling Strategies for Parallel Processing, pages 44--60. Springer, 2003.
[11]
Yong Zhao, Mihael Hategan, Ben Clifford, Ian Foster, Gregor Von Laszewski, Veronika Nefedova, Ioan Raicu, Tiberiu Stef-Praun, and Michael Wilde. Swift: Fast, reliable, loosely coupled parallel computation. In Services, 2007 IEEE Congress on, pages 199--206. IEEE, 2007.
[12]
Songnian Zhou. Lsf: Load sharing in large heterogeneous distributed systems. In I Workshop on Cluster Computing, 1992.

Cited By

View all
  • (2025)a priori uncertainty quantification of reacting turbulence closure models using Bayesian neural networksEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.109821141(109821)Online publication date: Feb-2025
  • (2021)Tools and Guidelines for Job Bundling on Modern SupercomputersPractice and Experience in Advanced Research Computing 2021: Evolution Across All Dimensions10.1145/3437359.3465569(1-8)Online publication date: 17-Jul-2021
  • (2019)The impact of photometric redshift errors on lensing statistics in ray-tracing simulationsMonthly Notices of the Royal Astronomical Society10.1093/mnras/stz1016486:2(2730-2753)Online publication date: 18-Apr-2019
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
XSEDE '14: Proceedings of the 2014 Annual Conference on Extreme Science and Engineering Discovery Environment
July 2014
445 pages
ISBN:9781450328937
DOI:10.1145/2616498
  • General Chair:
  • Scott Lathrop,
  • Program Chair:
  • Jay Alameda
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

In-Cooperation

  • NSF: National Science Foundation
  • Drexel University
  • Indiana University: Indiana University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2014

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Parametric studies
  2. Scalable applications
  3. Software frameworks

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

XSEDE '14

Acceptance Rates

XSEDE '14 Paper Acceptance Rate 80 of 120 submissions, 67%;
Overall Acceptance Rate 129 of 190 submissions, 68%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)0
  • Downloads (Last 6 weeks)0
Reflects downloads up to 01 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2025)a priori uncertainty quantification of reacting turbulence closure models using Bayesian neural networksEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.109821141(109821)Online publication date: Feb-2025
  • (2021)Tools and Guidelines for Job Bundling on Modern SupercomputersPractice and Experience in Advanced Research Computing 2021: Evolution Across All Dimensions10.1145/3437359.3465569(1-8)Online publication date: 17-Jul-2021
  • (2019)The impact of photometric redshift errors on lensing statistics in ray-tracing simulationsMonthly Notices of the Royal Astronomical Society10.1093/mnras/stz1016486:2(2730-2753)Online publication date: 18-Apr-2019
  • (2017)Launcher: A simple tool for executing high throughput computing workloadsThe Journal of Open Source Software10.21105/joss.002892:16(289)Online publication date: Aug-2017
  • (2017)mD3DOCKxbProceedings of the 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing10.1109/CCGRID.2017.131(671-676)Online publication date: 14-May-2017
  • (2016)Computational science education focused on future domain scientistsProceedings of the Workshop on Education for High Performance Computing10.5555/3018088.3018092(19-24)Online publication date: 13-Nov-2016
  • (2016)Computational Science Education Focused on Future Domain Scientists2016 Workshop on Education for High-Performance Computing (EduHPC)10.1109/EduHPC.2016.008(19-24)Online publication date: Nov-2016
  • (2016)Using Managed High Performance Computing Systems for High-Throughput ComputingConquering Big Data with High Performance Computing10.1007/978-3-319-33742-5_4(61-79)Online publication date: 17-Sep-2016
  • (2015)NCBI-BLAST programs optimization on XSEDE resources for sustainable aquacultureProceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure10.1145/2792745.2792749(1-5)Online publication date: 26-Jul-2015

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media