skip to main content
10.1145/2488551.2488561acmotherconferencesArticle/Chapter ViewAbstractPublication PageseurompiConference Proceedingsconference-collections
research-article

Dataflow coordination of data-parallel tasks via MPI 3.0

Published: 15 September 2013 Publication History

Abstract

Scientific applications are often complex collections of many large-scale tasks. Mature tools exist for describing task-parallel workflows consisting of serial tasks, and a variety of tools exist for programming a single data-parallel operation. However, few tools cover the intersection of these two models. In this work, we extend the load balancing library ADLB to support parallel tasks. We demonstrate how applications can easily be composed of parallel tasks using Swift dataflow scripts, which are compiled to ADLB programs with performance comparable to hand-coded equivalents. By combining this framework with data-parallel analysis libraries, we are able to dynamically execute many instances of a parallel data analysis application in support of a parameter exploration workload.

References

[1]
M. H. Arafat, J. Dinan, S. Krishnamoorthy, T. Windus, and P. Sadayappan. Load balancing of dynamical nucleation theory Monte Carlo simulations through resource sharing barriers. In Proc. 26th Intl. Parallel and Distributed Processing Symp., IPDPS '12, May 2012.
[2]
T. G. Armstrong, J. M. Wozniak, M. Wilde, and I. T. Foster. Compiler optimization for distributed dynamic data flow programs, 2013. MCS Tech. Report: ANL/MCS-P4045-0313 Under review for: Int'l Conference on Parallel Architectures and Compilation Techniques.
[3]
P. Balaprakash, S. M. Wild, and B. Norris. SPAPT: Search problems in automatic performance tuning. Procedia Computer Science, 9(0):1959--1968, 2012. Proceedings of the International Conference on Computational Science, ICCS 2012.
[4]
J. Dinan, D. Goodell, W. Gropp, R. Thakur, and P. Balaji. Efficient multithreaded context ID allocation in MPI. In Proc. Recent Adv. in MPI - 19th European MPI Users' Group Meeting, EuroMPI '12, September 2012.
[5]
J. Dinan, S. Krishnamoorthy, P. Balaji, J. R. Hammond, M. Krishnan, V. Tipparaju, and A. Vishnu. Noncollective communicator creation in MPI. In Recent Advances in the Message Passing Interface - 18th European MPI Users' Group Meeting, EuroMPI '11, 2011.
[6]
M. Hategan, J. Wozniak, and K. Maheshwari. Coasters: uniform resource provisioning and access for scientific computing on clouds and grids. In Proc. Utility and Cloud Computing, 2011.
[7]
J. Lofstead, F. Zheng, S. Klasky, and K. Schwan. Adaptable, metadata rich IO methods for portable high performance IO. In Proceedings of the 2009 IEEE International Symposium on Parallel&Distributed Processing, pages 1--10, Washington, DC, USA, 2009. IEEE Computer Society.
[8]
E. L. Lusk, S. C. Pieper, and R. M. Butler. More scalability, less pain: A simple programming model and its implementation for extreme computing. SciDAC Review, 17:30--37, January 2010.
[9]
E. Merzari, W. Pointer, A. Obabko, and P. Fischer. On the numerical simulation of thermal striping in the upper plenum of a fast reactor. In Proceedings of ICAPP 2010, San Diego, CA, 2010.
[10]
T. Peterka, D. Goodell, R. Ross, H.-W. Shen, and R. Thakur. A Configurable Algorithm for Parallel Image-Compositing Applications. In Proceedings of SC 09, Portland OR, 2009.
[11]
T. Peterka and R. Ross. Versatile communication algorithms for data analysis. In EuroMPI Special Session on Improving MPI User and Developer Interaction IMUDI'12, Vienna, Austria, 2012.
[12]
T. Peterka, R. Ross, W. Kendall, A. Gyulassy, V. Pascucci, H.-W. Shen, T.-Y. Lee, and A. Chaudhuri. Scalable Parallel Building Blocks for Custom Data Analysis. In Proceedings of the 2011 IEEE Large Data Analysis and Visualization Symposium LDAV'11, Providence, RI, 2011.
[13]
T. Peterka, R. Ross, B. Nouanesengsy, T.-Y. Lee, H.-W. Shen, W. Kendall, and J. Huang. A Study of Parallel Particle Tracing for Steady-State and Time-Varying Flow Fields. In Proceedings of IPDPS 11, Anchorage AK, 2011.
[14]
The MPI Forum. MPI: A message-passing interface standard, 2012.
[15]
M. Wilde, M. Hategan, J. M. Wozniak, B. Clifford, D. S. Katz, and I. Foster. Swift: A language for distributed parallel scripting. Parallel Computing, 37:633--652, 2011.
[16]
J. M. Wozniak, T. G. Armstrong, E. L. Lusk, D. S. Katz, M. Wilde, and I. T. Foster. Turbine: A distributed memory data flow engine for many-task applications. In Int'l Workshop Scalable Workflow Enactment Engines and Technologies (SWEET) 2012, 2012.
[17]
J. M. Wozniak, T. G. Armstrong, M. Wilde, D. S. Katz, E. Lusk, and I. T. Foster. Swift/T: Scalable data flow programming for many-task applications. In Proc. CCGrid, 2013.
[18]
Y. Zhao, M. Hategan, B. Clifford, I. Foster, G. von Laszewski, I. Raicu, T. Stef-Praun, and M. Wilde. Swift: Fast, reliable, loosely coupled parallel computation. In Proc. Workshop on Scientific Workflows, 2007.

Cited By

View all
  • (2021)A population data-driven workflow for COVID-19 modeling and learningThe International Journal of High Performance Computing Applications10.1177/1094342021103516435:5(483-499)Online publication date: 10-Sep-2021
  • (2021)ExaWorks: Workflows for Exascale2021 IEEE Workshop on Workflows in Support of Large-Scale Science (WORKS)10.1109/WORKS54523.2021.00012(50-57)Online publication date: Nov-2021
  • (2020)Improving the Runtime Performance of Non-linear Mixed-Effects Model EstimationEuro-Par 2019: Parallel Processing Workshops10.1007/978-3-030-48340-1_43(560-571)Online publication date: 29-May-2020
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
EuroMPI '13: Proceedings of the 20th European MPI Users' Group Meeting
September 2013
289 pages
ISBN:9781450319034
DOI:10.1145/2488551
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

  • ARCOS: Computer Architecture and Technology Area, Universidad Carlos III de Madrid

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 September 2013

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. ADLB
  2. MPI
  3. dataflow
  4. parallel tasks
  5. swift

Qualifiers

  • Research-article

Funding Sources

Conference

EuroMPI '13
Sponsor:
  • ARCOS
EuroMPI '13: 20th European MPI Users's Group Meeting
September 15 - 18, 2013
Madrid, Spain

Acceptance Rates

EuroMPI '13 Paper Acceptance Rate 22 of 47 submissions, 47%;
Overall Acceptance Rate 66 of 139 submissions, 47%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)4
  • Downloads (Last 6 weeks)0
Reflects downloads up to 17 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2021)A population data-driven workflow for COVID-19 modeling and learningThe International Journal of High Performance Computing Applications10.1177/1094342021103516435:5(483-499)Online publication date: 10-Sep-2021
  • (2021)ExaWorks: Workflows for Exascale2021 IEEE Workshop on Workflows in Support of Large-Scale Science (WORKS)10.1109/WORKS54523.2021.00012(50-57)Online publication date: Nov-2021
  • (2020)Improving the Runtime Performance of Non-linear Mixed-Effects Model EstimationEuro-Par 2019: Parallel Processing Workshops10.1007/978-3-030-48340-1_43(560-571)Online publication date: 29-May-2020
  • (2017)Supporting task-level fault-tolerance in HPC workflows by launching MPI jobs inside MPI jobsProceedings of the 12th Workshop on Workflows in Support of Large-Scale Science10.1145/3150994.3151001(1-11)Online publication date: 12-Nov-2017
  • (2016)From desktop to large-scale model exploration with Swift/TProceedings of the 2016 Winter Simulation Conference10.5555/3042094.3042132(206-220)Online publication date: 11-Dec-2016
  • (2016)From desktop to Large-Scale Model Exploration with Swift/T2016 Winter Simulation Conference (WSC)10.1109/WSC.2016.7822090(206-220)Online publication date: Dec-2016
  • (2015)Lessons Learned from Building In Situ Coupling FrameworksProceedings of the First Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization10.1145/2828612.2828622(19-24)Online publication date: 15-Nov-2015
  • (2015)Interlanguage parallel scripting for distributed-memory scientific computingProceedings of the 10th Workshop on Workflows in Support of Large-Scale Science10.1145/2822332.2822338(1-11)Online publication date: 15-Nov-2015
  • (2015)Toward Interlanguage Parallel Scripting for Distributed-Memory Scientific ComputingProceedings of the 2015 IEEE International Conference on Cluster Computing10.1109/CLUSTER.2015.74(482-485)Online publication date: 8-Sep-2015
  • (2014)Compiler techniques for massively scalable implicit task parallelismProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1109/SC.2014.30(299-310)Online publication date: 16-Nov-2014
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media