skip to main content
10.1145/2616498.2616565acmotherconferencesArticle/Chapter ViewAbstractPublication PagesxsedeConference Proceedingsconference-collections
research-article

Once you SCOOP, no need to fork

Published: 13 July 2014 Publication History

Abstract

This paper presents SCOOP, a new Python framework for automatically distributing dynamic task hierarchies. A task hierarchy refers to tasks that can recursively spawn an arbitrary number of subtasks. The underlying computing infrastructure consists of a simple list of resources. The typical use case is to run the user's main program under the umbrella of the SCOOP module, where it becomes a root task that can spawn any number of subtasks through the standard "futures" API of Python, and where these subtasks may themselves spawn other subsubtasks, etc. The full task hierarchy is dynamic in the sense that it is unknown until the end of the last running task. SCOOP automatically distributes tasks amongst available resources using dynamic load balancing. A task is nothing more than a Python callable object in conjunction with its arguments. The user need not worry about message passing implementation details; all communications are implicit.

References

[1]
K. Asanovic, R. Bodik, and B. Catanzaro. The landscape of parallel computing research: A view from Berkeley. Vol. 2. Technical Report UCB/EECS-2006-183, 2006.
[2]
L. Dagum and R. Menon. OpenMP: an industry standard API for shared-memory programming. Computational Science & Engineering, pages 46--55, 1998.
[3]
J. Dean and S. Ghemawat. MapReduce: simplified data processing on large clusters. Communications of the ACM, pages 1--13, 2008.
[4]
F.-A. Fortin, F.-M. De Rainville, M.-A. Gardner, M. Parizeau, and C. Gagné. DEAP: Evolutionary algorithms made easy. Journal of Machine Learning Research, 13:2171--2175, 2012.
[5]
D. Friedman and D. Wise. The impact of applicative programming on multiprocessing. Indiana University, 1976.
[6]
H. González-Vélez and M. Leyton. A survey of algorithmic skeleton frameworks: high-level structured parallel programming enablers. Software: Practice and Experience, pages 1--26, 2010.
[7]
P. Hibbard. Parallel processing facilities. New Directions in Algorithmic Languages, 1976.
[8]
H. B. Jr and C. Hewitt. The incremental garbage collection of processes. ACM SIGART Bulletin, 1977.
[9]
L. Kale and S. Krishnan. CHARM++: a portable concurrent object oriented system based on C++. ACM, 1993.
[10]
T. E. Oliphant. Python for Scientific Computing. Computing in Science & Engineering, 9(3):10--20, 2007.
[11]
M. Snir, S. Otto, and D. Walker. MPI: the complete reference. MIT press, 1995.

Cited By

View all
  • (2023)A robust simulator of pressure-dependent consumption in PythonJournal of Hydroinformatics10.2166/hydro.2023.218Online publication date: 9-Dec-2023
  • (2023)Landscape of High-Performance Python to Develop Data Science and Machine Learning ApplicationsACM Computing Surveys10.1145/361758856:3(1-30)Online publication date: 5-Oct-2023
  • (2023)Multi-Objective Policy Evolution for a Same-Day Delivery Problem with Soft DeadlinesProceedings of the Companion Conference on Genetic and Evolutionary Computation10.1145/3583133.3596381(1941-1949)Online publication date: 15-Jul-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
XSEDE '14: Proceedings of the 2014 Annual Conference on Extreme Science and Engineering Discovery Environment
July 2014
445 pages
ISBN:9781450328937
DOI:10.1145/2616498
  • General Chair:
  • Scott Lathrop,
  • Program Chair:
  • Jay Alameda
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

  • NSF: National Science Foundation
  • Drexel University
  • Indiana University: Indiana University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2014

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Distributed computing
  2. High performance computing
  3. Parallel programming
  4. Software libraries

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

XSEDE '14

Acceptance Rates

XSEDE '14 Paper Acceptance Rate 80 of 120 submissions, 67%;
Overall Acceptance Rate 129 of 190 submissions, 68%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)16
  • Downloads (Last 6 weeks)4
Reflects downloads up to 01 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2023)A robust simulator of pressure-dependent consumption in PythonJournal of Hydroinformatics10.2166/hydro.2023.218Online publication date: 9-Dec-2023
  • (2023)Landscape of High-Performance Python to Develop Data Science and Machine Learning ApplicationsACM Computing Surveys10.1145/361758856:3(1-30)Online publication date: 5-Oct-2023
  • (2023)Multi-Objective Policy Evolution for a Same-Day Delivery Problem with Soft DeadlinesProceedings of the Companion Conference on Genetic and Evolutionary Computation10.1145/3583133.3596381(1941-1949)Online publication date: 15-Jul-2023
  • (2023)mpi4py.futures: MPI-Based Asynchronous Task Execution for PythonIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2022.322548134:2(611-622)Online publication date: 1-Feb-2023
  • (2023)Instance Selection-Based Surrogate-Assisted Genetic Programming for Feature Learning in Image ClassificationIEEE Transactions on Cybernetics10.1109/TCYB.2021.310569653:2(1118-1132)Online publication date: Feb-2023
  • (2023)Engineering the fracture resistance of 2H-transition metal dichalcogenides using vacancies: An in-silico investigation based on HRTEM imagesMaterials Today10.1016/j.mattod.2023.10.00270(17-32)Online publication date: Nov-2023
  • (2022)Parallel Genetic Algorithms’ Implementation Using a Scalable Concurrent Operation in PythonSensors10.3390/s2206238922:6(2389)Online publication date: 20-Mar-2022
  • (2022)Integrated Design and Optimization of Water-Energy Nexus: Combining Wastewater Treatment and Energy SystemFrontiers in Sustainable Cities10.3389/frsc.2022.8569964Online publication date: 31-Mar-2022
  • (2022)Exploring Parameter and Hyper-Parameter Spaces of Neuroscience Models on High Performance Computers With Learning to LearnFrontiers in Computational Neuroscience10.3389/fncom.2022.88520716Online publication date: 27-May-2022
  • (2022)Evolving programmable computational metamaterialsProceedings of the Genetic and Evolutionary Computation Conference10.1145/3512290.3528861(122-129)Online publication date: 8-Jul-2022
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media