David Simonetti
ABSTRACT
Modern scientific workflows desire to mix several different computing modalities: self-contained computational tasks, data-intensive transformations, and serverless function calls. To date, these modalities have required distinct system architectures with different scheduling objectives and constraints. In this paper, we describe how TaskVine, a new workflow execution platform, combines these modalities into an execution platform with shared abstractions. We demonstrate results of the system executing a machine learning workflow with combined standalone tasks and serverless functions.
- Y. Babuji. Parsl: Pervasive parallel programming in python. HPDC '19, New York, NY, USA, 2019. Association for Computing Machinery.Google Scholar
Digital Library
- R. Chard. Funcx: A federated function serving fabric for science. HPDC '20, New York, NY, USA, 2020. Association for Computing Machinery.Google ScholarDigital Library
- T. A. S. Foundation. Apache openwhisk.Google Scholar
- S. Hendrickson. Serverless computation with openlambda. In 8th USENIX workshop on hot topics in cloud computing (HotCloud 16), 2016.Google ScholarDigital Library
- P. Moritz. Ray: A distributed framework for emerging AI applications. In OSDI, 2018.Google Scholar
Index Terms
- Mixed Modality Workflows in TaskVine
Recommendations
Challenges of Running Scientific Workflows in Cloud Environments
ScienceCloud '15: Proceedings of the 6th Workshop on Scientific Cloud ComputingThis talk will touch upon the challenges of running scientific workflows in distributed environments such as academic and commercial clouds. It will describe the Pegasus Workflow Management System [1] and how it manages the execution of a variety of ...
Predicting the Costs of Serverless Workflows
ICPE '20: Proceedings of the ACM/SPEC International Conference on Performance EngineeringFunction-as-a-Service (FaaS) platforms enable users to run arbitrary functions without being concerned about operational issues, while only paying for the consumed resources. Individual functions are often composed into workflows for complex tasks. ...
Scalable script-based data analysis workflows on clouds
WORKS '13: Proceedings of the 8th Workshop on Workflows in Support of Large-Scale ScienceData analysis workflows are often composed by many concurrent and compute-intensive tasks that can be efficiently executed only on scalable computing infrastructures, such as HPC systems, Grids and Cloud platforms. The use of Cloud services for the ...
Comments