skip to main content
10.1145/3624062.3624279acmotherconferencesArticle/Chapter ViewAbstractPublication PagesscConference Proceedingsconference-collections
research-article

Delivering Rules-Based Workflows for Science

Published: 12 November 2023 Publication History

Abstract

Rules-based workflow scheduling is a recently developed method for constructing an analysis structure, in a far more dynamic manner than traditional graph based systems. However, rules-based workflows are still in their relative infancy and lack the breadth of features available in traditional scientific workflow systems. This paper will address some of these missing features by introducing the new meow_base library for generic construction of rules-based systems while meeting the requirements of a scientific workflow management system. It will also present two example workflows, showing how rules-based systems can better enable analysis loops or human-in-the-loop interactions than more traditional workflow systems.

Supplemental Material

MP4 File - Conference presentation recording
Recording of "Delivering Rules-Based Workflows for Science" presentation at the The 18th Workshop on Workflows in Support of Large-Scale Science (WORKS23)

References

[1]
I. Altintas, C. Berkley, E. Jaeger, M. Jones, B. Ludascher, and S. Mock. 2004. Kepler: an extensible system for design and execution of scientific workflows. In Proceedings. 16th International Conference on Scientific and Statistical Database Management, 2004.423–424. https://doi.org/10.1109/SSDM.2004.1311241
[2]
Peter Amstutz, Michael R. Crusoe, Nebojša Tijanić, Brad Chapman, John Chilton, Michael Heuer, Andrey Kartashov, Dan Leehr, Hervé Ménager, Maya Nedeljkovich, and et al.2016. Common Workflow Language, v1.0. https://doi.org/10.6084/m9.figshare.3115156.v2
[3]
bids 2023. Brain Imaging Data Structure. https://bids.neuroimaging.io/.
[4]
Shawn Bowers and Bertram Ludäscher. 2005. Actor-Oriented Design of Scientific Workflows. In Conceptual Modeling – ER 2005, Lois Delcambre, Christian Kop, Heinrich C. Mayr, John Mylopoulos, and Oscar Pastor (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 369–384.
[5]
Ryan Chard, Kyle Chard, Jason Alt, Dilworth Y. Parkinson, Steve Tuecke, and Ian Foster. 2017. Ripple: Home Automation for Research Data Management. In 2017 IEEE 37th International Conference on Distributed Computing Systems Workshops (ICDCSW). 389–394. https://doi.org/10.1109/ICDCSW.2017.30
[6]
Ewa Deelman, James Blythe, Yolanda Gil, and Carl Kesselman. 2002. Pegasus: Planning for Execution in Grids. Technical Report Technical Report 2002-20. GriPhyN. http://pegasus.isi.edu/publications/ewa/pegasus_overview.pdf
[7]
Jack Deslippe, Abdelilah Essiari, Simon J. Patton, Taghrid Samak, Craig E. Tull, Alexander Hexemer, Dinesh Kumar, Dilworth Parkinson, and Polite Stewart. 2014. Workflow Management for Real-Time Analysis of Lightsource Experiments. In 2014 9th Workshop on Workflows in Support of Large-Scale Science. 31–40. https://doi.org/10.1109/WORKS.2014.9
[8]
fanotify 2021. fanotify(7) - Linux manual page. https://man7.org/linux/man-pages/man7/fanotify.7.htmlpublisher: man7.org.
[9]
J. E. Ferreira, Q. Wu, S. Malkowski, and C. Pu. 2010. Towards Flexible Event-Handling in Workflows through Data States. In 2010 6th World Congress on Services. 344–351. https://doi.org/10.1109/SERVICES.2010.60
[10]
Bas P. Harenslak and Julian Rutger de Ruiter. 2021. Data Pipelines with Apache Airflow (1 ed.). Manning.
[11]
Charles Antony Richard Hoare. 1978. Communicating sequential processes. Commun. ACM 21, 8 (1978), 666–677.
[12]
inotify 2021. inotify(7) - Linux manual page. https://man7.org/linux/man-pages/man7/inotify.7.htmlpublisher: man7.org.
[13]
Thomas Leibovici. 2015. Taking back control of HPC file systems with Robinhood Policy Engine. https://doi.org/10.48550/ARXIV.1505.01448
[14]
Pedro García López, Aitor Arjona, Josep Sampé, Aleksander Slominski, and Lionel Villard. 2020. Triggerflow. In Proceedings of the 14th ACM International Conference on Distributed and Event-based Systems. ACM. https://doi.org/10.1145/3401025.3401731
[15]
David Marchant. 2022. Events as a Basis for Workflow Scheduling. In 2022 IEEE/ACM Workshop on Workflows in Support of Large-Scale Science (WORKS). 52–59. https://doi.org/10.1109/WORKS56498.2022.00012
[16]
David Marchant. 2022. mig_meow on GitHub. https://github.com/PatchOfScotland/mig_meow.
[17]
David Marchant. 2023. meow_base. https://github.com/PatchOfScotland/meow_base.
[18]
David Marchant. 2023. meow_benchmarks on GitHub. https://github.com/PatchOfScotland/meow_benchmarks.
[19]
David Marchant. 2023. meow_bids. https://github.com/PatchOfScotland/meow_bids.
[20]
David Marchant, Rasmus Munk, Elise O. Brenne, and Brian Vinter. 2020. Managing Event Oriented Workflows. In 2020 IEEE/ACM 2nd Annual Workshop on Extreme-scale Experiment-in-the-Loop Computing (XLOOP). 23–28. https://doi.org/10.1109/XLOOP51963.2020.00009
[21]
Christopher J Markiewicz, Krzysztof J Gorgolewski, Franklin Feingold, Ross Blair, Yaroslav O Halchenko, Eric Miller, Nell Hardcastle, Joe Wexler, Oscar Esteban, Mathias Goncavles, Anita Jwa, and Russell Poldrack. 2021. The OpenNeuro resource for sharing of neuroscience data. eLife 10 (oct 2021), e71774. https://doi.org/10.7554/eLife.71774
[22]
Marta Mattoso, Jonas Dias, Kary A.C.S.Ocaña, Eduardo Ogasawara, Flavio Costa, Felipe Horta, Vitor Silva, and Daniel de Oliviera. 2015. Dynamic Steering of HPC scientific workflows: A survey. Future Generation Computer Systems 46 (2015), 100–113.
[23]
Timothy McPhillips, Shawn Bowers, Daniel Zinn, and Bertram Ludäscher. 2008. Scientific workflow design for mere mortals. Future Generation Computer Systems 25 (2008), 541–551.
[24]
mriqc 2023. MRIQC 23.1.0 documentation. https://mriqc.readthedocs.io/en/latest/.
[25]
Hoang Anh Nguyen, David Abramson, Timoleon Kiporous, Andrew Janke, and Graham Galloway. 2014. WorkWays: Interacting with Scientific Workflows. In 2014 9th Gateway Computing Environments Workshop. 21–24. https://doi.org/10.1109/GCE.2014.6
[26]
Martin Norgaard. 2023. "The MN dataset". https://doi.org/
[27]
D.M. Pelt, A.A Hendriksen, and K.J. Batenburg. 2022. Foam-like phantoms for comparing tomography algorithms.29(1) (2022), 254–265.
[28]
perf 2021. perf-trace(1) - Linux manual page. https://man7.org/linux/man-pages/man1/perf-trace.1.htmlpublisher: man7.org.
[29]
Stefanie Rinderle, Manfred Reichert, and Peter Dadam. 2004. Correctness criteria for dynamic changes in workflow systems––a survey. Data & Knowledge Engineering 50, 1 (2004), 9–34. https://doi.org/10.1016/j.datak.2004.01.002 Advances in business process management.
[30]
strace 2022. strace(1) - Linux manual page. https://man7.org/linux/man-pages/man1/strace.1.htmlpublisher: man7.org.
[31]
The SPADE project 2016. The SPADE project. http://nest.lbl.gov/projects/spade/html/
[32]
wdl 2023. wdl: Workflow Description Language - Specification and Implementations. https://github.com/openwdl/wdl.
[33]
yawl 2023. YAWL BPM. https://yawlfoundation.github.io/.
[34]
A. Yoo, M. Jetter, and M. Grondona. 2015. Slurm: Simple Linux Utility for Resource Management. Lecture Notes in Computer Science 2862 (2015), 44–60.
[35]
Yong Zhao, Ioan Raicu, and Ian Foster. 2008. Scientific Workflow Systems for the 21st Century, New Bottle or New Wine?. In Proceedings of the 2008 IEEE Congress on Services - Part I, SERVICES ’08. Washington, DC, USA, 467–471.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
SC-W '23: Proceedings of the SC '23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis
November 2023
2180 pages
ISBN:9798400707858
DOI:10.1145/3624062
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 November 2023

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. BIDS
  2. Heterogeneous
  3. MEOW
  4. Rules-based
  5. Workflows

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

SC-W 2023

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 54
    Total Downloads
  • Downloads (Last 12 months)36
  • Downloads (Last 6 weeks)7
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media