Article

Integrating existing scientific workflow systems: the Kepler/Pegasus example

Authors:

Nandita Mandal,

Karan VahiAuthors Info & Claims

WORKS '07: Proceedings of the 2nd workshop on Workflows in support of large-scale science

Pages 21 - 28

https://doi.org/10.1145/1273360.1273365

Published: 25 June 2007 Publication History

Abstract

Scientific workflows have become an important tool used by scientists to conduct large-scale analysis in distributed environments. Today thereare a variety of workflow systems that provide an often disjoint set of capabilities and expose different workflow modeling semantics to the users. In this paper we examine the possibility of integrating two well-known workflow systems Kepler and Pegasus and examine the opportunities and challenges presented by such an integration. We illustrate the combined system on a workflow used as a basis of a provenance challenge.

References

[1]

Shawn Bowers et al., Actor Oriented Design for Scientific Workflows, Lecture Notes in Computer Science, Vol. 3716 (November 2005), pp. 369--384.

Digital Library

[2]

Betram Ludascher et al., Scientific Workflow Management and the Kepler System. Concurrency and Computation: Practice and Experience, Special Issue on Scientific Workflows, 2005.

Digital Library

[3]

Ewa Deelman et al., "Pegasus: a Framework for Mapping Complex Scientific Workflows onto Distributed Systems". Scientific Programming Journal, Vol 13(3), 2005, Pages 219--237

Digital Library

[4]

S. Miles, First Provenance Challenge, http://twiki.ipaw.info/bin/view/Challenge/FirstProvenanceChallenge, August 2006.

[5]

Ilkay Altintas et al., Kepler: An Extensible System for Design and Execution of Scientific Workflows, 2004.

[6]

Ilkay Altintas et al., A Framework for the Design and Reuse of Grid Workflows, Spring 2005.

[7]

PTOLEMY II project and system. Department of EECS, UC Berkeley, 2004. http://ptolemy.eecs.berkeley.edu/ptolemyII/.

[8]

C. Brooks et al., Heterogeneous Concurrent Modeling and Design in Java (Volume 3: Ptolemy II Domains): SDF Domain, Technical Memorandum UCB/ERL M04/17, University of California, Berkeley, CA USA 94720, June 24, 2004.

[9]

Edward A. Lee and Steve Neuendorffer. MoML . A Modeling Markup Language in XML . Version 0.4. Technical report, University of California at Berkeley, March, 2000.

[10]

Yolanda Gil et al., Wings for Pegasus: A Semantic Approach to Creating Very Large Scientific Workflows, OWL: Experiences and Directions 2006

[11]

I. Foster et al., Chimera: A Virtual Data System for Representing, Querying, and Automating Data Derivation, Proceedings of Scientific and Statistical Database Management, 2002.

Digital Library

[12]

Ewa Deelman et al., Pegasus: Mapping Scientific Workflows onto the Grid, Across Grids Conference 2004, Nicosia, Cyprus

[13]

E. Deelman, C. Kesselman, et al., "Transformation Catalog Design for GriPhyN," Technical Report GriPhyN-2001-17, 2001.

[14]

Condor Team, The directed acyclic graph manager, www.cs.wisc.edu/condor/dagman, 2002

[15]

Jia Yu and Rajkumar Buyya, A Taxonomy of Workflow anagement Systems for Grid Computing, Technical Report, GRIDS-TR-2005-1, Grid Computing and Distributed Systems Laboratory, University of Melbourne, Australia, March 10, 2005.

[16]

Gurmeet Singh et al., The Pegasus Portal: Web Based Grid Computing, Symposium on Applied Computing archive Proceedings of the 2005.

Digital Library

[17]

Gurmeet Singh et al., A Metadata Catalog Service for Data Intensive Applications, ACM/IEEE conference on Supercomputing 2003.

Digital Library

[18]

A. Chervenak, E. Deelman, et al., Giggle: A Framework for Constructing Scalable Replica Location Services, Proceedings of Supercomputing 2002 (SC2002), Baltimore, MD. 2002.

Digital Library

[19]

Carole Goble, Using the Semantic Web for e-Science: Inspiration, Incubation, Irritation Lecture Notes in Computer Science 3729:1--3

Digital Library

[20]

J. Mocenigo, Grappa: A Java Graph Package, October 2006.

[21]

http://www.mygrid.org.uk/linkup

[22]

Berriman, G. et al., Montage: The Architecture and Scientific Applications of a National Virtual Observatory Service for Computing Astronomical Image Mosaics, Proceedings of Earth Sciences Technology Conference, 2006

[23]

Ewa Deelman et al., Managing Large-Scale Workflow Execution from Resource Provisioning to Provenance tracking: The CyberShake Example, e-Science 2006, Amsterdam, December 4--6, 2006

[24]

S. McGough et al., Workflow Enactment in ICENI. In UK e-Science All Hands Meeting, Nottingham, UK, IOP Publishing Ltd, Bristol, UK, Sep. 2004; 894--900.

[25]

F. Berman et al., The GrADS Project: Software Support for High-Level Grid Application Development. International Journal of High Performance Computing Applications(JHPCA), 15(4):327--344, SAGE Publications Inc., London, UK, Winter 2001.

Digital Library

[26]

G. von Laszewski, M. Hategan. Java CoG Kit Karajan/GridAnt Workflow Guide. Technical Report, Argonne National Laboratory, Argonne, IL, USA, 2005.

[27]

G. von Laszewski, K. Amin, M. Hategan, N. J. Zaluzec, S. Hampton, and A. Rossi. GridAnt: A Client-Controllable Grid Workflow System. In 37th Annual Hawaii International Conference on System Sciences (HICSS'04), Big Island, Hawaii: IEEE CS Press, Los Alamitos, CA, USA, January 5--8, 2004.

Digital Library

[28]

I. Taylor, M. Shields, and I. Wang. Resource Management of Triana P2P Services. Grid Resource Management, Kluwer, Netherlands, June 2003.

[29]

T. Oinn et al., Taverna: a tool for the composition and enactment of bioinformatics workflows. Bioinformatics, 20(17): 3045--3054, Oxford University Press, London, UK, 2004.

Digital Library

[30]

Y. Gil, et al., "Wings for Pegasus: Creating Large-Scale Scientific Applications Using Semantic Representations of Computational Workflows," in Proceedings of the 19th Annual Conference on Innovative Applications of Artificial Intelligence (IAAI) Vancouver, British Columbia, Canada, 2007 (to appear)

Digital Library

Cited By

Guo JXu YFu HXue WGan LTan MWu TShen YWu XHu LChe X(2022)GEO-WMS: an improved approach to geoscientific workflow management system on HPCCCF Transactions on High Performance Computing10.1007/s42514-022-00131-x5:4(360-373)Online publication date: 14-Nov-2022
https://doi.org/10.1007/s42514-022-00131-x
Alqaoud A(2013)Performance evaluation for scientific workflow interoperability38th Annual IEEE Conference on Local Computer Networks - Workshops10.1109/LCNW.2013.6758536(941-947)Online publication date: Oct-2013
https://doi.org/10.1109/LCNW.2013.6758536
Tschager TSchmidt HHidders JMissier PSroka J(2012)DAGwomanProceedings of the 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies10.1145/2443416.2443419(1-6)Online publication date: 20-May-2012
https://dl.acm.org/doi/10.1145/2443416.2443419
Show More Cited By

Index Terms

Integrating existing scientific workflow systems: the Kepler/Pegasus example
1. Software and its engineering
  1. Software creation and management
    1. Software development techniques

Recommendations

A taxonomy of scientific workflow systems for grid computing

With the advent of Grid and application technologies, scientists and engineers are building more and more complex applications to manage and process large data sets, and execute scientific experiments on distributed resources. Such application scenarios ...
A Reference Architecture for Scientific Workflow Management Systems and the VIEW SOA Solution

Scientific workflows have recently emerged as a new paradigm for scientists to formalize and structure complex and distributed scientific processes to enable and accelerate many scientific discoveries. In contrast to business workflows, which are ...
A Performance Evaluation of X-Ray Crystallography Scientific Workflow Using SciCumulus
CLOUD '11: Proceedings of the 2011 IEEE 4th International Conference on Cloud Computing

X-ray crystallography is an important field due to its role in drug discovery and its relevance in bioinformatics experiments of comparative genomics, phylogenomics, evolutionary analysis, ortholog detection, and three-dimensional structure ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WORKS '07: Proceedings of the 2nd workshop on Workflows in support of large-scale science

June 2007

86 pages

ISBN:9781595937155

DOI:10.1145/1273360

General Chairs:
Ewa Deelman
USC Information Sciences Institute
,
Ian Taylor
Cardiff University and the Center for Computation and Technology, Louisiana State University

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 June 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

HPDC07

Sponsor:

HPDC07: International Symposium on High Performance Distributed Computing

June 25, 2007

California, Monterey, USA

Acceptance Rates

Overall Acceptance Rate 30 of 54 submissions, 56%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
603
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Guo JXu YFu HXue WGan LTan MWu TShen YWu XHu LChe X(2022)GEO-WMS: an improved approach to geoscientific workflow management system on HPCCCF Transactions on High Performance Computing10.1007/s42514-022-00131-x5:4(360-373)Online publication date: 14-Nov-2022
https://doi.org/10.1007/s42514-022-00131-x
Alqaoud A(2013)Performance evaluation for scientific workflow interoperability38th Annual IEEE Conference on Local Computer Networks - Workshops10.1109/LCNW.2013.6758536(941-947)Online publication date: Oct-2013
https://doi.org/10.1109/LCNW.2013.6758536
Tschager TSchmidt HHidders JMissier PSroka J(2012)DAGwomanProceedings of the 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies10.1145/2443416.2443419(1-6)Online publication date: 20-May-2012
https://dl.acm.org/doi/10.1145/2443416.2443419
Chen JYang Y(2011)Temporal dependency-based checkpoint selection for dynamic verification of temporal constraints in scientific workflow systemsACM Transactions on Software Engineering and Methodology10.1145/2000791.200079320:3(1-23)Online publication date: 26-Aug-2011
https://dl.acm.org/doi/10.1145/2000791.2000793
Cruz SBatista VSilva ETosta FVilela CCuadrat RTschoeke DDavila ACampos MMattoso M(2010)Detecting distant homologies on protozoans metabolic pathways using scientific workflowsInternational Journal of Data Mining and Bioinformatics10.1504/IJDMB.2010.0335204:3(256-280)Online publication date: 1-Jun-2010
https://dl.acm.org/doi/10.1504/IJDMB.2010.033520
Ludascher BAltintas IBowers SCummings JCritchlow TDeelman ERoure DFreire JGoble CJones MKlasky SMcPhillips TPodhorszki NSilva CTaylor IVouk M(2010)Scientific Process Automation and Workflow ManagementScientific Data Management10.1201/9781420069815-c13Online publication date: 6-May-2010
https://doi.org/10.1201/9781420069815-c13
Wu QMengxia Zhu Xukang Lu Brown PYunyue Lin Gu YFei Cao Reuter M(2010)Automation and management of scientific workflows in distributed network environments2010 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum (IPDPSW)10.1109/IPDPSW.2010.5470720(1-8)Online publication date: Apr-2010
https://doi.org/10.1109/IPDPSW.2010.5470720
Wang FDeng HGuo LJi K(2010)A Batch Astronomical Data Sharing SystemProceedings of the 2010 International Forum on Information Technology and Applications - Volume 0110.1109/IFITA.2010.211(413-416)Online publication date: 16-Jul-2010
https://dl.acm.org/doi/10.1109/IFITA.2010.211
Azimi AParsa S(2010)Architectural plan for constructing fault tolerable workflow engines based on grid service2010 2nd International Conference on Education Technology and Computer10.1109/ICETC.2010.5529657(V4-384-V4-389)Online publication date: Jun-2010
https://doi.org/10.1109/ICETC.2010.5529657
Chen JYang Y(2010)Localising temporal constraints in scientific workflowsJournal of Computer and System Sciences10.1016/j.jcss.2009.11.00776:6(464-474)Online publication date: 1-Sep-2010
https://dl.acm.org/doi/10.1016/j.jcss.2009.11.007
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten