SciFlow: A dataflow-driven model architecture for scientific computing using Hadoop | IEEE Conference Publication | IEEE Xplore

SciFlow: A dataflow-driven model architecture for scientific computing using Hadoop


Abstract:

Many computational science applications utilize complex workflow patterns that generate an intricately connected set of output files for subsequent analysis. Some types o...Show More

Abstract:

Many computational science applications utilize complex workflow patterns that generate an intricately connected set of output files for subsequent analysis. Some types of applications, such as rare event sampling, additionally require guaranteed completion of all subtasks for analysis, and place significant demands on the workflow management and execution environment. SciFlow is a user interface built over the Hadoop infrastructure that provides a framework to support the complex process and data interactions and guaranteed completion requirements of scientific workflows. It provides an efficient mechanism for building a parallel scientific application with dataflow patterns, and enables the design, deployment, and execution of data intensive, many-task computing tasks on a Hadoop platform. The design principles of this framework emphasize simplicity, scalability and fault-tolerance. A case study using the forward flux sampling rare event simulation application validates the functionality, reliability and effectiveness of the framework.
Date of Conference: 06-09 October 2013
Date Added to IEEE Xplore: 23 December 2013
Electronic ISBN:978-1-4799-1293-3
Conference Location: Silicon Valley, CA, USA

References

References is not available for this document.