Scheduling parameter sweep workflow in the Grid based on resource competition

Abstract

Workflow technology has been adopted in scientific domains to orchestrate and automate scientific processes in order to facilitate experimentation. Such scientific workflows often involve large data sets and intensive computation that necessitate the use of the Grid. To execute a scientific workflow in the Grid, tasks within the workflow are assigned to Grid resources. Thus, to ensure efficient execution of the workflow, Grid workflow scheduling is required to manage the allocation of Grid resources.

Although many Grid workflow scheduling techniques exist, they are mainly designed for the execution of a single workflow. This is not the case with parameter sweep workflows, which are used for parametric study and optimisation. A parameter sweep workflow is executed numerous times with different input parameters in order to determine the effect of each parameter combination on the experiment. While executing multiple instances of a parameter sweep workflow in parallel can reduce the time required for the overall execution, this parallel execution introduces new challenges to Grid workflow scheduling. Not only is a scheduling algorithm that is able to manage multiple workflow instances required, but this algorithm also needs the ability to schedule tasks across multiple workflow instances judiciously, as tasks may require the same set of Grid resources. Without appropriate resource allocation, resource competition problem could arise.

We propose a new Grid workflow scheduling technique for parameter sweep workflow called the Besom scheduling algorithm. The scheduling decision of our algorithm is based on the resource dependencies of tasks in the workflow, as well as conventional Grid resource-performance metrics. In addition, the proposed technique is extended to handle loop structures in scientific workflows without using existing loop-unrolling techniques. The Besom algorithm is evaluated using simulations with a variety of scenarios. A comparison between the simulation results of the Besom algorithm and of the three existing Grid workflow scheduling algorithms shows that the Besom algorithm is able to perform better than the existing algorithms for workflows that have complex structures and that involve overlapping resource dependencies of tasks.

Introduction

Computational science that involves large data sets and intensive computation is performed on distributed computing networks. Such scientific applications necessitate the use and management of a supercomputing infrastructure. The Grid [1], which offers supercomputing power through shared distributed resources, can be used to fulfil this necessity. However, several different functions, such as resource discovery and security management [1], [2], are required in order to manage and provide access to distributed Grid resources. To address this need, software tools have been developed for these functionalities and are implemented in multilayered architecture [1], [3]. Software at a lower layer provides services to the upper layer with end-user applications–specifically, e-Science applications–at the top layer.

To facilitate e-Science applications, workflow technology has been introduced into scientific domains and has become a necessary tool to help scientists perform their work in the Grid environment [4], [5], [6], [7]. Traditionally, workflow is a representation of business processes that consists of a set of tasks and the dependencies between them, specified in control structures or patterns [8]. Workflow has the ability to orchestrate services from heterogeneous sources, such as web services and existing software packages, to streamline business processes. In the same manner, scientific workflows can be utilised to orchestrate heterogeneous computation procedures developed by different parties to represent scientific processes in e-Science. Scientific workflow management systems such as Kepler [9] and Pegasus [10] (which usually provide a graphical user interface for composing workflows) can then be used to automatically execute scientific processes in the Grid.

In order to execute a scientific workflow in the Grid, a workflow scheduler is required. The workflow scheduler employs various Grid workflow scheduling algorithms to decide which task in the workflow is to be executed by which Grid resource so that the execution of the scientific workflow can meet constraints or criteria such as performance and cost. However, scheduling workflows in the Grid can be challenging. Unlike traditional scheduling problems in which resources are likely to be static [11], the distributed resources in the Grid are heterogeneous and may change their availability. Because the Grid has less centralised control [1], [12], a resource may join the Grid and make itself available to run applications, but may also become unavailable due to the decision of its owner or hardware failure. Scheduling in the Grid therefore usually needs to be done rapidly in order to make use of the immediate resources.

In some scientific experiments, the same process is repeated several times with varying input parameters to study or to optimise the parameters in the experiments [13]. For example, quantum chemical calculations optimise four parameters to give the best pseudo-atom surface [14]. This process is called parameter sweep and can also be automated using a workflow. Abramson, Enticott, and Altintas [14] proposed a technique to parallelise the execution of a parameter sweep workflow running in the Grid environment by cloning the repeated tasks to improve the overall execution time. This can be viewed as concurrently executing multiple workflow instances of the same workflow specification. As this results in that every instance competes for the same set of resources, it is therefore necessary to make sure that the scheduling of tasks and the allocation of Grid resources allow the execution of these multiple workflow instances to meet execution constraints.

Although many Grid workflow scheduling techniques exist, most of these scheduling algorithms only deal with a single Grid workflow at a time and assume that Grid workflows contain no loop structure [15], [16]. These techniques neglect the issues that may arise when multiple instances of a scientific workflow are executed in parallel and in iteration. Another issue is that existing Grid workflow scheduling algorithms do not take into account the dependency of tasks on specialised resources; nor do they consider the popularity of a resource [16], [17], [18]. For example, assigning a lengthy task to a resource that is shared by many tasks might delay the overall execution. This limitation is amplified with parallel execution of parameter sweep workflows due to the fact that every instance of a parameter sweep workflow, as mentioned earlier, requires the same set of resources. The situation may be exacerbated if there is competition for a scarce special resource.

In this paper, we propose a scheduling technique for multiple scientific workflow instances, based on the resource dependencies of tasks, that is also able to handle loop structure. The proposed algorithm aims to minimise the makespan of the parameter sweep workflow execution. We use parameter sweep workflows as a case study to illustrate the execution of workflow with high resource competition.

The rest of this paper is structured as follows. Section 2 describes the related work in the area of Grid workflow scheduling. The proposed parameter sweep workflow scheduling technique is explained Section 3, and the extension for loop handling is explained in Section 4. The scheduler that implements the proposed technique is briefly described in Section 5 while the simulation results and the evaluation are presented in Section 6. Finally, Section 7 concludes this paper along with potential future research directions.