Execution coordination in mobile agent-based distributed job workflow execution

Abstract

Mobile agent-based distributed job workflow execution requires the use of execution coordination techniques to ensure that an agent executing a subjob can locate its predecessors’ execution results. This paper describes the classification, implementation, and evaluation of execution coordination techniques in the mobile agent-based distributed job workflow execution system. First, a classification of the existing execution coordination techniques is developed for mobile agent systems. Second, to put the discussion into perspective, our framework for mobile agent-based distributed job workflow execution over the Grid (that is, MCCF: Mobile Code Collaboration Framework) is described. How the existing coordination techniques can be applied in the MCCF is also discussed. Finally, a performance study has been conducted to evaluate three coordination techniques using real and simulated job workflows. The results are presented and discussed in the paper.

Introduction

Complex scientific applications can generally be viewed as job workflows in which subjobs (i.e., nodes or activities) represent application components and dependencies represent the interactions between the components. In a Service Oriented Architecture (SOA), common application components are usually deployed as services. A workflow engine coordinates the data and control flow amongst multiple services and allows the enactment of workflows to take advantages of distributed services and resources.

Data intensive scientific applications, such as bioinformatics [21] and digital imaging survey (e.g., Sloan Digital Sky Survey (SDSS) [1]), often involve diverse, high volume, and distributed data sets. They can be generally expressed as a workflow of a number of analysis modules, each of which acts on specific sets of data and performs cross multidisciplinary computations.

To reduce the communication overhead caused by data movement and to provide decentralized control of execution during the workflow enactment, the Mobile Code Collaboration Framework (MCCF) is developed to map the execution of subjobs to the distributed resources and to coordinate the subjobs’ execution at runtime according to the abstract workflow provided by users [16]. Light-weight Mobile Agent (LMA) [6] and Code-on-Demand (CoD) [18] techniques are adopted in the development of the MCCF, so that an analysis module in data intensive scientific applications can be executed at a computational resource close to where the required data set is located.

The MCCF, which does not have a centralized engine, is different from the existing scientific workflow engines (e.g., Condor’s DAGMan,¹ SCIRun²). When multiple data independent subjobs can be executed concurrently, replicas of an exiting LMA will be generated so that there is one LMA for each subjob. The LMAs will then be migrated to the different computational resources for the execution of these data independent subjobs in parallel. Because of the data dependencies in a job workflow, before an LMA executes a subjob, it needs to locate the execution results of its predecessors. When multiple concurrently executing subjobs have a common immediate successor, only one of the corresponding LMAs should be selected for the latter’s execution. Others should be discarded. Due to the lack of a centralized engine, execution coordination is therefore required in the MCCF. So, the objective of the paper is to investigate and evaluate execution coordination technique for the MCCF.

Execution coordination of mobile agents is a well researched area. There are many techniques that have been developed. Section 2 gives a classification of these techniques according to how they can be applied in the mobile agent-based distributed job workflow execution. To further analyze their applicability, the MCCF and how these coordination techniques can be implemented in the MCCF are presented in Section 3. A comparative, experimental study of these coordination techniques is presented in Section 4. Section 5 concludes the paper and outlines the future work.