A novel multi-agent reinforcement learning approach for job scheduling in Grid computing

Abstract

Grid computing utilizes distributed heterogeneous resources to support large-scale or complicated computing tasks, and an appropriate resource scheduling algorithm is fundamentally important for the success of Grid applications. Due to the complex and dynamic properties of Grid environments, traditional model-based methods may result in poor scheduling performance in practice. Scalability and adaptability are among the key objectives of Grid job scheduling. In this paper, a novel multi-agent reinforcement learning method, called ordinal sharing learning (OSL) method, is proposed for job scheduling problems, especially, for realizing load balancing in Grids. The approach circumvents the scalability problem by using an ordinal distributed learning strategy, and realizes multi-agent coordination based on an information-sharing mechanism with limited communication. Simulation results show that the OSL method can achieve the goal of load balancing effectively, and its performance is even comparable to some centralized scheduling algorithm in most cases. The convergence property and adaptability of the proposed method are also illustrated.

Introduction

Multi-agent resource allocation is the process of distributing a number of items amongst a number of agents, and acts as a central matter of concern in both computer science and economics [1]. It is relevant to a wide range of application domains, such as network routing [2], public transportation [3] and Grid computing [4], [5], [6], where Grid computing is one of the most important applications of resource allocation or scheduling [7].

Grid computing enables the sharing, selection, and aggregation of geographically distributed heterogeneous resources and becomes an important solution paradigm for supporting complicated computing problems. However, there are still some technical challenges for Grids [5]. For a majority of Grid systems, the real and specific problem that underlies Grid computing is coordinated resource scheduling and problem solving in dynamic, multi-institutional virtual organizations, where an effective and efficient scheduling algorithm is fundamentally important [8], [9]. Only with the help of a feasible scheduling policy, can the Grids speed up the task process and provide non-trivial services to users [10]. In the following, the job scheduling problem, which is the key issue for balancing the entire system load while completing all the jobs at hand as soon as possible, is studied (see Fig. 1).

In the past decade, there have been many advances in Grid job scheduling techniques. Various scheduling approaches, including model-based or model-free methods, either using centralized or decentralized mechanisms, have been developed for Grids. On the one hand, lots of algorithms have been studied for job scheduling problems in traditional parallel and distributed systems, such as FPLTF (Fastest Processor to Largest Task First), WQR (Work Queue with Replication) and FCFS (First Come First Serve) [11]. On the other hand, extensive research has been done for Grid scheduling problems, too. In traditional resource scheduling systems, such as Condor [12], PBS [13] and SGE [14], centralized schedulers work effectively since accurate and global information can be obtained. However, centralized or hierarchical resource allocation methods may suffer from the lack of scalability and fault-tolerance ability as well as having a single point of failure [15]. To overcome the scalability problem, some decentralized scheduling algorithms have been proposed. However, most existing decentralized schedulers, for example, in Condor-G [16] and AppleS [17], perform individual scheduling policies regardless of the other schedulers’ decisions and may lead to serious synchronization problems in resource management. Finally, a Herd behavior will arise since schedulers run without central oversight and communication [18], [19]. However, if job scheduling is carried out under the assumption of coordination, such as in Legion Federation [20] and Condor Flock P2P [21], the strong dependency on negotiation among schedulers and resources may lead to high communication overhead. Therefore, how to coordinate the scheduling among decentralized schedulers with a moderate communication cost is an important and open problem. A recent work to deal with the above problem has been done in [22], where a collaborative model is proposed based on the Random Early Detection (RED) strategies via gossiping and good scheduling performance is achieved.

Moreover, to meet the need for scheduling adaptation, which comes from the heterogeneity of resources, the variations of resource performance, and the diversity of applications, an adaptive scheduling method is deserved. Recently, a promising approach based on reinforcement learning (RL) has been studied for job scheduling and resource allocation in Grids [23]. As an important class of machine learning methods, RL aims to solve uncertain decision-making problems by interacting with the environment and near-optimal or suboptimal policies can be obtained in a data-driven way [24]. Therefore, RL provides a model-free methodology and is very promising to solve the difficulties of Grid resource scheduling. According to different learning mechanisms, existing RL approaches to resource scheduling can be mainly divided into two types. One is based on policy gradient learning algorithms [6], [25], [26] and the other uses value-function-based learning algorithms [5], [23], [27]. However, the learning efficiency and scalability of existing RL methods in Grid resource allocation still need to be improved for large-scale applications of Grid computing.

In this paper, to realize learning-based coordination and generalization in large-scale Grid environments, a novel multi-agent reinforcement learning method, called the ordinal sharing learning (OSL) method, is proposed to solve the job scheduling problem for Grid computing. In the OSL method, a fast distributed learning algorithm is designed based on an ordinal information-sharing mechanism. Compared with previous multi-agent RL (MARL) methods for job scheduling, the OSL method has two aspects of innovations. One aspect simplifies the modeling of optimal decision-making in job scheduling, where only a utility table is learned online to estimate the resources’ efficiency, instead of building the complex Grid Information System (GIS). The other aspect circumvents the scalability and coordination problem by an efficient information-sharing mechanism with limited communication for multi-agent systems, where an ordinal sharing strategy makes all agents share their utility tables and make decisions in turn. The proposed approach was evaluated in a simulated large-scale Grid computing environment and the results show its validity and feasibility.

The remainder of this paper is organized as follows. Section 2 introduces a general model for job scheduling in Grid computing, and discusses the performance measures. Section 3 discusses the basic idea of multi-agent reinforcement learning and presents the OSL method for Grid job scheduling. Section 4 makes performance evaluation and comparisons of different job scheduling methods in a simulated Grid computing environment and the results illustrate the effectiveness of the proposed method. Section 5 gives a further overview of the related works. Finally, conclusions are made in Section 6.