Adaptive data-aware utility-based scheduling in resource-constrained systems

Abstract

This paper addresses the problem of the dynamic scheduling of data-intensive multiprocessor jobs. Each job requires some number of CPUs and some amount of data that needs to be downloaded into a local storage. The completion of each job brings some benefit (utility) to the system, and the goal is to find the optimal scheduling policy that maximizes the average utility per unit of time obtained from all completed jobs. A co-evolutionary solution methodology is proposed, where the utility-based policies for managing local storage and for scheduling jobs onto the available CPUs mutually affect each other’s environments, with both policies being adaptively tuned using the Reinforcement Learning (RL) methodology. The simulation results demonstrate that the performance of the scheduling policies increases significantly as a result of being tuned with RL, to the point that they significantly outperform the best scheduling algorithm suggested in the literature for jobs with soft-deadline utility functions.

Introduction

Scheduling in High Performance Computing (HPC) systems is becoming an increasingly important and difficult task. An HPC system can have as many as 10⁵ multi-threaded processors and can cost many millions of dollars [6]. Correspondingly, it is desirable to operate such systems as efficiently as possible.

Most of the HPC jobs need to access some data stored on local disk or remote storage, and some jobs need to access very large amounts of data, especially in applications such as high-energy physics, natural language processing, astronomy, and bioinformatics. As noted in [4], the amount of data processed by scientific applications has been increasing exponentially since 1990, at an even faster rate than predicted by the Moore’s law. Thus, efficient data-aware scheduling will become a critical issue in scientific computing in the very near future.

If the scheduling system does not explicitly account for the amount of data each job needs to access, then some jobs will occupy the CPU resources for much longer than necessary, since they will spend significant amounts of time idling and waiting for the data to be read from remote storage. Thread-level schedulers (in Unix, Solaris, etc.) solve this problem by placing multiple threads on a single CPU, so that while some threads are waiting for their data to come in, the CPU can work on the other threads. However, job-level schedulers in HPC systems usually have strict security requirements, where jobs usually do not share the CPU resources. Despite the potentially large losses in system’s productivity associated with ignoring the job data requirements, the data-aware scheduling problem has received little attention so far.

This paper addresses this problem in the context of utility-based optimization, where completion of each job is assumed to bring some benefit (utility) to the system (a decreasing function of the job response time) and the goal is to maximize the average benefit obtained per unit of time from all completed jobs. This utility accrual (UA) formulation was initially proposed by Jensen in [3] and is a generalization of the standard deadline-based scheduling, where the benefit received from each job is 1 if the job is completed before its deadline and 0 otherwise.

As suggested in the recent overview of the UA real-time scheduling domain [7], the only algorithm that allows jobs to share (mutually exclusively) a finite amount of a certain resource (e.g., local storage space) is presented in [11]. However, this algorithm is not forward-looking in a sense that it simply orders the existing jobs according to their expected utility upon completion divided by the time remaining to completion (potential utility density) without accounting for what might happen in the near future (ideally, an algorithm should not schedule a large job that has a low utility upon completion if the probability of a new high-utility job arriving is large enough). We use this algorithm as a benchmark in our data scheduling experiments in Section 7, calling it NDP-RUU. We also use a preemption-enabled version of this algorithm, calling it BDP-RUU. The only UA real-time scheduling algorithm we are aware of that explicitly tries to maximize the expected future utility received by the system (by statistically estimating the job arrival probabilities) is presented in [10]. However, this algorithm does not consider any data constraints, and we are not aware of any other utility-maximizing real-time scheduling algorithms that consider multi-unit resource constraints.

As noted in [5], the problem of scheduling jobs in the UA paradigm even on a single CPU with single-unit resource constraints is NP-hard. Thus, a heuristic needs to be developed for solving this problem in the “static” case of focusing only on the currently arrived jobs (as done in [5], [11]). Alternatively, an adaptive policy tuning algorithm needs to be used in the “dynamic” case when jobs arrive stochastically and the scheduler needs to learn their arrival pattern and make appropriate forward-looking scheduling decisions.

The complexity of the policy tuning process increases (usually exponentially) with the number of input variables used by the policy. In the data-aware scheduling problem, the input variables need to describe the state of the local storage, of the CPU module executing jobs as well as of the jobs waiting for their data to be downloaded and those waiting to be executed on the CPU module. Instead of using a centralized learning algorithm (which will be impractically slow given the large number of input variables it needs to consider), we propose a distributed learning approach, where the CPU module and the local storage module are self-managing and self-optimizing. In particular, each module uses the Reinforcement Learning (RL) methodology [8] to adaptively tune the policy for managing its main resource (CPUs or local storage space). The CPU management policy and the corresponding storage management policy mutually affect each other’s environments, making the self-optimization process co-evolutionary.

Co-evolutionary RL has already been used successfully in some domains (e.g., [2], [1]). However, all co-evolutionary RL frameworks that we are aware of have used agents that directly affect the global environment and receive either individual reinforcement signals from it or a single reinforcement signal for all agents. In this paper we are faced with a sequential multi-agent learning task, where the actions of the first agent (the one managing the local storage) affect the state of the second agent (the one managing the CPU resources), and the feedback signal received from the environment (the total utility of jobs executed over some period of time) is affected directly only by the actions of the second agent. In order to resolve the difficulty of the first agent learning without a direct feedback signal, we propose a new idea of letting it use the “value function” $V\left(x\right)$ of the second agent in order to define its feedback signal. This process is explained in a greater detail in Section 6. To the best of our knowledge, this is the first co-evolutionary feedback learning architecture that uses this idea.

The purpose of this paper is to present a feasibility study of applying the co-evolutionary RL solution approach described above to the data-aware scheduling problem. While this approach did outperform noticeably the best non-adaptive policy described in the literature for this domain (NDP-RUU, as was mentioned above), an extensive comparison of the co-evolutionary RL solution with the other possible solutions for the data-aware scheduling problem within the UA paradigm is outside the scope of this paper. Our hope is that the current feasibility study will inspire other researchers to apply this methodology to new domains and perform in-context comparisons with other techniques.

The rest of the paper is organized as follows. Section 2 formulates the scheduling problem to be solved. Section 3 gives an overview of the solution framework used in this paper. Section 6 explains the specific scheduling algorithms that were developed. Section 7 presents numerical simulations that demonstrate the value of the proposed algorithms and gives some intuition about the observed results. Finally, Section 8 concludes the paper.