Modeling and simulation of hierarchical task allocation system for energy-aware HPC clouds

https://doi.org/10.1016/j.simpat.2020.102221Get rights and content

Abstract

The paper presents a hierarchical approach to the problem of energy-aware task allocation in a distributed HPC system consisting of several computing clusters. The solution involves the control of clusters and the use of network traffic engineering methods to minimize overall energy consumption. The advantages of the hierarchical approach are twofold: first entities controlling parts of computing infrastructure may decide how much of their responsibility is delegated to the coordinator, next the computational complexity of related mathematical programming problems may be reduced, and their solution may be done in parallel. The mathematical programming problem of task allocation is defined in the centralized form first. After discussing its computation complexity and related organizational difficulties, it is decomposed to form a hierarchy. Finally, heuristics for allocation are proposed and verified via numerical simulation.

Introduction

High-Performance Computing (HPC) and cloud systems become part of our life, playing a vital role in providing many widely adopted services from social media and entertainment to e-commerce and being one of the main tools for large scale scientific experiments. Apart from apparent benefits, they present several challenges to engineers and society, one of the most significant being excessive energy consumption. Although the energy consumed by HPC systems is rising slower now than ten years ago [1] it still amounts to approximately 1.5% of the total electrical energy usage. Such immense power demand raises operational costs (OPEX) and is accompanied by environmentally endangering CO2 emissions [2], [3]. While the adoption of green power sources may help protect the environment (see e.g., Netflix efforts to eliminate traditional energy sources [4]), an improvement of power efficiency remains the primary means of reducing OPEX. The power efficiency of HPC systems is typically measured in GFlops per watt, the state of art values found in the Green500 list [5] reach as high numbers as 16.9 GFlops/W for the Japan A64FX prototype. However, it must be noted that the most powerful clusters provide hundreds of PFlops consuming lots of energy [6]. Therefore, limiting energy consumption and related thermal emission has become a key part of the European energy policies [7], [8]. The efficiency of service provisioning by distributed computing clouds and data centers is significantly influenced by data processing speed and the bandwidth of a network. As the predicted traffic grows exponentially, the high-level energy-aware optimization of network usage becomes an important tool to exploit network power-saving features fully.

On the other hand, the utilization of computing servers in data centers rarely approaches 100% [9]. Typical usage profiles reach no higher than 10%–50% of server capacity for most of the operation time due to computing resources over-subscription strategy applied to guarantee the expected quality of service and provide adaptation of services to sudden changes in the workload. The resulting increased consumption of electricity requires advanced resource allocation methodologies to reduce the gap between the capacity provided by data centers and the requirements of users, especially during low workload periods. As low-level technologies and mechanisms (e.g., ACPI [10] enabled power saving functions of storage devices and CPUs) are widely adopted, today, the main challenge is to arrange and adapt all of them to develop high-level, energy-efficient, and flexible power control systems encompassing all elements of HPC systems. The rapid growth of energy demand by computing and networking infrastructures must be mitigated on both software and hardware levels.

This paper proposes a coordination framework for power saving task allocation to a distributed cloud system consisting of several computing clusters interconnected with data centers via the backbone network. Various entities may own the analyzed system elements; they all incorporate power-saving mechanisms, possibly controlled locally. The proposed solution allows for energy-efficient management of the available power budget while assuring the required quality of services. In the next section, the motivation for this research is presented shortly, then in Section 3, a short survey of approaches for energy-aware computation is provided and discussed, together with related work in Section 4. The optimization problem of power-saving tasks allocation is presented in Section 5, first in centralized form — Section 6 and then coordination scheme together with its sub-problems definition is presented in Section 7. Section 8 covers the numerical simulation verification and performance evaluation of the proposed approach. Finally, Section 9 concludes and proposes some areas of improving the discussed solution.

Section snippets

Motivation

Based on the previous research related to power saving in backbone network [11], [12], [13] and HPC systems [14], [15], [16], [17] as well as CPU level energy consumption control [18], [19], [20] it may be envisaged that low-level power saving control techniques, although very effective, cannot limit energy consumed by ICT equipment adequately to needs resulting from dynamic usage growth. The way to circumvent these problems may be to incorporate available mechanisms and systems into a

Local control mechanism

Modern devices use two basic mechanisms [22]: (1) smart standby — when a device can sleep for some time, (2) dynamic voltage and frequency scaling (DVFS) — when supply voltage and the clock frequency is adjusted. The former method disables some of the device components for a period when no workload is present, retaining its ability to start processing on workload arrival. The latter allows lowering device service rate (i.e., the processor frequency or network interface bandwidth) when the

Related work on power-saving resource allocation

While power-saving mechanisms described in Section 3 allow to limit the energy consumption of a single server, further reduction may be attained by their integration with HPC system management software. The main advantage of applying power-saving control to a cluster is the possibility to coordinate task execution in a manner maximizing the effectiveness of equipment used by adjusting its energy states and task allocation. The typical result is a consolidation of tasks by moving them to a set

Description of the problem

The subject of the work is energy-saving control of a distributed HPC system consisting of a number of computing clusters connected to some data centers providing the information needed for task processing. Such space decomposition is used when data must be stored in distinct locations due to their excessive size requiring initial preprocessing close to the source or reliability considerations resulting in various replication schemes. Typical examples may be large scale scientific experiments

Centralized cluster workload allocation

Before developing a decomposition scheme, the overall, centralized energy-aware HPC system control problem will be defined to describe all its components and control targets. Various approaches to the definition and solution of HPC systems’ energy-aware control were discussed in Section 4. The one presented here involves control of network equipment and several computing clusters. The resulting control framework provides top-level of coordination and control and cooperates with cluster

Two level cluster workload allocation

The solution proposed in the previous section models all aspects of the controlled HPC system in a single set of formulas (1)–(14). The allocations found this way minimize the power demanded by the whole system; however, their implementation requires deep and direct interaction with management systems of all clusters. When different institutions run the clusters, such a scheme may be difficult to accept. The reasons for this may be the incompatibility of management systems but also the

Simulation experiments

Several groups of simulation experiments were carried out to evaluate the proposed solution. Initially, the basic properties of algorithms were analyzed, and exhaustive tests were performed to assess algorithm sensitivity to parameters values. Then the calculation time and power effectiveness were evaluated. The simulations were run for several scenarios involving varying system resources, workload, and network topologies. Two variants of backbone networks were considered to check the influence

Conclusions and future work

The problem of energy saving allocation of tasks to the distributed computing system is relatively complex. The mathematical programming model of the problem has been developed, and its solution has been discussed in Section 6. The analysis provided in Sections 6 Centralized cluster workload allocation, 7 Two level cluster workload allocation demonstrates that solving this task by the global controller with an appropriate method (e.g., specialized MIP solver) is able to deliver an optimal

References (68)

  • The top500

    (2019)
  • Energy efficiency

    (2016)
  • Code of conduct for energy efficiency in data centres

    (2016)
  • BarrosoL.A. et al.

    The case for energy-proportional computing

    Computer

    (2007)
  • Hewlett-PackardL.A. et al.

    Advanced Configuration and Power Interface Specification, Revision 5.0

    (2011)
  • KarpowiczM.P. et al.

    Energy-aware multilevel control system for a network of linux software routers: Design and implementation

    IEEE Syst. J.

    (2018)
  • KamolaM. et al.

    Shortest path green routing and the importance of traffic matrix knowledge

  • KarpowiczM. et al.

    Energy and power efficiency in cloud

  • Niewiadomska-SzynkiewiczE. et al.

    Resource management system for HPC computing

  • ArabasP.

    Energy aware data centers and networks: a survey

    J. Telecommun. Inf. Technol.

    (2019)
  • P. Arabas, E. Niewiadomska-Szynkiewicz, Energy-efficient workload allocation in distributed HPC system, in: The 2019...
  • ArabasP. et al.

    Server power consumption: Measurements and modeling with MSRs

  • KarpowiczM.

    Energy-efficient CPU frequency control for the Linux system

    Concurr. Comput.: Pract. Exper.

    (2015)
  • IBMM.

    User’s manual for CPLEX

    (2017)
  • PiersonJ.M.

    Large-scale Distributed Systems and Energy Efficiency: A Holistic View

    (2015)
  • BollaR. et al.

    Theoretical and technological limitations of power scaling in network devices

  • V. Pallipadi, A. Starikovskiy, The ondemand governor, in: Proc. Linux Symposium, Vol. 2, 2006, pp....
  • V. Pallipadi, S. Li, A. Belay, cpuidle: Do nothing, efficiently, in: Proc. Linux Symposium, Vol. 2, 2007, pp....
  • IntelR.

    64 and IA-32 architectures software developer’s manual

    (2015)
  • T. D., H. Jagode, H. You, J. Dongarra, Collecting performance data with PAPI-C, in: Proc. 3rd Parallel Tools Workshop,...
  • IntelR. et al.

    Intelligent platform management interface specification, second generation

    (2015)
  • BollaR. et al.

    The green abstraction layer: A standard power-management interface for next-generation network devices

    IEEE Internet Comput.

    (2013)
  • BollaR. et al.

    Large-scale validation and benchmarking of a network of power-conservative systems using ETSI’s Green Abstraction Layer

    Trans. Emerg. Telecommun. Technol.

    (2016)
  • ETSI ES 203 237 V1.1.1 (2014–03) standard

    (2014)
  • View full text