Energy saving strategies for parallel applications with point-to-point communication phases

Highlights

• We present a runtime procedure to detect communication phases.

• We propose novel frequency scaling strategies to save energy.

• We explore the opportunity of energy saving in quantum chemistry software GAMESS.

• Different process binding strategies are proposed for GAMESS.

Abstract

Although high-performance computing traditionally focuses on the efficient execution of large-scale applications, both energy and power have become critical concerns when approaching exascale. Drastic increases in the power consumption of supercomputers affect significantly their operating costs and failure rates. In modern microprocessor architectures, equipped with dynamic voltage and frequency scaling (DVFS) and CPU clock modulation (throttling), the power consumption may be controlled in software. Additionally, network interconnect, such as Infiniband, may be exploited to maximize energy savings while the application performance loss and frequency switching overheads must be carefully balanced. This paper advocates for a runtime assessment of such overheads by means of characterizing point-to-point communications into phases followed by analyzing the time gaps between the communication calls. Certain communication and architectural parameters are taken into consideration in the three proposed frequency scaling strategies, which differ with respect to their treatment of the time gaps. The experimental results are presented for NAS parallel benchmark problems as well as for the realistic parallel electronic structure calculations performed by the widely used quantum chemistry package GAMESS. For the latter, three different process-to-core mappings were studied as to their energy savings under the proposed frequency scaling strategies and under the existing state-of-the-art techniques. Close to the maximum energy savings were obtained with a low performance loss of 2% on the given platform.

Introduction

Power consumption has become a major concern for modern and future-generation supercomputers consuming tens of megawatts of power. For example, the petascale K supercomputer, which was at the top of the TOP500 list in 2011, consumes around 12 MW of power for its 705,024 cores. At the exascale, supercomputers having 100 million cores are supposed to operate on “only” 100 MW of power  [3]. Their tremendous power demands drastically increase the operating costs as well as limit scalability and sustainability. Therefore, energy conservation must be employed at all levels of high-performance computing (HPC): applications, system software, and hardware.

The current generation of Intel processors provides various P-states for dynamic voltage and frequency scaling (DVFS) and T-states for introducing processor idle cycles (throttling). For example, the Intel “Penryn” microarchitecture provides four P-states ($f_1,\dots ,f_4$, where $f_i>f_j$ for $i<j$) and eight T-states $T_j$ with $j(j=0,\dots ,7)$ idle cycles per eight cycles in the CPU execution.Table 1 depicts the various P-states and the associated core voltages offered by the Intel Xeon E5450 processor, which is used in this work. The delay of switching from one state to another depends on the relative ordering of the current and desired states, as discussed, e.g., in  [24]. The user may write a specific value to model-specific registers (MSRs) to change the P- or T-state of the system. Note that throttling can be viewed as equivalent to dynamic frequency scaling (DFS)  [1] because, by inserting idle cycles in the CPU execution, a particular frequency is obtained without changing the operating voltage of the cores. Therefore, it is often less effective than DVFS.

Various methods to employ the DVFS exist. The more sophisticated techniques scale processor frequency on different intervals of application runtime while attempting to predict accurately the performance effects from the DVFS. Such techniques may be broadly classified into two types: one that first divides the application into execution intervals of predefined duration and then uses the performance counters to determine a suitable frequency for them  [10], [14], [15], and the other that first determines communication intervals in parallel applications that use either explicit message passing  [9], [20] or global address-space primitives  [36] and then scales the frequency for those intervals, usually based on the MIPS metric.

Modern interconnects, such as Infiniband,¹ provide operating-system kernel bypass mechanisms for a full CPU offload during communications. Note that the term “CPU offload” means “offloading from the CPU” and is used here consistently with Infiniband vendor presentations (see footnote 1). With the CPU offload, more cycles are provided to the CPU for computational tasks or CPU is put in a low-power mode when waiting for data, as in blocking communications, for example. In this work, the CPU offloading during the communications and the performance-counter information on the computations are unified to extract the maximum energy savings without the need for any changes to the user application or to the communication library.

A wide range of HPC applications are developed using communication libraries implementing the Message Passing Interface² (MPI), which has become a de facto standard for the design of parallel applications. It defines point-to-point and collective communication primitives for sending and receiving data explicitly within a parallel application. In the authors’ previous work  [32], energy efficiency in the MPI collective operations, such as all-to-all exchange, was addressed. Specifically, a collective operation was considered as a multitude of point-to-point communications grouped together, essentially presenting a single communication stream. Since the rank³ sequences of the point-to-point transfers within a collective communication are typically known during the collective algorithm execution, the places to apply DVFS and throttling may be determined in advance for a given message size. For example, an a priori algorithmic analysis of the MPI_Alltoall algorithms reveals that, for a few initial iterations, every core undergoes intranode communications after which the communication becomes purely internode. Thus, different throttling states may be preselected while the DVFS is lowered to the maximum at the start of MPI_Alltoall and raised back to the highest level in the end. A single point-to-point communication is, on the other hand, just one call per given processor rank. Therefore, its frequency scaling on the percall basis, as was done for collectives, may easily result in a significant parallel performance degradation due to the switching overhead. Nevertheless, it is desirable to decrease the energy consumption during the point-to-point communications, in addition to collectives, because they may constitute a significant portion of the execution (often more than 10%), and applications communicating heavily in the point-to-point fashion are abundant. Therefore, this work focuses on a class of point-to-point communications as provided by the MPI standard.

By considering test cases from the NAS benchmark suite  [2], this work validates the proposed runtime procedure that groups several point-to-point communications, aiming to reduce the overhead from the DVFS and throttling. Next, it applies this procedure to realistic electronic structure calculations performed by the widely-used GAMESS quantum chemistry package  [12], [29], which is capable of performing molecular structure and property calculations by a rich variety of ab initio methods. An estimated user base of 150,000 comes from more than 100 countries. The GAMESS communication library, which has been custom-built, is based on the partitioned global address space (PGAS) concepts.

The rest of paper is organized as follows. Section  2 describes the design of the runtime procedure for analyzing the point-to-point communications during the application execution. Section  3 discusses the communication model of GAMESS and techniques to apply efficiently the runtime communication analysis to GAMESS. Section  4 presents the experimental results while Sections  5 Related work, 6 Conclusions and future work provide the related work and conclusions, respectively.