Cache simulation for irregular memory traffic on multi-core CPUs: Case study on performance models for sparse matrix–vector multiplication

Highlights

• Method for estimating irregular data traffic in multi-core memory hierarchies.

• Detailed performance modelling for bandwidth-limited computations.

• Experiments quantifying bottlenecks of sparse matrix–vector multiplication.

Abstract

Parallel computations with irregular memory access patterns are often limited by the memory subsystems of multi-core CPUs, though it can be difficult to pinpoint and quantify performance bottlenecks precisely. We present a method for estimating volumes of data traffic caused by irregular, parallel computations on multi-core CPUs with memory hierarchies containing both private and shared caches. Further, we describe a performance model based on these estimates that applies to bandwidth-limited computations. As a case study, we consider two standard algorithms for sparse matrix–vector multiplication, a widely used, irregular kernel. Using three different multi-core CPU systems and a set of matrices that induce a range of irregular memory access patterns, we demonstrate that our cache simulation combined with the proposed performance model accurately quantifies performance bottlenecks that would not be detected using standard best- or worst-case estimates of the data traffic volume.

Introduction

Performance is a high priority in scientific computations, and so meticulous work is devoted to optimising the underlying code. During such optimisation efforts, performance models are valuable tools for directing attention towards pressure points, and indicating when optimisations are good enough and expending further effort would be unproductive. For instance, the popular Roofline model [41] bounds performance in terms of a CPU’s peak computational capacity and memory bandwidth together with an algorithm’s computational intensity. Because CPUs have hierarchical memories, the bandwidth and computational intensity can vary depending on the memory hierarchy level that is considered. Moreover, the computational intensity depends not only on parameters such as cache size, but also on the memory access pattern of the computation. Recently, more elaborate performance models have been developed for stencil codes [8], [32], [46], where they have been used to evaluate the effectiveness of spatial and temporal blocking optimisations. In these cases, the amounts of data transferred between levels of the memory hierarchy are known in advance, because memory accesses are predictable and depend only on the problem size and the order of the stencil. Unfortunately, this is not the case for irregular computations, where memory access patterns depend on data that may only be known at runtime.

When faced with irregular access patterns, the typical approach is to derive estimates of the memory traffic for the worst- or best-case scenarios. These are “paper and pencil” estimates that have the advantage of being cheap to produce, not requiring any implementation or actual machine to run. On the other hand, such estimates are crude and can in reality be far from the true data traffic volumes, thereby rendering little help in understanding the actual performance that is achieved. For example, Fig. 1 shows worst- and best-case estimates for sparse matrix–vector multiplication (SpMV), a widely used computational kernel that suffers from both irregularity and low computational intensity. Due to the considerable difference between the best- and worst-case data traffic, these estimates cannot provide much confidence if they are used to evaluate whether the performance of a given kernel implementation is good enough. In this case, more accurate estimation of data traffic volumes is needed for performance validation. In general, numerous computational kernels face the same issues due to irregular memory accesses that arise through the use of sparse data structures, such as graphs or unstructured meshes.

In this paper, we present a method for quantifying the amounts of data transferred between levels of a multi-core CPU’s memory hierarchy during irregular computations. The estimated data traffic volumes are produced by a trace-driven cache simulation that relies on a few basic assumptions and a simplified model of the memory hierarchy. Moreover, the method applies to memory hierarchies with shared caches, a common feature of contemporary multi-core CPUs, and a case that is not always addressed by existing analytical cache models [1], [12]. Because the proposed method is based on tracing a sequence of memory references, it requires some amount of computation that is likely to be at least as much as the cost of executing the kernel itself. However, the method remains applicable in cases where the actual machine in question is not available, or the data traffic cannot be quantified directly through hardware monitoring facilities, for example, because these facilities are unavailable, unreliable or the results are not easily interpreted.

Because of its importance and familiarity as an irregular computational kernel, we use SpMV to demonstrate that our cache simulation accurately quantifies the volumes of data transfers in the memory hierarchies of two Intel-based multi-core CPU systems. In turn, these data transfer volumes are used to give accurate performance predictions that are unavailable through the use of simple worst- or best-case estimates. We also give performance predictions for an AMD Epyc CPU, and explore some limitations of the proposed method using a variant of SpMV that not only includes irregular reads, but also irregular writes. Ultimately, these predictions result in a quantitative understanding of SpMV performance, which, for example, can be used to check that the observed performance of a given implementation matches our expectations, and that the implementation is free from hidden performance issues.

The remainder of this paper is organised as follows. In the next section, we describe our cache simulation approach for estimating the data traffic volumes of computations with irregular memory accesses. In Section 3, we present a performance model for bandwidth-limited computations, where the relevant data traffic volumes are used together with realistic memory and CPU cache bandwidths. Next, in Section 4, we recall standard SpMV algorithms for matrices in the compressed sparse row (CSR) and coordinate (COO) storage formats. We also review known bounds on the volume of data traffic generated by the CSR SpMV algorithm, which is later used to compare with the results of our cache simulation method. Then, in Section 5, we describe experiments that are used to validate the estimated data traffic volumes and the performance model for the studied SpMV algorithms. Finally, we briefly discuss related work in Section 6 and draw our conclusions in Section 7.