A high performance matrix multiplication algorithm for MPPs

Abstract

A 3-dimensional (3-D) matrix multiplication algorithm for massively parallel processing systems is presented. Performing the product of two matrices C=β C+α A B is viewed as solving a 2-dimensional problem in the 3-dimensional computational space. The three dimensions correspond to the matrices dimensions m, k, and n: A ∈ R^m×k, B ∈ R ^k×n, and C ∈ R ^m×n. The p processors are configured as a “virtual” processing cube with dimensions p ₁, P ₂, and p ₃. The cube's dimensions are proportional to the matrices' dimensions-m, n, and k. Each processor performs a local matrix multiplication of size m/p ₁ × n/p ₂ × k/p ₃, on one of the sub-cubes in the computational space. Before the local computation can be carried out, each sub-cube needs to receive sub-matrices corresponding to the planes where A and B reside. After the single matrix multiplication has completed, the sub-matrices of C have to be reassigned to their respective processors. The 3-D parallel matrix multiplication approach has, to the best of our knowledge, the least amount of communication among all known parallel algorithms for matrix multiplication. Furthermore, the single resulting sub-matrix computation gives the best possible performance from the uni-processor matrix multiply routine. The 3-D approach achieves high performance for even relatively small matrices and/or a large number of processors (massively parallel). This algorithm has been implemented on IBM Power-parallel SP-2 systems (up to 216 nodes) and have yielded close to the peak performance of the machine. For large matrices, the algorithm can be combined with Winograd's variant of Strassen's algorithm to achieve “super-linear” speed-up. When the Winograd approach is used, the performance achieved per processor exceeds the theoretical peak of the system.