HeteroMPI: Towards a message-passing library for heterogeneous networks of computers

doi:10.1016/j.jpdc.2005.08.002

Journal of Parallel and Distributed Computing

Volume 66, Issue 2, February 2006, Pages 197-220

https://doi.org/10.1016/j.jpdc.2005.08.002 Get rights and content

Abstract

The paper presents Heterogeneous MPI (HeteroMPI), an extension of MPI for programming high-performance computations on heterogeneous networks of computers. It allows the application programmer to describe the performance model of the implemented algorithm in a generic form. This model allows the specification of all the main features of the underlying parallel algorithm, which have an impact on its execution performance. These features include the total number of parallel processes, the total volume of computations to be performed by each process, the total volume of data to be transferred between each pair of the processes, and how exactly the processes interact during the execution of the algorithm. Given a description of the performance model, HeteroMPI tries to create a group of processes that executes the algorithm faster than any other group. The principal extensions to MPI are presented. We demonstrate the features of the library by performing experiments with parallel simulation of the interaction of electric and magnetic fields and parallel matrix multiplication.

Section snippets

References (23)

J. Choi et al.
ScaLAPACK: a portable linear algebra library for distributed memory computers-design issues and performance
Comput. Phys. Commun.
(1996)
G.E. Fagg et al.
HARNESS and fault tolerant MPI
in: Parallel Comput.
(October 2001)
I. Foster et al.
Managing multiple communication methods in high-performance networked computing systems
Journal Parallel Distrib. Comput.
(January 1997)
W. Gropp et al.
A high-performance, portable implementation of the MPI message passing interface standard
Parallel Comput.
(September 1996)
A. Kalinov et al.
Heterogeneous distribution of computations solving linear algebra problems on networks of heterogeneous computers
Journal Parallel Distrib. Comput.
(April 2001)
A. Lastovetsky
Adaptive parallel computing on heterogeneous networks with mpC
Parallel Comput.
(October 2002)
A. Lastovetsky et al.
On performance analysis of heterogeneous parallel algorithms
Parallel Comput.
(2004)
O. Aumage, L. Bouge, R. Namyst, A portable and adaptative multi-protocol communication library for multithreaded...
R. Batchu, J. Neelamegam, Z. Cui, M. Beddhu, A. Skjellum, Y.S. Dandass, M. Apte, MPI/FT: architecture and taxonomies...
O. Beaumont et al.
A proposal for a heterogeneous cluster ScaLAPACK (Dense Linear Solvers)
IEEE Trans. Comput.
(October 2001)

M. Bhandarkar, L.V. Kale, E. de Sturler, J. Hoeflinger, object-based adaptive load balancing for MPI programs, in:...

Cited by (45)

Model-based selection of optimal MPI broadcast algorithms for multi-core clusters
2022, Journal of Parallel and Distributed Computing
Citation Excerpt :
The target architecture for the presented method is a cluster of multi-core processors. While this is the most common architecture where MPI is used, there are other architectures such as heterogeneous clusters [23], [42] or Internet of Things [22], [15], where MPI either is used or can be used in the future. Such platforms are highly heterogeneous, represent a significant challenge for accurate modeling of MPI collective algorithms and are out of the scope of this paper.
The performance of collective communication operations determines the overall performance of MPI applications. Different algorithms have been developed and implemented for each MPI collective operation, but none proved superior in all situations. Therefore, MPI implementations have to solve the problem of selecting the optimal algorithm for the collective operation depending on the platform, the number of processes involved, the message size(s), etc. The current solution method is purely empirical.
Recently, an alternative solution method using analytical performance models of collective algorithms has been proposed and proved both accurate and efficient for one-process-per-CPU configurations. The method derives the analytical performance models of algorithms from their code implementation rather than from high-level mathematical definitions, and estimates the parameters of the models separately for each algorithm. The method is network and topology oblivious and uses the Hockney model for point-to-point communications. In this paper, we extend that selection method to the case of clusters of multi-core processors, where each core of the platform runs a process of the MPI application.
We present the proposed approach using Open MPI broadcast algorithms, and experimentally validate it on three different clusters of multi-core processors, Grisou, Gros and MareNostrum4.
Accelerating Communications in Federated Applications with Transparent Object Proxies
2023, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2023
Accelerating Communications in Federated Applications with Transparent Object Proxies
2023, arXiv
Programming Reconfigurable Heterogeneous Computing Clusters Using MPI with Transpilation
2020, Proceedings of H2RC 2020: 6th International Workshop on Heterogeneous High-Performance Reconfigurable Computing, Held in conjunction with SC 2020: The International Conference for High Performance Computing, Networking, Storage and Analysis
Toward a BLAS library truly portable across different accelerator types
2019, Journal of Supercomputing
PRODA: improving parallel programs on GPUs through dependency analysis
2019, Cluster Computing

View all citing articles on Scopus

Alexey Lastovetsky received a Ph.D. from the Moscow Aviation Institute in 1986, and a Doctor of Sciences degree from the Russian Academy of Sciences in 1997. His main research interests include algorithms, models and programming tools for high performance heterogeneous computing. He published over 60 technical papers in refereed journals and international conferences. He authored the monograph “Parallel computing on heterogeneous networks” published by Wiley in 2003. He is currently a lecturer in the Department of Computer Science at University College Dublin, National University of Ireland. At UCD, he also leads Heterogeneous Computing Laboratory. He is on the editorial boards of the research journals “Parallel Computing” and “Programming and Computer Software”.

Ravi Reddy is currently a Ph.D. student in the Department of Computer Science at University College Dublin, National University of Ireland. His main research interests are design of algorithms and tools for parallel and distributed computing systems.

View full text

HeteroMPI: Towards a message-passing library for heterogeneous networks of computers

Abstract

Section snippets

Comput. Phys. Commun.

in: Parallel Comput.

Journal Parallel Distrib. Comput.

Parallel Comput.

Journal Parallel Distrib. Comput.

Parallel Comput.

Parallel Comput.

A proposal for a heterogeneous cluster ScaLAPACK (Dense Linear Solvers)

IEEE Trans. Comput.