Abstract
Clusters of SMPs are hybrid-parallel architectures that combine the main concepts of distributed-memory and shared-memory parallel machines. Although SMP clusters are widely used in the high performance computing community, there exists no single programming paradigm that allows exploiting the hierarchical structure of these machines. Most parallel applications deployed on SMP clusters are based on MPI, the standard API for distributed-memory parallel programming, and thus may miss a number of optimization opportunities offered by the shared memory available within SMP nodes. In this paper we present extensions to the data parallel programming language HPF and associated compilation techniques for optimizing HPF programs on clusters of SMPs. The proposed extensions enable programmers to control key aspects of distributed-memory and shared-memory parallelization at a high-level of abstraction. Based on these language extensions, a compiler can adopt a hybrid parallelization strategy which closely reflects the hierarchical structure of SMP clusters by automatically exploiting shared-memory parallelism based on OpenMP within cluster nodes and distributed-memory parallelism utilizing MPI across nodes. We describe the implementation of these features in the VFC compiler and present experimental results which show the effectiveness of these techniques.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
REFERENCES
Message Passing Interface Forum, MPI: A Message-Passing Interface Standard, Vers. 1.1, June 1995, MPI-2: Extensions to the Message-Passing Interface, 1997.
The OpenMP Forum, OpenMP Fortran Application Program Interface, Version 1.1, November 1999, http://www.openmp.org.
F. Cappello and D. Etieble, MPI versus MPI+ on the IBM SP for the NAS Benchmarks, In Proceedings of SC 2000: High Performance Networking and Computing Conference, Dallas (November 2000).
O. Haan, Matrix Transpose with Hybrid OpenMP/MPI Parallelization. Technical Report, http://www.spscicomp.org/2000/userpres.html#haan, 2000.
R. D. Loft, S. J. Thomas, and J. M. Dennis, Terascale Spectral Element Dynamical Core for Atmospheric General Circulation Models. In Proceedings SC2001, Denver (November 2001).
P. V. Luong, C. P. Breshears, and L. N. Ly, Costal Ocean Modeling of the U.S. West Coast with Multiblock Grid and Dual-Level Parallelism. In Proceedings of SC2001, Denver (November 2001).
High Performance Fortran Forum, High Performance Fortran Language Specification, Version 2.0, Department of Computer Science, Rice University (1997).
S. Benkner, VFC: The Vienna Fortran Compiler, Scientific Programming, 7(1):67-81 (1999).
S. Benkner, HPF+High Performance Fortran for Advanced Scientific and Engineering Applications, Future Generation Computer Systems, Vol. 15 (3) (1999).
S. Benkner, P. Mehrotra, J. Van Rosendale, and H. Zima, High-Level Management of Communication Schedules in HPF-like Languages, In Proceedings of the International Conference on Supercomputing (ICS'98), pp. 109-116, Melbourne, Australia, ACM Press (July 13–17, 1998).
S. Benkner, Optimizing Irregular HPF Applications Using Halos, Concurrency: Practice and Experience, Wiley (2000).
S. Benkner and T. Brandes, Exploiting Data Locality on Scalable Shared Memory Machines with Data Parallel Programs, In Euro-Par 2000 Parallel Processing, Lecture Notes in Computer Science 1900, Munich, Germany (September 2000).
H. Moritsch and S. Benkner, High Performance Numerical Pricing Methods, In Fourth Int'l. HPF Users Group Meeting, Tokyo (October 2000).
E. Dockner, H. Moritsch, G. Ch. Pflug, and A. Swietanowski, AURORA financial management system: From Model Design to Implementation, Technical report AURORA TR1998-08, University of Vienna (June 1998).
S. Benkner and T. Brandes. High-Level Data Mapping for Clusters of SMPs, In Proceedings 6th International Workshop on High-Level Parallel Programming Models and Supportive Environments, San Francisco, Springer-Verlag (April 2001).
T. Brandes and F. Zimmermann, ADAPTOR—A Transformation Tool for HPF Programs, In K. M. Decker and R. M. Rehmann (Eds.), Programming Environments for Massively Parallel Distributed Systems, Birkhäuser Verlag (1994).
D. S. Henty, Performance of Hybrid Message-Passing and Shared-Memory Parallelism for Discrete Element Modeling, In Proceedings of SC 2000: High Performance Networking and Computing Conference, Dallas (November 2000).
Silicon Graphics Inc. MIPSpro Power Fortran 77 Programmer's Guide: OpenMP Multiprocessing Directives, Technical Report Document 007-2361-007 (1999).
J. Bircsak, P. Craig, R. Crowell, Z. Cvetanovic, J. Harris, C. Nelson, and C. Offner, Extending OpenMP for NUMA Machines, In Proceedings of SC 2000: High Performance Networking and Computing Conference, Dallas (November 2000).
B. Chapman, P. Mehrotra, and H. Zima, Enhancing OpenMP with Features for Locality Control, In Proc. ECWMF Workshop “Towards Teracomputing—The Use of Parallel Processors in Meteorology” (1998).
M. Leair, J. Merlin, S. Nakamoto, V. Schuster, and M. Wolfe, Distributed OMP—A Programming Model for SMP Clusters, In Eighth International Workshop on Compilers for Parallel Computers, pp. 229-238, Aussois, France (January 2000).
J. Merlin, D. Miles, and V. Schuster, Extensions to OpenMP for SMP Clusters. In Proceedings of the Second European Workshop on OpenMP, EWOMP (2000).
Y. Hu, H. Lu, A. Cox, and W. Zwaenepel, Openmp for networks of smps, In Proceedings of IPPS. (1999).
M. Sato, S. Satoh, K. Kusano, and Y. Tanaka, Design of openmp compiler for an smp cluster, In Proceedings EWOMP '99, pp. 32-39 (1999).
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Benkner, S., Sipkova, V. Exploiting Distributed-Memory and Shared-Memory Parallelism on Clusters of SMPs with Data Parallel Programs. International Journal of Parallel Programming 31, 3–19 (2003). https://doi.org/10.1023/A:1021782118861
Issue Date:
DOI: https://doi.org/10.1023/A:1021782118861