Abstract
This paper presents an optimization of MPI communications, called CoMPI, based on run-time compression of MPI messages exchanged by applications. A broad number of compression algorithms have been fully implemented and tested for both MPI collective and point to point primitives. In addition, this paper presents a study of several compression algorithms that can be used for run-time compression, based on the datatype used by applications. This study has been validated by using several MPI benchmarks and real HPC applications. Show that, in most of the cases, using compression reduces the application communication time enhancing application performance and scalability. In this way, CoMPI obtains important improvements in the overall execution time for many of the considered scenarios.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Gropp, W., Lusk, E.: Sowing MPICH: A case study in the dissemination of a portable environment for parallel scientific computing. The International Journal of Supercomputer Applications and High Performance Computing 11(2), 103–114 (1997)
Balkanski, D., Trams, M., Rehm, W.: Heterogeneous computing with mpich/madeleine and pacx mpi: a critical comparison (2003)
Keller, R.,, M.L.: Using pacx-mpi in metacomputing applications. In: 18th Symposium Simulationstechnique, Erlangen, September 12-15 (2005)
Paruj, R., Jian, K., Martin, B.: Fast lossless compression of scientific floating-point data. In: DCC 2006: Proceedings of the Data Compression Conference, Washington, DC, USA, pp. 133–142. IEEE Computer Society Press, Los Alamitos (2006)
Jian, K., Martin, B., Evan, S.: Runtime compression of mpi messanes to improve the performance and scalability of parallel applications. In: SC 2004: Proceedings of the 2004 ACM/IEEE conference on Supercomputing, Washington, DC, USA, p. 59. IEEE Computer Society Press, Los Alamitos (2004)
Carretero, J., No, J., Park, S.-S., Choudhary, A., Chen, P.: Compassion: a parallel I/O runtime system including chunking and compression for irregular applications. In: Proceedings of the International Conference on High-Performance Computing and Networking, April 1998, pp. 668–677 (1998)
Garíca, F., Galderón, A., Carretero, J.: MiMPI: A multithread-safe implementation of MPI. In: Margalef, T., Dongarra, J., Luque, E. (eds.) PVM/MPI 1999. LNCS, vol. 1697, pp. 207–214. Springer, Heidelberg (1999)
Gropp, W.D., Thakur, R.: Issues in developing a thread-safe MPI implementation. In: Mohr, B., Träff, J.L., Worringen, J., Dongarra, J. (eds.) PVM/MPI 2006. LNCS, vol. 4192, pp. 12–21. Springer, Heidelberg (2006)
Zigon, R.: Run length encoding. j-DDJ 14(2) (February 1989)
Knuth, D.E.: Dynamic huffman coding. J. Algorithms 6(2), 163–180 (1985)
Coco, S., DArrigo, D.G.: A rice-based lossless data compression system for space. In: Proceedings of the 2000 IEEE Nordic Signal Processing Symposium, pp. 133–142 (2000)
Burtscher, M., Ratanaworabhan, P.: Fpc: A high-speed compressor for double-precision floating-point data. IEEE Transactions on Computers 58(1), 18–31 (2009)
Oberhumer, M.F.X.J.: Lzo real-time data compression library (2005)
Bell, T., Powell, M., J.H.R.A.: The canterbury corpus (2009)
Pichel, J.C., Singh, D.E., Rivera, F.F.: Image segmentation based on merging of sub-optimal segmentations. Pattern Recogn. Lett. 27(10), 1105–1116 (2006)
Mourino, J., Martin, M., Doallo, R., Singh, D., Rivera, F., Bruguera, J.: The stem-ii air quality model on a distributed memory system (2004)
Carter, R., Ciotti, B., Fineberg, S., Nitzberg, B.: NHT-1 I/O benchmarks. Technical Report RND-92-016, NAS Systems Division, NASA Ames (1992)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Filgueira, R., Singh, D.E., Calderón, A., Carretero, J. (2009). CoMPI: Enhancing MPI Based Applications Performance and Scalability Using Run-Time Compression. In: Ropo, M., Westerholm, J., Dongarra, J. (eds) Recent Advances in Parallel Virtual Machine and Message Passing Interface. EuroPVM/MPI 2009. Lecture Notes in Computer Science, vol 5759. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-03770-2_27
Download citation
DOI: https://doi.org/10.1007/978-3-642-03770-2_27
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-03769-6
Online ISBN: 978-3-642-03770-2
eBook Packages: Computer ScienceComputer Science (R0)