COMPASSION: A parallel I/O runtime system including chunking and compression for irregular applications

Abstract

In this paper we present two designs, namely, “Collective I/O” and “Pipelined Collective I/O”, of a runtime library for irregular applications based on the two-phase collective 1/O technique. We also present the optimization of both models by using chunking and compression mechanisms. In the first scheme, all processors participate in compressions and I/O at the same time, making scheduling of I/O requests simpler but creating a possibility of contention at the I/O nodes. In the second approach, processors are grouped into several groups, overlapping communication, compression, and I/O to reduce I/O contention dynamically. Finally, evaluation results are shown that demonstrates that we can obtain significantly high-performance for I/O above what has been possible so far.

This work was supported in part by Sandia National Labs award AV-6193 under the ASCI program, and in part by NSF Young Investigator Award CCR-9357840 and NSF CCR-9509143. Jesus Carretero is a postdoctoral fellow at NWU supported by the NATO Science Fellowships Programme.