Hierarchical Read–Write Optimizations for Scientific Applications with Multi-variable Structured Datasets

Abstract

Large-scale scientific applications spend a significant amount of time in reading and writing data. These simulations run on supercomputers which are architected with high-bandwidth, low-latency, and complex topology interconnects. Yet, few efforts exist that fully exploit the interconnect features for I/O. MPI-IO optimizations suffer from significant network contention at large core counts making I/O a critical bottleneck at extreme scales. We propose HieRO, which leverages the fast interconnect and performs hierarchical optimizations for I/O in scientific applications with structured datasets. HieRO performs reads/writes in multiple stages using carefully chosen leader processes who invoke the MPI-IO calls. Additionally, HieRO considers the application’s domain decomposition and access patterns and fully utilizes the on-chip interconnect at each multicore node. We evaluate the efficacy of our optimizations with two scientific applications, WRF and S3D, with I/O access patterns commonly used in a wide gamut of applications. We evaluate our approaches on two supercomputers, the Edison Cray XC30 and the Mira Blue Gene/Q, representing systems with diverse interconnects and parallel filesystems. We demonstrate that algorithmic changes can lead to significant improvements in parallel read/write. HieRO is able to achieve more than $40\times$ read time improvements for WRF and achieve up to $40\times$ read and $13\times$ write time improvements for S3D on 524288 cores.