Adaptive Spatially Aware I/O for Multiresolution Particle Data Layouts
Large-scale simulations on nonuniform particle distributions that evolve over time are widely used in cosmology, molecular dynamics, and engineering. Such data are often saved in an unstructured format that neither preserves spatial locality nor provides metadata for accelerating spatial or attribute subset queries, leading to poor performance of visualization tasks. Furthermore, the parallel I/O strategy used typically writes a file per process or a single shared file, neither of which is portable or scalable across different HPC systems. We present a portable technique for scalable, spatially aware adaptive aggregation that preserves spatial locality in the output. We evaluate our approach on two supercomputers, Stampede2 and Summit, and demonstrate that it outperforms prior approaches at scale, achieving up to 2.5× faster writes and reads for nonuniform distributions. Furthermore, the layout written by our method is directly suitable for visual analytics, supporting low-latency reads and attribute-based filtering with little overhead.
- Research Organization:
- Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC)
- DOE Contract Number:
- AC05-00OR22725
- OSTI ID:
- 1820816
- Resource Relation:
- Conference: 35th IEEE International Parallel and Distributed Processing Symposium (IPDPS) - Virtual, Tennessee, United States of America - 5/17/2021 8:00:00 AM-5/21/2021 4:00:00 AM
- Country of Publication:
- United States
- Language:
- English
Similar Records
Using Pilot Jobs and CernVM File System for Simplified Use of Containers and Software Distribution
Fast Multiresolution Reads of Massive Simulation Datasets