Master of Puppets: Cooperative Multitasking for In Situ Processing
- Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Modern scientific and engineering simulations track the time evolution of billions of elements. For such large runs, storing most time steps for later analysis is not a viable strategy. It is far more efficient to analyze the simulation data while it is still in memory. Here, we present a novel design for running multiple codes in situ: using coroutines and position-independent executables we enable cooperative multitasking between simulation and analysis, allowing the same executables to post-process simulation output, as well as to process it on the fly, both in situ and in transit. We present Henson, an implementation of our design, and illustrate its versatility by tackling analysis tasks with different computational requirements. This design differs significantly from the existing frameworks and offers an efficient and robust approach to integrating multiple codes on modern supercomputers. The techniques we present can also be integrated into other in situ frameworks.
- Research Organization:
- Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
- DOE Contract Number:
- AC02-05CH11231
- OSTI ID:
- 1371719
- Report Number(s):
- LBNL-1005724; ir:1005724
- Resource Relation:
- Conference: 25th ACM International Symposium on High-Performance Parallel and Distributed Computing, Kyoto (Japan), 31 May - 4 Jun 2016
- Country of Publication:
- United States
- Language:
- English
The cosmological simulation code gadget-2
|
journal | December 2005 |
Similar Records
Multitasked embedded multigrid for three-dimensional flow simulation. Final report
Co-scheduling Ensembles of In Situ Workflows