I/O in WRF: A Case Study in Modern Parallel I/O Techniques
Authors: 
Zanhua Huang, Kaiyuan Hou, Ankit Agrawal, 
Alok Choudhary, Robert Ross, Wei-Keng LiaoAuthors Info & Claims
Zanhua Huang, Kaiyuan Hou, Ankit Agrawal, Alok Choudhary, Robert Ross, Wei-Keng LiaoAuthors Info & ClaimsArticle No.: 94, Pages 1 - 13
Abstract
Large-scale parallel applications can face significant I/O performance bottlenecks, making efficient I/O crucial. This work presents a comparative study of several parallel I/O implementations in the Weather Research and Forecasting model, including PnetCDF blocking and non-blocking I/O options, netCDF4, HDF5 Log VOL, and ADIOS. For I/O methods creating files in a canonical data layout, PnetCDF's non-blocking option offers up to 2x improvement over its blocking option and up to 4.5x over HDF5 via netCDF4, demonstrating the effectiveness of the write request aggregation technique. The HDF5 Log VOL outperforms ADIOS with a 4x improvement in write performance when creating files in the log layout, although both require non-negligible time to convert the file back to canonical order for post-run analysis. From these results we extract some observations that can guide I/O strategies for modern parallel codes.
Supplemental Material
MP4 File - SC23 paper presentation recording for "I/O in WRF: A Case Study in Modern Parallel I/O Techniques"
SC23 paper presentation recording for "I/O in WRF: A Case Study in Modern Parallel I/O Techniques", by Zanhua Huang, Kaiyuan Hou, Ankit Agrawal, Alok Choudhary, Robert Ross and Wei-Keng Liao
- Download
- 194.11 MB
References
[1]
[n. d.]. Community Multiscale Air Quality Modeling System (CMAQ). Zenodo. Software.
[2]
Ehtesham Ahmed, Naeem Saddique, Firas Al Janabi, Klemens Barfus, Malik Rizwan Asghar, Abid Sarwar, and Peter Krebs. 2023. Flood Predictability of One-Way and Two-Way WRF Nesting Coupled Hydrometeorological Flow Simulations in a Transboundary Chenab River Basin, Pakistan. Remote Sens. 15, 2 (2023), 457.
[3]
Katie Antypas, Nicholas Wright, Nicholas P Cardo, Allison Andrews, and Matthew Cordery. 2014. Cori: A Cray XC pre-exascale system for NERSC. Cray User Group Proceedings. Cray (2014).
[4]
Tricia Balle and Pete Johnsen. 2016. Improving I/O Performance of the Weather Research and Forecast (WRF) Model.
[5]
Suren Byna, Mohamad Chaarawi, Quincey Koziol, John Mainzer, and Frank Willmore. 2017. Tuning HDF5 Subfiling Performance on Parallel File Systems. In the Cray User Group Meeting.
[6]
Philip Carns, Kevin Harms, William Allcock, Charles Bacon, Samuel Lang, Robert Latham, and Robert Ross. 2011. Understanding and Improving Computational Science Storage Access through Continuous Characterization. ACM Trans. Storage 7, 3, Article 8 (oct 2011), 26 pages.
[7]
Jacqueline Chen, Alok Choudhary, Bronis R. de Supinski, Matt DeVries, Evatt Hawkes, Scott Klasky, Wei-keng Liao, Kwan-Liu Ma, Jim Crummey, Norbert Podhorszki, Ramanan Sankaran, Sameer Shende, and Chialin Yoo. 2009. Terascale Direct Numerical Simulations of Turbulent Combustion Using S3D. Computational Science and Discovery 2 (January 2009).
[8]
Jacob Finkenrath, Giannis Koutsou, Swen Metzger, Hendrik Elbern, and Jonas Berndt. 2019. Approaching exascale with the Weather Research and Forecasting Solar model.
[9]
Kui Gao, Chen Jin, Alok Choudhary, and Wei-Keng Liao. 2011. Supporting computational data model representation with high-performance I/O in parallel netCDF. In 2011 18th International Conference on High Performance Computing. 1--10.
[10]
Kui Gao, Wei-Keng Liao, Alok Choudhary, Robert Ross, and Robert Latham. 2009. Combining I/O operations for multiple array variables in parallel netCDF. In 2009 IEEE International Conference on Cluster Computing and Workshops. 1--10.
[11]
Kui Gao, Wei-Keng Liao, Arifa Nisar, Alok N. Choudhary, Robert B. Ross, and Robert Latham. 2009. Using Subfiling to Improve Programming Flexibility and Performance of Parallel Shared-file I/O. In the International Conference Parallel Processing.
[12]
William F. Godoy, Norbert Podhorszki, Ruonan Wang, Chuck Atkins, Greg Eisenhauer, Junmin Gu, Philip Davis, Jong Choi, Kai Germaschewski, Kevin Huck, Axel Huebl, Mark Kim, James Kress, Tahsin Kurc, Qing Liu, Jeremy Logan, Kshitij Mehta, George Ostrouchov, Manish Parashar, Franz Poeschel, David Pugmire, Eric Suchyta, Keichi Takahashi, Nick Thompson, Seiji Tsutsumi, Lipeng Wan, Matthew Wolf, Kesheng Wu, and Scott Klasky. 2020. ADIOS 2: The Adaptable Input Output System. A framework for high-performance data management. SoftwareX 12 (2020), 100561.
[13]
The HDF Group. 1997--2023. Hierarchical Data Format, version 5. https://www.hdfgroup.org/HDF5.
[14]
The HDF Group. 2022. HDF5 release 1.14.0. https://portal.hdfgroup.org/display/support/HDF5+1.14.0.
[15]
Qiao Kang, Sunwoo Lee, Kaiyuan Hou, Robert Ross, Ankit Agrawal, Alok Choudhary, and Wei-Keng Liao. 2020. Improving MPI Collective I/O for High Volume Non-Contiguous Requests with Intra-Node Aggregation. IEEE Transactions on Parallel and Distributed Systems 31, 11 (2020), 2682--2695.
[16]
Akira Kyle. 2018. Weather Research and Forecast (WRF) Scaling, Performance Assessment and Optimization. https://akirakyle.github.io/WRF_benchmarks/. Accessed on June 2022.
[17]
Rob Latham, Chris Daley, Wei-keng Liao, Kui Gao, Rob Ross, Anshu Dubey, and Alok Choudhary. 2012. A case study for scientific I/O: improving the FLASH astrophysics code. Computational Science & Discovery 5, 1 (2012), 015001.
[18]
Michael Laufer. 2022. WRF-ADIOS2-to-NetCDF4. https://github.com/MichaelLaufer/WRF-ADIOS2-to-NetCDF4. Accessed on Mar, 2023.
[19]
Michael Laufer and Erick Fredj. 2022. High Performance Parallel I/O and In-Situ Analysis in the WRF Model with ADIOS2. CoRR abs/2201.08228 (2022). arXiv:2201.08228 https://arxiv.org/abs/2201.08228
[20]
Jianwei Li, Wei-Keng Liao, Alok Choudhary, Robert Ross, Rajeev Thakur, William Gropp, Rob Latham, Andrew Siegel, Brad Gallagher, and Michael Zingale. 2003. Parallel netCDF: A High-Performance Scientific I/O Interface. In SC Conference. IEEE Computer Society, Los Alamitos, CA, USA, 39.
[21]
Wei-Keng Liao and Alok Choudhary. 2008. Dynamically Adapting File Domain Partitioning Methods for Collective I/O Based on Underlying Parallel File System Locking Protocols. In the ACM/IEEE Conference on Supercomputing.
[22]
Wei-Keng Liao, Kaiyuan Hou, and Zanhua Huang. 2019--2023. Log VOL. https://github.com/DataLib-ECP/vol-log-based.
[23]
Barry H. Lynn, Seth Cohen, Leonard Druyan, Adam S. Phillips, Dennis Shea, Haim-Zvi Krugliak, and Alexander P. Khain. 2020. An Examination of the Impact of Grid Spacing on WRF Simulations of Wintertime Precipitation in the Mid-Atlantic United States. Weather and Forecasting 35, 6 (2020), 2317--2343.
[24]
Message Passing Interface Forum. 2021. MPI: A Message-Passing Interface Standard Version 4.0. https://www.mpi-forum.org
[25]
OpenFOAM Foundation. [n. d.]. OpenFOAM - The Open Source Computational Fluid Dynamics (CFD) Toolbox. https://www.openfoam.org.
[26]
Juan Rosario, Rajesh Bordawekar, and Alok Choudhary. 1993. Improved parallel I/O via a two-phase run-time access strategy. ACM SIGARCH Computer Architecture News 21 (12 1993), 31--38.
[27]
Manu Shantharam, Mahidhar Tatineni, Dongju Choi, and Amitava Majumdar. 2018. Understanding I/O Bottlenecks and Tuning for High Performance I/O on Large HPC Systems: A Case Study. In Proceedings of the Practice and Experience on Advanced Research Computing (Pittsburgh, PA, USA) (PEARC '18). Association for Computing Machinery, New York, NY, USA, Article 54, 6 pages.
[28]
William C. Skamarock, Joseph B. Klemp, Jimy Dudhia, David O. Gill, Zhiquan Liu, Judith Berner, Wei Wang, Jordan G. Powers, Michael G. Duda, Dale M. Barker, and Xiang-yu Huang. 2019. A description of the advanced research WRF Version 4. techreport NCAR/TN-556+STR. NCAR Tech. 145 pages.
[29]
R Thakur, E Lusk, and W Gropp. 1997. Users guide for ROMIO: A high-performance, portable MPI-IO implementation. (10 1997).
[30]
Unidata. 2022. NetCDF version 4.9.0 [software].
[31]
Xiwen Wang, Weijia Wang, Yuan He, Shulei Zhang, Wei Huang, R. Iestyn Woolway, Kun Shi, and Xiaofan Yang. 2023. Numerical simulation of thermal stratification in Lake Qiandaohu using an improved WRF-Lake model. Journal of Hydrology 618 (2023), 129184.
[32]
D. C. Wong, C. E. Yang, J. S. Fu, K. Wong, and Y. Gao. 2015. An approach to enhance PnetCDF performance in environmental modeling applications. Geoscientific Model Development 8, 4 (2015), 1033--1046.
Index Terms
- I/O in WRF: A Case Study in Modern Parallel I/O Techniques
Recommendations
Performance Evaluation of Software RAID vs. Hardware RAID for Parallel Virtual File System
ICPADS '02: Proceedings of the 9th International Conference on Parallel and Distributed SystemsLinux clusters of commodity computer systems and interconnectshave become the fastest growing choice for buildingcost-effective high-performance parallel computing systems.The Parallel Virtual File System (PVFS) could potentially fulfillthe requirements ...
Large files, small writes, and pNFS
ICS '06: Proceedings of the 20th annual international conference on SupercomputingWorkload characterization studies highlight the prevalence of small and sequential data requests in scientific applications. Parallel file systems excel at large data transfers but sometimes at the expense of small I/O performance. pNFS is an NFSv4.1 ...
MPI-IO/Gfarm: An Optimized Implementation of MPI-IO for the Gfarm File System
CCGRID '11: Proceedings of the 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid ComputingThis paper proposes a design and implementation of an MPI-IO implementation of the Gfarm file system, called MPI-IO/Gfarm. The Gfarm file system is a global file system that federates the local storage of compute nodes among several clusters. It has a ...
Comments
Information & Contributors
Information
Published In
November 2023
1428 pages
ISBN:9798400701092
DOI:10.1145/3581784
- Chair:
- Dorian Arnold,
- Program Chair:
- Rosa M Badia,
- Program Co-chair:
- Kathryn Mohror
Copyright © 2023 ACM.
Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 November 2023
Check for updates
Badges
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
SC '23
Sponsor:
SC '23: International Conference for High Performance Computing, Networking, Storage and Analysis
November 12 - 17, 2023
CO, Denver, USA
Acceptance Rates
Overall Acceptance Rate 1,516 of 6,373 submissions, 24%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 556Total Downloads
- Downloads (Last 12 months)296
- Downloads (Last 6 weeks)59
Reflects downloads up to 27 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in