Abstract
An in situ co-processing visualization pipeline based on the Universal Data Junction (UDJ) library and Inshimtu is presented and used for processing data from Weather Research and Forecasting (WRF) simulations. For the common case of analyzing just a number of fields during simulation, UDJ transfers and redistributes the data in approximately \(6\%\) of the time needed by WRF for a MPI-IO output of all variables upon which a previous method with Inshimtu is based. The relative cost of transport and redistribution compared to IO remains approximately constant up to the highest considered node count without obvious impediments to scale further.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ayachit, U., et al.: Paraview catalyst: enabling in situ data analysis and visualization. In: ISAV2015: Proceedings of the First Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization, vol. 1, no. (1), pp. 25–29, November 2015. https://doi.org/10.1145/2828612.2828624. https://dl.acm.org/doi/10.1145/2828612.2828624
Bauer, A.C., et al.: In situ methods, infrastructures, and applications on high performance computing platforms. Comput. Graph. Forum 35(3), 577–597 (2016)
Boyuka, D.A., et al.: Transparent in situ data transformations in adios. In: Proceedings of the 14th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, CCGRID 2014, pp. 256–266. IEEE Press (2014). https://doi.org/10.1109/CCGrid.2014.73
Godoy, W.F., et al.: ADIOS 2: the adaptable input output system. A framework for high-performance data management. SoftwareX 12, 100561 (2020). https://doi.org/10.1016/j.softx.2020.100561. https://www.sciencedirect.com/science/article/pii/S2352711019302560
Holst, G., Dasari, H.P., Markomanolis, G., Hoteit, I., Theussl, T.: Inshimtu - a lightweight in-situ visualization “shim” (2017). https://woiv.gitlab.io/woiv17/ISC_WOIV_Holst.pdf
Loring, B., et al.: Improving performance of m-to-n processing and data redistribution in in transit analysis and visualization. Technical report, Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States) (2020)
Moreland, K.: The tensions of in situ visualization. IEEE Comput. Graphics Appl. 36(2), 5–9 (2016). https://doi.org/10.1109/MCG.2016.35
Skamarock, W.C., et al.: A description of the advanced research WRF version 3. National Center for Atmospheric Research: Boulder, CO, USA, June 2008. https://doi.org/10.5065/D68S4MVH
Skamarock, W.C., et al.: A description of the advanced research WRF model version 4. National Center for Atmospheric Research: Boulder, CO, USA, p. 145 (2019). https://doi.org/10.5065/1dfh-6p97
Acknowledgment
This work is part of the HPE/Cray center of excellence collaboration at KAUST. UDJ development has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 773897. We want to thank Hari Dasari colleagues for helping with the test case as well as Tim Dykes and Utz Uwe Haus from the HPE EMEA research lab for support with UDJ.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Esposito, A., Holst, G. (2021). In Situ Visualization of WRF Data Using Universal Data Junction. In: Jagode, H., Anzt, H., Ltaief, H., Luszczek, P. (eds) High Performance Computing. ISC High Performance 2021. Lecture Notes in Computer Science(), vol 12761. Springer, Cham. https://doi.org/10.1007/978-3-030-90539-2_32
Download citation
DOI: https://doi.org/10.1007/978-3-030-90539-2_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-90538-5
Online ISBN: 978-3-030-90539-2
eBook Packages: Computer ScienceComputer Science (R0)