ABSTRACT
The volume of data generated by exascale simulations requires scalable tools for analysis and visualization. Due to the relatively low I/O bandwidth of modern HPC systems, it is crucial to work as close as possible with simulated data via in situ approaches. In situ visualization provides insights into simulation data and, with the help of additional interactive analysis tools, can support the scientific discovery process at an early stage. Such in situ visualization tools need to be hardware-independent given the ever-increasing hardware diversity of modern supercomputers. We present a new in situ 3D vector field visualization algorithm for particle-in-cell (PIC) simulations and performance evaluation of the solution developed at large-scale. We create a solution in a hardware-agnostic approach to support high throughput and interactive in situ processing on leadership class computing systems. To that end, we demonstrate performance portability on Summit's and the Frontier's pre-exascale testbed at the Oak Ridge Leadership Computing Facility.
- Félicie Albert, M E Couprie, Alexander Debus, Mike C Downer, Jérôme Faure, Alessandro Flacco, Leonida A Gizzi, Thomas Grismayer, Axel Huebl, Chan Joshi, M Labat, Wim P Leemans, Andreas R Maier, Stuart P D Mangles, Paul Mason, François Mathieu, Patric Muggli, Mamiko Nishiuchi, Jens Osterhoff, P P Rajeev, Ulrich Schramm, Jörg Schreiber, Alec G R Thomas, Jean-Luc Vay, Marija Vranic, and Karl Zeil. 2021. 2020 roadmap on plasma accelerators. New Journal of Physics 23, 3 (3 2021), 031101. Google Scholar
Cross Ref
- AMD. 2022. AMD CDNA 2 architecture White Paper. Advanced Micro Devices, Inc.Google Scholar
- Kush Amerasinghe. 2010. H.264 for the rest of us. Technical Report. Adobe.Google Scholar
- Utkarsh Ayachit, Brad Whitlock, Matthew Wolf, Burlen Loring, Berk Geveci, David Lonie, and E. Wes Bethel. 2016. The SENSEI Generic In Situ Interface. In 2016 Second Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization (ISAV). 40--44. Google ScholarCross Ref
- S. Bachthaler, M. Strengert, D. Weiskopf, and T. Ertl. 2006. Parallel Texture-Based Vector Field Visualization on Curved Surfaces Using GPU Cluster Computers. In Proceedings of the 6th Eurographics Conference on Parallel Graphics and Visualization (Braga, Portugal) (EGPGV '06). Eurographics Association, Goslar, DEU, 75--83.Google Scholar
- A. C. Bauer, H. Abbasi, J. Ahrens, H. Childs, B. Geveci, S. Klasky, K. Moreland, P. O'Leary, V. Vishwanath, B. Whitlock, and E. W. Bethel. 2016. In Situ Methods, Infrastructures, and Applications on High Performance Computing Platforms. Comput. Graph. Forum 35, 3 (jun 2016), 577--597.Google Scholar
- A. C. Bauer, B. Geveci, and W. Schroeder. 2013. The ParaView Catalyst user's guide. Kitware Inc. 67 pages.Google Scholar
- James F. Blinn. 1977. Models of Light Reflection for Computer Synthesized Pictures. In Proceedings of the 4th Annual Conference on Computer Graphics and Interactive Techniques (San Jose, California) (SIGGRAPH '77). Association for Computing Machinery, New York, NY, USA, 192--198. Google ScholarDigital Library
- Paul Bourke. 1997. Trilinear interpolation. http://paulbourke.net/miscellaneous/\interpolation/. Accessed: 2022-03-26.Google Scholar
- K. J. Bowers, B. J. Albright, L. Yin, B. Bergen, and T. J. T. Kwan. 2008. Ultrahigh performance three-dimensional electromagnetic relativistic kinetic plasma simulation. Physics of Plasmas 15, 5 (2008), 055703. Google ScholarCross Ref
- M. Bussmann, H. Burau, T. E. Cowan, A. Debus, A. Huebl, G. Juckeland, T. Kluge, W. E. Nagel, R. Pausch, F. Schmitt, U. Schramm, J. Schuchart, and R. Widera. 2013. Radiative signatures of the relativistic Kelvin-Helmholtz instability. In SC '13 Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 5-1 -- 5-12. Google ScholarDigital Library
- Brian Cabral and Leith Casey Leedom. 1993. Imaging Vector Fields Using Line Integral Convolution. In Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques (Anaheim, CA) (SIGGRAPH '93). Association for Computing Machinery, New York, NY, USA, 263--270. Google ScholarDigital Library
- Hank Childs, Sean D. Ahern, James Ahrens, Andrew C. Bauer, Janine Bennett, E. Wes Bethel, Peer-Timo Bremer, Eric Brugger, Joseph Cottam, Matthieu Dorier, Soumya Dutta, Jean M. Favre, Thomas Fogal, Steffen Frey, Christoph Garth, Berk Geveci, William F. Godoy, Charles D. Hansen, Cyrus Harrison, Bernd Hentschel, Joseph Insley, Chris R. Johnson, Scott Klasky, Aaron Knoll, James Kress, Matthew Larsen, Jay Lofstead, Kwan-Liu Ma, Preeti Malakar, Jeremy Meredith, Kenneth Moreland, Paul Navrátil, Patrick O'Leary, Manish Parashar, Valerio Pascucci, John Patchett, Tom Peterka, Steve Petruzza, Norbert Podhorszki, David Pugmire, Michel Rasquin, Silvio Rizzi, David H. Rogers, Sudhanshu Sane, Franz Sauer, Robert Sisneros, Han-Wei Shen, Will Usher, Rhonda Vickery, Venkatram Vishwanath, Ingo Wald, Ruonan Wang, Gunther H. Weber, Brad Whitlock, Matthew Wolf, Hongfeng Yu, and Sean B. Ziegeler. 2020. A terminology for in situ visualization and analysis systems. The International Journal of High Performance Computing Applications 34, 6 (2020), 676--691. Google ScholarDigital Library
- S. Corde, K. Ta Phuoc, G. Lambert, R. Fitour, V. Malka, A. Rousse, A. Beck, and E. Lefebvre. 2013. Femtosecond x rays from laser-plasma accelerators. Reviews of Modern Physics 85, 1 (2013), 1--48. Google ScholarCross Ref
- J. P. Couperus, R. Pausch, A. Köhler, O. Zarini, J. M. Krämer, M. Garten, A. Huebl, R. Gebhardt, U. Helbig, S. Bock, K. Zeil, A. Debus, M. Bussmann, U. Schramm, and A. Irman. 2017. Demonstration of a beam loaded nanocoulomb-class laser wakefield accelerator. Nature Communications 8, 1 (2017), 1--7. Google ScholarCross Ref
- Alexander Debus, Richard Pausch, Axel Huebl, Klaus Steiniger, René Widera, Thomas E. Cowan, Ulrich Schramm, and Michael Bussmann. 2019. Circumventing the Dephasing and Depletion Limits of Laser-Wakefield Acceleration. Physical Review X 9, 3 (9 2019), 031044. Google ScholarCross Ref
- Zi'ang Ding, Zhanping Liu, Yang Yu, and Wei Chen. 2015. Parallel unsteady flow line integral convolution for high-performance dense visualization. In 2015 IEEE Pacific Visualization Symposium (PacificVis). 25--30. Google ScholarCross Ref
- E. Esarey, C. B. Schroeder, and W. P. Leemans. 2009. Physics of laser-driven plasma-based electron accelerators. Reviews of Modern Physics 81, 3 (8 2009), 1229--1285. Google ScholarCross Ref
- Martin Falk and Daniel Weiskopf. 2008. Output-Sensitive 3D Line Integral Convolution. IEEE Transactions on Visualization and Computer Graphics 14, 4 (2008), 820--834. Google ScholarDigital Library
- L.K. Forssell and S.D. Cohen. 1995. Using line integral convolution for flow visualization: curvilinear grids, variable-speed animation, and unsteady flows. IEEE Transactions on Visualization and Computer Graphics 1, 2 (1995), 133--141. Google ScholarDigital Library
- GTrunc. 2022. OpenGL Mathematics (GLM). https://github.com/g-truc/glm. Accessed: 2022-03-26.Google Scholar
- William F. Godoy, Norbert Podhorszki, Ruonan Wang, Chuck Atkins, Greg Eisenhauer, Junmin Gu, Philip Davis, Jong Choi, Kai Germaschewski, Kevin Huck, Axel Huebl, Mark Kim, James Kress, Tahsin Kurc, Qing Liu, Jeremy Logan, Kshitij Mehta, George Ostrouchov, Manish Parashar, Franz Poeschel, David Pugmire, Eric Suchyta, Keichi Takahashi, Nick Thompson, Seiji Tsutsumi, Lipeng Wan, Matthew Wolf, Kesheng Wu, and Scott Klasky. 2020. ADIOS 2: The Adaptable Input Output System. A framework for high-performance data management. Software X 12 (2020), 100561.Google ScholarCross Ref
- A. V. Pascal Grosset, Manasa Prasad, Cameron Christensen, Aaron Knoll, and Charles Hansen. 2017. TOD-Tree: Task-Overlapped Direct Send Tree Image Compositing for Hybrid MPI Parallelism and GPUs. IEEE Transactions on Visualization and Computer Graphics 23, 6 (2017), 1677--1690. Google ScholarDigital Library
- J. H. Halton. 1964. Algorithm 247: Radical-Inverse Quasi-Random Point Sequence. Commun. ACM 7, 12 (dec 1964), 701--702. Google ScholarDigital Library
- Benjamín Hernández and Tim Biedert. 2019. GPU Enhanced Remote Collaborative Scientific Visualization. Presented at the NVIDIA GPU Technology Conference, Silicon Valley, CA.Google Scholar
- I. Hotz, L. Feng, H. Hagen, B. Hamann, K. Joy, and B. Jeremic. 2004. Physically based methods for tensor field visualization. In IEEE Visualization 2004. 123--130. Google ScholarDigital Library
- Axel Huebl, René Widera, Felix Schmitt, Alexander Matthes, Norbert Podhorszki, Jong Youl Choi, Scott Klasky, and Michael Bussmann. 2017. On the Scalability of Data Reduction Techniques in Current and Upcoming HPC Systems from an Application Perspective. In High Performance Computing, Julian M. Kunkel, Rio Yokota, Michela Taufer, and John Shalf (Eds.). Springer International Publishing, Cham, 15--29.Google Scholar
- Intel. 2022. An Open, Scalable, Portable, Ray Tracing Based Rendering Engine for High-Fidelity Visualization. https://github.com/ospray/OSPRay. Accessed: 2022-03-26.Google Scholar
- Bruno Jobard, Gordon Erlebacher, and M. Yousuff Hussaini. 2000. Hardware-Accelerated Texture Advection for Unsteady Flow Visualization. In Proceedings of the Conference on Visualization '00 (Salt Lake City, Utah, USA) (VIS '00). IEEE Computer Society Press, Washington, DC, USA, 155--162.Google Scholar
- T. Kurz, T. Heinemann, M. F. Gilljohann, Y. Y. Chang, J. P. Couperus Cabadağ, A. Debus, O. Kononenko, R. Pausch, S. Schöbel, R. W. Assmann, M. Bussmann, H. Ding, J. Götzfried, A. Köhler, G. Raj, S. Schindler, K. Steiniger, O. Zarini, S. Corde, A. Döpp, B. Hidding, S. Karsch, U. Schramm, A. Martinez de la Ossa, and A. Irman. 2021. Demonstration of a compact plasma accelerator powered by laser-accelerated electron beams. Nature Communications 12, 1 (5 2021). Google ScholarCross Ref
- Matthew Larsen, James Ahrens, Utkarsh Ayachit, Eric Brugger, Hank Childs, Berk Geveci, and Cyrus Harrison. 2017. The ALPINE In Situ Infrastructure: Ascending from the Ashes of Strawman. In Proceedings of the In Situ Infrastructures on Enabling Extreme-Scale Analysis and Visualization (Denver, CO, USA) (ISAV'17). Association for Computing Machinery, New York, NY, USA, 42--46. Google ScholarDigital Library
- Matthew Leinhauser, René Widera, Sergei Bastrakov, Alexander Debus, Michael Bussmann, and Sunita Chandrasekaran. 2022. Metrics and Design of an Instruction Roofline Model for AMD GPUs. ACM Trans. Parallel Comput. 9, 1, Article 1 (jan 2022), 14 pages. Google ScholarDigital Library
- Matthew Leinhauser, Jeffrey Young, Sergei Bastrakov, Rene Widera, Ronnie Chatterjee, and Sunita Chandrasekaran. 2021. Performance Analysis of PIConGPU: Particle-in-Cell on GPUs using NVIDIA's NSight Systems and NSight Compute. Technical Report. Oak Ridge National Laboratory. Google Scholar
- Guo-Shi Li, Xavier Tricoche, and Charles Hansen. 2006. GPUFLIC: Interactive and Accurate Dense Visualization of Unsteady Flows. In Proceedings of the Eighth Joint Eurographics / IEEE VGTC Conference on Visualization (Lisbon, Portugal) (EUROVIS'06). Eurographics Association, Goslar, DEU, 29--34.Google Scholar
- Yangguang Liao. 2019. Visualization and Analysis of Vector Field Simulation Data. Ph. D. Dissertation. University of California Davis.Google Scholar
- Zhanping Liu and R.J. Moorhead. 2005. Accelerated unsteady flow line integral convolution. IEEE Transactions on Visualization and Computer Graphics 11, 2 (2005), 113--125. Google ScholarDigital Library
- Alexander Matthes, Axel Huebl, Rene Widera, Sebastian Grottel, Stefan Gumhold, and Michael Bussmann. 2016. In Situ, Steerable, Hardware-Independent and Data-Structure Agnostic Visualization with ISAAC. Supercomput. Front. Innov.: Int. J. 3, 4 (dec 2016), 30--48. Google ScholarDigital Library
- Melzani, Mickaël, Winisdoerffer, Christophe, Walder, Rolf, Folini, Doris, Favre, Jean M., Krastanov, Stefan, and Messmer, Peter. 2013. Apar-T: code, validation, and physical interpretation of particle-in-cell results. A&A 558 (2013), A133. Google ScholarCross Ref
- Kenneth Moreland, Christopher Sewell, William Usher, Li-ta Lo, Jeremy Meredith, David Pugmire, James Kress, Hendrik Schroots, Kwan-Liu Ma, Hank Childs, Matthew Larsen, Chun-Ming Chen, Robert Maynard, and Berk Geveci. 2016. VTK-m: Accelerating the Visualization Toolkit for Massively Threaded Architectures. IEEE Computer Graphics and Applications 36, 3 (2016), 48--58. Google ScholarDigital Library
- Kenneth D Moreland. 2009. IceT users' guide and reference. (6 2009). Google ScholarCross Ref
- NVIDIA. 2022. NVIDIA H100 Tensor Core GPU Architecture White Paper. NVIDIA Corporation.Google Scholar
- NVIDIA. 2022. VisRTX - NVIDIA RTX based implementation of ANARI. https://github.com/NVIDIA/VisRTX. Accessed: 2022-03-26.Google Scholar
- Oakridge Leadership Computing Facility. 2022. Crusher Quick-Start Guide. https://docs.olcf.ornl.gov/systems/crusher_quick_start_guide.html. [Online; accessed 09-December-2022].Google Scholar
- Oakridge Leadership Computing Facility. 2022. Summit User Guide. https://docs.olcf.ornl.gov/systems/summit_user_guide.html. [Online; accessed 09-December-2022].Google Scholar
- Nobuaki Ohno and Hiroaki Ohtani. 2015. Development of In-Situ Visualization Tool for PIC Simulation. Plasma and Fusion Research 9 (2015), 3401071--3401071. Google ScholarCross Ref
- OLCF. 2021. Storage Specifications for Frontier Exascale System. https://www.olcf.ornl.gov/2021/05/20/\olcf-announces-storage-specifications-for-frontier-exascale-system/. Accessed: 2022-03-24.Google Scholar
- R. Pausch, M. Bussmann, A. Huebl, U. Schramm, K. Steiniger, R. Widera, and A. Debus. 2017. Identifying the linear phase of the relativistic Kelvin-Helmholtz instability and measuring its growth rate via radiation. Physical Review E 96, 1 (2017). Google ScholarCross Ref
- Tom Peterka, Robert Ross, Attila Gyulassy, Valerio Pascucci, Wesley Kendall, Han-Wei Shen, Teng-Yok Lee, and Abon Chaudhuri. 2011. Scalable parallel building blocks for custom data analysis. In 2011 IEEE Symposium on Large Data Analysis and Visualization. 105--112. Google ScholarCross Ref
- Franz Poeschel, Juncheng E, William F. Godoy, Norbert Podhorszki, Scott Klasky, Greg Eisenhauer, Philip E. Davis, Lipeng Wan, Ana Gainaru, Junmin Gu, Fabian Koller, René Widera, Michael Bussmann, and Axel Huebl. 2022. Transitioning from File-Based HPC Workflows to Streaming Data Pipelines with openPMD and ADIOS2. In Driving Scientific and Engineering Discoveries Through the Integration of Experiment, Big Data, and Modeling and Simulation, Jeffrey Nichols, Arthur `Barney' Maccabe, James Nutaro, Swaroop Pophale, Pravallika Devineni, Theresa Ahearn, and Becky Verastegui (Eds.). Springer International Publishing, Cham, 99--118.Google Scholar
- Frits H. Post, Benjamin Vrolijk, Helwig Hauser, Robert S. Laramee, and Helmut Doleisch. 2002. Feature Extraction and Visualisation of Flow Fields. In Eurographics 2002 - STARs. Eurographics Association. Google ScholarCross Ref
- Alexancer Pukhov and Jürgen Meyer-ter Vehn. 2002. Laser wake field acceleration: the highly non-linear broken-wave regime. Applied Physics B 74, 4 (2002), 355--361.Google ScholarCross Ref
- C. Rezk-Salama, P. Hastreiter, C. Teitzel, and T. Ertl. 1999. Interactive exploration of volume line integral convolution based on 3D-texture mapping. In Proceedings Visualization '99. 233--528. Google ScholarCross Ref
- Marzia Rivi, Luigi Calori, Giuseppa Muscianisi, and Vladimir Slavnic. 2011. In-situ visualization: state-of-the-art and some use cases. Technical Report. Partnership for Advanced Computing in Europe (PRACE).Google Scholar
- Oliver Rübel, Burlen Loring, Jean-Luc Vay, David P. Grote, Remi Lehe, Stepan Bulanov, Henri Vincenti, and E. Wes Bethel. 2016. WarpIV: In Situ Visualization and Analysis of Ion Accelerator Simulations. IEEE Computer Graphics and Applications 36, 3 (2016), 22--35. Google ScholarDigital Library
- Christopher Sewell, Jeremy Meredith, Kenneth Moreland, Tom Peterka, Dave DeMarle, Li-ta Lo, James Ahrens, Robert Maynard, and Berk Geveci. 2012. The SDAV Software Frameworks for Visualization and Analysis on Next-Generation Multi-Core and Many-Core Architectures. In 2012 SC Companion: High Performance Computing, Networking Storage and Analysis. 206--214. Google ScholarDigital Library
- Han-Wei Shen and D.L. Kao. 1998. A new line integral convolution algorithm for visualizing time-varying flow fields. IEEE Transactions on Visualization and Computer Graphics 4, 2 (1998), 98--108. Google ScholarDigital Library
- Detlev Stalling and Hans-Christian Hege. 1995. Fast and Resolution Independent Line Integral Convolution. In Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '95). Association for Computing Machinery, New York, NY, USA, 249--256. Google ScholarDigital Library
- The Khronos Group. 2022. The ANARI (Analytic Rendering Interface) API. https://www.khronos.org/anari. Accessed: 2022-03-26.Google Scholar
- The WarpX Team. 2022. In situ Visualization with Ascent. https://warpx.readthedocs.io/en/latest/dataanalysis/ascent.html. Accessed: 2022-03-26.Google Scholar
- The WarpX Team. 2022. In situ Visualization with SENSEI. https://warpx.readthedocs.io/en/latest/dataanalysis/sensei.html. Accessed: 2022-03-26.Google Scholar
- Greg Turk and David Banks. 1996. Image-Guided Streamline Placement. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '96). Association for Computing Machinery, New York, NY, USA, 453--460. Google ScholarDigital Library
- R. Wegenkittl, E. Groller, and W. Purgathofer. 1997. Animating flow fields: rendering of oriented line integral convolution. In Proceedings. Computer Animation '97 (Cat. No.97TB100120). 15--21. Google ScholarCross Ref
- Brad Whitlock, Jean M. Favre, and Jeremy S. Meredith. 2011. Parallel in Situ Coupling of Simulation with a Fully Featured Visualization System. In Proceedings of the 11th Eurographics Conference on Parallel Graphics and Visualization (Llandudno, UK) (EGPGV '11). Eurographics Association, Goslar, DEU, 101--109.Google Scholar
- Erik Zenker, Benjamin Worpitz, René Widera, Axel Huebl, Guido Juckeland, Andreas Knüpfer, Wolfgang E Nagel, and Michael Bussmann. 2016. Alpaka-An Abstraction Library for Parallel Kernel Acceleration. In 2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). IEEE, 631--640.Google Scholar
Index Terms
- Hardware-Agnostic Interactive Exascale In Situ Visualization of Particle-In-Cell Simulations
Recommendations
Visualization for Exascale: Portable Performance is Critical
Researchers face a daunting task to provide scientific visualization capabilities for exascale computing. Of the many fundamental changes we are seeing in HPC systems, one of the most profound is a reliance on new processor types optimized for execution ...
Towards in situ visualization of extreme-scale, agent-based, worldwide disease-spreading simulations
SA '15: SIGGRAPH Asia 2015 Visualization in High Performance ComputingExascale computing is regarded as a major milestone for a number of application fields. However, utilizing the massively parallel power of such architectures requires a significant paradigm shift in the software structure of both simulation and ...
Manycore challenge in particle-in-cell simulation
Display Omitted Toughness of inter-node, intra-node and intra-core parallelism is discussed.Story change by manycore processors is exemplified by PIC simulation code.Manycore- and SIMD-aware implementation improves the performance 10-fold. This paper ...
Comments