In Situ Methods, Infrastructures, and Applications on High Performance Computing Platforms

Bauer, A. C.; Abbasi, H.; Ahrens, J.; Childs, H.; Geveci, B.; Klasky, S.; Moreland, K.; O'Leary, P.; Vishwanath, V.; Whitlock, B.; Bethel, E. W.

doi:10.1111/cgf.12930

Title: In Situ Methods, Infrastructures, and Applications on High Performance Computing Platforms

Journal Article · Wed Jun 01 00:00:00 EDT 2016 · Computer Graphics Forum

DOI:https://doi.org/10.1111/cgf.12930· OSTI ID:1595010

Bauer, A. C. ^[1]; Abbasi, H. ^[2]; Ahrens, J. ^[3]; Childs, H. ^[4]; Geveci, B. ^[1]; Klasky, S. ^[2]; Moreland, K. ^[5]; O'Leary, P. ^[1]; Vishwanath, V. ^[6]; Whitlock, B. ^[7]; Bethel, E. W. ^[8]

Kitware, Inc., Clifton Park, NY (United States)
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Univ. of Oregon, Eugene, OR (United States)
Sandia National Lab. (SNL-CA), Livermore, CA (United States)
Argonne National Lab. (ANL), Argonne, IL (United States)
Intelligent Light, USA, Rutherford, NJ (United States)
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)

The considerable interest in the high performance computing (HPC) community regarding analyzing and visualization data without first writing to disk, i.e., in situ processing, is due to several factors. First is an I/O cost savings, where data is analyzed /visualized while being generated, without first storing to a filesystem. Second is the potential for increased accuracy, where fine temporal sampling of transient analysis might expose some complex behavior missed in coarse temporal sampling. Third is the ability to use all available resources, CPU’s and accelerators, in the computation of analysis products. This STAR paper brings together researchers, developers and practitioners using in situ methods in extreme-scale HPC with the goal to present existing methods, infrastructures, and a range of computational science and engineering applications using in situ analysis and visualization.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Kitware, Inc., Clifton Park, NY (United States); Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF); Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE Office of Nuclear Energy (NE); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: SC0012037; AC02-05CH11231

OSTI ID:: 1595010

Alternate ID(s):: OSTI ID: 1439995

Journal Information:: Computer Graphics Forum, Vol. 35, Issue 3; ISSN 0167-7055

Publisher:: WileyCopyright Statement

Country of Publication:: United States

Language:: English

Citation Metrics:

Cited by: 83 works

Citation information provided by
Web of Science

References (55)

FLASH: An Adaptive Mesh Hydrodynamics Code for Modeling Astrophysical Thermonuclear Flashes Fryxell, B.; Olson, K.; Ricker, P. The Astrophysical Journal Supplement Series, Vol. 131, Issue 1 https://doi.org/10.1086/317361	journal	November 2000
High end scientific codes with computational I/O pipelines: improving their end-to-end performance Zheng, Fang; Cao, Jianting; Dayal, Jai Proceedings of the 2nd international workshop on Petascal data analytics: challenges and opportunities - PDAC '11 https://doi.org/10.1145/2110205.2110210	conference	January 2011
Increasing Scientific Data Insights about Exascale Class Simulations under Power and Storage Constraints Ahrens, James IEEE Computer Graphics and Applications, Vol. 35, Issue 2 https://doi.org/10.1109/MCG.2015.35	journal	March 2015
In-situ multi-resolution and temporal data compression for visual exploration of large-scale scientific simulations Lehmann, Henry; Jung, Bernhard 2014 IEEE 4th Symposium on Large Data Analysis and Visualization (LDAV) https://doi.org/10.1109/LDAV.2014.7013204	conference	November 2014
Improved post hoc flow analysis via Lagrangian representations Agranovsky, Alexy; Camp, David; Garth, Christoph 2014 IEEE 4th Symposium on Large Data Analysis and Visualization (LDAV) https://doi.org/10.1109/LDAV.2014.7013206	conference	November 2014
Space-time volumetric depth images for in-situ visualization Fernandes, Oliver; Frey, Steffen; Sadlo, Filip 2014 IEEE 4th Symposium on Large Data Analysis and Visualization (LDAV) https://doi.org/10.1109/LDAV.2014.7013205	conference	November 2014
Distributed merge trees Morozov, Dmitriy; Weber, Gunther ACM SIGPLAN Notices, Vol. 48, Issue 8 https://doi.org/10.1145/2517327.2442526	journal	August 2013
Topology-aware data movement and staging for I/O acceleration on Blue Gene/P supercomputing systems Vishwanath, Venkatram; Hereld, Mark; Morozov, Vitali Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '11 https://doi.org/10.1145/2063384.2063409	conference	January 2011
In situ depth maps based feature extraction and tracking Ye, Yucong Chris; Wang, Yang; Miller, Robert 2015 IEEE 5th Symposium on Large Data Analysis and Visualization (LDAV) https://doi.org/10.1109/LDAV.2015.7348065	conference	October 2015
Runtime volume visualization for parallel CFD Ma, Kwan-Liu Parallel Computational Fluid Dynamics 1995 https://doi.org/10.1016/B978-044482322-9/50092-X	book	January 1996
Consuming Network Bandwidth with Visapult Bethel, Wes; Shalf, John Visualization Handbook https://doi.org/10.1016/B978-012387582-2/50031-9	book	January 2005
Adaptable, metadata rich IO methods for portable high performance IO Lofstead, Jay; Zheng, Fang; Klasky, Scott Distributed Processing (IPDPS), 2009 IEEE International Symposium on Parallel & Distributed Processing https://doi.org/10.1109/IPDPS.2009.5161052	conference	May 2009
Extending scalability of collective IO through nessie and staging Lofstead, Jay; Oldfield, Ron; Kordenbrock, Todd Proceedings of the sixth workshop on Parallel Data Storage - PDSW '11 https://doi.org/10.1145/2159352.2159355	conference	January 2011
ActiveSpaces: Exploring dynamic code deployment for extreme scale data processing: ActiveSpaces: Exploring dynamic code deployment for extreme scale data processing Docan, Ciprian; Zhang, Fan; Jin, Tong Concurrency and Computation: Practice and Experience, Vol. 27, Issue 14 https://doi.org/10.1002/cpe.3407	journal	October 2014
Visualization by Proxy: A Novel Framework for Deferred Interaction with Volume Data Tikhonova, A.; Correa, C. D. IEEE Transactions on Visualization and Computer Graphics, Vol. 16, Issue 6 https://doi.org/10.1109/TVCG.2010.215	journal	November 2010
Visualization at Supercomputing Centers: The Tale of Little Big Iron and the Three Skinny Guys Bethel, E. W.; van Rosendale, J.; Southard, D. IEEE Computer Graphics and Applications, Vol. 31, Issue 1 https://doi.org/10.1109/MCG.2011.13	journal	January 2011
Site remediation in a virtual environment Bethel, E. W.; Jacobsen, Janet; Holland, Preston IS&T/SPIE 1994 International Symposium on Electronic Imaging: Science and Technology, SPIE Proceedings https://doi.org/10.1117/12.172079	conference	April 1994
Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks: HELLO ADIOS Liu, Qing; Logan, Jeremy; Tian, Yuan Concurrency and Computation: Practice and Experience, Vol. 26, Issue 7 https://doi.org/10.1002/cpe.3125	journal	August 2013
EPIC - An Extract Plug-In Components Toolkit for In-Situ Data Extracts Architecture Duque, Earl P.; Hiepler, `Daniel E.; Haimes, Robert 22nd AIAA Computational Fluid Dynamics Conference https://doi.org/10.2514/6.2015-3410	conference	June 2015
Petroleum Reservoir Simulation in a Virtual Environment Jacobsen, J. S.; Bethel, E. W.; Datta-Gupta, Akhil SPE Reservoir Simulation Symposium https://doi.org/10.2118/29118-MS	conference	April 2013
Research Challenges for Visualization Software Childs, Hank; Geveci, Berk; Schroeder, Will Computer, Vol. 46, Issue 5 https://doi.org/10.1109/MC.2013.179	journal	May 2013
I/O Containers: Managing the Data Analytics and Visualization Pipelines of High End Codes Dayal, Jai; Cao, Jianting; Eisenhauer, Greg 2013 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum (IPDPSW) https://doi.org/10.1109/IPDPSW.2013.198	conference	May 2013
The Tensions of In Situ Visualization Moreland, Kenneth IEEE Computer Graphics and Applications, Vol. 36, Issue 2 https://doi.org/10.1109/MCG.2016.35	journal	March 2016
Computational steering. Software systems and strategies Parker, S. G.; Johnson, C. R.; Beazley, D. IEEE Computational Science and Engineering, Vol. 4, Issue 4 https://doi.org/10.1109/99.641609	journal	January 1997
A multi-resolution approach to global ocean modeling Ringler, Todd; Petersen, Mark; Higdon, Robert L. Ocean Modelling, Vol. 69 https://doi.org/10.1016/j.ocemod.2013.04.010	journal	September 2013
Interactive, Internet Delivery of Visualization via Structured Prerendered Multiresolution Imagery IEEE Transactions on Visualization and Computer Graphics, Vol. 14, Issue 2 https://doi.org/10.1109/TVCG.2007.70428	journal	March 2008
Cinema image-based in situ analysis and visualization of MPAS-ocean simulations O’Leary, Patrick; Ahrens, James; Jourdain, Sébastien Parallel Computing, Vol. 55 https://doi.org/10.1016/j.parco.2015.10.005	journal	July 2016
In Situ Visualization at Extreme Scale: Challenges and Opportunities IEEE Computer Graphics and Applications, Vol. 29, Issue 6 https://doi.org/10.1109/MCG.2009.120	journal	November 2009
In Situ Visualization for Large-Scale Combustion Simulations IEEE Computer Graphics and Applications, Vol. 30, Issue 3 https://doi.org/10.1109/MCG.2010.55	journal	May 2010
Parallel netCDF: A High-Performance Scientific I/O Interface Li, Jianwei; Zingale, Michael; Liao, Wei-keng Proceedings of the 2003 ACM/IEEE conference on Supercomputing - SC '03 https://doi.org/10.1145/1048935.1050189	conference	January 2003
Computer-made perspective movies as a scientific and communication tool Zajac, E. E. Communications of the ACM, Vol. 7, Issue 3 https://doi.org/10.1145/363958.363993	journal	March 1964
HPCCloud: A Cloud/Web-Based Simulation Environment OLeary, Patrick; Christon, Mark; Jourdain, Sebastien 2015 IEEE 7th International Conference on Cloud Computing Technology and Science (CloudCom) https://doi.org/10.1109/CloudCom.2015.33	conference	November 2015
In Situ Pathtube Visualization with Explorable Images Ye, Yucong; Miller, Robert; Ma, Kwan-Liu The Eurographics Association https://doi.org/10.2312/EGPGV/EGPGV13/009-016	null	January 2013
Efficient parallel volume rendering of large-scale adaptive mesh refinement data Leaf, Nick; Vishwanath, Venkatram; Insley, Joseph 2013 IEEE Symposium on Large-Scale Data Analysis and Visualization (LDAV) https://doi.org/10.1109/LDAV.2013.6675156	conference	October 2013
Evaluation of methods to integrate analysis into a large-scale shock shock physics code Oldfield, Ron A.; Moreland, Kenneth; Fabian, Nathan Proceedings of the 28th ACM international conference on Supercomputing - ICS '14 https://doi.org/10.1145/2597652.2597668	conference	January 2014
A survey of computational steering environments Mulder, Jurriaan D.; van Wijk, Jarke J.; van Liere, Robert Future Generation Computer Systems, Vol. 15, Issue 1 https://doi.org/10.1016/S0167-739X(98)00047-8	journal	February 1999
yt: A MULTI-CODE ANALYSIS TOOLKIT FOR ASTROPHYSICAL SIMULATION DATA Turk, Matthew J.; Smith, Britton D.; Oishi, Jeffrey S. The Astrophysical Journal Supplement Series, Vol. 192, Issue 1 https://doi.org/10.1088/0067-0049/192/1/9	journal	December 2010
Scalable Implicit Flow Solver for Realistic Wing Simulations with Flow Control Rasquin, Michel; Smith, Cameron; Chitale, Kedar Computing in Science & Engineering, Vol. 16, Issue 6 https://doi.org/10.1109/MCSE.2014.75	journal	November 2014
WarpIV: In Situ Visualization and Analysis of Ion Accelerator Simulations Rubel, Oliver; Loring, Burlen; Vay, Jean-Luc IEEE Computer Graphics and Applications, Vol. 36, Issue 3 https://doi.org/10.1109/MCG.2016.62	journal	May 2016
Interactive Exploration and Analysis of Large-Scale Simulations Using Topology-Based Data Segmentation Bremer, P-T; Weber, G.; Tierny, J. IEEE Transactions on Visualization and Computer Graphics, Vol. 17, Issue 9 https://doi.org/10.1109/TVCG.2010.253	journal	September 2011
Optimal scheduling of in-situ analysis for large-scale scientific simulations Malakar, Preeti; Vishwanath, Venkatram; Munson, Todd Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '15 https://doi.org/10.1145/2807591.2807656	conference	January 2015
pV3 - A distributed system for large-scale unsteady CFD visualization Haimes, Robert 32nd Aerospace Sciences Meeting and Exhibit https://doi.org/10.2514/6.1994-321	conference	February 2013
Experiences Applying Data Staging Technology in Unconventional Ways Lofstead, J.; Oldfield, R.; Kordenbrock, T. 2013 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), 2013 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing https://doi.org/10.1109/CCGrid.2013.27	conference	May 2013
DataSpaces: an interaction and coordination framework for coupled simulation workflows Docan, Ciprian; Parashar, Manish; Klasky, Scott Cluster Computing, Vol. 15, Issue 2 https://doi.org/10.1007/s10586-011-0162-y	journal	February 2011
From Mesh Generation to Scientific Visualization: An End-to-End Approach to Parallel Supercomputing Tu, Tiankai; Yu, Hongfeng; Ramirez-Guzman, Leonardo ACM/IEEE SC 2006 Conference (SC'06) https://doi.org/10.1109/SC.2006.32	conference	November 2006
PreDatA – preparatory data analytics on peta-scale machines Zheng, Fang; Abbasi, Hasan; Docan, Ciprian 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPS) https://doi.org/10.1109/IPDPS.2010.5470454	conference	April 2010
An Approach to Lowering the In Situ Visualization Barrier Fogal, Thomas; Krüger, Jens Proceedings of the First Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization - ISAV2015 https://doi.org/10.1145/2828612.2828618	conference	January 2015
PARAMESH: A parallel adaptive mesh refinement community toolkit MacNeice, Peter; Olson, Kevin M.; Mobarry, Clark Computer Physics Communications, Vol. 126, Issue 3 https://doi.org/10.1016/S0010-4655(99)00501-9	journal	April 2000
Strawman: A Batch In Situ Visualization and Analysis Infrastructure for Multi-Physics Simulation Codes Larsen, Matthew; Brugger, Eric; Childs, Hank Proceedings of the First Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization - ISAV2015 https://doi.org/10.1145/2828612.2828625	conference	January 2015
Oh, $#*@! Exascale! The Effect of Emerging Architectures on Scientific Discovery Moreland, Kenneth 2012 SC Companion: High Performance Computing, Networking, Storage and Analysis (SCC), 2012 SC Companion: High Performance Computing, Networking Storage and Analysis https://doi.org/10.1109/SC.Companion.2012.38	conference	November 2012
Concurrent Visualization in a Production Supercomputing Environment Ellsworth, D.; Green, B.; Henze, C. IEEE Transactions on Visualization and Computer Graphics, Vol. 12, Issue 5 https://doi.org/10.1109/TVCG.2006.128	journal	September 2006
An Image-Based Approach to Extreme Scale in Situ Visualization and Analysis Ahrens, James; Jourdain, Sebastien; OLeary, Patrick SC14: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2014.40	conference	November 2014
In Situ Infrastructure Enhancements for Data Extract Generation Whitlock, Brad J.; Legensky, Steve M.; Forsythe, Jim 54th AIAA Aerospace Sciences Meeting https://doi.org/10.2514/6.2016-1928	conference	January 2016
Parallel Computational Steering for HPC Applications Using HDF5 Files in Distributed Shared Memory Biddiscombe, J.; Soumagne, J.; Oger, G. IEEE Transactions on Visualization and Computer Graphics, Vol. 18, Issue 6 https://doi.org/10.1109/TVCG.2012.63	journal	June 2012
A Multi-Code Analysis Toolkit for Astrophysical Simulation Data Turk, Matthew J.; Smith, Britton D.; Oishi, Jeffrey S. arXiv https://doi.org/10.48550/arxiv.1011.3514	text	January 2010

Cited By (8)

Design of a Flexible In Situ Framework with a Temporal Buffer for Data Processing and Visualization of Time-Varying Datasets Ono, Kenji; Nonaka, Jorji; Yoshikawa, Hiroyuki High Performance Computing. Lecture Notes in Computer Science https://doi.org/10.1007/978-3-030-02465-9_17	book	January 2018
A novel in situ compression method for CFD data based on generative adversarial network Liu, Yang; Wang, Yueqing; Deng, Liang Journal of Visualization, Vol. 22, Issue 1 https://doi.org/10.1007/s12650-018-0519-x	journal	October 2018
Scientific post hoc and in situ visualisation of high-order polynomial solutions from massively parallel simulations Rasquin, M.; Bauer, A.; Hillewaert, K. International Journal of Computational Fluid Dynamics, Vol. 33, Issue 4 https://doi.org/10.1080/10618562.2019.1618453	journal	April 2019
Data Reduction Techniques for Simulation, Visualization and Data Analysis: Survey on Scientific Data Reduction Techniques Li, S.; Marsaglia, N.; Garth, C. Computer Graphics Forum, Vol. 37, Issue 6 https://doi.org/10.1111/cgf.13336	journal	March 2018
A decade of particle-based scientific visualization Reina, Guido; Gralka, Patrick; Ertl, Thomas The European Physical Journal Special Topics, Vol. 227, Issue 14 https://doi.org/10.1140/epjst/e2019-800172-4	journal	March 2019
Scalable In situ Analysis of Molecular Dynamics Simulations Malakar, Preeti; Knight, Christopher; Munson, Todd Proceedings of the In Situ Infrastructures on Enabling Extreme-Scale Analysis and Visualization - ISAV'17 https://doi.org/10.1145/3144769.3144772	conference	January 2017
In situ data-driven adaptive sampling for large-scale simulation data summarization Biswas, Ayan; Dutta, Soumya; Pulido, Jesus Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization - ISAV '18 https://doi.org/10.1145/3281464.3281467	conference	January 2018
A Violently Tornadic Supercell Thunderstorm Simulation Spanning a Quarter-Trillion Grid Volumes: Computational Challenges, I/O Framework, and Visualizations of Tornadogenesis Orf, Leigh Atmosphere, Vol. 10, Issue 10 https://doi.org/10.3390/atmos10100578	journal	September 2019

Similar Records

A report documenting the completion of the Los Alamos National Laboratory portion of the ASC level II milestone ""Visualization on the supercomputing platform

Journal Article · Mon Jan 24 00:00:00 EST 2011 · OSTI ID:1595010

Ahrens, James P; Patchett, John M; Lo, Li - Ta; +3 more

Final Report Scalable Analysis Methods and In Situ Infrastructure for Extreme Scale Knowledge Discovery

Technical Report · Wed Sep 13 00:00:00 EDT 2017 · OSTI ID:1595010

O'Leary, Patrick

Scalable Analysis Methods and In Situ Infrastructure for Extreme Scale Knowledge Discovery

Technical Report · Sun Jul 24 00:00:00 EDT 2016 · OSTI ID:1595010

Bethel, Wes

Related Subjects

97 MATHEMATICS AND COMPUTING

Title: In Situ Methods, Infrastructures, and Applications on High Performance Computing Platforms

Citation Formats

References (55)

Cited By (8)

Similar Records

Related Subjects