skip to main content
10.1145/2828612.2828620acmconferencesArticle/Chapter ViewAbstractPublication PagesscConference Proceedingsconference-collections
research-article

In Situ Analysis as a Parallel I/O Problem

Published: 15 November 2015 Publication History

Abstract

This paper presents a combination of state-of-the-practice for extraction-based in situ analysis, the treatment of in situ analysis as a form of I/O, and a proposal to the community regarding the handling of in situ and I/O libraries. Extraction in situ analysis extracts subsets of data using certain techniques (e.g., isosurfacing) and saves only the extracted information. It was found that this approach, while reducing output, still poses parallel I/O issues. An initial solution was to rework ParaView Catalyst output to use ADIOS, though that only brought about minor improvements. Further aggregation, even for large-scale extraction I/O, was the primary key to solving that issue. Finally, the paper presents the position that a single library could provide I/O, in situ analysis, code coupling, and resiliency capabilities because of the overlap in steps necessary to bootstrap each.

References

[1]
Liu, Q., Logan, J., Tian, Y., Abbasi, H., Podhorszki, N., Choi, J. Y., Klasky, S., Tchoua, R., Lofstead, J., Oldfield, R., Parashar, M., Samatova, N., Schwan, K., Shoshani, A., Wolf, M., Wu, K. and Yu, W. 2014. Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks. Concurrency Computat: Pract. Exper., 26 (Aug. 2013), 1453--1473. DOI=http://dx.doi.org/10.1002/cpe.3125.
[2]
Nowell, L. 2014. Science at Extreme Scale: Architectural challenges and opportunities. DoE Computer Graphics Forum, Apr. 2014. http://www.doecgf.org/Meetings/.
[3]
Ahrens, J., Jourdain, S., O'Leary, P., Patchett, J., Rogers, D. H., and Petersen, M. 2014. An image-based approach to extreme scale in situ visualization and analysis. Proc. of the Intl. Conf. for High Performance Computing. (Goa, India, Dec. 2014).
[4]
Vahala, G., Yepez, J., Vahala, L., Soe, M., Zhang, B., and Ziegeler, S. 2011. Poincare recurrence and spectral cascades in three-dimensional quantum turbulence. APS Physical Review E: Statistical, Non-linear and Soft Matter Physics. 84 (Oct. 2011), 4, 17p.
[5]
Oganesov, A., Vahala, G., Vahala, L., Yepez, J., and Soe, M. 2014. Unitary qubit highly parallelized lattice algorithms for non-linear systems. Proc. Intl. Conf. for Mesoscopic Methods in Eng. And Science. 11 (New York, New York, Jul. 2014). https://www.icmmes.org/icmmes2014/.
[6]
Ahrens, J., Geveci, B., and Law, C. 2005. Paraview: an end-user tool for large data visualization, Visualization Handbook, Elsevier, ISBN-13: 978-0123875822.
[7]
Schroeder, W., Martin, K., and Lorensen, B., 2006. The Visualization Toolkit. Kitware. ISBN 978-1-930934-19-1.
[8]
Jin, T., Zhang, F., Sun, Q., Bui, H., Podhorszki, N., Klasky, S., Kolla, H., Chen, J., Hager, R., Chang, C.S., and Parahsar, M. 2015. Exploring data staging across deep memory hierarchies for coupled data intensive simulation workflows. Proc. IEEE International Parallel and Distributed Processing Symposium, 29 (Hyderabad, India, May 2015).
[9]
Valcke, S., Balaji, V., Craig, A., DeLuca, C., Dunlap, R., Ford, R. W., Jacob, R., Larson, J., O'Kuinghttons, R., Riley, G. D., and Vertenstein, M. 2012, Coupling technologies for Earth system modeling, Geosci. Model Dev., 5 (2012), 1589--1596.

Cited By

View all
  • (2020)A terminology for in situ visualization and analysis systemsThe International Journal of High Performance Computing Applications10.1177/1094342020935991(109434202093599)Online publication date: 14-Aug-2020
  • (2020)ETH: An Architecture for Exploring the Design Space of In-situ Scientific Visualization2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS47924.2020.00060(515-526)Online publication date: May-2020
  • (2018)SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS.2018.00156(1507-1512)Online publication date: Jul-2018
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ISAV2015: Proceedings of the First Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization
November 2015
51 pages
ISBN:9781450340038
DOI:10.1145/2828612
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 November 2015

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. In situ analysis
  2. in situ visualization
  3. parallel I/O

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

SC15

Acceptance Rates

ISAV2015 Paper Acceptance Rate 8 of 19 submissions, 42%;
Overall Acceptance Rate 23 of 63 submissions, 37%

Upcoming Conference

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)4
  • Downloads (Last 6 weeks)0
Reflects downloads up to 12 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2020)A terminology for in situ visualization and analysis systemsThe International Journal of High Performance Computing Applications10.1177/1094342020935991(109434202093599)Online publication date: 14-Aug-2020
  • (2020)ETH: An Architecture for Exploring the Design Space of In-situ Scientific Visualization2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS47924.2020.00060(515-526)Online publication date: May-2020
  • (2018)SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS.2018.00156(1507-1512)Online publication date: Jul-2018
  • (2016)In Situ Methods, Infrastructures, and Applications on High Performance Computing PlatformsComputer Graphics Forum10.5555/3071534.307159035:3(577-597)Online publication date: 1-Jun-2016
  • (2016)An I/O Mini-App dedicated to in situ visualizationProceedings of the 2nd Workshop on In Situ Infrastructures for Enabling Extreme-scale Analysis and Visualization10.5555/3018859.3018865(29-34)Online publication date: 13-Nov-2016
  • (2016) In Situ Methods, Infrastructures, and Applications on High Performance Computing Platforms Computer Graphics Forum10.1111/cgf.1293035:3(577-597)Online publication date: 4-Jul-2016
  • (2016)An I/O Mini-App Dedicated to In Situ Visualization2016 Second Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization (ISAV)10.1109/ISAV.2016.011(29-34)Online publication date: Nov-2016

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media