Skip to main content
SpringerLink
Log in
Book cover

International Conference on High Performance Computing

ISC High Performance 2022: High Performance Computing. ISC High Performance 2022 International Workshops pp 324–343Cite as

Cinema Transfer: A Containerized Visualization Workflow

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13387))

Abstract

We present a containerized workflow demonstrating in situ analysis of simulation data rendered by a ParaView/Catalyst adapter for the generic SENSEI in situ interface, then streamed to a remote site for visualization. We use Cinema, a database approach for navigating the metadata produced in situ. We developed a web socket tool, cinema_transfer, for transferring the generated cinema databases to a remote machine while the simulation is running. We evaluate the performance of this containerized workflow and identify bottlenecks for large scale runs, in addition to testing identical containers at different sites with differing hardware and Message Passing Interface (MPI) implementations.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   74.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Ahrens, J., Jourdain, S., O’Leary, P., Patchett, J., Rogers, D.H., Petersen, M.: An image-based approach to extreme scale in situ visualization and analysis. In: SC 2014: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 424–434. IEEE (2014)

    Google Scholar 

  2. Ahrens, J., et al.: In situ mpas-ocean image-based visualization. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, Visualization & Data Analytics Showcase (2014)

    Google Scholar 

  3. Argonne Leadership Computing Facility. https://alcf.anl.gov, Accessed Mar 2022

  4. Ayachit, U., et al.: Performance analysis, design considerations, and applications of extreme-scale in situ infrastructures. In: SC 2016: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 921–932. IEEE (2016)

    Google Scholar 

  5. Ayachit, U., et al.: Paraview catalyst: Enabling in situ data analysis and visualization. In: Proceedings of the First Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization, pp. 25–29 (2015)

    Google Scholar 

  6. Cobalt HPC management suite. https://github.com/ido/cobalt, Accessed Mar 2022

  7. Docker open platform for developing, shipping, and running applications. https://www.docker.com, Accessed Mar 2022

  8. Docker tmpfs mounts. https://docs.docker.com/storage/tmpfs/, Accessed Mar 2022

  9. Kubernetes: Production-Grade Container Orchestration. https://kubernetes.io, Accessed Mar 2022

  10. LAAMPS Lennard Jones Benchmark. https://www.lammps.org/bench.html, Accessed Mar 2022

  11. McMillan, S.: Making containers easier with hpc container maker (2018). https://github.com/HPCSYSPROS/Workshop18/blob/master/Making_Containers_Easier_with_HPC_Container_Maker/ws_hpcsysp103.pdf

  12. MPICH: High Performance Portable MPI. https://www.mpich.org, Accessed Mar 2022

  13. Nealey, I.: https://github.com/inealey/cinema_transfer, Accessed Mar 2022

  14. Nealey, I.: https://github.com/inealey/sensei/tree/lammps, Accessed Mar 2022

  15. Nealey, I., Ferrier, N., Insley, J., Mateevitsi, V.A., Papka, M.E., Rizzi, S.: Artifacts for woiv (2022). https://doi.org/10.5281/zenodo.6336286

  16. OpenSSH: https://www.openssh.com, Accessed Mar 2022

  17. Pacific Research Platform. https://pacificresearchplatform.org, Accessed Sept 2021

  18. Singularity open source container platform. https://github.com/sylabs/singularity, Accessed Mar 2022

  19. Singularity Persistent Overlays. https://sylabs.io/guides/3.5/user-guide/persistent_overlays.html, Accessed Mar 2022

  20. Snyder, P.: tmpfs: a virtual memory file system. In: Proceedings of the Autumn 1990 EUUG Conference, pp. 241–248 (1990)

    Google Scholar 

  21. Will, M., Wofford, Q., Patchett, J., Rogers, D., Lukasczyk, J., Garth, C.: Developing and evaluating in situ visualization algorithms using containers, pp. 6–11. Association for Computing Machinery, New York (2021). https://doi.org/10.1145/3490138.3490141

Download references

Acknowledgments

This work was supported by and used resources of the Argonne Leadership Computing Facility, which is a U.S. Department of Energy Office of Science User Facility supported under Contract DE-AC02-06CH11357. This work was supported in part by the Director, Office of Science, Office of Advanced Scientific Computing Research, of the U.S. Department of Energy under Contract DE-AC02-06CH11357, through the grant “Scalable Analysis Methods and In Situ Infrastructure for Extreme Scale Knowledge Discovery”. This project also used resources at the California Institute for Telecommunications and Information Technology (Calit2) at UCSD. These facilities are supported by the following National Science Foundation awards: CC*DNI DIBBs: The Pacific Research Platform- NSF Award Number:1541349; CC* NPEO: Toward the National Research Platform- NSF Award Number:1826967; CI-New: Cognitive Hardware and Software Ecosystem Community Infrastructure (CHASE-CI)- NSF Award Number:1730158; MRI: Development of Advanced Visualization Instrumentation for the Collaborative Exploration of Big Data- NSF Award Number:1338192; and Development of the Sensor Environment Imaging (SENSEI) Instrument- NSF Award Number:1456638.

Author information

Authors and Affiliations

  1. University of California-San Diego, La Jolla, CA, USA

    Isaac Nealey

  2. Argonne National Laboratory, Lemont, IL, USA

    Nicola Ferrier, Joseph Insley, Victor A. Mateevitsi, Michael E. Papka & Silvio Rizzi

Authors
  1. Isaac Nealey
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nicola Ferrier
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Joseph Insley
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Victor A. Mateevitsi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Michael E. Papka
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Silvio Rizzi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Isaac Nealey .

Editor information

Editors and Affiliations

  1. University of Tennessee, Knoxville, TN, USA

    Hartwig Anzt

  2. University of New Mexico, Albuquerque, NM, USA

    Amanda Bienz

  3. University of Tennessee, Knoxville, TN, USA

    Piotr Luszczek

  4. Université Paris-Saclay, Orsay, France

    Marc Baboulin

Appendix: Reproducibility

Appendix: Reproducibility

We have aimed at making this work completely reproducible. For this purpose, a Zenodo repository of artifacts is available [15].

The repository contains our producer and consumer container recipes as Dockerfiles. These Docker images can be easily converted to Singularity images. In addition, we also include LAMMPS, SENSEI, and Catalyst configuration files to reproduce our experiments. The results of our experiments are also presented in log files and spreadsheets. Finally, we present videos that illustrate our runs with the producer and consumer running concurrently.

In addition to our Zenodo archive, we will present our container recipes below in an effort to make this workflow as reproducible and as transparent as possible. It may be informative to compare these with the block diagram above [Fig. 1] to ascertain how the containers are built to support this portable workflow.

GCC Base Container:

figure b
figure c

Producer Build Container:

figure d
figure e
figure f

Consumer Build Container:

figure g
figure h
figure i

Producer Runtime Container:

figure j

Consumer Runtime Container:

figure k

It is our hope that the scientific visualization community will benefit from this work and build on this material for future research.

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Nealey, I., Ferrier, N., Insley, J., Mateevitsi, V.A., Papka, M.E., Rizzi, S. (2022). Cinema Transfer: A Containerized Visualization Workflow. In: Anzt, H., Bienz, A., Luszczek, P., Baboulin, M. (eds) High Performance Computing. ISC High Performance 2022 International Workshops. ISC High Performance 2022. Lecture Notes in Computer Science, vol 13387. Springer, Cham. https://doi.org/10.1007/978-3-031-23220-6_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-23220-6_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-23219-0

  • Online ISBN: 978-3-031-23220-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation