research-article

Using hybrid parallelism to improve memory use in the Uintah framework

Authors:

Martin Berzins,

John SchmidtAuthors Info & Claims

TG '11: Proceedings of the 2011 TeraGrid Conference: Extreme Digital Discovery

Article No.: 24, Pages 1 - 8

https://doi.org/10.1145/2016741.2016767

Published: 18 July 2011 Publication History

Abstract

The Uintah Software framework was developed to provide an environment for solving fluid-structure interaction problems on structured adaptive grids on large-scale, long-running, data-intensive problems. Uintah uses a combination of fluid-flow solvers and particle-based methods for solids together with a novel asynchronous task-based approach with fully automated load balancing. Uintah's memory use associated with ghost cells and global meta-data has become a barrier to scalability beyond O(100K) cores. A hybrid memory approach that addresses this issue is described and evaluated. The new approach based on a combination of Pthreads and MPI is shown to greatly reduce memory usage as predicted by a simple theoretical model, with comparable CPU performance.

References

[1]

A. Almgren, J. Bell, D. Kasen, M. Lijewski, A. Nonaka, P. Nugent, C. Rendlemano, R. Thomas, and M. Zingale. MAESTRO, CASTRO and SEDONA - petascale codes for astrophysical applications. submitted for publication also arXiv:1008:2801v1, July 2010.

[2]

Shameen Akhter and Jason Roberts. Multi-core Programming. Intel Press, 2006.

[3]

A. Ali, L. Johnsson, and J. Subhlok. Scheduling fft computations on smp and multicore systems. In ICS 07 Conference, New York, NY, USA, 2007. ACM.

Digital Library

[4]

P. Balaji, A. Chan, and E. Lusk W. Gropp, R. Thakur. Non-data-communication overheads in MPI: analysis on Blue Gene/P. In Proc. of the 15th Euro. PVM/MPI Users' Group Meeting on Recent Advances in PVM and MPI, pages 13--22, Berlin, Heidelberg, 2008. Springer-Verlag.

Digital Library

[5]

K. Bergman, S. Borkar, D. Campbell, W. Carlson, W. Dally, M. Denneau, P. Franzon, W. Harrod, J. Hiller, S. Karp, P. Kogge (Editor, Study Lead) nd S. Keckler, D. Klein, R. Lucas, M. Richards, A. Scarpelli, S. Scott, A. Snavely, T. Sterling, R. S. Williams, and K. Yelick. Exascale computing study: Technology challenges in achieving exascale systems. Technical Report TR-2008-13, Department of Computer Science, Notre Dame University, 2008.

[6]

M. Berzins, J. Luitjens, Q. Meng, T. Harman, C. A. Wight, and J. R. Peterson. Uintah - a scalable framework for hazard analysis. In TG '10: Proc. of 2010 TeraGrid Conference, New York, NY, USA, 2010. ACM.

Digital Library

[7]

B. Fryxell, K. Olson, P. Ricker, F. X. Timmes, M. Zingale, D. Q. Lamb, P. Macneice, R. Rosner, J. W. Rosner, J. W. Truran, and H. Tufo. FLASH an adaptive mesh hydrodynamics code for modeling astrophysical thermonuclear flashes. The Astrophysical Journal Supplement Series, 131:273--334, November 2000.

[8]

C. Burstedde, L. C. Wilcox, and O. Ghattas. p4est: Scalable algorithms for parallel adaptive mesh refinement on forests of octrees. SIAM Journal on Scientific Computing, 33(3):1103--1133, 2011.

Digital Library

[9]

J. D. de St. Germain, J. McCorquodale, S. G. Parker, and C. R. Johnson. Uintah: A massively parallel problem solving environment. In Ninth IEEE International Symposium on High Performance and Distributed Computing, pages 33--41. IEEE, Piscataway, NJ, November 2000.

Digital Library

[10]

G. Bronevetsky, J. Gyllenhaal, and B. de Supinski. CLOMP: Accurately characterizing openmp application overheads. In R. Eigenmann and B. de Supinski, editors, OpenMP in a New Era of Parallelism, volume 5004 of Lecture Notes in Computer Science, pages 13--25. Springer Berlin/Heidelberg, 2008.

Digital Library

[11]

T. Goodale, G. Allen, G. Lanfermann, J. Masso, T. Radke, E. Seidel, and J. Shalf. The Cactus framework and toolkit: Design and applications. In Vector and Parallel Processing VECPAR 2002, Lecture Notes in Computer Science, Berlin, 2003. Springer.

Digital Library

[12]

J. E. Guilkey, T. B. Harman, and B. Banerjee. An eulerian-lagrangian approach for simulating explosions of energetic devices. Computers and Structures, 85:660--674, 2007.

Digital Library

[13]

L. V. Kale, E. Bohm, C. L. Mendes, T. Wilmarth, and G. Zheng. Programming petascale applications with Charm++ and AMPI. Petascale Computing: Algorithms and Applications, 1:421--441, 2007.

[14]

B. A. Kashiwa. A multifield model and method for fluid-structure interaction dynamics. Technical Report LA-UR-01-1136, Los Alamos National Laboratory, Los Alamos, 2001.

[15]

G. Krishnamoorthy, S. Borodai, R. Rawat, J. P. Spinti, and P. J. Smith. Numerical modeling of radiative heat transfer in pool fire simulations. In ASME 2005 International Mechanical Engineering Congress (IMECE2005), November 2005.

[16]

J. Luitjens and M. Berzins. Improving the performance of Uintah: A large-scale adaptive meshing computational framework. In Proc. of the 24th IEEE Int. Parallel and Distributed Processing Symposium (IPDPS10), 2010.

[17]

J. Luitjens, M. Berzins, and T. Henderson. Parallel space-filling curve generation through sorting. Concurr. Comput.: Pract. Exper., 19(10):1387--1402, 2007.

Digital Library

[18]

Q. Meng, J. Luitjens, and M. Berzins. Dynamic task scheduling for the uintah framework. In Proceedings of the 3rd IEEE Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS10), 2010.

[19]

B. O'Shea, G. Bryan, J. Bordner, M. Norman, T. Abel, R. Harkness, and A. Kritsuk. Introducing Enzo, an amr cosmology application. In Adaptive Mesh Refinement - Theory and Applications, volume 41 of Lecture Notes in Computational Science and Engineering, pages 341--350, Berlin, Heidelberg, 2005. Springer-Verlag.

[20]

C. D. Ott, E. Schnetter, G. Allen, E. Seidel, J. Tao, and B. Zink. A case study for petascale applications in astrophysics: simulating gamma-ray bursts. In Proc. of the 15th ACM Mardi Gras conference: From lightweight mash-ups to lambda grids., MG '08, pages 18:1--18:9, New York, NY, USA, 2008. ACM.

Digital Library

[21]

S. G. Parker. A component-based architecture for parallel multi-physics PDE simulation. Future Generation Comput. Sys., 22:204--216, 2006.

Digital Library

[22]

R. Rawat, J. Spinti, W. Yee, and P. J. Smith. Parallelization of a large scale hydrocarbon pool fire in the Uintah PSE. In ASME 2002 International Mechanical Engineering Congress and Exposition (IMECE2002), pages 49--55, November 2002.

[23]

H-Y. Schive, U.-H. Zhang, and T. Chiueh. Directionally unsplit hydrodynamic schemes with hybrid MPI/OpenMP/GPU parallelization in amr. 2011.

[24]

D. Sulsky, S. Zhou, and H. L. Schreyer. Application of a particle-in-cell method to solid mechanics. Computer Physics Communications, 87:236--252, 1995.

[25]

J. Tao, G. Allen, I. Hinder, E. Schnetter, and Y. Zlochower. XIREL:standard benchmarks for numericla relativity codes using Cactus and Carpet. Technical Report CCT-TR-2008-5, Center for Computationa and Technology, Louisiana State University, Baton Rouge, Louisiana, 2008.

Cited By

Zambre RSahasrabudhe DZhou HBerzins MChandramowlishwaran ABalaji P(2021)Logically Parallel Communication for Fast MPI+Threads ApplicationsIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2021.307515732:12(3038-3052)Online publication date: 1-Dec-2021
https://doi.org/10.1109/TPDS.2021.3075157
Peterson BHumphrey ASunderland DSutherland JSaad TDasari HBerzins M(2018)Automatic Halo Management for the Uintah GPU-Heterogeneous Asynchronous Many-Task RuntimeInternational Journal of Parallel Programming10.1007/s10766-018-0619-1Online publication date: 7-Dec-2018
https://doi.org/10.1007/s10766-018-0619-1
Holmen JHumphrey ASunderland DBerzins MHart D(2017)Improving Uintah's Scalability Through the Use of Portable Kokkos-Based Data Parallel TasksPractice and Experience in Advanced Research Computing 2017: Sustainability, Success and Impact10.1145/3093338.3093388(1-8)Online publication date: 9-Jul-2017
https://dl.acm.org/doi/10.1145/3093338.3093388
Show More Cited By

Index Terms

Using hybrid parallelism to improve memory use in the Uintah framework

Recommendations

Preliminary experiences with the uintah framework on Intel Xeon Phi and stampede
XSEDE '13: Proceedings of the Conference on Extreme Science and Engineering Discovery Environment: Gateway to Discovery

In this work, we describe our preliminary experiences on the Stampede system in the context of the Uintah Computational Framework. Uintah was developed to provide an environment for solving a broad class of fluid-structure interaction problems on ...
Radiation modeling using the Uintah heterogeneous CPU/GPU runtime system
XSEDE '12: Proceedings of the 1st Conference of the Extreme Science and Engineering Discovery Environment: Bridging from the eXtreme to the campus and beyond

The Uintah Computational Framework was developed to provide an environment for solving fluid-structure interaction problems on structured adaptive grids on large-scale, long-running, data-intensive problems. Uintah uses a combination of fluid-flow ...
Investigating applications portability with the Uintah DAG-based runtime system on PetaScale supercomputers
SC '13: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis

Present trends in high performance computing present formidable challenges for applications code using multicore nodes possibly with accelerators and/or co-processors and reduced memory while still attaining scalability. Software frameworks that execute ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

TG '11: Proceedings of the 2011 TeraGrid Conference: Extreme Digital Discovery

July 2011

256 pages

ISBN:9781450308885

DOI:10.1145/2016741

General Chair:
John Towns
NCSA-Illinois
,
Program Chairs:
Shawn Brown
PSC
,
Daniel S. Katz
UC/ANL

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

University of Illinois: University of Illinois

In-Cooperation

SIGAPP: ACM Special Interest Group on Applied Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 July 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

TG'11

Sponsor:

University of Illinois

TG'11: TeraGrid 2011

July 18 - 21, 2011

Utah, Salt Lake City

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
96
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)1

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zambre RSahasrabudhe DZhou HBerzins MChandramowlishwaran ABalaji P(2021)Logically Parallel Communication for Fast MPI+Threads ApplicationsIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2021.307515732:12(3038-3052)Online publication date: 1-Dec-2021
https://doi.org/10.1109/TPDS.2021.3075157
Peterson BHumphrey ASunderland DSutherland JSaad TDasari HBerzins M(2018)Automatic Halo Management for the Uintah GPU-Heterogeneous Asynchronous Many-Task RuntimeInternational Journal of Parallel Programming10.1007/s10766-018-0619-1Online publication date: 7-Dec-2018
https://doi.org/10.1007/s10766-018-0619-1
Holmen JHumphrey ASunderland DBerzins MHart D(2017)Improving Uintah's Scalability Through the Use of Portable Kokkos-Based Data Parallel TasksPractice and Experience in Advanced Research Computing 2017: Sustainability, Success and Impact10.1145/3093338.3093388(1-8)Online publication date: 9-Jul-2017
https://dl.acm.org/doi/10.1145/3093338.3093388
Van der Wijngaart RGeorganas EMattson TWissink A(2017)A New Parallel Research Kernel to Expand Research on Dynamic Load-Balancing CapabilitiesHigh Performance Computing10.1007/978-3-319-58667-0_14(256-274)Online publication date: 12-May-2017
https://doi.org/10.1007/978-3-319-58667-0_14
Humphrey ASunderland DHarman TBerzins M(2016)Radiative Heat Transfer Calculation on 16384 GPUs Using a Reverse Monte Carlo Ray Tracing Approach with Adaptive Mesh Refinement2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)10.1109/IPDPSW.2016.93(1222-1231)Online publication date: May-2016
https://doi.org/10.1109/IPDPSW.2016.93
Georganas EWijngaart RMattson T(2016)Design and Implementation of a Parallel Research Kernel for Assessing Dynamic Load-Balancing Capabilities2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS.2016.65(73-82)Online publication date: May-2016
https://doi.org/10.1109/IPDPS.2016.65
Beckvermit JHarman TWight CBerzins M(2016)Packing Configurations of PBX‐9501 Cylinders to Reduce the Probability of a Deflagration to Detonation Transition (DDT)Propellants, Explosives, Pyrotechnics10.1002/prep.20150033141:6(1070-1078)Online publication date: 4-Aug-2016
https://doi.org/10.1002/prep.201500331
Peterson BDasari HHumphrey ASutherland JSaad TBerzins M(2015)Reducing overhead in the Uintah framework to support short-lived tasks on GPU-heterogeneous architecturesProceedings of the 5th International Workshop on Domain-Specific Languages and High-Level Frameworks for High Performance Computing10.1145/2830018.2830023(1-8)Online publication date: 15-Nov-2015
https://dl.acm.org/doi/10.1145/2830018.2830023
Humphrey AHarman TBerzins MSmith P(2015)A Scalable Algorithm for Radiative Heat Transfer Using Reverse Monte Carlo Ray TracingHigh Performance Computing10.1007/978-3-319-20119-1_16(212-230)Online publication date: 20-Jun-2015
https://doi.org/10.1007/978-3-319-20119-1_16
Meng QBerzins M(2014)Scalable large-scale fluid-structure interaction solvers in the Uintah framework via hybrid task-based parallelism algorithmsConcurrency and Computation: Practice & Experience10.1002/cpe.309926:7(1388-1407)Online publication date: 1-May-2014
https://dl.acm.org/doi/10.1002/cpe.3099
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten