skip to main content
10.1145/3437959.3459257acmconferencesArticle/Chapter ViewAbstractPublication PagespadsConference Proceedingsconference-collections
research-article
Open access

Speculative Parallel Execution for Local Timestepping

Published: 01 June 2021 Publication History

Abstract

Currently, synchronous timestepping for fluid and plasma simulations requires selection of a global time step that conservatively satisfies stability conditions everywhere. However, this approach causes substantial unnecessary work in the presence of large variations of element sizes or local wavespeeds. Local timestepping can significantly reduce work by allowing subdomains to take steps according to local rather than global stability constraints. However, parallelizing this algorithm presents considerable difficulty. Since the stability condition depends on the state of the submesh and its neighbors, dependencies become irregular and may dynamically change as neighbors take smaller or larger timesteps. Furthermore, coarsening and refining timesteps introduces dynamic load imbalance. In order to correctly resolve these dependencies in a distributed setting, we parallelize the local timestepping algorithm using an optimistic (Timewarp-based) parallel discrete event simulation. We introduce waiting heuristics to eliminate misspeculation when dependencies can be identified early, and present a semi-static load balancing strategy to improve scalability. We present detailed performance characterizations of event overheads, misspeculation, and scalability of our approach. Our numerical experiments demonstrate up to a 2.8x speedup versus a baseline unoptimized approach; a 4x improvement in per-node throughput compared to an MPI parallelization of synchronous timestepping; and scalability up to 3,072 cores on NERSC Cori's Haswell partition.

Supplementary Material

MP4 File (SIGSIM-PADS21-pads589.mp4)
Presentation video of "Speculative Parallel Execution for Local Timestepping" presented by Maximilian Bremer

References

[1]
Emmanuel Audusse, Francc ois Bouchut, Marie-Odile Bristeau, Rupert Klein, and Benoît Perthame. 2004. A fast and stable well-balanced scheme with hydrostatic reconstruction for shallow water flows. SIAM Journal on Scientific Computing, Vol. 25, 6 (2004), 2050--2065.
[2]
Onno Bokhove. 2005. Flooding and Drying in Discontinuous Galerkin Finite-Element Discretizations of Shallow-Water Equations. Part 1: One Dimension. Journal of Scientific Computing, Vol. 22, 1 (2005), 47--82.
[3]
Dan Bonachea and Paul H. Hargrove. 2018. GASNet-EX: A High-Performance, Portable Communication Library for Exascale. In Proceedings of Languages and Compilers for Parallel Computing (LCPC'18) (Lecture Notes in Computer Science, Vol. 11882). Springer International Publishing, 138--158. https://doi.org/10.25344/S4QP4W Lawrence Berkeley National Laboratory Technical Report (LBNL-2001174).
[4]
Maximilian Bremer. 2020. Task-Based Parallelism for Hurricane Sotrm Surge Modeling. Ph.D. Dissertation. The University of Texas at Austin.
[5]
Maximilian H. Bremer, John D. Bachan, and Cy P. Chan. 2018. Semi-Static and Dynamic Load Balancing for Asynchronous Hurricane Storm Surge Simulations. In Proceedings of the Parallel Applications Workshop, Alternatives To MPI (PAW-ATM '18). IEEE Computer Society, Los Alamitos, CA, USA, 44--56.
[6]
Alexander Breuer, Alexander Heinecke, and Michael Bader. 2016. Petascale Local Time Stepping for the ADER-DG Finite Element Method. In International Parallel and Distributed Processing Symposium (IPDPS '16). IEEE Computer Society, Los Alamitos, CA, USA, 854--863.
[7]
Steven R. Brus, Damrongsak Wirasaet, Ethan J. Kubatko, Joannes J. Westerink, and Clint Dawson. 2019. High-order discontinuous Galerkin methods for coastal hydrodynamics applications. Computer Methods in Applied Mechanics and Engineering, Vol. 355 (2019), 860 -- 899.
[8]
George F. Carrier and Harvey P. Greenspan. 1958. Water waves of finite amplitude on a sloping beach. Journal of Fluid Mechanics, Vol. 4, 1 (1958), 97--109.
[9]
Francc ois E Cellier, Ernesto Kofman, Gustavo Migoni, and Mario Bortolotto. 2008. Quantized state system simulation. Proc. GCMS'08, Grand Challenges in Modeling and Simulation (2008), 504--510.
[10]
Cy Chan, Bin Wang, John Bachan, and Jane Macfarlane. 2018. Mobiliti: scalable transportation simulation using high-performance parallel computing. In 2018 21st International Conference on Intelligent Transportation Systems (Maui, HI) (ITSC). IEEE, New York, NY, USA, 634--641.
[11]
Emil M. Constantinescu and Adrian Sandu. 2007. Multirate Timestepping Methods for Hyperbolic Conservation Laws. Journal of Scientific Computing, Vol. 33, 3 (01 December 2007), 239--278.
[12]
Clint Dawson and Robert Kirby. 2001. High Resolution Schemes for Conservation Laws with Locally Varying Time Steps. SIAM Journal on Scientific Computing, Vol. 22, 6 (2001), 2256--2281.
[13]
Clint Dawson, Ethan J. Kubatko, Joannes J. Westerink, Corey Trahan, Christopher Mirabito, Craig Michoski, and Nishant Panda. 2011. Discontinuous Galerkin methods for modeling hurricane storm surge. Advances in Water Resources, Vol. 34, 9 (2011), 1165--1176.
[14]
Clint Dawson, Corey Jason Trahan, Ethan J. Kubatko, and Joannes J. Westerink. 2013. A parallel local timestepping Runge-Kutta discontinuous Galerkin method with applications to coastal ocean modeling. Computer Methods in Applied Mechanics and Engineering, Vol. 259 (2013), 154 -- 165.
[15]
Xenofon Floros, Federico Bergero, Francc ois E Cellier, and Ernesto Kofman. 2011. Automated simulation of modelica models with QSS methods-the discontinuous case. In 8th International Modelica Conference. 657--667.
[16]
Richard M Fujimoto. 1990. Parallel discrete event simulation. Commun. ACM, Vol. 33, 10 (1990), 30--53.
[17]
Ernst Hairer, Syvert P. Nørsett, and Gerhard Wanner. 1993. Solving Ordinary Differential Equations I .Springer-Verlag, Berlin Heidelberg.
[18]
Thi-Thao-Phuong Hoang, Wei Leng, Lili Ju, Zhu Wang, and Konstantin Pieper. 2019. Conservative explicit local time-stepping schemes for the shallow water equations. J. Comput. Phys., Vol. 382 (2019), 152 -- 176.
[19]
Jen-Chih Huang, Xiangmin Jiao, Richard M. Fujimoto, and Hongyuan Zha. 2007. DAG-guided Parallel Asynchronous Variational Integrators with Super-elements. In Proceedings of the 2007 Summer Computer Simulation Conference (San Diego, California) (SCSC '07). Society for Computer Simulation International, San Diego, CA, USA, 691--697.
[20]
George Karypis and Vipin Kumar. 1998. A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM Journal on Scientific Computing, Vol. 20, 1 (1998), 359--392.
[21]
Ernesto Kofman and Sergio Junco. 2001. Quantized-state systems: a DEVS Approach for continuous system simulation. Transactions of The Society for Modeling and Simulation International, Vol. 18, 3 (2001), 123--132.
[22]
Lilia Krivodonova. 2010. An efficient local time-stepping scheme for solution of nonlinear conservation laws. J. Comput. Phys., Vol. 229, 22 (2010), 8537 -- 8551.
[23]
Randall J. LeVeque. 1992. Numerical methods for conservation laws .Birk"auser Verlag, Basel.
[24]
A. Lew, J. E. Marsden, M. Ortiz, and M. West. 2003. Asynchronous Variational Integrators. Archive for Rational Mechanics and Analysis, Vol. 167, 2 (01 April 2003), 85--146.
[25]
Yuri A. Omelchenko and Homa Karimabadi. 2006. Event-driven, hybrid particle-in-cell simulation: A new paradigm for multi-scale plasma modeling. J. Comput. Phys., Vol. 216, 1 (2006), 153 -- 178.
[26]
Yuri A. Omelchenko and Homa Karimabadi. 2007. A time-accurate explicit multi-scale technique for gas dynamics. J. Comput. Phys., Vol. 226, 1 (2007), 282--300.
[27]
Yuri A. Omelchenko and Homa Karimabadi. 2012. HYPERS: A unidimensional asynchronous framework for multiscale hybrid simulations. J. Comput. Phys., Vol. 231, 4 (2012), 1766 -- 1780.
[28]
Stanley Osher and Richard Sanders. 1983. Numerical Approximations to Nonlinear Conservation Laws with Locally Varying Time and Space Grids. Math. Comp., Vol. 41, 164 (1983), 321--336.
[29]
John R Rice. 1960. Split Runge-Kutta Method for Simultaneous Equations. Journal of Research of the National Bureau of Standards: Mathematics and mathematical physics. B, Vol. 64B, 3 (1960), 151--170.
[30]
Martin Schlegel, Oswald Knoth, Martin Arnold, and Ralf Wolke. 2009. Multirate Runge-Kutta schemes for advection equations. J. Comput. Appl. Math., Vol. 226, 2 (2009), 345 -- 357. Special Issue: Large scale scientific computations.
[31]
Carsten Uphoff, Sebastian Rettenberger, Michael Bader, Elizabeth H. Madden, Thomas Ulrich, Stephanie Wollherr, and Alice-Agnes Gabriel. 2017. Extreme Scale Multi-physics Simulations of the Tsunamigenic 2004 Sumatra Megathrust Earthquake. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (Denver, Colorado) (SC '17). ACM, New York, NY, USA, Article 21, 16 pages.

Cited By

View all
  • (2024)Devastator: A Scalable Parallel Discrete Event Simulation Framework for Modern C++Proceedings of the 38th ACM SIGSIM Conference on Principles of Advanced Discrete Simulation10.1145/3615979.3656061(35-46)Online publication date: 24-Jun-2024
  • (2024)An explicit local space-time adaptive framework for monodomain models in cardiac electrophysiologyComputer Methods in Applied Mechanics and Engineering10.1016/j.cma.2024.116806422(116806)Online publication date: Mar-2024
  • (2022)Performance Analysis of Speculative Parallel Adaptive Local Timestepping for Conservation LawsACM Transactions on Modeling and Computer Simulation10.1145/354599632:4(1-30)Online publication date: 5-Nov-2022

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SIGSIM-PADS '21: Proceedings of the 2021 ACM SIGSIM Conference on Principles of Advanced Discrete Simulation
May 2021
181 pages
ISBN:9781450382960
DOI:10.1145/3437959
This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 June 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. hurricane storm surge
  2. local timestepping
  3. parallel discrete event simulation
  4. shallow water equations
  5. timewarp

Qualifiers

  • Research-article

Funding Sources

Conference

SIGSIM-PADS '21
Sponsor:

Acceptance Rates

Overall Acceptance Rate 398 of 779 submissions, 51%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)84
  • Downloads (Last 6 weeks)7
Reflects downloads up to 08 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Devastator: A Scalable Parallel Discrete Event Simulation Framework for Modern C++Proceedings of the 38th ACM SIGSIM Conference on Principles of Advanced Discrete Simulation10.1145/3615979.3656061(35-46)Online publication date: 24-Jun-2024
  • (2024)An explicit local space-time adaptive framework for monodomain models in cardiac electrophysiologyComputer Methods in Applied Mechanics and Engineering10.1016/j.cma.2024.116806422(116806)Online publication date: Mar-2024
  • (2022)Performance Analysis of Speculative Parallel Adaptive Local Timestepping for Conservation LawsACM Transactions on Modeling and Computer Simulation10.1145/354599632:4(1-30)Online publication date: 5-Nov-2022

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media