Skip to main content
SpringerLink
Log in

Efficient GPU-Implementation of Adaptive Mesh Refinement for the Shallow-Water Equations

  • Published:
Journal of Scientific Computing Aims and scope Submit manuscript

Abstract

The shallow-water equations model hydrostatic flow below a free surface for cases in which the ratio between the vertical and horizontal length scales is small and are used to describe waves in lakes, rivers, oceans, and the atmosphere. The equations admit discontinuous solutions, and numerical solutions are typically computed using high-resolution schemes. For many practical problems, there is a need to increase the grid resolution locally to capture complicated structures or steep gradients in the solution. An efficient method to this end is adaptive mesh refinement (AMR), which recursively refines the grid in parts of the domain and adaptively updates the refinement as the simulation progresses. Several authors have demonstrated that the explicit stencil computations of high-resolution schemes map particularly well to many-core architectures seen in hardware accelerators such as graphics processing units (GPUs). Herein, we present the first full GPU-implementation of a block-based AMR method for the second-order Kurganov–Petrova central scheme. We discuss implementation details, potential pitfalls, and key insights, and present a series of performance and accuracy tests. Although it is only presented for a particular case herein, we believe our approach to GPU-implementation of AMR is transferable to other hyperbolic conservation laws, numerical schemes, and architectures similar to the GPU.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

Notes

  1. On the June 2013 list, there were 43 GPU-powered machines, and 12 machines using the Intel Xeon Phi co-processor.

  2. It should be noted that some parts of the code, such as launching kernels on the GPU, is necessarily performed on the CPU.

  3. Here, we use a midpoint integration rule, central-upwind numerical flux, and a piecewise-linear reconstruction with slopes limited by a generalized minmod function, see [18] for details. The weight is 0.5 for both steps of the particular stability-preserving, second-order Runge–Kutta scheme used herein. If other schemes are used, this weight must be altered accordingly.

  4. Only the array containing boundary values for the end of the time-step series is filled in the initialization. The boundary values for the start of the time-step series is simply set to be the end-values from the previous time-step series.

References

  1. Berger, M., LeVeque, R.: Adaptive mesh refinement for two-dimensional hyperbolic systems and the AMRCLAW software. SIAM J. Numer. Anal. 35, 2298–2316 (1998)

    Article  MATH  MathSciNet  Google Scholar 

  2. Berger, M.J., Colella, P.: Local adaptive mesh refinement for shock hydrodynamics. J. Comput. Phys. 82, 64–84 (1989). doi:10.1016/0021-9991(89)90035-1

    Article  MATH  Google Scholar 

  3. Berger, M.J., Oliger, J.: Adaptive mesh refinement for hyperbolic partial differential equations. J. Comput. Phys. 53(3), 484–512 (1984). doi:10.1016/0021-9991(84)90073-1

    Article  MATH  MathSciNet  Google Scholar 

  4. Brodtkorb, A., Dyken, C., Hagen, T., Hjelmervik, J., Storaasli, O.: State-of-the-art in heterogeneous computing. J. Sci. Program. 18(1), 1–33 (2010)

    Google Scholar 

  5. Brodtkorb, A.R., Hagen, T.R., Lie, K.A., Natvig, J.R.: Simulation and visualization of the Saint-Venant system using GPUs. Comput. Vis. Sci. 13(7), 341–353 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  6. Brodtkorb, A.R., Hagen, T.R., Sætra, M.L.: Graphics processing unit (GPU) programming strategies and trends in GPU computing. J. Parallel Distrib. Comput. 73(1), 4–13 (2013). doi:10.1016/j.jpdc.2012.04.003

    Article  Google Scholar 

  7. Brodtkorb, A.R., Sætra, M.L.: Explicit shallow water simulations on GPUs: guidelines and best practices. In: XIX International Conference on Water Resources, CMWR 2012, June 17–22, 2012. University of Illinois at Urbana-Champaign (2012)

  8. Brodtkorb, A.R., Sætra, M.L., Altinakar, M.: Efficient shallow water simulations on GPUs: implementation, visualization, verification, and validation. Comput. Fluids 55, 1–12 (2012). doi:10.1016/j.compfluid.2011.10.012

    Article  MATH  MathSciNet  Google Scholar 

  9. Burstedde, C., Ghattas, O., Gurnis, M., Isaac, T., Stadler, G., Warburton, T., Wilcox, L.: Extreme-scale AMR. In: Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–12. IEEE Computer Society (2010)

  10. Colella, P., Graves, D.T., Johnson, J.N., Johansen, H.S., Keen, N.D., Ligocki, T.J., Martin, D.F., McCorquodale, P.W., Modiano, D., Schwartz, P.O., Sternberg, T.D., Van Straalen, B.: Chombo software package for AMR applications design document. Technical report, Lawrence Berkeley National Laboratory (2012)

  11. Delestre, O., Lucas, C., Ksinant, P.A., Darboux, F., Laguerre, C., Vo, T.N.T., James, F., Cordier, S.: SWASHES: a compilation of shallow water analytic solutions for hydraulic and environmental studies. Int. J. Numer. Methods Fluids 72(3), 269–300 (2013). doi:10.1002/fld.3741

    Article  MathSciNet  Google Scholar 

  12. Goutal, N., Maurel, F.: Proceedings of the 2nd Workshop on Dam-Break Wave Simulation. Technical report, Groupe Hydraulique Fluviale, Département Laboratoire National d’Hydraulique, Electricité de France (1997)

  13. Harris, M.: NVIDIA GPU computing SDK 4.1: Optimizing parallel reduction in CUDA (2011)

  14. Holdahl, R., Holden, H., Lie, K.A.: Unconditionally stable splitting methods for the shallow water equations. BIT Numer. Math. 39(3), 451–472 (1999). doi:10.1023/A:1022366502335

    Article  MATH  MathSciNet  Google Scholar 

  15. Hornung, R.D., Kohn, S.R.: Managing application complexity in the SAMRAI object-oriented framework. Concurr. Comput. Pract. Exp. 14(5), 347–368 (2002). doi:10.1002/cpe.652

    Article  MATH  Google Scholar 

  16. Jones, S.: Introduction to dynamic parallelism. In: GPU Technology Conference Presentation S0338 (2012)

  17. Khronos Group: OpenCL. http://www.khronos.org/opencl/

  18. Kurganov, A., Petrova, G.: A second-order well-balanced positivity preserving central-upwind scheme for the Saint-Venant system. Commun. Math. Sci. 5, 133–160 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  19. Lawrence Berkeley National Laboratory, Center for Computational Sciences and Engineering: BoxLib. https://ccse.lbl.gov/BoxLib/index.html

  20. LeVeque, R.J., George, D.L., Berger, M.J.: Tsunami modelling with adaptively refined finite volume methods. Acta Numer. 20, 211–289 (2011). doi:10.1017/S0962492911000043

    Article  MATH  MathSciNet  Google Scholar 

  21. Li, R., Wu, S.: H-adaptive mesh method with double tolerance adaptive strategy for hyperbolic conservation laws. J. Sci. Comput. 56(3), 616–636 (2013). doi:10.1007/s10915-013-9692-1

    Article  MATH  MathSciNet  Google Scholar 

  22. Lie, K.A., Noelle, S.: On the artificial compression method for second-order nonoscillatory central difference schemes for systems of conservation laws. SIAM J. Sci. Comput. 24(4), 1157–1174 (2003)

    Article  MATH  MathSciNet  Google Scholar 

  23. MacNeice, P., Olson, K.M., Mobarry, C., de Fainchtein, R., Packer, C.: PARAMESH: a parallel adaptive mesh refinement community toolkit. Comput. Phys. Commun. 126(3), 330–354 (2000). doi:10.1016/S0010-4655(99)00501-9

    Article  MATH  Google Scholar 

  24. Meuer, H., Strohmaier, E., Dongarra, J., Simon, H.: Top 500 supercomputer sites (2013). http://www.top500.org/

  25. Nessyahu, H., Tadmor, E.: Non-oscillatory central differencing for hyperbolic conservation laws. J. Comput. Phys. 87(2), 408–463 (1990)

    Article  MATH  MathSciNet  Google Scholar 

  26. Nicholaeff, D., Davis, N., Trujillo, D., Robey, R.W.: Cell-based adaptive mesh refinement implemented with general purpose graphics processing units. Technical report, Los Alamos National Laboratory (2012)

  27. NVIDIA: NVIDIA CUDA programming guide 5.0 (2012)

  28. NVIDIA: NVIDIA GeForce GTX 680. Technical report, NVIDIA Corporation (2012)

  29. NVIDIA: CUDA community showcase (2013). http://www.nvidia.com/object/cuda_showcase_html.html

  30. Owens, J., Houston, M., Luebke, D., Green, S., Stone, J., Phillips, J.: GPU computing. Proc. IEEE 96(5), 879–899 (2008). doi:10.1109/JPROC.2008.917757

    Article  Google Scholar 

  31. Sætra, M.L.: Shallow water simulation on GPUs for sparse domains. In: A. Cangiani, R.L. Davidchack, E. Georgoulis, A.N. Gorban, J. Levesley, M.V. Tretyakov (eds.) Numerical Mathematics and Advanced Applications 2011, pp. 673–680. Springer, Berlin (2013). doi:10.1007/978-3-642-33134-3_71.

  32. Sætra, M.L., Brodtkorb, A.R.: Shallow water simulations on multiple GPUs. In: K. Jónasson (ed.) Applied Parallel and Scientific Computing, Lecture Notes in Computer Science, vol. 7134, pp. 56–66. Springer, Berlin (2012). doi:10.1007/978-3-642-28145-7_6.

  33. Schive, H.Y., Tsai, Y.C., Chiueh, T.: GAMER: a graphic processing unit accelerated adaptive-mesh-refinement code for astrophysics. Astrophys. J. Suppl. Ser. 186(2), 457–484 (2010). doi:10.1088/0067-0049/186/2/457

    Article  Google Scholar 

  34. Sweby, P.K.: High resolution schemes using flux limiters for hyperbolic conservation laws. SIAM J. Numer. Anal. 21(5), 995–1011 (1984)

    Article  MATH  MathSciNet  Google Scholar 

  35. The Enzo Project: Enzo. http://enzo-project.org/

  36. van Leer, B.: Towards the ultimate conservative difference scheme. V. A second-order sequel to Godunov’s method. J. Comput. Phys. 32(1), 101–136 (1979). doi:10.1016/0021-9991(79)90145-1

    Article  Google Scholar 

  37. Wang, P., Abel, T., Kaehler, R.: Adaptive mesh fluid simulations on GPU. New Astron. 15(7), 581–589 (2010). doi:10.1016/j.newast.2009.10.002

    Article  Google Scholar 

Download references

Acknowledgments

This work is supported in part by the Research Council of Norway through Grant No. 180023 (Parallel3D), and in part by the Centre of Mathematics for Applications at the University of Oslo (Sætra and Lie).

Author information

Authors and Affiliations

  1. Centre of Mathematics for Applications, University of Oslo, P.O. Box 1053, Blindern, 0316 , Oslo, Norway

    Martin L. Sætra & Knut-Andreas Lie

  2. Norwegian Meteorological Institute, P.O. Box 43, Blindern, 0313 , Oslo, Norway

    Martin L. Sætra

  3. Department of Applied Mathematics, SINTEF ICT, P.O. Box 124, Blindern, 0314 , Oslo, Norway

    André R. Brodtkorb & Knut-Andreas Lie

Authors
  1. Martin L. Sætra
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. André R. Brodtkorb
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Knut-Andreas Lie
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Martin L. Sætra.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sætra, M.L., Brodtkorb, A.R. & Lie, KA. Efficient GPU-Implementation of Adaptive Mesh Refinement for the Shallow-Water Equations. J Sci Comput 63, 23–48 (2015). https://doi.org/10.1007/s10915-014-9883-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10915-014-9883-4

Keywords

Mathematics Subject Classification

Search

Navigation