Skip to main content
SpringerLink
Log in

Hybrid Parallel Multigrid Methods for Geodynamical Simulations

  • Conference paper
  • First Online:
Book cover Software for Exascale Computing - SPPEXA 2013-2015

Part of the book series: Lecture Notes in Computational Science and Engineering ((LNCSE,volume 113))

Abstract

Even on modern supercomputer architectures, Earth mantle simulations are so compute intensive that they are considered grand challenge applications. The dominating roadblocks in this branch of Geophysics are model complexity and uncertainty in parameters and data, e.g., rheology and seismically imaged mantle heterogeneity, as well as the enormous space and time scales that must be resolved in the computational models. This article reports on a massively parallel all-at-once multigrid solver for the Stokes system as it arises in mantle convection models. The solver employs the hierarchical hybrid grids framework and demonstrates that a system with coupled velocity components and with more than a trillion (1. 7 ⋅ 1012) degrees of freedom can be solved in about 1,000 s using 40,960 compute cores of JUQUEEN. The simulation framework is used to investigate the influence of asthenosphere thickness and viscosity on upper mantle velocities in a static scenario. Additionally, results for a time-dependent simulation with a time-variable temperature-dependent viscosity model are presented.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    listed top 9 of the TOP500 list, Nov. 2015.

References

  1. Bangerth, W., Burstedde, C., Heister, T., Kronbichler, M.: Algorithms and data structures for massively parallel generic adaptive finite element codes. ACM Trans. Math. Soft. 38 (2), 14:1–14:28 (2011)

    Google Scholar 

  2. Bank, R.E., Welfert, B.D., Yserentant, H.: A class of iterative methods for solving saddle point problems. Numer. Math. 56 (7), 645–666 (1990)

    Article  MathSciNet  MATH  Google Scholar 

  3. Baumgardner, J.R.: Three-dimensional treatment of convective flow in the Earth’s mantle. J. Stat. Phys. 39 (5/6), 501–511 (1985)

    Article  Google Scholar 

  4. Becker, T.W., Boschi, L.: A comparison of tomographic and geodynamic mantle models. Geochem. Geophy. Geosy. 3, 1525–2027 (2002)

    Article  Google Scholar 

  5. Bergen, B., Gradl, T., Rüde, U., Hülsemann, F.: A massively parallel multigrid method for finite elements. Comput. Sci. Eng. 8 (6), 56–62 (2006)

    Article  Google Scholar 

  6. Bergen, B., Wellein, G., Hülsemann, F., Rüde, U.: Hierarchical hybrid grids: achieving TERAFLOP performance on large scale finite element simulations. Int. J. Parallel Emergent Distrib. Syst. 22 (4), 311–329 (2007)

    Article  MathSciNet  Google Scholar 

  7. Brandt, A.: Guide to multigrid development. In: Multigrid methods, pp. 220–312. Springer, Berlin/Heidelberg (1982). Republished as: Multigrid Techniques: 1984 guide with applications to fluid dynamics, revised edition, SIAM, 2011

    Google Scholar 

  8. Brandt, A., Dinar, N.: Multigrid solutions to elliptic flow problems. In: Numerical methods for partial differential equations (Proc. Adv. Sem., Math. Res. Center, Univ. Wisconsin, Madison, Wis., 1978), Publ. Math. Res. Center Univ. Wisconsin, vol. 42, pp. 53–147. Academic Press, New York/London (1979)

    Google Scholar 

  9. Brezzi, F., Douglas, Jr., J.: Stabilized mixed methods for the Stokes problem. Numer. Math. 53 (1–2), 225–235 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  10. Brezzi, F., Fortin, M.: Mixed and Hybrid Finite Element Methods. Springer, New York (1991)

    Book  MATH  Google Scholar 

  11. Bunge, H.P., Baumgardner, J.R.: Mantle convection modeling on parallel virtual machines. Comput. Phys. 9 (2), 207–215 (1995)

    Article  Google Scholar 

  12. Bunge, H.P., Hagelberg, C.R., Travis, B.J.: Mantle circulation models with variational data assimilation: inferring past mantle flow and structure from plate motion histories and seismic tomography. Geophys. J. Int. 152 (2), 280–301 (2003). http://www.geophysik.uni-muenchen.de/Members/bunge/download/adjoint-paper.pdf

    Article  Google Scholar 

  13. Bunge, H.P., Richards, M.A., Baumgardner, J.R.: A sensitivity study of three-dimensional spherical mantle convection at 108 Rayleigh number: effects of depth-dependent viscosity, heating mode, and an endothermic phase change. J. Geophys. Res. 102, 11991–12007 (1997)

    Article  Google Scholar 

  14. Bunge, H.P., Richards, M., Lithgow-Bertelloni, C., Baumgardner, J.R., Grand, S., Romanowicz, B.: Time scales and heterogeneous structure in geodynamic earth models. Science 280, 91–95 (1998). http://www.geophysik.uni-muenchen.de/~bunge/downloads/gemlab.pdf

    Article  Google Scholar 

  15. Burstedde, C., Stadler, G., Alisic, L., Wilcox, L.C., Tan, E., Gurnis, M., Ghattas, O.: Large-scale adaptive mantle convection simulation. Geophys. J. Internat. 192 (3), 889–906 (2013)

    Article  Google Scholar 

  16. Burstedde, C., Wilcox, L.C., Ghattas, O.: p4est: Scalable algorithms for parallel adaptive mesh refinement on forests of octrees. SIAM J. Sci. Comp. 33 (3), 1103–1133 (2011)

    Google Scholar 

  17. CIG – Computational Infrastructure for Geodynamics: ASPECT: Advanced Solver for Problems in Earth’s ConvecTion, User Manual (2015), version 1.3

    Google Scholar 

  18. Council, N.R.: Origin and Evolution of Earth: Research Questions for a Changing Planet. The National Academies Press, Washington, DC (2008). http://www.nap.edu/catalog/12161/origin-and-evolution-of-earth-research-questions-for-a-changing

    Google Scholar 

  19. Davies, D.R., Davies, J.H., Bollada, P.C., Hassan, O., Morgan, K., Nithiarasu, P.: A hierarchical mesh refinement technique for global 3-D spherical mantle convection modelling. Geosci. Model Dev. 6 (4), 1095–1107 (2013)

    Article  Google Scholar 

  20. Davies, D.R., Goes, S., Davies, J.H., Schuberth, B.S.A., Bunge, H.P., Ritsema, J.: Reconciling dynamic and seismic models of Earth’s lower mantle: the dominant role of thermal heterogeneity. Earth Planet. Sci. Lett. 353–354 (1), 253–269 (2012)

    Article  Google Scholar 

  21. Dziewonski, A.M., Anderson, D.L.: Preliminary reference Earth model. Phys. Earth Plan. Int. 25, 297–356 (1981)

    Article  Google Scholar 

  22. Elman, H.C., Silvester, D.J., Wathen, A.J.: Finite Elements and Fast Iterative Solvers: With Applications in Incompressible Fluid Dynamics. Oxford University Press, New York (2005)

    MATH  Google Scholar 

  23. Engelman, M.S., Sani, R.L., Gresho, P.M.: The implementation of normal and/or tangential boundary conditions in finite element codes for incompressible fluid flow. Int. J. Numer. Methods Fluids 2 (3), 225–238 (1982)

    Article  MathSciNet  MATH  Google Scholar 

  24. Fichtner, A., Kennett, B.L.N., Igel, H., Bunge, H.P.: Full seismic waveform tomography for upper-mantle structure in the Australasian region using adjoint methods. Geophys. J. Int. 179 (3), 1703–1725 (2009)

    Article  Google Scholar 

  25. Gaspar, F.J., Notay, Y., Oosterlee, C.W., Rodrigo, C.: A simple and efficient segregated smoother for the discrete Stokes equations. SIAM J. Sci. Comput. 36 (3), A1187–A1206 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  26. Girault, V., Raviart, P.A.: Finite Element Methods for Navier-Stokes Equations. Springer, New York (1986)

    Book  MATH  Google Scholar 

  27. Gmeiner, B., Huber, M., John, L., Rüde, U., Waluga, C., Wohlmuth, B.: Massively parallel large scale stokes flow simulation. In: Binder, K., Müller, M., Kremer, M., Schnurpfeil, A. (eds.) NIC Symposium 2016. Schriften des Forschungszentrums Jülich, NIC Series, vol. 48, pp. 333–341. ISBN:978-3-95806-109-5

    Google Scholar 

  28. Gmeiner, B., Huber, M., John, L., Rüde, U., Wohlmuth, B.: A quantitative performance analysis for Stokes solvers at the extreme scale (submitted, arXiv:1511.02134)

    Google Scholar 

  29. Gmeiner, B., Rüde, U., Stengel, H., Waluga, C., Wohlmuth, B.: Performance and scalability of hierarchical hybrid multigrid solvers for stokes systems. SIAM J. Sci. Comput. 37 (2), C143–C168 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  30. Gmeiner, B., Rüde, U., Stengel, H., Waluga, C., Wohlmuth, B.: Towards textbook efficiency for parallel multigrid. Numer. Math. Theory Methods Appl. 8, 22–46 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  31. Gmeiner, B., Waluga, C., Wohlmuth, B.: Local mass-corrections for continuous pressure approximations of incompressible flow. SIAM J. Numer. Anal. 52 (6), 2931–2956 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  32. Grand, S.P., van der Hilst, R.D., Widiyantoro, S.: Global seismic tomography: a snapshot of convection in the earth. GSA Today 7, 1–7 (1997)

    Google Scholar 

  33. Hager, G., Treibig, J., Habich, J., Wellein, G.: Exploring performance and power properties of modern multi-core chips via simple machine models. Concurr. Comput. 28, 1–2 (2014)

    Google Scholar 

  34. Hartley, R.A., Roberts, G.G., White, N., Richardson, C.: Transient convective uplift of an ancient buried landscape. Nat. Geosci. 4, 562–565 (2011)

    Article  Google Scholar 

  35. Haskell, N.A.: The motion of a fluid under a surface load. Physics 6, 265–269 (1935)

    Article  MATH  Google Scholar 

  36. Höink, T., Lenardic, A.: Three-dimensional mantle convection simulations with a low-viscosity asthenosphere and the relationship between heat flow and the horizontal length scale of convection. Geophys. Res. Lett. 35, L10304 (2008)

    Article  Google Scholar 

  37. Huber, M., Gmeiner, B., Rüde, U., Wohlmuth, B.: Resilience for multigrid software at the extreme scale (preprint, arXiv:1506.06185)

    Google Scholar 

  38. Huber, M., John, L., Pustejovska, P., Rüde, U., Waluga, C., Wohlmuth, B.: Solution Techniques for the Stokes System: a priori and a posteriori modifications, resilient algorithms. In Proceedings of the ICIAM, Beijing (2015). arXiv:151105759

    Google Scholar 

  39. Mitrovica, J.X.: Haskell [1935] revisited. J. Geophys. Res. 101, 555–569 (1996)

    Article  Google Scholar 

  40. Müller, R.D., Sdrolias, M., Gaina, C., Roest, W.R.: Age, spreading rates, and spreading asymmetry of the world’s ocean crust. Geochem. Geophy. Geosy. 9, 1525–2027 (2008)

    Article  Google Scholar 

  41. Oeser, J., Bunge, H.P., Mohr, M.: Cluster Design in the Earth Sciences: TETHYS. In: Gerndt, M., Kranzlmüller, D. (eds.) High Performance Computing and Communications – Second International Conference, HPCC 2006, Munich. Lecture Notes in Computer Science, vol. 4208, pp. 31–40. Springer (2006). http://www.springerlink.com/content/l18628n708k11127

  42. Parnell-Turner, R., White, N., Henstock, T., Murton, B., Maclennan, J., Jones, S.M.: A continuous 55 million year record of transient mantle plume activity beneath Iceland. Nat. Geosci. 7, 914–919 (2014)

    Article  Google Scholar 

  43. Resovsky, J., Trampert, J.: Using probabilistic seismic tomography to test mantle velocity–density relationships. Earth Planet. Sci. Lett. 215 (1), 121–134 (2003)

    Article  Google Scholar 

  44. Ricard, Y.: Physics of mantle convection. In: Schubert, G. (ed.) Treatise on Geophysics, vol. 7. Elsevier, Amsterdam (2007)

    Google Scholar 

  45. Ritsema, J., von Heijst, H.J., Woodhouse, J.H.: Global transition zone tomography. J. Geophys. Res. 109, B02302 (2004)

    Article  Google Scholar 

  46. Rudi, J., Malossi, A.C.I., Isaac, T., Stadler, G., Gurnis, M., Staar, P.W.J., Ineichen, Y., Bekas, C., Curioni, A., Ghattas, O.: An Extreme-scale Implicit Solver for Complex PDEs: Highly Heterogeneous Flow in Earth’s Mantle. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’15), pp. 5:1–5:12. ACM, New York (2015). http://doi.acm.org/10.1145/2807591.2807675

  47. Schöberl, J., Zulehner, W.: On Schwarz-type smoothers for saddle point problems. Numer. Math. 95 (2), 377–399 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  48. Seton, M., Müller, R.D., Zahirovic, S., Gaina, C., Torsvik, T.H., Shephard, G., Talsma, A., Gurnis, M., Turner, M., Maus, S., Chandler, M.: Global continental and ocean basin reconstructions since 200 ma. Earth-Sci. Rev. 113, 212–270 (2012)

    Article  Google Scholar 

  49. Stixrude, L., Lithgow-Bertelloni, C.: Thermodynamics of mantle minerals – I. Physical properties. Geophys. J. Int. 162, 610–632 (2005)

    Article  Google Scholar 

  50. Sundar, H., Stadler, G., Biros, G.: Comparison of multigrid algorithms for high-order continuous finite element discretizations. Numer. Linear Algebra Appl. 22 (4), 664–680 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  51. Tackley, P.J.: Effects of strongly variable viscosity on three-dimensional compressible convection in planetary mantles. J. Geophys. Res. 101, 3311–3332 (1996)

    Article  Google Scholar 

  52. Tackley, P.J.: Mantle convection and plate tectonics: toward an integrated physical and chemical theory. Science 16, 2002–2007 (2000)

    Article  Google Scholar 

  53. Tackley, P.J., Stevenson, D.J., Glatzmaier, G.A., Schubert, G.: Effects of multiple phase transitions in a three-dimensional spherical model of convection in earth’s mantle. J. Geophys. Res. 99 (B8), 15877–15901 (1994)

    Article  Google Scholar 

  54. Tan, E., Choi, E., Thoutireddy, P., Gurnis, M., Aivazis, M.: GeoFramework: coupling multiple models of mantle convection within a computational framework. Geochem. Geophy. Geosy. 7 (6), Q06001 (2006)

    Article  Google Scholar 

  55. Urquiza, J.M., Garon, A., Farinas, M.I.: Weak imposition of the slip boundary condition on curved boundaries for Stokes flow. J. Comput. Phys. 256, 748–767 (2014)

    Article  MathSciNet  Google Scholar 

  56. Verfürth, R.: Finite element approximation of incompressible Navier-Stokes equations with slip boundary condition. Numer. Math. 50 (6), 697–721 (1987)

    Article  MathSciNet  MATH  Google Scholar 

  57. Vogt, P.R.: Asthenosphere motion recorded by the ocean floor south of Iceland. Earth Planet. Sci. Lett. 13, 153–160 (1971), http://www.sciencedirect.com/science/article/pii/0012821X7190118X

    Article  Google Scholar 

  58. Waluga, C., Wohlmuth, B., Rüde, U.: Mass-corrections for the conservative coupling of flow and transport on collocated meshes. J. Comp. Phys. 305, 319–332 (2016)

    Article  MathSciNet  Google Scholar 

  59. Weismüller, J., Gmeiner, B., Ghelichkhan, S., Huber, M., John, L., Wohlmuth, B., Rüde, U., Bunge, H.P.: Fast asthenosphere motion in high-resolution global mantle flow models. Geophys. Res. Lett. 42 (18), 7429–7435 (2015)

    Article  Google Scholar 

  60. Williams, S.W., Waterman, A., Patterson, D.A.: Roofline: an insightful visual performance model for floating-point programs and multicore architectures. Tech. Rep. UCB/EECS-2008-134, EECS Department, University of California, Berkeley (Oct 2008)

    Google Scholar 

  61. Zhong, S., McNamara, A., Tan, E., Moresi, L., Gurnis, M.: A benchmark study on mantle convection in a 3-D spherical shell using CitcomS. Geochem. Geophy. Geosy. 9, Q10017 (2008)

    Article  Google Scholar 

  62. Zhong, S., Zuber, M.T., Moresi, L., Gurnis, M.: The role of temperature-dependent viscosity and surface plates in spherical shell models of mantle convection. J. Geophys. Res. 105 (B5), 11063–11082 (2000)

    Article  Google Scholar 

  63. Zulehner, W.: Analysis of iterative methods for saddle point problems: a unified approach. Math. Comput. 71 (238), 479–505 (2002)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

This work was supported (in part) by the German Research Foundation (DFG) through the Priority Programme 1648 “Software for Exascale Computing” (SPPEXA) and grant WO 671/11-1. The authors gratefully acknowledge the Gauss Centre for Supercomputing (GCS) for providing computing time through the John von Neumann Institute for Computing (NIC) on the GCS share of the supercomputer JUQUEEN at Jülich Supercomputing Centre (JSC).

Author information

Authors and Affiliations

  1. Department of Earth and Environmental Sciences, Ludwig-Maximilians-Universität München, München, Germany

    Simon Bauer, Hans-Peter Bunge, Marcus Mohr & Jens Weismüller

  2. Institute for Numerical Mathematics, Technische Universität München, München, Germany

    Daniel Drzisga, Markus Huber, Lorenz John, Christian Waluga & Barbara Wohlmuth

  3. Department of Computer Science 10, Friedrich-Alexander-University Erlangen-Nuremberg, Erlangen, Germany

    Björn Gmeiner & Ulrich Rüde

  4. Erlangen Regional Computing Center (RRZE), Friedrich-Alexander-University Erlangen-Nuremberg, Erlangen, Germany

    Holger Stengel, Gerhard Wellein & Markus Wittmann

Authors
  1. Simon Bauer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hans-Peter Bunge
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Daniel Drzisga
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Björn Gmeiner
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Markus Huber
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Lorenz John
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Marcus Mohr
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Ulrich Rüde
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Holger Stengel
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Christian Waluga
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Jens Weismüller
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Gerhard Wellein
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Markus Wittmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Barbara Wohlmuth
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lorenz John .

Editor information

Editors and Affiliations

  1. Technische Universität München Institut für Informatik, Garching, Bayern, Germany

    Hans-Joachim Bungartz

  2. Technische Universität München Institut für Informatik, Garching, Germany

    Philipp Neumann

  3. Technische Universität Dresden, Dresden, Germany

    Wolfgang E. Nagel

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this paper

Cite this paper

Bauer, S. et al. (2016). Hybrid Parallel Multigrid Methods for Geodynamical Simulations. In: Bungartz, HJ., Neumann, P., Nagel, W. (eds) Software for Exascale Computing - SPPEXA 2013-2015. Lecture Notes in Computational Science and Engineering, vol 113. Springer, Cham. https://doi.org/10.1007/978-3-319-40528-5_10

Download citation

Publish with us

Policies and ethics

Search

Navigation