Skip to main content
SpringerLink
Log in

Tools to support mesh adaptation on massively parallel computers

  • Original Article
  • Published:
Engineering with Computers Aims and scope Submit manuscript

Abstract

The scalable execution of parallel adaptive analyses requires the application of dynamic load balancing to repartition the mesh into a set of parts with balanced work load and minimal communication. As the adaptive meshes being generated reach billions of elements and the analyses are performed on massively parallel computers with 100,000’s of computing cores, a number of complexities arise that need to be addressed. This paper presents procedures developed to deal with two of them. The first is a procedure to support multiple parts per processor which is used as the mesh increases in size and it is desirable to partition the mesh to a larger number of computing cores than are currently being used. The second is a predictive load balancing method that sets entity weights before dynamic load balancing steps so that the mesh is well balanced after the mesh adaptation step thus avoiding excessive memory spikes that would otherwise occur during mesh adaptation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Flaherty J, Loy R, Özturan C, Shephard M, Szymanski B, Teresco J, Ziantz L (1996) Parallel structure and dynamic load balancing for adaptive finite element computation. Appl Numer Math 26:241–263

    Article  Google Scholar 

  2. Shephard M, Flaherty J, Bottasso C, de Cougny H, Ozturan C, Simone M (1997) Parallel automated adaptive analysis. Parallel Comput 23:1327–1347

    Article  MathSciNet  MATH  Google Scholar 

  3. Sahni O, Zhou M, Shephard M, Jansen K (2009) Scalable implicit finite element solver for massively parallel processing with demonstration to 160k cores. In: Proceedings of the SC09. Springer, Berlin

  4. Shephard M, Jansen K, Sahni O, Diachin L (2007) Parallel adaptive simulations on unstructured meshes. J Phys Conf Ser 78:012053

    Article  Google Scholar 

  5. Zhou M, Sahni O, Kim H, Figueroa C, Taylor C, Shephard M, Jansen (2010) Cardiovascular flow simulation at extreme scale. Comput Mech 46:71–82 doi:10.1007/s00466-009-0450-z

    Article  MathSciNet  MATH  Google Scholar 

  6. Zhou M, Sahni O, Shephard M, Devine K, Jansen K (2010) Controlling unstructured mesh partitions for massively parallel simulations. SIAM J Sci Comput 32(6):3201–3227

    Google Scholar 

  7. Beall MW, Shephard MS (1997) A general topology-based mesh data structure. Int J Numer Methods Eng 40:1573–1596

    Article  MathSciNet  Google Scholar 

  8. Celes ERW, Paulino GH (2005) A compact adjacency-based topological data structure for finite element mesh representation. Int J Numer Methods Eng 64:1529–1565

    Article  MATH  Google Scholar 

  9. Garimella R (2002), Mesh data structure selection for mesh generation and fea applications. Int J Numer Methods Eng 55:451–478

    Article  MATH  Google Scholar 

  10. Seol E, Shephard M (2006) Efficient distributed mesh data structure for parallel automated adaptive analysis. Eng Comput 22(3–4):197–213

    Article  Google Scholar 

  11. Chand K, Diachin L, Li X, Ollivier-Gooch C, Seol E, Shephard M, Tautges T, Trease H (2008) Toward interoperable mesh, geometry and field components for PDE simulation development. Eng Comput 24(2):165–182

    Article  Google Scholar 

  12. Devine K, Diachin L, Kraftcheck J, Jansen KE, Leung V, Luo X, Miller M, Ollivier-Gooch C, Ovcharenko A, Sahni O, Shephard M, Tautges T, Xie T, Zhou M (2009) Interoperable mesh components for large-scale, distributed-memory simulations. J Phys Conf Ser 180:012011

    Article  Google Scholar 

  13. Boman E, Devine K, Fisk L, Heaphy R, Hendrickson B, Leung V, Vaughan C, Catalyurek U, Bozdag D, Mitchell W (1999) Zoltan home page. http://www.cs.sandia.gov/Zoltan

  14. Devine K, Boman E, Heaphy R, Hendrickson B, Vaughan C (2002) Zoltan data management services for parallel dynamic applications. Comput Sci Eng 4(2):90–97

    Article  Google Scholar 

  15. Devine K, Boman E, Heaphy R, Bisseling R, Catalyurek U (2006) Parallel hypergraph partitioning for scientific computing. In: Proceedings of 20th international parallel and distributed processing symposium (IPDPS06), IEEE

  16. Karypis G, Kumar V (1999) Parallel multilevel k-way partitioning scheme for irregular graphs. SIAM Rev 41(2):278–300

    Article  MathSciNet  MATH  Google Scholar 

  17. Pellegrini F (2007) Scotch 5.0 user’s guide. Tech. rep., LaBRI

  18. Walshaw C, Cross M (2007) JOSTLE: parallel multilevel graph-partitioning software—an overview. In: Mesh partitioning techniques and domain decomposition techniques, pp 27–58

  19. Teresco J, Devine K, Flaherty J (2005) Partitioning and dynamic load balancing for the numerical solution of partial differential equations. In: Numerical solution of partial differential equations on parallel computers. Springer, Berlin

  20. Karypis G, Kumar V (1996) A parallel algorithm for multilevel graph partitioning and sparse matrix ordering. In: 10th intlernational parallel processing symposium, pp 314–319

  21. Karypis G, Kumar V (1998) Multilevel algorithms for multi-constraint graph partitioning. In: Proceedings of the 1998 ACM/IEEE conference on supercomputing, pp 1–13

  22. Womersley J (1955) Method for the calculation of velocity, rate of flow and viscous drag in arteries when the pressure gradient is known. J Physiol 127:553–563

    Google Scholar 

  23. Vignon-Clementel I, Figueroa C, Jansen K, Taylor C (2006) Outflow boundary conditions for three-dimensional finite element modeling of blood flow and pressure in arteries. Comput Methods Appl Mech Eng 195(29–32):3776–3796

    Article  MathSciNet  MATH  Google Scholar 

  24. Flaherty J, Loy R, Shephard M, Szymanski B, Teresco J, Ziantz L (1997) Predictive load balancing for parallel adaptive finite element computation. In: Proceedings of the international conference on parallel and distributed processing techniques and applications, PDPTA’97, vol 1, pp 460–469

  25. Oliker L, Biswas R, Strawn R (1996) Parallel implementation of an adaptive scheme for 3D unstructured grids on the SP2. In: Proceedings of the 3rd international workshop on parallel algorithm for irregularly structured problems

  26. Li X, Shephard M, Beall M (2005) 3D anisotropic mesh adaptation by mesh modification. Comput Methods Appl Mech Eng 194:4915–4950

    Article  MathSciNet  MATH  Google Scholar 

  27. Sahni O, Müller Y, Jansen K, Shephard M, Taylor C (2006) Efficient anisotropic adaptive discretization of the cardiovascular system. Comput Methods Appl Mech Eng 195:5634–5655

    Article  MATH  Google Scholar 

  28. Borouchaki H, George P, Hecht F, Laug P, Saltel E (1997) Delaunay mesh generation governed by metric specifications. part i. algorithms. Finite Elem Anal Des 25(1–2):61–83

    Article  MathSciNet  MATH  Google Scholar 

  29. Frey P, George PL, Hecht F (2008) Mesh generation: application to finite elements, 2nd edn. ISTE, London

    MATH  Google Scholar 

  30. Labbe P, Dompierre J, Vallet M-G, Guibault F, Trpanier J-Y (2004) A universal measure of the conformity of a mesh with respect to an anisotropic metric field. Int J Numer Methods Eng 61(16):2675–2695

    Article  MATH  Google Scholar 

  31. Li X (2003) Mesh modification procedures for general 3D non-manifold domains. Ph.D. thesis, Rensselaer Polytechnic Institute

  32. Pain C, Umpleby A, de Oliveira C, Goddard A (2001) Tetrahedron mesh optimization and adaptivity for steady-state and transient finite element calculations. Comput Methods Appl Mech Eng 190:3771–3796

    Article  MATH  Google Scholar 

Download references

Acknowledgments

We gratefully acknowledge the support of this work by NSF’s PetaApps program under grant OCI-0749152 and by the DOE Office of Science’s SciDAC/ITAPS program under grant DE-FC02-06ER25769. This research used computing resources provided by: (a) Rensselaer’s Computational Center for Nanotechnology Innovations that is funded by the State of New York, IBM and Rensselaer Polytechnic Institute, (b) Argonne Leadership Computing Facility at Argonne National Laboratory, which is supported by the Office of Science of the DOE under contracts DE-AC02-06CH11357, and (c) NSF through TeraGrid resources provided by National Institute for Computational Sciences (NICS); Kraken system was used. We would like to acknowledge that results presented in this article made use of software components provided by ACUSIM Software Inc. and Simmetrix Inc.

Author information

Authors and Affiliations

  1. Scientific Computation Research Center, Rensselaer Polytechnic Institute, Troy, NY, 12180, USA

    Min Zhou, Ting Xie, Seegyoung Seol, Mark S. Shephard & Onkar Sahni

  2. Department of Aerospace Engineering Sciences, University of Colorado at Boulder, 429 UCB, Boulder, CO, 80309, USA

    Kenneth E. Jansen

Authors
  1. Min Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ting Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Seegyoung Seol
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mark S. Shephard
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Onkar Sahni
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kenneth E. Jansen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mark S. Shephard.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zhou, M., Xie, T., Seol, S. et al. Tools to support mesh adaptation on massively parallel computers. Engineering with Computers 28, 287–301 (2012). https://doi.org/10.1007/s00366-011-0218-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00366-011-0218-x

Keywords

Search

Navigation