skip to main content
research-article

PUMI: Parallel Unstructured Mesh Infrastructure

Published: 10 May 2016 Publication History

Abstract

The Parallel Unstructured Mesh Infrastructure (PUMI) is designed to support the representation of, and operations on, unstructured meshes as needed for the execution of mesh-based simulations on massively parallel computers. In PUMI, the mesh representation is complete in the sense of being able to provide any adjacency of mesh entities of multiple topologies in O(1) time, and fully distributed to support relationships of mesh entities across multiple memory spaces in a manner consistent with supporting massively parallel simulation workflows. PUMI's mesh maintains links to the high-level model definition in terms of a model topology as produced by CAD systems, and is specifically designed to efficiently support evolving meshes as required for mesh generation and adaptation. To support the needs of parallel unstructured mesh simulations, PUMI also supports a specific set of services such as the migration of mesh entities between parts while maintaining the mesh adjacencies, maintaining read-only mesh entity copies from neighboring parts (ghosting), repartitioning parts as the mesh evolves, and dynamic mesh load balancing.
Here we present the overall design, software structures, example programs, and performance results. The effectiveness of PUMI is demonstrated by its applications to massively parallel adaptive simulation workflows.

Supplementary Material

a17-ibanez-app.pdf (ibanez.zip)
Supplemental movie, appendix, image and software files for, PUMI: Parallel Unstructured Mesh Infrastructure

References

[1]
ACIS Web. 2015. 3D ACIS Modeling. (2015). http://spatial.com/products/3d-acis-modeling.
[2]
Frédérik Alauzet, Xiangrong Li, E. Seegyoung Seol, and Mark S. Shephard. 2006. Parallel anisotropic 3D mesh adaptation by mesh modification. Engineering with Computers 21, 3 (Jan. 2006), 247--258.
[3]
Albany Web. 2015. Albany: a component-based partial differential equation code built on Trilinos. (2015). https://github.com/gahansen/Albany/wiki.
[4]
Micheal Amitay, Barton L. Smith, and Ari Glezer. 1998. Aerodynamic flow control using synthetic jet technology. In 36th AIAA Aerospace Sciences Meeting and Exhibit. American Institute of Aeronautics and Astronautics.
[5]
APF Web. 2015. APF: A Parallel Field Library. (2015). http://www.scorec.rpi.edu/apf.
[6]
ASCAC Web. 2015. ASCAC: Advanced Scientific Commuting Advisory Committee of U.S. Department of Energy. Retrieved from http://science.energy.gov/ascr/ascac.
[7]
Mark W. Beall. 1999. An Object-Oriented Framework for the Reliable Automated Solution of Problems in Mathematical Physics. Ph.D. Dissertation. Rensselaer Polytechnic Institute, Troy, NY.
[8]
Mark W. Beall and Mark S. Shephard. 1997. A general topology-based mesh data structure. International Journal of Numerical Methods in Engineering 40, 9 (May 1997), 1573--1596.
[9]
Mark W. Beall, Joe Walsh, and Mark S. Shephard. 2004. A comparison of techniques for geometry access related to mesh generation. Engineering with Computers 20, 3 (Sep. 2004), 210--221.
[10]
Erik G. Boman, Ümit V. Çatalyürek, Cédric Chevalier, and Karen D. Devine. 2012. The Zoltan and Isorropia parallel toolkits for combinatorial scientific computing: Partitioning, ordering, and coloring. Scientific Programming 20, 2 (2012), 129--150.
[11]
CCI Web. 2015. CCI: Center for Computational Innovations, Rensselaer Polytechnic Institute. Retrieved from http://cci.rpi.edu.
[12]
Waldemar Celes, Glaucio H. Paulino, and Rodrigo Espinha. 2005a. A compact adjacency-based topological data structure for finite element mesh representation. International Journal of Numerical Methods in Engineering 64, 11 (2005), 1529--1556.
[13]
Waldemar Celes, Glaucio H. Paulino, and Rodrigo Espinha. 2005b. A compact adjacency-based topological data structure for finite element mesh representation. International Journal of Numerical Methods in Engineering. 64, 11 (Nov. 2005), 1529--1556.
[14]
Kedar C. Chitale, Onkar Sahni, Mark S. Shephard, Saurabh Tendulkar, and Kenneth E. Jansen. 2014. Anisotropic adaptation for transonic flows with turbulent boundary layers. AIAA Journal 53, 2 (2014), 367--378.
[15]
Karen Devine, Erik Boman, Robert Heaphy, Bruce Hendrickson, and Courtenay Vaughan. 2002. Zoltan data management services for parallel dynamic applications. Computing in Science and Engineering 4, 2 (2002), 90--97.
[16]
Vladimir Dyedov, Navamita Ray, Daniel Einstein, Xiangmin Jiao, and Timothy J. Tautges. 2014. AHF: Array-based half-facet data structure for mixed-dimensional and non-manifold meshes. In Proceedings of the 22nd International Meshing Roundtable. Springer, 445--464.
[17]
H. Carter Edwards, Alan B. Williams, Gregory D. Sjaardema, David G. Baur, and William K. Cochran. 2010. SIERRA toolkit computational mesh conceptual model. Sandia National Laboratories SAND Series, SAND2010-1192 (2010).
[18]
FASTMath DOE SciDAC Web. 2015. FASTMath: Applied Mathematics Algorithms, Tools, and Software for HPC Applications. Retrieved from http://www.fastmath-scidac.org.
[19]
Joe E. Flaherty, Raymond M. Loy, Mark S. Shephard, Boleslaw K. Szymanski, James D. Teresco, and Louis H. Ziantz. 1997. Predictive load balancing for parallel adaptive finite element computation. In Proceedings of International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA'97), Xiangmin Jiao and Jean-Christophe Weill (Eds.), Vol. 1. 460--469.
[20]
Azat Yu Galimov, Onkar Sahni Jr., Richard T. Lahey, Mark S. Shephard, Donald A. Drew, and Kenneth E. Jansen. 2010. Parallel adaptive simulation of a plunging liquid jet. Acta Mathematica Scientia 30, 2 (Mar. 2010), 522--538.
[21]
Rao V. Garimella. 2002. Mesh data structure selection for mesh generation and FEA applications. International Journal of Numerical Methods in Engineering. 55 (2002), 451--478.
[22]
GeomSim Web. 2015. GeomSim: Direct Geometry Access. Retrieved from http://www.simmetrix.com/products/Simul-ationModelingSuite/geomsim/geomsim.html.
[23]
Ari Glezer and Micheal Amitay. 2002. Synthetic jets. Annual Review of Fluid Mechanics 34 (Jan. 2002), 503--529.
[24]
GRUMMP Web. 2015. Generation and Refinement of Unstructured, Mixed-Element Meshes. (2015). http://tetra.mech.ubc.ca/GRUMMP.
[25]
Glen Hansen and Steve Owen. 2008. Mesh generation technology for nuclear reactor simulation; barriers and opportunities. Journal of Nuclear Engineering and Design 238, 10 (Oct. 2008), 2590--2605.
[26]
Michael A. Heroux, Roscoe A. Bartlett, Vicki E. Howle, Robert J. Hoekstra, Jonathan J. Hu, Tamara G. Kolda, Richard B. Lehoucq, Kevin R. Long, Roger P. Pawlowski, Eric T. Phipps, Andrew G. Salinger, Heidi K. Thornquist, Ray S. Tuminaro, James M. Willenbring, Alan Williams, and Kendall S. Stanley. 2005. An overview of the Trilinos project. ACM Transaction on Mathematical Software (TOMS) - Special Issue on the Advanced ComputaTional Software (ACTS) Collection 31, 3 (Sep. 2005), 397--423.
[27]
Torsten Hoefler, Christian Siebert, and Andrew Lumsdaine. 2010. Scalable communication protocols for dynamic sparse data exchange. ACM Sigplan Notices 45, 5 (2010), 159--168.
[28]
Thomas J. R. Hughes, Luca Mazzei, and Kenneth E. Jansen. 2000. Large-eddy simulation and the variational multiscale method. Computing and Visualization in Science 3, 1--2 (May 2000), 47--59.
[29]
Daniel A. Ibanez, Ian Dunn, and Mark S. Shephard. 2016. Hybrid MPI-thread parallelization of adaptive mesh operations. Parallel Comput. 52 (feb. 2016), 133--143.
[30]
iMeshP Web. 2015. iMeshP: SciDAC ITAPS Parallel Mesh Interface. Retrieved from http://www.itaps.org/software/iMeshP_html.
[31]
ITAPS Web. 2015. ITAPS: Interoperable Technologies for Advanced Petascale Simulations of Department of Energy's Scientific Discovery through Advanced Computing (SciDAC). Retrieved from http://www.itaps.org.
[32]
Kenneth E. Jansen, Christian H. Whiting, and Gregory M. Hulbert. 2000. A generalized-α method for integrating the filtered Navier-Stokes equations with a stabilized finite element method. Computer Methods in Applied Mechanics and Engineering 190, 3--4 (Oct. 2000), 305--319.
[33]
Anil K. Karanam, Kenneth E. Jansen, and Christian H. Whiting. 2008. Geometry based pre-processor for parallel fluid dynamic simulations using a hierarchical basis. Engineering with Computers 24, 1 (Jan. 2008), 17--26.
[34]
Orion S. Lawlor, Sayantan Chakravorty, Terry L. Wilmarth, Nilesh Choudhury, Issac Dooley, Gengbin Zheng, and Laxmikant V. Kalé. 2006. Parfum: A parallel framework for unstructured meshes for scalable dynamic physics applications. Engineering with Computers 22, 3 (Dec. 2006), 215--235.
[35]
Xiangrong Li, Mark S. Shephard, and Mark W. Beall. 2005. 3D anisotropic mesh adaptation by mesh modification. Computer Methods in Applied Mechanics and Engineering 194, 48--49 (Nov. 2005), 4915--4950.
[36]
LLNL HPC Web. 2015. High Performance Computing Training, Lawrence Livermore National Laboratory. Retrieved from http://computing.llnl.gov/training.
[37]
Xiao-Juan Luo, Mark S. Shephard, Lie-Quan Lee, Lixin Ge, and Cho Ng. 2011. Moving curved mesh adaption for higher-order finite element simulations. Engineering with Computers 27 (2011), 41--50.
[38]
MeshAdapt Web. 2015. MeshAdapt: Parallel Unstructured Mesh Adaptation Library. Retrieved from http://www.scorec.rpi.edu/meshadapt.
[39]
MeshSim Web. 2015. MeshSim: Mesh Matching. Retrieved from http://www.simmetrix.com/products/SimulationMod-elingSuite/MeshSim/MeshMatching/MeshMatching.html.
[40]
NetCDF Web. 2015. NetCDF: Network Common Data Form. Retrieved from http://www.unidata.ucar.edu/software/netcdf.
[41]
Robert M. O'Bara, Mark W. Beall, and Mark S. Shephard. 2002. Attribute management system for engineering analysis. Engineering with Computers 18 (2002), 339--351.
[42]
Carl Ollivier-Gooch, Lori F. Diachin, Mark S. Shephard, Tim J. Tautges, Jason A. Kraftcheck, Vitus Leung, Xiaojuan Luo, and Mark Miller. 2010. An interoperable, data-structure-neutral component for mesh query and manipulation. ACM Transactions on Mathematical Software 37, 3 (Sep. 2010).
[43]
Aleksandr Ovcharenko, Kedar C. Chitale, Onkar Sahni, Kenneth E. Jansen, and Mark S. Shephard. 2013. Parallel adaptive boundary layer meshing for CFD analysis. In Proceedings of the 21st International Meshing Roundtable, Xiangmin Jiao and Jean-Christophe Weill (Eds.). Springer, Berlin, 437--455.
[44]
Malcolm J. Panthaki, Raikanta Sahu, and Walter H. Gerstle. 1997. An object-oriented virtual geometry interface. In Proceedings of the 6th International Meshing Roundtable. Springer, Berlin, 67--82. http://www.imr.sandia.gov/papers/abstracts/Pa54.html.
[45]
Parasolid Web. 2015. Parasolid: 3D Geometric Modeling Engine. (2015). http://www.plm.automation.siemens.com/en_us/products/open/parasolid.
[46]
PUMI Web. 2015. PUMI: Parallel Unstructured Mesh Infrastructure. (2015). http://www.scorec.rpi.edu/pumi.
[47]
Michel Rasquin, Cameron W. Smith, Kedar Chitale, E. Seegyoung Seol, Ben Matthews, J. Martin, Onkar Sahni, Raymond Loy, Mark S. Shephard, and Kenneth E. Jansen. 2015. Scalable fully implicit finite element flow solver with application to high-fidelity flow control simulations on a realistic wind design. Computing in Science and Engineering (2015). accepted.
[48]
Jean-François Remacle and Mark S. Shephard. 2003. An algorithm oriented mesh database. International Journal of Numerical Methods in Engineering 58, 2 (Sep. 2003), 349--374.
[49]
Onkar Sahni, Kenneth E. Jansen, Mark S. Shephard, Charles A. Taylor, and Mark W. Beall. 2008. Adaptive boundary layer meshing for viscous flow simulations. Engineering with Computers 24, 3 (Sep. 2008), 267--285.
[50]
Onkar Sahni, Kenneth E. Jansen, Charles A. Taylor, and Mark S. Shephard. 2009. Automated adaptive cardiovascular flow simulations. Engineering with Computers 25, 1 (2009), 25--36.
[51]
Onkar Sahni, Jens Müller, Kenneth E. Jansen, Mark S. Shephard, and Charles A. Taylor. 2006. Efficient anisotropic adaptive discretization of cardiovascular system. Computer Methods in Applied Mechanics and Engineering 195, 41--43 (Aug. 2006), 5634--5655.
[52]
Onkar Sahni, Joshua Wood, Kenneth E. Jansen, and Michael Amitay. 2011. Three-dimensional interactions between a finite-span synthetic jet and a crossflow. Journal of Fluid Mechanics 671 (2011), 254--287. http://dx.doi.org/10.1017/S0022112010005604
[53]
Kirk Schloegel, George Karypis, and Vipin Kumar. 2002. Parallel static and dynamic multi-constraint graph partitioning. Concurrency and Computation: Practice and Experience 14, 3 (Mar. 2002), 219--240.
[54]
Larry A. Schoof and Victor R. Yarberry. 1994. EXODUS II: A Finite Element Data Model. Technical Report SAND92-2137. Sandia National Laboratories, Albuquerque, NM 87158 and Livermore, CA 94550.
[55]
E. Seegyoung Seol. 2005. FMDB: Flexible Distributed Mesh Database for Parallel Automated Adaptive Analysis. Ph.D. Dissertation. Rensselaer Polytechnic Institute, Troy, NY.
[56]
E. Seegyoung Seol and Mark S. Shephard. 2006a. Efficient distributed mesh data structure for parallel automated adaptive analysis. Engineering with Computers 22, 3--4 (2006), 197--213.
[57]
E. Seegyoung Seol and Mark S. Shephard. 2006b. Efficient distributed mesh data structure for parallel automated adaptive analysis. Engineering with Computers 22, 3 (Dec. 2006), 197--213.
[58]
E. Seegyoung Seol, Cameron W. Smith, Daniel A. Ibanez, and Mark S. Shephard. 2012. A parallel unstructured mesh infrastructure. High Performance Computing, Networking, Storage and Analysis (SCC), 2012 SC Companion (Nov. 2012), 1124--1132.
[59]
Mark S. Shephard. 2000. Meshing environment for geometry-based analysis. International Journal of Numerical Methods in Engineering 47, 1--3 (Jan. 2000), 169--190.
[60]
Mark S. Shephard and E. Seegyoung Seol. 2009. Flexible distributed mesh data structure for parallel adaptive analysis. In Advanced Computational Infrastructures for Parallel and Distributed Applications, Manish Parashar and Xiaolin Li (Eds.). Wiley, 407--435.
[61]
Mark S. Shephard, Cameron Smith, E. Seegyoung Seol, and Onkar Sahni. 2013. Methods and tools for parallel anisotropic mesh adaptation and analysis. In VI International Conference on Adaptive Modeling and Simulation (ADMOS'13), J. P. Moitinho de Almeida, P. Díez, C. Tiago, and N. Parés (Eds.). The European Community in Computational Methods in Applied Sciencies (ECCOMAS), Lisbon, Portugal.
[62]
Simmetrix Web. 2015. Simmetrix: Simulation Modeling and Application Suite. Retrieved from http://www.simmetrix.com.
[63]
Tim J. Tautges. 2001. CGM: A geometry interface for mesh generation, analysis and other applications. Engineering with Computers 17, 3 (Oct. 2001), 299--314.
[64]
T. J. Tautges, R. Meyers, K. Merkley, C. Stimpson, and C. Ernst. 2004a. MOAB: A Mesh-Oriented Database. SAND2004-1592. Sandia National Laboratories. Report.
[65]
Tim J. Tautges, Ray Meyers, Karl Merkley, Clint Stimpson, and Corey Ernst. 2004b. MOAB: A Mesh-Oriented Database. Technical Report SAND2004-1592. Sandia National Laboratories, Albuquerque, NM 87158 and Livermore, CA 94550.
[66]
Andrés E. Tejada-Martínez and Kenneth E. Jansen. 2005. On the interaction between dynamic model dissipation and numerical dissipation due to streamline upwind/Petrov-Galerkin stabilization. Computer Methods in Applied Mechanics and Engineering 194, 9--11 (Mar. 2005), 1225--1248.
[67]
Andrés E. Tejada-Martínez and Kenneth E. Jansen. 2006. A parameter-free dynamic subgrid-scale model for large-eddy simulation. Computer Methods in Applied Mechanics and Engineering 195, 23 (Apr. 2006), 2919--2938.
[68]
Saurabh Tendulkar, Mark W. Beall, Mark S. Shephard, and Kenneth E. Jansen. 2011. Parallel mesh generation and adaptation for CAD geometries. In Proceedings of the NAFEMS World Congress. Boston, MA.
[69]
Trilinos Web. 2015. The Trilinos Project: Sandia National Laboratories. (2015). http://trilinos.sandia.gov.
[70]
VTK Web. 2015. VTK: Visualization Toolkit. (2015). http://www.vtk.org.
[71]
K. J. Weiler. 1988. The radial-edge structure: A topological representation for non-manifold geometric boundary representations. In Geometric Modeling for CAD Applications: Selected and Expanded Papers from the lfip Wg 5.2 Working Conference, M. J. Wozny, H. W. McLaughlin, and Jose L. Encarnacao (Eds.). Elsevier Science, 3--36.
[72]
Christian H. Whiting, Kenneth E. Jansen, and Saikat Dey. 2003. Hierarchical basis for stabilized finite element methods for compressible flows. Computer Methods in Applied Mechanics and Engineering 192, 47--48 (2003), 5167--5185. http://dx.doi.org/10.1016/j.cma.2003.07.011
[73]
Ting Xie, E. Seegyoung Seol, and Mark S. Shephard. 2014. Generic components for petascale adaptive unstructured mesh-based simulations. Engineering with Computers 30, 1 (Jan. 2014), 79--95.
[74]
Min Zhou, Onkar Sahni, Karen D. Devine, Mark S. Shephard, and Kenneth E. Jansen. 2010a. Controlling unstructured mesh partitions for massively parallel simulations. SIAM Journal on Scientific Computing 32, 6 (Nov. 2010), 3201--3227.
[75]
Min Zhou, Onkar Sahni, H. Jin Kim, C. Alberto Figueroa, Charles A. Taylor, Mark S. Shephard, and Kenneth E. Jansen. 2010b. Cardiovascular flow simulation at extreme scale. Computational Mechanics 46, 1 (2010), 71--82.
[76]
Min Zhou, Onkar Sahni, Ting Xie, Mark S. Shephard, and Kenneth E. Jansen. 2012a. Unstructured mesh partition improvement for implicit finite element at extreme scale. Journal of Supercomputing 59, 3 (Mar. 2012), 1218--1228.
[77]
Min Zhou, Ting Xie, E. Seegyoung Seol, Mark S. Shephard, Onkar Sahni, and Kenneth E. Jansen. 2012b. Tools to support mesh adaptation on massively parallel computers. Engineering with Computers 28, 3 (Jul. 2012), 287--301.

Cited By

View all
  • (2025)Development of novel collision detection algorithms for the estimation of fast ion losses in tokamak fusion deviceComputer Physics Communications10.1016/j.cpc.2024.109490309(109490)Online publication date: Apr-2025
  • (2024)TTK is Getting MPI-ReadyIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.339021930:8(5875-5892)Online publication date: 1-Aug-2024
  • (2024)Coherent mesh representation for parallel I/O of unstructured polyhedral meshesThe Journal of Supercomputing10.1007/s11227-024-06051-780:11(16112-16132)Online publication date: 1-Jul-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Mathematical Software
ACM Transactions on Mathematical Software  Volume 42, Issue 3
June 2016
208 pages
ISSN:0098-3500
EISSN:1557-7295
DOI:10.1145/2935754
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 May 2016
Accepted: 01 August 2015
Revised: 01 April 2015
Received: 01 March 2014
Published in TOMS Volume 42, Issue 3

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Unstructured mesh
  2. hybrid MPI/thread
  3. massively parallel
  4. partial differential equation simulation

Qualifiers

  • Research-article
  • Research
  • Refereed

Funding Sources

  • Department of Energy (DOE) Office of Science's Scientific Discovery through Advanced Computing (SciDAC) Institute as part of the Frameworks, Algorithms, and Scalable Technologies for Mathematics (FASTMath)

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)40
  • Downloads (Last 6 weeks)3
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Development of novel collision detection algorithms for the estimation of fast ion losses in tokamak fusion deviceComputer Physics Communications10.1016/j.cpc.2024.109490309(109490)Online publication date: Apr-2025
  • (2024)TTK is Getting MPI-ReadyIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.339021930:8(5875-5892)Online publication date: 1-Aug-2024
  • (2024)Coherent mesh representation for parallel I/O of unstructured polyhedral meshesThe Journal of Supercomputing10.1007/s11227-024-06051-780:11(16112-16132)Online publication date: 1-Jul-2024
  • (2024)A physics preserving neural network based approach for constitutive modeling of isotropic fibrous materialsEngineering with Computers10.1007/s00366-024-02095-8Online publication date: 18-Dec-2024
  • (2024)Speculative anisotropic mesh adaptation on shared memory for CFD applicationsEngineering with Computers10.1007/s00366-024-01994-040:6(3801-3827)Online publication date: 1-Dec-2024
  • (2024)Unstructured mesh tools for magnetically confined fusion system simulationsEngineering with Computers10.1007/s00366-024-01976-240:5(3319-3336)Online publication date: 28-Apr-2024
  • (2024)A new open-source framework for multiscale modeling of fibrous materials on heterogeneous supercomputersEngineering with Computers10.1007/s00366-023-01934-440:3(1337-1355)Online publication date: 1-Jun-2024
  • (2023)The Potential of Deep Learning to Advance Clinical Applications of Computational BiomechanicsBioengineering10.3390/bioengineering1009106610:9(1066)Online publication date: 9-Sep-2023
  • (2023)GTMesh: A Highly Efficient C++ Template Library for Numerical Schemes on General Topology MeshesApplied Sciences10.3390/app1315874813:15(8748)Online publication date: 28-Jul-2023
  • (2023) A Posterior h/p Adaptive Refinement Algorithm for 3-D Goal-Oriented Electromagnetic Problems IEEE Transactions on Microwave Theory and Techniques10.1109/TMTT.2023.325766171:10(4206-4217)Online publication date: Oct-2023
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media