Coarse-grid treatment in parallel AMG for coupled systems in CFD applications
Highlights
► Implementation of agglomeration scheme for coupled systems. ► Benchmarks on different coarse-grid treatment techniques. ► Efficient method for coupled systems in fluid dynamics.
Introduction
The numerical kernel of fluid flow simulations is an algorithm for the solution of the Navier–Stokes equations. This is a non-linear system of coupled partial differential equations. The unknowns (which are functions of time and space in general) are the velocity field, the pressure, and eventually the temperature. With an appropriate discretisation technique the continuous problem is rendered a discrete one, i.e. the unknown functions are approximated by discrete values at certain points in space and time. It is possible to set up a single linear system for all unknown discrete values, but a suitable linearisation technique with an iteration is necessary since the discrete problem is still non-linear; this means that a large linear system has to be set up and solved several times in order to obtain a solution to the non-linear system. For CFD (computational fluid dynamics) software in the industrial practice these methods do not play a big role; an obvious reason for this is that, historically, the memory of the computers accessible to the users in their every-day professional practice was quite limited and also that the efficiency of the numerical techniques for the solution of the large linear systems was not sufficient. Moreover, in many simulations, the solution of the Navier–Stokes equations is only a part of the considerably larger complete physical model, e.g. if reactions of several dozens of species are simulated; solving linear systems for all the discrete unknowns at once would be beyond the possibilities of even modern computer systems.
Until recently the common practice in industrial CFD software which usually employs finite volume schemes for spacial discretisation was therefore to apply implicit segregated approaches such as SIMPLE, see Patankar and Spalding [1], or the fractional-step method, see Chorin [2]: for these approaches it is sufficient to setup and solve a linear system for each physical variable, e.g. velocity components, pressure, enthalpy, turbulence quantities, species concentrations, etc., separately. The size of the linear systems that are solved within these schemes corresponds to the number of finite volumes and the unknown discrete values are associated with a single physical unknown; we refer to such systems as scalar systems. Since perfect scalability of linear solutions on parallel computers is not easy to obtain, it is still a clear advantage that this practice reduces the number of unknowns of the linear system to the number of finite volumes. Moreover, matrix properties are more easily exploited, e.g. the matrix of the pressure-correction equation for both mentioned methods is symmetric positive definite in many cases. Linear solvers based on algebraic multigrid (AMG) are among the most efficient methods to solve the systems in the context of industrial software, see Emans [3]. The price to to be paid for this decoupling of the equations is usually a slower convergence of the outer algorithm, e.g. of SIMPLE.
Since parallel computers are frequently used in engineering practice, limited memory is no longer strictly prohibitive to the coupling of equations in practical applications; the employment of segregated approaches is therefore no longer an obligation. The demand for faster solution techniques has driven the development of robust algorithms with rapid convergence that are based on the coupling of equations which previously have been solved in a segregated manner. Recent examples are the methods proposed by Chen et al. [4] where the pressure and the velocity components are coupled, and SIMPLE-H by Emans et al. [5], where pressure and enthalpy are coupled. In both approaches not all unknowns of the simulation are coupled through the linear system, but those that are strongly linked by physics. We refer to these kind of linear systems as coupled systems. These linear systems are generally more difficult to deal with. They are nonsymmetric and methods to improve the condition, e.g. by strengthening the diagonal through an appropriate formulation of the underrelaxation, see Patankar [6], cannot be applied easily. The demand for efficient linear solvers for these techniques, however, is as urgent as that for the conventional segregated approaches. Due to their favourable scalability properties algebraic multigrid (AMG) methods are among the preferred techniques for this kind of system.
AMG techniques for similar linear problems, namely the discretisation of other coupled partial differential equations, have been developed in the past decade; examples are oil reservoir simulations, see Clees and Ganzer [7] and semiconductor device simulations, see Clees and Stüben [8]. A recursive formulation of multigrid algorithms relies on the coarse-grid correction, i.e. for a certain grid a coarser grid is employed to solve a defect equation, and the correction, projected back to the finer grid, is utilised to improve the solution on the initial grid. Theoretically this formulation can be applied until the coarser grid has only one point and its solution is obtained by only one floating-point operation. Practically it is recommendable to stop the recursive application of the multigrid algorithm at a sufficiently small grid where the system can be solved efficiently by an appropriate direct or iterative solver; this is often faster than the operations needed to perform the multigrid cycle. For parallel AMG implementations it is more difficult to find a sufficiently efficient linear solver: Since the solution of the small coarse-grid systems that are distributed to different parallel processors usually becomes quite inefficient due to communication overhead, this part of the program can be the reason for poor parallel efficiency of an AMG algorithm. For very large problems this is unlikely to pose any serious noticeable problem since the time spent on operations on the finest grid dominates the total computing time, but for smaller systems where it is already difficult to obtain good parallel efficiency by parallel AMG, the parallel solution of the coarse-grid problem can significantly contribute to the total computing time. In practical application of CFD software, problems of such a critical size are the rule rather than the exception. Nevertheless, it seems that the application of a parallel solver for the coarsest grid still reflects the level of the art since it is the method of choice in recent publications on parallel AMG, see e.g. Notay [9].
The coarse-grid treatment in multigrid algorithms, in particular for parallel schemes, is a technical problem which requires practical solutions. Trottenberg et al. [10] describe the principal options: either the distributed coarse-grid system is solved in parallel or, at some stage, the system is combined on one of the processors, eventually additional coarse-grids are constructed and system of the coarsest grid is solved directly by one processor. As far as we know, a systematic examination of the practical consequences of the choice of the coarse-grid treatment strategy has not been published yet; according to our own experience, the latter method, referred to as agglomeration in this paper, is appropriate for scalar systems, see Emans [3]. It could be demonstrated that AMG with agglomeration has good convergence properties and shows good parallel efficiency. However, the agglomeration scheme requires considerable additional effort for the implementation: It is an entirely new part of the algorithm that cannot be composed of existing code of a multigrid algorithm.
If an agglomeration scheme is applied to coupled systems, an additional difficulty appears: it is the necessity to deal with the association of the nodes to physical unknowns. We address this difficulty by a method that relies on an appropriate reordering of the components of the unknown vector. This particular agglomeration technique is integrated into an efficient implementation of an AMG scheme for coupled systems. The practical benefit of our approach will be demonstrated by means of CFD benchmarks where we compare the new method to existing techniques based on the parallel solution of the linear system associated with the coarsest grid. In order to preserve a certain amount of generality it will be shown that the observations apply equally to different coarsening schemes and to different cycling strategies. Although our benchmarks are taken from a specific problem of fluid dynamics, we do not see any reason that would prevent the conclusion to be transferred to any other application of AMG for linear systems arising from a discretisation of coupled partial differential equations.
Section snippets
Algorithm and implementation
Suppose we want to solve the linear problemon a parallel computer where is non-singular and sparse, is some right-hand side vector and the solution where n is the rank of A and thus the number of components of the unknown solution vector . These components are referred to as “nodes” in a large part of the literature on AMG; since we will use this term frequently in the following, we would like to remind that, in the context of coupled systems, it refers to the
Background and details of the simulations
Our benchmark cases are two short periods of an unsteady three-dimensional simulation of a full cycle of a four cylinder gasoline engine. The fluid dynamics is modelled by the Navier–Stokes equations (including the energy equation, in non-conservative formulation in terms of enthalpy) and a standard k–ɛ turbulence model. The material properties are those of air. The conservation laws are discretised on an unstructured finite volume mesh using a collocated variable arrangement. The method to
Conclusions
The efficient parallelisation of the agglomeration scheme for coupled system requires a reordering of the components in the unknown vector. The implementation of such a scheme is advantageous since its behaviour comes closer to that of the ideal multigrid behaviour than that of reasonable alternatives. In practical calculations with moderate number of unknowns the agglomeration scheme shows therefore a better performance than the iterative or direct coarse-grid solver. The application of an
Acknowledgement
Part of this work has been supported in the framework of “Industrielle Kompetenzzentren” by the Austrian Ministerium für Wirtschaft, Jugend und Familie and by the government of Upper Austria.
Maximilian Emans received his PhD from the Technische Universitt München in Computer Science. He is currently employed at the Johann Radon Institute for Computational and Applied Mathematics (RICAM) of the Austrian Academy of Sciences and at IMCC Linz as research engineer; his research interests are numerical methods for fluid dynamics and algebraic multigrid solvers for linear systems.
References (24)
- et al.
A calculation procedure for heat, mass and momentum transfer in three-dimensional parabolic flows
International Journal Heat Mass Transfer
(1972) - et al.
Hybrid scheduling for the parallel solution of linear systems
Parallel Computing
(2006) - et al.
Parallel multi-frontal solver for p adaptive finite element modeling of multi-physics computational problems
Journal of Computational Science
(2010) - et al.
Performance analysis of a hybrid MPI/OpenMP application on multi-core clusters
Journal of Computational Science
(2010) Performance parameters and benchmarking of supercomputers
Parallel Computing
(1991)Performance of parallel AMG-preconditioners on CFD-codes for weakly compressible flows
Parallel Computing
(2010)Numerical solution of the Navier–Stokes equations
Mathematics of Computation
(1968)Efficient parallel AMG methods for approximate solutions of linear systems in CFD applications
SIAM Journal on Scientific Computing
(2010)- et al.
A finite volume method of pressure-based coupled solver for incompressible/compressible flows
- M. Emans, S. Frolov, B. Lidskii, V. Posvyanskii, B. Basara, SIMPLE-H: a finite-volume pressure–enthalpy coupling...
Numerical Heat and Mass Transfer
An efficient algebraic multigrid solver strategy for adaptive implicit methods in oil reservoir simulation
SPE Journal
Cited by (7)
A massively parallel solver for discrete Poisson-like problems
2015, Journal of Computational PhysicsCitation Excerpt :This observation is the starting point of the enhancements presented in the next section. Any parallel multigrid or algebraic multigrid solver needs a carefully designed bottom level solver [11]. Typically, only few unknowns per process are left on the coarsest grid, making it difficult, if not impossible, to achieve good scalability for that part of the algorithm.
An adjoint-based scalable algorithm for time-parallel integration
2014, Journal of Computational ScienceCitation Excerpt :Gander and Vanderwalle [7] cast ‘Parareal’ as a multiple shooting method and a multigrid method. There have been attempts to parallelize different portions of the solution procedure in related problems in the areas of optimal control [8], parameter estimation [9], CFD applications [10]. The traditional ‘Parareal’ method obtains the fine solutions in parallel, but requires a serial coarse solution to be computed in every iteration.
Exciting times in computational science
2011, Journal of Computational ScienceScaling structured multigrid to 500K+ cores through coarse-grid redistribution
2018, SIAM Journal on Scientific ComputingExtreme-scale multigrid components within PETSc
2016, PASC 2016 - Proceedings of the Platform for Advanced Scientific Computing ConferenceAggregation algorithms for k-cycle AMG in computational fluid dynamics
2015, Progress in Computational Fluid Dynamics
Maximilian Emans received his PhD from the Technische Universitt München in Computer Science. He is currently employed at the Johann Radon Institute for Computational and Applied Mathematics (RICAM) of the Austrian Academy of Sciences and at IMCC Linz as research engineer; his research interests are numerical methods for fluid dynamics and algebraic multigrid solvers for linear systems.