Coarse-grid treatment in parallel AMG for coupled systems in CFD applications

https://doi.org/10.1016/j.jocs.2011.08.001Get rights and content

Abstract

We present an agglomeration approach for the solution of the coarse-grid problems in algebraic multigrid for coupled systems. Our implementation relies on an appropriate reordering of the variables of the merged systems. A benchmark from fluid dynamics, representing the important class of mixed elliptic–hyperbolic problems, is used to demonstrate that the performance of the suggested agglomeration scheme comes much closer to the desired behaviour of the ideal multigrid than that of alternatives described in the literature.

Highlights

► Implementation of agglomeration scheme for coupled systems. ► Benchmarks on different coarse-grid treatment techniques. ► Efficient method for coupled systems in fluid dynamics.

Introduction

The numerical kernel of fluid flow simulations is an algorithm for the solution of the Navier–Stokes equations. This is a non-linear system of coupled partial differential equations. The unknowns (which are functions of time and space in general) are the velocity field, the pressure, and eventually the temperature. With an appropriate discretisation technique the continuous problem is rendered a discrete one, i.e. the unknown functions are approximated by discrete values at certain points in space and time. It is possible to set up a single linear system for all unknown discrete values, but a suitable linearisation technique with an iteration is necessary since the discrete problem is still non-linear; this means that a large linear system has to be set up and solved several times in order to obtain a solution to the non-linear system. For CFD (computational fluid dynamics) software in the industrial practice these methods do not play a big role; an obvious reason for this is that, historically, the memory of the computers accessible to the users in their every-day professional practice was quite limited and also that the efficiency of the numerical techniques for the solution of the large linear systems was not sufficient. Moreover, in many simulations, the solution of the Navier–Stokes equations is only a part of the considerably larger complete physical model, e.g. if reactions of several dozens of species are simulated; solving linear systems for all the discrete unknowns at once would be beyond the possibilities of even modern computer systems.

Until recently the common practice in industrial CFD software which usually employs finite volume schemes for spacial discretisation was therefore to apply implicit segregated approaches such as SIMPLE, see Patankar and Spalding [1], or the fractional-step method, see Chorin [2]: for these approaches it is sufficient to setup and solve a linear system for each physical variable, e.g. velocity components, pressure, enthalpy, turbulence quantities, species concentrations, etc., separately. The size of the linear systems that are solved within these schemes corresponds to the number of finite volumes and the unknown discrete values are associated with a single physical unknown; we refer to such systems as scalar systems. Since perfect scalability of linear solutions on parallel computers is not easy to obtain, it is still a clear advantage that this practice reduces the number of unknowns of the linear system to the number of finite volumes. Moreover, matrix properties are more easily exploited, e.g. the matrix of the pressure-correction equation for both mentioned methods is symmetric positive definite in many cases. Linear solvers based on algebraic multigrid (AMG) are among the most efficient methods to solve the systems in the context of industrial software, see Emans [3]. The price to to be paid for this decoupling of the equations is usually a slower convergence of the outer algorithm, e.g. of SIMPLE.

Since parallel computers are frequently used in engineering practice, limited memory is no longer strictly prohibitive to the coupling of equations in practical applications; the employment of segregated approaches is therefore no longer an obligation. The demand for faster solution techniques has driven the development of robust algorithms with rapid convergence that are based on the coupling of equations which previously have been solved in a segregated manner. Recent examples are the methods proposed by Chen et al. [4] where the pressure and the velocity components are coupled, and SIMPLE-H by Emans et al. [5], where pressure and enthalpy are coupled. In both approaches not all unknowns of the simulation are coupled through the linear system, but those that are strongly linked by physics. We refer to these kind of linear systems as coupled systems. These linear systems are generally more difficult to deal with. They are nonsymmetric and methods to improve the condition, e.g. by strengthening the diagonal through an appropriate formulation of the underrelaxation, see Patankar [6], cannot be applied easily. The demand for efficient linear solvers for these techniques, however, is as urgent as that for the conventional segregated approaches. Due to their favourable scalability properties algebraic multigrid (AMG) methods are among the preferred techniques for this kind of system.

AMG techniques for similar linear problems, namely the discretisation of other coupled partial differential equations, have been developed in the past decade; examples are oil reservoir simulations, see Clees and Ganzer [7] and semiconductor device simulations, see Clees and Stüben [8]. A recursive formulation of multigrid algorithms relies on the coarse-grid correction, i.e. for a certain grid a coarser grid is employed to solve a defect equation, and the correction, projected back to the finer grid, is utilised to improve the solution on the initial grid. Theoretically this formulation can be applied until the coarser grid has only one point and its solution is obtained by only one floating-point operation. Practically it is recommendable to stop the recursive application of the multigrid algorithm at a sufficiently small grid where the system can be solved efficiently by an appropriate direct or iterative solver; this is often faster than the operations needed to perform the multigrid cycle. For parallel AMG implementations it is more difficult to find a sufficiently efficient linear solver: Since the solution of the small coarse-grid systems that are distributed to different parallel processors usually becomes quite inefficient due to communication overhead, this part of the program can be the reason for poor parallel efficiency of an AMG algorithm. For very large problems this is unlikely to pose any serious noticeable problem since the time spent on operations on the finest grid dominates the total computing time, but for smaller systems where it is already difficult to obtain good parallel efficiency by parallel AMG, the parallel solution of the coarse-grid problem can significantly contribute to the total computing time. In practical application of CFD software, problems of such a critical size are the rule rather than the exception. Nevertheless, it seems that the application of a parallel solver for the coarsest grid still reflects the level of the art since it is the method of choice in recent publications on parallel AMG, see e.g. Notay [9].

The coarse-grid treatment in multigrid algorithms, in particular for parallel schemes, is a technical problem which requires practical solutions. Trottenberg et al. [10] describe the principal options: either the distributed coarse-grid system is solved in parallel or, at some stage, the system is combined on one of the processors, eventually additional coarse-grids are constructed and system of the coarsest grid is solved directly by one processor. As far as we know, a systematic examination of the practical consequences of the choice of the coarse-grid treatment strategy has not been published yet; according to our own experience, the latter method, referred to as agglomeration in this paper, is appropriate for scalar systems, see Emans [3]. It could be demonstrated that AMG with agglomeration has good convergence properties and shows good parallel efficiency. However, the agglomeration scheme requires considerable additional effort for the implementation: It is an entirely new part of the algorithm that cannot be composed of existing code of a multigrid algorithm.

If an agglomeration scheme is applied to coupled systems, an additional difficulty appears: it is the necessity to deal with the association of the nodes to physical unknowns. We address this difficulty by a method that relies on an appropriate reordering of the components of the unknown vector. This particular agglomeration technique is integrated into an efficient implementation of an AMG scheme for coupled systems. The practical benefit of our approach will be demonstrated by means of CFD benchmarks where we compare the new method to existing techniques based on the parallel solution of the linear system associated with the coarsest grid. In order to preserve a certain amount of generality it will be shown that the observations apply equally to different coarsening schemes and to different cycling strategies. Although our benchmarks are taken from a specific problem of fluid dynamics, we do not see any reason that would prevent the conclusion to be transferred to any other application of AMG for linear systems arising from a discretisation of coupled partial differential equations.

Section snippets

Algorithm and implementation

Suppose we want to solve the linear problemAx=bon a parallel computer where ARn×n is non-singular and sparse, bRn is some right-hand side vector and xRn the solution where n is the rank of A and thus the number of components of the unknown solution vector x. These components are referred to as “nodes” in a large part of the literature on AMG; since we will use this term frequently in the following, we would like to remind that, in the context of coupled systems, it refers to the

Background and details of the simulations

Our benchmark cases are two short periods of an unsteady three-dimensional simulation of a full cycle of a four cylinder gasoline engine. The fluid dynamics is modelled by the Navier–Stokes equations (including the energy equation, in non-conservative formulation in terms of enthalpy) and a standard kɛ turbulence model. The material properties are those of air. The conservation laws are discretised on an unstructured finite volume mesh using a collocated variable arrangement. The method to

Conclusions

The efficient parallelisation of the agglomeration scheme for coupled system requires a reordering of the components in the unknown vector. The implementation of such a scheme is advantageous since its behaviour comes closer to that of the ideal multigrid behaviour than that of reasonable alternatives. In practical calculations with moderate number of unknowns the agglomeration scheme shows therefore a better performance than the iterative or direct coarse-grid solver. The application of an

Acknowledgement

Part of this work has been supported in the framework of “Industrielle Kompetenzzentren” by the Austrian Ministerium für Wirtschaft, Jugend und Familie and by the government of Upper Austria.

Maximilian Emans received his PhD from the Technische Universitt München in Computer Science. He is currently employed at the Johann Radon Institute for Computational and Applied Mathematics (RICAM) of the Austrian Academy of Sciences and at IMCC Linz as research engineer; his research interests are numerical methods for fluid dynamics and algebraic multigrid solvers for linear systems.

References (24)

  • S. Patankar

    Numerical Heat and Mass Transfer

    (1980)
  • T. Clees et al.

    An efficient algebraic multigrid solver strategy for adaptive implicit methods in oil reservoir simulation

    SPE Journal

    (2007)
  • Cited by (7)

    • A massively parallel solver for discrete Poisson-like problems

      2015, Journal of Computational Physics
      Citation Excerpt :

      This observation is the starting point of the enhancements presented in the next section. Any parallel multigrid or algebraic multigrid solver needs a carefully designed bottom level solver [11]. Typically, only few unknowns per process are left on the coarsest grid, making it difficult, if not impossible, to achieve good scalability for that part of the algorithm.

    • An adjoint-based scalable algorithm for time-parallel integration

      2014, Journal of Computational Science
      Citation Excerpt :

      Gander and Vanderwalle [7] cast ‘Parareal’ as a multiple shooting method and a multigrid method. There have been attempts to parallelize different portions of the solution procedure in related problems in the areas of optimal control [8], parameter estimation [9], CFD applications [10]. The traditional ‘Parareal’ method obtains the fine solutions in parallel, but requires a serial coarse solution to be computed in every iteration.

    • Exciting times in computational science

      2011, Journal of Computational Science
    • Extreme-scale multigrid components within PETSc

      2016, PASC 2016 - Proceedings of the Platform for Advanced Scientific Computing Conference
    View all citing articles on Scopus

    Maximilian Emans received his PhD from the Technische Universitt München in Computer Science. He is currently employed at the Johann Radon Institute for Computational and Applied Mathematics (RICAM) of the Austrian Academy of Sciences and at IMCC Linz as research engineer; his research interests are numerical methods for fluid dynamics and algebraic multigrid solvers for linear systems.

    View full text