CPL library — A minimal framework for coupled particle and continuum simulation

Abstract

We present an open-source library for coupling particle codes, such as molecular dynamics (MD) or the discrete element method (DEM), and grid based computational fluid dynamics (CFD). The application is focused on domain decomposition coupling, where a particle and continuum software model different parts of a single simulation domain with information exchange. This focus allows a simple library to be developed, with core mapping and communication handled by just four functions. Emphasis is on scaling on supercomputers, a tested cross-language library, deployment with containers and well-documented simple examples. Building on this core, a template is provided to facilitate the user development of common features for coupling, such as averaging routines and functions to apply constraint forces. The interface code for LAMMPS and OpenFOAM is provided to both include molecular detail in a continuum solver and model fluids flowing through a granular system. Two novel development features are highlighted which will be useful in the development of the next generation of multi-scale software: (i) The division of coupled code into a smaller blocks with testing over a range of processor topologies. (ii) The use of coupled mocking to facilitate coverage of various parts of the code and allow rapid prototyping. These two features aim to help users develop coupled models in a test-driven manner and focus on the physics of the problem instead of just software development. All presented code is open-source with detailed documentation on the dedicated website (cpl-library.org) permitting useful aspects to be evaluated and adopted in other projects.

Program summary

Program Title: $\mathbf{CPL}\mathbf{LIBRARY}$

Program Files doi: http://dx.doi.org/10.17632/9dh8w97d2x.1

Licensing provisions: GPLv3

Programming languages: Fortran/C/C++/Python

External routines/libraries: Message Passing Interface (MPI)

Nature of problem: Coupling of particle and continuum software to enable simulations not possible with either code alone. In particular, handling communication and interaction of computational fluid dynamics (CFD) software with either molecular dynamics (MD) or discrete element method (DEM) solvers on high performance computing (HPC) platforms.

Solution method: A shared library with a minimal set of functions to enable coupling, together with an entire infrastructure to facilitate development of validated coupled software. This includes a minimal Python interface to encourage mock testing, libraries to help develop coupled tools along with pre-coupled examples including OpenFOAM, LAMMPS, Flowmol (Smith, 2014) and interactive plotting using wxPython and matplotlib.

Unusual features: Minimal interface with simple setup. A $CPL$_$Mocks$ framework to facilitate debugging and test driven development. Communication established between independent executables using MPI_Open_Port, reducing required changes to core source code. Coupled codes track the git repository with validation through continuous integration testing and deployment on DockerHub. All communication based on MPI_Cart and MPI_Graph so optimisation is possible through MPI implementation on supercomputers.

Introduction

Coupled simulations have the potential to revolutionise science. Consider the flow inside a chemical reactor, countless molecules driven by waves of fluid; the turbulent thermodynamics in a boiling vessel; the evaporation at the wall essential to the transfer of heat; the multi-fractal structure of soil soaked in fluid, driving and seeping until the interplay causes the failure of an embankment dam; the flow of blood, cells and viruses driven by a cascade of fluid, but fundamentally governed by details at the molecular scale. It is clear that such simulations are not possible with one modelling paradigm alone; however the combination of different models is often an overwhelmingly difficult problem.

The modelling framework proposed here aims to couple computational fluid dynamics (CFD) to a particle method, either molecular dynamics (MD) or the discrete element method (DEM). Coupling two simulation tools is a very challenging software development problem. CFD simulations require appropriate choice of boundary conditions and numerical scheme [1], with appropriate stability and monotony considerations [2]. MD and DEM solvers require appropriate integration and force lists [3], choice of inter-atomic potentials, building of initial configurations and careful equilibration [4]. Coupling brings in the implementation of constraint algorithms [5], [6], [7], appropriate averaging methods [8], [9], [10] as well as domain termination [11] and particle insertion considerations [12]. Furthermore, both techniques are non-linear, making the origin of problems in either of the two codes very tricky to identify. $\mathbf{CPL}\mathbf{LIBRARY}$ is a minimal framework to link two codes, with tools to enable the user to systematically test the software, allowing them to isolate and identify problems.

The two main application cases for $\mathbf{CPL}\mathbf{LIBRARY}$ are shown graphically in Fig. 1, molecular-continuum coupling labelled ($a$) and overlapping DEM and continuum coupling for granular flows labelled ($b$). $\mathbf{CPL}\mathbf{LIBRARY}$ originally focused on simulations where the domain is decomposed into two parts, as in Fig. 1 ($a$) a CFD and an MD region with an interface between them. This type of coupling is known as domain decomposition, an approach first applied by O’Connell [13] that has been widely adopted over the last two decades [6], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23]. Other names for this type of coupling used in the literature include geometrical coupling [24], concurrent coupling [25] or extreme scaling [26]. The two parts of the domains overlap in order to stagger the interface between them and provide a buffer to facilitate coupling. Domain decomposition coupling involves the direct exchange of mass, momentum, energy and other fields between a particle and continuum simulation, where each piece of software simulates part of the total domain [27]. Other coupling approaches are possible to link nano-scale molecular models to the macro-scale continuum, including the embedding of particle simulations into a continuum solver [28], [29] or running lower-level models to establish values for lookup tables or closure relations. These alternate approaches aim to remove elements of empiricism in higher level models, for example by calculating quantities in the MD which are then set as coefficients in the CFD, e.g. viscosity [30], [31], contact-line dynamics [32], surface tension [33] or heat flux [34]. These are outside the focus of $\mathbf{CPL}\mathbf{LIBRARY}$, which solely aims to facilitate the exchange of fields between continuum and particle simulations in the region in which they overlap.

The approach of exchanging fields is, however, applicable to granular mechanics simulations of the type shown in Fig. 1 ($b$), where the average of particle fields influences fluid flow features in the CFD and the resulting drag is simply applied as a force field to the particle DEM system. For this type of coupling, the theoretical foundations were set out in Anderson and Jackson (1967) [35] and is a commonly applied method for coupled fluid-granular systems with a large numbers of particles per CFD cell [36], [37], [38], [39], [40], [41], [42]. The basic premise can be considered a special case of domain-decomposition coupling in which the overlap between both codes is the whole domain. In this case, DEM and CFD model the particle and fluid parts of the granular simulation respectively.

Fig. 2 gives an overview of three types of existing coupling software. The first type of software is termed monolithic, which involves pre-packaging two pieces of software as a single application. This provides an advantage for the user as it simplifies the installation and allows a combined interface to both codes. However, a potential disadvantage is that both codes are adapted from their natural state to bring them together, so the versions are frozen at the point at which they were coupled and can no longer be used separately. Examples of monolithic codes typically occur in more applied applications, where the focus is on providing complete tools to solve engineering problems. For granular systems, open source examples include direct linking of LAMMPS-OpenFOAM [43], adaptations like SediFOAM [44] and mixed commercial open-source such as CFDEM [45].

In the second software type, libraries, all coupling is handled through shared libraries which facilitate information exchange between software. Libraries are lightweight and minimise the impact of coupling on either code, also making any exchange transparent through an explicit library call. The disadvantage is added complexity in compiling and the requirement to adapt new codes to include the libraries for each coupled application. $\mathbf{CPL}\mathbf{LIBRARY}$ falls into this category. Other examples include MaMiCo [46] for Lattice Boltzmann-MD coupling and MUI [25] which allows general coupling of non-uniform grids and particle–particle coupling such as linking dissipative particle dynamics (DPD) and MD [47]. (See Fig. 2, [48], [49].)

The Final classification considered are frameworks, codes which provide a series of tools and techniques as an environment for general coupling of two or more packages. Many of these models are summarised in reviews [55], [56], [57] so they will be discussed only briefly here. The MUSCLE framework is part of pan-European projects (MAPPER) for multi-scale simulation [51]. Similarly, preCICE is a very large collaborative project, focused on fluid–solid interaction with a very comprehensive set of functionality and pre-coupled codes [54]. OASIS3 [52] aims to link various fluid projects for climate modelling. MCT [53] is a Fortran project written to link codes while MPH library allows the user to run a series of executables [58]. These frameworks aim to provide a holistic solution for multi-scale modelling and, in the longer term, may represent all-purpose solutions. However, the complexity of these frameworks can make it difficult to learn and identify the most suitable software component needed for a particular task, such as the direct topological mapping and information exchange targeted in this work.

The focus of $\mathbf{CPL}\mathbf{LIBRARY}$ is direct exchange between a fluid solver and a particle solver, linking in as a lightweight library and providing direct information exchange through MPI. The key is to provide ease of implementation and testing, which in turn facilitates the use of $\mathbf{CPL}\mathbf{LIBRARY}$ to enable the coupling of complex software on supercomputers with minimal knowledge of their individual coding strategy and algorithmic deployment. This manuscript outlines the key features of $\mathbf{CPL}\mathbf{LIBRARY}$ in Section 2. Examples applications of $\mathbf{CPL}\mathbf{LIBRARY}$ are then given in Section 3 and the four functions required in order to implement coupling are outlined. A range of validation and verification examples as well as automated tests on continuous integration (CI) platforms are discussed in the results (Section 4), followed by scaling test on supercomputers. Finally, we summarise the software in Section 5.