MCore: A non-hydrostatic atmospheric dynamical core utilizing high-order finite-volume methods
Introduction
In recent years, the exponential growth of computing power and trend towards massive parallelization of computing systems has had a profound influence on the atmospheric modeling community. Atmospheric cloud-resolving models are now pushing towards scales of only a few kilometers, meanwhile utilizing thousands to hundreds of thousands of processors. At these small scales many of the approximations that have been previously used in developing dynamical cores, such as the hydrostatic approximation, are no longer valid. As a consequence, there has been a trend towards developing atmospheric models which incorporate the full unapproximated hydrodynamic equations of motion. These developments have required a substantial paradigm shift in the way developers think about the algorithms and software behind geophysical models. Many design decisions that worked well in the past, including the use of the regular latitude-longitude grid and polar Fourier filtering, are no longer acceptable on large parallel systems since they either limit the choices for the parallel domain decompositions or necessitate additional parallel communication and thereby increase the computational overhead. Therefore, modifications must be made to accommodate this new generation of massively parallel hardware. As a consequence, the past ten years have seen substantial innovation in the modeling community as they push forward with efforts to determine the best candidates for the next-generation of atmospheric models.
Our focus in this paper is on non-hydrostatic modeling: That is, we are interested in models which treat the vertical velocity as a prognostic variable. In this case the vertical velocity has its own evolution equation and is not diagnosed from the other flow variables. Non-hydrostatic models are valid on essentially any horizontal scale and so can be used in cloud-resolving simulations. Several non-hydrostatic models are now in use, having been largely developed in the past ten years in response to growing availability of computing power. These include the UK Met Office’s Unified Model [6], [41], the Non-hydrostatic ICosahedral Atmospheric Model (NICAM), which was developed in Tomita and Satoh [44] and Satoh et al. [38] in cooperation with the Center for Climate System Research (CCSR, Japan), the NOAA Non-hydrostatic Icosahedral Model (NIM) [15] and the Ocean–Land–Atmosphere Model (OLAM) [48]. Recently, the Geophysical Fluid Dynamics Laboratory (GFDL) and the NASA Goddard Space Flight Center (GSFC) have also developed a non-hydrostatic dynamical core on the cubed-sphere [8], [30] based on the work of Putman and Lin [29]. These models all make use of some sort of conservative finite-difference or finite-volume formulation to ensure conservation of mass and most adopt the Arakawa C-grid staggering [2]. Other non-hydrostatic models include the Integrated Forecast System (IFS) [4], [52] which is a semi-Lagrangian spectral transform model used at the European Centre for Medium-Range Weather Forecasts (ECMWF), the Canadian semi-Lagrangian Global Environmental Multiscale (GEM) model [57], the Model for Prediction Across Scales (MPAS) [39] under development at the National Center for Atmospheric Research (NCAR) and Los Alamos National Laboratory, and the ‘ICOsahedral Non-hydrostatic’ (ICON) model [11], [49] which is a joint model by the Max-Planck Institute for Meteorology (MPI-M) and the German Weather Service.
In developing models for large-scale parallel computers, the choice of grid is of particular importance. Although non-hydrostatic dynamical cores have been developed on the regular latitude-latitude (RLL) grid, including the UK Met Office model, it is well known that the RLL grid suffers from the convergence of grid lines at the north and south poles. As a consequence, models using the RLL grid require the use of polar filters to remove instabilities associated with small grid elements, which can in turn severely degrade performance on parallel systems. Many recently developed hydrostatic and non-hydrostatic models have tended away from this grid, instead using quasi-uniform grids such as the icosahedral or cubed-sphere grids. Several hydrostatic models are now built on geodesic grids, including the icosahedral German Weather Service model GME [24], [25] and the icosahedral-hexagonal model of Ringler et al. [32]. Non-hydrostatic models that use the icosahedral grid include NICAM, NIM, OLAM and ICON. The icosahedral grid has been shown to perform well on large parallel systems and is among the most uniform options for spherical grids. Another choice of quasi-uniform grid is the cubed-sphere grid, which was originally developed by Sadourny [37] and revived by Ronchi et al. [34]. In fact, the work of Ronchi et al. [34] introduced a precursor to some of the techniques described in this paper, including the fourth-order collocated stencils and treatment of the cubed-sphere edges. The cubed-sphere was later used as the basis for a shallow-water model by Rančić et al. [31]. Since then, shallow-water models have been developed using the cubed-sphere grid that utilize finite-volume methods [35], [46], multi-moment finite-volume [5], the discontinuous Galerkin method [26] and the spectral element method [42]. The spectral element method was successfully extended to a full hydrostatic atmospheric model (the Spectral Element Atmosphere Model, SEAM) [9], [43], which is part of the High-Order Method Modeling Environment (HOMME). HOMME incorporates both the spectral element method and an experimental implementation of the discontinuous Galerkin method, and has proven to scale efficiently to hundreds of thousands of processors [7]. Recently HOMME has become an optional dynamical core in the Community Atmosphere Model version 5 (CAM5) [27] which is under development at the NCAR and Sandia National Laboratories. The GFDL/NASA finite-volume dynamical core has been modified to use a cubed-sphere grid [29], and has been demonstrated to also be very effective at high resolutions.
This paper continues a series that describes the development of an atmospheric model based on unstaggered high-order finite-volume methods. In Ullrich et al. [46] a shallow-water model utilizing cell-centered third-and fourth-order finite-volume methods was described. This approach was demonstrated to be robust and highly competitive with existing methods when tested against the shallow-water test cases of [55]. The high-order finite-volume method was later extended to non-hydrostatic simulations in 3D Cartesian geometry in Ullrich and Jablonowski [45]. Therein the authors demonstrated an accuracy-preserving technique for splitting horizontal and vertical motions using interleaved explicit and implicit time steps. The work of this paper is a combination of Ullrich et al. [46] and Ullrich and Jablonowski [45], and describes the high-order finite-volume formulation in spherical geometry. The fluid model that arises from this work has been named MCore. MCore is an atmospheric dynamical core that provides support for both the shallow-water equations and the full non-hydrostatic fluid equations. However, our emphasis in this paper will be on the non-hydrostatic dynamical core under the shallow-atmosphere approximation.
MCore makes use of a fully Eulerian cell-centered finite-volume formulation that has been proven to be robust for problems from other fields, including aerospace, computational biology and high-energy physics. It uses a fourth-order-accurate numerical method in the horizontal and a second-order accurate method in the vertical. However, it should be noted that although MCore is designed to be fourth-order-accurate in the horizontal, this design decision has been made largely to ensure that the model accurately captures the correct dispersive behavior of low-to mid-frequency horizontally propagating waves. We do not generally expect to obtain fourth-order convergence except in very idealized circumstances.
The outline of this paper is as follows. In Section 2 we introduce the cubed-sphere grid, which underlies the MCore model. The non-hydrostatic fluid equations under the shallow-atmosphere approximation are introduced in Section 3. The numerical approach underlying the MCore model is presented in Section 4. Numerical results and test cases are described in Section 5. Finally, our conclusions and future work are discussed in Section 6. A list of variables used in this paper can be found in Table 1. A list of the constants used in this paper and their corresponding values can be found in Table 2. The appendices describe the mathematical formulation of the primitive equations in cubed-sphere coordinates, under both the deep-and shallow-atmosphere approximation. Throughout this paper we will make use of Einstein summation notation, especially when describing geometric relations, under which summation is implied over repeated indices.
Section snippets
The cubed-sphere
The MCore model is implemented on a cubed-sphere grid, which can be imagined as the product of projecting a cube with regularly gridded faces onto the surface of a sphere. The cubed-sphere grid was originally suggested by Sadourny [37], but was not used in developing full geophysical codes until the work of Ronchi et al. [34]. There are several advantages to the cubed-sphere grid, such as grid regularity on each panel. Further, the cubed-sphere grid avoids the so-called “pole-problem,” which
The non-hydrostatic fluid equations in cubed-sphere coordinates
MCore utilizes the full non-hydrostatic fluid equations in terms of conserved variables density ρ, momentum ρu (with 3D velocity vector u = (uα, uβ, ur)) and potential temperature density ρθ (with potential temperature θ). Although this paper focuses on the shallow-atmosphere approximation to the equations of motion, MCore also implements the full equations of motion in a deep atmosphere. In the deep atmosphere, the equations of motion require a different treatment of the gravity and Coriolis
Numerical method
In this section we present the numerical methodology used to solve the non-hydrostatic equations of motion in a discrete context. MCore uses the method-of-lines to split the spatial and temporal components of the equations. Further, it splits the horizontal and vertical component of the fluid motion, solving for the former using a temporally explicit approach and the later using an implicit scheme. Coupling of these terms is managed via a Strang-carryover strategy, which ensures
Numerical results
Several test cases have been chosen to demonstrate the robustness and accuracy of MCore, including a baroclinic instabilty, 3D Rossby–Haurwitz wave, mountain-induced Rossby wave train, gravity wave test and Held–Suarez climatology. Rayleigh friction, as described in Section 4.10, is only used for the mountain-induced Rossby wave train test case. Most test runs make use of a rT = 30 km model top unless noted otherwise (such as in Section 5.4) and a vertical grid spacing which we have chosen to be
Conclusions and future work
In this paper we have developed a new atmospheric dynamical core which uses high-order finite-volume methods for solving the non-hydrostatic equations of motion under the shallow-atmosphere approximation. The model is built on a cubed-sphere grid with a height-based vertical coordinate. Under the upwind finite-volume methodology, a sub-grid-scale reconstruction is built within each element using neighboring element values. The reconstruction we propose is novel, incorporating geometric terms
Acknowledgments
Support for this work has been provided by the Office of Science, US Department of Energy, Award No. DE-SC0003990 and a University of Michigan Rackham Predoctoral Fellowship. We would also like to thank the two anonymous reviewers for their constructive comments in improving this work.
References (57)
- et al.
Computational design and the basic dynamical processes of the UCLA general circulation model
Methods Comput. Phys.
(1977) - et al.
A fourth-order accurate local refinement method for Poisson’s equation
J. Comput. Phys.
(2005) - et al.
Shallow water model on cubed-sphere by multi-moment finite volume method
J. Comput. Phys.
(2008) - et al.
On the use of a coordinate transformation for the solution of the Navier–Stokes equations
J. Comput. Phys.
(1975) A sequel to AUSM, Part II: AUSM+-up for all speeds
J. Comput. Phys.
(2006)- et al.
Finite-volume transport on various cubed-sphere grids
J. Comput. Phys.
(2007) Approximate Riemann solvers, parameter vectors, and difference schemes
J. Comput. Phys.
(1981)- et al.
The “cubed sphere”: a new method for the solution of partial differential equations in spherical geometry
J. Comput. Phys.
(1996) A wave propagation method for hyperbolic systems on the sphere
J. Comput. Phys.
(2006)- et al.
Nonhydrostatic icosahedral atmospheric model (NICAM) for global cloud resolving simulations
J. Comput. Phys.
(2008)