Heading for an Asynchronous Parallel Ocean Model

Abstract

The Swedish Meterological and Hydrological Institute (SMHI) makes daily forecasts of currents, temperature, salinity, water level, and ice conditions in the Baltic Sea. These forecasts are based on data from a High Resolution Operational Model of the Baltic Sea (HiROMB) currently calculated on one processor CRAY C90 vector computer and has to be ported to the distributed memory parallel CRAY T3E to save operation expenses and even more important allow a refinement of the grid resolution from currently 3 nautical miles (nm) to 1 nm within acceptable execution times.

The 3 nm grid covers the waters east of 6°E and includes the Skagerrak, Kattegat, Belt Sea and Baltic Sea. Boundary values at the open western border for flux, temperature, salinity, and ice properties are provided by a coarser 12 nm grid covering the whole North Sea and Baltic Sea region.

In a rectangular block covering the 3 nm grid only 25% of the surface points and less than 10% of the volume points are active, t.m. water points which have to be considered in the calculation. To avoid indirect addressing, the grid is decomposed into a set of smaller rectangular blocks discarding all blocks without water points. This reduces the fraction of inactive points significantly to the expense that one layer of ghost points is added around each block. The remaining blocks may be assigned to processors in a parallel environment. Introduction of a minimal depth for each block further reduces the number of inactive points.

Three different parts may be identified in the ocean model. In the baroclinic part water temperature and salinity are calculated for the whole sea including all depth levels. Explicit two-level time-stepping is used for horizontal diffusion and advection. Vertical exchange of momentum, salinity, and temperature are computed implicitly.

A semi-implicit scheme is used in the barotropic part for the vertically integrated flow, resulting in a system of linear equations (the Helmholtz equations) over the whole surface for water level changes. This system is sparse and non-symmetric, rejecting the 9-point stencil used to discretize the differential equations over the water surface. It is factorized with a direct solver once at the start of the simulation and then solved for a new right-hand side in each time step.

Ice dynamics includes ice formation and melting, changes in ice thickness and compactness, and is taking place on a very slow time scale. The equations are highly nonlinear and are solved with Newton iterations using a sequence of linearizations. A new equation system is factorized and solved in each iteration using a direct sparse solver. Convergence of the nonlinear system is achieved after at most a dozen iterations. The linear equation systems are typically small. In the mid winter season however, the time spent in ice dynamics calculations may dominate the whole computation time.