Parallel SSOR preconditioning implemented on dynamic SMP clusters with communication on the fly

Abstract

The paper presents a comparative analysis of parallel implementation of the preconditioned conjugate gradient method with the symmetric-successive over-relaxation preconditioner. Two parallel implementations of the matrix solver are compared. The first one is a message-passing version executed on a cluster of workstations. The other one is an efficient version simulated on a novel architecture of dynamically reconfigurable shared memory clusters with a new paradigm of inter-processor communication called communication on the fly. The presented example shows high suitability of the proposed architecture for fine grain numerical computations. It can be very useful in the simulation of physical phenomena described as numerical problems suitable for fine grain parallel execution.

Introduction

The finite element (FE) technique is one of the most popular approximated methods in scientific computations and computer aided engineering software. Different kinds and mixed formulations of the FE method are especially useful in computational electromagnetics (CEM) [1], [2]. The method replaces the original continuous problem, described by the partial differential equations, by some large linear or nonlinear matrix equations. An efficient formulation of the matrix subroutine is extremely important in the time domain electromagnetic computations (TD-CEM). The computation of transient states requires step-by-step integration of the algebraic matrix equation

$${\mathrm{Ae}}_{\tau }=b_{\tau }$$

derived from Maxwell’s equations [1], where $e_{\tau }$ denotes a computed distribution of time-dependent electromagnetic field. A wide spectrum of electromagnetic phenomena and material properties results in a large diversity of the form of the matrix $A$. The properties of the matrix $A$ depend on the type and order of the implemented elements, quality of the mesh, types of boundary conditions, and some others factors. The efficient solution of the CEM problems requires some smart solver. It should be flexible and portable to different problem formulations as well as work on different hardware platforms.

A preconditioned conjugate gradient algorithm (PCG) is one of the most suitable methods in CEM [3], [4]. The ill conditioning of the matrix $A$ results in a deterioration of the convergence of the CG solver. The performance of computations can be improved by an appropriate and efficiently implemented preconditioner

$${\mathrm{Ae}}_{\tau }=b\stackrel{\mathrm{preconditioning}}{⟶}{\mathrm{MAe}}_{\tau }=\mathrm{Mb}\text{,}$$

where $M$ is a preconditioner matrix. The selection of the proper solver and/or preconditioner is not a trivial task. A useful definition of the proper preconditioner (sequential and parallel) implies absolute minimization of time of computation of the matrix, see Eq. (2). In this way the improvement of convergence outweighs the computational cost of the preconditioning. There are no simple rules for the selection of a preconditioner, therefore an expert system seems to be the best tool. Some features of the computed problems (e.g. structure of the matrix $A$, its norms, conditioning number and eigenvalues) can be analyzed using either some artificial intelligence methods or heuristic algorithms. This kind of an environment and expert system for distributed platforms is developed within the SANS project (Self-Adapting Numerical Software) [5]. It can be used as a user friendly, transparent tool for the selection of the proper solver. The significant part of the project is a database, where some information about solvers as well as some features of the problems can be collected.

The next question is the efficient implementation of the solver [6], [7], [8]. Since the data and the tasks are partitioned in the parallel implementation, some specific properties of the applied hardware platform must be taken into account. The optimization of distributed preconditioning algorithms is aimed at decreasing communication cost as well as improving convergence of iterative calculation. Besides some universal methods (derived from sequential versions) [7], [9], the parallel formulations are developed with some peculiar techniques [4], [5], [10], [11], [12], [13]:

• Optimization of partitioning scheme to obtain the desired locality of data and ideal parallelization of matrix processing. The partitioning techniques connected with the finite element method can be based on the geometry of the computed model and some physical constraints (e.g. boundary conditions in the model).

• Implementation of a partitioning paradigm and formulation of some multilevel methods.

• Optimization of parallel processing of sparse matrices using some direct algebraic transformations of the assembled matrices, e.g. some block and block-striped mappings, some red–black and multi-coloring schemes.

• Modification of the structure of tasks and communication pattern in the algorithm, e.g. overlapping and optimization of relations between the local/sequential data processing and data transfers, implementation of non-blocking data transfers, optimization of load balancing.

• Implementation and adjustment of some heuristic modifications of the complete matrix equation (e.g. a localized threshold value-based methods, lumping and collocation of the matrix).

The parallel formulation of the symmetric-successive over-relaxation (SSOR) preconditioner is discussed in this paper. It is based on the complete form of the algorithm and no methods of the matrix reordering or problem modification are implemented. This form of the algorithm is particularly hard to parallelize, because the triangular matrix equation must be solved in each iteration [9], [10]. The distributed version of the SSOR looses some advantages of the sequential form. A two-step domain and task decomposition paradigms are used in the implemented SSOR subroutines. The applied two-dimensional matrix decomposition enables flexible modification of communication pattern, and tuning the relation between data transfers and sequential computations. The granulation of the algorithm can be flexibly adjusted even during execution. In this way, the performance of this stage can be adjusted to current available computational power and specific properties of the communication network. From the general point of view, this form of the algorithm can be applied on any hardware multicomputer/multiprocessor platform with distributed memory. The cluster of workstations (COW) platform as a common, popular testbed in practical real simulation, was first used to check the performance of the proposed algorithm [14]. The algorithm is used to approximate a time-dependent distribution of high-frequency electromagnetic wave in a benchmark model with a perfect conducting sphere suspended in a free space. Next, the algorithm was ported onto a system of dynamic shared memory multiprocessor clusters (SMP) with communication on the fly [15]. This runtime-configurable multiprocessor platform constitutes a new class of hardware. Its architecture should be particularly helpful in computation and data transfer intensive algorithms. The idea of this platform enables the overlapping data processing and data access commands. In this system, a new method for data exchange between processor clusters (communication on the fly) is applied. It is based on the combination of porting the data in caches of processors switched between clusters with the distribution of data in target clusters by means of simultaneous reads into many processor data caches by snooping of data exposed on the cluster shared memory bus. Such architecture can be used to build a dynamically configurable parallel embedded subsystem aimed in speeding up time consuming numerical computations, particularly oriented to fine grain numerical simulation. Specific features of the platform coincide with the presented structure of the SSOR preconditioner and the PCG algorithm. Simulation experiments with implementation of the distributed SSOR preconditioner have confirmed strong advantage of the proposed architecture over a classical cluster of workstations.