TTPM – An efficient deadlock-free algorithm for multicast communication in 2D torus networks

Abstract

A torus network has become increasingly important to multicomputer design because of its many features including scalability, low bandwidth and fixed degree of nodes. A multicast communication is a significant operation in multicomputer systems and can be used to support several other collective communication operations. This paper presents an efficient algorithm, TTPM, to find a deadlock-free multicast wormhole routing in two-dimensional torus parallel machines. The introduced algorithm is designed such that messages can be sent to any number of destinations within two start-up communication phases; hence the name Torus Two Phase Multicast (TTPM) algorithm. An efficient routing function is developed and used as a basis for the introduced algorithm. Also, TTPM allows some intermediate nodes that are not in the destination set to perform multicast functions. This feature allows flexibility in multicast path selection and therefore improves the performance. Performance results of a simulation study on torus networks are discussed to compare TTPM algorithm with a previous algorithm.

Introduction

In high performance multicomputers, processors communicate between each other by passing messages via the communication network. Torus networks are widely used in high performance parallel computing systems and have been implemented in many research and commercial multicomputer systems such as Cray T3D, the Torus Routing Chip [1], and AP1000 [2]. They over useful edge connectivety and they can be partitioned into meshes. The two dimension (2D) torus topology is important in multicomputer design because of its many properties including constant node degree, constant length channel wires, high channel bandwidth, and low contention latency. In torus network topologies, wraparound channels have been added to connect each edge node to the corresponding node on the opposite edge. Under random traffic, the symmetry of torus networks leads to a more balanced utilization of communication channels than in mesh networks. With wormhole switching techniques and lower dimensional networks, performance analysis of direct networks offers improved latency and throughput results for the same network bandwidth [3].

The performance of multicast communication is measured in terms of its latency in delivering a message to all destinations. In wormhole-routed networks, the communication latency consists of two parts, start-up latency and network latency. The start-up latency is the time required to start a message, which involves operation system overheads. The network latency is a combination of propagation delay, router delay, and contention delay. The contention latency is computed for all delays associated with contention for routing resources among the various worms in the network. In multicast communication where an operation might consist of several communication phases, the start-up latency has a significant impact on the performance [4]. Deadlock in the interconnection network occurs when a set of messages is blocked forever because each message in the set holds one or more resources needed by another message in this set [5]. Collective communication, which involves a group of intercommunicating nodes, is useful in several applications. Some parallel programming languages provide efficient support for these applications. The importance of collective communication is further demonstrated by their inclusion in the message passing interface (MPI) standards [6]. An important primitive among collective communication operations is multicast communication, which is defined as sending a single message from a source node to a set of destination nodes.

The multicast communication can be used as a basis for many collective operations, such as barrier synchronization and global reduction, as well as cache invalidation in shared-memory multiprocessor systems [7]. The design and performance of multicast communication depends on several characteristics of the network architecture, including the switching strategy and network topology. The four main switching techniques are circuit switching, store-and-forward, virtual-cut-through, and wormhole switching [8]. The popular switching technique is wormhole routing in which a message is decomposed into a number of packets, which are further subdivided into flits. Flits are pipelined among several nodes by way of routers at the nodes [9]. A flit is the smallest unit of information that a channel can accept or refuse. The flit length is often affected by the network size. A 256-node network requires 8 bits per flit [5], [8]. Routing algorithms have been developed for wormhole communications networks [8], [10].

The basic node architecture of a 2D torus is illustrated in Fig. 1. The router provides communication services to the host processor. The incoming/outgoing internal channels to/from the router are usually referred as injection/consumption channels. The number of injection/consumption channels implies the number of messages that can be sent/received concurrently by the processor. To reduce the latency, multicast communication needs to be supported by the hardware. Hardware multicasting requires some additional functionality implemented along with the abilities associated with unicast communication. The communication services as forward and absorb, etc., which are required in wormhole-routed networks for multicasting are defined in the next Section 3.2. The importance of hardware supported multicasting was reported by Ni [11]. Also, in order to reduce the complexity of the communication hardware, many wormhole-routed systems support a so-called one-port communications architecture, in which each node can access the network through a single communication channel. One-port architectures occur frequently in massively parallel computers. Most wormhole-routed, torus networks use one-port architectures [6], [12].

An n × m 2D torus, denoted by T_n×m, is a 2D torus network computer system, where n, m represent the numbers of columns, and rows respectively. For each node p_i = (x_i, y_i) ∈ T_n×m, 0 ⩽ x_i ⩽ n − 1, 0 ⩽ y_i ⩽ m − 1, and deg [p_i] = 4 is the degree of node p_i. The absorb and the forward capability introduces additional resource dependencies that can lead to deadlock situations. The upper bound of the number of consumption channels required to avoid such deadlocks is equal to nv where n is the network dimension ofT_n×n and v is the number of virtual channels per direction [6].

In this paper, torus networks with bi-directional channels are used. The neighbor nodes in a bi-directional torus may be connected by either single, bi-directional physical channels, or by pairs of unidirectional physical channels in opposite directions. For simplicity, the torus networks will be drawn without channels.

This paper is organized as follows. The next section summarizes related research. The types of multicast phases and some properties of the introduced algorithm are discussed in Section 3. A deterministic TTPM algorithm based on the dimension order routings is reported in Sections 4 Torus Two Phase Multicast (TTPM) algorithm, 5 Performance evaluation. Finally, results and discussion are given in Section 6.