Distinguishing sharing types to minimize communication in software distributed shared memory systems

Abstract

Using thread migration to redistribute threads to processors is a common scheme for minimizing communication needed to maintain data consistency in software distributed shared memory (DSM) systems. In order to minimize data-consistency communication, the number of shared pages is used to identify the pair of threads that will cause the most communication. This pair of threads is then co-located on the same node. Thread pairs sharing a given page can be classified into thee types, i.e., read/read (r/r), read/write (r/w) and write/write (w/w). Based on memory-consistency protocol, these three types of sharing generate distinct amounts of data-consistency communication. Ignoring this factor will mispredict the amount of communication caused by cross-node sharing and generate wrong decisions in thread migration. This paper presents a new policy called distinguishing of types sharing (DOTS) for DSM systems. The basic concept of this policy is to classify sharing among threads as r/r, r/w or w/w, each with a different weight, and then evaluate communication cost in terms of these weights. Experiments show that considering sharing types is necessary for minimization of data-consistency communication in DSM. Using DOTS for thread mapping produces more communication reduction than considering only the number of shared pages.

Introduction

Software distributed shared memory (DSM) Li, 1988, Bershad and Zekauskas, 1993, Amza et al., 1996 provides an easy user interface for programmers to cluster personal computers or workstations in networks for massive computation. The threads of applications running on DSM communicate with each other via shared variables instead of message passing. Whenever the threads intend to access shared data, data consistency is transparently maintained by the DSM mechanism. This makes the programming on a distributed system easier since programmers do not need to put much effort into data transfer. However, program performance is not as good as systems employing explicit message passing. One of main reasons is that DSM usually generates more network traffic than message-passing systems due to the problem of data consistency (Lu et al., 1995). Accordingly, minimizing data-consistency communication is an important issue in DSM systems. One solution is the use of thread migration to co-locate threads that need data communication with each other on the same node.

Previous work (Thitikamol and Keleher, 1999a) used sharing degree, i.e., the number of shared pages, to predict the amount of communication caused by a pair of threads if they are located on different nodes. Then, they locate to the same node the pairs of threads that show the highest degree of mutual sharing, expecting thereby a maximum communication reduction. In addition, the communication cost of a mapping between threads and processors is estimated by cut cost, i.e., the total sharing degree of the pairs of threads on different nodes. Whether a new mapping is better for communication reduction than an old one is determined by comparing their cut costs.

However, thread pairs with the highest sharing degree located on different nodes do not necessarily create the most communication. The type of operation being performed on a shared page is also an important factor for determining the amount of created communication. Considering the possible operations that can be performed concurrently by two threads on one page yields three types: read/read (r/r), read/write (r/w) and write/write (w/w). Based on release memory consistency Gharachorloo et al., 1990, Keleher et al., 1995, a processor can delay making visible to other threads located on other processors any data modification by local threads until the next synchronization call takes effect. If a pair of threads located on distinct nodes r/r share a page, then their sharing need induce no data-consistency communication at synchronization points. For r/w sharing, an update needs to be sent from one node to another. For w/w sharing, updates need to be exchanged between both execution nodes. Obviously, the three types of sharing require different amounts of communication for shared page consistency. Ignoring this factor will mispredict the amount of communication caused by cross-node sharing and generate erroneous thread migration decisions.

Assuming the computational capability of each node and the computational needs of each thread are identical, load balance is achieved by assigning the same number of threads to each node, as in the example shown in Fig. 1. However, the thread mapping of this example is not optimal for minimizing communication cost. The system first tries to map t₂ and t₃ on the same node since the sharing degree of this thread pair is the highest, and then tries to locate t₁ and t₄ to another. The cut-cost of this new mapping is 6, which is less than the previous, so the system decides this mapping will reduce communication. In fact, however, the amount of data-consistency communication is not reduced by this thread mapping. A better mapping for this example is achieved by assigning t₁ and t₃ to one node and t₂ and t₄ to another. The main reason for the above mistaken decision is that the amount of communication caused by each type of sharing is regarded as equal. The correct thread migration decision requires consideration of the sharing type.

Based on the above discussion, we propose distinguishing of types sharing (DOTS), a policy that utilizes data sharing type when making decisions for minimizing data-consistency communication in DSM systems. The basic concept of DOTS is to classify sharing among threads as r/r, r/w or w/w, each with a different weight, and then evaluate communication cost in terms of these weights. DOTS was experimentally implemented in a DSM system called Cohesion. The fault handler of the system was extended to track the types of sharing among threads. Two new metrics, type-sharing degree and type-sharing cost, were used to identify maximum communication thread pairs, and to judge whether an alternative thread mapping would be better for communication reduction. A set of loop applications was tested on the system for performance evaluation, and will be discussed below. The experimental results confirm that distinguishing of sharing type is significant for communication minimization in DSM systems. Thus, DOTS thread mapping is more effective for reducing data-consistency communication than considering only the number of shared pages.

The rest of this paper is organized as follows. Section 2 is the overview of test bed. Section 3 is the analysis of data-consistency communication on Cohesion. The implementation and performance of DOTS on Cohesion are described in 4 Implementation, 5 Performance, respectively. Section 6 discusses previous work related to exploiting data sharing to minimize communication for software DSM systems. Finally, Section 7 presents conclusions and future work.