Design of a scalable multiprocessor architecture and its simulation

Abstract

Performance enhancement and system scalability are two of the most important issues in the design of multiprocessor systems. A scalable cluster-based multiprocessor architecture and its simulation environment called SEECMA are proposed. Several new issues in our architecture, including scalable cache coherence protocols, relaxed memory consistency models, memory optimization techniques and several types of processors are considered. It has been developed to meet current trends in clustering architecture design. Additionally, the SEECMA environment is presented as a helpful investigation tool for both education and research. In addition to many simulation options, it is provided with a user-friendly graphic interface. SEECMA can automatically collect data from several simulation runs and display the results for comparison. So far, we have evaluated several vital issues of cluster-based multiprocessors on SEECMA including effective prefetching and replacement policies, and optimization of migratory sharing using both hardware and software mechanisms. On average, these enhance system performance by up to 8%, 9% and 7%, respectively. Our cluster-based multiprocessor architecture also scales more readily than the current general, or cluster-based, multiprocessor environments.

Introduction

The gap between the computing power of microprocessors and that of the largest supercomputers is shrinking, while the price/performance advantage of microprocessors is increasing. This clearly points to using microprocessors as the computation engines in large multiprocessor systems. The challenge lies in building a machine that can scale up its performance while maintaining the initial price/performance advantage of the individual processors. Scalability allows a parallel architecture to leverage commodity microprocessors and small-scale multiprocessors to build larger scale machines. These larger machines offer substantially higher performance, which provides the impetus for programmers to port their sequential applications to parallel architectures instead of waiting for the next higher performance uni-processor. Due to the rapid progress in VLSI and packaging technologies, they have become a driving force in the design and development of highly scalable parallel systems using cluster-based architecture, simultaneously exploiting local communication. Currently there are commercial supercomputers constructed with a cluster-based architecture, such as the CONVEX SPP series (CONVEX, 1994). Meanwhile, system designers must rely upon a convenient and accurate simulation environment to verify their designs or determine the most cost-effective strategies. Our cluster multiprocessor and its simulation environment, called simulation and evaluation environment for cluster-based multiprocessor architecture (SEECMA), are designed to address the above issues. SEECMA is the outcome of follow-up work on simulation and evaluation environment for shared-memory multiprocessor architecture (SEESMA) (Wu et al., 1998c).

Our architecture is designed for high performance and scalability. Our main purpose is to construct effective memory and interconnection architectures. After our evaluation, we found that our architecture has the following desirable attributes.

• A multithreaded processor architecture in a cluster-based system is possible, which is more effective when combined with our proposed mechanisms.

• Several memory consistency models (Wu, 1998a, Wu and Chen, 1998b) to improve the parallelism of the system were constructed, all resulting in improved system performance.

• We propose several new cache coherence protocols and directory structures in the linked-base type of cluster-based architecture, and these also perform better than existing protocols. We also demonstrate that some cache coherence protocols that perform well in non-cluster-based systems perform poorly in cluster-based systems.

• Several new prefetching and replacement policies are presented to improve the efficiency of the cache system, and simulation results prove these to be effective.

• With the assistance of these effective mechanisms, we find our architecture has high scalability when evaluating performance.

As SEECMA is extended from SEESMA (Wu et al., 1998c), the kernel simulator is the MINT package (Veenstra and Fowler, 1994), which was originally developed at the University of Rochester. SEECMA aims to provide a simulation and evaluation environment for cluster-based multiprocessor systems. It supports the following simulation functions:

• two types of processor architectures;

• two-level caches with write caches (WCs);

• a message-passing-based interconnection network;

• four types of memory consistency models;

• five types of cache coherence protocols;

• three types of cache coherence directory structures;

• effective replacement policies; and

• effective prefetching schemes.

SEECMA is equipped with versatile simulation options and users may customize the target memory architecture by clicking on the appropriate buttons in a user-friendly X-windows interface.

Users are provided with either a menu or graphic input environment through a Graphic User Interface (GUI). The menu-based interface is similar to conventional windows functions while the graph-based interface is more convenient for beginners. Users move the cursor around various architectural components and click on them to select the required simulation options. Each time a simulation option is located, the graph is updated. On-line help is also supported. Other than the above, SEECMA automatically collects numerical data from several simulation results, and displays the corresponding statistical graphs. So far, both bar charts and line charts are available. Using these graphs, users are able to compare the performance of different architectural parameters.

SEECMA, with its abundant simulation features, serves as a valuable research environment for system designers who are interested in cluster-based multiprocessor architecture. We have used SEECMA on many research issues related to memory subsystems, including cache coherence protocols, memory consistency models, interconnection networks, cache hierarchies, migratory sharing (Su et al., 1996), as well as prefetching and replacement policies (Pean et al., 1998a).

As a result of simulation, we found that our new mechanisms, developed in our cluster-based architecture, performed much better than current methods. We illustrate some of them in the following. Note that all the comparisons, including non-cluster system architecture and DASH (Lenoski and Laudon, 1989) system architecture, are made based on the simulations performed using SEECMA. Our inter-clustered cache prefetching mechanism performs, on average, about 8% better than the original non-inter-clustered cache system for benchmarks in the SPLASH benchmark suite (Singh et al., 1992; Woo et al., 1995). Our effective replacement policy performs, on average, about 9% better than the original replacement policy. We propose a migratory-sharing optimization mechanism; the total system performs 7% better, on average, than the system without our optimization. Other effective mechanisms also perform well, and we describe them in detail in the following sections.

The rest of the paper is organized as follows. Section 2 gives an overview of the overall system architecture. Section 3 describes the main system features and design issue considerations. An overview of the SEECMA simulation and evaluation environment is given in Section 4. Section 5 evaluates the performance of our architecture on SEECMA, and Section 6 gives our conclusions and outlines future work.