Manual Parallelization Versus State-of-the-Art Parallelization Techniques

Abstract

Being multiprocessors (both on-chip and/or off-chip), modern computer systems can automatically exploit the benefits of parallel programs, but their resources remain underutilized in executing still-prevailing sequential applications. An obvious solution is in the parallelization of such applications. The first part overviews the broad issues in parallelization. Various parallelization approaches and contemporary software and hardware tools for extracting parallelism from sequential applications are studied. It also attempts to identify typical code patterns amenable for parallelization. The second part represents a case study where the SPEC CPU2006 suite is considered as a representative collection of typical sequential applications. Following that, it discusses the possibilities and potentials of automatic parallelization and vectorization of the sequential C++ applications from the CPU2006 suite. Since these potentials are generally limited, it explores the issues in manual parallelization of these applications. After previously identified patterns are applied by source-to-source code modifications, the effects of parallelization are evaluated by profiling and executing on two representative parallel machines. Finally, the presented results are carefully discussed.

Introduction

There is an everlasting strive for an increasing power and speed of computer systems. Parallel systems have long been considered as a promising solution for both throughput-oriented and speedup-oriented computing. Parallel processing has been used predominantly for demanding scientific applications and for large-scale systems over the years, but the architecture, technology, and application trends have been forcing it rapidly towards the commercial computing in medium-scale and even small-scale systems [1].

The importance of parallel processing has been boosted in the last decade by current trends in processor architecture and technology. Until recently, the performance of the processors grew steadily following Moore's law primarily as a consequence of an ever-increasing number of progressively faster transistors. However, inability to obtain further benefits of instruction-level parallelism (ILP), the problems with power dissipation, technology constraints, and design and verification difficulties in complex superscalars gave rise to chip multiprocessors (CMP) [2]. Since a CMP includes multiple simple superscalar cores on a chip, in addition to having the benefits of ILP, a multicore processor can issue multiple instructions per cycle from instruction streams (threads) [3]. Hence, parallel processing is brought further down to the laptop and embedded-system level. While parallelism in superscalars is extracted on the instruction level transparently to a software designer, an increasing level of parallelism in multicores (thread-level parallelism) requires the more intensive involvement of a programmer.

The first part of this chapter represents a survey on parallelization issues and approaches. It reviews some important issues in solving data dependences by loop transformations in order to improve the parallelization abilities. Then, a broad spectrum of parallelization approaches and contemporary tools is explored in order to provide the state-of-the-art in the field. A special attention is devoted to typical parallelization patterns found within sequential applications. Speedups obtained by applying these simple yet powerful patterns are significant, as we show later in our case study.

In a situation where all contemporary processors are multicores, the benefits of multithreaded, parallel workloads are easily exploited. However, the real challenge nowadays is to fully utilize the resources of multicore processors to improve the performance of a considerable amount of existing general-purpose sequential applications. Since the nature of such applications is best reflected in benchmark suites, the second part of this chapter is a case study on parallelization of the SPEC CPU2006 benchmark suite [4], [5] as one of the most representative and widely used suites for uniprocessors. It examines the potentials of autoparallelization and vectorization of the SPEC CPU2006 benchmarks in the state-of-the-art compilers and the efforts to parallelize its applications outside this suite. In order to achieve an additional level of parallelism, this study is oriented towards making manual source-to-source code modifications based on profiling information. To this end, the SPEC CPU2006 benchmarks are carefully examined for places where typical parallelization patterns can be efficiently applied. Finally, the resulting speedups are evaluated on two large parallel machines. The evaluation environment and methodology are also described in order to illustrate the details of entire process. Based on this experience, some general indications on where to find the parallelization potential are discussed.