The Structure of Parallel Algorithms

doi:10.1016/S0065-2458(08)60033-9

Advances in Computers

Volume 19, 1980, Pages 65-112

https://doi.org/10.1016/S0065-2458(08)60033-9 Get rights and content

Publisher Summary

This chapter presents many examples of parallel algorithms and studies them under a uniform framework. The chapter explains a parallel algorithm as a collection of independent task modules that can be executed in parallel and that communicate with each other during the execution of the algorithm. The chapter explains the three important attributes of a parallel algorithm and classifies parallel algorithms in terms of these attributes. Three orthogonal dimensions of the space of parallel algorithms: concurrency control, module granularity, and communication geometry. The classification of parallel algorithms corresponds naturally to that of parallel architectures. Algorithms for synchronous parallel computers are considered, where examples of algorithms using various communication geometries are presented. Algorithms for asynchronous parallel computers are also considered in the chapter. A number of techniques dealing with the difficulties arising from the asynchronous behavior of computation and the examples are mainly drawn from results in concurrent database systems. This chapter deals with the basic issues and techniques in designing parallel algorithms for various architectures. The chapter concludes that issues concerning algorithms for synchronous parallel computers are quite different from those for asynchronous parallel computers.

References (88)

E.A. Ashcroft
Proving assertions about parallel programs
J. Comput. Syst. Sci.
(1975)
D. Chazan et al.
Chaotic relaxation
Linear Algebra Appl.
(1969)
B. Samadi
B-trees in a system with multiple users
Inf. Process. Lett.
(1976)
A. Aho et al.
“The Design and Analysis of Computer Algorithms.”
(1975)
G.A. Anderson et al.
Computer interconnection structures: Taxonomy, characteristics, and examples
ACM Comput. Surv.
(1975)
G.H. Barnes et al.
The ILLIAC IV computer
IEEE Trans. Comput.
(1968)
K.E. Batcher
Sorting networks and their applications
1968 Spring Joint Comput. Conf.
(1968)
G.M. Baudet
Asynchronous iterative methods for multiprocessors
J. Assoc. Comput. Mach.
(1978)
G.M. Baudet
The design and analysis of algorithms for asynchronous multiprocessors
(1978)
G.M. Baudet et al.
Optimal sorting algorithms for parallel computers
IEEE Trans. Comput.
(1978)

G.M. Baudet et al.

“Parallel Execution of a Sequence of Tasks on an Asynchronous Multiprocessor,”

(1977)

R. Bayer et al.

Organization and maintenance of large ordered indexes

Acta Inf.

(1972)

R. Bayer et al.

Concurrency of operations on B-trees

Acta Inf.

(1977)

V.E. Benes

“Mathematical Theory of Connecting Networks and Telephone Traffic.”

(1965)

J.L. Bentley, H.T. Kung, (1979A tree machine for searching problems. Proc. 1979 Internat. Conf. Parallel Processing,...

P.A. Berstein et al.

A system of distributed databases (the fully redundant case)

IEEE Trans. Software Eng.

(1978)

R.P. Brent

The parallel evaluation of general arithmetic expressions

J. Assoc. Comput. Mach.

(1974)

R.P. Brent, H.T. Kung, (1980b) In...

R.P. Brent et al.

“The Area-Time Complexity of Binary Multiplication,”

(1980)

S. Browning

Algorithms for the tree machine

T.C. Chen

Overlap and pipeline processing

T.C. Chen, V.Y. Lum, C. Tung, (1978) The rebound sorter: An efficient sort engine for large files. Proc. 4th Internat....

D. Cohen

“Mathematical Approach to Computational Networks,”

(1978)

E.W. Dijkstra et al.

On-the-fly garbage collection: An exercise in cooperation

Commun. ACM

(1978)

P.H. Enslow

Multiprocessor organization: A survey

ACM Comput. Surv.

(1977)

K.P. Eswaran et al.

The notions of consistency and predicate locks in a database system

Commun. ACM

(1976)

R.W. Floyd, (1967) Assigning meanings to programs. Proc. Symp. Applied Mathematics, pp. 19–32. Am. Math. Soc.,...

M.J. Flynn

Very high-speed computing systems

Proc. IEEE

(1966)

M. Foster et al.

The design of VLSI special purpose chip: Example and opinions

Res. Computer

(1980)

S.H. Fuller et al.

“The Cm* Review Report,”

(1977)

J. Gray

Notes on data base operating systems

“Lecture Notes in Computer Science 60: Operating Systems,”

(1978)

L.J. Guibas et al.

Direct VLSI implementation of combinatorial algorithms

(1979)

T.G. Hallin et al.

Pipelining of arithmetic functions

IEEE Trans, on Comput.

(1972)

F.E. Heart, S.M. Ornstein, W.R. Crowther, W.B. Barker, (1973) A new minicomputer/multiprocessor for the ARPA network....

D. Heller

A survey of parallel algorithms in numerical linear algebra

SIAM Rev.

(1978)

A.K. Jones et al.

“Experience using multiprocessor systems: A status report,”

(1979)

A.K. Jones et al.

Programming issues raised by a multiprocessor

Proc. IEEE

(1978)

R.M. Karp

Reducibility among combinational problems

“Complexity of Computer Computations,”

(1972)

W.H. Kautz et al.

Cellular interconnection arrays

IEEE Trans. Comput.

(1968)

R.M. Keller

Formal verification of parallel programs

Commun. ACM

(1976)

D.E. Knuth

(1973)

S.R. Kosaraju

Speed of recognition of context-free languages by array automata

SIAM J. Comput.

(1975)

D.J. Kuck

ILLIAC IV software and application programming

IEEE Trans. Comput.

(1968)

D.J. Kuck

A survey of parallel machine organization and programming

ACM Comput. Surv.

(1977)

Cited by (135)

A parallel 3D unsteady incompressible flow solver on VPP700
2001, Parallel Computing
Citation Excerpt :
A lot of work is in progress in evolving efficient parallel algorithms to exploit the potential of modern computing trends. Heller [1], Kung [2], Ortega and Voigt [3], etc., have presented a survey of parallel numerical methods that can be used to solve a variety of engineering problems. Parallel computational strategies for the numerical solution of unsteady incompressible flow through Navier–Stokes (NS) equations are being developed by several researchers both on SIMD and MIMD machines.
In this study, MIMD parallel computation of coupled non-linear partial differential equations governing the mass and momentum conservation of a 3D viscous incompressible unsteady flow on distributed memory-based vector parallel processing system (VPP700) under UXP/VPPF90 has been discussed. The parallel numerical implementation is based on a time accurate cell centered finite volume method in conjunction with Roe's flux difference splitting of non-linear terms and pseudo-compressibility technique. Data flow graphs necessary for combined parallelization and vectorization have been generated to provide a logical basis for parallel code development. The performance of the parallel implementation measured in terms of speedup and efficiency factors is found to be good. The parallel code is tested on three test problems of practical significance.
A new efficient pipelined-systolic matrix-vector multiplication algorithm for a tree-structured machine
1997, Nonlinear Analysis, Theory, Methods and Applications
New computing systems and future high-performance computing environment and their impact on structural analysis and design
1997, Computers and Structures
Recent advances in computer technology that are likely to impact structural analysis and design are reviewed. A brief summary is given of the advances in microelectronics and networking technologies, and in the human-computer interaction paradigms and techniques. The major features of new and projected computing systems, including high-performance computers, parallel processing machines, and small systems are described. Advances in programming environment, numerical algorithms, and computational strategies for new computing systems are reviewed. The impact of the advances in computer technology on structural analysis and design is described. A scenario for future computing paradigm is presented and the near-term needs in the computational structures area are outlined.
A simulation-based scalability study of parallel systems
1994, Journal of Parallel and Distributed Computing
Scalability studies of parallel architectures have used scalar metrics to evaluate their performance. Very often, it is difficult to glean the sources of inefficiency resulting from the mismatch between the algorithmic and architectural requirements using such scalar metrics. Low-level performance studies of the hardware are also inadequate for predicting the scalability of the machine on real applications. We propose a top-down approach to scalability study that alleviates some of these problems. We characterize applications in terms of the frequently occurring kernels and their interaction with the architecture in terms of overheads in the parallel system. An overhead function is associated with each artifact of the parallel system that limits its scalability. We present a simulation platform called SPASM (Simulator for Parallel Architectural Scalability Measurements) that quantifies these overhead functions. SPASM separates the algorithmic overhead into its components (such as serial and work-imbalance overheads), and interaction overhead into its components (such as latency and contention). Such a separation is novel and has not been addressed in any previous study using real applications. We illustrate the top-down approach by considering a case study in implementing three NAS parallel kernels on two simulated message-passing platforms.
An Empirical Methodology for Exploring Reconfigurable Architectures
1993, Journal of Parallel and Distributed Computing
Recent research in reduced instruction set computer architectures has emphasized the importance of the empirical approach to designing computer architectures: architectural features are analyzed for utility and cost with respect to the system software that uses them. This approach has resulted in architectural simulators that allow computer designers to vary the features of the architecture being simulated and to analyze how the addition or removal of these features affects the cost and performance of the architecture. In this paper we apply this technique to a new area: reconfigurable architectures. Our approach is to use an empirical methodology that emphasizes the interaction between the target software and the reconfigurability features of parallel architectures. We have developed a set of tools, the reconfigurable architecture workbench, that assists in this methodology by allowing parallel programs to be simulated on a target architecture in order to study the performance implications of various reconfigurability features. The workbench is based on a framework, the PCI model, which describes the range of parallel programs, parallel architectures, and reconfiguration features. We present details of the design and implementation of a prototype workbench, GT-RAW. GT-RAW is being used to study the utility of one dimension of reconfiguration for image processing and image understanding applications. We present an example of the experiments that are being conducted with GT-RAW as a demonstration of our empirical methodology.
Running ASCEND, DESCEND and PIPELINE algorithms in parallel using small processors
1993, Information Processing Letters

View all citing articles on Scopus

View full text

The Structure of Parallel Algorithms

Publisher Summary

J. Comput. Syst. Sci.

Linear Algebra Appl.

Inf. Process. Lett.

“The Design and Analysis of Computer Algorithms.”

Computer interconnection structures: Taxonomy, characteristics, and examples

ACM Comput. Surv.

The ILLIAC IV computer

IEEE Trans. Comput.

Sorting networks and their applications

1968 Spring Joint Comput. Conf.

Asynchronous iterative methods for multiprocessors

J. Assoc. Comput. Mach.

The design and analysis of algorithms for asynchronous multiprocessors

Optimal sorting algorithms for parallel computers

IEEE Trans. Comput.

“Parallel Execution of a Sequence of Tasks on an Asynchronous Multiprocessor,”

Organization and maintenance of large ordered indexes

Acta Inf.

Concurrency of operations on B-trees

Acta Inf.

“Mathematical Theory of Connecting Networks and Telephone Traffic.”

A system of distributed databases (the fully redundant case)

IEEE Trans. Software Eng.

The parallel evaluation of general arithmetic expressions

J. Assoc. Comput. Mach.

“The Area-Time Complexity of Binary Multiplication,”

Algorithms for the tree machine

Overlap and pipeline processing

“Mathematical Approach to Computational Networks,”

On-the-fly garbage collection: An exercise in cooperation

Commun. ACM

Multiprocessor organization: A survey

ACM Comput. Surv.

The notions of consistency and predicate locks in a database system

Commun. ACM

Very high-speed computing systems

Proc. IEEE

The design of VLSI special purpose chip: Example and opinions

Res. Computer

“The Cm* Review Report,”

Notes on data base operating systems

“Lecture Notes in Computer Science 60: Operating Systems,”

Direct VLSI implementation of combinatorial algorithms

Pipelining of arithmetic functions

IEEE Trans, on Comput.

A survey of parallel algorithms in numerical linear algebra

SIAM Rev.

“Experience using multiprocessor systems: A status report,”

Programming issues raised by a multiprocessor

Proc. IEEE

Reducibility among combinational problems

“Complexity of Computer Computations,”

Cellular interconnection arrays

IEEE Trans. Comput.

Formal verification of parallel programs

Commun. ACM

Speed of recognition of context-free languages by array automata

SIAM J. Comput.

ILLIAC IV software and application programming

IEEE Trans. Comput.

A survey of parallel machine organization and programming

ACM Comput. Surv.