Elsevier

Neurocomputing

Volume 31, Issues 1–4, March 2000, Pages 67-85
Neurocomputing

Back-propagation learning algorithm and parallel computers: The CLEPSYDRA mapping scheme

https://doi.org/10.1016/S0925-2312(99)00151-4Get rights and content

Abstract

This paper deals with the parallel implementation of the back-propagation of errors learning algorithm. To obtain the partitioning of the neural network on the processor network the author describes a new mapping scheme that uses a mixture of synapse parallelism, neuron parallelism and training examples parallelism (if any). The proposed mapping scheme allows to describe the back-propagation algorithm as a collection of SIMD processes, so that both SIMD and MIMD machines can be used. The main feature of the obtained parallel algorithm is the absence of point-to-point communication; in fact, for each training pattern, an all-to-one broadcasting with an associative operator (combination) and an one-to-all broadcasting (that can be both realized in log P time) are needed. A performance model is proposed and tested on a ring-connected MIMD parallel computer. Simulation results on MIMD and SIMD parallel machines are also shown and commented.

Section snippets

Preliminaries

An artificial neural network (ANN) is a parallel and distributed information processing structure consisting of a large number of Processing Elements interconnected via unidirectional signal channels called connections; its behavior is determined by parameters denoted weights and a learning procedure is used to compute these parameters. Such a learning procedure tends to be very time consuming and is therefore obvious to try to develop faster learning algorithms and/or to capitalize on the

Related papers

A first glance at the standard equations used to describe the BPA reveals that there are several degrees of parallelism in such an algorithm. First, if the learning is by-epoch, there is the parallel processing of many training examples (training examples parallelism, [12]). Here, on each processor a replica of the whole neural network and some training patterns are mapped; each replica evaluates partial weight changes that are then summed. Secondly, there is the parallel processing performed

The proposed approach

As it has been shown in the previous section, the exploitation of neuron parallelism (vertical slicing), eventually combined with the training examples parallelism, represents the most frequently used method to obtain fast simulations of the learning phase of feed-forward networks. The approach here proposed aims to improve the performance through the use of a mixture of neuron parallelism, synapse parallelism and training examples parallelism (if any).

With reference to the on-line BPA, suppose

Performance analysis

As a first step, analyze the time complexity of a sequential simulation. In the following we will indicate with Tia the time complexity of phase i (where i is the label of the formula in Section 2) in the sequential case (a=S) or in the parallel one (a=P). It is of courseT1S=k1NiNh+k2Nh,T2S=k1NhNo+k2No,T3S=k3No,T6S=k4No,T7S=k1NhNo+k4No,T8S=k5(NiNh),T9S=k5(NhNo).The k#'s clearly represent dimensional constants. In the parallel case it isT1P=T1SPsince there is no communication and

Simulation results

In this paper are shown simulation results on a MIMD machine (the MEIKO CS-1) and on a SIMD machine (the MASPAR MP-1).

As regards the MIMD machine, the Transputer array programmed in OCCAM2 has been configured as a ring and a network with 256 input neurons, 128 hidden neurons and 16 output neurons has been chosen as a case-study; moreover, it has been supposed that the training set is composed by 100 examples (each consisting of 256 single precision floating point values) and the algorithm has

Concluding remarks

In this paper a new mapping scheme has been proposed for the parallel implementation of the learning phase of feed-forward neural networks. This mapping scheme (Clepsydra) allows to develop implementations oriented both to MIMD and SIMD parallel computers; simulation results on a Transputer-based MIMD machine and on a SIMD machine (the MASPAR MP-1) have been also sketched and commented.

The Clepsydra mapping scheme is based on the use of all-to-one (vectorial) associative broadcasting and on the

Antonio d'Acierno received the “laurea” degree cum laude in Electronic Engineering at the University of Naples (Italy) in 1988. Since then he has been with the “Istituto per la Ricerca sui Sistemi Informatici Paralleli” of the Italian National Research Council. His research activity concerns parallel architectures for neural networks simulation, parallel architectures for image processing, distributed object oriented programming and object oriented data base management systems. Antonio

References (19)

There are more references available in the full text version of this article.

Cited by (5)

  • Neural networks: An overview of early research, current frameworks and new challenges

    2016, Neurocomputing
    Citation Excerpt :

    The most advanced simulators described in Section 2.2.1 include facilities or versions to be executed in multi-computers, using, for example, MPI interfaces. To carry out simulations in multi-computers it is necessary to work widely in the parallelization of the algorithms of various neural models, (back propagation [197,198], ART, neocognitron [199], recurrent NN, self-organizing NN, Bolzmann Machines [200], spiking models [201], etc.). Thus, in the EPFL, within the Blue Brain Project [202], a parallel version of the NEURON code has been developed, running on an IBM Blue Gene/P supercomputer with 16,384 computing cores and a peak performance of 56 TFlops, with this configuration facilitating the simulation of models at a cellular-level with up to 1 million detailed, multi-compartment neurons.

  • Artificial neural networks in hardware: A survey of two decades of progress

    2010, Neurocomputing
    Citation Excerpt :

    We cover these topics by including most of the important works which have appeared in the literature with an optimistic perspective. However, owing to space limitations, there are topics, which will not be covered in this survey including hardware friendly learning algorithms (e.g., perturbation learning [71], constructive learning [72], cascade error projection learning [73,74], and local learning [75] with its special case of spike based Hebbian learning [76]), HNN designs focused on specific ANN models (e.g., MLP with back propagation [77,78], radial basis function networks [79,31,80], and Neocognitron [81]), and neurocomputers [49,82]. Rest of the paper is organized as follows: Issues related to the parameters used for evaluating an HNN system are highlighted in Section 2.

  • Real Life Applications of Soft Computing

    2010, Real Life Applications of Soft Computing

Antonio d'Acierno received the “laurea” degree cum laude in Electronic Engineering at the University of Naples (Italy) in 1988. Since then he has been with the “Istituto per la Ricerca sui Sistemi Informatici Paralleli” of the Italian National Research Council. His research activity concerns parallel architectures for neural networks simulation, parallel architectures for image processing, distributed object oriented programming and object oriented data base management systems. Antonio d'Acierno is a member of IEEE Computer Society and of ACM.

View full text