Back-propagation learning algorithm and parallel computers: The CLEPSYDRA mapping scheme
Section snippets
Preliminaries
An artificial neural network (ANN) is a parallel and distributed information processing structure consisting of a large number of Processing Elements interconnected via unidirectional signal channels called connections; its behavior is determined by parameters denoted weights and a learning procedure is used to compute these parameters. Such a learning procedure tends to be very time consuming and is therefore obvious to try to develop faster learning algorithms and/or to capitalize on the
Related papers
A first glance at the standard equations used to describe the BPA reveals that there are several degrees of parallelism in such an algorithm. First, if the learning is by-epoch, there is the parallel processing of many training examples (training examples parallelism, [12]). Here, on each processor a replica of the whole neural network and some training patterns are mapped; each replica evaluates partial weight changes that are then summed. Secondly, there is the parallel processing performed
The proposed approach
As it has been shown in the previous section, the exploitation of neuron parallelism (vertical slicing), eventually combined with the training examples parallelism, represents the most frequently used method to obtain fast simulations of the learning phase of feed-forward networks. The approach here proposed aims to improve the performance through the use of a mixture of neuron parallelism, synapse parallelism and training examples parallelism (if any).
With reference to the on-line BPA, suppose
Performance analysis
As a first step, analyze the time complexity of a sequential simulation. In the following we will indicate with Tia the time complexity of phase i (where i is the label of the formula in Section 2) in the sequential case (a=S) or in the parallel one (a=P). It is of courseThe k#'s clearly represent dimensional constants. In the parallel case it issince there is no communication and
Simulation results
In this paper are shown simulation results on a MIMD machine (the MEIKO CS-1) and on a SIMD machine (the MASPAR MP-1).
As regards the MIMD machine, the Transputer array programmed in OCCAM2 has been configured as a ring and a network with 256 input neurons, 128 hidden neurons and 16 output neurons has been chosen as a case-study; moreover, it has been supposed that the training set is composed by 100 examples (each consisting of 256 single precision floating point values) and the algorithm has
Concluding remarks
In this paper a new mapping scheme has been proposed for the parallel implementation of the learning phase of feed-forward neural networks. This mapping scheme (Clepsydra) allows to develop implementations oriented both to MIMD and SIMD parallel computers; simulation results on a Transputer-based MIMD machine and on a SIMD machine (the MASPAR MP-1) have been also sketched and commented.
The Clepsydra mapping scheme is based on the use of all-to-one (vectorial) associative broadcasting and on the
Antonio d'Acierno received the “laurea” degree cum laude in Electronic Engineering at the University of Naples (Italy) in 1988. Since then he has been with the “Istituto per la Ricerca sui Sistemi Informatici Paralleli” of the Italian National Research Council. His research activity concerns parallel architectures for neural networks simulation, parallel architectures for image processing, distributed object oriented programming and object oriented data base management systems. Antonio
References (19)
- et al.
Speeding up back-propagation training on a hypercube computer
Neurocomputing
(1992) - et al.
Mapping artificial neural networks to a tree shape neurocomputer, Microprocessors Microsystems
(1996) - et al.
A unified systolic architecture for artificial neural networks
J. Parallel Distrib. Comput.
(1989) - et al.
Using and designing massively parallel computers for artificial neural networks
J. Parallel Distrib. Comput.
(1992) Implementations of artificial neural networks on the connection machine
Parallel Comput.
(1990)- et al.
An implementation of back-propagation learning on GF11, a large SIMD parallel computer
Parallel Comput.
(1990) - et al.
The back-propagation algorithm on grid and hypercube architectures
Parallel Comput.
(1990) - et al.
Algorithmic mapping of feedforward neural networks onto multiple bus systems
IEEE Trans. Parallel Distrib. Systems
(1997) - M. Besch, H.W. Pohl, Flexible data parallel training of neural networks using MIMD-computers, in Third Euromicro...
Cited by (5)
Neural networks: An overview of early research, current frameworks and new challenges
2016, NeurocomputingCitation Excerpt :The most advanced simulators described in Section 2.2.1 include facilities or versions to be executed in multi-computers, using, for example, MPI interfaces. To carry out simulations in multi-computers it is necessary to work widely in the parallelization of the algorithms of various neural models, (back propagation [197,198], ART, neocognitron [199], recurrent NN, self-organizing NN, Bolzmann Machines [200], spiking models [201], etc.). Thus, in the EPFL, within the Blue Brain Project [202], a parallel version of the NEURON code has been developed, running on an IBM Blue Gene/P supercomputer with 16,384 computing cores and a peak performance of 56 TFlops, with this configuration facilitating the simulation of models at a cellular-level with up to 1 million detailed, multi-compartment neurons.
Artificial neural networks in hardware: A survey of two decades of progress
2010, NeurocomputingCitation Excerpt :We cover these topics by including most of the important works which have appeared in the literature with an optimistic perspective. However, owing to space limitations, there are topics, which will not be covered in this survey including hardware friendly learning algorithms (e.g., perturbation learning [71], constructive learning [72], cascade error projection learning [73,74], and local learning [75] with its special case of spike based Hebbian learning [76]), HNN designs focused on specific ANN models (e.g., MLP with back propagation [77,78], radial basis function networks [79,31,80], and Neocognitron [81]), and neurocomputers [49,82]. Rest of the paper is organized as follows: Issues related to the parameters used for evaluating an HNN system are highlighted in Section 2.
Quantum-Inspired Stacked Auto-encoder-Based Deep Neural Network Algorithm (Q-DNN)
2018, Arabian Journal for Science and EngineeringReal Life Applications of Soft Computing
2010, Real Life Applications of Soft Computing
Antonio d'Acierno received the “laurea” degree cum laude in Electronic Engineering at the University of Naples (Italy) in 1988. Since then he has been with the “Istituto per la Ricerca sui Sistemi Informatici Paralleli” of the Italian National Research Council. His research activity concerns parallel architectures for neural networks simulation, parallel architectures for image processing, distributed object oriented programming and object oriented data base management systems. Antonio d'Acierno is a member of IEEE Computer Society and of ACM.