Abstract
To meet the requirements of big data processing, this paper presents an efficient mapping scheme for a fully connected multilayered neural network, which is trained by using back-propagation (BP) algorithm based on Map-Reduce of cloud computing clusters. The batch-training (or epoch-training) regimes are used by effective segmentation of samples on the clusters, and are adopted in the separated training method, weight summary to achieve convergence by iterating. For a parallel BP algorithm on the clusters and a serial BP algorithm on an uniprocessor, the required time for implementing the algorithms is derived. The performance parameters, such as speedup, optimal number and minimum of data nodes are evaluated for the parallel BP algorithm on the clusters. Experiment results demonstrate that the proposed parallel BP algorithm in this paper has better speedup, faster convergence rate, less iterations than that of the existed algorithms.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-015-1599-3/MediaObjects/500_2015_1599_Fig1_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-015-1599-3/MediaObjects/500_2015_1599_Fig2_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-015-1599-3/MediaObjects/500_2015_1599_Fig3_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-015-1599-3/MediaObjects/500_2015_1599_Fig4_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-015-1599-3/MediaObjects/500_2015_1599_Fig5_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-015-1599-3/MediaObjects/500_2015_1599_Fig6_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-015-1599-3/MediaObjects/500_2015_1599_Fig7_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-015-1599-3/MediaObjects/500_2015_1599_Fig8_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00500-015-1599-3/MediaObjects/500_2015_1599_Fig9_HTML.gif)
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Alham NK et al (2013) A MapReduce-based distributed SVM ensemble for scalable image classification and annotation. Comput Math Appl 66(10):1920–1934
Apache Hadoop (2015). http://hadoop.apache.org/
Apache Hadoop Distributed File System (2015). http://hadoop.apache.org/docs/r1.0.4/hdfs_design.html
Bengio Y, Lamblin P, Popovici D, Larochelle H (2006) Greedy layer-wise training of deep networks. In: Proceedings of advances in neural information processing systems, pp 153–160
Chen G (2005) Parallel computing architecture, algorithm, programming [M]. Higher Education Press, Beijing, pp 1–366
Chu LC, Wah BW (1992) Optimal mapping of neural network learning on message-passing multicomputers. J Parallel Distrib Comput 14:319–339
Dean J, Ghemawat S (2008) Mapreduce: simplified data processing on large clusters. In: OSDI ’04, pp 137–150
Deprit E (1989) Implementing recurrent backpropagation on the connection machine. Neural Netw 2:295–314
El-Amawy A, Kulasinghe P (1997) Algorithmic mapping of feedforward neural networks onto multiple bus systems. IEEE Trans Parallel Distrib Syst 8(2):130–136
Erhan D, Manzagol P, Bengio Y, Bengio S, Vincent P (2009) The difficulty of training deep architectures and the effect of unsupervised pre-training. In: Proceedings of the international conference on artificial intelligence and statistics, pp 153–160
Eskil MT, Benli KS (2014) Facial expression recognition based on anatomy. Comput Vis Image Underst 119:1–14
Fahlman SE, Hinton GE (1987) Connectionist architectures for artificial intelligence. IEEE Comput 20:100–109
Feldman JA et al (1988) Computing with structured connectionist networks. Commun ACM 31(2):170–187
Fischer Asja, Igel C (2014) Training restricted Boltzmann machines: an introduction. Pattern Recognit 47:25–39
Foo SK, Saratchandran P, Sundararajan N (1995) Application of genetic algorithm for parallel implementation of backpropagation neural networks. In: Proceedings of the international symposium on intelligent robotic systems, pp 76–79
Han L (2007) Artificial neural networks tutorial [M]. Beijing University of Posts and Telecommunications Press of China, Beijing
Haykin S, Deng C (1991) Classification of radar clutter using neural networks. IEEE Trans Neural Netw 2:589–600
Haykins S (1999) Neural networks–a comprehensive foundation. Prentice Hall Int’l, Englewood Cliffs
Heo H, Park H, Kim N, Lee J (2009) Prediction of credit delinquents using locally transductive multi-layer perceptron. Neurocomputing 73:169–175
Isa NAM, Mamat WMFW (2011) Clustered-hybrid multilayer perceptron network for pattern recognition application. Appl Soft Comput 11:1457–1466
Jin X, Yang G, Yin Y, Man H, He H (2014) Sparse-representation-based classification with structure-preserving dimension reduction. Cognit Comput 6(3):608–621
Khan ER (1991) Systonlic architectures for artificial neural nets. In: IEEE international joint conference on neural networks, vol 1, pp 620–627, 18–21 Nov 1991
Kosko B (ed) (1992) Neural networks for signal processing. Prentice Hall, Englewood Cliffs
Kumar V, Shekhar S, Amin MB (1994) A scalable parallel formulation of the backpropagation algorithm for hypercubes and related architectures. IEEE Trans Parallel Distrib Syst 5:1073–1090
Kumar V, Shekhar S, Amin MB (1994) A scalable parallel formulation of the backpropagation algorithm for hypercubes and related architectures. IEEE Trans Parallel Distrib Syst 5:1073–1090
Leighton T (1992) Introduction to parallel algorithms and architectures. Morgan Kaufmann Publishers, San Mateo
Li H, Li H, Wei Y, Tang Y, Wang Q (2014) Sparse-based neural response for image classification. Neurocomputing 144:198–207
Liew WS, Seera M, Loo CK, Lim E (2015) Affect classification using genetic-optimized ensembles of fuzzyARTMAPs. Appl Soft Comput 27:53–63
Lin Wenhui, Liu Jun (2013) Performance analysis of map-reduce program in heterogeneous cloud computing. J Netw 8(8):1734–1741
Luttrell SP (1989) Image compression using a multilayer neural network. Pattern Recognit 10:1–7
Madhyastha TM, Reed DA (2002) Learning to classify parallel input/output access patterns. IEEE Trans Parallel Distrib Syst 13(8):802–813
Malalur SS, Manry MT, Jesudhas P (2015) Multiple optimal learning factors for the multi-layer perceptron. Neurocomputing 149:1490–1501
Malkin J, Subramanya A, Bilmes J (2009) On the semi-supervised learning of multi-layered perceptrons. In: Proceedings of the annual conference of the International Speech Communication Association, Brighton, UK, September 2009
Markowska-Kaczmar U, Koldowski M (2014) Spiking neural network vs multilayer perceptron: who is the winner in the racing car computer game. Soft Comput. doi:10.1007/s00500-014-1515-2
Matei Zaharia et al (2012) Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: Proceedings of the 9th USENIX conference on networked systems design and implementation, USENIX association
Murphey YL, Luo Y (2002) Feature extraction for a multiple pattern classification neural network system. Pattern Recognit Proc. 2:220–223
Ng A, Ngiam J et al (2014) Backpropagation Algorithm [DB/OL]. http://ufldl.stanford.edu/wiki/index.php/Backpropagation_Algorithm
Nikoonahad M, Liu DC (1990a) Medical ultrasound imaging using neural networks. Electron Lett 26:545–546
Nikoonahad M, Liu DC (1990b) Medical ultra sound imaging using neural networks. Electron Lett 2(6):18–23
Rumelhart DE, McClelland JL (eds) (1986a) Parallel and distributed processing (PDP): explorations in the microstructure of the cognition. MIT Press, Cambridge
Rumelhart DE, McClelland JL (eds) (1986b) Explorations in the micro structure of the cognition, parallel and distributed processing. MIT Press, Cambridge
Salakhutdinov R (2012) An efficient learning procedure for deep boltzmann machines. Neural Comput 24:1967–2006
Schmidhuber J (2015) Deep learning in neural networks: an overview. Neural Netw 61:85–117
Sejnowski TJ, Rosenberg CR (1987) Parallel networks that learn to pronounce English text. Complex Syst 1:145–168
Shams S, Przytula KW (1991) Implementation of multilayer neural networks on parallel programmable digital computers. In: Bayoumi M (ed) Parallel algorithms and architectures for DSP applications. Kluwer, Norwell, pp 225–253
Sudhakar V, Siva C, Murthy R (1998) Efficient mapping of back-propagation algorithm onto a network of workstations. IEEE Trans Man Mach Cybern Part B Cybern 28(6):841–848
Suk H-I, Lee S-W, Shen D (2014) Hierarchical feature representation and multimodal fusion with deeplearning for AD/MCI diagnosis. NeuroImage 101:569–582
Sundaram V (1990) PVM: a framework for parallel and distributed computing. Concurr Pract Exp 12:315–319
Sundararajan N, Saratchandran P (1998) Parallel architecture for artificial neural networks: paradigms and implementations. IEEE CS Press, Los Alamitos
Suresh S, Omkar SN, Mani V (2005) Parallel implementation of back-propagation algorithm in networks of workstations. IEEE Trans Parallel Distrib Syst 16(1):24–34
The Apache Software Foundation (2015) The Apache Hadoop Project. http://www.apache.org/
Torija AJ, Ruiz DP (2015) A general procedure to generate models for urban environmental-noise pollution using feature selection and machine learning methods. Sci Total Environ 505:680–693
Triguero I, Peralta D, Bacardit J, García S, Herrera F (2015) MRPR: a MapReduce solution for prototype reduction in big data classification. Neurocomputing 150:331–345
UCI dataset[EB/OL] (2015). http://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Original)
Xu J, Man H (2011) Dictionary learning based on Laplacian score in sparse coding. Lect Notes Comput Sci 6871:253–264
Xu J, He H, Man H (2012a) DCPE co-training for classification. Neurocomputing 86:75–85
Xu J, Yin Y, Man H (2012b) Feature selection based on sparse imputation. In: The 2012 international joint conference on neural networks (IJCNN), pp 1–7
Yoon H, Nang JH (1990) Multilayer neural networks on distributed memory multiprocessors. In: Proceedings of the international neural network conference, Paris, France
Zeng X, Luo S, Li Q (2010) An associative sparse coding neural network and applications. Neurocomputing 73:684–9
Zhou ZH, Li M (2010) Semi-supervised learning by disagreement. Knowl Inf Syst 24(3):415–439
Zhu C, Rao R (2012) The improved BP algorithm based on MapReduce and genetic algorithm. In: IEEE international conference on computer science and service system, pp 1567–1570
Acknowledgments
This work is supported by National Natural Science Foundation of China (No. 61171141) and the Provincial Department Cooperation Research Funds of Guangdong Province (No. 2012B091100448).
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by V. Loia.
Rights and permissions
About this article
Cite this article
Zhang, Hj., Xiao, Nf. Parallel implementation of multilayered neural networks based on Map-Reduce on cloud computing clusters. Soft Comput 20, 1471–1483 (2016). https://doi.org/10.1007/s00500-015-1599-3
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-015-1599-3