Skip to main content

Advertisement

Log in

Parallel implementation of multilayered neural networks based on Map-Reduce on cloud computing clusters

  • Methodologies and Application
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

To meet the requirements of big data processing, this paper presents an efficient mapping scheme for a fully connected multilayered neural network, which is trained by using back-propagation (BP) algorithm based on Map-Reduce of cloud computing clusters. The batch-training (or epoch-training) regimes are used by effective segmentation of samples on the clusters, and are adopted in the separated training method, weight summary to achieve convergence by iterating. For a parallel BP algorithm on the clusters and a serial BP algorithm on an uniprocessor, the required time for implementing the algorithms is derived. The performance parameters, such as speedup, optimal number and minimum of data nodes are evaluated for the parallel BP algorithm on the clusters. Experiment results demonstrate that the proposed parallel BP algorithm in this paper has better speedup, faster convergence rate, less iterations than that of the existed algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  • Alham NK et al (2013) A MapReduce-based distributed SVM ensemble for scalable image classification and annotation. Comput Math Appl 66(10):1920–1934

  • Apache Hadoop (2015). http://hadoop.apache.org/

  • Apache Hadoop Distributed File System (2015). http://hadoop.apache.org/docs/r1.0.4/hdfs_design.html

  • Bengio Y, Lamblin P, Popovici D, Larochelle H (2006) Greedy layer-wise training of deep networks. In: Proceedings of advances in neural information processing systems, pp 153–160

  • Chen G (2005) Parallel computing architecture, algorithm, programming [M]. Higher Education Press, Beijing, pp 1–366

  • Chu LC, Wah BW (1992) Optimal mapping of neural network learning on message-passing multicomputers. J Parallel Distrib Comput 14:319–339

    Article  Google Scholar 

  • Dean J, Ghemawat S (2008) Mapreduce: simplified data processing on large clusters. In: OSDI ’04, pp 137–150

  • Deprit E (1989) Implementing recurrent backpropagation on the connection machine. Neural Netw 2:295–314

    Article  Google Scholar 

  • El-Amawy A, Kulasinghe P (1997) Algorithmic mapping of feedforward neural networks onto multiple bus systems. IEEE Trans Parallel Distrib Syst 8(2):130–136

    Article  Google Scholar 

  • Erhan D, Manzagol P, Bengio Y, Bengio S, Vincent P (2009) The difficulty of training deep architectures and the effect of unsupervised pre-training. In: Proceedings of the international conference on artificial intelligence and statistics, pp 153–160

  • Eskil MT, Benli KS (2014) Facial expression recognition based on anatomy. Comput Vis Image Underst 119:1–14

    Article  Google Scholar 

  • Fahlman SE, Hinton GE (1987) Connectionist architectures for artificial intelligence. IEEE Comput 20:100–109

    Article  Google Scholar 

  • Feldman JA et al (1988) Computing with structured connectionist networks. Commun ACM 31(2):170–187

    Article  Google Scholar 

  • Fischer Asja, Igel C (2014) Training restricted Boltzmann machines: an introduction. Pattern Recognit 47:25–39

    Article  MATH  Google Scholar 

  • Foo SK, Saratchandran P, Sundararajan N (1995) Application of genetic algorithm for parallel implementation of backpropagation neural networks. In: Proceedings of the international symposium on intelligent robotic systems, pp 76–79

  • Han L (2007) Artificial neural networks tutorial [M]. Beijing University of Posts and Telecommunications Press of China, Beijing

    Google Scholar 

  • Haykin S, Deng C (1991) Classification of radar clutter using neural networks. IEEE Trans Neural Netw 2:589–600

    Article  Google Scholar 

  • Haykins S (1999) Neural networks–a comprehensive foundation. Prentice Hall Int’l, Englewood Cliffs

    Google Scholar 

  • Heo H, Park H, Kim N, Lee J (2009) Prediction of credit delinquents using locally transductive multi-layer perceptron. Neurocomputing 73:169–175

    Article  Google Scholar 

  • Isa NAM, Mamat WMFW (2011) Clustered-hybrid multilayer perceptron network for pattern recognition application. Appl Soft Comput 11:1457–1466

    Article  Google Scholar 

  • Jin X, Yang G, Yin Y, Man H, He H (2014) Sparse-representation-based classification with structure-preserving dimension reduction. Cognit Comput 6(3):608–621

    Article  Google Scholar 

  • Khan ER (1991) Systonlic architectures for artificial neural nets. In: IEEE international joint conference on neural networks, vol 1, pp 620–627, 18–21 Nov 1991

  • Kosko B (ed) (1992) Neural networks for signal processing. Prentice Hall, Englewood Cliffs

  • Kumar V, Shekhar S, Amin MB (1994) A scalable parallel formulation of the backpropagation algorithm for hypercubes and related architectures. IEEE Trans Parallel Distrib Syst 5:1073–1090

    Article  Google Scholar 

  • Kumar V, Shekhar S, Amin MB (1994) A scalable parallel formulation of the backpropagation algorithm for hypercubes and related architectures. IEEE Trans Parallel Distrib Syst 5:1073–1090

    Article  Google Scholar 

  • Leighton T (1992) Introduction to parallel algorithms and architectures. Morgan Kaufmann Publishers, San Mateo

    MATH  Google Scholar 

  • Li H, Li H, Wei Y, Tang Y, Wang Q (2014) Sparse-based neural response for image classification. Neurocomputing 144:198–207

    Article  Google Scholar 

  • Liew WS, Seera M, Loo CK, Lim E (2015) Affect classification using genetic-optimized ensembles of fuzzyARTMAPs. Appl Soft Comput 27:53–63

    Article  Google Scholar 

  • Lin Wenhui, Liu Jun (2013) Performance analysis of map-reduce program in heterogeneous cloud computing. J Netw 8(8):1734–1741

    Google Scholar 

  • Luttrell SP (1989) Image compression using a multilayer neural network. Pattern Recognit 10:1–7

    Article  MATH  Google Scholar 

  • Madhyastha TM, Reed DA (2002) Learning to classify parallel input/output access patterns. IEEE Trans Parallel Distrib Syst 13(8):802–813

    Article  Google Scholar 

  • Malalur SS, Manry MT, Jesudhas P (2015) Multiple optimal learning factors for the multi-layer perceptron. Neurocomputing 149:1490–1501

    Article  Google Scholar 

  • Malkin J, Subramanya A, Bilmes J (2009) On the semi-supervised learning of multi-layered perceptrons. In: Proceedings of the annual conference of the International Speech Communication Association, Brighton, UK, September 2009

  • Markowska-Kaczmar U, Koldowski M (2014) Spiking neural network vs multilayer perceptron: who is the winner in the racing car computer game. Soft Comput. doi:10.1007/s00500-014-1515-2

  • Matei Zaharia et al (2012) Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: Proceedings of the 9th USENIX conference on networked systems design and implementation, USENIX association

  • Murphey YL, Luo Y (2002) Feature extraction for a multiple pattern classification neural network system. Pattern Recognit Proc. 2:220–223

    Google Scholar 

  • Ng A, Ngiam J et al (2014) Backpropagation Algorithm [DB/OL]. http://ufldl.stanford.edu/wiki/index.php/Backpropagation_Algorithm

  • Nikoonahad M, Liu DC (1990a) Medical ultrasound imaging using neural networks. Electron Lett 26:545–546

  • Nikoonahad M, Liu DC (1990b) Medical ultra sound imaging using neural networks. Electron Lett 2(6):18–23

  • Rumelhart DE, McClelland JL (eds) (1986a) Parallel and distributed processing (PDP): explorations in the microstructure of the cognition. MIT Press, Cambridge

  • Rumelhart DE, McClelland JL (eds) (1986b) Explorations in the micro structure of the cognition, parallel and distributed processing. MIT Press, Cambridge

  • Salakhutdinov R (2012) An efficient learning procedure for deep boltzmann machines. Neural Comput 24:1967–2006

    Article  MathSciNet  MATH  Google Scholar 

  • Schmidhuber J (2015) Deep learning in neural networks: an overview. Neural Netw 61:85–117

    Article  Google Scholar 

  • Sejnowski TJ, Rosenberg CR (1987) Parallel networks that learn to pronounce English text. Complex Syst 1:145–168

    MATH  Google Scholar 

  • Shams S, Przytula KW (1991) Implementation of multilayer neural networks on parallel programmable digital computers. In: Bayoumi M (ed) Parallel algorithms and architectures for DSP applications. Kluwer, Norwell, pp 225–253

    Chapter  Google Scholar 

  • Sudhakar V, Siva C, Murthy R (1998) Efficient mapping of back-propagation algorithm onto a network of workstations. IEEE Trans Man Mach Cybern Part B Cybern 28(6):841–848

    Article  Google Scholar 

  • Suk H-I, Lee S-W, Shen D (2014) Hierarchical feature representation and multimodal fusion with deeplearning for AD/MCI diagnosis. NeuroImage 101:569–582

    Article  Google Scholar 

  • Sundaram V (1990) PVM: a framework for parallel and distributed computing. Concurr Pract Exp 12:315–319

    Article  Google Scholar 

  • Sundararajan N, Saratchandran P (1998) Parallel architecture for artificial neural networks: paradigms and implementations. IEEE CS Press, Los Alamitos

    Google Scholar 

  • Suresh S, Omkar SN, Mani V (2005) Parallel implementation of back-propagation algorithm in networks of workstations. IEEE Trans Parallel Distrib Syst 16(1):24–34

    Article  Google Scholar 

  • The Apache Software Foundation (2015) The Apache Hadoop Project. http://www.apache.org/

  • Torija AJ, Ruiz DP (2015) A general procedure to generate models for urban environmental-noise pollution using feature selection and machine learning methods. Sci Total Environ 505:680–693

    Article  Google Scholar 

  • Triguero I, Peralta D, Bacardit J, García S, Herrera F (2015) MRPR: a MapReduce solution for prototype reduction in big data classification. Neurocomputing 150:331–345

    Article  Google Scholar 

  • UCI dataset[EB/OL] (2015). http://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Original)

  • Xu J, Man H (2011) Dictionary learning based on Laplacian score in sparse coding. Lect Notes Comput Sci 6871:253–264

    Article  Google Scholar 

  • Xu J, He H, Man H (2012a) DCPE co-training for classification. Neurocomputing 86:75–85

  • Xu J, Yin Y, Man H (2012b) Feature selection based on sparse imputation. In: The 2012 international joint conference on neural networks (IJCNN), pp 1–7

  • Yoon H, Nang JH (1990) Multilayer neural networks on distributed memory multiprocessors. In: Proceedings of the international neural network conference, Paris, France

  • Zeng X, Luo S, Li Q (2010) An associative sparse coding neural network and applications. Neurocomputing 73:684–9

    Article  Google Scholar 

  • Zhou ZH, Li M (2010) Semi-supervised learning by disagreement. Knowl Inf Syst 24(3):415–439

    Article  Google Scholar 

  • Zhu C, Rao R (2012) The improved BP algorithm based on MapReduce and genetic algorithm. In: IEEE international conference on computer science and service system, pp 1567–1570

Download references

Acknowledgments

This work is supported by National Natural Science Foundation of China (No. 61171141) and the Provincial Department Cooperation Research Funds of Guangdong Province (No. 2012B091100448).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Hai-jun Zhang.

Additional information

Communicated by V. Loia.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Hj., Xiao, Nf. Parallel implementation of multilayered neural networks based on Map-Reduce on cloud computing clusters. Soft Comput 20, 1471–1483 (2016). https://doi.org/10.1007/s00500-015-1599-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-015-1599-3

Keywords

Navigation