Skip to main content
SpringerLink
Log in

Designing a Neural Network from Scratch for Big Data Powered by Multi-node GPUs

  • Chapter
  • First Online:
Handbook of Deep Learning Applications

Part of the book series: Smart Innovation, Systems and Technologies ((SIST,volume 136))

Abstract

Lately, Machine Learning has taken a crucial role in the society in different vertical sectors. For complex problems with high-dimensionality, Deep Learning has become an efficient solution for learning in the context of supervisioned learning. Deep Learning [1] consists in using Artificial Neural Networks (ANN or NN) with several hidden layers, typically also with a large number of nodes in each layer.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. J. Schmidhuber, Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015)

    Article  Google Scholar 

  2. W.S. McCulloch, W. Pitts, A logical calculus of the ideas immanent in nervous activity. Bull. Math. Biophys. 5(4), 115–133 (1943)

    Article  MathSciNet  Google Scholar 

  3. A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep convolutional neural networks, in Advances in Neural Information Processing Systems (2012), pp. 1097–1105

    Google Scholar 

  4. G.L. Martin, J.A. Pittman, Recognizing hand-printed letters and digits using backpropagation learning. Neural Comput. 3(2), 258–267 (1991)

    Article  Google Scholar 

  5. G. Zhang, M.Y. Hu, B.E. Patuwo, D.C. Indro, Artificial neural networks in bankruptcy prediction: general framework and cross-validation analysis. Eur. J. Oper. Res. 116(1), 16–32 (1999)

    Article  Google Scholar 

  6. M.H. Ebell, Artificial neural networks for predicting failure to survive following in-hospital cardiopulmonary resuscitation. J. Family Pract. 36(3), 297–304 (1993)

    Google Scholar 

  7. D.A. Pomerleau, Efficient training of artificial neural networks for autonomous navigation. Neural Comput. 3(1), 88–97 (1991)

    Article  Google Scholar 

  8. W.S. Sarle, On Computing Number of Neurons in Hidden Layer (1995)

    Google Scholar 

  9. A. Jacobs, The pathologies of big data. Commun. ACM 52(8), 36–44 (2009)

    Article  Google Scholar 

  10. M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D.G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, X. Zheng, Tensorflow: a system for large-scale machine learning, in Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation. OSDI’16, Berkeley, CA, USA, USENIX Association (2016), pp. 265–283

    Google Scholar 

  11. J. Bergstra, O. Breuleux, P. Lamblin, R. Pascanu, O. Delalleau, G. Desjardins, I. Goodfellow, A. Bergeron, Y. Bengio, P. Kaelbling, Theano: Deep Learning on GPUs with Python

    Google Scholar 

  12. R. Collobert, S. Bengio, J. Marithoz, Torch: A Modular Machine Learning Software Library (2002)

    Google Scholar 

  13. B.J. Erickson, P. Korfiatis, Z. Akkus, T. Kline, K. Philbrick, Toolkits and libraries for deep learning. J. Digit. Imaging 30(4), 400–405 (2017)

    Article  Google Scholar 

  14. S. Lawrence, C.L. Giles, A.C. Tsoi, What size neural network gives optimal generalization? convergence properties of backpropagation (1998)

    Google Scholar 

  15. F. Cao, T. Xie, The construction and approximation for feedforword neural networks with fixed weights, in International Conference on Machine Learning and Cybernetics, ICMLC 2010, Qingdao, China, 11–14 July 2010, Proceedings (2010), pp. 3164–3168

    Google Scholar 

  16. N. Guliyev, V. Ismailov, On the approximation by single hidden layer feedforward neural networks with fixed weights (2017)

    Google Scholar 

  17. S. Koturwar, S. Merchant, Weight initialization of deep neural networks (DNNs) using data statistics. arXiv:1710.10570 (2017)

  18. S.K. Kumar, On weight initialization in deep neural networks. arXiv:1704.08863 (2017)

  19. D. Mishkin, J. Matas, All you need is a good init. arXiv:1511.06422 (2015)

  20. I. Goodfellow, Y. Bengio, A. Courville, Y. Bengio, Deep Learning, vol. 1 (MIT Press, Cambridge, 2016)

    Google Scholar 

  21. V. Nair, G.E. Hinton, Rectified linear units improve restricted Boltzmann machines, in Proceedings of the 27th International Conference on Machine Learning (ICML-10) (2010), pp. 807–814

    Google Scholar 

  22. D.E. Rumelhart, G.E. Hinton, R.J. Williams, Learning representations by back-propagating errors. Nature 323(6088), 533 (1986)

    Article  Google Scholar 

  23. M. Lichman, UCI Machine Learning Repository (2013)

    Google Scholar 

  24. K.O. Stanley, D.B. D’Ambrosio, J. Gauci, A hypercube-based encoding for evolving large-scale neural networks. Artif. Life 15(2), 185–212 (2009)

    Article  Google Scholar 

  25. W. McKinney et al., Data structures for statistical computing in python, in Proceedings of the 9th Python in Science Conference, vol. 445 (2010), pp. 51–56

    Google Scholar 

  26. A. Trask, A Neural Network in 11 Lines of Python (2013)

    Google Scholar 

  27. L. Gu, H. Li, Memory or time: performance evaluation for iterative operation on hadoop and spark, in 2013 IEEE 10th International Conference on High Performance Computing and Communications and 2013 IEEE International Conference on Embedded and Ubiquitous Computing (HPCC\(\_\)EUC) (IEEE, 2013), pp. 721–727

    Google Scholar 

  28. G. Dahl, A. McAvinney, T. Newhall et al., Parallelizing neural network training for cluster systems, in Proceedings of the IASTED International Conference on Parallel and Distributed Computing and Networks (ACTA Press, 2008), pp. 220–225

    Google Scholar 

  29. S. Sanfilippo, P. Noordhuis, Redis (2009)

    Google Scholar 

  30. J. Reese, S. Zaranek, GPU programming in MATLAB, in MathWorks News & Notes (The MathWorks Inc., Natick, MA, 2012), pp. 22–25

    Google Scholar 

  31. M.M. Chakravarty, G. Keller, S. Lee, T.L. McDonell, V. Grover, Accelerating Haskell array codes with multicore GPUs, in Proceedings of the Sixth Workshop on Declarative Aspects of Multicore Programming (ACM, 2011), pp. 3–14

    Google Scholar 

  32. A. Fonseca, B. Cabral, Æminiumgpu: an intelligent framework for GPU programming, in Facing the Multicore-Challenge III (Springer, 2013), pp. 96–107

    Google Scholar 

  33. B. Catanzaro, M. Garland, K. Keutzer, Copperhead: compiling an embedded data parallel language. ACM SIGPLAN Not. 46(8), 47–56 (2011)

    Article  Google Scholar 

  34. S.K. Lam, A. Pitrou, S. Seibert, Numba: a LLVM-based python JIT compiler, in Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC (ACM, 2015), p. 7

    Google Scholar 

  35. B. Snyder, D. Bosnanac, R. Davies, ActiveMQ in Action, vol. 47 (Manning, 2011)

    Google Scholar 

  36. E.D. Karnin, A simple procedure for pruning back-propagation trained neural networks. IEEE Trans. Neural Netw. 1(2), 239–242 (1990)

    Article  Google Scholar 

  37. D.B. Fogel, L.J. Fogel, V. Porto, Evolving neural networks. Biol. Cybern. 63(6), 487–493 (1990)

    Article  Google Scholar 

  38. J.F. de Freitas, M. Niranjan, A.H. Gee, A. Doucet, Sequential Monte Carlo methods to train neural network models. Neural Comput. 12(4), 955–993 (2000)

    Article  Google Scholar 

  39. V. Sze, Y. Chen, T. Yang, J.S. Emer, Efficient processing of deep neural networks: a tutorial and survey. arXiv:1703.09039 (2017)

Download references

Acknowledgements

The first author was supported by the LASIGE Research Unit (UID/CEC/00408/2013).

Author information

Authors and Affiliations

  1. LASIGE, Faculdade de Ciências da Universidade de Lisboa, Lisbon, Portugal

    Alcides Fonseca

  2. CISUC, Departamento de Engenharia Informática, Faculdade de Ciências e Tecnologia da Universidade de Coimbra, Coimbra, Portugal

    Bruno Cabral

Authors
  1. Alcides Fonseca
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bruno Cabral
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alcides Fonseca .

Editor information

Editors and Affiliations

  1. Aurel Vlaicu University of Arad, Arad, Romania

    Valentina Emilia Balas

  2. School of Computer Science and Engineering, Vellore Institute of Technology, Vellore, Tamil Nadu, India

    Sanjiban Sekhar Roy

  3. University of Canberra, Bruce, ACT, Australia

    Dharmendra Sharma

  4. Department of Civil Engineering, National Institute of Technology Patna, Patna, Bihar, India

    Pijush Samui

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Fonseca, A., Cabral, B. (2019). Designing a Neural Network from Scratch for Big Data Powered by Multi-node GPUs. In: Balas, V., Roy, S., Sharma, D., Samui, P. (eds) Handbook of Deep Learning Applications. Smart Innovation, Systems and Technologies, vol 136. Springer, Cham. https://doi.org/10.1007/978-3-030-11479-4_1

Download citation

Publish with us

Policies and ethics

Search

Navigation