Skip to main content

SPMD-Based Neural Network Simulation with Golang

  • Conference paper
  • First Online:
Computational Science – ICCS 2023 (ICCS 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14075))

Included in the following conference series:

  • 631 Accesses

Abstract

This paper describes the design and implementation of parallel neural networks (PNNs) with the novel programming language Golang. We follow in our approach the classical Single-Program Multiple-Data (SPMD) model where a PNN is composed of several sequential neural networks, which are trained with a proportional share of the training dataset. We used for this purpose the MNIST dataset, which contains binary images of handwritten digits. Our analysis focusses on different activation functions and optimizations in the form of stochastic gradients and initialization of weights and biases. We conduct a thorough performance analysis, where network configurations and different performance factors are analyzed and interpreted. Golang and its inherent parallelization support proved very well for parallel neural network simulation by considerable decreased processing times compared to sequential variants.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Ben-Nun, T., Hoefler, T.: Demystifying parallel and distributed deep learning: An in-depth concurrency analysis. ACM Comput. Surv. (CSUR) 52(4), 1–43 (2019)

    Article  Google Scholar 

  2. Brezany, P., Mueck, T.A., Schikuta, E.: A software architecture for massively parallel input-output. In: Waśniewski, J., Dongarra, J., Madsen, K., Olesen, D. (eds.) PARA 1996. LNCS, vol. 1184, pp. 85–96. Springer, Heidelberg (1996). https://doi.org/10.1007/3-540-62095-8_10

    Chapter  Google Scholar 

  3. Darema, F.: The SPMD Model: past, present and future. In: Cotronis, Y., Dongarra, J. (eds.) EuroPVM/MPI 2001. LNCS, vol. 2131, pp. 1–1. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-45417-9_1

    Chapter  Google Scholar 

  4. Goyal, P., et al.: Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 (2017)

  5. Hoare, C.A.R.: Communicating sequential processes. In: The origin of concurrent programming, pp. 413–443. Springer (1978)

    Google Scholar 

  6. Huqqani, A.A., Schikuta, E., Ye, S., Chen, P.: Multicore and gpu parallelization of neural networks for face recognition. Proc. Comput. Sci. 18(Supplement C), 349–358 (2013): 2013 International Conference on Computational Science

    Google Scholar 

  7. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  8. Meyerson, J.: The go programming language. IEEE Softw. 31(5), 104–104 (2014)

    Article  Google Scholar 

  9. Schikuta, E., Weishaupl, T.: N2grid: neural networks in the grid. In: 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541), vol. 2, pp. 1409–1414 (2004)

    Google Scholar 

  10. Schikuta, E.: Structural data parallel neural network simulation. In: Proceedings of 11th Annual International Symposium on High Performance Computing Systems (HPCS 1997), Winnipeg, Canada (1997)

    Google Scholar 

  11. Schikuta, E., Fuerle, T., Wanek, H.: Vipios: The vienna parallel input/output system. In: Pritchard, D., Reeve, J. (eds.) Euro-Par 1998 Parallel Processing, pp. 953–958. Springer, Berlin (1998)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Daniela Kalwarowskyj .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kalwarowskyj, D., Schikuta, E. (2023). SPMD-Based Neural Network Simulation with Golang. In: Mikyška, J., de Mulatier, C., Paszynski, M., Krzhizhanovskaya, V.V., Dongarra, J.J., Sloot, P.M. (eds) Computational Science – ICCS 2023. ICCS 2023. Lecture Notes in Computer Science, vol 14075. Springer, Cham. https://doi.org/10.1007/978-3-031-36024-4_43

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-36024-4_43

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-36023-7

  • Online ISBN: 978-3-031-36024-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics