Skip to main content
SpringerLink
Log in

Distributed Training from Multi-sourced Data

  • Conference paper
  • First Online:
Advanced Information Networking and Applications (AINA 2022)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 450))

Abstract

A distributed system is a set of logical or physical units capable of performing calculations and communicating with each other. Nowadays, these systems are at the heart of technologies such as the Internet of Things IoT, the Internet of Vehicles IoV, etc. These systems collect data, perform calculations and make decisions. On the other hand, deep learning (DL) has led to enormous progress in the field of artificial intelligence. Since the precision of DL to form a set of reference data on a single machine is known, it becomes more interesting to form several models and distribute the intelligence over the different nodes of the system by different calculation strategies.

In this paper we propose a method using deep neural networks on several machines by distributing the dataset before starting the training, ensuring communication between them in order to improve the calculation time and accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 219.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 279.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Dahl, G., Yu, D., Deng, L., Acero, A.: Context-dependent pre-trained deep neural networks for large vocabulary speech recognition. IEEE Trans. Audio Speech Lang. Process. 20, 30–42 (2012)

    Article  Google Scholar 

  2. Hinton, G., et al.: Deep neural networks for acoustic modeling in speech recognition. IEEE Sig. Process. Mag. 29, 82–97 (2012)

    Article  Google Scholar 

  3. Ciresan, D.C., Meier, U., Gambardella, L.M., Schmidhuber, J.: Deep big simple neural nets excel on handwritten digit recognition. CoRR (2010)

    Google Scholar 

  4. Coates, A., Lee, H., Ng, A.Y.: An analysis of single-layer networks in unsupervised feature learning. In: AISTATS 2011 (2011)

    Google Scholar 

  5. Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C.: A neural probabilistic language model. J. Mach. Learn. Res. 3, 1137–1155 (2003)

    MATH  Google Scholar 

  6. Collobert, R., Weston, J.: A unified architecture for natural language processing: deep neural networks with multitask learning. In: ICML 2008 (2008)

    Google Scholar 

  7. Le, Q.V., Ngiam, J., Coates, A., Lahiri, A., Prochnow, B., Ng, A.Y.: On optimization methods for deep learning. In: ICML 2011 (2011)

    Google Scholar 

  8. Raina, R., Madhavan, A., Ng, A.Y.: On optimization methods for deep learning. In: ICML 2009 (2009)

    Google Scholar 

  9. Martens, J.: Deep learning via hessian-free optimization. In: ICML 2010 (2010)

    Google Scholar 

  10. Bergstra, J., et al.: Theano : a CPU and GPU math expression compiler. In: SciPy 2010 (2010)

    Google Scholar 

  11. Ciresan, D., Meier, U., Schmidhuber, J.: Multi-column deep neural networks for image classification. Technical report. In: IDSIA 2012 (2012)

    Google Scholar 

  12. Shi, Q., et al.: Hash kernels. In: AISTATS 2009 (2009)

    Google Scholar 

  13. Langford, J., Smola, A., Zinkevich, M.: Slow learners are fast. In: NIPS 2009 (2009)

    Google Scholar 

  14. Mann, G., McDonald, R., Mohri, M., Silberman, N., Walker, D.: Efficient large-scale distributed training of conditional maximum entropy models. In: NIPS 2009 (2009)

    Google Scholar 

  15. LeCun, Y., Cortes, C.: The MNIST database of handwritten digits. In: NIPS 1998 (1998 ). yann.lecun.com/exdb/mnist/

  16. Li, S., et al.: PyTorch distributed: experiences on accelerating data parallel trainings. In: VLDB 2020 (2020)

    Google Scholar 

  17. He, M.L.Z., Rahayu, W., Xue, Y.: Distributed training of deep learning models: a taxonomic perspective. IEEE Trans. Parallel Distrib. Syst. 31(12), 2802–2818 (2020)

    Article  Google Scholar 

  18. Mayer, R., Jacobsen, H.-A.: Distributed training of deep learning models: scalable deep learning on distributed infrastructures: challenges, techniques and tools. ACM Comput. Surv. 53, 1–37 (2019)

    Article  Google Scholar 

  19. Grinberg, M.: Flask Web Development: Developing Web Applications with Python. O’Reilly Media, Inc. (2018)

    Google Scholar 

  20. Merkel, D.: Docker: lightweight Linux containers for consistent development and deployment. Linux J. 2014(239), 2 (2014)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. CNRS, LaBRI, UMR 5800, University of Bordeaux, Bordeaux INP, F-33400, Talence, France

    Ibrahim Dahaoui, Mohamed Mosbah & Akka Zemmari

Authors
  1. Ibrahim Dahaoui
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohamed Mosbah
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Akka Zemmari
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ibrahim Dahaoui .

Editor information

Editors and Affiliations

  1. Department of Information and Communication Engineering, Fukuoka Institute of Technology, Fukuoka, Japan

    Leonard Barolli

  2. University of Technology Sydney, Sydney, NSW, Australia

    Farookh Hussain

  3. Faculty of Bussiness Administration, Rissho University, Tokyo, Japan

    Tomoya Enokido

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Dahaoui, I., Mosbah, M., Zemmari, A. (2022). Distributed Training from Multi-sourced Data. In: Barolli, L., Hussain, F., Enokido, T. (eds) Advanced Information Networking and Applications. AINA 2022. Lecture Notes in Networks and Systems, vol 450. Springer, Cham. https://doi.org/10.1007/978-3-030-99587-4_29

Download citation

Publish with us

Policies and ethics

Search

Navigation