Skip to main content
SpringerLink
Log in

OSGAN: One-shot distributed learning using generative adversarial networks

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

With the advancements in mobile technology, a large amount of data is generated by end devices, which has created a renewed interest in developing new AI-based applications for gaining insights into these data. However, most of these distributed applications need data aggregated at a central server, which has posed severe bandwidth, latency, security, and privacy issues. This paper presents OSGAN (One-Shot distributed learning algorithm using Generative Adversarial Networks), a generic framework that trains a generative adversarial network (GAN) at each client and uses GAN’s generative capabilities to create sample data at the server. The server aggregates these data from various clients, builds a deep learning model and sends its parameters back to the clients in one communication round, i.e. the exchange of information between the clients and the server happens only once. In this paper, we present the design and implementation of OSGAN and evaluate its performance by comparing it with the state-of-the-art federated learning (FL) and central training algorithms for both IID and non-IID distribution of data. Our experiments on multiple datasets show that our proposed approach achieves a similar accuracy when compared with both FL and central training algorithms. Specifically, the accuracy drop with OSGAN is maintained within 2% for multiple datasets and multiple numbers of clients. Our results show that the proposed approach reduces the amount of data transfer by almost 98% when compared with federated learning and close to 80% when compared with the central learning approach thereby providing substantial benefit in terms of saving the bandwidth.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data Availability Statement

The image datasets that support the findings of this study are available in [40] while the non-image datasets can be found in [42].

References

  1. Khan AR, Mahmood A, Safdar A, Khan ZA, Khan NA (2016) Load forecasting, dynamic pricing and dsm in smart grid: a review. Renew Sustain Energy Rev 54:1311–1322

    Article  Google Scholar 

  2. Zhou K, Yang S (2016) Understanding household energy consumption behavior: the contribution of energy big data analytics. Renew Sustain Energy Rev 56:810–819

    Article  Google Scholar 

  3. Bui N, Zorzi M (2011) Health care applications: a solution based on the internet of things. In: Proceedings of the 4th International Symposium on Applied Sciences in Biomedical and Communication Technologies 1–5

  4. Bergstra J, Breuleux O, Bastien F, Lamblin P, Pascanu R, Desjardins G, Turian J, Warde-farley D, Bengio Y (2010) Theano: A cpu and gpu math compiler in python. In: Proceedings of the 9th Python in Science Conference 3–10

  5. Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) Caffe: Convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM International Conference on Multimedia. MM ’14, pp. 675–678. Association for Computing Machinery, New York, NY, USA https://doi.org/10.1145/2647868.2654889

  6. Collobert R, Kavukcuoglu K, Farabet C (2011) Torch7: A Matlab-Like Environment for Machine Learning. In: BigLearn, NIPS Workshop

  7. Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Mané D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Viégas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X: TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Software available from tensorflow.org (2015). https://www.tensorflow.org/

  8. Li M, Zhou L, Yang Z, Li A, Xia F, Andersen DG, Smola A (2013) Parameter server for distributed machine learning. In: Big Learning NIPS Workshop 6:2

  9. McDonald R, Hall K, Mann G (2010) Distributed training strategies for the structured perceptron. In: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 456–464

  10. Povey D, Zhang X, Khudanpur S (2014) Parallel training of deep neural networks with natural gradient and parameter averaging. arXiv preprint arXiv:1410.7455

  11. Shokri R, Shmatikov V (2015) Privacy-preserving deep learning. In: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security 1310–1321

  12. McMahan B, Moore E, Ramage D, Hampson S, y Arcas BA (2017) Communication-efficient learning of deep networks from decentralized data. In: Artificial Intelligence and Statistics, pp 1273–1282 PMLR

  13. Konecnỳ J, McMahan HB, Ramage D, Richtárik P (2016) Federated optimization: distributed machine learning for on-device intelligence

  14. Li T, Sahu AK, Talwalkar A, Smith V (2020) Federated learning: challenges, methods, and future directions. IEEE Signal Process Mag 37(3):50–60

    Article  Google Scholar 

  15. Zhao Y, Li M, Lai L, Suda N, Civin D, Chandra V (2018) Federated learning with non-iid data arXiv preprint arXiv:1806.00582

  16. Hard A, Rao K, Mathews R, Ramaswamy S, Beaufays F, Augenstein S, Eichner H, Kiddon C, Ramage D (2018) Federated learning for mobile keyboard prediction arXiv preprint arXiv:1811.03604

  17. Zhao Y, Zhao J, Jiang L, Tan R, Niyato D (2019) Mobile edge computing, blockchain and reputation-based crowdsourcing iot federated learning: a secure, decentralized and privacy-preserving system. arXiv preprint arXiv:1906.10893

  18. Konečnỳ J, McMahan HB, Yu FX, Richtárik P, Suresh AT, Bacon D (2016) Federated learning: strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492

  19. Sharifnassab A, Salehkaleybar S, Golestani SJ (2019) Order optimal one-shot distributed learning. In: Advances in Neural Information Processing Systems 2168–2177

  20. Kasturi A, Ellore AR, Hota C (2020) Fusion learning: a one shot federated learning. In: International Conference on Computational Science 424–436 Springer

  21. Bonawitz K, Salehi F, Konečnỳ J, McMahan B, Gruteser M (2019) Federated learning with autotuned communication-efficient secure aggregation. In: 2019 53rd Asilomar Conference on Signals, Systems, and Computers 1222–1226 IEEE

  22. Reisizadeh A, Mokhtari A, Hassani H, Jadbabaie A, Pedarsani R (2020) Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization. In: International Conference on Artificial Intelligence and Statistics 2021–2031

  23. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in Neural Information Processing Systems 2672–2680

  24. Karras T, Laine S, Aila T (2019) A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 4401–4410

  25. Vondrick C, Pirsiavash H, Torralba A (2016) Generating videos with scene dynamics. In: Advances in Neural Information Processing Systems 613–621

  26. Yang Z, Dong J, Liu P, Yang Y, Yan S (2019) Very long natural scenery image prediction by outpainting. In: Proceedings of the IEEE International Conference on Computer Vision 10561–10570

  27. Chen J, Pan X, Monga R, Bengio S, Jozefowicz R (2016) Revisiting distributed synchronous sgd. arXiv preprint arXiv:1604.00981

  28. Zhang S, Choromanska AE, LeCun Y (2015) Deep learning with elastic averaging sgd. In: Advances in Neural Information Processing Systems 685–693

  29. Zhang Y, Duchi JC, Wainwright MJ (2013) Communication-efficient algorithms for statistical optimization. J Machine Learn Res 14(1):3321–3363

    MathSciNet  MATH  Google Scholar 

  30. Suresh AT, Felix XY, Kumar S, McMahan HB (2017) Distributed mean estimation with limited communication. In: International Conference on Machine Learning 3329–3337

  31. Konečnỳ J, Richtárik P (2018) Randomized distributed mean estimation: accuracy vs. communication. Front Appl Mathemat Stat 4:62

    Article  Google Scholar 

  32. Khaled A, Richtárik P (2019) Gradient descent with compressed iterates. arXiv preprint arXiv:1909.04716

  33. Caldas S, Konečny J, McMahan HB, Talwalkar A (2018) Expanding the reach of federated learning by reducing client resource requirements. arXiv preprint arXiv:1812.07210

  34. Guha N, Talwalkar A, Smith V (2019) One-shot federated learning. arXiv preprint arXiv:1902.11175

  35. Fan C, Liu P (2020) Federated generative adversarial learning. arXiv preprint arXiv:2005.03793

  36. Rasouli M, Sun T, Rajagopal R (2020) Fedgan: Federated generative adversarial networks for distributed data. arXiv preprint arXiv:2006.07228

  37. Hardy C, Le Merrer E, Sericola B (2019) Md-gan: Multi-discriminator generative adversarial networks for distributed datasets. In: 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS) 866–877 IEEE

  38. Hardy C, Le Merrer E, Sericola B (2018) Gossiping gans. In: Proceedings of the Second Workshop on Distributed Infrastructures for Deep Learning: DIDL, vol. 22

  39. Mirza M, Osindero S (2014) Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784

  40. LeCun Y, Cortes C, Burges C (2010) Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist2

  41. Xiao H, Rasul K, Vollgraf R (2017) Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. CoRR abs/1708.07747arXiv:1708.07747

  42. Dua D, Graff C (2017). UCI Machine Learning Repository http://archive.ics.uci.edu/ml

  43. Li X, Huang K, Yang W, Wang S, Zhang Z (2019) On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189

  44. Sattler F, Wiedemann S, Müller K-R, Samek W (2019) Robust and communication-efficient federated learning from non-iid data. IEEE transactions on neural networks and learning systems

  45. Zinkevich M, Weimer M, Li L, Smola AJ (2010) Parallelized stochastic gradient descent. In: Advances in Neural Information Processing Systems, pp. 2595–2603

Download references

Funding

No significant financial assistance for this work has been provided that could have influenced its outcome.

Author information

Authors and Affiliations

  1. Computer Science Department, BITS Pilani, Hyderabad Campus, Hyderabad, Telangana, 500078, India

    Anirudh Kasturi & Chittaranjan Hota

Authors
  1. Anirudh Kasturi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chittaranjan Hota
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

We check that all named authors read and approved the article and that there are no other people who met the criteria for authorship but are not included. We also affirm that all have agreed the authorship order indicated in the manuscript of us.

Corresponding author

Correspondence to Anirudh Kasturi.

Ethics declarations

Ethical approval

This article does not contain any studies with human participants performed by the authors.

Conflict of interest

We want to confirm that there are no known conflicts of interest linked with this publication.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kasturi, A., Hota, C. OSGAN: One-shot distributed learning using generative adversarial networks. J Supercomput 79, 13620–13640 (2023). https://doi.org/10.1007/s11227-023-05182-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-023-05182-7

Keywords

Search

Navigation