Skip to main content
SpringerLink
Log in

Consolidating incentivization in distributed neural network training via decentralized autonomous organization

  • S.I.: Deep learning modelling in real life: (Anomaly Detection, Biomedical, Concept Analysis, Finance, Image analysis, Recommendation)
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Big data has reignited research interest in machine learning. Massive quantities of data are being generated regularly as a consequence of the development in the Internet, social networks, and online sensors. Particularly deep neural networks benefited greatly from this unprecedented data availability. Large models with millions of parameters are becoming common, and big data has been proved to be essential for their effective training. The scientific community has come up with a number of methods to create more accurate models, but most of these methods require high-performance infrastructure. There is also the issue of privacy, since anyone using leased processing power from a remote data center is putting their data in the hands of a third party. Studies on decentralized and non-binding methods among individuals with commodity hardware are scarce, though. Our work on LEARNAE seeks to respond to this challenge by creating a totally distributed and fault-tolerant framework of artificial neural network training. In our recent work, we demonstrated a method for incentivizing peers to participate to collaborative process, even if they are not interested in the neural network produced. For this, LEARNAE included a subsystem that rewards participants proportionately to their contribution using digital assets. In this article we add another important piece to the puzzle: A decentralized mechanism to mitigate the effect of bad actors, such as nodes that attempt to exploit LEARNAE’s network power without following the established rewarding rules. This is achieved by a novel reward mechanism, which takes into account the overall contribution of each node to the entire swarm. The network collaboratively builds a contribution profile for every participant, and the final rewards are dictated by these profiles. Taking for granted that the majority of the peers are benevolent, the whole process is tamper-proof, since it is implemented on blockchain and thus is protected by distributed consensus. All codebase is structured as a decentralized autonomous organization, which allows LEARNAE to embed new features like digital asset locking, proposal submitting, and voting.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. https://docs.ipfs.io/concepts/bitswap.

  2. https://www.statista.com/statistics/802690/worldwide-connected-devices-by-access-technology.

  3. https://goerli.net.

  4. https://www.rinkeby.io.

References

  1. Nikolaidis S, Refanidis I (2019) Learnae: distributed and resilient deep neural network training for heterogeneous peer to peer topologies. Eng Appl Neural Netw 25:286–298. https://doi.org/10.1007/978-3-030-20257-6_24

    Article  Google Scholar 

  2. Nikolaidis S, Refanidis I (2020) Privacy preserving distributed training of neural networks. Neural Comput Applic 32(23):17333–17350. https://doi.org/10.1007/s00521-020-04880-0

    Article  Google Scholar 

  3. Nikolaidis S, Refanidis I (2021) Using distributed ledger technology to democratize neural network training. Appl Intell. https://doi.org/10.1007/s10489-021-02340-3

    Article  Google Scholar 

  4. Benet J (2021) “IPFS - Content addressed, versioned, P2P file system,” [cs], Jul. 2014, Accessed: Apr. 01, [Online]. Available: http://arxiv.org/abs/1407.3561

  5. Popov S, Saa O, Finardi P (2019) Equilibria in the Tangle. Comput Ind Eng 136:160–172. https://doi.org/10.1016/j.cie.2019.07.025

    Article  Google Scholar 

  6. Zhang X, Trmal J, Povey D, and Khudanpur S (2014) “Improving deep neural network acoustic models using generalized maxout networks,” in 2014 IEEE International conference on acoustics, speech and signal processing (ICASSP), , pp. 215–219. doi: https://doi.org/10.1109/ICASSP.2014.6853589

  7. Miao Y, Zhang H, and Metze F (2014) Distributed learning of multilingual dnn feature extractors using gpus

  8. Dean J et al (2021) “Large scale distributed deep networks,” in advances in neural information processing systems, 2012, vol. 25. Accessed: Aug. 11, [Online]. Available: https://papers.nips.cc/paper/2012/hash/6aca97005c68f1206823815f66102863-Abstract.html

  9. Mashtizadeh AJ, Bittau A, Huang YF, and Mazières D (2013) “Replication, history, and grafting in the Ori file system,” in Proceedings of the twenty-fourth ACM symposium on operating systems principles, New York, NY, USA, pp. 151–166. https://doi.org/10.1145/2517349.2522721

  10. Cohen B (2003) “Incentives build robustness in BitTorrent,” Workshop on Economics of PeertoPeer systems, vol. 6

  11. Baumgart I and Mies S (2007) “S/Kademlia: A practicable approach towards secure key-based routing,” in 2007 International conference on parallel and distributed systems, pp. 1–8. https://doi.org/10.1109/ICPADS.2007.4447808

  12. Freedman MJ, Freudenthal E, and Mazières D (2004) “Democratizing content publication with coral,” in Proceedings of the 1st conference on symposium on networked systems design and implementation - Volume 1, USA, p. 18

  13. Wang L and Kangasharju J (2013) “Measuring large-scale distributed systems: case of bittorrent mainline DHT,” presented at the IEEE International conference on peer-to-peer computing,. https://doi.org/10.1109/P2P.2013.6688697

  14. Nofer M, Gomber P, Hinz O et al (2017) Blockchain. Bus Inf Syst Eng 59:183–187. https://doi.org/10.1007/s12599-017-0467-3

    Article  Google Scholar 

  15. Pilkington M (2016) Blockchain technology: principles and applications. Research hand-book on digital transformations. Edward Elgar Publishing

    Google Scholar 

  16. Mingxiao D, Ma X, Zhang Z, Wang X, and Chen Q (2017) A review on consensus algo-rithm of blockchain. IEEE International conference on systems, man, and cybernetics (SMC) 2567–2572

  17. Phillip A, Chan JS, Peiris S (2018) A new look at cryptocurrencies. Econ Lett 163:6–9

    Article  Google Scholar 

  18. Hildenbrandt E, Saxena M, Rodrigues N, Zhu X, Daian P, Guth D, Rosu G (2018) Kevm: A complete formal semantics of the ethereum virtual machine. IEEE 31st computer security foundations symposium (CSF) 204–217

  19. Luu L, Chu D H, Olickel H, Saxena P, Hobor A (2016) Making smart contracts smarter. Proceedings of the 2016 ACM SIGSAC conference on computer and communications security 254–269

  20. Christidis K, Devetsikiotis M (2016) Blockchains and smart contracts for the internet of things. IEEE Access 4:2292–2303

    Article  Google Scholar 

  21. Dannen C (2017) Introducing Ethereum and solidity, vol 318. Apress, Berkeley

    Book  Google Scholar 

  22. Dean J, Corrado GS, Monga R, Chen K, Devin M, Le QV, Mao M, Razato M, Senior A, Tucker P, Yang K, Ng AY (2012) Large scale distributed deep networks. Adv Neural Inf Process Syst 25:1223–1231

    Google Scholar 

  23. Dekel O, Gilad-Bachrach R, Shamir O, Xiao L (2012) Optimal distributed online prediction using mini-batches. J Mach Learn Res 87:165–202

    MathSciNet  MATH  Google Scholar 

  24. Li M, Andersen DG, Park JW, Smola AJ, Ahmed A, Josifovski V, Long J, Shekita EJ, Su BY (2014) Scaling distributed machine learning with the parameter server. 11th USENIX symposium on operating systems design and implementation 583–598

  25. Iandola FN, Ashraf K, Moskewicz MW, Keutzer K (2015) FireCaffe: near-linear acceleration of deep neural network training on compute clusters. http://arxiv.org/abs/1511.00175

  26. Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick RB, Guadarrama S, Darrell T (2014) Caffe: convolutional architecture for fast feature embedding. ACM international conference on multimedia 675–678

  27. Feng A, Shi J, Jain M (2016) CaffeOnSpark open sourced for distributed deep learning on big data clusters

  28. Wang Y, Zhang X, Wong I, Dai J, Zhang Y et al (2017) BigDL programming guide

  29. Dean J, Corrado GS, Monga R, Chen K, Devin M, Le QV, Mao M, Razato M, Senior A, Tucker P, Yang K, Ng AY (2012) large scale distributed deep networks

  30. Chilimbi T, Suzue Y, Apacible Y, Kalyanaraman K (2014) Project adam: building an efficient and scalable deep learning training system. 11th USENIX symposium on operating systems design and implementation 571–582

  31. Zhang S, Choromanska A, LeCun Y (2015) Deep learning with elastic averaging SGD. Adv Neural Inf Process Syst 98:685–693

    Google Scholar 

  32. Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M, Kudlur M, Levenberg J, Monga R, Moore S, Murray DG, Steiner B, Tucker P, Vasudevan V, Warden P, Wicke M, Yu Y, Zheng X (2016) TensorFlow: A system for large-scale machine learning. 12th USENIX symposium on operating systems design and implementation 265–283

  33. Moritz P, Nishihara R, Stoica I, Jordan MI (2016) SparkNet: training deep networks in spark. International conference on learning representations

  34. Lian X, Zhang C, Zhang H, Hsieh CJ, Zhang W, Liu J (2017) Can decentralized algorithms outperform centralized algorithms? A case study for decentralized parallel stochastic gradient descent. Adv Neural Inf Process Syst (NIPS) 65:871

    Google Scholar 

  35. Blot M, Picard D, Cord M, Thome N (2016) Gossip training for deep learning. http://arxiv.org/abs/1611.09726

  36. Boyd S, Ghosh A, Prabhakar B, Shah D (2006) Randomized gossip algorithms. IEEE Trans Inf Theory 52:2508–2530

    Article  MathSciNet  Google Scholar 

  37. Kim H, Park J, Jang J, Yoon S (2016) DeepSpark: Spark-based deep learning supporting asynchronous updates and caffe compatibility. http://arxiv.org/abs/1602.08191

  38. Lian X, Zhang W, Zhang C, Liu J (2018) Asynchronous decentralized parallel stochastic gradient descent international conference on machine learning (ICML)

  39. Peeters R, Schuilenburg M (2018) Machine justice: Governing security through the bureaucracy of algorithms. Inf Polity 23(3):267–280. https://doi.org/10.3233/IP-180074

    Article  Google Scholar 

  40. Kitchin R (2014) Big Data, new epistemologies and paradigm shifts. Big Data Soc. https://doi.org/10.1177/2053951714528481

    Article  Google Scholar 

  41. Veale M and Brass I (2019) administration by algorithm? public management meets public sector machine learning (2019). In: algorithmic Regulation (Karen Yeung and Martin Lodge eds., Oxford University Press, 2019), Available at SSRN: https://ssrn.com/abstract=3375391

  42. Veale M, Brass I (2019) Administration by Algorithm? Public Management meets Public Sector Machine Learning. In: Yeung K, Lodge M (eds) Algorithmic Regulation. Oxford University Press, Oxford, pp 121–149

    Chapter  Google Scholar 

  43. Mintzberg H (1980) Structure in 5’s: a synthesis of the research on organization design. Manage Sci 26:322–341

    Article  Google Scholar 

  44. Zuurmond A (1994). De infocratie: Een theoretische en empirische heroriëntatie op Weber's ideaaltype in het informatietijdperk. Den Haag: Phaedrus

  45. Lessig, Lawrence. (2006). Code: and other laws of cyberspace, Version 2.0

  46. Aneesh A (2009) Global labor: algocratic modes of organization. Sociol Theory 27(4):347–370. https://doi.org/10.1111/j.1467-9558.2009.01352.x

    Article  Google Scholar 

  47. Cormen, T. H. (2013). Algorithms unlocked

  48. Wirtz J, Patterson PG, Kunz WH, Gruber T, Lu VN, Paluch S, Martins A (2018) Brave new world: service robots in the frontline. J Serv Manag 29(5):907–931. https://doi.org/10.1108/JOSM-04-2018-0119

    Article  Google Scholar 

  49. Mohabbat Kar R, Thapa BEP, Parycek P (Hrsg.). (2018). (Un)berechenbar? Algorithmen und Automatisierung in Staat und Gesellschaft. Berlin: Fraunhofer-Institut für Offene Kommunikationssysteme FOKUS, Kompetenzzentrum Öffentliche IT (ÖFIT). https://nbn-resolving.org/urn:nbn:de:0168-ssoar-57518-2

  50. Danaher J, Hogan MJ, Noone C et al (2017) Algorithmic governance: developing a research agenda through the power of collective intelligence. Big Data Soc. https://doi.org/10.1177/2053951717726554

    Article  Google Scholar 

  51. Prusty, Narayan (2017). Building blockchain projects. Birmingham, UK: Packt. p. 9. ISBN 9781787125339

  52. The decentralized autonomous organization and governance issues regulation of Financial institutions Journal: Social Science Research Network (SSRN). 5 December 2017

  53. Vigna P, Casey MJ (2015) The age of cryptocurrency: how bitcoin and the blockchain are challenging the global economic order. St. Martin’s Press

    Google Scholar 

  54. Hodson H (2013) Bitcoin moves beyond mere money. New Sci 219:20

    Google Scholar 

  55. "The DAO of accrue: A new, automated investment fund has attracted stacks of digital money". The Economist. 21 May 2016

  56. Popper N (2016) A venture fund with plenty of virtual capital, but no capitalist. New York Times

    Google Scholar 

  57. Pangburn DJ (2015) The humans who dream of companies that won’t need us. FastCompany

    Google Scholar 

  58. Evans J (2015) Vapor no more: ethereum has launched. TechCrunch

    Google Scholar 

  59. Deegan P (2014) From bitcoin to burning man and beyond: the quest for identity and autonomy in a digital society. Institute for institutional innovation

    Google Scholar 

  60. Kevin H, David Z, Cristina N-R (2009) A survey of attack and defense techniques for reputation systems. ACM Comput Surv 42:1–31

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Applied Informatics, University of Macedonia, 54636, Thessaloniki, Greece

    Spyridon Nikolaidis & Ioannis Refanidis

Authors
  1. Spyridon Nikolaidis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ioannis Refanidis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Spyridon Nikolaidis.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nikolaidis, S., Refanidis, I. Consolidating incentivization in distributed neural network training via decentralized autonomous organization. Neural Comput & Applic 34, 19599–19613 (2022). https://doi.org/10.1007/s00521-022-07374-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07374-3

Keywords

Search

Navigation