Skip to main content

Advertisement

Log in

Vector-Valued Hopfield Neural Networks and Distributed Synapse Based Convolutional and Linear Time-Variant Associative Memories

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

The Hopfield network is an example of an artificial neural network used to implement associative memories. A binary digit represents the neuron’s state of a traditional Hopfield neural network. Inspired by the human brain’s ability to cope simultaneously with multiple sensorial inputs, this paper presents three multi-modal Hopfield-type neural networks that treat multi-dimensional data as a single entity. In the first model, called the vector-valued Hopfield neural network, the neuron’s state is a vector of binary digits. Synaptic weights are modeled as finite impulse response (FIR) filters in the second model, yielding the so-called convolutional associative memory. Finally, the synaptic weights are modeled by linear time-varying (LTV) filters in the third model. Besides their potential applications for multi-modal intelligence, the new associative memories may also be used for signal and image processing and solve optimization and classification tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Lecun Y (1989). In: Pfeifer R, Schreter Z, Fogelman F, Steels L (eds) Generalization and network design strategies. Elsevier, Amsterdam

    Google Scholar 

  2. Géron A (2017) Hands-on machine learning with scikit-learn and tensorflow: concepts, tools, and techniques to build intelligent systems. O’Reilly Media, Incorporated

  3. Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press. http://www.deeplearningbook.org

  4. Schmidhuber J (2015) Deep Learning in neural networks: An overview. Elsevier Ltd. https://doi.org/10.1016/j.neunet.2014.09.003

  5. Zhao L, Song Y, Zhang C, Liu Y, Wang P, Lin T, Deng M, Li H (2020) T-GCN: a temporal graph convolutional network for traffic prediction. IEEE Trans Intell Transp Syst 21(9):3848–3858. https://doi.org/10.1109/TITS.2019.2935152

    Article  Google Scholar 

  6. Li C, Dong Z, Chen G, Zhou B, Zhang J, Yu X (2021) Data-driven planning of electric vehicle charging infrastructure: a case study of Sydney. IEEE transactions on smart grid, Australia. https://doi.org/10.1109/TSG.2021.3054763

  7. Garimella RM (2008) Finite impulse response (FIR) filter model of synapses: Associated neural networks. In: 2008 fourth international conference on natural computation, vol. 2, pp 370–374

  8. Zhang C, Yang Z, He X, Deng L (2020) Multimodal intelligence: representation learning, information fusion, and applications. IEEE J Selected Topics Signal Process 14(3):478–493. https://doi.org/10.1109/JSTSP.2020.2987728

    Article  Google Scholar 

  9. Kohonen T (1987) Self-organization and associative memory, 2nd edn. Springer, New York

    MATH  Google Scholar 

  10. Hassoun MH, Watta PB (1997) Associative memory networks. In: Fiesler E, Beale R (eds) Handbook of neural computation. Oxford University Press, Oxford, pp 1–311314

    Google Scholar 

  11. Gan J (2017) Discrete Hopfield neural network approach for crane safety evaluation. In: 2017 international conference on mechanical, system and control engineering (ICMSC), pp 40–43. https://doi.org/10.1109/ICMSC.2017.7959439

  12. Song Y, Xing B, Guo L, Xu X (2017) System parameter identification experiment based on Hopfield neural network for self balancing vehicle. In: 2017 36th Chinese control conference (CCC), pp 6887–6890. https://doi.org/10.23919/ChiCC.2017.8028442

  13. Wang Q, Shi W, Atkinson PM, Li Z (2015) Land cover change detection at subpixel resolution with a Hopfield neural network. IEEE J Selected Topics Appl Earth Observ Remote Sens 8(3):1339–1352. https://doi.org/10.1109/JSTARS.2014.2355832

    Article  Google Scholar 

  14. Li J, Li X, Huang B, Zhao L (2016) Hopfield neural network approach for supervised nonlinear spectral unmixing. IEEE Geosci Remote Sens Lett 13(7):1002–1006. https://doi.org/10.1109/LGRS.2016.2560222

    Article  Google Scholar 

  15. Valle ME, Lobo RA (2021) Hypercomplex-valued recurrent correlation neural networks. Neurocomputing 432:111–123. https://doi.org/10.1016/j.neucom.2020.12.034

    Article  Google Scholar 

  16. Pajares G, Guijarro M, Ribeiro A (2010) A Hopfield neural network for combining classifiers applied to textured images. Neural Netw 23(1):144–153. https://doi.org/10.1016/j.neunet.2009.07.019

    Article  Google Scholar 

  17. Zhang H, Hou Y, Zhao J, Wang L, Xi T, Li Y (2017) Automatic welding quality classification for the spot welding based on the Hopfield associative memory neural network and chernoff face description of the electrode displacement signal features. Mech Syst Signal Process 85:1035–1043. https://doi.org/10.1016/j.ymssp.2016.06.036

    Article  Google Scholar 

  18. Lobo RA, Valle ME (2020) Ensemble of binary classifiers combined using recurrent correlation associative memories vol. 12320 LNAI

  19. Hopfield JJ, Tank DW (1985) Neural computation of decisions in optimization problems. Biol Cybern 52:141–152

    Article  MATH  Google Scholar 

  20. Serpen G (2008) Hopfield network as static optimizer: learning the weights and eliminating the guesswork. Neural Process Lett 27(1):1–15. https://doi.org/10.1007/s11063-007-9055-8

    Article  Google Scholar 

  21. Li C, Yu X, Huang T, Chen G, He X (2015) A generalized Hopfield network for nonsmooth constrained convex optimization: Lie derivative approach. IEEE Trans Neural Netw Learn Syst 27:1–14. https://doi.org/10.1109/TNNLS.2015.2496658

    Article  MathSciNet  Google Scholar 

  22. Hopfield JJ (1982) Neural networks and physical systems with emergent collective computational abilities. Proc National Acad Sci 79:2554–2558

    Article  MathSciNet  MATH  Google Scholar 

  23. Smith K, Palaniswami M, Krishnamoorthy M (1998) Neural techniques for combinatorial optimization with applications. IEEE Trans Neural Netw 9(6):1301–1318

    Article  Google Scholar 

  24. Li C, Yu X, Huang T, Chen G, He X (2015) A generalized Hopfield network for nonsmooth constrained convex optimization: lie derivative approach. IEEE Trans Neural Netw Learn Syst 27:1–14. https://doi.org/10.1109/TNNLS.2015.2496658

    Article  MathSciNet  Google Scholar 

  25. Pajares G (2006) A Hopfield neural network for image change detection. IEEE Trans Neural Netw 17(5):1250–1264

    Article  Google Scholar 

  26. Shen D, Ip HHS (1997) A Hopfield neural network for adaptive image segmentation: An active surface paradigm1. Pattern Recogn Lett 18(1):37–48. https://doi.org/10.1016/S0167-8655(96)00117-1

    Article  Google Scholar 

  27. Widrich M, Schäfl B, Ramsauer H, Pavlović M, Gruber L, Holzleitner M, Brandstetter J, Sandve GK, Greiff V, Hochreiter S, Klambauer G (2020) Modern Hopfield networks and attention for immune repertoire classification

  28. Ramsauer H, Schäfl B, Lehner J, Seidl P, Widrich M, Gruber L, Holzleitner M, Pavlović M, Sandve G.K, Greiff V, Kreil D, Kopp M, Klambauer G, Brandstetter J, Hochreiter S (2020) Hopfield networks is all you need

  29. Oppenheim AV, Schafer RW (1989) Discrete-time signal processing. Prentice-Hall, Englewood Cliffs, NJ

    MATH  Google Scholar 

  30. Haykin S (2009) Neural networks and learning machines, 3rd edn. Prentice-Hall, Upper Saddle River, NJ

    Google Scholar 

  31. Aizenberg IN (2011) Complex-valued neural networks with multi-valued neurons. studies in computational intelligence, vol. 353. Springer, Berlin Heidelberg. https://doi.org/10.1007/978-3-642-20353-4

  32. Hirose A (2012) Complex-valued neural networks, 2nd edn. Studies in Computational Intelligence. Springer, Heidelberg, Germany

    Book  MATH  Google Scholar 

  33. Vieira G, Valle ME (2022) A general framework for hypercomplex-valued extreme learning machines. J Comput Math Data Sci 3:100032. https://doi.org/10.1016/J.JCMDS.2022.100032

    Article  Google Scholar 

  34. Aizenberg I, Gonzalez A (2018) Image recognition using mlmvn and frequency domain features. In: 2018 international joint conference on neural networks (IJCNN), pp. 1–8. https://doi.org/10.1109/IJCNN.2018.8489301

  35. Ding T, Hirose A (2020) Online regularization of complex-valued neural networks for structure optimization in wireless-communication channel prediction. IEEE Access 8:143706–143722. https://doi.org/10.1109/ACCESS.2020.3013940

    Article  Google Scholar 

  36. Greenblatt AB, Agaian SS (2018) Introducing quaternion multi-valued neural networks with numerical examples. Inf Sci 423:326–342. https://doi.org/10.1016/j.ins.2017.09.057

    Article  MathSciNet  Google Scholar 

  37. Hongo S, Isokawa T, Matsui N, Nishimura H, Kamiura N (2020) Constructing convolutional neural networks based on quaternion. In: Proceedings of the International joint conference on neural networks. Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/IJCNN48605.2020.9207325

  38. Keohane O, Aizenberg I (2020) Impulse noise filtering using MLMVN. In: Proceedings of the international joint conference on neural networks. institute of electrical and electronics engineers Inc. https://doi.org/10.1109/IJCNN48605.2020.9207691

  39. Kinugawa K, Shang F, Usami N, Hirose A (2018) Isotropization of quaternion-neural-network-based PolSAR adaptive land classification in poincare-sphere parameter space. IEEE Geosci Remote Sens Lett 15(8):1234–1238. https://doi.org/10.1109/LGRS.2018.2831215

    Article  Google Scholar 

  40. Sunaga Y, Natsuaki R, Hirose A (2020) Similar land-form discovery: Complex absolute-value max pooling in complex-valued convolutional neural networks in interferometric synthetic aperture radar. In: Proceedings of the international joint conference on neural networks. institute of electrical and electronics engineers Inc. https://doi.org/10.1109/IJCNN48605.2020.9207122

  41. Valous NA, Moraleda RR, Jäger D, Zörnig I, Halama N (2020) Interrogating the microenvironmental landscape of tumors with computational image analysis approaches. Seminars in Immunology 101411. https://doi.org/10.1016/j.smim.2020.101411

  42. Wang G, Xue R (2019) Quaternion filtering based on quaternion involutions and its application in signal processing. IEEE Access 7:149068–149079. https://doi.org/10.1109/ACCESS.2019.2944666

    Article  Google Scholar 

  43. Granero MA, Hernández CX, Valle ME (2021) Quaternion-valued convolutional neural network applied for acute lymphoblastic leukemia diagnosis. In: Brazilian conference on intelligent systems. BRACIS 2021. Lecture Notes in Computer Science, pp. 280–293. Springer

  44. Garimella RM (2006) Some novel real/complex-valued neural network models. In: Reusch B (ed) Computational intelligence, theory and applications. Springer, Berlin, pp 473–483

    Chapter  Google Scholar 

  45. Garimella RM (2007) Multi dimensional neural networks-unified theory, p. 168. New Age International Pvt Ltd Publishers

  46. Isokawa T, Nishimura H, Matsui N (2013) Quaternionic neural networks for associative memories. In: Hirose A (ed) Complex-valued neural networks. Wiley, New York, pp 103–131. https://doi.org/10.1002/9781118590072.ch5

    Chapter  Google Scholar 

  47. Kobayashi M (2020) Hyperbolic-valued hopfield neural networks in synchronous mode. Neural Comput 32(9):1–12

    Article  MathSciNet  MATH  Google Scholar 

  48. Kobayashi M (2020) Noise robust projection rule for hyperbolic hopfield neural networks. IEEE Trans Neural Netw Learn Syst 31(1):352–356. https://doi.org/10.1109/TNNLS.2019.2899914

    Article  MathSciNet  Google Scholar 

  49. Kobayashi M (2020) Hopfield neural networks using klein four-group. Neurocomputing. https://doi.org/10.1016/j.neucom.2019.12.127

  50. Kobayashi M (2020) Two-level complex-valued hopfield neural networks. IEEE Trans Neural Netw Learn Syst, 1–5. https://doi.org/10.1109/tnnls.2020.2995413

  51. Minemoto T, Isokawa T, Nishimura H, Matsui N (2016) Quaternionic multistate Hopfield neural network with extended projection rule. Artif Life Robot 21(1):106–111. https://doi.org/10.1007/s10015-015-0247-4

    Article  Google Scholar 

  52. Valle ME, Castro FZ (2018) On the dynamics of Hopfield neural networks on unit quaternions. IEEE Trans Neural Netw Learn Syst 29(6):2464–2471. https://doi.org/10.1109/TNNLS.2017.2691462

    Article  Google Scholar 

  53. de Castro FZ, Valle ME (2020) A broad class of discrete-time hypercomplex-valued Hopfield neural networks. Neural Netw 122:54–67. https://doi.org/10.1016/j.neunet.2019.09.040

    Article  MATH  Google Scholar 

  54. Popa C-A (2016) Matrix-valued Hopfield neural networks. In: Cheng L, Liu Q, Ronzhin A (eds.) Advances in neural networks – ISNN 2016: 13th international symposium on neural networks, ISNN 2016, St. Petersburg, Russia, July 6-8, 2016, Proceedings, pp. 127–134. Springer, Cham. https://doi.org/10.1007/978-3-319-40663-3_15

  55. Karbasi A, Salavati AH, Shokrollahi A (2014) Convolutional neural associative memories: massive capacity with noise tolerance. arXiv: 1407.6513

  56. Bruck J, Goodman JW (1988) A generalized convergence theorem for neural networks. IEEE Trans Inf Theory 34(5):1089–1092

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the International Institute of Information Technology–Hyderabad, National Council for Scientific and Technological Development (CNPq) under Grant No 310118/2017-4, the São Paulo Research Foundation (FAPESP) under Grant No 2019/02278-2, and the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Marcos Eduardo Valle.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Garimella, R.M., Valle, M.E., Vieira, G. et al. Vector-Valued Hopfield Neural Networks and Distributed Synapse Based Convolutional and Linear Time-Variant Associative Memories. Neural Process Lett 55, 4163–4182 (2023). https://doi.org/10.1007/s11063-022-11035-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-022-11035-w

Keywords

Navigation