Skip to main content
SpringerLink
Log in

Convolutional Neural Networks with Hebbian-Based Rules in Online Transfer Learning

  • Conference paper
  • First Online:
Advances in Soft Computing (MICAI 2020)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12468))

Included in the following conference series:

Abstract

In 1949, Donald Hebb proposed its Neurophysiologic Principle, which models the weight change of connected neurons. Although similar mechanisms have been experimentally proven to exists in several brain areas, traditional Deep Learning methods do not implement it, generally using Gradient-based algorithms instead. On the other hand, Convolutional Neural Networks (CNNs) has slight inspiration on the structure of Visual Cortex, particularly on the Hierarchical model of Hubel-Wiesel. Using convolutional layers it is possible to perform feature extraction and a final classification with dense layers. In this paper, we propose a combined technique of using pre-trained convolutional layers and a final classification using Hebbian-based rules (Basic Hebb, Covariance, Oja, and BCM). Once the feature extraction is done, Hebbian rules can discriminate the classes with high accuracy. These theoretical ideas were tested using this MNIST database, reaching 99.43% of test accuracy training CNNs layers with a final Hebb layer. Similar results were found using Hebbian-based rules in other datasets with RGB images and Transfer Learning. Even when these results are slightly lower than Gradient-based methods, Hebbian learning can perform online learning, which suggests that this combined strategy might be useful to design Online Machine Learning Algorithms for Image Classification.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Hebbian Learning optimizes specific functions, see [9].

  2. 2.

    Links of the videos and full code can be found in https://github.com/Pherjev/Hebbian-CNN.

References

  1. Aguilar Canto, F.J.: Eficacia de diferentes reglas hebbianas en el aprendizaje supervisado. Tecnología Educativa Revista CONAIC 7(1), 92–97 (2020)

    Google Scholar 

  2. Amato, G., Carrara, F., Falchi, F., Gennaro, C., Lagani, G.: Hebbian learning meets deep convolutional neural networks. In: Ricci, E., Rota Bulò, S., Snoek, C., Lanz, O., Messelodi, S., Sebe, N. (eds.) ICIAP 2019. LNCS, vol. 11751, pp. 324–334. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-30642-7_29

    Chapter  Google Scholar 

  3. Bahroun, Y., Hunsicker, E., Soltoggio, A.: Building efficient deep Hebbian networks for image classification tasks. In: Lintas, A., Rovetta, S., Verschure, P.F.M.J., Villa, A.E.P. (eds.) ICANN 2017. LNCS, vol. 10613, pp. 364–372. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68600-4_42

    Chapter  Google Scholar 

  4. Bahroun, Y., Soltoggio, A.: Online representation learning with single and multi-layer Hebbian networks for image classification. In: Lintas, A., Rovetta, S., Verschure, P.F.M.J., Villa, A.E.P. (eds.) ICANN 2017. LNCS, vol. 10613, pp. 354–363. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68600-4_41

    Chapter  Google Scholar 

  5. Baldominos, A., Saez, Y., Isasi, P.: A survey of handwritten character recognition with MNIST and EMNIST. Appl. Sci. 9(15), 3169 (2019)

    Article  Google Scholar 

  6. Bienenstock, E.L., Cooper, L.N., Munro, P.W.: Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. J. Neurosci. 2(1), 32–48 (1982)

    Article  Google Scholar 

  7. Bliss, T.V., Cooke, S.F.: Long-term potentiation and long-term depression: a clinical perspective. Clinics 66, 3–17 (2011)

    Article  Google Scholar 

  8. Bromer, C., et al.: Long-term potentiation expands information content of hippocampal dentate gyrus synapses. Proc. Natl. Acad. Sci. 115(10), E2410–E2418 (2018)

    Article  Google Scholar 

  9. Choe, Y.: Hebbian Learning. Springer, Incorporated (2015)

    Google Scholar 

  10. Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 1251–1258 (2017)

    Google Scholar 

  11. Cohen, G., Afshar, S., Tapson, J., Van Schaik, A.: EMNIST: extending MNIST to handwritten letters. In: 2017 International Joint Conference on Neural Networks (IJCNN), pp. 2921–2926. IEEE (2017)

    Google Scholar 

  12. Cooper, L.N., Bear, M.F.: The BCM theory of synapse modification at 30: interaction of theory with experiment. Nat. Rev. Neurosci. 13(11), 798–810 (2012)

    Article  Google Scholar 

  13. Dayan, P., Abbott, L.: Computational neuroscience (2002)

    Google Scholar 

  14. Dorfer, M., Kelz, R., Widmer, G.: Deep linear discriminant analysis. arXiv preprint arXiv:1511.04707 (2015)

  15. Elson, J., Douceur, J.J., Howell, J., Saul, J.: Asirra: a CAPTCHA that exploits interest-aligned manual image categorization. In: Proceedings of 14th ACM Conference on Computer and Communications Security (CCS). Association for Computing Machinery, Inc., October 2007

    Google Scholar 

  16. Erhan, D., Bengio, Y., Courville, A., Vincent, P.: Visualizing higher-layer features of a deep network. Univ. Montreal 1341(3), 1 (2009)

    Google Scholar 

  17. Hahnloser, R.H., Sarpeshkar, R., Mahowald, M.A., Douglas, R.J., Seung, H.S.: Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit. Nature 405(6789), 947–951 (2000)

    Article  Google Scholar 

  18. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  19. Hebb, D.O.: The Organization of Behavior: A Neuropsychological Theory. Wiley, Chapman & Hall, New York (1949)

    Google Scholar 

  20. Holca-Lamarre, R., Lücke, J., Obermayer, K.: Models of acetylcholine and dopamine signals differentially improve neural representations. Front. Comput. Neurosci. 11, 54 (2017)

    Article  Google Scholar 

  21. Hubel, D.H., Wiesel, T.N.: Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J. Physiol. 160(1), 106 (1962)

    Article  Google Scholar 

  22. Keck, C., Savin, C., Lücke, J.: Feedforward inhibition and synaptic scaling-two sides of the same coin? PLoS Comput. Biol. 8(3), e1002432 (2012)

    Article  MathSciNet  Google Scholar 

  23. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  24. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)

    Google Scholar 

  25. Kuśmierz, Ł., Isomura, T., Toyoizumi, T.: Learning with three factors: modulating Hebbian plasticity with errors. Curr. Opin. Neurobiol. 46, 170–177 (2017)

    Article  Google Scholar 

  26. Kussul, E., Baidyk, T., Kasatkina, L., Lukovich, V.: Rosenblatt perceptrons for handwritten digit recognition. In: International Joint Conference on Neural Networks. Proceedings (Cat. No. 01CH37222), IJCNN 2001, vol. 2, pp. 1516–1520. IEEE (2001)

    Google Scholar 

  27. Law, C.C., Cooper, L.N.: Formation of receptive fields in realistic visual environments according to the Bienenstock, Cooper, and Munro (BCM) theory. Proc. Natl. Acad. Sci. 91(16), 7797–7801 (1994)

    Article  Google Scholar 

  28. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  29. LeCun, Y., Cortes, C., Burges, C.: Mnist handwritten digit database (2010) http://yann.lecun.com/exdb/mnist 7, 23 (2010)

  30. Li, G., Liu, M., Dong, M.: A new online learning algorithm for structure-adjustable extreme learning machine. Comput. Math. Appl. 60(3), 377–389 (2010)

    Article  MathSciNet  Google Scholar 

  31. Liu, D., Yue, S.: Visual pattern recognition using unsupervised spike timing dependent plasticity learning. In: 2016 International Joint Conference on Neural Networks (IJCNN), pp. 285–292. IEEE (2016)

    Google Scholar 

  32. Lomo, T.: Frequency potentiation of excitatory synaptic activity in dentate area of hippocampal formation. In: Acta Physiologica Scandinavica. p. 128. BLACKWELL SCIENCE LTD PO BOX 88, OSNEY MEAD, OXFORD OX2 0NE, OXON, ENGLAND (1966)

    Google Scholar 

  33. Lüscher, C., Malenka, R.C.: NMDA receptor-dependent long-term potentiation and long-term depression (LTP/LTD). Cold Spring Harb. Perspect. Biol. 4(6), a005710 (2012)

    Article  Google Scholar 

  34. Magotra, A., kim, J.: Transfer learning for image classification using Hebbian plasticity principles. In: Proceedings of the 2019 3rd International Conference on Computer Science and Artificial Intelligence, pp. 233–238 (2019)

    Google Scholar 

  35. Mannor, S., Peleg, D., Rubinstein, R.: The cross entropy method for classification. In: Proceedings of the 22nd international conference on Machine Learning, pp. 561–568 (2005)

    Google Scholar 

  36. Nicoll, R.A.: A brief history of long-term potentiation. Neuron 93(2), 281–290 (2017)

    Article  Google Scholar 

  37. Nocedal, J., Wright, S.: Numerical Optimization. Springer, Heidelberg (2006)

    MATH  Google Scholar 

  38. Oja, E.: Simplified neuron model as a principal component analyzer. J. Math. Biol. 15(3), 267–273 (1982)

    Article  MathSciNet  Google Scholar 

  39. Palacios-Filardo, J., Mellor, J.R.: Neuromodulation of hippocampal long-term synaptic plasticity. Curr. Opin. Neurobiol. 54, 37–43 (2019)

    Article  Google Scholar 

  40. Rosenblatt, F.: The perceptron: a probabilistic model for information storage and organization in the brain. Psychol. Rev. 65(6), 386 (1958)

    Article  Google Scholar 

  41. Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vision 115(3), 211–252 (2015)

    Article  MathSciNet  Google Scholar 

  42. Sahoo, D., Pham, Q., Lu, J., Hoi, S.C.: Online deep learning: learning deep neural networks on the fly. arXiv preprint arXiv:1711.03705 (2017)

  43. Seeliger, K., et al.: Convolutional neural network-based encoding and decoding of visual object recognition in space and time. NeuroImage 180, 253–266 (2018)

    Article  Google Scholar 

  44. Sejnowski, T.J., Tesauro, G.: Building network learning algorithms from Hebbian synapses. In: Brain Organization and Memory: Cells, Systems, and Circuits, pp. 338–355. Oxford University Press, New York (1989)

    Google Scholar 

  45. Serre, T.: Hierarchical Models of the Visual System. Springer Publishing Company, Incorporated (2015)

    Google Scholar 

  46. Shim, H.G., et al.: Long-term depression of intrinsic excitability accompanied by synaptic depression in cerebellar Purkinje cells. J. Neurosci. 37(23), 5659–5669 (2017)

    Article  Google Scholar 

  47. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  48. Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.A.: Inception-v4, inception-resnet and the impact of residual connections on learning. In: Thirty-First AAAI Conference on Artificial Intelligence (2017)

    Google Scholar 

  49. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826 (2016)

    Google Scholar 

  50. Wadhwa, A., Madhow, U.: Bottom-up deep learning using the Hebbian principle (2016)

    Google Scholar 

Download references

Acknowledgments

I would like to thank Carlos Francisco Brito-Loeza for his comments and support.

Author information

Authors and Affiliations

  1. Facultad de Matemáticas, Universidad Autónoma de Yucatán, Anillo Periférico Norte, Tablaje Cat. 13615, Colonia Chuburná Hidalgo Inn, Mérida, Yucatán, Mexico

    Fernando Javier Aguilar Canto

Authors
  1. Fernando Javier Aguilar Canto
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fernando Javier Aguilar Canto .

Editor information

Editors and Affiliations

  1. Facultad de Ingeniería, Universidad Panamericana, Mexico City, Mexico

    Lourdes Martínez-Villaseñor

  2. Universidad Autónoma Metropolitana, Mexico City, Mexico

    Oscar Herrera-Alcántara

  3. Facultad de Ingeniería, Universidad Panamericana, Mexico City, Mexico

    Hiram Ponce

  4. Universidad Autónoma del Estado de Hidalgo, Hidalgo, Mexico

    Félix A. Castro-Espinoza

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Aguilar Canto, F.J. (2020). Convolutional Neural Networks with Hebbian-Based Rules in Online Transfer Learning. In: Martínez-Villaseñor, L., Herrera-Alcántara, O., Ponce, H., Castro-Espinoza, F.A. (eds) Advances in Soft Computing. MICAI 2020. Lecture Notes in Computer Science(), vol 12468. Springer, Cham. https://doi.org/10.1007/978-3-030-60884-2_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-60884-2_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-60883-5

  • Online ISBN: 978-3-030-60884-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation