Skip to main content

Active Inference in Hebbian Learning Networks

  • Conference paper
  • First Online:
Active Inference (IWAI 2023)

Abstract

This work studies how brain-inspired neural ensembles equipped with local Hebbian plasticity can perform active inference (AIF) in order to control dynamical agents. A generative model capturing the environment dynamics is learned by a network composed of two distinct Hebbian ensembles: a posterior network, which infers latent states given the observations, and a state transition network, which predicts the next expected latent state given current state-action pairs. Experimental studies are conducted using the Mountain Car environment from the OpenAI gym suite, to study the effect of the various Hebbian network parameters on the task performance. It is shown that the proposed Hebbian AIF approach outperforms the use of Q-learning, while not requiring any replay buffer, as in typical reinforcement learning systems. These results motivate further investigations of Hebbian learning for the design of AIF networks that can learn environment dynamics without the need for revisiting past buffered experiences.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Note that we are using our Hebbian scheme to amortize variational inference. This means we are optimizing amortization (encoding) parameters, not the (decoding) parameters of the generative model. An alternative approach would be to treat the model parameters as random variables and derive the update rules for minimizing variational free energy, in the form of a Hebbian update. In this instance, over-fitting would be precluded because of the complexity term in (6). However, this would not be amortization; this would be an implementation of AIF as described in [34].

  2. 2.

    Although we have referred to the AIF scheme as unsupervised, there is an implicit constraint on behavior that is implemented, in this instance, by the goal states in (16). In AIF, goal-directed behavior emerges from inferring the right courses of action that lead to preferred outcomes. In amortized AIF, this planning as inference is learned; as we have demonstrated. In contrast, reinforcement learning ignores inference and simply learns rewarded behaviors, which can take a very long time - because there is no learning of a generative model, or the constraints that it affords.

References

  1. Olshausen, B.A., Field, D.J.: Sparse coding with an overcomplete basis set: a strategy employed by V1? Vis. Res. 37(23), 3311–3325 (1997)

    Article  Google Scholar 

  2. Fang, M.S., Mudigonda, M., Zarcone, R., Khosrowshahi, A., Olshausen, B.: Learning and inference in sparse coding models with Langevin dynamics. Neural Comput. 34(8), 1676–1700 (2022)

    Article  MathSciNet  MATH  Google Scholar 

  3. Lee, H., Battle, A., Raina, R., Ng, A.: Efficient sparse coding algorithms. In: Advances in Neural Information Processing Systems. MIT Press (2006)

    Google Scholar 

  4. Safa, A., Ocket, I., Bourdoux, A., Sahli, H., Catthoor, F., Gielen, G.: A new look at spike-timing-dependent plasticity networks for spatio-temporal feature learning (2022)

    Google Scholar 

  5. Friston, K., Kiebel, S.: Predictive coding under the free-energy principle. Philos. Trans. R. Soc. Lond. Ser. B Biol. Sci. 364, 1211–21 (2009)

    Google Scholar 

  6. Friston, K.: Does predictive coding have a future? Nat. Neurosci. 21, 1019–1021 (2018). https://doi.org/10.1038/s41593-018-0200-7

    Article  Google Scholar 

  7. Zahid, U., Guo, Q., Fountas, Z.: Predictive coding as a neuromorphic alternative to backpropagation: a critical evaluation (2023)

    Google Scholar 

  8. Olshausen, B., Field, D.: Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381, 607–609 (1996)

    Article  Google Scholar 

  9. Safa, A., Ocket, I., Bourdoux, A., Sahli, H., Catthoor, F., Gielen, G.: Event camera data classification using spiking networks with spike-timing-dependent plasticity. In: 2022 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2022)

    Google Scholar 

  10. Bi, G., Poo, M.: Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J. Neurosci. 18(24), 10464–10472 (1998)

    Article  Google Scholar 

  11. Rao, R., Ballard, D.: Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat. Neurosci. 2(1), 79–87 (1999)

    Article  Google Scholar 

  12. Safa, A., Ocket, I., Bourdoux, A., Sahli, H., Catthoor, F., Gielen, G.G.E.: STDP-driven development of attention-based people detection in spiking neural networks. IEEE Transactions on Cognitive and Developmental Systems (2022). https://doi.org/10.1109/TCDS.2022.3210278

  13. Neftci, E., Das, S., Pedroni, B., Kreutz-Delgado, K., Cauwenberghs, G.: Event-driven contrastive divergence for spiking neuromorphic systems. Front. Neurosci. 7 (2014)

    Google Scholar 

  14. Krotov, D., Hopfield, J.J.: Unsupervised learning by competing hidden units. Proc. Natl. Acad. Sci. 116(16), 7723–7731 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  15. Parr, T., Pezzulo, G., Friston, K.: Active Inference: The Free Energy Principle in Mind, Brain, and Behavior. The MIT Press, Cambridge (2022)

    Book  Google Scholar 

  16. Isomura, T., Shimazaki, H., Friston, K.J.: Canonical neural networks perform active inference. Commun. Biol. 5, 55 (2022). https://doi.org/10.1038/s42003-021-02994-2

    Article  Google Scholar 

  17. Çatal, O., Wauthier, S., De Boom, C., Verbelen, T., Dhoedt, B.: Learning generative state space models for active inference. Front. Comput. Neurosci. 14 (2020)

    Google Scholar 

  18. Ueltzhöffer, K.: Deep active inference. Biol. Cybern. 112, 547–573 (2018). https://doi.org/10.1007/s00422-018-0785-7

    Article  MathSciNet  MATH  Google Scholar 

  19. Fountas, Z., Sajid, N., Mediano, P., Friston, K.: Deep active inference agents using Monte-Carlo methods. In: Advances in Neural Information Processing Systems, pp. 11662–11675. Curran Associates Inc. (2020)

    Google Scholar 

  20. Van de Maele, T., Verbelen, T., Çatal, O., De Boom, C., Dhoedt, B.: Active vision for robot manipulators using the free energy principle. Front. Neurorobotics 15 (2021)

    Google Scholar 

  21. Brockman, G., et al.: Openai gym (2016). arXiv preprint arXiv:1606.01540

  22. Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. The MIT Press, Cambridge (2018)

    MATH  Google Scholar 

  23. Ororbia, A.G., Mali, A.: Backprop-free reinforcement learning with active neural generative coding. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 1, pp. 29–37 (2022)

    Google Scholar 

  24. Ororbia, A., Mali, A.: Active predicting coding: brain-inspired reinforcement learning for sparse reward robotic control problems. In: IEEE International Conference on Robotics and Automation (ICRA) (2023)

    Google Scholar 

  25. Liang, Y., et al.: Can a fruit fly learn word embeddings? In: International Conference on Learning Representations (2021)

    Google Scholar 

  26. Ablin, P., Moreau, T., Massias, M., Gramfort, A.: Learning step sizes for unfolded sparse coding. In: Advances in Neural Information Processing Systems. Curran Associates Inc. (2019)

    Google Scholar 

  27. Lin, T.H., Tang, P.T.P.: Sparse Dictionary Learning by Dynamical Neural Networks. Presented at the (2019)

    Google Scholar 

  28. Friston, K.: The free-energy principle: a unified brain theory? Nat. Rev. Neurosci. 11, 127–138 (2010). https://doi.org/10.1038/nrn2787

    Article  Google Scholar 

  29. Hershey, J.R., Olsen, P.A.: Approximating the Kullback Leibler divergence between gaussian mixture models. In: 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP ’07, Honolulu, HI, USA, 2007, pp. IV-317–IV-320 (2007). https://doi.org/10.1109/ICASSP.2007.366913

  30. Kim, E., Lawson, E., Sullivan, K., Kenyon, G.: Spatiotemporal sequence memory for prediction using deep sparse coding. In: Proceedings of the 7th Annual Neuro-Inspired Computational Elements Workshop. Association for Computing Machinery (2019)

    Google Scholar 

  31. Werbos, P.J.: Backpropagation through time: what it does and how to do it. Proc. IEEE 78(10), 1550–1560 (1990). https://doi.org/10.1109/5.58337

    Article  Google Scholar 

  32. Schwartenbeck, P., FitzGerald, T., Dolan, R., Friston, K.: Exploration, novelty, surprise, and free energy minimization. Front. Psychol. 4 (2013)

    Google Scholar 

  33. https://gist.github.com/gkhayes/3d154e0505e31d6367be22ed3da2e955. Accessed 1 May 2023

  34. Friston, K.: Hierarchical models in the brain. PLoS Comput. Biol. 4(11), 1–24 (2008)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgement

This research was partially funded by a Long Stay Abroad grant from the Flemish Fund of Research - Fonds Wetenschappelijk Onderzoek (FWO) - grant V413023N. This research received funding from the Flemish Government under the “Onderzoeksprogramma Artificiële Intelligentie (AI) Vlaanderen” programme.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ali Safa .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Safa, A. et al. (2024). Active Inference in Hebbian Learning Networks. In: Buckley, C.L., et al. Active Inference. IWAI 2023. Communications in Computer and Information Science, vol 1915. Springer, Cham. https://doi.org/10.1007/978-3-031-47958-8_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-47958-8_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-47957-1

  • Online ISBN: 978-3-031-47958-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics