Abstract
This work studies how brain-inspired neural ensembles equipped with local Hebbian plasticity can perform active inference (AIF) in order to control dynamical agents. A generative model capturing the environment dynamics is learned by a network composed of two distinct Hebbian ensembles: a posterior network, which infers latent states given the observations, and a state transition network, which predicts the next expected latent state given current state-action pairs. Experimental studies are conducted using the Mountain Car environment from the OpenAI gym suite, to study the effect of the various Hebbian network parameters on the task performance. It is shown that the proposed Hebbian AIF approach outperforms the use of Q-learning, while not requiring any replay buffer, as in typical reinforcement learning systems. These results motivate further investigations of Hebbian learning for the design of AIF networks that can learn environment dynamics without the need for revisiting past buffered experiences.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Note that we are using our Hebbian scheme to amortize variational inference. This means we are optimizing amortization (encoding) parameters, not the (decoding) parameters of the generative model. An alternative approach would be to treat the model parameters as random variables and derive the update rules for minimizing variational free energy, in the form of a Hebbian update. In this instance, over-fitting would be precluded because of the complexity term in (6). However, this would not be amortization; this would be an implementation of AIF as described in [34].
- 2.
Although we have referred to the AIF scheme as unsupervised, there is an implicit constraint on behavior that is implemented, in this instance, by the goal states in (16). In AIF, goal-directed behavior emerges from inferring the right courses of action that lead to preferred outcomes. In amortized AIF, this planning as inference is learned; as we have demonstrated. In contrast, reinforcement learning ignores inference and simply learns rewarded behaviors, which can take a very long time - because there is no learning of a generative model, or the constraints that it affords.
References
Olshausen, B.A., Field, D.J.: Sparse coding with an overcomplete basis set: a strategy employed by V1? Vis. Res. 37(23), 3311–3325 (1997)
Fang, M.S., Mudigonda, M., Zarcone, R., Khosrowshahi, A., Olshausen, B.: Learning and inference in sparse coding models with Langevin dynamics. Neural Comput. 34(8), 1676–1700 (2022)
Lee, H., Battle, A., Raina, R., Ng, A.: Efficient sparse coding algorithms. In: Advances in Neural Information Processing Systems. MIT Press (2006)
Safa, A., Ocket, I., Bourdoux, A., Sahli, H., Catthoor, F., Gielen, G.: A new look at spike-timing-dependent plasticity networks for spatio-temporal feature learning (2022)
Friston, K., Kiebel, S.: Predictive coding under the free-energy principle. Philos. Trans. R. Soc. Lond. Ser. B Biol. Sci. 364, 1211–21 (2009)
Friston, K.: Does predictive coding have a future? Nat. Neurosci. 21, 1019–1021 (2018). https://doi.org/10.1038/s41593-018-0200-7
Zahid, U., Guo, Q., Fountas, Z.: Predictive coding as a neuromorphic alternative to backpropagation: a critical evaluation (2023)
Olshausen, B., Field, D.: Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381, 607–609 (1996)
Safa, A., Ocket, I., Bourdoux, A., Sahli, H., Catthoor, F., Gielen, G.: Event camera data classification using spiking networks with spike-timing-dependent plasticity. In: 2022 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2022)
Bi, G., Poo, M.: Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J. Neurosci. 18(24), 10464–10472 (1998)
Rao, R., Ballard, D.: Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat. Neurosci. 2(1), 79–87 (1999)
Safa, A., Ocket, I., Bourdoux, A., Sahli, H., Catthoor, F., Gielen, G.G.E.: STDP-driven development of attention-based people detection in spiking neural networks. IEEE Transactions on Cognitive and Developmental Systems (2022). https://doi.org/10.1109/TCDS.2022.3210278
Neftci, E., Das, S., Pedroni, B., Kreutz-Delgado, K., Cauwenberghs, G.: Event-driven contrastive divergence for spiking neuromorphic systems. Front. Neurosci. 7 (2014)
Krotov, D., Hopfield, J.J.: Unsupervised learning by competing hidden units. Proc. Natl. Acad. Sci. 116(16), 7723–7731 (2019)
Parr, T., Pezzulo, G., Friston, K.: Active Inference: The Free Energy Principle in Mind, Brain, and Behavior. The MIT Press, Cambridge (2022)
Isomura, T., Shimazaki, H., Friston, K.J.: Canonical neural networks perform active inference. Commun. Biol. 5, 55 (2022). https://doi.org/10.1038/s42003-021-02994-2
Çatal, O., Wauthier, S., De Boom, C., Verbelen, T., Dhoedt, B.: Learning generative state space models for active inference. Front. Comput. Neurosci. 14 (2020)
Ueltzhöffer, K.: Deep active inference. Biol. Cybern. 112, 547–573 (2018). https://doi.org/10.1007/s00422-018-0785-7
Fountas, Z., Sajid, N., Mediano, P., Friston, K.: Deep active inference agents using Monte-Carlo methods. In: Advances in Neural Information Processing Systems, pp. 11662–11675. Curran Associates Inc. (2020)
Van de Maele, T., Verbelen, T., Çatal, O., De Boom, C., Dhoedt, B.: Active vision for robot manipulators using the free energy principle. Front. Neurorobotics 15 (2021)
Brockman, G., et al.: Openai gym (2016). arXiv preprint arXiv:1606.01540
Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. The MIT Press, Cambridge (2018)
Ororbia, A.G., Mali, A.: Backprop-free reinforcement learning with active neural generative coding. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 1, pp. 29–37 (2022)
Ororbia, A., Mali, A.: Active predicting coding: brain-inspired reinforcement learning for sparse reward robotic control problems. In: IEEE International Conference on Robotics and Automation (ICRA) (2023)
Liang, Y., et al.: Can a fruit fly learn word embeddings? In: International Conference on Learning Representations (2021)
Ablin, P., Moreau, T., Massias, M., Gramfort, A.: Learning step sizes for unfolded sparse coding. In: Advances in Neural Information Processing Systems. Curran Associates Inc. (2019)
Lin, T.H., Tang, P.T.P.: Sparse Dictionary Learning by Dynamical Neural Networks. Presented at the (2019)
Friston, K.: The free-energy principle: a unified brain theory? Nat. Rev. Neurosci. 11, 127–138 (2010). https://doi.org/10.1038/nrn2787
Hershey, J.R., Olsen, P.A.: Approximating the Kullback Leibler divergence between gaussian mixture models. In: 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP ’07, Honolulu, HI, USA, 2007, pp. IV-317–IV-320 (2007). https://doi.org/10.1109/ICASSP.2007.366913
Kim, E., Lawson, E., Sullivan, K., Kenyon, G.: Spatiotemporal sequence memory for prediction using deep sparse coding. In: Proceedings of the 7th Annual Neuro-Inspired Computational Elements Workshop. Association for Computing Machinery (2019)
Werbos, P.J.: Backpropagation through time: what it does and how to do it. Proc. IEEE 78(10), 1550–1560 (1990). https://doi.org/10.1109/5.58337
Schwartenbeck, P., FitzGerald, T., Dolan, R., Friston, K.: Exploration, novelty, surprise, and free energy minimization. Front. Psychol. 4 (2013)
https://gist.github.com/gkhayes/3d154e0505e31d6367be22ed3da2e955. Accessed 1 May 2023
Friston, K.: Hierarchical models in the brain. PLoS Comput. Biol. 4(11), 1–24 (2008)
Acknowledgement
This research was partially funded by a Long Stay Abroad grant from the Flemish Fund of Research - Fonds Wetenschappelijk Onderzoek (FWO) - grant V413023N. This research received funding from the Flemish Government under the “Onderzoeksprogramma Artificiële Intelligentie (AI) Vlaanderen” programme.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Safa, A. et al. (2024). Active Inference in Hebbian Learning Networks. In: Buckley, C.L., et al. Active Inference. IWAI 2023. Communications in Computer and Information Science, vol 1915. Springer, Cham. https://doi.org/10.1007/978-3-031-47958-8_15
Download citation
DOI: https://doi.org/10.1007/978-3-031-47958-8_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-47957-1
Online ISBN: 978-3-031-47958-8
eBook Packages: Computer ScienceComputer Science (R0)