Abstract
Spiking neural network (SNN) has been shown to be a biologically plausible and energy efficient alternative to Deep Neural Network in Reinforcement Learning (RL). In prevailing SNN models for RL, fully-connected architectures with inter-layer connections are commonly employed. However, the incorporation of intra-layer connections is neglected, which impedes the feature representation and information processing capacities of SNNs in the context of reinforcement learning. To address these limitations, we propose Lateral Interactions Spiking Actor Network (LISAN) to improve decision-making in reinforcement learning tasks with high performance. LISAN integrates lateral interactions between neighboring neurons into the spiking neuron membrane potential equation. Moreover, we incorporate soft reset mechanism to enhance model’s functionality recognizing the significance of residual potentials in preserving valuable information within biological neurons. To verify the effectiveness of our proposed framework, LISAN is evaluated using four continuous control tasks from OpenAI gym as well as different encoding methods. The results show that LISAN substantially improves the performance compared to state-of-the-art models. We hope that our work will contribute to a deeper understanding of the mechanisms involved in information capturing and processing in the brain.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Arulkumaran, K., Deisenroth, M.P., Brundage, M., Bharath, A.A.: A brief survey of deep reinforcement learning. arXiv preprint arXiv:1708.05866 (2017)
Cheng, X., Hao, Y., Xu, J., Xu, B.: Lisnn: Improving spiking neural networks with lateral interactions for robust object recognition. In: IJCAI, pp. 1519–1525 (2020)
Evanusa, M., Sandamirskaya, Y., et al.: Event-based attention and tracking on neuromorphic hardware. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (2019)
Florian, R.V.: Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Comput. 19(6), 1468–1502 (2007)
Frémaux, N., Sprekeler, H., Gerstner, W.: Reinforcement learning using a continuous time actor-critic framework with spiking neurons. PLoS Comput. Biol. 9(4), e1003024 (2013)
Han, B., Srinivasan, G., Roy, K.: RMP-SNN: residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13558–13567 (2020)
Harris, K.D., Csicsvari, J., Hirase, H., Dragoi, G., Buzsáki, G.: Organization of cell assemblies in the hippocampus. Nature 424(6948), 552–556 (2003)
Hu, S., Zhu, F., Chang, X., Liang, X.: UPDeT: universal multi-agent reinforcement learning via policy decoupling with transformers. arXiv preprint arXiv:2101.08001 (2021)
Lahijanian, M., et al.: Resource-performance tradeoff analysis for mobile robots. IEEE Robot. Autom. Lett. 3(3), 1840–1847 (2018)
Memmesheimer, R.M., Rubin, R., Ölveczky, B.P., Sompolinsky, H.: Learning precisely timed spikes. Neuron 82(4), 925–938 (2014)
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)
Niroui, F., Zhang, K., Kashino, Z., Nejat, G.: Deep reinforcement learning robot for search and rescue applications: exploration in unknown cluttered environments. IEEE Robot. Autom. Lett. 4(2), 610–617 (2019)
Ratliff, F., Hartline, H.K., Lange, D.: The dynamics of lateral inhibition in the compound eye of limulus. I. Studies on Excitation and Inhibition in the Retina: A Collection of Papers from the Laboratories of H. Keffer Hartline, p. 463 (1974)
Sandamirskaya, Y.: Dynamic neural fields as a step toward cognitive neuromorphic architectures. Front. Neurosci. 7, 276 (2014)
Schöner, G., Spencer, J.P.: Dynamic Thinking: A Primer on Dynamic Field Theory. Oxford University Press, Oxford (2016)
Silver, D., et al.: Mastering the game of go with deep neural networks and tree search. Nature 529(7587), 484–489 (2016)
Tang, G., Kumar, N., Yoo, R., Michmizos, K.: Deep reinforcement learning with population-coded spiking neural network for continuous control. In: Conference on Robot Learning, pp. 2016–2029. PMLR (2021)
Wu, Z., et al.: Modeling learnable electrical synapse for high precision spatio-temporal recognition. Neural Netw. 149, 184–194 (2022)
Yuan, M., Wu, X., Yan, R., Tang, H.: Reinforcement learning in spiking neural networks with stochastic and deterministic synapses. Neural Comput. 31(12), 2368–2389 (2019)
Zhang, D., Zhang, T., Jia, S., Xu, B.: Multi-sacle dynamic coding improved spiking actor network for reinforcement learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 59–67 (2022)
Acknowledgements
This work was supported by the National Natural Science Foundation of China (Grant No. 62206188), the China Postdoctoral Science Foundation (Grant No. 2022M712237), Sichuan Province Innovative Talent Funding Project for Postdoctoral Fellows and the 111 Project under grant B21044.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Chen, X., Xiao, R., Yang, Q., Lv, J. (2024). Lateral Interactions Spiking Actor Network for Reinforcement Learning. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Communications in Computer and Information Science, vol 1962. Springer, Singapore. https://doi.org/10.1007/978-981-99-8132-8_14
Download citation
DOI: https://doi.org/10.1007/978-981-99-8132-8_14
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8131-1
Online ISBN: 978-981-99-8132-8
eBook Packages: Computer ScienceComputer Science (R0)