Challenges for large-scale implementations of spiking neural networks on FPGAs
Introduction
Inspired by the way the brain processes information, scientists and engineers have been researching neural networks (NNs) since the early 1940s [44]. NNs are an information processing paradigm inspired by the way biological nervous systems, such as the brain, process information. The key element of this paradigm is the novel structure of the information processing system. It is composed of a large number of highly interconnected processing elements, neurons, working in parallel to solve a specific problem.
The neuron is made up of four main parts; dendrites, synapses, axon and the cell body. A neuron is essentially a system that accepts electrical currents which arrive on its dendrites. It sums these and if they exceed a certain threshold it issues a new pulse which propagates along an axon. The information is transmitted from an axon to a dendrite via a synapse, which is done by means of chemical neurotransmitters across the synaptic membrane. An illustration of a biological neuron can be found in Fig. 1.
Research into artificial neural networks (ANNs), has seen the development of a plethora of neuron models from the initial McCulloch and Pitts concept [44] to the more biologically realistic spike models [77]. Recent trends in computational intelligence have indicated a strong tendency towards forming a better understanding of biological systems and the details of neuronal signal processing [31], [39], [62], [63]. Such research is motivated by the desire to form a more comprehensive understanding of information processing in biological networks and to investigate how this understanding could be used to improve traditional information processing techniques [41], [67]. Spiking neurons differ from conventional ANN models as information is transmitted by means of spikes rather than by firing rates [7], [22], [27], [55], [60]. It is believed that this allows spiking neurons to have richer dynamics as they can exploit the temporal domain to encode or decode data in the form of spike trains [58], [59]. However, this has demanded the development of new learning rules drawing again on inspiration from biology. For example, Hebbian learning has been identified as a closely biologically related learning rule [28] and more recent research has reported a spike timing dependent variation of this rule called spike timing dependent plasticity (STDP) [4], [51], [52], [70] which modulates the synaptic efficiency of synapses.
Software simulation of network topologies and connection strategies provides a platform for the investigation of how arrays of spiking neurons can be used to solve computational tasks. Such simulations face the problem of scalability in that biological systems are inherently parallel in their architecture whereas commercial PCs are based on the sequential Von Neumann serial processing architecture. Thus, it is difficult to assess the efficiency of these models to solve complex problems [10]. When implemented on parallel hardware, NNs can take full advantage of their inherent parallelism and run orders of magnitude faster than software simulations, thus becoming, appropriate for real-time applications. Developing custom application-specific integrated circuit (ASIC) devices for NNs however is both time consuming and expensive. These devices are also inflexible in that a modification of the basic neuron model would require a new development cycle to be undertaken. Field programmable gate arrays (FPGAs) are devices that permit the implementation of digital systems, providing an array of logic components that can be configured in a desired way by a configuration bitstream [72]. The device is reconfigurable such that a change to the system is easily achieved and an updated configuration bitstream can be downloaded to the device. Previous work has indicated that these devices provide a suitable platform for the implementation of conventional AANs [5], [19].
This paper aims to report on the issues arising from the authors’ experience in implementing spiking neural networks on reconfigurable hardware. This enables the identification of a number of challenges facing the area in terms of creating large-scale implementations of spiking neural networks on reconfigurable hardware, particularly that operate in real time, and yet demonstrate biological plausibility in terms of the adaptability of the architecture. The paper is structured as follows: Section 2 provides a review of the literature in terms of the related work in realising both classical and spiking neural networks on hardware. Section 3 summarises a number of approaches that the authors are proposing to realise large-scale implementations of spiking neural networks on FPGAs, and Section 4 concludes the paper with a discussion on the range of different approaches and remaining challenges.
Section snippets
Background research
There has been a range of research results reported in the literature on the implementation of NNs on both ASIC and reconfigurable platforms. ASIC based approaches are traditionally referred to as neuro-processors or neuron chips. FPGA based implementations are still a fairly new approach since the early 1990s [80]. There are three important aspects in terms of implementing NNs on reconfigurable hardware. In this section the authors classify and review them as implementation of learning
Implementation approaches
Considerable research has been undertaken in developing an understanding of the behaviour of a biological neuron. However, there are less research results available on how large arrays of these interconnected neurons combine to form powerful processing arrays. The HH spiking neuron model [30] is representative of the characteristics of a real biological neuron. The model consists of four coupled non-linear differential equations which are associated with time consuming software simulations and
Discussion and remaining challenges
One of the immediate issues in the time-efficient FPGA implementation of SNNs is the availability and usability of an appropriate development environment. Understanding the simulation tool always pays rich dividends particularly with the system complexity associated with large scale SNNs. Tool support is needed throughout the design flow, starting with capture of system level models all the way down to detailed implementation. Design decisions at system level have the most impact on the final
Conclusion
This paper has provided a comprehensive review of the reported research on FPGA implementations of classical and spiking neural networks on FPGAs. Spiking neural networks have received considerable recent attention as engineers aim to form a more comprehensive understanding of information processing in biological networks, and to investigate how this understanding could be used to improve traditional information processing techniques. The authors have provided a brief summary of two alternative
Liam P. Maguire received the M.Eng. and Ph.D. degrees in Electrical and Electronic Engineering from the Queen's University of Belfast, Belfast, UK, in 1988 and 1991, respectively. He is a Reader and Acting Head of School of the School of Computing and Intelligent Systems, University of Ulster, Derry, UK He is also a Member of the Intelligent Systems Engineering Laboratory at the University of Ulster. His current research interests are in two primary areas: fundamental research in bio-inspired
References (87)
- et al.
“SpikeNET: a simulator for modeling large networks of integrate and fire neurons”
Neurocomputing
(1999) - et al.
Intrinsic and extrinsic implementation of a bio-inspired hardware system
Inf. Sci.
(2004) - et al.
Hardware description of multi-layer perceptrons with different abstraction levels
Microprocessors and Microsystems
(2006) - et al.
Spike-timing-dependent synaptic plasticity: from single spikes to spike trains
Neurocomputing
(2004) - et al.
“A neural simulation system based on biologically realistic electronic neurons”
Inf. Sci.
(2004) - et al.
Advances in design and application of spiking neural networks
Springer J. Soft Comput.—Fusion Found. Methodol. Appl.
(2007) - et al.
Evolutionary design of spiking neural networks
J. New Math. Nat. Comput. (World Scientific Publishing)
(2006) - J.-L. Beuchat J.O. Haenni, E. Sanchez, “Hardware reconfigurable neural networks”, in: 5th Reconfigurable Architectures...
- et al.
“Synaptic modifications by correlated activity: Hebb's postulated revisited”
Ann. Rev. Neurosci.
(2001) - et al.
The implementation of fuzzy systems, neural networks and fuzzy neural networks using FPGAs
Inform. Sci.
(1998)
Foundations of Physiological Psychology
SpikeNET: an event-driven simulation package for modeling large networks of spiking neurons
Network. Comput. Neural Syst.
Efficient computation and cue integration with noisy population codes
Nature Neuroscience
“The Cam-brain machine (CBM): an FPGA based hardware tool which evolves a 1000 neuron net circuit module in seconds and updates a 75 million neuron artificial brain for real time robot control”
Neurocomputing J
“An FPGA-based approach to high-speed simulation of conductance-based neuron models”
Neuroinf.
“Fast digital simulation of spiking neural networks and neuromorphic integration with SPIKELAB”
Int. J. Neural Syst.
“Organization of behavior”
“A quantitative description of membrane current and its application to conduction and excitation in nerve”
J. Physiol.
“Simple model of spiking neurons”
IEEE Trans. Neural Networks
“Which model to use for cortical spiking neurons?”
IEEE Trans. Neural Networks,
“A SIMD/dataflow architecture for a neurocomputer for spike-processing neural networks (NESPINN)”
in: MicroNeuro’
Hierarchical mixtures of experts and the EM algorithm
Neural Computation
Analog circuits for modeling biological neural networks: design and applications
IEEE Trans. Biomed. Eng.
Cited by (165)
Architectural model of the human neuroregulator system based on multi-agent systems and implementation of system-on-chip using FPGA.
2022, Microprocessors and MicrosystemsTowards neuromorphic FPGA-based infrastructures for a robotic arm
2023, Autonomous RobotsDeep Brain Stimulation on Ca2+ Signalling and Neuro-Glial Network: A Digital Implementation
2023, IEEE Transactions on Medical Robotics and Bionics
Liam P. Maguire received the M.Eng. and Ph.D. degrees in Electrical and Electronic Engineering from the Queen's University of Belfast, Belfast, UK, in 1988 and 1991, respectively. He is a Reader and Acting Head of School of the School of Computing and Intelligent Systems, University of Ulster, Derry, UK He is also a Member of the Intelligent Systems Engineering Laboratory at the University of Ulster. His current research interests are in two primary areas: fundamental research in bio-inspired intelligent systems (such as the development of computational effective spiking neural networks) and the application of existing intelligent techniques in different domains (industrial process control, augmentative technologies and finally other disciplines such as Supply Chain Management). He is the author or co-author of over 100 research papers.
Martin McGinnity has been a member of the University of Ulster academic staff since 1992, and holds the post of Professor of Intelligent Systems Engineering within the Faculty of Engineering. He has a first class honours degree in physics, and a doctorate from the University of Durham, is a Fellow of the IET, member of the IEEE, and a Chartered Engineer. He has 27 years experience in teaching and research in electronic and computer engineering, leads the research activities of the Intelligent Systems Engineering Laboratory at the Magee campus of the University, and is currently Acting Associate Dean of the Faculty of Engineering, with responsibility for Research and Development, and Knowledge and Technology Transfer. His current research interests relate to the creation of intelligent computational systems in general, particularly in relation to hardware and software implementations of neural networks, fuzzy systems, genetic algorithms, embedded intelligent systems utilising re-configurable logic devices and bio-inspired cognitive systems.
Brendan Glackin received the 1st Class Honours degree in electronics and computing from the University of Ulster, Magee, UK and is currently pursuing the Ph.D. degree at this university. He is a Research Associate in the School of Computing and Intelligent Systems, University of Ulster, Derry, UK and is also a member of the Intelligent Systems Engineering Laboratory of the same university. His current research interests relate to the implementation in embedded systems of bio-inspired and hybrid intelligent systems.
Arfan Ghani received the degree of Bachelor of Electronic engineering with distinction from NED Engineering University, Pakistan and M.Sc. in Computer Systems Engineering from the Technical University of Denmark. He has more than three years industrial experience with M/S SIEMENS Pakistan, M/S VITESSE Semiconductors Denmark, M/S Microsoft Denmark and Intel Research Cambridge, UK. Currently he is a Doctoral candidate at the Intelligent Systems Engineering Laboratory, University of Ulster, UK. His research interests include design and implementation of bio inspired architectures on reconfigurable platforms (FPGAs), hardware implementation of neural networks, VLSI system design and digital signal processing.
Ammar Belatreche received the degree of ‘Ingenieur d’Etat’ (B.Eng.) in computer systems from the National Institute of Informatics (INI), Algiers, Algeria in 1998. He is currently a lecturer at the School of Computing and Intelligent Systems, University of Ulster, UK. He is also a member of the Intelligent Systems Engineering Laboratory at the University of Ulster where he is undertaking a PhD in computer science. His research interests include intelligent systems, spiking neural networks, artificial neural networks, evolutionary computing, pattern recognition, and machine learning and computer vision.
Jim Harkin is a Lecturer in the School of Computing and Intelligent Systems at the University of Ulster, UK. He holds a B.Tech. (1996), M.Sc. (1997) and Ph.D. (2001) in electronic engineering from the University of Ulster, and is a member of the IET. He is also a member of the Intelligent Systems Engineering Laboratory at the University of Ulster and his current research interests relate to the design of intelligent embedded systems to support self-repairing capabilities; and FPGA-based hardware implementation strategies for spiking neural networks. He has published 30+ articles in peer-reviewed journals and conferences.