Elsevier

Neurocomputing

Volume 71, Issues 1–3, December 2007, Pages 13-29
Neurocomputing

Challenges for large-scale implementations of spiking neural networks on FPGAs

https://doi.org/10.1016/j.neucom.2006.11.029Get rights and content

Abstract

The last 50 years has witnessed considerable research in the area of neural networks resulting in a range of architectures, learning algorithms and demonstrative applications. A more recent research trend has focused on the biological plausibility of such networks as a closer abstraction to real neurons may offer improved performance in an adaptable, real-time environment. This poses considerable challenges for engineers particularly in terms of the requirement to realise a low-cost embedded solution. Programmable hardware has been widely recognised as an ideal platform for the adaptable requirements of neural networks and there has been considerable research reported in the literature. This paper aims to review this body of research to identify the key lessons learned and, in particular, to identify the remaining challenges for large-scale implementations of spiking neural networks on FPGAs.

Introduction

Inspired by the way the brain processes information, scientists and engineers have been researching neural networks (NNs) since the early 1940s [44]. NNs are an information processing paradigm inspired by the way biological nervous systems, such as the brain, process information. The key element of this paradigm is the novel structure of the information processing system. It is composed of a large number of highly interconnected processing elements, neurons, working in parallel to solve a specific problem.

The neuron is made up of four main parts; dendrites, synapses, axon and the cell body. A neuron is essentially a system that accepts electrical currents which arrive on its dendrites. It sums these and if they exceed a certain threshold it issues a new pulse which propagates along an axon. The information is transmitted from an axon to a dendrite via a synapse, which is done by means of chemical neurotransmitters across the synaptic membrane. An illustration of a biological neuron can be found in Fig. 1.

Research into artificial neural networks (ANNs), has seen the development of a plethora of neuron models from the initial McCulloch and Pitts concept [44] to the more biologically realistic spike models [77]. Recent trends in computational intelligence have indicated a strong tendency towards forming a better understanding of biological systems and the details of neuronal signal processing [31], [39], [62], [63]. Such research is motivated by the desire to form a more comprehensive understanding of information processing in biological networks and to investigate how this understanding could be used to improve traditional information processing techniques [41], [67]. Spiking neurons differ from conventional ANN models as information is transmitted by means of spikes rather than by firing rates [7], [22], [27], [55], [60]. It is believed that this allows spiking neurons to have richer dynamics as they can exploit the temporal domain to encode or decode data in the form of spike trains [58], [59]. However, this has demanded the development of new learning rules drawing again on inspiration from biology. For example, Hebbian learning has been identified as a closely biologically related learning rule [28] and more recent research has reported a spike timing dependent variation of this rule called spike timing dependent plasticity (STDP) [4], [51], [52], [70] which modulates the synaptic efficiency of synapses.

Software simulation of network topologies and connection strategies provides a platform for the investigation of how arrays of spiking neurons can be used to solve computational tasks. Such simulations face the problem of scalability in that biological systems are inherently parallel in their architecture whereas commercial PCs are based on the sequential Von Neumann serial processing architecture. Thus, it is difficult to assess the efficiency of these models to solve complex problems [10]. When implemented on parallel hardware, NNs can take full advantage of their inherent parallelism and run orders of magnitude faster than software simulations, thus becoming, appropriate for real-time applications. Developing custom application-specific integrated circuit (ASIC) devices for NNs however is both time consuming and expensive. These devices are also inflexible in that a modification of the basic neuron model would require a new development cycle to be undertaken. Field programmable gate arrays (FPGAs) are devices that permit the implementation of digital systems, providing an array of logic components that can be configured in a desired way by a configuration bitstream [72]. The device is reconfigurable such that a change to the system is easily achieved and an updated configuration bitstream can be downloaded to the device. Previous work has indicated that these devices provide a suitable platform for the implementation of conventional AANs [5], [19].

This paper aims to report on the issues arising from the authors’ experience in implementing spiking neural networks on reconfigurable hardware. This enables the identification of a number of challenges facing the area in terms of creating large-scale implementations of spiking neural networks on reconfigurable hardware, particularly that operate in real time, and yet demonstrate biological plausibility in terms of the adaptability of the architecture. The paper is structured as follows: Section 2 provides a review of the literature in terms of the related work in realising both classical and spiking neural networks on hardware. Section 3 summarises a number of approaches that the authors are proposing to realise large-scale implementations of spiking neural networks on FPGAs, and Section 4 concludes the paper with a discussion on the range of different approaches and remaining challenges.

Section snippets

Background research

There has been a range of research results reported in the literature on the implementation of NNs on both ASIC and reconfigurable platforms. ASIC based approaches are traditionally referred to as neuro-processors or neuron chips. FPGA based implementations are still a fairly new approach since the early 1990s [80]. There are three important aspects in terms of implementing NNs on reconfigurable hardware. In this section the authors classify and review them as implementation of learning

Implementation approaches

Considerable research has been undertaken in developing an understanding of the behaviour of a biological neuron. However, there are less research results available on how large arrays of these interconnected neurons combine to form powerful processing arrays. The HH spiking neuron model [30] is representative of the characteristics of a real biological neuron. The model consists of four coupled non-linear differential equations which are associated with time consuming software simulations and

Discussion and remaining challenges

One of the immediate issues in the time-efficient FPGA implementation of SNNs is the availability and usability of an appropriate development environment. Understanding the simulation tool always pays rich dividends particularly with the system complexity associated with large scale SNNs. Tool support is needed throughout the design flow, starting with capture of system level models all the way down to detailed implementation. Design decisions at system level have the most impact on the final

Conclusion

This paper has provided a comprehensive review of the reported research on FPGA implementations of classical and spiking neural networks on FPGAs. Spiking neural networks have received considerable recent attention as engineers aim to form a more comprehensive understanding of information processing in biological networks, and to investigate how this understanding could be used to improve traditional information processing techniques. The authors have provided a brief summary of two alternative

Liam P. Maguire received the M.Eng. and Ph.D. degrees in Electrical and Electronic Engineering from the Queen's University of Belfast, Belfast, UK, in 1988 and 1991, respectively. He is a Reader and Acting Head of School of the School of Computing and Intelligent Systems, University of Ulster, Derry, UK He is also a Member of the Intelligent Systems Engineering Laboratory at the University of Ulster. His current research interests are in two primary areas: fundamental research in bio-inspired

References (87)

  • N. Carlson

    Foundations of Physiological Psychology

    (1992)
  • Y. Chen, S. Hall, L. McDaid, O. Buiu, P. Kelly, On the design of a low power compact spiking neuron cell based on...
  • Y. Chen, S. Hall, L.J. McDaid, P.M. Kelly, “Silicon synapse for hebbian learning application”, Proceedings of PREP,...
  • Y. Chen, S. Hall, L.J. McDaid, P.M. Kelly, “A silicon synapse based on a charge transfer device for spiking neural...
  • A. Delorme et al.

    SpikeNET: an event-driven simulation package for modeling large networks of spiking neurons

    Network. Comput. Neural Syst.

    (2003)
  • S. Deneve et al.

    Efficient computation and cue integration with noisy population codes

    Nature Neuroscience

    (2001)
  • J.G. Eldredge, “FPGA density enhancement of a neural network through run-time reconfiguration”, Master's Thesis,...
  • A. Ferrucci, “Acme: a field-programmable gate array implementation of a self adapting and scalable connectionist...
  • S. Furber, S. Temple, A. Brown, “On-chip and inter-chip networks for modelling large-scale neural systems”, in: IEEE...
  • H. De Garis et al.

    “The Cam-brain machine (CBM): an FPGA based hardware tool which evolves a 1000 neuron net circuit module in seconds and updates a 75 million neuron artificial brain for real time robot control”

    Neurocomputing J

    (2002)
  • A. Ghani, T.M. McGinnity, L.P. Maguire, “Approaches to the implementation of large scale spiking neural networks on...
  • A. Ghani, T. Martin McGinnity, L.P. Maguire, “Area efficient architecture for large scale implementation of...
  • B. Glackin, T.M. McGinnity, L.P. Maguire, A. Belatreche, Q.X. Wu, Implementation of a biologically realistic spiking...
  • B. Glackin, T.M. McGinnity, L.P. Maguire, A. Wu, Q.X. Ghani, A. Belatreche, “An event based large scale spiking neural...
  • B. Glackin, T.M. McGinnity, L.P. Maguire, Q.X. Wu, A. Belatreche, “A novel approach for the implementation of large...
  • N.H. Goddard, K.J. Lynne, T. Mintz, The Rochester Connectionist Simulator, Technical Report 233 (revised), Computer...
  • E.L. Graas et al.

    “An FPGA-based approach to high-speed simulation of conductance-based neuron models”

    Neuroinf.

    (2004)
  • C. Grassmann et al.

    “Fast digital simulation of spiking neural networks and neuromorphic integration with SPIKELAB”

    Int. J. Neural Syst.

    (1999)
  • G. Hartmann, G. Frank, M. Schaefer, C. Wolff, “SPIKE128K—an accelerator for dynamic simulation of large pulse-coded...
  • S. Haykin, Neural Networks. A Comprehensive Foundation, Second ed., Q6 Prentice-Hall, NJ, Englewood cliffs,...
  • D.O. Hebb

    “Organization of behavior”

    (1949)
  • H.H. Hellmich, H. Klar, “SEE: a concept for an FPGA based emulation engine for spiking neurons with adaptive weights,”...
  • A.L. Hodgkin et al.

    “A quantitative description of membrane current and its application to conduction and excitation in nerve”

    J. Physiol.

    (1952)
  • G. Indiveri, P. Verschure, “Autonomous vechicle guidance using analog VLSI neuromorphic sensor”, in: Proceedings of the...
  • E.M. Izhikevich

    “Simple model of spiking neurons”

    IEEE Trans. Neural Networks

    (2003)
  • E.M. Izhikevich

    “Which model to use for cortical spiking neurons?”

    IEEE Trans. Neural Networks,

    (2004)
  • A. Jahnke et al.

    “A SIMD/dataflow architecture for a neurocomputer for spike-processing neural networks (NESPINN)”

    in: MicroNeuro’

    (1996)
  • S. Johnston, G. Prasad, L.P. Maguire, T.M. McGinnity, “Comparative investigation into classical and spiking neuron...
  • S.P.J. Johnston, G. Prasad, L.P. Maguire, T.M. McGinnity, A. Belatreche, “Investigation into the pragmatism of...
  • M.I. Jordan et al.

    Hierarchical mixtures of experts and the EM algorithm

    Neural Computation

    (1994)
  • S. Le Masson et al.

    Analog circuits for modeling biological neural networks: design and applications

    IEEE Trans. Biomed. Eng.

    (1999)
  • M.A. Lewis, R. Etinne-Cummings, A.H. Cohen, M. Hartmann, “Toward biomorphic control using custom aVLSI CPG chips”, in:...
  • W.B Ligon III, S. McMillan, G. Monn, K. Schoonover, F. Stivers, K.D. Underwood, “A re-evaluation of the practicality of...
  • Cited by (165)

    View all citing articles on Scopus

    Liam P. Maguire received the M.Eng. and Ph.D. degrees in Electrical and Electronic Engineering from the Queen's University of Belfast, Belfast, UK, in 1988 and 1991, respectively. He is a Reader and Acting Head of School of the School of Computing and Intelligent Systems, University of Ulster, Derry, UK He is also a Member of the Intelligent Systems Engineering Laboratory at the University of Ulster. His current research interests are in two primary areas: fundamental research in bio-inspired intelligent systems (such as the development of computational effective spiking neural networks) and the application of existing intelligent techniques in different domains (industrial process control, augmentative technologies and finally other disciplines such as Supply Chain Management). He is the author or co-author of over 100 research papers.

    Martin McGinnity has been a member of the University of Ulster academic staff since 1992, and holds the post of Professor of Intelligent Systems Engineering within the Faculty of Engineering. He has a first class honours degree in physics, and a doctorate from the University of Durham, is a Fellow of the IET, member of the IEEE, and a Chartered Engineer. He has 27 years experience in teaching and research in electronic and computer engineering, leads the research activities of the Intelligent Systems Engineering Laboratory at the Magee campus of the University, and is currently Acting Associate Dean of the Faculty of Engineering, with responsibility for Research and Development, and Knowledge and Technology Transfer. His current research interests relate to the creation of intelligent computational systems in general, particularly in relation to hardware and software implementations of neural networks, fuzzy systems, genetic algorithms, embedded intelligent systems utilising re-configurable logic devices and bio-inspired cognitive systems.

    Brendan Glackin received the 1st Class Honours degree in electronics and computing from the University of Ulster, Magee, UK and is currently pursuing the Ph.D. degree at this university. He is a Research Associate in the School of Computing and Intelligent Systems, University of Ulster, Derry, UK and is also a member of the Intelligent Systems Engineering Laboratory of the same university. His current research interests relate to the implementation in embedded systems of bio-inspired and hybrid intelligent systems.

    Arfan Ghani received the degree of Bachelor of Electronic engineering with distinction from NED Engineering University, Pakistan and M.Sc. in Computer Systems Engineering from the Technical University of Denmark. He has more than three years industrial experience with M/S SIEMENS Pakistan, M/S VITESSE Semiconductors Denmark, M/S Microsoft Denmark and Intel Research Cambridge, UK. Currently he is a Doctoral candidate at the Intelligent Systems Engineering Laboratory, University of Ulster, UK. His research interests include design and implementation of bio inspired architectures on reconfigurable platforms (FPGAs), hardware implementation of neural networks, VLSI system design and digital signal processing.

    Ammar Belatreche received the degree of ‘Ingenieur d’Etat’ (B.Eng.) in computer systems from the National Institute of Informatics (INI), Algiers, Algeria in 1998. He is currently a lecturer at the School of Computing and Intelligent Systems, University of Ulster, UK. He is also a member of the Intelligent Systems Engineering Laboratory at the University of Ulster where he is undertaking a PhD in computer science. His research interests include intelligent systems, spiking neural networks, artificial neural networks, evolutionary computing, pattern recognition, and machine learning and computer vision.

    Jim Harkin is a Lecturer in the School of Computing and Intelligent Systems at the University of Ulster, UK. He holds a B.Tech. (1996), M.Sc. (1997) and Ph.D. (2001) in electronic engineering from the University of Ulster, and is a member of the IET. He is also a member of the Intelligent Systems Engineering Laboratory at the University of Ulster and his current research interests relate to the design of intelligent embedded systems to support self-repairing capabilities; and FPGA-based hardware implementation strategies for spiking neural networks. He has published 30+ articles in peer-reviewed journals and conferences.

    View full text