Elsevier

Neurocomputing

Volume 174, Part A, 22 January 2016, Pages 31-41
Neurocomputing

Review of advances in neural networks: Neural design technology stack

https://doi.org/10.1016/j.neucom.2015.02.092Get rights and content

Abstract

This review provides a high-level synthesis of significant recent advances in artificial neural network research, as well as multi-disciplinary concepts connected to the far-reaching goal of obtaining intelligent systems. We assume that a global outlook of these interconnected fields can benefit researchers by providing alternative viewpoints. Therefore, we present different network and neuron models, we discuss model parameters and the means to obtain them, and we draw a quick outline of information encoding, before proceeding to an overview of the relevant learning mechanisms, ranging from established approaches to novel ideas. We specifically focus on comparing the classical artificial model with the biologically-feasible spiking neuron, and we take this comparison further into a discussion on the biological plausibility of various learning approaches.

Section snippets

Introduction and background

Recently, we have been witnessing increasing interest in understanding how the brain functions and how it could be modeled. This momentum is fueled by major research initiatives, including the Human Brain Project [44], the BRAIN Initiative, as well as significant commercial efforts, such as IBM Watson [20].

In this paper we provide a high-level overview of the state-of-the-art in neural networks, the main building block of the brain. Computer scientists typically focus on artificial neural

Neural network definitions

From the structural point of view, a brain consists of a network of neurons, densely interconnected via synapses – a structure which artificial neural networks are attempting to recreate. In this section we present different neuron and network models, as well as discuss the information and feature encoding in these networks.

Establishing model parameters

A neural network is a meta-model for computation. Applying it to solve a concrete problem requires a particular set of parameters, whose tuning and life cycle are presented in the following subsections.

Learning

Learning is a data-driven mechanism for obtaining model parameters. In this section, we describe the learning process and we make a classification of learning algorithms, with a focus on a number of interesting learning trends.

Machine learning techniques aim to shift the work burden from human effort to machine computation, for specific classes of problems. With the right algorithm, the manual effort of designing solutions can be traded off for memory and computational power. Algorithms, in the

Advancing neural networks research

A number of recent advancements have helped neural networks to regain mainstream attention. Firstly, novel hardware has made it possible to verify concepts at a much larger scale, e.g. by simulating a system with 109 neurons [2], comparable in scale to a cat׳s brain. Secondly, the unsupervised feature learning paradigm offered a logic that could utilize large unpreprocessed datasets, providing results such as a self-emerging cat image detector [43]. Having already discussed the latter in

Summary

In this paper, we have provided a high-level overview of current trends in neural network research. By starting from the definitions of neural network models, we compare the artificial neuron model with the biologically-plausible spiking neuron. Furthermore, we explain the basic concepts behind the encoding and decoding of information in a neural network model, be it an artificial mathematical abstraction, or a biological neuroscience model. Moreover, we present the model parameters that need

Adela-Diana Almási obtained her Bachelor׳s and Master׳s degrees in Computer Science from the “Politehnica” University of Bucharest, Romania, where she is currently a PhD student under the supervision of Prof. Valentin Cristea. She is conducting her doctoral research in collaboration with the IBM Research Laboratory in Zurich, Switzerland. Her main research interests are machine learning, neural networks and data analytics.

References (68)

  • M. Collins, Discriminative training methods for hidden Markov models: theory and experiments with perceptron...
  • G.E. Dahl et al.

    Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition

    IEEE Trans. Audio Speech Lang. Process.

    (2012)
  • O.E. David, I. Greental, Genetic algorithms for evolving deep neural networks, In: Proceedings of the 2014 Conference...
  • P. Dayan et al.

    Theoretical neuroscience: computational and mathematical modeling of neural systems

    J. Cognit. Neurosci.

    (2003)
  • D. Debanne et al.

    Axon physiology

    Physiol. Rev.

    (2011)
  • W. Duch, Coloring black boxes: visualization of neural network decisions, in: 2003 Proceedings of the International...
  • C. Eliasmith

    How to Build a BrainA Neural Architecture for Biological Cognition

    (2013)
  • C. Eliasmith et al.

    Neural EngineeringComputation, Representation, and Dynamics in Neurobiological Systems

    (2004)
  • C. Eliasmith et al.

    A large-scale model of the functioning brain

    Science

    (2012)
  • C. Ferreira

    Designing neural networks using gene expression programming

    Appl. Soft Comput. Technol.: Chall. Complex.

    (2006)
  • D. Ferrucci et al.

    Building Watsonan overview of the deepqa Project

    AI Mag.

    (2010)
  • N. Fremaux et al.

    Functional requirements for reward-modulated spike-timing-dependent plasticity

    J. Neurosci.

    (2010)
  • P. Gastaldo et al.

    Combining ELMs with random projections

    IEEE Intell. Syst.

    (2013)
  • W. Gerstner et al.

    A neuronal learning rule for sub-millisecond temporal coding

    Nature

    (1996)
  • W. Gerstner et al.

    Spiking Neuron ModelsSingle Neurons, Populations, Plasticity

    (2002)
  • W. Gerstner et al.

    How good are neuron models?

    Science

    (2009)
  • W. Gerstner et al.

    Theory and simulation in neuroscience

    Science

    (2012)
  • A. Graves et al.

    A novel connectionist system for unconstrained handwriting recognition

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2009)
  • A. Graves, A.-r. Mohamed, G. Hinton, Speech recognition with deep recurrent neural networks, in: 2013 IEEE...
  • A. Gupta, L.N. Long, Character recognition using spiking neural networks, in: 2007 Neural Networks, IJCNN, 2007, pp....
  • S.-J. Han et al.

    Evolutionary neural networks for anomaly detection based on the behavior of a program

    IEEE Trans. Syst. Man Cybern. Part B: Cybern.

    (2005)
  • S. L. Hill, Y. Wang, I. Riachi, F. Schürmann, H. Markram, Statistical connectivity provides a sufficient foundation for...
  • M.L. Hines et al.

    ModelDBa database to support computational neuroscience

    J. Comput. Neurosci.

    (2004)
  • G. Hinton et al.

    A fast learning algorithm for deep belief nets

    Neural Comput.

    (2006)

    • Cited by (64)

    • An integrate-and-fire neuron with capacitive trans-impedance amplifier for improving linearity in Spiking Neural Networks

      2022, Solid-State Electronics

    • Challenges in implementing data-driven approaches for building life cycle energy assessment: A review

      2022, Renewable and Sustainable Energy Reviews

    • Deep learning for predictions of hydrolysis rates and conditional molecular design of esters

      2021, Journal of the Taiwan Institute of Chemical Engineers

    • Development of a Novel Feedforward Neural Network Model Based on Controllable Parameters for Predicting Effluent Total Nitrogen

      2021, Engineering

    • Task-Adaptive Neuromorphic Computing Using Reconfigurable Organic Neuristors with Tunable Plasticity and Logic-in-Memory Operations

      2024, Journal of Physical Chemistry Letters

    • Diagnostic reasoning in the age of artificial intelligence: Synergy or opposition?

      2024, Journal of Hospital Medicine
    View all citing articles on Scopus

    Adela-Diana Almási obtained her Bachelor׳s and Master׳s degrees in Computer Science from the “Politehnica” University of Bucharest, Romania, where she is currently a PhD student under the supervision of Prof. Valentin Cristea. She is conducting her doctoral research in collaboration with the IBM Research Laboratory in Zurich, Switzerland. Her main research interests are machine learning, neural networks and data analytics.

    Stanisław Woźniak received his BSc and MSc degrees in Computer Science from Poznan University of Technology, Poland. Currently he is pursuing a PhD degree at École Polytechnique Fédérale de Lausanne, Switzerland, in collaboration with IBM Research – Zurich, Switzerland. His research interests include neural networks, machine learning, computational neuroscience and cognitive systems.

    Valentin Cristea is the Head of the Computer Science and Engineering Department and Professor at the Politehnica University of Bucharest. His main fields of expertise are Computer networks, E-Services, and Large Scale Distributed Systems, topics on which he teaches courses and supervises PhD students. He is the Director of the National Center for Information Technology, leader of the CoLaborator, Distributed Systems and Grid, and e-Business/e-Government laboratories. He has a long experience in the development and management of international and national research projects on e-services, dependable large scale distributed systems, Grid and Cloud computing, and smart environments. He received the IBM Faculty Award in 2003 and 2011, he is an IEEE and ACM member and a Phare IT expert.

    Yusuf Leblebici received the B.Sc. and M.Sc. degrees in electrical engineering from Istanbul Technical University, Istanbul, Turkey, in 1984 and 1986, respectively, and the Ph.D. degree in electrical and computer engineering from the University of Illinois, Urbana-Champaign (UIUC), in 1990. Since 2002, he is a Chair Professor at the Swiss Federal Institute of Technology in Lausanne (EPFL), and director of Microelectronic Systems Laboratory. His research interests include design of high-speed CMOS digital and mixed-signal integrated circuits, computer-aided design of VLSI systems, intelligent sensor interfaces, modeling and simulation of semiconductor devices, and VLSI reliability analysis. He is the coauthor of six textbooks, as well as more than 300 articles published in various journals and conferences. He is a Fellow of IEEE since 2010, and he has been elected as Distinguished Lecturer of the IEEE Circuits and Systems Society for 2010–2011.

    Ton Engbersen has been with the IBM Research since 1980. His career spanned such diverse areas as Image processing, chip design, communications technology, server technology, legacy management, Innovation in Outsourcing and Data center Energy management. Throughout the years he has held a range of management positions in R&D in Switzerland and in the US. As member of the IBM Academy of Technology he led the European branch from 2009 to 2011. Ton has published more than 50 articles in refereed scientific journals. Currently he is the Scientific Director for the ASTRON-IBM Center of Exascale Technology, and leading the DOME Project http://www-03.ibm.com/press/us/en/pressrelease/37361.wss.

    View full text
    Copyright © 2015 Published by Elsevier B.V.