Elsevier

Neurocomputing

Volume 55, Issues 1–2, September 2003, Pages 383-401
Neurocomputing

N4: computing with neural receptive fields

https://doi.org/10.1016/S0925-2312(02)00630-6Get rights and content

Abstract

In this study, we introduce a new neural architecture called N4 that is based on a collection of local receptive fields realized in the form of referential neural networks. While the network exhibits some similarities to other structures of modular neural networks (such as expert networks), it comes with a number of unique features. Especially, its receptive fields exhibit high flexibility by being formed by neural networks. Subsequently, the processing therein is of referential nature. A ”skeleton” (structure) of the network is completed through unsupervised learning that is aimed at “discovering” and structuring the main dependencies in data. More specifically, the design of the network consists of two phases. First, a blueprint of the network is formed and this involves the prototypes obtained through clustering of training data. This structural development of the network is followed by further refinement in a form of parametric training of the individual neural receptive fields. The study provides a detailed analysis and learning of the network and includes experimental investigations.

Introduction

Commonly, neural networks are aimed at global processing by developing nonlinear mappings between input and output spaces [3], [4], [5]. Radial basis function (RBF) neural networks focus on a “local” character of processing by exploiting a collection of local receptive fields and concentrating all computations around them. There has been a vast body of research and comparative analysis about a local and global nature of processing in neural networks [3], [4], [6]. Furthermore a broad class of modular neural networks (coming under different names such as hierarchical mixtures of networks, committees of networks and alike) [7], [8], [9] has been studied as an intermediate category whose processing falls in-between full global neural processing and a local one. In this study, we revisit and augment (generalize) the RBF neural networks by introducing a concept of a neural nearest neighbor network (abbreviated as N4). This comes as a generalization of the RBF neural networks in which neural networks play a role of the generalized and flexible receptive fields. There are two underlying computing mechanisms in the N4 architecture. The highly nonlinear character of processing is realized by the local neural networks at the first layer. Next, the linear type of processing is completed at the second layer of the network.

The design of the N4 structure is comprised of two fundamental phases. The first one focuses on the structural optimization of the network and involves fuzzy clustering. The structure of the network spans over the prototypes of the already determined clusters. The number of the clusters is equal to the number of the local neural networks that generalize the idea of the receptive fields. We refer to them as neural receptive fields. The ensuing learning of these neural networks occurs at a parametric level and forms the second phase of the parametric learning. The neural receptive fields help modularize an overall learning problem by being naturally “focused” on processing a portion of data in the input space that fall within a close neighborhood of the prototypes of the individual networks.

The paper consists of six sections. First, we introduce a detailed architecture of N4 (Section 2) that is followed by a learning scheme (Section 3). The N4 processing scheme is compared with other mechanisms of learning and processing as being available in neural networks and this analysis is included in Section 4. Intensive experimental studies illustrating the main features of N4 are reported in Section 5. They involve synthetic two-dimensional data as well as multivariable Boston housing data. Conclusions are covered in Section 6.

Section snippets

The architecture

The topology of the neural network, as visualized in Fig. 1, consists of the two types of functional blocks arranged in a two-layer architecture.

The first functional layer is formed by a series of neural networks (called local neural networks) driven by two sources of inputs information. Each of these networks accepts an input pattern (x) that is common to all of them. The second input is a prototype (say, vi). Each network comes with its own prototype. The processing involved there implies a

The learning scheme

As already stated, the learning scheme consists of two main phases that is structural learning and parametric optimization (learning). The structural learning is concerned with a development of a “blueprint” of the network. This comprises of the prototypes in the input space and the corresponding connections (prototypes in the output space) of the linear unit situated in the output layer.

The comparative architectural and functional analysis of the N4 network

It is instructive to contrast the introduced neural nearest neighbor network with the standard and commonly used neural network architectures. In particular, these include feedforward neural networks equipped with standard sigmoid nonlinearities and RBF neural networks. Obviously, the first category of the networks does not exploit any local receptive fields. N4 differs from the standard neural network with respect to the form of processing. N4 is about a local form of processing, revolves

Experiments

This section is concerned with a series of experiments which objective is to present the computational side of the neural architecture and contrast this network with other well-known models of two standard neural networks. First, we discuss two-dimensional synthetic data and afterwards proceed with one of the standard Machine Learning data available on the Internet such as a Boston housing data.

Conclusions

In this study, we have introduced a new neural network model based on an idea of the generalized neural receptive fields. While being a close relative of the RBF neural networks, the generalization helps us achieve higher flexibility and promotes a notion of generalized proximity (similarity) between patterns in the input space thus addressing possible heterogeneous nature of the input space. The development of the N4 involves two phases that is unsupervised learning (during which a blueprint

Acknowledgements

The support from the Natural Sciences and Engineering Research Council of Canada (NSERC) and ASERC (Alberta Software Engineering Research Consortium) is gratefully acknowledged. This work was also supported by Grant No. 2000-1-51200-001-2 from the Basic Research Program of Korea Science & Engineering Foundation.

Witold Pedrycz is a Professor and Chair in the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, Canada. He is also a Canada Research Chair (CRC) in Computational Intelligence.

He is actively pursuing research in Computational Intelligence, fuzzy modeling, knowledge discovery and data mining, fuzzy control including fuzzy controllers, pattern recognition, knowledge-based neural networks, relational computation, and Software Engineering. He has published numerous

References (12)

  • K. Hirota et al.

    Implicitly—supervised learning and its application to fuzzy pattern classifiers

    Inform. Sci.

    (1998)
  • J.C. Bezdek

    Pattern Recognition with Fuzzy Objective Function Algorithms

    (1981)
  • G. Bortolan, W. Pedrycz, Fuzzy clustering preprocessor in neural classifiers, Kybernetes, to...
  • R.M. Golden

    Mathematical Methods for Neural Network Analysis and Design

    (1996)
  • M.H. Hassoun

    Fundamentals of Artificial Neural Networks

    (1995)
  • S. Haykin

    Neural Networks: A Comprehensive Foundation

    (1994)
There are more references available in the full text version of this article.

Cited by (6)

  • Noise Boosted Neural Receptive Fields

    2021, 2021 IEEE Symposium Series on Computational Intelligence, SSCI 2021 - Proceedings
  • CNN LSTM Network Architecture for Modeling Software Reliability

    2019, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
  • Generalized cluster based fuzzy model tree for data modeling and prediction

    2010, 2010 IEEE World Congress on Computational Intelligence, WCCI 2010
  • The Design of granular-based radial basis function neural network by context-based clustering

    2009, Transactions of the Korean Institute of Electrical Engineers

Witold Pedrycz is a Professor and Chair in the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, Canada. He is also a Canada Research Chair (CRC) in Computational Intelligence.

He is actively pursuing research in Computational Intelligence, fuzzy modeling, knowledge discovery and data mining, fuzzy control including fuzzy controllers, pattern recognition, knowledge-based neural networks, relational computation, and Software Engineering. He has published numerous papers in this area. He is also an author of 7 research monographs covering various aspects of Computational Intelligence and Software Engineering.

Witold Pedrycz has been a member of numerous program committees of IEEE conferences in the area of fuzzy sets and neurocomputing. He currently serves as an Associate Editor of IEEE Transactions on Systems Man and Cybernetics and IEEE Transactions on Fuzzy Systems.

Giancarlo Succi is a professor in the Faculty of Computer Science of the Free University of Bozen. His researches in the area of software engineering, with specific focus on extreme programming and flexible methodologies, empirical methods in software engineering, software engineering economics, software metrics, software product lines, software reuse, software engineering over the Internet. He is also an active member of the XP community and Program co-chair for XP 2000, XP 2001, and XP2002.

Dr. Succi is author of more than 100 publications in international journals and conference proceedings, and of one book. He has been principal investigator for very large grants. He consults for private and public bodies in Europe and North America. He is member of the Italian Association of Professional Engineers, ACM, and the Computer Society of IEEE.

Myung Geun Chun received the B.S. degree in electronics engineering from Pusan National University, Pusan, Korea in 1987 and the M.S. and Ph.D. degrees in electrical engineering from the Korea Advanced Science and Technology (KAIST), Taejeon, Korea, in 1989 and 1983, respectively. Prior to joining Chungbuk National University, he worked at the Samsung Electronics as a senior researcher. Since 1996, he has been working at School of Electrical and Computer Engineering, Chungbuk National University, as an associate professor. He visited the Department of Electrical and Computer Engineering, University of Alberta, Canada during 2000-2001, where he worked with Professor Pedrycz. He serves now as the treasurer of the Korea Fuzzy Logic and Intelligent System Society. His current research interests include speech recognition, face recognition, iris pattern recognition, and intelligent systems.

View full text