Elsevier

Neurocomputing

Volume 62, December 2004, Pages 79-91
Neurocomputing

Reducing the number of neurons in radial basis function networks with dynamic decay adjustment

https://doi.org/10.1016/j.neucom.2003.12.004Get rights and content

Abstract

Classification is a common task for supervised neural networks. A specific radial basis function network for classification is the so-called RBF network with dynamic decay adjustment (RBFN-DDA). Fast training and good classification performance are properties of this network. RBFN-DDA is a dynamically growing network, i.e. neurons are inserted during training. A drawback of RBFN-DDA is its greedy insertion behavior. Too many superfluous neurons are inserted for noisy data, overlapping data or for outliers. We propose an online technique to reduce the number of neurons during training. We achieve our goal by deleting neurons after each training of one epoch. By using the improved algorithm on benchmark data and current medical data, the number of neurons is reduced clearly (up to 93.9% less neurons). Thus, we achieve a network with less complexity compared to the original RBFN-DDA.

Introduction

Many different variants of radial basis function networks are used for classification tasks. One of these variants is the so-called RBF network with dynamic decay adjustment (RBFN-DDA) [1], [2]. The RBFN-DDA training algorithm is fast, and classification performance is high. It is a growing radial basis function network, i.e. neurons are inserted during training to adapt network topology. RBFN-DDA is described in Section 2. We discuss shortly its advantages and disadvantages. RBFN-DDA was developed using the probabilistic nature of probabilistic neural networks (PNN) [16] and the idea of shrinking neuron radii of restricted Coulomb energy networks (RCEN) [15], cf. Section 2.

Despite its advantages RBFN-DDA is built of too many neurons because many neurons are inserted for noisy, overlapping data or for outliers (greedy insertion). These neurons are superfluous. Our idea is to delete neurons in addition during the training procedure. Recently, a strategy for pruning a neuro-fuzzy network after training with a new training was proposed [3] to overcome the outlier problem. Of course, a disadvantage of this method, applied to RBFN-DDA, would be a new complete training procedure.

For reducing model complexity of RBFN-DDA we prune neurons after each training epoch. Thus, no additional training is necessary. We solve the problem of inserting too many neurons for outliers, in overlapping regions of the data and in noisy data regions. We reach our goal by marking neurons as temporary neurons as long as they do not cover a sufficient number of neurons. The superfluous recognized neurons are pruned after each learning epoch. The centers of these RBF neurons are not used anymore as training data in the following learning epochs. In Section 3, we explain in detail how we will use temporary neurons, and we call the resulting RBFN-DDA algorithm “RBFN-DDA with temporary neurons” (RBFN-DDA-T).

In Section 4, we present the results of our experiments. First, in Section 4.1 we use different well-known benchmark data [6], [9], [14].2 As an important application we apply and compare the networks using medical data of septic shock patients in Section 4.2. Septic shock is one of the 10 most frequent causes for death in intensive care units of developed nations. We give results for outcome classification of this patient data in addition to the benchmark data.

With our new variant RBFN-DDA-T of the RBFN-DDA algorithm the number of neurons is reduced clearly (on average 57.7%, and up to 93.9% less neurons) without loss of classification performance. Superfluous neurons are pruned online after each learning epoch, and we obtain network models with much less complexity.

Section snippets

Radial basis functions with dynamic decay adjustment

Radial basis function networks (RBFN) were first introduced in [11]. Overlapping local radial basis functions are used to model function output (approximation) or class assignments (classification). For an introduction to RBF networks, see [8] for example.

Here, we will describe the learning procedure RBFN-DDA [1], [2]. Originally RBFN-DDA was proposed as an extension of static probabilistic neural networks (PNN) [16] by using techniques of restricted Coulomb energy networks (RCEN) [15]. PNN

Reducing model complexity

At first, we describe our modifications of the RBFN-DDA algorithm. Then, we note the improved algorithm in detail.

The algorithm is modified in steps 3 and 4. An additional step 7 is added. Steps 1, 2, 5 and 6 are identical to the RBFN-DDA algorithm. After insertion of a neuron it is immediately marked as “temporary”. During training a neuron may cover more and more samples. Then it is marked as “permanent”. After each training epoch, all superfluous, not representative neurons are deleted.

Results

In Section 4.1 we present results using well-known benchmark data [6], [9], [14]. In Section 4.2, we present results with septic shock patient data from intensive care units.

Conclusion

We have approached the problem of model complexity (=number of neurons) for RBFN-DDA. To reduce the number of neurons for this kind of network we generated neurons temporarily. Neurons are deleted after each training epoch if they do not cover a sufficient number of samples. If the neurons cover a sufficient number of samples, they are marked as “permanent” after the training epoch. This mechanism shows a remarkable effect. The number of neurons can be reduced up to 93.9% having a similar and

Jürgen Paetz attended an industrial training in computer science/telecommunication/applied mathematics until 1993. He received his diploma degree in mathematics from FernUniversität Hagen, Germany in 1997. His diploma thesis in the subject area of complex function theory was awarded with the award for excellence from the “Gesellschaft der Freunde der FernUniversität.” Until 2003 he worked at the hospital of the J.W. Goethe-University Frankfurt am Main, Germany with additional engagement in R.

References (17)

  • M.R. Berthold

    J. Diamond, Constructive training of probabilistic neural networks

    Neurocomputing

    (1998)
  • D.F. Specht

    Probabilistic neural networks

    Neural Networks

    (1990)
  • M.R. Berthold, J. Diamond, Boosting the performance of RBF networks with dynamic decay adjustment, in: Proceedings of...
  • M.R. Berthold, Fuzzy-models and potential outliers, in: Proceedings of the 18th International Conference of the North...
  • R. Brause, F. Friedrich, A neuro-fuzzy approach as medical interface, in: Proceedings of the Eighth European Symposium...
  • A.M. Fein, et al. (Eds.), Sepsis and Multiorgan Failure, Williams & Wilkins, Baltimore, MD,...
  • R.A. Fisher, The use of multiple measurements in axonomic problems, Ann. Eugenics 7 (1936) 179–188,...
  • E. Hanisch et al.

    Intensive care management in abdominal surgical patients with septic complications

There are more references available in the full text version of this article.

Cited by (0)

Jürgen Paetz attended an industrial training in computer science/telecommunication/applied mathematics until 1993. He received his diploma degree in mathematics from FernUniversität Hagen, Germany in 1997. His diploma thesis in the subject area of complex function theory was awarded with the award for excellence from the “Gesellschaft der Freunde der FernUniversität.” Until 2003 he worked at the hospital of the J.W. Goethe-University Frankfurt am Main, Germany with additional engagement in R. Brause's adaptive systems working group at the computer science institute of J.W. Goethe-University. In 2002 he completed his doctoral thesis about “Septic Shock Diagnosis by Adaptive Rule Based Systems” within the project MEDAN (Medical Data Analysis with Neural Networks) that was supported by the Deutsche Forschungsgemeinschaft (DFG). At present, he is postdoctoral researcher and lecturer in soft computing and applications at the J.W. Goethe-University, Department of biology and informatics.

1

The medical data were provided for academic research by the project MEDAN www.medan.de, a cooperation with the Klinikum der J.W. Goethe-Universität Frankfurt am Main, supported by the Deutsche Forschungsgemeinschaft (DFG).

View full text