Stabilization and speedup of convergence in training feedforward neural networks - ScienceDirect

Neurocomputing

Volume 10, Issue 1, January 1996, Pages 7-31

Neurocomputing

https://doi.org/10.1016/0925-2312(94)00026-3 Get rights and content

Abstract

We review the training problem for feedforward neural networks and discuss various techniques for accelerating and stabilizing the convergence during training. Among other techniques, these include a self-adjusting step gain, bipolar sigmoid activation functions, training on all classes in parallel, adjusting the exponential rates in the sigmoids, bounding the sigmoid derivatives away from zero, training on exemplars to which noise has been added, adjusting the initial weight set to a subdomain of low values of the sum-squared error, and adjusting the momentum coefficient over the iterations. We also examine methods to assure the generalization of the learning, which include the pruning of unimportant weights and adding noise to exemplars for training.

References (33)

G.E. Hinton
Connectionist learning procedures
Artificial Intelligence
(1989)
M. Caudill
Neural network training tips and techniques
IEEE AI Expert
(Jan. 1991)
S.E. Fahlman
An empirical study of learning speed in backpropagation
K.F. Gauss
S.Ye. Gilev et al.
Internal conflicts in neural networks
A.N. Gorban
R. Hecht-Nielsen
K. Hornik et al.
Multilayer feedforward networks are universal approximators
Neural Networks
(1988)
B. Kosko
Y. Le Cun et al.
Optimal brain damage

Y. Lee et al.

The effect of initial weights on premature saturation in backpropagation learning

A.M. Legendre

Methodes des mondres quarres, pour trouver le milieu le plus probable entre les resultats de differences observations

Mem. Inst. France

(1810)

G. Li et al.

Acceleration of backpropagation through initial weight pre-training with delta rule

P. Linz

C.G. Looney

Stabilization and speedup of convergence in training feedforward neural networks

K. Matsuoka

Noise injection into inputs in backpropagation

IEEE Trans. Systems Man Cybernet.

(1992)

Cited by (28)

TransMorph: Transformer for unsupervised medical image registration
2022, Medical Image Analysis
In the last decade, convolutional neural networks (ConvNets) have been a major focus of research in medical image analysis. However, the performances of ConvNets may be limited by a lack of explicit consideration of the long-range spatial relationships in an image. Recently, Vision Transformer architectures have been proposed to address the shortcomings of ConvNets and have produced state-of-the-art performances in many medical imaging applications. Transformers may be a strong candidate for image registration because their substantially larger receptive field enables a more precise comprehension of the spatial correspondence between moving and fixed images. Here, we present TransMorph, a hybrid Transformer-ConvNet model for volumetric medical image registration. This paper also presents diffeomorphic and Bayesian variants of TransMorph: the diffeomorphic variants ensure the topology-preserving deformations, and the Bayesian variant produces a well-calibrated registration uncertainty estimate. We extensively validated the proposed models using 3D medical images from three applications: inter-patient and atlas-to-patient brain MRI registration and phantom-to-CT registration. The proposed models are evaluated in comparison to a variety of existing registration methods and Transformer architectures. Qualitative and quantitative results demonstrate that the proposed Transformer-based model leads to a substantial performance improvement over the baseline methods, confirming the effectiveness of Transformers for medical image registration.
Neural network classifier optimization using Differential Evolution with Global Information and Back Propagation algorithm for clinical datasets
2016, Applied Soft Computing Journal
Citation Excerpt :
The training outputs of the NN are entirely dependent on the initial weights [5–8]. The local search with faster convergence of ANN for classification has been improved by various researchers [9–11]. Particle Swarm Optimization (PSO) developed by Kennedy and Eberhart [12,13] can be applied to overcome the local minima problem occurring in any optimization problems.
A Computer-Aided Diagnostic (CAD) system that uses Artificial Neural Network (ANN) trained by drawing in the relative advantages of Differential Evolution (DE), Particle Swarm Optimization (PSO) and gradient descent based backpropagation (BP) for classifying clinical datasets is proposed. The DE algorithm with a modified best mutation operation is used to enhance the search exploration of PSO. The ANN is trained using PSO and the global best value obtained is used as a seed by the BP. Local search is performed using BP, in which the weights of the Neural Network (NN) are adjusted to obtain an optimal set of NN weights. Three benchmark clinical datasets namely, Pima Indian Diabetes, Wisconsin Breast Cancer and Cleveland Heart Disease, obtained from the University of California Irvine (UCI) machine learning repository have been used. The performance of the trained neural network classifier proposed in this work is compared with the existing gradient descent backpropagation, differential evolution with backpropagation and particle swarm optimization with gradient descent backpropagation algorithms. The experimental results show that DEGI-BP provides 85.71% accuracy for diabetes, 98.52% for breast cancer and 86.66% for heart disease datasets. This CAD system can be used by junior clinicians as an aid for medical decision support.
Hourly solar radiation forecasting using optimal coefficient 2-D linear filters and feed-forward neural networks
2008, Solar Energy
In this work, the hourly solar radiation data collected during the period August 1, 2005–July 30, 2006 from the solar observation station in Iki Eylul campus area of Eskisehir region are studied. A two-dimensional (2-D) representation model of the hourly solar radiation data is proposed. The model provides a unique and compact visualization of the data for inspection, and enables accurate forecasting using image processing methods. Using the hourly solar radiation data mentioned above, the image model is formed in raster scan form with rows and columns corresponding to days and hours, respectively. Logically, the between-day correlations along the same hour segment provide the vertical correlations of the image, which is not available in the regular 1-D representation. To test the forecasting efficiency of the model, nine different linear filters with various filter-tap configurations are optimized and tested. The results provide the necessary correlation model and prediction directions for obtaining the optimum prediction template for forecasting. Next, the 2-D forecasting performance is tested through feed-forward neural networks (NN) using the same data. The optimal linear filters and NN models are compared in the sense of root mean square error (RMSE). It is observed that the 2-D model has pronounced advantages over the 1-D representation for both linear and NN prediction methods. Due to the capability of depicting the nonlinear behavior of the input data, the NN models are found to achieve better forecasting results than linear prediction filters in both 1-D and 2-D.
Stability analysis of a three-term backpropagation algorithm
2005, Neural Networks
Efficient learning by the backpropagation (BP) algorithm is required for many practical applications. The BP algorithm calculates the weight changes of artificial neural networks, and a common approach is to use a two-term algorithm consisting of a learning rate (LR) and a momentum factor (MF). The major drawbacks of the two-term BP learning algorithm are the problems of local minima and slow convergence speeds, which limit the scope for real-time applications. Recently the addition of an extra term, called a proportional factor (PF), to the two-term BP algorithm was proposed. The third increases the speed of the BP algorithm. However, the PF term also reduces the convergence of the BP algorithm, and criteria for evaluating convergence are required to facilitate the application of the three terms BP algorithm. This paper analyzes the convergence of the new three-term backpropagation algorithm. If the learning parameters of the three-term BP algorithm satisfy the conditions given in this paper, then it is guaranteed that the system is stable and will converge to a local minimum. It is proved that if at least one of the eigenvalues of matrix F (compose of the Hessian of the cost function and the system Jacobian of the error vector at each iteration) is negative, then the system becomes unstable. Also the paper shows that all the local minima of the three-term BP algorithm cost function are stable. The relationship between the learning parameters are established in this paper such that the stability conditions are met.
Artificial neural network classification based on high-performance liquid chromatography of urinary and serum nucleosides for the clinical diagnosis of cancer
2002, Journal of Chromatography B: Analytical Technologies in the Biomedical and Life Sciences
Nucleosides in human urine and serum have frequently been studied as a possible biomedical marker for cancer, acquired immune deficiency syndrome (AIDS) and the whole-body turnover of RNAs. Fifteen normal and modified nucleosides were determined in 69 urine and 42 serum samples using high-performance liquid chromatography (HPLC). Artificial neural networks have been used as a powerful pattern recognition tool to distinguish cancer patients from healthy persons. The recognition rate for the training set reached 100%. In the validating set, 95.8 and 92.9% of people were correctly classified into cancer patients and healthy persons when urine and serum were used as the sample for measuring the nucleosides. The results show that the artificial neural network technique is better than principal component analysis for the classification of healthy persons and cancer patients based on nucleoside data.
Radial basis functional link nets and fuzzy reasoning
2002, Neurocomputing
Citation Excerpt :
Table 4 shows the results of training on the same data with an MLP. We used the more efficient fullpropagation [20,21] mode rather than the epochal mode of backpropagation. The en route technique adjusted the learning rates η1 and η2 for the training of weights at the hidden and output neurodes, respectively.
We modify the architecture of radial basis function neural networks so as to also model linear as well as the usual nonlinear input–output relationships. The resulting network learns with fewer iterations and is more accurate than radial basis function neural networks or multiple layered perceptrons. Two training algorithms are presented for the new network: quick training and full training. The full training algorithm adjusts more parameters and requires more computation, but quick training is simpler and faster while also being very accurate. These networks, like radial basis function networks, are at least as powerful as the Takagi–Sugeno type of fuzzy rule-based systems. We compare their training results with those of multiple layered perceptrons and radial basis function neural networks on three data sets to show the advantage of our architecture and algorithm.

View all citing articles on Scopus

Copyright © 1996 Published by Elsevier B.V.

Swarm intelligence based MSMOPSO for optimization of resource provisioning in Internet of Things
Recent Trends in Computational Intelligence Enabled Research, 2021, pp. 61-82
Daneshwari I. Hatti, Ashok V. Sutagundar
A novel robust hybrid gravitational search algorithm for reusable launch vehicle approach and landing trajectory optimization
Neurocomputing, Volume 162, 2015, pp. 116-127
Zikang Su, Honglun Wang
Transit stop inspection and maintenance scheduling: A GPU accelerated metaheuristics approach
Transportation Research Part C: Emerging Technologies, Volume 55, 2015, pp. 246-260
Nikolaos Ath. Kallioras, …, Nikos D. Lagaros
Coping with the stochasticity of collision cascades in Molecular Dynamics simulations
Nuclear Instruments and Methods in Physics Research Section B: Beam Interactions with Materials and Atoms, Volumes 500–501, 2021, pp. 1-9
Thomas Jarrin, …, Anne Hémeryck
Molecular dynamics modeling of helium bubbles in austenitic steels
Nuclear Instruments and Methods in Physics Research Section B: Beam Interactions with Materials and Atoms, Volume 425, 2018, pp. 50-54
A. Jelea
Enhancing Decision Tree Classification Accuracy through Genetically Programmed Attributes for Wart Treatment Method Identification
Procedia Computer Science, Volume 132, 2018, pp. 1685-1694
Sabita Khatri, …, Anil Kumar