Accelerating backpropagation through dynamic self-adaptation

doi:10.1016/0893-6080(95)00144-1

Neural Networks

Volume 9, Issue 4, June 1996, Pages 589-601

https://doi.org/10.1016/0893-6080(95)00144-1 Get rights and content

Abstract

Standard backpropagation and many procedures derived from it use the steepest-descent method to minimize a cost function. In this paper, were present a new genetic algorithm, dynamic self-adaptation, to accelerate steepest descent as it is used in iterative procedures. The underlying idea is to take the learning rate of the previous step, to increase and decrease it slightly, to evaluate the cost function for both new values of the learning rate, and to choose the one that gives the lower value of the cost function. In this way, the algorithm adapts itself locally to the cost function landscape. We present a convergence proof, estimate the convergence rate, and test the algorithm on several hard problems. As compared to standard backpropagation, the convergence rate can be improved by several orders of magnitude. Furthermore, dynamic self-adaptation can also be applied to several parameters simultaneously, such as the learning rate and momentum.

References (16)

R.A. Jacobs
Increased rates of convergence through learning rate adaption
Neural Networks
(1988)
A. Rigler et al.
Rescaling of variables in backpropagation learning
Neural Networks
(1991)
A. Van Ooyen et al.
Improving the convergence of the back-propagation algorithm
Neural Networks
(1992)
R. Battiti
Accelerated backpropagation learning: two optimization methods
Complex Systems
(1989)
S. Becker et al.
Improving the convergence of back-propagation learning with second order methods
P.E. Gill et al.
Practical optimization
(1981)
D.R. Hush et al.
Improving the learning rate of backpropagation with the gradient reuse algorithm
A.H. Kramer et al.
Efficient parallel learning algorithms for neural networks

There are more references available in the full text version of this article.

Cited by (71)

Practical options for selecting data-driven or physics-based prognostics algorithms with reviews
2015, Reliability Engineering and System Safety
Citation Excerpt :
For these reasons, there have been many efforts to improve the drawbacks of the BPNN algorithm, such as a dynamic self-adaptation algorithm [55] a simulated annealing algorithm [56] combined genetic and differential evolution algorithm [57], and a technique combining the conjugate gradient optimization algorithm with the BPNN algorithm [58]. There are many ensemble techniques to improve the performance of algorithms [59–64], as well as other efforts in Refs. [40,55,58]. While the aforementioned methods are to find the weight parameters in a deterministic sense, there have been probabilistic approaches based on the Bayesian learning technique [65,66], in which the weight parameters are obtained by using a sampling method.
This paper is to provide practical options for prognostics so that beginners can select appropriate methods for their fields of application. To achieve this goal, several popular algorithms are first reviewed in the data-driven and physics-based prognostics methods. Each algorithm’s attributes and pros and cons are analyzed in terms of model definition, model parameter estimation and ability to handle noise and bias in data. Fatigue crack growth examples are then used to illustrate the characteristics of different algorithms. In order to suggest a suitable algorithm, several studies are made based on the number of data sets, the level of noise and bias, availability of loading and physical models, and complexity of the damage growth behavior. Based on the study, it is concluded that the Gaussian process is easy and fast to implement, but works well only when the covariance function is properly defined. The neural network has the advantage in the case of large noise and complex models but only with many training data sets. The particle filter and Bayesian method are superior to the former methods because they are less affected by noise and model complexity, but work only when physical model and loading conditions are available.
The intensity change of urban development land: Implications for the city master plan of Guangzhou, China
2014, Land Use Policy
Citation Excerpt :
It is reported that the application requires the forecasting result that the model supplies (Jiao et al., 2009). The Back Propagation (BP) neural network is one of the most widely used neural network models (Salomon and Hemmen, 1996). A BP neural network is a feed-forward multilayer neural network, which trains the weight of the differentiable nonlinear functions based on an error BP (Dong, 2005).
The Chinese economy has experienced substantial development over the last thirty years. This growth has resulted in an enormous expansion of China's urban areas and it has also led to a growing scarcity of land resources. Consequently, there is currently an urgent need to mitigate the conflict between the needs of growing urban areas and the shrinking supply of land resources. Understanding land-use intensity and its changes can provide important information to find mitigating measures for this conflict in the demand for land. Previous studies have found that increasing the utilization efficiency of land resources is one of the most effective ways to resolve this issue. This study focuses on the city of Guangzhou, which is an intensively developed megalopolis. A number of different data sources have been analyzed to find the characteristics and changes in urban land use in Guangzhou, including: Landsat data from 1979, 1990, 2000, and 2009; relative socioeconomic data from the Guangzhou statistic yearbooks; and the master plans of Guangzhou. The results indicate that the area of developed land has continued to increase. Urban development land intensity and its change were then explored using a Back Propagation neural network model of the city. Although the analysis revealed that urban development land did not have a higher intensity overall, it also showed that there was a sectional upward trend throughout the study period. Consequently, there is a potential to improve the land-use intensity of Guangzhou. A linear regression model was then adopted to explore the mechanism of land use change. The results reveal that rapid industrialization and urbanization have improved land-use intensity in Guangzhou. The per capita urban road area and per capita GDP show a strong relationship when compared with land-use intensity. Using the relative city master plans, the authors have argued that local government in Guangzhou has taken active steps to address land issues to promote the city's socioeconomic development; however, a scientific city master plan in the study area is still necessary to ensure the effective utilization of the city's limited land resources.
Suitability evaluation of urban construction land based on geo-environmental factors of Hangzhou, China
2011, Computers and Geosciences
Citation Excerpt :
Neurons at the same layer are not connected with each other, but neurons at adjacent layers are connected with the weight. The multilayer back-propagation algorithm, namely the BP algorithm, is most widely used (Rumelhart et al., 1986; Salomon and Hemmen, 1996) for its self-adaptive learning in a given environment. According to the urban geo-environment and construction land characteristics of Hangzhou, the BP neural network model of geo-environmental suitability evaluation of urban construction land was established in this study (Fig. 5).
Suitability evaluation of urban construction land based on geo-environmental factors is the process of determining the fitness of a given tract of land for construction. This process involves a consideration of the geomorphology, geology, engineering geology, geological hazards, and other geological factors and is the basis of urban construction land planning and management. With the support of Geographic Information Systems (GIS), grid analysis, and geo-spatial analysis techniques, four factor groups comprising nine separate subfactors of geo-environmental attributes were selected to be used in the evaluation of the suitability level for construction land in Hangzhou. This was based on K-means clustering and back-propagation (BP) neural network methods due to their advantages in fast computing, unique adaptive capacity, and self-organization. Simultaneously, the evaluation results based on K-means clustering and BP neural network were compared and analyzed, and the accuracy evaluation was set. The results showed that the geo-environmental suitability evaluation results of construction land based on K-means clustering and BP neural network were similar in terms of the distribution and scale of construction land suitability level. At the same time, the results of the two evaluation methods were consistent with the variability in suitability level, engineering geology, and hydrogeology of Hangzhou. The results also showed that the real advantage of the methods proposed in this paper lies in their capacity to streamline the mapping process and to ensure that the results are consistent throughout. The suitability level of the urban construction land based on the geo-environment in Hangzhou was divided into four construction sites: land for building super high-rise and high-rise buildings, land for building multistorey buildings, land for low-rise buildings, and nonbuilding land. The results of the suitability evaluation for each category will provide a scientific basis for decision-making in urban development in Hangzhou.
A Hybrid Neural Network for Predictive Model in A Plastic Injection Molding Process
2022, International Journal of Intelligent Engineering and Systems
A New Derivative-Free Linear Approximation for Solving the Network Water Flow Problem With Convergence Guarantees
2020, Water Resources Research
A new derivative-free linear approximation for solving the network water flow problem with convergence guarantees
2020, arXiv

View all citing articles on Scopus

View full text

Contributed articleAccelerating backpropagation through dynamic self-adaptation

Abstract

Neural Networks

Neural Networks

Neural Networks

Accelerated backpropagation learning: two optimization methods

Complex Systems

Improving the convergence of back-propagation learning with second order methods

Practical optimization

Improving the learning rate of backpropagation with the gradient reuse algorithm

Efficient parallel learning algorithms for neural networks

Contributed article
Accelerating backpropagation through dynamic self-adaptation