Abstract
This paper presents a novel Heuristic Global Learning (HER-GBL) algorithm for multilayer neural networks. The algorithm is based upon the least squares method to maintain the fast convergence speed, and the penalized optimization to solve the problem of local minima. The penalty term, defined as a Gaussian-type function of the weight, is to provide an uphill force to escape from local minima. As a result, the training performance is dramatically improved. The proposed HER-GBL algorithm yields excellent results in terms of convergence speed, avoidance of local minima and quality of solution.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
E. Aarts and J. Korst, Simulated Annealing and Boltzmann Machines: A Stochastic Approach to Combinatorial Optimization and Neural Computing, John Wiley & Sons, 1989.
R. Battiti and G. Tecchiolli, “Training neural nets with the reactive tabu search, ” IEEE Transactions on Neural Networks, Vol. 6, No. 5, pp. 1185–1200, September 1995.
N. Dodd, “Optimization of network structure using genetic techniques”, Int. Joint. Conf. on Neural Networks, San Diego, 1990.
S.E. Fahlman and C. Lebiere, “The cascade-correlation learning architecture”, Technical Report, CMU-CS-90-100, School of Computer Science, Carnegie Mellon University, 1990.
M. Gori and A. Tesi, “On the problem of local minima in backpropagation, ” IEEE Tran. Pattern Ana. Machine Intell., Vol. 14, No. 1, pp. 76–86, Jan. 1992.
M. Gori and A. Tesi, “Some examples of local minima during learning with back-propagation, ” Parallel Architectures and Neural Networks, Vietri sul Mare(IT), May 1990.
R. Horst and P.M. Pardalos, Handbooks of Global Optimization, Kluwer Academic Publishers, The Netherlands, 1995.
J. Hwang, S. Lay, M. Maechler, R.D. Martin and J. Schimert, “Regression modeling in backpropagation and projection pursuit learning”, IEEE Trans. on Neural Networks, Vol. 5, pp. 1–24, May 1994.
C. Jones and C. Tsang, “On the convergence of feedforward neural networks incorporating terminal attractors, ” IEEE International Conference on Neural Networks, Vol. 2, pp. 929–935, San Francisco, CA, March-April 1993.
K.J. Lang and M.J. Witbrock, “Learning to tell two spiral apart”, in Proc. of the 1988 Connectionist Models Summer School, Morgan Kaufmann, 1988.
J.R. McDonnell and D. Waagen, “Determining neural networks connectivity using evolutionary programming”, Twenty-sixth Asilormar Conf. on Signals, Systems, and Computers, Monterey, CA, 1992.
S.S. Rao, Optimization theory and applications (2nd edn), JohnWiley& Sons, New York, 1984.
Y. Shang and B.W. Wah, “Global optimization for neural network training, ” IEEE Computer, Vol. 29, No. 3, pp. 45–54, March 1996.
A. Torn and A. Zillinskas, Global Optimization, Springler-Verlag, 1987.
Q. Zheng and D. Zhuang, “Integral global optimization of constrained problems in functional spaces with discontinuous penalty functions, ” in C.A. Floudas and P.M. Pardalos, Recent Advances in Global Optimization, Princeton University Press, pp. 298–320, 1992.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Cho, Sy., Chow, T.W. A Fast Heuristic Global Learning Algorithm for Multilayer Neural Networks. Neural Processing Letters 9, 177–187 (1999). https://doi.org/10.1023/A:1018685627113
Issue Date:
DOI: https://doi.org/10.1023/A:1018685627113