Abstract
An algorithm for fast minimum search is proposed, which achieves very satisfying performance harmonising the Vogl's and the Conjugate Gradient algorithms. Such effectiveness is achieved by making adaptive, in a very simple and satisfactory way, both the learning rate and the momentum term, and by executing controls and corrections both on the possible cost function increase and on moves opposite to the direction of the negative of the gradient. Thanks to these improvements, we can obtain a good scaling relationship in learning. As regards the real world context, a musical application showed favourable results: besides the good convergence speed, a high generalisation capability has been achieved, as confirmed both by subjective musical evaluations and by objective tests.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Rumelhart DE, McClelland JL. (Eds) Parallel Distributed Processing, MIT Press, Cambridge MA
Rumelhart ER, Hinton GE, Williams RJ. Learning representations by back-propagating errors, Nature 1986;323: 533–536
Le Cun Y. A learning procedure for asymmetric network. In: Proceedings of Cognitiva, Paris, France, 1985; 599–604
Parker DB. Learning-logic. Invention report, S81-64, file 1, Stanford University: Office of Technology Licensing
Werbos PJ. Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences, Master thesis, Harvard University, 1974.
Hecht-Nielsen R. Theory of the backpropagation neural network. In: Proceedings of the International Joint Conference on Neural Networks, San Diego, 1989, 593–605
Mingolla E, Bullock D. Neurocomputing: foundation of research (book review), Neural Networks 1989;2: 405–409
Fletcher R, Reeves CM, Function minimization by conjugate gradients. Computer J 1964;7: 149–154
Bazaraa MS, Shetty CM. Non Linear Programming, Wiley, New York: 1979
Luenberger DG. Introduction to Linear and Nonlinear Programming, Second Edition, Addison-Wesley, Reading, MA, 1984
Battiti R. Accelerated backpropagation learning: two optimization methods. Complex System 1989;3: 331–342
Drago GP, Ridella S. Statistically controlled activation weight initialization (SCAWI). IEEE Trans Neural Network 1992;3 (4)
Hertz J, Krogh A, Palmer GR. Introduction to the Theory of Neural Computation, Addison-Wesley, Reading, 1991
Kramer AH, Sangiovanni-Vincentelli A. Efficient parallel learning algorithms for neural networks. In: Advances in Neural Information Processing Systems, Morgan Kaufmann, San Mateo CA, 1989; 40–48
Fahlman SE. An empirical study of learning speed in back propagation networks. In: Proceedings of the Connectionist Models Summer School, Carnegie Mellon, CMU-CS-88-162, June 1988
Vogl TPet al. Accelerating the convergence of the back-propatation method. Biol Cybern 1988;59: 257–263
Drago GP, Martini C, Morando M, Ridella S. A neural network for music composition. In: MH Hamza (Ed) Proceedings of the Tenth IASTED International Conference on Applied Informatics, Zurich, 1992; 211–214
Arrigo P, Corana A, Guiliano F, Marconi L, Morando M, Ridella S, Rolando C, Scalia F. Neural networks: computer simulation and biomedical applications. In ER Caianiello (Ed) Proceedings of the Second Italian workshop on Parallel Architectures and Neural Networks, World Scientific, Singapore, 1990; 205–212
Marconi L, Scalia F, Ridella S, Arrigo P. An application of backpropagation to medical diagnosis. In: Proceedings of the International Conference on Neural Networks, Washington, DC 1988; 577
Widrow B, Lehr MA. 30 years of adaptive neural networks: perceptron, madaline and backpropagation. Proc IEEE 1990;78: 1415–1442
Tesauro G, Janssens B. Scaling relationships in back-propagation learning. Complex System 1988;2: 39–44
Corana A, Rolando C, Ridella S. A highly efficient implementation of back-propagation algorithm on SIMD computers. In: JL Delhaye, E Gelembe (Eds) High Performance Computing, North Holland, Amsterdam, 1989; 181–190
Drago GP, Ridella S. A comparison between a SCAWI backpropagation and an improved cascade correlation. In: Proceedings of the VI Italian Workshop on Neural Nets; Wirn Vietri-93, World Scientific, Singapore, 1994; 212–217
Møller MF. A scaled conjugate gradient algorithm for fast supervised learning. Neural Networks 1993;6: 525–533
Johansson EM, Dowla FU, Goodman DM. Backpropagation learning for multi-layer feedforward neural networks using the conjugate gradient method. Int J Neural Syst 1991;2(4): 291–302
Battiti R. Optimization methods for back-propagation: automatic parameter tuning and faster convergence. In: IJNNC-90-Washington, DC, 1990; 593–596
Fahlman SE, Lebiere C. The Cascade-Correlation Learning Architecture. Carnegie Mellon, CMU-CS-90-100, February 1990
Fritzke B. Growing cell structures — a self-organizing network for unsupervised and supervised learning. Neural Networks 1994;7(9): 1441–1460
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Drago, G.P., Morando, M. & Ridella, S. An Adaptive Momentum Back Propagation (AMBP). Neural Comput & Applic 3, 213–221 (1995). https://doi.org/10.1007/BF01414646
Received:
Revised:
Issue Date:
DOI: https://doi.org/10.1007/BF01414646