Abstract
Natural gradient learning algorithm, which originated from information geometry, is known to provide a good solution for the problem of slow learning speed of gradient descent learning methods. Whereas the natural gradient learning algorithm is inspired from the geometric structure of the space of learning systems, there have been other approaches to acceleration of learning by using the second order information of error surface. Although the second order methods cannot give as successful solutions as the natural gradient learning method, their results showed the usefulness of the second order information of error surface in the learning process. In this paper, we develop a method of combining these two different approaches to propose a more efficient learning algorithm. At each learning step, we calculate a search direction by means of the natural gradient. When we apply the search direction to parameter-updating process, the second order information of error surface is applied to determine an efficient learning rate. Through a simple experiment on a real world problem, we confirmed that the proposed learning algorithm show faster convergence than the pure natural gradient learning algorithm.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
S. Amari, Natural Gradient Works Efficiently in Learning, Neural Computation, 10, 251–276, 1998.
S. Amari and H. Nagaoka, Information Geometry, AMS and Oxford University Press, 1999.
Amari, S., Park, H., and Fukumizu, F., Adaptive method of realizing natural gradient learning for multilayer perceptrons, Neural Computation, 12, xx–xx, 2000.
D. P. Bertsekas, Nonlinear Programming, Athena Scientific, Belmout, Massachusetts, 1995.
C. Bishop, Neural Networks for Pattern Recognition, Oxford University Press, 1995.
Y. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller, Neural Networks: Tricks of the Trade, ed. G. B. Orr and K. R.Müller, (pp. 5–50), Springer Lecture Notes in Computer Sciences, 1524, Springer Heidelberg, 1998
H. Park, Efficient On-line Learning Algorithms Based on Information Geometry for Stochastic Neural Networks, Ph.D. Thesis, Dept. of CS, Yonsei University, 2000.
M. Rattray and D. Saad, Transient and Asymptotics of Natural Gradient Learning, Proceedings of the 8th International Conference on Artificial Neural Networks, 165–170, 1998.
M. Rattray, D. Saad, and S. Amari, Natural Gradient Descent for On-line Learning, Physical Review Letters, 81, 5461–5464, 1998.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2000 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Park, H., Fukumizu, K., Amari, Si., Lee, Y. (2000). An Efficient Learning Algorithm Using Natural Gradient and Second Order Information of Error Surface. In: Mizoguchi, R., Slaney, J. (eds) PRICAI 2000 Topics in Artificial Intelligence. PRICAI 2000. Lecture Notes in Computer Science(), vol 1886. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-44533-1_23
Download citation
DOI: https://doi.org/10.1007/3-540-44533-1_23
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-67925-7
Online ISBN: 978-3-540-44533-3
eBook Packages: Springer Book Archive