Abstract
In this article, we present a simple, effective method to learning for an MLP that is based on approximating the Hessian using only local information, specifically, the correlations of output activations from previous layers of hidden neurons. This approach of training the hidden layer weights with the Hessian approximation combined with the training of the final output layer of weights using the pseudoinverse [1] yields improved performance at a fraction of the computational and structural complexity of conventional learning algorithms.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Huang, G., Zhu, Q., Siew, C.: Extreme Learning Machine: A New Learning Scheme of Feedforward Neural Networks. In: Proceedings of the International Joint Conference on Neural Networks (IJCNN). IEEE, Los Alamitos (2004)
Bishop, C.: Exact Calculation of the Hessian Matrix for the Multi-Layer Perceptron. Neural Computation 4(4), 494–501 (1992)
Buntine, W., Weigend, A.: Computing Second Derivatives in Feed-Forward Networks: A Review. IEEE Trans. Neural Networks 5(3), 480–488 (1994)
Press, W., Flannery, B., Teukolsky, S., Vetterling, W.: Numerical Recipes in C Example Book: The Art of Scientific Computing, 2nd edn. Cambridge University Press, Cambridge (1994)
Scalero, R., Tepedelenlioglu, N.: A Fast New Algorithm for Training Feedforward Neural Networks. IEEE Trans. Signal Processing 40(1) (1992)
Parisi, R., Claudio, E.D., Orlandi, G., Rao, B.: A Generalized Learning Paradigm Exploiting the Structure of Feedforward Neural Networks. IEEE Trans. Neural Networks 7(6), 1450–1460 (1996)
Bartlett, P.: The Sample Complexity of Pattern Classication with Neural Networks: The Size of the Weights is More Important than the Size of the Network. IEEE Trans. Information Theory 44(2), 525–536 (1998)
Cover, T.: Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition. IEEE Trans. Electronic. Comput. 14, 326–334 (1965)
Lowe, D.: Adaptive Radial Basis Function Nonlinearities, and the Problem of Generalization. In: 1st IEE International Conference on Artificial Neural Networks, pp. 171–175 (1989)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Teoh, E.J., Xiang, C., Tan, K.C. (2006). A Fast Learning Algorithm Based on Layered Hessian Approximations and the Pseudoinverse. In: Wang, J., Yi, Z., Zurada, J.M., Lu, BL., Yin, H. (eds) Advances in Neural Networks - ISNN 2006. ISNN 2006. Lecture Notes in Computer Science, vol 3971. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11759966_79
Download citation
DOI: https://doi.org/10.1007/11759966_79
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-34439-1
Online ISBN: 978-3-540-34440-7
eBook Packages: Computer ScienceComputer Science (R0)