Abstract
The value of the output function gradient of a neural network, calculated in the training points, plays an essential role for its generalization capability.
In this paper a feed forward neural architecture (αNet) that can learn the activation function of its hidden units during the training phase is presented. The automatic learning is obtained through the joint use of the Hermite regression formula and the CGD optimization algorithm with the Powell restart conditions. This technique leads to a smooth output function of αNet in the nearby of the training points, achieving an improvement of the generalization capability and the flexibility of the neural architecture.
Experimental results, obtained comparing α Net with traditional architectures with sigmoidal or sinusoidal activation functions, show that the former is very flexible and has good approximation and classification capabilities.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Gioiello G.A.M., Tarantino A., Sorbello F., Vassallo G.: Simple Techniques for an Efficient Recognition of Handwritten Characters Using a MLP. The Journal of Intelligent Systems. Vol 6 No 3/4 (1997) 199–221
Pilato G., Sorbello F., Vassallo G.: An Innovative Way to Measure the Quality of a Neural Network without the Use of the Test Set. Proc. of IEEE Int. Workshop on Intelligent Signal Processing, Budapest (Hungary) 4–7 Sept. 1999 pp. 278–282
Pilato G., Sorbello F., Vassallo G.: Using the Hermite Regression Algorithm to Improve the Generalization Capability of a Neural Network. XI Italian Workshop on Neural Nets WIRN’99, Vietri sul Mare (Italy) 20–22 May 1999 (in press)
Simard P., Le Cun Y. Denker J. Victorri B.: An Efficient Algorithm for Learning Invariances in Adaptative Classifiers. Proc of 11th IAPR International Conference on Pattern Recognition, (1994) 651–655
Chen T., Chen H., Liu R.: Approximation Capability in C(Rn) by Multilayer Feed-forward Networks and Related Problems. IEEE Trans. Neural Networks, vol 6 No 1, (1995) 25–30
Russell R., Robert J., Marks I., Seho O.: Similarities of Error Regularization, Sigmoid Gain Scaling, Target Smoothing, and Training with Jitter. IEEE Trans. on Neural Networks Vol.6 No. 3, (1995) 529–538
Hwang J. N., Lay S. R., Meachler M., Martin D., Schimert J.: Regression Modeling in Back-Propagation and Projection Pursuit Learning. IEEE Trans. Neural Networks, Vol 5 No 3, (1994) 342–353
Hwang J. N., You S. S., Lay S. R., Jou I. C.: What’s Wrong with a Cascaded Correlation Learning Network: A Projection Pursuit Learning Perspective. ftp://ftp.cis.ohio.state.edu/pub/neuroprose/
Friedman J. H., Stuetzle W.: Projection Pursuit Regression. Journal of the American Statistical Association. Vol.76 No.376 (1981) 817–823
Powell M. J. D.: Restart Procedures for the Conjugate Gradient Method. Mathematical Programming Vol 12 (1977) 241–254
Friedman J.: Multivariate Adaptative Regression Splines. Technical Report No. 102, Laboratory for Computational Statistics, Department of Statistics, Stanford University, Nov. 1988
Minor J. M.: Parity With Two Layer Feed-Forward Nets. Neural Networks Vol.6 No.5 (1993) 705–707
Fahlman S.E., Lebiere C.: The Cascade-Correlation Learning Architecture Advances in Neural Information Processing Systems-D.S. Touretzky ed, San Mateo Calif: Morgan Kaufmann. Vol.2 (1990) 524–532
Hornik K., Multilayer Feed-forward Networks Are Universal Approximators, Neural Networks, Vol. 2, (1989) 359–366
Zhang M., Suen C.Y., Bui T.D., An Optimal Pairing Scheme in Associative Memory Classifier and its Application in Character Recognition, Proc. Of 11th IAPR Int. Conf. On Pattern Recognition (1994) 50–53
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2000 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Gaglio, S., Pilato, G., Sorbello, F., Vassallo, G. (2000). Using the Hermite Regression Formula to Design a Neural Architecture with Automatic Learning of the “Hidden” Activation Functions. In: Lamma, E., Mello, P. (eds) AI*IA 99: Advances in Artificial Intelligence. AI*IA 1999. Lecture Notes in Computer Science(), vol 1792. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-46238-4_20
Download citation
DOI: https://doi.org/10.1007/3-540-46238-4_20
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-67350-7
Online ISBN: 978-3-540-46238-5
eBook Packages: Springer Book Archive