Abstract
Most of the gradient based optimisation algorithms employed during training process of back propagation networks use negative gradient of error as a gradient based search direction. A novel approach is presented in this paper for improving the training efficiency of back propagation neural network algorithms by adaptively modifying this gradient based search direction. The proposed algorithm uses the value of gain parameter in the activation function to modify the gradient based search direction. It has been shown that this modification can significantly enhance the computational efficiency of training process. The proposed algorithm is generic and can be implemented in almost all gradient based optimisation processes. The robustness of the proposed algorithm is shown by comparing convergence rates for gradient descent, conjugate gradient and quasi- Newton methods on many benchmark examples.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
A. van Ooyen and B. Nienhuis, Improving the convergence of the back-propagation algorithm. Neural Networks, 1992. 5: p. 465–471.
M. Ahmad and F.M.A. Salam, Supervised learning using the cauchy energy function. International Conference on Fuzzy Logic and Neural Networks, 1992.
Pravin Chandra and Yogesh Singh, An activation function adapting training algorithm for sigmoidal feedforward networks. Neurocomputing, 2004. 61: p. 429–437.
Krzyzak A., Dai W., and Suen C. Y., Classification of large set of handwritten characters using modified back propagation model. Proceedings of the International Joint Conference on Neural Networks, 1990. 3: p. 225–232.
Sang Hoon Oh, Improving the Error Backpropagation Algorithm with a Modified Error Function. IEEE TRANSACTIONS ON NEURAL NETWORKS, 1997. 8(3): p. 799–803.
Hahn-Ming Lee, Tzong-Ching Huang, and Chih-Ming Chen, Learning Efficiency Improvement of Back Propagation Algorithm by Error Saturation Prevention Method. IJCNN’ 99, 1999. 3: p. 1737–1742.
Sang-Hoon Oh and Youngjik Lee, A Modified Error Function to Improve the Error Back-Propagation Algorithm for Multi-Layer Perceptrons. ETRI Journal, 1995. 17(1): p. 11–22.
S. M. Shamsuddin, M. Darus, and M. N. Sulaiman, Classification of Reduction Invariants with Improved Back Ppropagation. IJMMS, 2002. 30(4): p. 239–247.
S. C. Ng, et al., Fast convergence for back propagation network with magnified gradient function. Proceedings of the International Joint Conference on Neural Networks 2003, 2003. 3: p. 1903–1908.
R.A. Jacobs, Increased rates of convergence through learning rate adaptation. Neural Networks, 1988. 1: p. 295–307.
M.K. Weir, A method for self-determination of adaptive learning rates in back propagation. Neural Networks, 1991. 4: p. 371–379.
X.H. Yu, G.A. Chen, and S.X. Cheng, Acceleration of backpropagation learning using optimized learning rate and momentum. Electronics Letters, 1993. 29(14): p. 1288–1289.
Bishop C. M., Neural Networks for Pattern Recognition. 1995: Oxford University Press.
R. Fletcher and M. J. D. Powell, A rapidly convergent descent method for nlinimization. British Computer J., 1963: p. 163–168.
Fletcher R. and Reeves R. M., Function minimization by conjugate gradients. Comput. J., 1964. 7(2): p. 149–160.
M. R. Hestenes and E. Stiefel, Methods of conjugate gradients for solving linear systems. J. Research NBS, 1952. 49: p.409.
HUANG H.Y., A unified approach to quadratically convergent algorithms for function minimization. J. Optim. Theory Appl., 1970. 5: p. 405–423.
D.E. Rumelhart, G.E. Hinton, and R.J. Williams, Learning internal representations by error propagation, in D.E. Rumelhart and J.L. McClelland (eds), Parallel Distributed Processing, 1986. 1: p. 318–362.
Rumelhart D. E., Hinton G. E., and Williams R. J., Learning internal representations by back-propagation errors. Parallel Distributed Processing, 1986. 1 (Rumelhart D.E. et al. Eds.): p. 318–362.
Adrian J. Sheperd, Second Order Methods for Neural Networks-Fast and Reliable Training Methods for Multi-layer Perceptrons, ed. J.G. Taylor. 1997: Springer. 143.
Byatt D., Coope I. D., and Price C. J., Effect of limited precision on the BFGS quasi-Newton algorithm. ANZIAM J, 2004. 45: p. 283–295.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2007 Springer-Verlag London Limited
About this paper
Cite this paper
Nawi, N.M., Ransing, M.R., Ransing, R.S. (2007). Improving the Gradient Based Search Direction to Enhance Training Efficiency of Back Propagation Based Neural Network Algorithms. In: Bramer, M., Coenen, F., Tuson, A. (eds) Research and Development in Intelligent Systems XXIII. SGAI 2006. Springer, London. https://doi.org/10.1007/978-1-84628-663-6_4
Download citation
DOI: https://doi.org/10.1007/978-1-84628-663-6_4
Publisher Name: Springer, London
Print ISBN: 978-1-84628-662-9
Online ISBN: 978-1-84628-663-6
eBook Packages: Computer ScienceComputer Science (R0)